Simultaneously segmenting and labeling images is a fundamental problem in Computer
Vision. In this paper, we introduce a hierarchical CRF model to deal with the
problem of labeling images of street scenes by several distinctive object classes.
In addition to learning a CRF model from all the labeled images, we group images
into clusters of similar images and learn a CRF model from each cluster separately.
When labeling a new image, we pick the closest cluster and use the associated CRF
model to label this image. Experimental results show that this hierarchical image
labeling method is comparable to, and in many cases superior to, previous methods
on benchmark data sets. In addition to segmentation and labeling results, we also
showed how to apply the image labeling result to rerank Google similar images.