Efficient semantic image segmentation with multi-class ranking prior

Computer Vision and Image Understanding 120 (2014) 81–90

Contents lists available at ScienceDirect

Computer Vision and Image Understanding journal homepage: www.elsevier.com/locate/cviu

Efficient semantic image segmentation with multi-class ranking prior q Deli Pei a,b,c,d, Zhenguo Li e, Rongrong Ji f,⇑, Fuchun Sun b,c,d,⇑ a

Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China c State Key Laboratory of Intelligent Technology and Systems, Beijing 100084, China d Tsinghua National Laboratory for Information Science and Technology, Beijing 100084, China e Huawei Noah’s Ark Lab, Hong Kong, China f Department of Cognitive Science, Xiamen University, Xiamen 361005, China b

a r t i c l e

i n f o

Article history: Received 3 October 2012 Accepted 6 October 2013 Available online 25 October 2013 Keywords: Computer vision Machine learning Semantic segmentation Structural SVMs

a b s t r a c t Semantic image segmentation is of fundamental importance in a wide variety of computer vision tasks, such as scene understanding, robot navigation and image retrieval, which aims to simultaneously decompose an image into semantically consistent regions. Most of existing works addressed it as structured prediction problem by combining contextual information with low-level cues based on conditional random fields (CRFs), which are often learned by heuristic search based on maximum likelihood estimation. In this paper, we use maximum margin based structural support vector machine (S-SVM) model to combine multiple levels of cues to attenuate the ambiguity of appearance similarity and propose a novel multiclass ranking based global constraint to confine the object classes to be considered when labeling regions within an image. Compared with existing global cues, our method is more balanced between expressive power for heterogeneous regions and the efficiency of searching exponential space of possible label combinations. We then introduce inter-class co-occurrence statistics as pairwise constraints and combine them with the prediction from local and global cues based on S-SVMs framework. This enables the joint inference of labeling within an image for better consistency. We evaluate our algorithm on two challenging datasets which are widely used for semantic segmentation evaluation: MSRC-21 dataset and Stanford Background dataset and experimental results show that we obtain high competitive performance compared with state-of-the-art methods, despite that our model is much simpler and efficient. Ó 2013 Elsevier Inc. All rights reserved.

1. Introduction Semantic segmentation is a fundamental but challenging problem in computer vision, which aims to assign each pixel in an image a pre-defined semantic label. It can be seen as an extension of the traditional object detection which aims at detecting prominent objects in the foreground of an image, with closed relation to some other fundamental computer vision tasks such as image segmentation and image classification. Semantic segmentation has many applications in practice, including scene understanding, robot navigation, and image retrieval. Semantic image segmentation algorithms in early stage typically solve this problem from a pixel-wise labeling perspective [1,2]. Although using pixels as labeling units is simple and straight-

q

This paper has been recommended for acceptance by Nicu Sebe.

⇑ Corresponding authors. Addresses: Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China (F. Sun). Department of Cognitive Science, School of Information Science and Technology, Xiamen University, Xiamen 361005, China (R. Ji). E-mail addresses: [email protected] (D. Pei), [email protected] (Z. Li), [email protected] (R. Ji), [email protected] (F. Sun). 1077-3142/$ - see front matter Ó 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.cviu.2013.10.005

forward, pixel itself contains limited and ambiguous information that cannot always be discriminative enough to determine its correct label. On the other hand, the proliferation of unsupervised image segmentation algorithms, such as mean shift [3], graph based segmentation [4,38], quick shift [5], TurboPixel [6] and SLIC [7], enables higher order features representation of regions. Therefore, more recently semantic segmentation approaches based on region-wise labeling [8–13] are also well investigated, which make use of region-level features that are not only more informative but also robust to noise, clutter, illuminate variance et al. In such a setting, an initial unsupervised segmentation is commonly adopted for pre-processing. However, image segmentation is still far away from being perfect without regard to the extensive attempts in the last several decades. From this point of view, how to make best use of these imperfect unsupervised image segmentation algorithms on the semantic segmentation problem is of fundamental importance yet is still unclear. Although higher order features extracted from regions are more expressive and informative than those from pixels, sematic ambiguity still exists because of the appearance similarity. A general consent is that contextual information within an image is a very useful cue to attenuate this ambiguity, which can be used to

82

D. Pei et al. / Computer Vision and Image Understanding 120 (2014) 81–90

suppress/encourage the presence of object classes during labeling. Context refers to any information that is not extracted directly from local appearance and can be summarized into two categories: pairwise constraints and global cues. Pairwise constraints, such as smoothness based on contrast [14,9], relative location [10,11] and co-occurrence [8,11,15] are used to model the pairwise relationship between regions within an image. Global constraints are usually used to enforce higher level consistency of region sets or image level. Some approaches are proposed to model these cues, such as using image classification results [13], Potts potential [12], pN Potts potential [16] and its improved versions robust PN potential [14], pN-based hierarchical CRFs [17], and Harmony potential [9]. These models will be further discussed in Section 2. In terms of the methodology, most of the existing methods [16,12,14,10,11,15,9] use conditional random fields (CRFs) to combine these constraints from different levels and make joint inference of labeling within an image, which is also known as structured prediction. In contrast to many sophisticated algorithms for inference, these models [10,11,9,14,15] are usually learned by gradient descent or heuristic search on validation set based on maximum likelihood estimation. On the other hand, Zhu et al. [18] showed that the max-margin based learning algorithm is more robust for structured prediction compared with the maximum likelihood estimation based learning algorithm in many machine learning applications. In this paper, we use maximum margin based structural support vector machine (S-SVMs) model to combine multiple levels of cues to attenuate the ambiguity of appearance similarity and we propose multi-class ranking based global constraints to confine the object classes to be considered when labeling regions within an image. For global cues, we first rank all the object classes for an image (class with higher probability present in the image gets larger score) using multi-class ranking algorithm [20] and transform the ranking scores into image-level soft constraint to confine the possible classes present in the image. The advantages of this global cues can be seen from two aspects: on the one hand, compared with robust pN potential [14] which limits their parent node to take only one single label, our method ranks all the classes for an image and thus is more representative to heterogeneous regions. On the other hand, since we compute the ranking scores for all the classes and transform them to soft constraint, we do not need to make hard decision for every class and thus avoid searching exponential space of possible label combination as harmony potential [9]. The global cues are integrated with the prediction obtained from region feature and logistic regression to encouraging more likely classes while suppressing the others. We then introduce inter-class co-occurrence statistics as pairwise constraints and combine them with the prediction from previous stage under S-SVMs framework. This enables the joint inference of labeling within an image for better consistency. Moreover, our model can be can be efficiently learned with cutting plane algorithm [19] instead of using heuristic search approach as in CRFs learning. Experimental results show that we obtain high competitive performance with state-of-the-art methods with a much simpler and efficient model on two challenging datasets: MSRS21 and Stanford Background Dataset. Probably the most related work is [21], which discussed the application of structural SVM in image semantic segmentation and compared with alternative maximum likelihood method. However, our model is different from their model in designing pairwise and global constraints as well as loss function in parameter learning. The standard contrast-dependent Potts model was used as pairwise constraint in contrast to our co-occurrence property. With regard to global constraints, they used very simple and straightforward K image-level classification results and the

advantage of multiple classes ranking over 1-VS-All classifiers is discussed in [20]. The remainder of the paper is organized as follows: In the next section we review the related work. Our model is presented in Section 3, including the problem formulation and model details. Sections 4 and 5 describe the inference and learning methods. Implementation details and performance evaluation are shown in Section 6 while conclusions are drawn in Section 7.

2. Related work Despite of the success in inferring pixel labels [1,2], more recent methods tend to infer labels over regions or superpixels for the sake of lower computational complexity and incorporating higher level semantic cues. For these approaches, traditional image segmentation algorithms such as Normalized Cut [8], meanshift [14,17,13], graph-based image segmentation [10], quick shift [9,22] are adopted to get initial segments. More recently, several over-segmentation algorithms [6,7] are developed to bypass the problem of tradition segmentation algorithms, such as the semantic ambiguity (regions span multiple object classes) and the difficulty to determine the optimal number of segment regions. These algorithms try to seek the trade-off between reducing image complexity through pixel-grouping and avoiding under-segmentation [6]. Images are decomposed into much smaller regions than object size, e.g. 100–300 regions. Many traditional segmentation algorithms can also be adopted to generate superpixels by setting a finer level region segments. Qualitative results of different segmentation algorithms are given in Fig. 1, where each image is decomposed into approximate 150 superpixels. It can be seen that over-segmentation algorithms tent to segment an image into regions with regions with approximate size while the region size of traditional segmentation may vary a lot with the complexity of the content. Although various powerful features have been proposed recently (e.g. color histogram, texture and SIFT, these feature are still not informative enough to achieve high classification performance because of the appearance similarity. To attenuate this ambiguity of feature representation, some pairwise constraints, such as smoothness [14,9,23], relative location [10,11] and co-occurrence [8,11,15], are further introduced to attenuate the ambiguity of feature representation: (i) The assumption for pairwise smoothing term is that adjacent regions tend to have same label, and subsequently spatially adjacent regions with different labels will be punished. To keep the boundary, appearance contrast is considered in smoothing term, by which regions with larger appearance contrast will be punished less for their inconsistent labels. However, the dilemma of this smoothing term is that regions with similar appearance will naturally tend to have same label. This is contradicted with the objective of smoothing term that expecting spatially adjacent regions with variant appearance to have same label. (ii) The co-occurrence statistics exploit the property that some classes (e.g. boat, water) are more likely to present within an image than others (e.g. car, water). Thus the existence of one class can be used as the evidence of expecting the presence of some highly related classes and suppress the presence of other unlikely classes. For instance, Rabinovich et al. [8,11] construct context matrices by counting the co-occurrence frequency among object labels in the training set to incorporate semantic contextual information. Ladicky et al. [15] claimed that the co-occurrence cost should depend only on the labels present in an image, it should be invariant to the number and location of pixels that object occupies. (iii) Gould et al. [10] encoded the inter-class spatial relationship as a local feature in a two-stage classification process. However, because of the 2D projection, relative location in images is usually uninformative and hence degenerates to co-occurrence constraint.

D. Pei et al. / Computer Vision and Image Understanding 120 (2014) 81–90

83

Fig. 1. Over-segmentation examples of different segmentation algorithms, with approximate 150 superpixels for each image.

Pairwise constraint can only capture local context information between regions. A more recent trend is building a hierarchical model by adding an extra global constraint to pairwise framework to incorporate constraints on higher level, such as the group of segments or image level. Plath et al. [12] proposed a Potts potential to model the label consistency of regions in a hierarchical tree structure, which punished all nodes that have inconsistent labels with their parent label. Kohli et al. [14] adapted the pN Potts potential proposed in [16] to a segment quality sensitive higher order potential, named robust PN potential. The cost of inconsistent labeling in high contrast region will be less compared to low contrast region. However, a drawback of both high order potentials [12,14] is that they both limited their parent node to take only one single label, which is often not the case and makes it unable to handle heterogeneous regions. Csurka and Perronnin [13] proposed to use image classification results to reduce the number of classes to be considered in an image. But this hard constraint schema did not take into account the classification accuracy and classification errors could propagate to following stages and affect the overall performance. The work [17] proposed a novel hierarchical CRF framework which allowed for integration of features computed at different levels to avoid single choice of quantization. Gonfaus et al. [9] proposed more expressive constraint named harmony potential, which restrict the power set over all possible labels on image level first and then use it as a higher order constraint. However, the exponential sized power set makes the exact inference infeasible. And heuristic method such as branch-and-bound sampling has to be applied to get an approximation of the best assignment, which results in taking into account a small subset only. Besides the context cues directly extracted from the image, priors from various vision tasks are also introduced to improve the performance. Several approaches considered jointing the object detection and multi-class image segmentation by feeding information from one task to the other [24–26,15,9]. Heitz et al. [24] developed Cascaded Classification Models (CCM) to combine the subtasks of scene categorization, object detection, multiclass image segmentation for holistic scene understanding. However, since these subtasks were only coupled by their input/output variables in a loose style, each of them is still optimized separately and information sharing was limited and may cause inconsistent representation. Gould et al. [25] proposed a hierarchical region-based approach that combined joint object detection with image segmentation to reason simultaneously about pixels, regions and ob-

jects in an image. Ladicky et al. [15] integrated the results from sliding window detectors with low-level pixel-based unary and pairwise relations into a conditional random field framework (CRF) for joint reasoning about regions, objects and their attributes and similar idea in [9]. 3. Model In this section we will specify our model for the structural prediction. First, we consider superpixels obtained by an unsupervised image segmentation, and use xi, i = 1, . . . , N to denote the feature vector of superpixel i and yi 2 C = {c1, c2, . . . , cK} for its corresponding label where N and K are the numbers of superpixels and classes, respectively. The whole image can then be represented as the collection of superpixel feature vectors, X = {xij i = 1, . . . , N}, and an assignment of labels to the set of superpixels is referred to as a labeling of the image, denoted by Y = {yij i = 1, . . . , N}. Our objective is to learn a function F(X, Y) that is able to capture the compatibility of the prediction Y and the observation X, such that the better the prediction Y describes the image content X, the higher value F(X, Y) becomes. Thus, given the observation X, the optimal prediction Y can be found by maximizing F(X, Y) over all possible labelings:

^ ¼ arg max FðX; YÞ: Y

ð1Þ

Y

Following the structural SVMs [27], we assume the compatibility function F is linear in terms of a combined feature representation of inputs and outputs u(X, Y) (also known as joint feature map):

FðX; YÞ ¼ hx; uðX; YÞi:

ð2Þ

The joint feature map u(X, Y) can be designed in order to capture multi-scale, multi-layer and contextual cues. Given the joint feature map, the task in learning is to train an optimal model parameter x using training set. The local constraint, also called unary potential, captures the local appearance evidence for labeling superpixels; the mid-level constraint usually exploits pairwise relationship, such as smoothness, relative location and co-occurrence, between superpixels. In some approaches, certain global constraint is also applied to infer possible labeling from image level rather than superpixels. We will specify how we define these constraints and combine them together in the following sections.

84

D. Pei et al. / Computer Vision and Image Understanding 120 (2014) 81–90

3.1. The unary potential First we detail our feature representation for superpixels. The raw features of a superpixel consist of two ingredients, appearance-based descriptors and bag-of-word (BoW) representation. Following [10], the appearance-based descriptors include color and texture features which compute mean, standard deviation, skewness, and kurtosis statistics of the superpixel’s color distribution and filter responses. In addition, we also extract the location and geometry features of the superpixel. For more details we refer the reader to [10]. The BoW representation has been shown useful in many state-of-the-art vision systems. Therefore we also incorporate it for superpixel representation. Moreover, as shown in [22,28], BoW features extracted not only inside superpixels, but also in their neighborhood can describe superpixels more effectively. Thus, for each superpixel wee extract BoW from both itself and its adjacent regions and then concatenate them together. The final representation of the raw features becomes:

superpixel casts a vote for its support to all the other superpixels’ class labels given its region size and the confidence of its initial guess, which is defined as follows:

usi ;sj ðyi ; yj Þ ¼

Pðyi jsi ÞSi þ Pðyj jsj ÞSj P ; i Si

where P(yijsi) is the probability of superpixel si taking label yi defined in (4) and Si is the size of the superpixel i. Thus, each superpixel i receives N  1 votes from all the other superpixels for its label assignment yi:

V si ðyi Þ ¼

N X

lyi ;yj usi ;sj ðyi ; yj Þ:

>

ð3Þ

sai

sbi

where is the appearance descriptor, is the concatenated BoW feature, and ha, hb are the weight parameters to be learned by cross validation. Instead of using the above raw feature, we compute an intermediate representation from these raw features via logistic regression, which makes feature more compact. Given the raw feature representation si of a superpixel, the probability of taking label l 2 C = {c1, c2, . . . , cK} can be computed by the following logistic regression model:

Pðljsi Þ ¼

8 expðbl0 þbTl si Þ > > < 1þPcK1 expðbt0 þbT si Þ if l ¼ c1 ; . . . ; cK1 ; t¼c1

t

1 > > : 1þPcK1 expðbt0 þbT si Þ if l ¼ cK ; t¼c1

ð4Þ

t

where b is the learned parameter for the logistic regression. We concatenate class probabilities to form the K-dimensional intermediate representation:

xi ¼ ðPðc1 jsi Þ; Pðc2 jsi Þ; . . . ; PðcK jsi ÞÞ> :

ð5Þ

Moreover, we assign the most probable label to the superpixel as an initial label guess for further joint inference: 

li ¼ arg max Pðljsi Þ:

ð6Þ

l2C

As a baseline, the performances of raw features under various over-segmentation algorithms are systematically evaluated in Section 6 and compared with those obtained from structured prediction using contextual information. The unary potential can be written as follows:

F unary ðX; YÞ ¼

X

xTyi xi ;

ð7Þ

i

where xc1 ; . . . ; xcK 2 RK are the model parameters for the unary potential. 3.2. The pairwise potential The unary potential part computes not only the intermediate representation of superpixels, but also the initial labeling of each superpixel based on local features. However the performance of such labeling may not be satisfactory due to the ambiguity on low level representation. To leverage the semantic context between superpixels and attenuate the ambiguity, we introduce a voting strategy to exploit co-occurrence property of objects within an image. Based on the initial label obtained in Section 3.1, each

ð9Þ

j¼1;j–i

We define the pairwise potential by aggregating votes of all superpixels for their label assignments Y and then we have

F pair ðX; YÞ ¼

X XX V si ðyi Þ ¼ lyi ;yj usi ;sj ðyi ; yj Þ; i

si ¼ ðha sai ; hb sbi Þ ;

ð8Þ

i

ð10Þ

j–i

where lci ;cj are K2 model parameters for pairwise potential, describing the preferences of co-occurrent class pair in the data. 3.3. The global constraint When most of the superpixels are correctly labeled, the cooccurrence property is beneficial to rectify the minor superpixels that are mislabeled. However, as the proportion of mislabeled superpixels increases, it is more likely that the error is propagated to other superpixels due to the voting scheme. To resolve this problem, global constraint on image level is further introduced to confine the possible classes present in an image. However, the existing global consistency potentials are either too simple in expressive power, which only allows regions have a single class label such as Potts [12] and Robust PN-based [14], or too complicated which has to search exponential space of likely combinations of labels such as Harmony potential [9]. We propose a new efficient global constraint which has a better trade-off between expressive power for to heterogeneous regions and the efficiency of searching exponential space of possible label combinations. With the help of multi-class ranking algorithm in [20], we first rank all the object classes from image level and then transform the ranking into soft constraint. To obtain the multi-class ranking score, each image is represented by kernel descriptor [29] and its corresponding binary label 1 K vector li ¼ fli ; . . . ; li g 2 f1; þ1gK , where K is the total number of j object classes, li ¼ þ1 denotes the presence of class j in image Ii j and li ¼ 1 denotes the absence. We aim to learn K classification function ft(Ii), Rd ? R, t 2 C = {c1, c2, . . . , cK}, one for each class, such that for any image I; fci ðIÞ scores higher than fcj ðIÞ when I is more likely belonging to class ci than class cj.1 The ranking score indicates the confidence of assigning specific label to a given image. Although it is informative, the result is still very rough. Therefore, rather than setting a threshold to binary this vector to obtain possible label set as in [13,30], we transform this ranking score into soft constraint using a sigmoid function:

ht ðIÞ ¼

1 þ q; 1 þ a  expðb  f t ðIÞÞ

ð11Þ

where a, b, q are parameters to be learned from a validation set. Now each image Ij can then be represented by a K dimensional vector r j ¼ fhc1 ðIj Þ; . . . ; hcK ðIj Þg. Then this soft constraint is integrated with the unary potential to impose image label prior 1

We use the code available at http://www.cse.msu.edu/bucakser/software.html.

85

D. Pei et al. / Computer Vision and Image Understanding 120 (2014) 81–90

to superpixels within image Ij. Thus the intermediate representation of superpixel defined in (5) can be revised as follows:

e x i ¼ ðhc1 ðIj Þ  Pðc1 j si Þ; hc2 ðIj Þ  Pðc2 j si Þ; . . . ; hcK ðIj Þ  PðcK jsi ÞÞ> : ð12Þ To illustrate the benefit of our proposed soft constraint strategy, we compare it with two alternative global constraint strategies: Top n labels hard constraint and Threshold-t constraint. The Top n constraint selects the most probable n labels for each image according to the rank score vector f computed in the above procedure, other labels are simply discarded. In Threshold-t strategy, instead of selecting a fixed number of labels for each image, we filter out unlikely labels by setting a threshold to the ranking score vector f. We compare with these strategies to show the efficiency of our proposed approach in Section 6.2. The advantages of transforming multi-label ranking to global constraints can be seen from two aspects: on one hand, the multi-label ranking score inferred from image level is more representative to heterogeneous regions by encouraging multiple labels, compared with the robust pN model [14] which limits their parent node to take only one single label. On the other hand, instead of inferring possible label set of an image from exponential sized power set of labels as in [22], which is intractable and can only be solved by sampling strategy, we can directly compute the ranking scores of every label for an image and it can be integrated directly with prediction results obtained from local features and logistic regression. 3.4. Overall compatibility function Combining all the above development together, we propose the compatibility function as follows:

FðX; YÞ ¼ F unary ðX; YÞ þ F pair ðX; YÞ X XX ¼ xTyi ex i þ xyi ;yj usi ;sj ðyi ; yj Þ: i

i

ð13Þ

j–i

The compatibility function combines local and global cues and contextual information in a unified framework and makes joint labeling inference, which efficiently attenuates the ambiguity of local appearance similarity and makes the labeling more consistent. We systematically evaluate our model on two challenging datasets for semantic segmentation and compare with state-of-the-art methods in Section 6. 4. Inference The inference process defined in (1) seeks the most compatible labeling Y for a given observation X. Typically the process of maximizing this compatible function can be formulated as an integer programming problem, which is NP-hard in general except some special cases (e.g. K = 1) and consequently can only be solved approximately. In this paper, we adopt a greedy search algorithm in an iterative style because of its simplicity. First we rewrite the compatibility function as follows:

FðX; YÞ ¼ ¼

X

xTyi xei þ

XX

lyi ;yj usi ;sj ðyi ; yj Þ

i

i

i

j–i

j–i

X X fxTyi xei þ lyi ;yj usi ;sj ðyi ; yj Þg

X ¼ gðyi jxi ; y1 ; . . . ; yi1 ; yiþ1 ; . . . ; yN Þ;

Thus the inference process is described as follows: (1) In each iteration we randomly choose one superpixel and fix all the other superpixels’ labels. (2) We compute the score function g of all K possible classes for this superpixel. (3) If the label with largest score is different with the previous label, then update the label. (4) The iteration stops when no more label changes or reaches max iterations. Like most greedy search algorithm, the initialization is crucial to the performance. For our case, we found that the local prediction obtained from logistic regression serves as a natural and good start prediction. We initialize our prediction with logistic regression results in Eq. (12) instead of random values. The pseudocode of the above operations is given as follows: Algorithm 1. inference algorithm 1: Input: 2: Image feature Ij, superpixels si, i = 1, . . . , N 3: Initialization: ^i ¼ maxyi 2C hyi ðIj Þ  Pðyi jsi Þ 4: y 5: xei ¼ ðhc1 ðIj Þ  Pðc1 jsi Þ; hc2 ðIj Þ  Pðc2 jsi Þ; . . . ; hcK ðIj Þ  PðcK jsi ÞÞ 6: rebeat 7: for all superpixel si do ^1 ; . . . ; y ^i1 ; y ^iþ1 ; . . . ; y ^N Þ 8: yi ¼ maxyi 2C gðyi j xei ; y ^i then 9: if yi –y ^i 10: update yi ! y 11: end if 12: end for 13: until no label changes OR reaches max interactions ^i ji ¼ 1; . . . ; Ng 14: return Y ¼ fy

Because the inference is conducted directly on superpixels instead of pixels, the number of variables is significantly reduced, typically from tens of thousands (e.g. an image of 400  300 pixels) to several hundred(usually 100–300 superpixels per image). Therefore the inference algorithm converges very fast, typically less than 15 iterations.

5. Learning In this section, we discuss how to learn the proposed model 14, i.e., the model parameters x. To find the optimal solution x⁄, we follow the idea in [27] for structured output prediction, and consider the following maximum-margin optimization problem:

min

1 CX kxk2 þ n 2 n i i

s:t: 8i ni P 0 b 2 X n Y; hx; duðX; Y; Y b Þi P Dð Y b ; YÞ  ni ; 8Y

b Þ ¼ uðX; YÞ  uðX; Y b Þ; ni is a slack variable which where duðX; Y; Y becomes non-zero when the margin is violated, Y is the ground truth label of the given image and X is the structured output space. b ; YÞ is the loss function that quantifies how incorrect the predicDð Y b is when Y is the correct output value. tion Y One intuitive form of the loss function is 0–1 loss on each superpixel:

ð14Þ

i

where g() is the potential of superpixel xi being labeled yi while the rests are y1, . . . , yi1, yi+1, . . . , yN.

ð15Þ

b ; YÞ ¼ Dð Y

N X ^i ; yi ÞÞ; ð1  dðy

ð16Þ

i

where d takes 1 when two values are identical and 0 otherwise.

86

D. Pei et al. / Computer Vision and Image Understanding 120 (2014) 81–90

However loss function defined in (16) penalizes incorrect superpixel labeling equally without taking into account the region size. Thus the loss of a large mislabeled superpixel is equal to the loss of a very small one. We then derive a more appropriate loss function as follows:

b ; YÞ ¼ Dð Y

PN i

gSi ð1  dðy^i ; yi ÞÞ P

i Si

;

ð17Þ

where Si is the area of superpixel i and g is a weight factor to be learned from cross validation. Because the structured output space X to be sought grows exponentially with the numbers of superpixels N and object classes K, the number of constraints in (15) is also exponentially large which makes it impossible to optimize directly. Current state-ofthe-art approaches typically use cutting plane algorithm proposed by Joachims et al. [19] and their implementation SVM Struct package.2 For a better efficiency, we follow a variant implementation of the cutting plan algorithm presented in [31]. The learning algorithm aims at finding a small set of constraints that ensures a sufficiently accurate solution. It starts with an unconstrained optimization problem as a relaxation of original problem and maintains a working set Wi. In each iteration through the training process, the ‘‘most violated’’ constraint is selected and then added to the existing working set if certain condition is satisfied. Once a constraint is added, we optimize the problem again to get new solution. Iteration stops when no constraint has changed or objective precision has reached. 6. Experimental results

the superpixels from different over-segmentation methods using Logistic Regression. Particularly, to stress the role of over-segmentation, no pairwise or global contextual information is incorporated. We use MSRC-21 in this experiment. For the feature representation of superpixels, we combine appearance-based and bag-of-word descriptors (see Section 3.1). The appearance-based descriptor has 238 features, consisting of (1) color features computing the mean, standard deviation, skewness, and kurtosis statistics of RGB, Lab, and YCrCb color-space channels and gray image (4  10 dimensions); (2) texture features computing the same statistics of 48 filter responses (4  48 dimensions), including first and second derivatives of Gaussian and Laplacian-of-Gaussian with various orientations and scales; (3) shape features (3 dimensions); and (4) location features (3 dimensions). To build a bag-of-word (BoW) representation, we divide an image into 16  16 pixel cells with 75% overlap. Each cell is captured by a 128-dimensional SIFT descriptor. The dictionary size is 400 visual words built with K-means clustering and these descriptors are then quantized using nearest neighbor. To represent a superpixel, we concatenate the BoW representations of the superpixel and the region around it, giving a BoW feature vector of length 2  400 = 800. The overall representation of each superpixel is thus of 238 + 800 = 1038 dimensions. A Logistic Regressor is trained on the training set obtained by standard split of MSRC-21 dataset, where the cost parameter is set to C = 25. For evaluation metric, we follow [17] to use the global accuracy, which is the proportion of correctly labeled pixels to all the pixels considered (excluding pixels with void label):

P Nii accuracy ¼ P i ; i;j N ij

ð18Þ

In this section, we evaluate the proposed method on two benchmarking datasets, the MSRC-21 Dataset [32] and the Stanford Background Dataset (SBD) [33], which are widely used for semantic image segmentation evaluation. MSRC-21 consists of 591 images in 21 classes: building, grass, tree, cow, sheep, sky, airplane, water, face, car, bicycle, flower, sign, bird, book, chair, road, cat, dog, body, and boat, where the ground truth is provided at pixel level. A void label is included to avoid the membership ambiguity of pixels on object boundaries, which is typically ignored in training and evaluation. Following [2], we divide MSRC-21 into 45% for training, 10% for validation, and 45% for test. SBD is mostly used for background understanding, where various foreground objects, including car, cow, book, boat, chair, person, et al., are merged into one foreground class. It contains 715 images chosen from the following public datasets: LabelMe [34], MSRC-21 [32], PASCAL VOC [35], and Geometric Context [36]. Eight category labels were obtained using Amazon’s Mechanical Turk (AMT), which include sky, tree, road, grass, water, building, mountain, and foreground.

where Nij is the number of pixels of label i (ground-truth) being labeled as j. The results are shown in Fig. 2(a), where different numbers of superpixels are tested. We can see that the results associated with FH, SLIC and TP are relatively robust when the number of superpixels is greater than 50, compared to that of MS. Overall, FH performs slightly better than the rest, and is adopted later in our structured prediction model. Moreover, we computer the performance of different segmentations by assigning the dominant labels to superpixels as shown in Fig. 2(b). It can be seen that the accuracies increase with the number of superpixels as expected. That’s because the greater the granularity, the better the segmentation coincides with the border. On the other side, we can see that the performance of image semantic segmentation (Fig. 2(a)) doesnt monotonically increase with the number of superpixels.

6.1. Influence of over-segmentation

6.2. Influence of global constraints

Though over-segmentation is widely adopted as a key preprocessing in semantic segmentation, its impact on subsequent learning is rarely evaluated. In this section, we test the influence of four popular over-segmentation techniques, including Mean Shift (MS) [3], Felzenszwalb and Huttenlocher’s efficient graph-based segmentation (FH) [4], SLIC [5], and TurboPixel (TP) [6]. Note that different methods perform segmentation differently, as shown in Fig. 1, where each image is segmented into about 150 superpixels, MS and FH tend to generate larger superpixels in coherent regions and smaller superpixels in complex regions, while SLIC and TurboPixel appear to produce grid-style balanced superpixels. The question is how such difference in over-segmentation can affect later superpixel labeling. To this end, we consider the task of labeling

In this section, we evaluate the impact of multi-label ranking as global constraints on semantic segmentation, by integrating the global constraints into the experiment in the last section. We compare three ways of using label ranking, Top n constraint, Threshold constraint, and our proposed soft constraint (see Section 3.3). In order to capture complementary image properties, we propose to use different features for multi-label ranking, w.r.t. the local features for superpixels (see Section 3.1). We adopt kernel descriptors [29] for holistic image representation, which construct kernel descriptors from gradient, color, and local binary pattern match kernels using kernel principal component analysis (KPCA). Following the setting in [29], an image is divided into 16  16 pixel patches with 50% overlap to extract low level features. We compute image-level features using efficient match kernels (EMK) on 1  1, 2  2, and 4  4 pyramid sub-regions, and perform

2

The code is available on http://svmlight.joachims.org/svm_struct.html.

87

D. Pei et al. / Computer Vision and Image Understanding 120 (2014) 81–90

(a)

SLIC TurboP MeanS GraphB

73

72

71

70 50

100

150

200

250

300

95

Global Accuracy (%)

Global Accuracy (%)

74

(b)

94 93 92 SLIC TurboP MeanS GraphB

91 90 50

100

Number of Superpixels

150

200

250

300

Number of Superpixels

Fig. 2. Influence of initial segmentation on semantic segmentation performance. (a) The performance of semantic segmentation by unary feature and linear classifier. (b) The performance of semantic segmentation by assigning the dominant labels to superpixels.

constrained kernel singular value decomposition (CKSVD) with 1000 visual words learned by K-means. Overall, each image is represented by a 84,000-dimensional feature vector. We adopt the efficient multi-class ranking algorithm [20] to learn K classification functions ft(Ii): Rd ? R, t 2 C = {c1, c2, . . . , cK}, one for each class, with the goal that for any image I; f ci ðIÞ scores higher than fcj ðIÞ when I is more likely to belong to class ci than to class cj. We compare the kernel descriptor with the widely used spatial pyramid matching (SPM) representation with similar settings. The results measured by ROC curve are shown in Fig. 3, where the area under curve (AUC) of SPM is 90.3%, while the AUC of the kernel descriptor increases to a higher 94.3%. Now we are ready to report results after the integration of multi-labeling ranking. Recall that the Top n constraint considers only the top n labels for each image according to the ranking scores, while Threshold-t constraint retains those with scores greater than t. In contrast, our method converts the ranking scores to soft constraint using the sigmoid function defined in Eq. (11) (here we set a = 3, b =  3, q =  0.4). Either hard or soft constraint is combined with Logistic Regression as in Eq. (12) and the labels of superpixels can be inferred by Eq. (6). The results are shown in Table 1. We can see that the proposed soft constraint method outperforms the other two hard constraint alternatives under various parameters.

6.3. Results for MSRC-21 In this section, we report our structured prediction results on MSRC-21. We also report the results obtained by combining local unary features and Logistic Regression, with or without global constraints, where no pairwise co-occurrence information is incorporated. For comparison, we show the results of six state-of-the-art methods, taken from [22,30,9,37,17]. The overall results are summarized in Table 2. From Table 2, we can see that using local unary features and Logistic Regression yields a baseline of 73% pixel-wise global accuracy and 59% average per-class accuracy. Note that in this baseline, the label of a region (i.e., a superpixel) is decided by its appearance alone. By integrating the multi-label ranking results, we improve the global accuracy by 6% and average accuracy by 9%. This shows that global cues can effectively guide the labeling of local regions by substantially reducing potential classes to be considered during labeling. This is because region labels will be strengthened if they are consistent with the global ranking and suppressed otherwise. By further refining the labeling with pairwise co-occurrence information using structural SVMs framework, we achieve 84% global accuracy and 76% average accuracy, which are highly competitive compared to the results reported in previous methods, although our model is much simpler and efficient

1 KDES SPM

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2

0

0.2

0.4

0.6

0.8

1

Fig. 3. The ROC curve of two feature representation in multi-class ranking. The area under curve (AUC) of kernel descriptor is 94.3% while the AUC of spatial pyramid matching is 90.3%.

Table 1 Comparison of different global constraint methods.

Local Local Local Local Local Local

Feature Feature + Top Feature + Top Feature + Top Feature + Top Feature + Top

Local Local Local Local Local

Feature + Threshold = 0 Feature + Threshold = 0.2 Feature + Threshold = 0.4 Feature + Threshold = 0.6 Feature + Threshold = 0.8

3 4 5 6 7

labels labels labels labels labels

Local Feature + Soft const.

Global

Average

72.8 72.6 76.1 77.5 76.8 76.3

58.6 59.8 64.5 65.0 64.9 63.1

75.9 76.7 77.8 78.0 74.3

63.0 64.5 66.0 66.7 60.8

79.1

67.7

in that we decouple the global constraint from pairwise potential in joint inference and instead integrate it with the local prediction from Logistic Regression (Section 3.1). Considering the per-class accuracy, we obtained very good performance on classes such as grass, sky, flower, which can be inferred easily from local appearance and their accuracies are above 95%. For some difficult classes, such as bird and boat, the accuracies are less than 40%, due to the similar appearance, various sizes, and complex background. Fig. 4 shows example results of our model. Consider the images shown in Fig. 4(a), the labeling results obtained by applying Logistic Regression on local appearance features are shown in Fig. 4(b), where the label of a region is decided by its appearance feature

88

D. Pei et al. / Computer Vision and Image Understanding 120 (2014) 81–90

Table 2 Quantitative results on the MSRC-21 data set. The computation of these scores follows the protocol defined in [17]. The best performance is highlighted in bold. Building Grass Tree Cow Sheep Sky Aeroplane Water Face Car Bicycle Flower Sign Bird Book Chair Road Cat Dog Body Boat Global Average Gould et al. [10] Ladicky et al. [17] Gonfaus et al. [9] Munoz et al. [37] Csurka and Perronnin [30] Lucchi et al. [21] Boix et al. [22]

72 80 60 63 75

95 96 78 93 93

81 86 77 88 78

66 74 91 84 70

71 87 68 65 79

93 99 88 89 88

74 74 87 69 66

70 87 76 78 63

70 86 73 74 75

69 87 77 81 76

72 82 93 84 81

68 97 97 80 74

55 95 73 51 44

23 30 57 55 25

83 86 95 84 75

40 31 81 80 24

77 95 76 69 79

60 51 81 47 54

50 69 46 59 55

50 66 56 71 43

14 09 46 24 18

77 86 77 78 77

64 75 75 71 64

64 66

94 87

91 84

72 81

87 83

97 90 93 81

76 82

72 78

83 86 86 94

88 96

93 87

62 48

90 90

89 81

85 82

97 0 82 75

83 70

0 52

85 83

77 80

LR w/o global LR w/global Structural SVMs

66 76 70

94 98 98

83 87 87

50 68 76

52 66 79

93 68 90 77 96 81

70 70 75

64 73 86

51 82 60 84 74 88

73 77 96

54 57 72

25 32 36

69 79 90

40 58 79

82 86 87

39 18 65 41 74 60

42 57 54

19 20 35

73 79 84

59 68 76

Fig. 4. Example results on MSRC-21 data set by our model. (a) Original images. (b) Logistic regression prediction. (c) Logistic regression prediction with global constraint. (d) Structured prediction results with multi-class labeling prior and contextual information. (e) Ground-truth labeling.

alone. Taking the first image for example, it can be seen that partial regions of the bird were mislabeled as dog, cat, sheep or even road because of the ambiguity of local appearance. Then the multi-label ranking results give higher confidence to labels like grass, bird and dog and suppress the presence of road, sheep and cat as in Fig. 4(c). Finally, by introducing co-occurrence property, most of regions labeled as bird would surpress the presence of dog in an image because these two classes rarely present at the same time. Fig. 4(d) shows the labeling results obtained by our final structured prediction and post-processing by grouping superpixels into a larger group, and it can be seen that the final results are much more clean and consistent. The proposed method is very efficient. It takes about 800 s for training structural SVMs on a training set of 335 samples in MSRC-21, and takes about 1 s for labeling one test image. These

results are running in MATLAB 7.10.0(R2010a) 64 bit on a laptop with 2.67 GHz i5 CPU and 8 GB RAM. 6.4. Results on Stanford Background dataset In this section, we report our results on SBD. We follow [33] to perform 5-fold cross-validation with the dataset randomly divided into 572 training images and 143 test images for each fold. The results are shown in Table 3. We can see that our structured prediction model preforms favorably compared to other state-ofthe-art methods. We also observed that the incorporation of the global label-ranking, although useful, did not improve the performance significantly. This probably can be explained from two aspects: First, in SBD the foreground class includes a wide range of object classes such as person, car, cow, sheep, bicycle, and their

Table 3 Quantitative results on Stanford Background dataset. The best performance is highlighted in bold.

Gould et al. [33] Munoz et al. [37] LR w/o global LR w/global Structural SVMs

Sky

Tree

Road

Grass

Water

Building

Mountain

Foreground

Global

Average

92.6 91.6 91.9 92.4 94.9

61.4 66.3 70.0 70.0 69.7

89.6 86.7 88.9 89.3 90.0

82.4 83.0 78.3 77.8 81.4

47.9 59.8 54.9 51.5 60.9

82.4 78.4 79.6 79.3 79.9

13.8 5.0 6.3 3.0 13.5

53.7 63.5 54.2 57.9 54.1

76.4 76.9 76.6 77.0 77.8

65.5 66.2 65.5 65.2 68.0

D. Pei et al. / Computer Vision and Image Understanding 120 (2014) 81–90

89

Fig. 5. Example results on Stanford Background dataset by our model. (a) Original images. (b) Logistic regression prediction. (c) Logistic regression prediction with global constraint. (d) Structured prediction results with multi-class labeling prior and contextual information. (e) Ground-truth labeling. Note that objects such as cars, person, horse are merged as one foreground class.

appearances vary drastically among classes, making it very difficult to model the appearance; Second, the number of classes in SBD is much less than MSRC-21 and therefore the multi-label ranking and co-occurrence statistics may be less informative. Beside forground, another challenging class is mountain, which has few instance in the dataset, making it very hard to label correctly. Some example results of our model are shown in Fig. 5, where we can see that local labeling is not sufficient to address the appearance ambiguity (Fig. 5(b)). In the presence of pairwise and global cues, the labeling becomes more robust, as shown in Fig. 4(d).

7. Conclusion We have presented a new structured prediction model for semantic segmentation. Traditional structured prediction frameworks using pairwise constraints alone suffer degeneration when a notable number of regions within an image are wrongly labeled in the early stage prediction by Logistic Regression, because the wrong contextual information of mislabeled regions may propagate to correct ones. Therefore it is necessary to confine possible labels from image-level. We utilized the multi-label ranking score and converted it to soft global constraint, which encourage the presence of some likely labels while suppress the presence of unlikely labels. Compared with other existing global constraint schemas, we decoupled the global constraint with pairwise constraint and integrated with unary potential directly, making it much simpler while remain efficiency. The proposed model was evaluated on two challenging datasets and experiments showed that our model obtained highly competitive performance compared with the state-of-the-art results.

In the future work, we plan to integrate multi-source cues such as depth into structural SVMs framework. So far we only consider extracting multi-scale cues from single source, that is the optic image. Features from multiple sources could contain complementary information and be potentially useful for prompting performance. Acknowledgments This work was supported by the National Key Project for Basic Research of China (2013CB329403), Nature Science Foundation of China (No. 61373076), the Fundamental Research Funds for the Central Universities (No.2013121026), and the 985 Project of Xiamen University. References [1] J. Shotton, J. Winn, C. Rother, A. Criminisi, Textonboost: joint appearance, shape and context modeling for multi-class object recognition and segmentation, in: Proceedings of European Conference on Computer Vision (ECCV), 2006, pp. 1–15. [2] J. Shotton, M. Johnson, R. Cipolla, Semantic texton forests for image categorization and segmentation, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1–8. [3] D. Comaniciu, P. Meer, Mean shift: a robust approach toward feature space analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (5) (2002) 603–619. [4] P. Felzenszwalb, D. Huttenlocher, Efficient graph-based image segmentation, International Journal of Computer Vision 59 (2) (2004) 167–181. [5] A. Vedaldi, S. Soatto, Quick shift and kernel methods for mode seeking, in: Proceedings of European Conference on Computer Vision (ECCV), 2008, pp. 705–718. [6] A. Levinshtein, A. Stere, K. Kutulakos, D. Fleet, S. Dickinson, K. Siddiqi, Turbopixels: fast superpixels using geometric flows, IEEE Transactions on Pattern Analysis and Machine Intelligence 31 (12) (2009) 2290–2297. [7] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, S. Süsstrunk, Slic Superpixels, Technical Report 149300 EPFL (June).

90

D. Pei et al. / Computer Vision and Image Understanding 120 (2014) 81–90

[8] A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora, S. Belongie, Objects in context, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2007, pp. 1–8. [9] J. Gonfaus, X. Boix, J. Van De Weijer, A. Bagdanov, J. Serrat, J. Gonzalez, Harmony potentials for joint classification and segmentation, in: Object Categorization Using Co-Occurrence, Location and Appearance (CVPR), 2010, pp. 3280–3287. [10] S. Gould, J. Rodgers, D. Cohen, G. Elidan, D. Koller, Multi-class segmentation with relative location prior, International Journal of Computer Vision 80 (3) (2008) 300–316. [11] C. Galleguillos, A. Rabinovich, S. Belongie, Object categorization using cooccurrence, location and appearance, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1–8. [12] N. Plath, M. Toussaint, S. Nakajima, Multi-class image segmentation using conditional random fields and global classification, in: Proceedings of International Conference on Machine Learning (ICML), 2009, pp. 817–824. [13] G. Csurka, F. Perronnin, A simple high performance approach to semantic segmentation, in: Proceedings of British Machine Vision Conference (BMVC), 2008. [14] P. Kohli, L. Ladicky`, P. Torr, Robust higher order potentials for enforcing label consistency, International Journal of Computer Vision 82 (3) (2009) 302– 324. [15] L. Ladicky, C. Russell, P. Kohli, P. Torr, Graph cut based inference with cooccurrence statistics, in: Proceedings of European Conference on Computer Vision (ECCV), 2010, pp. 239–253. [16] P. Kohli, M. Kumar, P. Torr, P3 & beyond: solving energies with higher order cliques, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007, pp. 1–8. [17] L. Ladicky, C. Russell, P. Kohli, P. Torr, Associative hierarchical CRFs for object class image segmentation, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2009, pp. 739–746. [18] J. Zhu, E. Xing, B. Zhang, Laplace maximum margin markov networks, in: Proceedings of International Conference on Machine Learning (ICML), 2008, pp. 1256–1263. [19] T. Joachims, T. Finley, C. Yu, Cutting-plane training of structural SVMs, Machine Learning 77 (1) (2009) 27–59. [20] S. Bucak, P. Kumar Mallapragada, R. Jin, A. Jain, Efficient multi-label ranking for multi-class learning: application to object recognition, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2009, pp. 2098– 2105. [21] A. Lucchi, Y. Li, X. Boix, K. Smith, P. Fua, Are spatial and global constraints really necessary for segmentation? in: IEEE International Conference on Computer Vision (ICCV), 2011, pp. 9–16.

[22] X. Boix, J. Gonfaus, J. van de Weijer, A. Bagdanov, J. Serrat, J. Gonzàlez, Harmony potentials, International Journal of Computer Vision 96 (1) (2012) 83–102. [23] S. Nowozin, P. Gehler, C. Lampert, On parameter learning in CRF-based approaches to object class image segmentation, in: Proceedings of European conference on Computer vision (ECCV), ECCV’10, 2010, pp. 98–111. [24] G. Heitz, S. Gould, A. Saxena, D. Koller, Cascaded classification models: combining models for holistic scene understanding, in: Proceedings of Neural Information Processing Systems (NIPS), 2008. [25] S. Gould, T. Gao, D. Koller, Region-based segmentation and object detection, in: Proceedings of Neural Information Processing Systems (NIPS), vol. 1, 2009. [26] D. Hoiem, A. Efros, M. Hebert, Closing the loop in scene interpretation, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1–8. [27] I. Tsochantaridis, T. Hofmann, T. Joachims, Y. Altun, Support vector machine learning for interdependent and structured output spaces, in: Proceedings of International Conference on Machine Learning (ICML), 2004, pp. 104–111. [28] B. Fulkerson, A. Vedaldi, S. Soatto, Class segmentation and object localization with superpixel neighborhoods, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2009, pp. 670–677. [29] L. Bo, X. Ren, D. Fox, Kernel descriptors for visual recognition, in: Proceedings of Neural Information Processing Systems (NIPS), 2010. [30] G. Csurka, F. Perronnin, An efficient approach to semantic segmentation, International Journal of Computer Vision 95 (2) (2011) 198–212. [31] C. Desai, D. Ramanan, C. Fowlkes, Discriminative models for multi-class object layout, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2009, pp. 229–236. [32] A. Criminisi, Micorsoft Research Cambridge Object Recognition Image Database. . [33] S. Gould, R. Fulton, D. Koller, Decomposing a scene into geometric and semantically consistent regions, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pp. 1–8. [34] B. Russell, A. Torralba, K. Murphy, W. Freeman, Labelme: a database and webbased tool for image annotation, International Journal of Computer Vision 77 (1–3) (2008) 157–173. [35] M. Everingham, L. Van Gool, C. Willianms, J. Winn, A. Zisserman, The Pascal Visual Object Classes Challenge 2007 (voc2007) Results (2007). [36] D. Hoiem, A. Efros, M. Hebert, Recovering surface layout from an image, International Journal of Computer Vision 75 (1) (2007) 151–172. [37] D. Munoz, J. Bagnell, M. Hebert, Stacked hierarchical labeling, in: Proceedings of European Conference on Computer Vision (ECCV), 2010, pp. 57–70. [38] Z. Li, X.-M. Wu, S.-F. Chang, Segmentation using superpixels: A bipartite graph partitioning approach, in: Proceedings of IEEE Conference on Computer Vision and Pattern 826 Recognition (CVPR), 2012, pp. 789–796.