Saliency detection by exploiting multi-features of color contrast and color distribution

ARTICLE IN PRESS

JID: CAEE

[m3Gsc;September 5, 2017;10:8]

Computers and Electrical Engineering 0 0 0 (2017) 1–16

Contents lists available at ScienceDirect

Computers and Electrical Engineering journal homepage: www.elsevier.com/locate/compeleceng

Saliency detection by exploiting multi-features of color contrast and color distributionR Mian Muhammad Sadiq Fareed a, Qi Chun a,∗, Gulnaz Ahmed b, Muhammad Rizwan Asif a, Muhammad Zeeshan Fareed b a b

School of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China School of Management, Xi’an Jiaotong University, Xi’an 710049, China

a r t i c l e

i n f o

Article history: Received 9 June 2016 Revised 26 August 2017 Accepted 28 August 2017 Available online xxx Keywords: Color contrast Color distribution Location prior Energy cost function Saliency map

a b s t r a c t Automatic salient object detection from a cluttered image using the object prior information related to the image enhances the accuracy of object detection which is very useful for many computer vision applications. In this work, we introduce a new bottom-up approach for salient object detection by incorporating the multi-features of color contrast with background connectivity weight and color distribution. Firstly, we extract coarse saliency map by using a color contrast with background connectivity weight and the color distribution. Secondly, we improve the coarse saliency map result through a multi-features global optimization energy function. This energy function is used to fuse several low-level measures, to evenly highlight the salient object and suppress the background efficiently. Extensive experiments on the benchmark datasets have been performed to demonstrate that the proposed model outperforms against the existed state-of-the-art methods with the higher values of precision and recall. © 2017 Elsevier Ltd. All rights reserved.

1. Introduction The human cortical cells are very sensitive to contrast and hastily capture salient object in a messy image through only selecting the important visual information. This type of capability is also engaged in computer visualization system to deal with the data handling problems. Many saliency detection techniques have been designed to check the attention mechanism of human visual system. Salient object detection is extensively utilized in several computer vision applications such as: image segmentation, object detection, image re-targeting, salient region detection, adaptive image display, and advertising design. The saliency detection schemes designed to deal with the human visual information are generally separated into two classes: the bottom-up saliency detection techniques and the top-down saliency detection techniques. The bottom-up saliency detection techniques are data-driven and non-task-driven. These designed techniques only simulate and pay more attention to the salient areas in the images. While the other class of saliency detection is top-down, which is task driven and designed to work according to the human visual information system with supervised learning.

R ∗

Reviews processed and recommended for publication to the Editor-in-Chief by Guest Editor Dr. E. Cabal-Yepez. Corresponding author. E-mail addresses: [email protected] (M.M. Sadiq Fareed), [email protected] (Q. Chun).

http://dx.doi.org/10.1016/j.compeleceng.2017.08.027 0045-7906/© 2017 Elsevier Ltd. All rights reserved.

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

ARTICLE IN PRESS

JID: CAEE 2

[m3Gsc;September 5, 2017;10:8]

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

Fig. 1. The comparison results of previously proposed contrast based methods on CSSD dataset [4]. The images are arranged in the order from left to right: (a) original image, (b) ground truth, (c) MR [5], (d) SF [3], (e) GS [6], and (f) the proposed model saliency map without refinement.

The majority of contrast-based techniques use more color information for their saliency computation. Zhai et al., [1] utilize the entire image color histograms for the region based saliency computation. In [2], a fast frequency-domain algorithm is proposed that utilizes the low-level characteristics of colors and luminance to compute their saliency maps. Saliency Filters (SF) [3] firstly, computes two color contrast measures like uniqueness and spatial distribution. Then by utilizing these two contrast measures a precise saliency map is derived. A common problem in the above-discussed methods is that they cannot distinguish between the similar features of the foreground and the background regions. So, they cannot properly highlight the salient part of the image. Some of them are equipped to tackle with the smooth background, however, sometime it is invalid to deal with the smooth background. Consequently, in the work presented in this paper, we estimate a new multi-features color contrast with background connectivity weight and a color distribution prior which are more accurate to extract salient objects. We employ the background connectivity with outer boundary information followed by a spatially weighted contrast to find out the color contrast prior with background connectivity weight. In contrast with most presented center based schemes, we utilize the objectbiased prior to enhance the foci of attention. We improve the coarse saliency map results through a global optimization energy function. This energy function is used to fuse several low-level measures, to evenly highlight the salient object and to suppress the background efficiently. Our saliency detection algorithm works in two steps. In the first step, we utilize the color contrast with background measure and color distribution to calculate the initial salient region maps. Then, we use the object-biased prior to highlight the initial contrast map and to achieve the enhance saliency map. Lastly, we engaged the energy function to refine the previous saliency maps by assigning the high values to the flat salient object regions. The framework of the proposed model is elaborated in Fig. 2. The rest of this paper is structured as follows. We discuss the current literature about the saliency detection in the Section 2. In Section 3, particulars of the presented algorithm are illustrated to show how the improvements are achieved from pixel to region level saliency. In Section 4, the experimental results and comparisons are discussed to validate the proposed scheme. Finally, the conclusion is drawn in Section 5.

2. Related work and background In addition to the contrast prior, more than a few recent methods utilize a boundary prior [6–8] to improve their saliency maps. However, these methods are not accurate for measuring saliency, when the objects are located near the image boundaries. To estimate the fine saliency results a global optimization formulation is discussed in [8]. The authors utilize an energy cost function directly to prominent the achieved saliency results. They assign the higher values to the background constraints and the lower values to the foreground constraints to get the smooth and uniform saliency results. However, some background pixels are attached to the foreground and the size of the salient object does not remain significant. To better suppress the background and to effectively highlight the saliency map an energy smoothness function is defined in [9]. They define a global optimization function through that they refine the initial saliency map results. According to that function, the greater the value of the initial saliency maps of a pixel, the greater the probability that it belongs to the foreground. However, they over smoothed the salient objects and the extracted saliency maps edges are not clearly observable. All of the above-mentioned issues like object attenuation and over smoothing are clearly described in Fig. 1. A Regional Contrast (RC) method is defined in [10], to calculate the pixel-wise full resolution saliency map which is simply based on the color separation. The authors select the histogram-based method to get the fast and efficient results. After that, they employ a smoothing procedure to overcome the problem of quantization artifacts. Initially, in RC’s maps, the spatial relation is integrated to get the region based contrast. Then the saliency value is computed by using the global contrast score and the spatial distance of a region from another region. However, this proposed technique is only for the natural images those are sub-optimal for computing salient regions of highly textured images. Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

ARTICLE IN PRESS

JID: CAEE

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

[m3Gsc;September 5, 2017;10:8] 3

Fig. 2. The framework of the proposed saliency detection model.

Fig. 3. The individual map of each stage of the proposed algorithm. The images in this visual comparison are taken from ASD dataset [14]. Images from left to right are: (a) input image, (b) segmented image, (c) color contrast prior with background connectivity weight map, (d) color distribution map, multi-features color contrast map, (e) multi-features color distribution map, (f) combined saliency map, and (g) the optimized final map.

In order to compute the salient regions, a center based color contrast with distribution prior is described in [1,11]. The authors define a smoothing procedure which only depends upon the superpixels segments. They believe that this smoothing process works very well in removing the distortion and highlighting the whole salient object. A refinement procedure is also included in this method to recover the unconnected parts of the pre-computed results. However, they do not include the background prior in the model for their result calculation which is an important measure for saliency detection. A new method for saliency detection by combining Simple Priors (SP) is discussed in [12]. The authors in this method simply combine the frequency prior, the color prior, and the location prior for their result calculation. SP priors are selected on the basis of two reasons: first, people pay more attention to the center of the image and second, the warm colors are more attractive. A two stage saliency detection method is defined in [13], in which the superpixel isolation and the distribution are two important measures for saliency detection. However, the computed results are not persuasive because they only used the low-level features for saliency detection. The saliency maps computed through the above-discussed schemes are not so accurate in the case when the foreground and the background regions are mixed up and the difference between the connective regions is very less. Different from all of the above-discussed schemes, we compute the boundary edge weights as discussed in [7]. This boundary edge weight describes the possibility that how a boundary segment is different from the background in an image. We employ the background connectivity with outer boundary information followed by a spatially weighted contrast to find out the color contrast prior with the background connectivity weight. We select colors which are more attractive to the human visual system and formulate a color distribution prior which is more effective in extracting the saliency results. 3. Saliency via multiple priors The individual steps of superpixels. In the second distribution for the whole of the color contrast and

our proposed model are shown in Fig. 3. In the first step, the original image is segmented into step, color contrast prior with background connectivity is computed. In the third step, the color image is calculated. Then the energy smoothness function is employed to get the multi-features the color distribution as revealed in the fifth and the sixth column of Fig. 3. After that, both

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

ARTICLE IN PRESS

JID: CAEE 4

[m3Gsc;September 5, 2017;10:8]

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

previously computed saliency results are integrated to generate the final saliency map as illustrated in the last column of Fig. 3. 3.0.1. Image segmentation Superpixels provide a more appropriate means to extract features from an image. To attain the best structural information about the image and to avoid the computation problem, we first produce the superpixels by using the Simple Linear Iterative Clustering (SLIC) algorithm [16], which segments the image into local, compact, edge-aware, and the perceptually homogeneous regions. The saliency map with the highest accuracy and the precision can be computed by using the RGB and the CIE LAB color models. We utilize the means of the CIE LAB color space, the RGB color model, and the image coordinates to describe a single superpixel i = [r, g, b, l, a, b, x, y]. 3.0.2. Color contrast with background connectivity weight Contrast prior based saliency computation is the most important step for our result calculation. Based on the consideration that the salient objects are always positioned at the center of the image, however, this supposition is not always accurate. On the contrary, the image boundary segments contain very important information which can be engaged to obtain the saliency maps. The color contrast with the background connectivity weight is used to compute the color contrast at each and every region level. The weights are associated with the spatial distances, the nearer regions are assigned a higher weight than the distant regions. The color space distance among the two regions in [11] is computed using the following expression:

Hc (ri , r j ) =

m1  m2 

P (r1,i )P (r2, j )D(r1,i , r2, j )

(1)

i=1 j=1

where, P(rk, i ) is the probability of the ith color rk, i among all nk colors in the kth region rk , with n=1,2,3 and k = 1, 2. Originally, we normalize the color histogram and applied the probability of a region as the weight for this color to highlight the color differences between dominant colors. We quantize the CIE Lab color space channels into 12, 12, 12, respectively. Then, we build a color histogram for all the superpixels in which each region holds 123 = 1728 components. Where, all of the components are expressed through a color combination of the Lab color channels. Storing and computing the regular matrix format histogram for each region is inefficient since each region typically contains a small number of colors in the color histogram of the whole image. Instead, we use a sparse histogram representation for efficient storage and computation. We further incorporate the spatial information by engaging a spatial weights expression to enhance the effects of the closer regions and reduce the effects of the away regions. Then, the contrast among salient regions is calculated using the notation as

Rc ( ri , r j ) =



D p (ri , r j )exp(

r j =ri

−Ds (ri , r j ) )(wi j )Hc (ri , r j ) 2δ 2

(2)

where, Ds (ri , rj ) is the spatial distance between two consecutive regions and δ is the measure of spatial weighting strength.



However, D p (ri , r j ) = exp

 pi −p j  σ p2



, where pi and pj , respectively, are the positions of two superpixels i and j, and σ p con-

trols the strength of spatial weighting. Spatially weighted contrast is very effective and enhances the saliency detection results. If we normalize a superpixel i between [0, 1], then its saliency can be computed as its spatially weighted contrast to all other superpixels as cont Smap (i ) = Rc (ri , r j ) × Bi × G(x, y )

(3)

The Bi term used in Eq. (3) is the background connectivity weight. The background connectivity weight is mapped from the boundary connectivity. We introduce a background connectivity term to prevent the overlapping of boundary superpixels in the image with the background. We also engage this boundary connectivity to highlight the salient object and to suppress the background. To significantly capture the saliency results without the object attenuation and any loss of object geometry. This extremely dependable background measure presents very valuable information for the saliency detection. Particularly, this background connectivity weight enhances the contrast computation. Superpixel boundary connectivity is measured using the following equation:

N Bconnt =

i=1

d ( p, q ).δ (q Bnd )

N



i=1 exp

−d2 ( p,q ) 2σ 2



(4)

First, we utilize this boundary connectivity to find the foreground and then by using this foreground term, we can find the background connectivity weight. The foreground connectivity term is calculated as follows:

Fi = exp



−

B2connt 2 2σconnt



(5)

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

ARTICLE IN PRESS

JID: CAEE

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

[m3Gsc;September 5, 2017;10:8] 5

Fig. 4. The flowchart describing the contrast maps before and after exploiting location prior for the proposed algorithm. The images in this visual comparison are taken from JUDD dataset.

where, σ connt controls the background connectivity. Then from this foreground connectivity the background connectivity is measured as:



Bi = 1 − Fi

(6)

In Eq. (3), G(x, y) is the object-biased location map which is used to recover the lost segments during the pre-processing. This function assigns high saliency values to the object-biased superpixels. In [9], the center-biased Gaussian blob is incorporated to amplify the saliency results. This method has persuasive results when the object is at the center of the scene. If the object is not appearing at the center of the scene the achieved results do not seem to be good. To accurately confine the object, Gaussian biased prior has been utilized in [13]. Even though this object-biased technique has improved the saliency results, however, they choose the size of the location prior too large that it incorporates some background pixels with the salient object. To overcome this issue, we tune the values of σx = 0.35H and σy = 0.35W, where W is the width and H is the height of the image. We first calculate the center of the input image by using the following equation:



Sc =

xc = yc =

n

i=0

S n i

j=0

S n j

n

j=1

i=1

Sj

xi

Si

yj

(7)

Then, the object-biased prior is computed by using the following equation to enhance and amplify each pixel of our salient region map as



G(x, y ) = exp −

x − x  yi − yc c i − 2σx2 2σy2

(8)

where, σx = xc and σy = yc represent the image center coordinates, while xi and yi are the superpixels coordinates. The contrast map without the location-biased map is not significant and some parts of the salient objects are attenuated during pre-processing. We can see in Fig. 4, the second column describes the contrast map in which some information is lost during processing, however, after exploiting this contrast map with object-biased location prior some of the lost information is recovered as revealed in the fourth column of Fig. 4. 3.0.3. Color distribution prior From the previous studies [12] and daily life experience, authors in SP found that the warm colors are more attractive to the human visual system than the colder colors. They believe that only a few of the colors grab the human attention and the rest of them do not take part in the salient region computation. They did not incorporate the cold colors for their saliency detection computation. Consequently, their resultant maps are not very exact. On the contrary, we deem that each and every color contains some part of information and can help us to find the exact and accurate results. So, in our model, we include all the color channels from CIE Lab color space for results calculation. We make base this analysis, and formulate a metric to estimate the color distribution map for a given segment. To find color distribution maps, we first perform a linear mapping for all the channels in Lab color space fL (i): | → FL (i) ∈ [0, 1], fa (i): | → Fa (i) ∈ [0, 1], and fb (i): | → Fb (i) ∈ [0, 1]. Then we normalize the CIE Lab color channels using the following mathematical expressions as

FL (i ) =

fL (i ) − min(L ) max(L ) − min(L )

(9)

Fa (i ) =

fa (i ) − min(a ) max(a ) − min(a )

(10)

Fb (i ) =

fb (i ) − min(b) max(b) − min(b)

(11)

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

ARTICLE IN PRESS

JID: CAEE 6

[m3Gsc;September 5, 2017;10:8]

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

FLab (i ) = [(FL (i ))2 + (Fa (i ))2 + (Fb (i ))2 ]

Cd (i ) = exp



−

FLab (i )

(12)

 (13)

2 σwc

where, σ wc controls the strength of the color distribution. The color distribution is employed to measure the distribution of a color over the entire input image. If the color of a segment is spread all along the scene, then this segment has the largest color distribution and vice versa [13]. They also supposed that the larger segments have more contribution than the smaller segments. We also follow the same pattern for saliency detection with the only difference that, we supposed color difference and color distribution for our result calculation. However, they only supposed colors similarity for their result computation. We estimate our color distribution measure using the following mathematical expression as dis Smap (i ) =

n 

Pr j − μ p wiCd (i )

(14)

i, j=1

where, Pr j is the position of a superpixel j, μp expresses the weighted mean position of the color Cri , and Wi illustrates the percentage of the area of rj to the area of the entire image. 3.0.4. Energy cost function for saliency map refinement A ranking method for arranging the data according to the initial queries is defined in [5,7–9,17]. In the beginning, few samples are taken and then these samples are regarded as basic queries for arranging the remaining data. The labels have continuous values between 0 and 1 to specify the confidences of queries. For a defined dataset, we construct a graph G = (V, E ), where we take V as nodes and E as undirected link edges. The ranking function is obtained by solving an optimization problem. The optimization function is introduced to accomplish the objective of fine saliency results. This optimization function assigns the higher values to the salient object part and the lesser values to the background cues to better suppress the background. This salient object smoothness function makes sure that the obtained salient object will be uniform and even in all the regions. We define our optimization function as

Fl = arg min

Fl ,l=0...k

k  l=1



n 

wlij

i, j=1



l

Fi −



2 Fjl

+β

n 

l

Fi −





2 ril

i=1

+γ

n  k 

Fil − 1

2

i=1 l=1

(15)

where, F(i) and F(j) are the pre-computed salient region maps values of node i and node j, respectively. While wij is the edge weight between two connected superpixels i and j as follows:

wi j = exp



−

( ci − c j )  2σc2

(16)

The first term on the right-hand side in the energy function is the smoothness constraint. For a good saliency map, the salient object should be even and smooth. This term is used here to assign smooth values to the foreground region. The other term is the fitting constraint, which means that for a good saliency map the connected superpixels should not change too much. The third expression is multi-features foreground constraint. This foreground term is indicated in [7], however, they just used it for a single feature calculation. But we employ this term for multi-features foreground computation. The last defined constraint enhances the foreground contrast and also helps in creating well-defined boundaries of the salient objects. We find the solution of this energy smoothness function as defined in [17]. We take the value of k = 2, because in this framework, we are only dealing with two features. After putting the value of k this optimization function can be written as



 1 1  F1 , F2 = arg min F1 (D1 − W1 )F1 + β F1 − D−1 D1 F1 − D−1 1 r1 1 r1 + F2 (D2 − W2 )F2 F1 ,F2 2 2 

+



1 1 β F2 − D−1 D2 F2 − D−1 (F1 − 1 ) (F1 − 1 ) 2 r2 2 r2 + γ 2 2

(17)

The optimal saliency values of superpixels are calculated by minimizing the above energy function, which can be solved by setting the derivative of the above energy function with respect to F1 and F2 to be zero. The resulting solution of the above function is

S1∗ = 2(D1 − W1 ) + β D1 + γ I

(18)

S2∗ = 2(D2 − W2 ) + β D2 + γ I

(19)

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

ARTICLE IN PRESS

JID: CAEE

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

[m3Gsc;September 5, 2017;10:8] 7

Possibly, almost all the salient region detection techniques [7,17,29] consume a color channel for their result calculation. A few of them have exploited one color channel [12], whereas, only some of them employ more than one color channel for their result computation [29]. We expostulate that use of only one color channel does not bring all the times to successful results. All of the color channels contain some part of the information. Consequently, we employ the two forms of color features for our ranking mechanism as given in Eqs. (18) and (19). This refinement enhances the contrast between foreground and background, so, the resultant map boundaries are well-defined and the size of the salient object is more significant. If a few isolated superpixels are incorrectly detected, then, after refinement these superpixels are repaired. In this way, we get consistent foreground containing all the information. 4. Experimental setup To evaluate the saliency results, there are a number of datasets exist that vary from each other in size, the number of objects in an image, image background is simple or complicated. To evaluate the performance of the proposed model in a better manner, it is essential to run the model over different datasets. The model should perform equally well on all the datasets. We evaluate our model on six different benchmark datasets, which include: 1) CSSD [4], 2) Judd DB [19], 3) MASRA10 0 0 [20], 4)ASD [14], 5) ECSSD [4], and 6) SED2 [21]. We choose these datasets because of the following characteristics: (1) easily available, (2) different complexity levels, (3) a large number of images, (4) varying number of objects in an image, and (5) potential benchmark datasets. The proposed scheme is analyzed against eighteen up to date models. We first perform the visual comparison on different datasets to assess the validity of our model and then draw a graphical comparison to check the performance of the given model. The methods we run against our model are elected due the subsequent four reasons: (1) citations, (2) recency, (3) variety, and (4) computation complexity. These models are: CA [15], AC [2], MR [5], GS [6], RB [7], SP [12], MF [17], AW [22], HF [23], GB [24], IT [25], RC [10], FT [14], MQ [26], MZ [27], HC [28], SF [3], and SR [18]. The source codes of the above-discussed techniques are easily available for public, however, we used the saliency results generated by Cheng et al. [29] for these models IT, SR, GB, AC, MZ, FT, CA, HC, and RC on ASD data set. Some of the saliency maps for other datasets are generated by [14] and are also accessible for public. Only a few of the source codes are directly downloaded from the authors web, consequently, we utilize their source codes to extract the saliency results for the comparison purpose. 4.1. Evaluation metrics Numerous techniques are used to evaluate the concurrence among saliency results and the ground truths. Several metrics consider human eye fixation ground truths to assess the accuracy of the model, while few of them used bounding box ground truths. Here, we use three evaluation metrics to evaluate the performance of the proposed model. Precision-Recall (PR) and Receiver Operating Characteristics (ROC) evaluation measures depend on the overlaid region between saliency result and the ground truth. From precision and recall, we also compute another metric f score which is frequently used in the literature to analyze the computed results. Furthermore, we employ another metric that can directly calculate the Mean Absolute Error (MAE) among the computed saliency result and the ground truth mask. Before computing the PR evaluation metric, the calculated saliency map should be converted into the binary form to evaluate the calculated map. We use the adaptive threshold as defined in [29], the proposed adaptive threshold for binarizing saliency map S, that is calculated as twice as the mean saliency of S:

Ta =

h w 2  S(x, y ) w×h

(20)

x=1 y=1

where, w is the width and h is the height of saliency map. 4.1.1. Precision-Recall (PR) We use notation S for saliency, after saliency S is converted to the binary mask B with the ground truth G the PR-curve can be calculated using the following:

P recision =

|B



G|

|B|



, Recall =

|G B| |G|

(21)

4.1.2. F-measure Precision and Recall jointly compute the F-measure because without F-measure the saliency map is not fully analyzed. The F-measure is the overall performance measurement calculated by the weighted harmonic of precision and recall.

Fη =

(1 + η2 ) × Precision × Recall η2 × (Precision + Recal l )

(22)

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

ARTICLE IN PRESS

JID: CAEE 8

[m3Gsc;September 5, 2017;10:8]

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

Fig. 5. The PR-curves and the ROC-curves for graphical performance evaluation of the final saliency map with object-biased location prior and without location prior using the MASRA-10 0 0 database.

where, the value of η = 0.3 is suggested in previous saliency detection [29] to increase importance to the precision value. We used the same value of η to check the performance of our protocol. 4.1.3. Receiver operating characteristics (ROC) curve After evaluating the performance using the F-measure, we as well computed True Positive Rate (TPR) and False Positive Rate (FPR). We draw the ROC curve after binarizing the saliency result with a fixed threshold

T PR =

|B



G|

|G|



, F PR =

|B G¯ | |G¯ |.

(23)

where, G¯ and B¯ represent the opposite of the ground-truth G and binary mask B, respectively. The ROC curve is drawn using TPR versus FPR and then by changing the value of fixed threshold. 4.1.4. Mean absolute error For a complete comparison, we consequently assess the Mean Absolute Error (MAE) between the continuous saliency map S and the ground truth G. The MAE value is defined as:

MAE =

h w 1  |S¯(x, y ) − G¯ (x, y )| w×h

(24)

x=1 y=1

where, S¯ and G¯ are the normalized values of saliency map S and the ground truth G in the range [0, 1]. MASRA-1000 dataset: First of all, we analyze our model graphically using PR-curve and ROC-curve on MASRA-10 0 0 dataset [20]. Commonly, all the saliency detection methods prefer this dataset because it has a very large collection of simple as well as complicated images. Initially, we check the effect of the location prior on the final saliency results as revealed in Fig. 5. The final saliency map with location prior shows improved results as compared to the final map without location map. We assess the effect of multiple features on the final saliency results as demonstrated in Fig. 6. The final map with multiple features illustrates better results as compared to final map with a single-feature. We also examine our salient region detection method graphically with different parameter values. The balancing parameters also affect the final saliency map as described in Fig. 7. So, there is a need of proper adjustment of these parameters to get an accurate map. Then we evaluate the effect of fitting and multi-label constraint parameters on the final saliency map using PR-curves as shown in Fig. 9. From the simulation results, we found that the fitting and multi-features constraints play a very important role in highlighting the salient objects and in suppressing the background part. ASD dataset: We also evaluate the performance of our model on ASD data set [14]. The reason for selecting the ASD data set is that the ASD dataset is a subset of the most famous MSRA dataset. Almost all salient object detection techniques used Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

JID: CAEE

ARTICLE IN PRESS M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

[m3Gsc;September 5, 2017;10:8] 9

Fig. 6. The PR-curves and the ROC-curves for graphical performance evaluation of the final saliency map with multiple-features and with single-feature on the MASRA-10 0 0 database.

Fig. 7. PR-curves for the validity of the color distribution and the color contrast priors on MASRA-10 0 0 dataset. The balancing parameters are tuned at different values to check their effect on the final saliency map.

this dataset for their model evaluation, so, it is also our preference to know the strength and bounds of our model on this dataset. The graphical comparison of our model with pre-processed and post-processed maps is elaborated in Fig. 10. Here, we can see that the curves of the initial saliency maps of color contrast and color distribution are lower than that of the saliency map generated through refinement. That is the reason that the designed energy smoothness function assigns higher weights to the object regions which are suppressed during initial map calculation and also enhances the contrast between the foreground and the background. The curve of the final saliency map is slightly higher than the both saliency maps obtained after refinement, the reason is that the final saliency map contains the color contrast map and the color distribution maps which are further enhanced in the final map. After that, we check the performance of our saliency detection model Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

JID: CAEE 10

ARTICLE IN PRESS

[m3Gsc;September 5, 2017;10:8]

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

Fig. 8. Visual performance comparison of the proposed model against fifteen state-of-the-art methods. The images in this comparison are arranged in sequence from left to right like the original image, CA [15], AC [2], GS [6], RB [7], SP [12], AW [22], HF [23], GB [24], IT [25], RC [10], FT [14], MQ [26], MZ [27], HC [28], SF [3], SR [18], and the ground truth. All the images used in this comparison are taken from ASD database.

for both visual and graphical with competitor models. We analyze the visual performance of our scheme against sixteen state-of-the-art models like CA [15], AC [2], MF [17], RB [7], SP [12], AW [22], HF [23], GB [24], IT [25], RC [10], FT [14], MQ [26], MZ [27], HC [28], SF [3], and SR [18] on ASD data set as depicted in Fig. 8. From this comparison, we can see that proposed model results are very close to the ground truth. We also used four different evaluation criteria to check the validity of our model. From the results revealed in Fig. 11, we can see that the proposed technique outperforms against all the other schemes, whereas RB [7], MF [17], RC [10], and SF [3] also perform well for saliency detection. From our experimental results, we found that the proposed saliency detection method is more efficient in highlighting the salient regions and more effective in suppressing the background than the other state-of-the-art methods. JuddDB dataset: We also used JuddDB dataset to graphically analyze our model. The purpose of selecting JuddDB dataset is to evaluate the performance of the proposed scheme over an image with multiple objects and high background clutter. We run our model against ten state-of-the-art models like AC [4], CA [15], SP [12], GC [30], FT [14], HC [16], IT [17], SF [5], SR [18], and RC [10] to assess the validity of our model. We used two criteria PR-curve and ROC curve to evaluate the performance. Our saliency detection model performs very well in both evaluation measures PR-curve and ROC curve as described in Fig. 12. Although the performance of RC [10] and SP [12] is good in Fig. 12, however, it is due to the fact that the background is suppressed inadequately and the saliency map contains the whole of the image with the low value of precision. SED2 dataset: Moreover, we used SED2 dataset [21] to analyze and validate our model graphically. The reason to pick SED2 dataset is to evaluate the performance of the proposed scheme over an image with two objects. We compare our model against ten state-of-the-art models like AC [4], CA [15], SP [12], MP [13], FT [14], HC [28], IT [25], RB [7], SR [18], and RC [10] to check the validity of our model. We employ two criteria PR-curve and ROC curve to estimate the performance. Our saliency detection model remains very well for both evaluation measures PR-curve and ROC curve as illustrated in Fig. 13. The performance of RB is up to the mark because the RB suppresses the background more accurately. Failure cases: The proposed model outperforms against the above-discussed state-of-the-art salient object detection methods with higher PR values. However, the performance of the proposed scheme is not very satisfactory in some cases. These typical sorts of cases are shown inFig. 14. The proposed model has not performed very well when the color of the salient object is similar to the background, in this scenario the object will not salient properly, some background pixels are atPlease cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

JID: CAEE

ARTICLE IN PRESS M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

[m3Gsc;September 5, 2017;10:8] 11

Fig. 9. PR-curves to validate the performance of our proposed salient region detection model with different parameters setting on MASRA-10 0 0 dataset. The fitting parameter and foreground parameter are tuned at different values to verify the refinement function and their effect on the final saliency map.

Fig. 10. Graphical performance evaluation of different priors and the validity of the smoothness energy function using the ASD database. PR-curves and ROC-curves are drawn from different maps of the proposed model. Table 1 Comparison of average execution time of different methods (seconds per image). Method

Time(s)

Code

MR [5] RC [10] GB [24] IT [25] FT [14] MF [17] MP [13] SP [12] Our

0.547 0.108 0.495 0.185 0.025 2.15 2.8 0.25 0.248

Matlab C++ Matlab Matlab C++ Matlab Matlab Matlab Matlab

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

JID: CAEE 12

ARTICLE IN PRESS

[m3Gsc;September 5, 2017;10:8]

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

Fig. 11. Graphical performance comparison of the proposed model against fourteen state-of-the-art methods like AC [2], AW [22], CA [15], HF [23], GB [24], RB [7], MF [17], HC [28], FT [14], IT [25], MQ [26], MZ [27], RC [10], SF [3], and SR [18] on the ASD database.

tached to the object and object size not remain significant. The salient object with the similar look in the background is very complicated to be detected, which is a well-known issue in object detection. Execution time The average running time of recent methods using Matlab and C ++ implementation on the ASD data set is described in Table 1. The execution time of all methods explained in this table is obtained using a machine with Intel dual core i3 − 2310 M, 2.10 GHz CPU, and 4 GB RAM. In general, the better saliency results can be attained at the cost of execution time. Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

ARTICLE IN PRESS

JID: CAEE

[m3Gsc;September 5, 2017;10:8]

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

1

0.7 0.6 0.5

0.9 0.8 True positive rate

0.8

Precision

1

Our AC SP CA FT HC SF SR IT RC GC

0.9

0.4 0.3

0.7 0.6

0.4 0.3 0.2

0.1

0.1 0

0.2

0.4

Recall

0.6

0.8

1

Our AC CA SP GC FT HC IT SF SR RC

0.5

0.2

0

13

0

0

0.2

0.4

0.6

False positive rate

0.8

1

Fig. 12. Performance analysis of our model using saliency evaluation criterion PR-curve and ROC-curve with ten state-of-the-art techniques on the Judd DB database. The competitors schemes are AC [4], CA [15], SP [12], GC [30], FT [14], HC [16], IT [17], SF [5], SR [18], and RC [10].

Fig. 13. Performance analysis of our model using saliency evaluation criteria PR-curve and ROC-curve with ten state-of-the-art techniques on the SED2 database. The other schemes are AC [4], CA [15], SP [12], MP [13], FT [14], HC [28], IT [25], RB [7], SR [18], and RC [10].

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

JID: CAEE 14

ARTICLE IN PRESS

[m3Gsc;September 5, 2017;10:8]

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

Fig. 14. Some failure cases in ECSSD dataset [4]. The images are in the order from left to right: (a) the original images are in the first column, (b) the failure examples of our model are in the second column, (c) CB failure examples are in the third column, (d) RC failure results are in the fourth column, (e) MNP failure maps are in the fifth column, and (f) failure maps of MP are given in the last column.

5. Conclusion In this paper, we have presented a novel bottom-up saliency detection method by utilizing the multi-feature of color contrast and distribution. We have introduced a color distribution with the background connectivity weight and the color contrast priors which are more accurate to extract the salient objects. We also engaged a multi-feature smoothness energy function to get better and very precise results. The experimental outcomes showed that, the saliency maps of the proposed algorithm clearly highlighted the salient objects and homogeneously suppressed the background regions. Additionally, the proposed algorithm performed very well against the mentioned state-of-the-art schemes in all evaluation measures. The proposed method performance remained equally well in graphical as well as for visual comparisons. In the future, we will consider a hierarchical abstraction base model to segment the image at different weights to detect the saliency map. Acknowledgment This work was supported in part by the National Natural Science Foundation of China (Grant no. 61572395), in part by the National High-tech Research and Development Program of China (Grant no. 2009AA01Z321) and in part by the Specialized Research Fund for the Doctoral Program of Higher Education of China (Grant no. 20110201110012). References [1] Zhai Y, Shah M. Visual attention detection in video sequences using spatiotemporal cues. In: Proceedings of ACM conference on multimedia; 2006. p. 815–24. [2] Achanta R, Susstrunk S. Saliency detection using maximum symmetric surround. Proceedings of the 17th IEEE international conference on image processing (ICIP) 2010;2653(2656):26–9. [3] Perazzi F, Krahenbuhl P, Pritch Y, Hornung A. Saliency filters: contrast-based filtering for salient region detection. In: Proceedings of computer vision and pattern recognition (CVPR); 2012. p. 733–40. 2012 IEEE Conference on. [4] Yan Q, Xu L, Shi J, Jia J. Hierarchical saliency detection 2013;1155(1162):23–8. [5] Yang C, Zhang L, Lu H, Ruan X, Yang M-H. Saliency detection via graph-based manifold ranking. Proceedings of IEEE conference on computer vision and pattern recognition (CVPR) 2013;3166(3173). [6] Wei Y, Wen F, Zhu W, Sun J. Geodesic saliency using background priors. In: Fitzgibbon A, Lazebnik S, Perona P, Sato Y, Schmid C, editors. Lecture notes in computer science, 7574. Heidelberg, Berlin: Springer; 2012. p. 29–42. [7] Zhu W, Liang S, Wei Y, Sun J. Saliency optimization from robust background detection 2014;2814(2821). [8] Zhang H, Xu M. Saliency detection based on boundary feature and smoothness energy function. Optik (Stuttg) 2015;126:81–6. [9] Fareed MMS, Ahmed G, Qi C. Salient region detection through sparse reconstruction and graph-based ranking. J. Vis. Commun. Image Represent. 2015;32C:144–55. doi:10.1016/j.jvcir.2015.08.002. [10] Cheng M, Mitra NJ, Huang X, Torr PHS, Hu S. Global contrast based salient region detection. IEEE Trans Pattern Anal Mach Intell 2015;37(3):569. 582 [11] Fu K, Gong C, Yang J, Zhou Y, Yu-Hua I. Superpixel based color contrast and color distribution driven salient object detection. Signal Process Image Commun 2013;28:1448–63. [12] Zhang L, Gu Z, Li H. SDSP: a novel saliency detection method by combining simple priors. Proceedings of IEEE international conference on image processing (ICIP) 2013;171(175):15–18. [13] Fan Q, Qi C. Two-stage salient region detection by exploiting multiple priors. J Vis Commun Image Represent 2014;25(8):1823–34. [14] Achanta R, Hemami S, Estrada F, Susstrunk S. Frequency-tuned salient region detection. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition workshops; 2009. p. 1597–604. [15] Goferman S, Zelnik-Manor L, Tal A. Context-aware saliency detection. Proceedings of IEEE conference on computer vision and pattern recognition (CVPR) 2010:2376–83. [16] Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Susstrunk S. SLIC Superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 2012;34(11):2274–82. [17] Zhang L, Zhao S, Liu W, Lu H. Saliency detection via sparse reconstruction and joint label inference in multiple features. Neurocomputing 2015;155:1–11. [18] Hou X, Zhang L. Saliency detection: a spectral residual approach. Proceedings of IEEE conference on computer vision and pattern recognition (CVPR) 2007;1(8):17–22. [19] Borji A. What is a salient object? a dataset and a baseline model for salient object detection. IEEE Trans Image Process 2015;24(2):742–56. [20] Liu T, Sun J, Zheng N-N, Tang X, Shum H-Y. Learning to detect a salient object. Proceedings of IEEE conference on computer vision and pattern recognition (CVPR) 2007:1–8. 2007.

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

JID: CAEE

ARTICLE IN PRESS M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

[m3Gsc;September 5, 2017;10:8] 15

[21] Alpert S, Galun M, Basri R, Brandt A. Image segmentation by probabilistic bottom-up aggregation and cue integration. Proceedings of IEEE conference on computer vision and pattern recognition (CVPR) 2007;1(8):17–22. [22] Garcia-Diaz A, Fdez-Vidal XR, Pardo XM, Dosil R. Saliency from hierarchical adaptation through decorrelation and variance normalization. Image Vis Comput 2012;30(1):51–64. [23] Li J, Levine MD, An X, Xu X, He H. Visual saliency based on scale-space analysis in the frequency domain. IEEE Trans Pattern Anal Mach Intell 2013;35(4):996–1010. [24] Harel J, Koch C, Perona P. Graph-based visual saliency. Adv Neural Inf Process Syst 2006;19(4–7):545–52. 2006 [25] Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 1998;20(11):1254–9. doi:10.1109/34.730558. [26] Schauerte B, Stiefelhagen R. Predicting human gaze using quaternion DCT image signature saliency and face detection. In: In Proceedings of the 12th IEEE workshop on the applications of computer vision (WACV). USA; 2012. p. 9–11. [27] Ma Y-F, Zhang H-J. Contrast-based image attention analysis by using fuzzy growing. In: In Proceedings of the eleventh ACM international conference on Multimedia (MULTIMEDIA ’03). New York, NY, USA: ACM; 2003. p. 374–81. doi:10.1145/957013.957094. [28] Cheng M-M, Zhang G-X, Mitra NJ, Huang X, Hu S-M. Global contrast based salient region detection. Proceedings of IEEE conference on computer vision and pattern recognition (CVPR) 2011(409–416):20–5. [29] Fareed MMS, Chun Q, Ahmed G, Asif MR, Bibi I. Maximum mean discrepancy regularized sparse reconstruction for robust salient regions detection. Signal Process Image Commun 2017;54(66–80):0923–5965. https://doi.org/10.1016/j.image.2017.02.013ISSN. [30] Cheng M-M, Warrell J, Lin W-Y, Zheng S, Vineet V, Crook N. Efficient salient region detection with soft image abstraction. Proceedings of IEEE international conference on computer vision (IEEE ICCV) 2013.

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027

JID: CAEE 16

ARTICLE IN PRESS

[m3Gsc;September 5, 2017;10:8]

M.M. Sadiq Fareed et al. / Computers and Electrical Engineering 000 (2017) 1–16

Mian Muhammad Sadiq Fareed is currently working as Post-doctoral fellow at School of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an, China. Previously, he completed Ph.D. Communication and Information Engineering from School of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an, China. He is interested in computer vision, object detection/ segmentation, wireless body area networks, and wireless sensor networks. Chun Qi received the Ph.D. degree from Xian Jiaotong University, Xian, China, in 20 0 0. He is currently a Professor and Ph.D. supervisor at School of Electronics and Information Engineering, Xi’an Jiaotong University. His current research interests mainly include image processing, pattern recognition and signal processing. Gulnaz Ahmed is currently pursuing post-doctorate from School of Management, Xi’an Jiaotong University, Xi’an, China. Previously, she completed Ph.D. from School of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an, China. Her research interests include: image processing, big data analysis, wireless body area networks, and wireless sensor networks. Muhammad Rizwan Asif is currently pursuing Ph.D. in Information and Communication Engineering from Xi’an Jiaotong University, China. He is currently on leave from his position of Lecturer at COMSATS Institute of Information Technology, Lahore, Pakistan. His research interests include image processing, computer vision and intelligent systems. Muhammad Zeeshan is enrolled in the Ph.D. in School of Management, Xi’an Jiaotong University, Xi’an, China. Previously, he completed MS Electrical Engineering from Northwestern Poly-technical University, Xi’an, China. His main research interests include wireless body area sensor networks, and wireless sensor networks.

Please cite this article as: M.M. Sadiq Fareed et al., Saliency detection by exploiting multi-features of color contrast and color distribution, Computers and Electrical Engineering (2017), http://dx.doi.org/10.1016/j.compeleceng.2017.08.027