Histogram-based automatic thresholding for bruise detection of apples by structured-illumination reflectance imaging

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

Available online at www.sciencedirect.com

ScienceDirect journal homepage: www.elsevier.com/locate/issn/15375110

Research Paper

Histogram-based automatic thresholding for bruise detection of apples by structured-illumination reflectance imaging Yuzhen Lu a, Renfu Lu b,* a b

Department of Biosystems and Agricultural Engineering, Michigan State University, East Lansing, MI 48824, USA United States Department of Agriculture Agricultural Research Service, East Lansing, MI 48824, USA

article info

Thresholding is an important step in the segmentation of image features, and the existing

Article history:

methods are not all effective when the image histogram exhibits a unimodal pattern,

Received 30 January 2017

which is common in defect detection of fruit. This study was aimed at developing a general

Received in revised form

automatic thresholding methodology for fast and effective segmentation of bruises from

15 May 2017

the images acquired by structured-illumination reflectance imaging (SIRI). SIRI images,

Accepted 24 May 2017

under sinusoidal patterns of illumination at a spatial frequency of 100 cycles m
1, were acquired from 120 apple samples of four varieties with artificially created bruises and from another 40 apples with naturally occurred bruises. Subsequently, three sets of images, i.e.,

Keywords:

amplitude component (AC), direct component (DC) and ratio (i.e., dividing AC by DC), were

Automatic thresholding

derived from the original SIRI images. A unimodal thresholding method, called UNIMODE,

Histogram

was first applied to DC images for background removal, and then nine automatic thresh-

Structured illumination

olding techniques, including one unimodal and eight bimodal, were applied to the ratio

Bruise

images for bruise segmentation. It was found that severe over-segmentation occurred

Apple

when using the bimodal thresholding methods, and this problem was mitigated by confining threshold selection to the lower part of the histogram that contained bruise information. Three bimodal thresholding techniques, i.e., INTERMODE (histogram valley emphasized), RIDLER (iterative thresholding), OTSU (clustering based) achieved the best bruise detection results with the overall accuracies of more than 90%. The overall detection results were further improved by integrating these techniques with the unimodal thresholding, due to reductions in the false positive error. The three bimodal thresholding techniques resulted in overall detection accuracies of 77e85% for naturally occurred bruises. This study has showed that the proposed automatic thresholding methodology provides a simple and effective tool for bruise detection of apples. Published by Elsevier Ltd on behalf of IAgrE.

* Corresponding author. E-mail addresses: [email protected] (Y. Lu), [email protected], [email protected] (R. Lu). http://dx.doi.org/10.1016/j.biosystemseng.2017.05.005 1537-5110/Published by Elsevier Ltd on behalf of IAgrE.

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

1.

Introduction

Automatic detection of defects for apples during sorting and grading has been a challenging issue (Lu & Lu, 2016). The difficulty is mainly attributed to the fact that there are a wide variety of defects, such as bruise, scar, physiological disorders, insect damage and contamination, and each of them is different in its morphological or physiological characteristics. The problem is further complicated by the fruit skin that may have great variations in colour and texture, which are dependent on variety as well as the degree of maturity or ripeness and pre- and post-harvest handling regimes. Bruise, especially fresh one, is among the most difficult defects to detect (Leemans & Destain, 2004; Leemans, Magein, & Destain, 2002; Shahin, Tollner, McClendon, & Arabnia, 2002; Unay & Gosselin, 2006), because it is often free of external symptoms or looks similar to the surrounding healthy tissue. Numerous imaging modalities have been investigated for detection of defects on apples over the past 30 years, from visible light based monochromatic or black/white imaging to colour or RGB (i.e., red, green and blue) imaging, and from near-infrared (NIR) broadband imaging to multi- and hyperspectral imaging (Li, Huang, & Zhao, 2015; Lu & Lu, 2016). Among these techniques, NIR imaging has shown good performance in detecting such defects as bruises (Baranowski, Mazurek, & Pastuszka-Wozniak, 2013; Bennedsen & Peterson, 2005; Lu, 2003; Luo, Takahashi, Kyo, & Zhang, 2011; Xing, Saeys, & De Baerdemaeker, 2007), due to better penetration of NIR light in tissues than the visible light. However, NIR imaging has been reported to be ineffective for the detection of fresh bruises of less than one day old (Martinsen et al., 2014), because bruised tissues have not fully developed in such a short time period. Our lab recently developed a SIRI technique, which uses sinusoidally-modulated structured light in place of uniform light that is predominantly employed in the existing imaging techniques, for defect detection of apples. SIRI has the capability of enhancing image contrast and resolution and better controlling the penetration of light in the tissue by varying the spatial frequency of illumination (Li, 2016; Lu, Li, & Lu, 2016c; Lu & Lu, in press). It was found to be more effective or superior to conventional imaging under uniform illumination, in ascertaining fresh bruises within hours after impact bruising. As in other imaging techniques, image processing, including image segmentation and classification, is critical in implementing SIRI technique for fruit defect detection. With image segmentation and classification methods, possible defect regions are first segmented by thresholding and then classified by using a detection criterion or model built upon extracted features (Li, Wang, & Gu, 2002; Throop, Aneshansley, Anger, & Peterson, 2005; Throop, Aneshansley, & Upchurch, 1995; Xing, Bravo, Jancsok, Ramon, & De Baerdemaeker, 2005; Xing et al., 2007). In defect segmentations of fruit, thresholding-based techniques are generally used to remove the background and then segment possible defect regions from normal regions; they are fast and simple to use and thus well suited for on-line applications. However, the techniques can be problematic in segmenting the images with weak contrast or uneven reflection due to the surface

31

curvature or irregular shape of fruit. Alternatively, the two steps of image segmentation and classification can be replaced by single-step pixel-based classification models, like neural networks and support vector machines (SVM) (Ariana, Guyer, & Shrestha, 2006; Bennedsen, Peterson, & Tabb, 2007; Guyer & Yang, 2000; Unay & Gosselin, 2006). Classificationbased techniques are able to address multi-class detection problems, but they are more complex and require intensive computation for model training at the pixel level, thus restricting their practical utility. In addition, the image background needs to be identified and removed prior to defect segmentation, which is traditionally done by applying a simple global threshold. In most reported research on detection of defects on apples using thresholding techniques, the user-defined fixed threshold method, i.e., manual thresholding, is commonly used to segment the background and/or defects (Huang, Li, Wang, & Chen, 2015; Li et al., 2002; Lu, 2003; Throop et al., 2005; Zhang et al., 2015). The method requires manual involvement, and the selected threshold can be suboptimal or even erroneous if there is a bias due to limited observations from fruit samples, or when there are unexpected defect features or large fluctuations in lighting. For this reason, automatic thresholding is more desirable in determining the optimal threshold for individual fruit. Unay and Gosselin (2006) reported on the use of four automatic thresholding techniques, including three global thresholding methods, Entropy, Isodata and Otsu (corresponding to ENTROPY, RIDLER and OTSU in this study) and one local thresholding method, for segmenting defects on ‘Jonagold’ apples, and their results showed that Isodata and Otsu performed better than other thresholding methods for defect segmentation. In segmenting bruises on ‘McIntosh’ apples, ElMasry, Wang, Vigneault, Qiao, and ElSayed (2008) performed adaptive thresholding for selecting multiple thresholds based on the intensity distribution of each local neighbourhood in an image. Mizushima and Lu (2013) proposed an automatic thresholding technique for background removal in colour images based on the Otsu thresholding (Otsu, 1979) and SVM. In these applications, researchers only used bimodal thresholding techniques like the Otsu thresholding for defect segmentation, which, however, are likely to fail if the histogram is or close to unimodal (Ng, 2006; Xu, Xu, Jin, & Song, 2011). Defects on fruit are, in most cases, small in size, compared to the surface area of fruit. Hence the unimodal image histograms are expected for most fruit. Research is thus needed to evaluate the performance of different bimodal thresholding methods when they are applied to the segmentation of images with a unimodal histogram. Furthermore, improvements to the existing automatic bimodal thresholding techniques should also be considered in order to achieve more accurate and robust defect segmentation. The main objective of this study was, therefore, to develop a general automatic thresholding methodology for effective bruise segmentation from SIRI images. Specific contributions of the study include: 1) applying a unimodal thresholding technique for automatic background removal; 2) comparing nine automatic thresholding techniques, coupled with histogram pretreatment, for bruise segmentation; and 3) integrating unimodal and bimodal thresholding techniques for improved bruise segmentation.

32

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

2. Overview of automatic thresholding methods Automatic thresholding has been a popular tool in object segmentation of gray-scale images. The basic idea of automatic thresholding is to automatically select an optimal threshold for segmenting objects of interest in an image that is commonly based on its gray-level histogram (Glasbey, 1993). Over the past five decades, considerable research has been reported on automatic thresholding, leading to many wellestablished algorithms. In-depth reviews of numerous automatic thresholding algorithms can be found in Weszka (1978), Sahoo, Soltani, Wong, and Chen (1988), and Sezgin and Sankur (2004). Automatic thresholding techniques can be roughly categorized into global and local thresholding. Global thresholding, also known as bi-level thresholding, selects a single threshold typically based on the histogram of an entire image for segmenting the image into the background and foreground parts. Local thresholding, also called multi-level thresholding, makes use of localized information to define multiple thresholds to segment the image into multiple regions. Compared to local thresholding, global thresholding is much more widely used due to its simplicity and fast speed. For automatic bruise detection of apples, image segmentation includes background removal and bruise segmentation. In both cases, global thresholding can be applied when the image to be segmented primarily consists of two regions, i.e., the background and fruit object, which, in turn, is composed of the defective and normal tissues. The present work, thus, only focuses on global thresholding. In the following section, we give an overview of nine simple and most popular non-parametric histogram-based global thresholding techniques.

2.1.

2.1.2.


ft ðx; yÞ ¼

n if f ðx; yÞ  t 0 if f ðx; yÞ < t

2.1.3.

(1)

s2B ¼

N1 N2 ðm
mT Þ2 þ ðm2
mT Þ2 N 1 N

(3)

where N1 and m1 are the pixel number and average gray level of C1, respectively; N2 and m2 are the pixel number and average gray level of C2, respectively; N and mT are the total pixel number and average gray level of the image. Then OTSU seeks an optimal threshold by maximizing s2B as follows:   t* ¼ argmax s2B ðtÞ

(4)

where arg max is an operator that finds a maximum point for a function and its complementary operator is arg min. The OTSU method remains one of the most widely used thresholding methods, but it is problematic in some cases where the histogram is close to unimodal, or there is a large difference in size between the targeted object and background (Xu et al., 2011), e.g., for detecting defects that are usually much smaller relative to the entire object. The drawback can be somewhat alleviated by emphasizing the within-class variance s2W . Let us assume C1,j and C2,j to be the gray level at pixel j of the classes C1 and C2, respectively. Then s2W can be calculated as follows (Lu & Lu, in press):

(2)

where N is the total number of pixels in an image, that is, P N ¼ L
1 i¼0 ni .

2.1.1.

Otsu's methods

Otsu (1979) proposed a threshold selection method based on discriminant analysis or clustering, which is herein denoted as OTSU. Let an image be portioned into two classes C1 and C2 by a threshold t. The between-class variance s2B can be calculated as follows:

0t < L
1

From the histogram, the probability of occurrence of gray level i is defined as follows: ni N

Iterative thresholding

Ridler and Calvard (1978) proposed one of the first iterative threshold-seeking schemes. From an initial threshold, typically the mean gray level of an image, a new threshold is calculated as the average of the foreground and background class means. The process is repeated iteratively until the change of updated thresholds is sufficiently small. We denote this method as RIDLER.

Algorithms

Let a histogram h of an image f(x,y) be represented by n0, n1, …, ni, …, nL
1, where (x,y) is the spatial coordinate, ni is the number of pixels in the image at gray level i, and L is the number of distinct gray levels (e.g., 256 for eight-bit grayscale images). Applying a global threshold t to f(x,y) results in a binary image ft(x,y) as follows:

pi ¼

finds the two local maxima from the histogram, say nj and nk, and then sets the optimal threshold t* ¼ ðj þ kÞ=2 (Glasbey, 1993). To implement the two methods, the histogram needs to be smoothed properly. To achieve this, a simple method, ITERATIVE SMOOTHING, which iteratively smooths the histogram using three-point moving average, was adopted in this study. ITERATIVE SMOOTHING proceeds until the histogram becomes bimodal. If the bimodality could not be achieved, the histogram was treated as unimodal and the threshold sets to be zero.

Valley-based methods

For images with two distinct objects, the histogram will be bimodal. Then an optimal threshold can be chosen as the gray level corresponding to the valley of the histogram. This method is denoted as MINIMUM (Glasbey, 1993; Prewitt & Mendelsohn, 1966). A simple alternative, called INTERMODE,

s2W ¼

N2 N1  2 1 X  2 1 X C1;j
m1 þ C2;j
m2 N j¼1 N j¼1

(5)

So, the optimal threshold is obtained by minimizing s2W as follows:   t* ¼ argmin s2W ðtÞ

(6)

0t < L
1

This method is denoted as OTSU(M), where (M) means modified.

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

2.1.4.

Maximum entropy method

Kapur, Sahoo, and Wong (1985) utilized information theory to develop a thresholding method by maximizing the entropy of the thresholded image. The authors considered the foreground and background of an image as two different signal sources, and an optimal threshold was obtained when the sum of the two class entropies reached its maximum. Assuming an image is segmented by a threshold t, the entropies E1 and E2 of the segmented two classes C1 and C2 are defined as follows: E1 ðtÞ ¼


E2 ðtÞ ¼


  pi ln P1 P1

t X pi i¼0

(7)

  pi pi ln 1
P1 1
P1 i¼tþ1 L
1 X

(8)

P where P1 ¼ ti¼0 pi is the probability of the class C1. Then the optimal threshold is obtained as follows: t* ¼ argmaxfE1 ðtÞ þ E2 ðtÞg

(9)

0t < L
1

This method is herein denoted as ENTROPY.

2.1.5.

Moment-preserving method

Tsai (1985) considered the grayscale image as the blurred vision of an ideal binary image. The optimal threshold was selected in such a way that the first three moments of the grayscale image are preserved in the resultant binary image, which is denoted as MOMENTS. The kth moment mk is defined as follows: mk ¼

33

L
1 1 X ik ni N i¼0

(10)

The optimal threshold is chosen as the p0-tile where p0 is given by: z
m1 p0 ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi c21
4c0

(11)

t* ¼ argmin fJðtÞg

(14)

0t < L
1

2.1.7.

Unimodal method

All the aforementioned methods operate under an explicit or implicit assumption that the histogram of a given image has two distinct peaks or is bimodally distributed. They may not produce desired segmentation if the histogram deviates significantly from the bimodal distribution. For fruit defect detection, the histogram of an acquired image tends to be unimodal, with a dominant peak located at the lower part due to the well-controlled dark background and an upper flat and broad tail due to the fruit sample. With the dark background being removed, the image is still likely to produce a unimodal histogram, since the defect to be detected is usually much smaller in size than the normal tissue. In addition, lowcontrasted images also tend to produce a unimodal histogram. To address the difficulty in segmenting the images with a unimodal histogram by using bimodal thresholding methods, we present a simple shape-based unimodal thresholding method, denoted as UNIMODE here, which was originally proposed by Rosin (2001). UNIMODE assumes that an image has a dominant peak located at the lower part of its histogram, as shown in Fig. 1(left), and that the main peak has a detectable corner at its base that corresponds to a suitable threshold. A straight line is drawn from the peak to the first empty bin of the histogram closely following the non-empty bin, and the optimal threshold is selected at the point that maximizes the distance between the histogram and the straight line. For the case of defect detection of fruit, this method is expected to work better for removing the image background that usually represents the large class with lower intensities in the image. Furthermore, we extend this method to the cases where the dominant peak is situated at the upper part of the histogram, as shown in Fig. 1(right), which would enable the segmentation of the defect region from the normal or healthy region after the background has been removed.

and, c0 ¼

m1 m3
m22 m1 m3
m3 1 ; c1 ¼ ; z¼ 2 m2
m21 m2
m21

2.1.6.

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi  c21
4c0
c1

(12)

Minimum error method

Kittler and Illingworth (1986) who considered thresholding as a classification problem, proposed the minimum error method, denoted as MINERROR, assuming that the gray levels of object and background pixels are normally distributed with distinct means and standard deviations. A criterion function J(t) was introduced as follows: JðtÞ ¼ 1 þ 2½H1 ðtÞln s1 ðtÞ þ H2 ðtÞln s2 ðtÞ
2½H1 ðtÞln H1 ðtÞ þ H2 ðtÞln H2 ðtÞ (13) Pt PL
1 where H1 ¼ i¼0 ni , H2 ¼ i¼tþ1 ni , and s1 and s2 are the graylevel standard deviations of the two objects segmented by the threshold t. Then the optimal threshold is obtained as follows:

3. Segmentation of SIRI images for bruise detection 3.1.

SIRI images of apples

Two sets of SIRI images from our two previous studies were used in this study. The first set of images were acquired by a multispectral SIRI system at the wavelength of 730 nm, from 120 apple samples of ‘Golden Delicious’, ‘Granny Smith’, ‘Royal Gala’ and ‘Delicious’ varieties (Lu & Lu, in press). Among these samples, 80 were artificially bruised through impact test with an impact energy of about 0.6 J, while the remaining 40 were not bruised. Evaluation of automatic thresholding methods for bruise detection was mainly conducted for these samples. The second set of images were acquired by a broadband light based SIRI system for 40 ‘Fuji’ apples that came from a commercial packinghouse in

34

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

Fig. 1 e A unimodal thresholding method for determining a threshold from a histogram with a single dominant peak located at lower end (left) or upper end (right) of the histogram. Michigan, USA (Li, 2016). Apart from pre-existing bruises, additional bruises had also been incurred on some of these apples after they were run through an in-field sorting machine for a separate study. These images were used for further validation of the thresholding techniques. Both sets of apples were imaged by SIRI under sinusoidallymodulated patterns of illumination at a spatial frequency of 100 cycles m
1, The acquired SIRI images were first subjected to a low-pass Gaussian filter to suppress image noise, followed by a two-phase based demodulation approach to obtain AC and DC images (Lu & Lu, in press). Further processing was done to obtain ratio images by dividing AC by DC. A detailed description of the SIRI system, image acquisition, preprocessing and demodulation can be found in (Lu, Li, & Lu, 2016a, 2016b; Lu & Lu, in press).

3.2.

Background removal

Although all the AC, DC and ratio images can be used for background removal, the DC images had much higher intensities due to less light attenuation within fruit tissues and thus exhibited stronger contrasts between fruit and background. Therefore, the DC images were used for background removal in this study. Figure 2 presents the DC images of four apples and their normalized histograms. The histograms exhibited strong unimodality with a predominant peak very close to zero, which was due to the well-controlled dark environment, and with an upper flat and broad tail due to the

fruit object. It was noted that there existed a small peak nearby the main peak, which was because part of the dark background was illuminated by the projected light. Overall, given the strong unimodal histograms, the UNIMODE method seemed well suited for background removal. Initial tests revealed that UNIMODE was somewhat sensitive to local variations of the histogram caused by image noise. To cope with this issue, the histogram was presmoothed by convolving it with a 3-point window, i.e., [1/3, 1/3, 1/3], before UNIMODE was applied for thresholding. It was found that a single convolution, which did not cause dramatic changes to the histogram, could yield desirable segmentation.

3.3.

Unimodal histogram

After background removal, a mask or template for the fruit region of interest (ROI) was obtained, and it was then applied to the ratio images that exhibited strong contrasts between defective and normal fruit tissues (Lu et al., 2016c; Lu & Lu, in press), for bruise segmentation. As shown in Fig. 3, bruises are clearly distinguishable on the ratio images, while they are not visible on the DC images for the same four apples as presented in Fig. 2. In spite of high contrasts between the bruise and normal regions, these ratio images, overall, still resulted in a dominant unimodal histogram, with or without the presence of a noticeable smaller peak in the lower part of the histogram caused by the bruise tissue (Fig. 3), because the two regions are highly unequal in size.

Fig. 2 e Direct component (DC) images (upper) of apples of ‘Golden Delicious’, ‘Royal Gala’, ‘Granny Smith’ and ‘Delicious’ apples (from left to right) and their corresponding normalized histograms (lower).

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

35

Fig. 3 e Ratio images (upper) of apples of ‘Golden Delicious’, ‘Royal Gala’, ‘Granny Smith’ and ‘Delicious’ (from left to right) and their corresponding normalized histograms (bottom).

Most automatic thresholding methods, including RIDLER, OTSU, ENTROPY, MOMENTS and MINIMUM, only work well for bimodal histograms, and they can result in poor thresholding results for unimodal histograms, as demonstrated later. To cope with the problem, we proposed a pretreatment method, called PARTIAL HISTOGRAM, in which only the partial histogram extracted from the lower part below its main peak position was used for threshold selection. As shown later, the pretreatment enabled the bimodal thresholding methods to yield more accurate bruise segmentation, when the bruise tissue had lower intensities than the normal tissue, which is because excluding the upper part of the histogram irrelevant to bruises significantly narrowed down the possibility of erroneous thresholds. In addition, it was noted that, in dealing with images of normal (i.e., non-bruised) apples, bimodal thresholding methods gave more false alarms than UNIMODE. Thus, further integration of UNIMODE and bimodal thresholding methods was done to improve segmentation results. Such integration was implemented by thresholding the unimodality degree of the histogram to be processed to determine whether a unimodal or bimodal thresholding method would be applied.

3.4.

Bruise detection

Figure 4 shows the flowchart of image processing procedures for bruise detection. A median filter was first used to remove fruit lenticel spots and other small irregularities, which was

followed by applying automatic thresholding for bruise segmentation. The initial segmentation gave a binary image with discrete white patches, commonly referred to as blobs, which might contain some artifacts such as holes and noise due to the imperfection of thresholding. Therefore, morphological processing, including opening and holes filling, was conducted to refine the segmentation. The remaining blobs were then evaluated based on their morphological features, and either dismissed or counted as bruises. Such evaluation can be performed by setting a simple judgement criterion or developing a feature based classification model, which would need a large amount of time for feature extraction and model training. Here, we turned to a simple judgement criterion based on the shape features of the segmented blobs for bruise detection. As the shape of a bruise region is close to circular or elliptical, its circularity can be used for evaluating segmented blobs. Previous investigations (Lu & Lu, in press; Throop et al., 1995) used circularity values ranging from 0.5 to 1.6 to identify bruise regions. We have also noted that naturally occurred bruises can be strongly elongated with a large circularity value. To avoid erroneous removal of such irregular bruise regions, the upper limit of the circularity was extended to 3.0, based on our initial observations. Finally, cleaning operations were applied to eliminate very small objects (smaller than 0.5% of the entire fruit area). As a side note, the segmented elongated region could also be the fruit's edge (Fig. 4), resulting from illumination nonuniformity. To avoid confusion, an erosion operation was

Fig. 4 e Image processing procedures for bruise segmentation and evaluation.

36

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

implemented after background removal to eliminate the far edges from the ROI and subsequent analysis. It should, however, be acknowledged that such processing is not preferred since it can miss the bruise that appears right on the edges. This limitation may be addressed by correcting the illumination artifact through reconstruction of the three-dimensional (3-D) shape of the fruit, which is a topic for further research. The above image processing procedures were implemented for bruise detection. If there existed no non-zero pixels in the final segmented binary image, the apple was considered normal, otherwise bruised. The accuracy of bruise detection was evaluated in terms of the percentage of false positive (FP), false negative (FN) and overall errors.

3.5.

Software

All the image processing procedures, including automatic thresholding techniques for background removal and bruise segmentation, were implemented in the Matlab environment (version R2016b, The MathWorks, Inc., Natick, MA, USA).

4.

Results and discussion

4.1.

Image segmentation

Figure 5 shows the results of threshold selection and background removal using the UNIMODE method for the four apples presented in Fig. 1. All the chosen thresholds, which occurred around the corner at the base of the histogram, achieved good identification of the fruit regions, and UNIMODE also worked well for other apples (results are not presented). The success of the method was attributed to the fact that the image histogram exhibited strong unimodality, due to the well-controlled background, with a flat and broad tail due to the fruit. All the nine thresholding methods described in Section 2 were applied to the fruit ROI for bruise segmentation. Figure 6 shows examples for a ‘Golden Delicious’ apple and a ‘Royal Gala’ apple (the same as presented in Fig. 3) without the pretreatment of PARTIAL HISTOGRAM. For the ‘Golden

Delicious’ apple, MINIMUM, INTERMODE, OTSU(M) and UNIMODE achieved good segmentation, while RIDLER, OTSU, ENTROPY and MOMENTS resulted in over-segmentation, and MINERROR failed to detect the bruise region. For the ‘Royal Gala’ apple, all the methods except UNIMODE, caused severe over-segmentation. These results were associated with the pattern of the histograms of the two apples (Fig. 3). The ‘Golden Delicious’ apple had a less unimodal histogram with two resolved peaks that could be identified by MINIMUM and INTERMODE methods through ITERATIVE SMOOTHING and thus obtained suitable thresholds; the ‘Royal Gala’ apple had a stronger unimodal histogram such that those bimodal-based thresholding methods, especially ENTROPY and OTSU, began to lose their efficacy, and led to over-segmentation. The UNIMODE method performed well for the two apples, but it would fail if the secondary peak of the histogram is relatively large. Figure 7 shows examples of threshold selection by the UNIMODE for four bruised apples. These histograms exhibited strong unimodality with noticeable secondary peaks that were comparable in size to their main peaks, but the UNIMODE was not able to yield meaningful threshold selection. Such histograms usually indicate strong contrasts between bruised and normal tissues, and can be well addressed by such methods as MINIMUM and INTERMODE, which choose a threshold located around the valley between the main and secondary peaks. The unimodality issue encountered with bimodal thresholding methods was greatly mitigated by applying PARTIAL HISTOGRAM. Figure 8 shows the bruise segmentations for the same two apples as shown in Fig. 6. The bruise on ‘Golden Delicious’ was accurately identified by all the methods, and it was also correctly identified on ‘Royal Gala’ except MINIMUM, INTERMODE and ENTROPY. The PARTIAL HISTOGRAM method made the remaining histogram more unimodal, thus greatly improving RIDLER, OTSU, MOMENTS and MINERROR; but it had limited effects on MINIMUM and INTERMODE, since they mainly relied on the peak features of a histogram for threshold selection. The method also had no effects on UNIMODE which had implicitly focused on the lower part of the histogram. It should be stressed that PARTIAL HISTOGRAM works well, provided that the region

Fig. 5 e Smoothed histograms (upper) and direct component (DC) images (bottom) of apples of ‘Golden Delicious’, ‘Royal Gala’, ‘Granny Smith’ and ‘Delicious’ apples (from left to right). The red cross in the histogram indicates the position of the threshold selected by the UNIMODE method and the region enclosed by the red curve in the DC image indicates the identified fruit object after dark background had been removed.

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

37

Fig. 6 e Bruise segmentation results by using the nine automatic thresholding methods applied to the full histograms for a ‘Golden Delicious’ apple (a) and a ‘Royal Gala’ apple (b). The region enclosed by the red curve in the image indicates the detected bruise. MINERROR does not detect the bruise of ‘Golden Delicious’.

Fig. 7 e Threshold selection by the UNIMODE method for four bruised apples. The red cross in histogram indicates the positions of the chosen threshold.

Fig. 8 e Bruise segmentation results by using the nine automatic thresholding methods applied to the partial histogram for a ‘Golden Delicious’ apple (a) and a ‘Royal Gala’ apple (b). The region enclosed by the red curve in the image indicates the detected bruise.

of interest (e.g., the bruise region) has consistently reduced or elevated intensities than normal tissues. In the following, only results from automatic thresholding coupled with PARTIAL HISTOGRAM are discussed for quantitative evaluation of bruise detection.

While the bimodal thresholding techniques, coupled with PARTIAL HISTOGRAM, generally demonstrated the ability to well segment bruises, some of them tended to produce oversegmentation for non-bruised apples that had strong unimodal histograms as well. As shown in Fig. 9, INTERMODE,

38

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

Fig. 9 e Threshold selection by eight bimodal thresholding methods for a normal, non-bruised apple. The red marker in the histogram indicates the position of the selected threshold.

RIDLER, OTSU, ENTROPY and MOMENTS could lead to oversegmentation at varying degrees, thus incurring FP errors; while MINIMUM, OTSU(M) and MINERROR gave more reasonable thresholds. The UNIMODE also performed well for healthy apples (results are not presented).

4.2.

Bruise detection

Table 1 summarizes the detection results by the nine thresholding methods for the 120 apples (40 normal and 80 artificially bruised). INTERMODE, RIDLER and OTSU showed better performance with overall accuracies of more than 90% and the lowest FN error of 5%. MINIMUM and OTSU(M) also performed well with overall accuracies of 90%. The results compared favourably to the errors of 11.7e27.5% reported in our previous study, which used a modified Otsu thresholding that emphasized both inter-class and with-class variances but without histogram pretreatment, for bruise segmentation (Lu & Lu, in press). ENTROPY, MINERROR and MOMENTS methods gave poorest accuracies due to a large FN error, suggesting their inefficacy of segmenting bruises in the case of strong unimodal histograms. UNIMODE, although having a mediocre overall accuracy, performed the best in terms of FP errors. The method tends to give conservative thresholds that are

Table 1 e Errors for detection of artificially created bruises for 120 apple samples of four varieties (‘Delicious’, ‘Golden Delicious’, ‘Granny Smith’ and ‘Royal Gala’) by nine automatic thresholding methods. Method MINIMUM INTERMODE RIDLER OTSU OTSU(M) ENTROPY MOMENTS MINERROR UNIMODE

False positive (%)

False negative (%)

Overall (%)

12.5 15.0 10.0 12.5 5.0 25.0 10.0 5.0 2.5

8.8 5.0 5.0 5.0 12.5 61.3 35.0 60.0 30

10.0 8.3 6.7 7.5 10.0 49.2 26.7 41.7 20.8

typically located at the base corner of the histogram, which may increase the FN error but decrease the FP error. Table 2 shows the detection results by combining UNIMODE with INTERMODE, RIDLER and OTSU individually. A reduction in the overall errors was achieved mainly due to reduced FP errors except for UNIMODE þ OTSU, compared to the three bimodal methods used alone. But it should be pointed out that combining unimodal and bimodal methods led to slightly increased computation time, compared to using them alone, due to the need for inspecting the modality of histograms. Finally, the INTERMODE, RIDLER and OTSU methods were further evaluated on 40 ‘Fuji’ apples with naturally occurred bruises. As shown in Fig. 10, the majority of these apples had more than one bruises, and these bruises were far more varied in size and shape than the artificially created bruises, some of which were difficult to recognize visually, thus adding more difficulty to the bruise segmentation. In view of the fact that all the apples were bruised (i.e., no FP error), the three thresholding methods were implemented without coupling with UNIMODE. It was found that both RIDLER and OTSU were able to detect all bruised apples, while INTERMODE performed slightly worse with an overall error of 10% (no FP error was observed in the three cases). Figure 11 shows the results by RIDLER. There were some bruises that were undetected or only partial detected, because of the weak-contrasted images whose histograms were extremely unimodal. Further evaluation of the detection results was done by comparing the number of detected bruises and all the bruises visually revealed in the images, and the overall detection errors were 15.0%, 16.7% and 26.7% for RIDLER, OTSU and INTERMODE, respectively. These are good results,

Table 2 e Errors for detection of artificially created bruises by integrating the unimodal and three bimodal thresholding methods. Methods UNIMODE þ INTERMODE UNIMODE þ RIDLER UNIMODE þ OTSU

FP (%)

FN (%)

Overall (%)

7.5 7.5 10.0

7.5 5.0 5.0

7.5 5.8 6.7

Note: FP and FN denote false positive and false negative errors, respectively.

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

39

Fig. 10 e Ratio images for 40 ‘Fuji’ apples with naturally occurred bruises.

Fig. 11 e Bruise detection by RIDLER coupled with PARTIAL HISTOGRAM for 40 ‘Fuji’ apples with naturally occurred bruises. The region enclosed by the red curve in the image indicates the detected bruise.

given the fact that many bruises were small, which would not degrade the fruit for the fresh market. The findings of this study demonstrate that all the methods except ENTROPY, MOMENTS and MINERROR are effective for bruise detection of apples, and that their performance depends on the histogram modality. Apples with or without bruises usually have histograms that are strongly unimodal, which pose a challenge for bruise segmentation when using such bimodal thresholding methods as OTSU and RIDLER. A simple solution to mitigating the problem is to

confine the histogram to its lower part that contains bruise information, based on prior knowledge that bruises lead to reduced intensities of reflectance compared to normal tissues. But it should be noted that the reflectance of bruised tissues tends to increase over time during cold storage (Lu, 2003), and may exceed that of normal tissues, which would complicate the bruise detection algorithm and thus warrant further investigation. Implementing MINIMUM and INTERMODE requires the histogram to display two well-resolved peaks representing bruise and normal tissues. INTERMODE is generally

40

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

more reliable in segmentation as it takes the average of the two peak positions. While UNIMODE is not well suited for bruise segmentation, it can be used in conjunction with other methods to suppress the FP error and thus improve the overall detection accuracies. Besides, it also provides an effective means for background removal. The present study mainly focuses on segmenting bruises of apples created under controlled conditions. More research is needed to evaluate automatic thresholding techniques for segmenting naturally occurred bruises that are formed during various postharvest handling operations (e.g., in-field sorting, transport and online sorting). The stem and calyx of apples, if present in the image, can affect thresholding for bruise segmentation, as they can also appear as dark, discoloured areas and be very similar to bruises. After thresholding performed for the initial segmentation, a further identification step is required to distinguish between stem/calyx and bruises, which will be explored in future work. Overall, automatic thresholding holds promise for detection of bruises on apples in a real-time inspection system, compared to manual thresholding and pixel-level classification based image segmentation that is computation intensive. The general thresholding methodology presented this study also has the potential for being used for detection of other types of defects for apples and other fruits. But it should be noted that thresholding techniques alone cannot be used for detecting bruises and other defects, which need be used in conjunction with an evaluation criterion or a supervised classification model, based upon the relevant features of defects under study, to achieve the final detection purpose.

5.

Conclusions

This study evaluated nine histogram-based automatic thresholding methods for segmenting bruises from the SIRI images of apples, from which a general thresholding methodology was proposed for bruise detection. The unimodal thresholding method was effective for removing background from the SIRI derived DC images. All thresholding methods, except ENTROPY, MOMENTS and MINERROR, achieved good detection accuracies, when they were applied to the lower or left part of the histograms of the ratio images that contain bruised information. The RIDLER, OTSU and INTERMODE methods, in particular, showed superior performance for detecting artificially created bruises with overall accuracies of more than 90%, and they also performed well in detecting naturally occurred bruises, with overall accuracies between 73% and 85%. Integration of UNIMODE and the three bimodal thresholding methods further improved bruise detection, resulting in lower FP errors. The proposed thresholding methodology is simple and effective for bruise detection of apples, and it could be used for detecting other types of defects on fruit.

Disclaimer Mention of commercial products is solely for providing factual information, and it does not imply the endorsement of the products by USDA over those not mentioned.

Acknowledgement The authors would like to thank Richard Li for providing broadband SIRI images from apples with naturally occurred bruises for further evaluation of thresholding techniques for bruise segmentation.

references

Ariana, D., Guyer, D. E., & Shrestha, B. (2006). Integrating multispectral reflectance and fluorescence imaging for defect detection on apples. Computers and Electronics in Agriculture, 50(2), 148e161. Baranowski, P., Mazurek, W., & Pastuszka-Wozniak, J. (2013). Supervised classification of bruised apples with respect to the time after bruising on the basis of hyperspectral imaging data. Postharvest Biology and Technology, 86, 249e258. Bennedsen, B. S., & Peterson, D. L. (2005). Performance of a system for apple surface defect identification in near-infrared images. Biosystems Engineering, 90(4), 419e431. Bennedsen, B. S., Peterson, D. L., & Tabb, A. (2007). Identifying apple surface defects using principal components analysis and artificial neural networks. Transactions of the ASABE, 50(6), 2257e2265. ElMasry, G., Wang, N., Vigneault, C., Qiao, J., & ElSayed, A. (2008). Early detection of apple bruises on different background colors using hyperspectral imaging. LWT e Food Science and Technology, 41(2), 337e345. Glasbey, C. A. (1993). An analysis of histogram-based thresholding algorithms. CVGIP e Graphical Models and Image Processing, 55(6), 532e537. Guyer, D., & Yang, X. K. (2000). Use of genetic artificial neural networks and spectral imaging for defect detection on cherries. Computers and Electronics in Agriculture, 29(3), 179e194. Huang, W. Q., Li, J. B., Wang, Q. Y., & Chen, L. P. (2015). Development of a multispectral imaging system for online detection of bruises on apples. Journal of Food Engineering, 146, 62e71. Kapur, J. N., Sahoo, P. K., & Wong, A. K. C. (1985). A new method for gray-level picture thresholding using the entropy of the histogram. Computer Vision, Graphics, and Image Processing, 29(3), 273e285. Kittler, J., & Illingworth, J. (1986). Minimum error thresholding. Pattern Recognition, 19(1), 41e47. Leemans, V., & Destain, M. F. (2004). A real-time grading method of apples based on features extracted from defects. Journal of Food Engineering, 61(1), 83e89. Leemans, V., Magein, H., & Destain, M. F. (2002). On-line fruit grading according to their external quality using machine vision. Biosystems Engineering, 83(4), 397e404. Li, R. (2016). Development of a structured illumination reflectance imaging system for enhanced detection of subsurface and surface defects in apple fruit (Master). East Lansing, MI, USA: Michigan State University. Li, J. B., Huang, W. Q., & Zhao, C. J. (2015). Machine vision technology for detecting the external defects of fruits e a review. Imaging Science Journal, 63(5), 241e251. Li, Q. Z., Wang, M. H., & Gu, W. K. (2002). Computer vision based system for apple surface defect detection. Computers and Electronics in Agriculture, 36(2e3), 215e223. Lu, R. (2003). Detection of bruises on apples using near-infrared hyperspectral imaging. Transactions of the ASAE, 46(2), 523e530. Lu, Y., Li, R., & Lu, R. (2016a). Fast demodulation of pattern images by spiral phase transform in structured-illumination

b i o s y s t e m s e n g i n e e r i n g 1 6 0 ( 2 0 1 7 ) 3 0 e4 1

reflectance imaging for detection of bruises in apples. Computers and Electronics in Agriculture, 127, 652e658. Lu, Y., Li, R., & Lu, R. (2016b). GrameSchmidt orthonormalization for retrieval of amplitude images under sinusoidal patterns of illumination. Applied Optics, 55(25), 6866e6873. Lu, Y., Li, R., & Lu, R. (2016c). Structured-illumination reflectance imaging (SIRI) for enhanced detection of fresh bruises in apples. Postharvest Biology and Technology, 117, 89e93. Lu, Y., & Lu, R. (2016). Quality evaluation of apples. In D. W. Sun (Ed.), Computer vision technology for food quality evaluation (pp. 273e304). Elsevier Inc. Lu, Y., & Lu, R. (2017). Development of a multispectral structuredillumination reflectance imaging (SIRI) system and its application to bruise detection of apples. Transactions of the ASABE. http://dx.doi.org/10.13031/trans.12158. Luo, X., Takahashi, T., Kyo, K., & Zhang, S. (2011). Wavelength selection in vis/NIR spectra for detection of bruises on apples. Journal of Food Engineering, 109, 457e466. Martinsen, P., Oliver, R., Seelye, R., McGlone, V. A., Holmes, T., Davy, M., et al. (2014). Quantifying the diffuse reflectance change caused by fresh bruises on apples. Transactions of the ASABE, 57(2), 565e572. Mizushima, A., & Lu, R. F. (2013). An image segmentation method for apple sorting and grading using support vector machine and Otsu's method. Computers and Electronics in Agriculture, 94, 29e37. Ng, H. F. (2006). Automatic thresholding for defect detection. Pattern Recognition Letters, 27(14), 1644e1649. Otsu, N. (1979). Threshold selection method from gray-level histograms. IEEE Transactions on Systems Man and Cybernetics, 9(1), 62e66. Prewitt, J. M. S., & Mendelsohn, M. L. (1966). The analysis of cell images. Annals of the New York Academy of Sciences, 128, 1035e1053. Ridler, T. W., & Calvard, S. (1978). Picture thresholding using an iterative selection method. IEEE Transactions on Systems Man and Cybernetics, 8(8), 630e632. Rosin, P. L. (2001). Unimodal thresholding. Pattern Recognition, 34(11), 2083e2096.

41

Sahoo, P. K., Soltani, S., Wong, A. K. C., & Chen, Y. C. (1988). A survey of thresholding techniques. Computer Vision, Graphics, and Image Processing, 41(2), 233e260. Sezgin, M., & Sankur, B. (2004). Survey over image thresholding techniques and quantitative performance evaluation. Journal of Electronic Imaging, 13(1), 146e168. Shahin, M. A., Tollner, E. W., McClendon, R. W., & Arabnia, H. R. (2002). Apple classification based on surface bruises using image processing and neural networks. Transactions of the ASAE, 45(5), 1619e1627. Throop, J. A., Aneshansley, D. J., Anger, W. C., & Peterson, D. L. (2005). Quality evaluation of apples based on surface defects: development of an automated inspection system. Postharvest Biology and Technology, 36(3), 281e290. Throop, J. A., Aneshansley, D. J., & Upchurch, B. L. (1995). An image processing algorithm to find new and old bruises. Applied Engineering in Agriculture, 11(5), 751e757. Tsai, W. H. (1985). Moment-preserving thresholding: a new approach. Computer Vision, Graphics, and Image Processing, 29(3), 377e393. Unay, D., & Gosselin, B. (2006). Automatic defect segmentation of ‘Jonagold’ apples on multi-spectral images: a comparative study. Postharvest Biology and Technology, 42(3), 271e279. Weszka, J. S. (1978). A survey of threshold selection techniques. Computer Graphics and Image Processing, 7(2), 259e265. Xing, J., Bravo, C., Jancsok, P. T., Ramon, H., & De Baerdemaeker, J. (2005). Detecting bruises on ‘Golden Delicious’ apples using hyperspectral imaging with multiple wavebands. Biosystems Engineering, 90(1), 27e36. Xing, J., Saeys, W., & De Baerdemaeker, J. (2007). Combination of chemometric tools and image processing for bruise detection on apples. Computers and Electronics in Agriculture, 56(1), 1e13. Xu, X. Y., Xu, S. Z., Jin, L. H., & Song, E. M. (2011). Characteristic analysis of Otsu threshold and its applications. Pattern Recognition Letters, 32(7), 956e961. Zhang, B. H., Huang, W. Q., Gong, L., Li, J. B., Zhao, C. J., Liu, C. L., et al. (2015). Computer vision detection of defective apples using automatic lightness correction and weighted RVM classifier. Journal of Food Engineering, 146, 143e151.