Hyperspectral imaging combined with multivariate analysis and band math for detection of common defects on peaches (Prunus persica)

Computers and Electronics in Agriculture 114 (2015) 14–24

Contents lists available at ScienceDirect

Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag

Hyperspectral imaging combined with multivariate analysis and band math for detection of common defects on peaches (Prunus persica) Baohua Zhang a,b, Jiangbo Li a, Shuxiang Fan a, Wenqian Huang a,⇑, Chunjiang Zhao a,b, Chengliang Liu b, Danfeng Huang b a b

Beijing Research Center of Intelligent Equipment for Agriculture, Beijing Academy of Agriculture and Forestry Sciences, Beijing 100097, China State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai 200240, China

a r t i c l e

i n f o

Article history: Received 24 November 2014 Received in revised form 16 March 2015 Accepted 17 March 2015

Keywords: Hyperspectral imaging Multivariate analysis Band math Common defects Peaches

a b s t r a c t Automatic detection of common defects on peaches by using imaging system is still a challenge due to the high variability of peach surface color, the similarity between the defects and stem, as well as the uneven distribution of lightness on peaches. In order to detect the common defects on peaches using hyperspectral imaging, defects were divided into two different types: artificial defects and non-artificial defects. For artificial defect detection, a two-step multivariate analysis method (Monte CarloUninformative Variable Elimination and successful projections algorithm) was conducted in the spectral domain for the discriminant wavelength (DW) selection, and then minimum noise fraction (MNF) transform was conducted on the images at DWs for image processing and artificial defect detection. For the candidate non-artificial defect detection, a pair of two characteristic wavelengths at 925 nm and 726 nm was selected by analyzing the full spectra of sound and non-artificial defective regions, and then a band math equation was constructed for differentiating the non-artificial defect regions and stems from the sound and physical damage regions, and the candidate non-artificial defects (including non-artificial defects and stems) could be segmented by using a simple threshold method. In order to distinguish the stem from the segmented candidate non-artificial defect regions, another band math equation was constructed based on another pair of two characteristic wavelengths at 650 nm and 675 nm for stem identification. Additionally, the uneven lightness distribution in the spectral images was also investigated and eliminated by the band math methods. The overall classification accuracy of 93.3% for the 120 samples indicated that the selected DWs and proposed method were suitable and efficient for the common defect detection. The limitation of our research is the static inspection in one single view. Ó 2015 Elsevier B.V. All rights reserved.

1. Introduction With the development of optical sensing and imaging techniques, hyperspectral imaging has been developed as a scientific and efficient tool for non-destructive inspection and assessment for quality and safety of a variety of food and agriculture products (Zhang et al., 2014a,b; Lee et al., 2014; ElMasry et al., 2012). A typical hyperspectral image is composed of a set of monochromatic images corresponding to almost continuous wavelengths. Therefore, hyperspectral imaging has the natural advantage to make it possible to conduct a more sophisticate spectral analysis and image processing to extract defect features which are not easy to detect with conventional computer vision system. Hyperspectral applications in quality and safety inspection of fruits include defect ⇑ Corresponding author. Tel.: +86 1051503491; fax: +86 1051503705. E-mail address: [email protected] (W. Huang). http://dx.doi.org/10.1016/j.compag.2015.03.015 0168-1699/Ó 2015 Elsevier B.V. All rights reserved.

detection such as common defect detection on oranges (Li et al., 2011), and defective feature detection in loquats (Yu et al., 2014a,b), fly infestation detection in mangoes (Haff et al., 2013), internally damaged almond nut detection (Nakariyakul and Casasent, 2011), physical damage detection of pears (Lee et al., 2014), bruise and cultivar detection (Siedliska et al., 2014), frass detection on tomatoes (Yang et al., 2014), and crack identification in jujubes (Yu et al., 2014a,b). Peach is a popularly cultivated fruit which is highly favored by consumers worldwide because of its rich nutrition and healthcare benefits. However, peaches are easily damaged by tree branches and insects during their growing. Therefore, common defects such as scars, insect damage, indentation, and spots are always observed. Additionally, physical damage is always observed in the picking and postharvest processing stage. The presence of surface defects is one of the most important sensory attribute of peaches, which could influence their market value, consumers’

B. Zhang et al. / Computers and Electronics in Agriculture 114 (2015) 14–24

preferences and choice (Zhang et al., 2015a; Xiao-bo et al. 2010; Pathare et al., 2013; Zhang et al., 2015b). Therefore, common defect detection of peaches is important and necessary in the postharvest handling and processing step. In our research, we consider to detect the common defects on peaches by using hyperspectral imaging combined with multivariate analysis and band math. In order to detect the defects efficiently, defects on peaches were divided into two categories: one is common defects caused by non-artificial factors including scars, insect damage, indentation, and spot defects, this type of defects presents dark in the color images; the other is physical damage caused by artificial factors, this type of defect is unobvious in the color images or even for human eyes. It is note that there are many common defects such as bruise defect, thin scar, light blemishes that caused by artificial factors. And they present the similar color and texture in appearance with sound tissues. In order to differ from the non- artificial defects, which presented low lightness and were easy to detect, the artificial defects (including bruise defect, thin scar, light blemishes, etc.) were considered to be artificial defects. Actually, for the detection of artificial defects, similar methods would be used, including the wavelength selection methods and image processing methods. In this paper, we just take the very common artificial defect-physical damage into consideration. This would be not comprehensive, but similar methods could be easily applied into other similar applications. Non-artificial defects would be detected by using band math methods at the two characteristic wavelengths selected by analyzing the peaks and valleys in the corrected spectra of defects and sound peel. Physical damage would be detected on the images at the discriminant wavelengths selected by using multivariate analysis in the spectral domain. Various attempts have been made to select the most discriminant wavelengths for external defect detection in the previous researches. The methods include principle components analysis (PCA) (Xing and De Baerdemaeker, 2005; Li et al., 2011; LópezGarcía et al., 2010), partial least squares regression (PLSR) (Yu et al., 2014a,b), stepwise regression (ElMasry et al., 2008) and some other more sophisticated methods such as F values (Cho et al., 2007), receiver operating characteristic (ROC) (Luo et al., 2012), and competitive adaptive reweighted sampling (CARS, Wu and Sun et al., 2013a,b; Yu et al., 2014a,b). Different wavelength selection methods result in different accuracies of the external quality inspection. Actually, for some non-artificial defects, which always represent dark in one or two monochromatic images, according to ‘Occam’s Razor’, it is not necessary to select the most discriminant wavelengths by using the above mentioned methods, and they can even be detected by conventional computer vision systems. However, in order to simplify the complexity of the computation, it is necessary and important to select 4–6 optimal wavelengths that carry the most important information for detection of the unobvious defects such as physical damage (Qin et al., 2013; Liu et al., 2014). In this paper, a pair of two characteristic wavelengths was selected for the band math calculation for the detection of non-artificial defects just according to the difference between the spectra of sound and defect regions on peaches. A two-step multivariate analysis in the spectral domain was used for the most discriminant wavelengths selection for the physical damage detection. In order to distinguish the true defects from the stem, stem was also identified by using band math method in our research. Additionally, the uneven distribution of lightness on the surface of peaches makes it difficult to inspect the defects near to the edge. There are few researches which were focused on the lightness correction for the defect detection on apples, pears and oranges by using hyperspectral imaging system. Therefore, the uneven distribution would also be investigated and corrected for the defect detection on peaches.

15

2. Objectives The specific objective of this paper was to detect the common defects by using hyperspectral imaging combined with multivariate analysis and band math methods. In order to achieve the main objective, several sub-objectives have to be fulfilled: (1) Acquiring the hyperspectral image of peaches in the spectral region of 400–1000 nm by using the hyperspectral imaging system; (2) Selecting the discriminant wavelengths for the artificial defect (physical damage) detection by using a two-step multivariate analysis in spectral domain; (3) Developing a band math method for the uneven lightness correction and the candidate non-artificial defect detection at characteristic wavelengths selected by analyzing the peaks, valleys and trends in the corrected spectra of defect and sound ROIs (regions of interest); (4) Developing a band math method to distinguish stems from true non-artificial defects for the stem identification; (5) Developing and testifying the whole image processing algorithm for the detection of common defects on peaches. 3. Materials and methods 3.1. Peaches and hyperspectral imaging system There are many cultivars of peaches grown in China, ‘Pinggu’ peach with different color varying from white to dark red is one of the most cultivars that are prone to external defects and physical damage, and common defect detection for ‘Pinggu’ peaches is still a challenging work. Thus, ‘Pinggu’ peaches from a local supermarket in Beijing, China, were selected as the experimental samples in our research. One hundred and sixty peach samples consisting 20 sound peaches, 80 peaches with various non-artificial defects (scars, insect damage, indentation, and spots), and 60 peaches with physical damage. Scars are the most common non-infectious defects. The scars present cracks in appearance. Many non-artificial factors can cause the scars in the peach growing season. The most common factor is high temperature from long time sun exposure. Insect damage is caused by insect, which generate surface blemishes and present a low lightness in appearance. Peach indentation is caused by unknown factors. The indentation regions present as shallow pits in the peach peel. Spots are caused by the collision between peaches and leaves or branches during their growing season. The spots commonly appear as deep dark speckle in peach peel. In conclusion, the non-artificial defects present low intensity in the images, the physical damage present almost the same texture and color with the sound tissues. The pictorial view of all defect patterns, stem-ends and sound tissues were shown in Fig. 1. Physical damage was produced in the equator position by using an iron ball (100 g) falling along a plastic pipe with a length of 50 mm, and then the peaches with controlled physical damage were stored in the laboratory at the room temperature (20– 30 °C) for 12 h before acquiring the hyperspectral images. The experimental set was divided into two subsets: one subset containing 20 peaches with various non-artificial defects and 20 peaches with physical damage was used for selecting the discriminant wavelengths, developing band math equations and training the detection algorithm; The other subset containing the rest of the peaches in the experimental set was used for verifying the performance of the developed detection algorithm. The hyperspectral imaging system used in our research consists of the following components: A computer (Dell, Inter(R) Core(TM) i5–2400 CPU @3.10 GHz, RAM 4.0 GB), an Andor monochrome liner EMCCD (Andor Luca DL-604M, Andor Technology plc., N. Ireland) with 1004  1000 pixels, an imaging spectrograph (ImSpector V10E-QE, Spectral Imaging Ltd., Finland) coupled with a standard

16

B. Zhang et al. / Computers and Electronics in Agriculture 114 (2015) 14–24

Fig. 1. The pictorial view of all defect patterns, stem-ends and sound tissues. (a) Sound tissues with red skin. (b) Sound tissues with white-red skin. (c) Physical damage (bruises). (d) Stem. (e) Scars. (f) Insect damage. (g) Indentation. (h) Spots.

Area camera Spectrograph Halogen lamps

Computer Sample Motor

Transportation plate

Fig. 2. The schematic diagram of the developed hyperspectral imaging system.

C-mount zoom lens (V23-f/2.4, Specim Ltd., Finland), two 150 W halogen lamps assemblies (3900-ER, Illumination Technologies, Inc., USA), and a mobile platform. The two halogen lamps were fixed at the both upsides of the mobile platform at an angle of 45 with the height of 40 cm. The spectrograph sampled at 0.772 nm increments and combined to produce a hyperspectral spectra with 1000 spectral samples from 326.7 nm to1098 nm. The detailed information of the hyperspectral imaging system is as following: exposure time: 26 ms, frame rate: 12.4, image size: 1004  1000, fruit speed: 0.8 mm/s, method of orientation: random, effective wavelength range: 400–1000 nm. The schematic diagram of the image acquisition system is shown in Fig. 2. The hyperspectral imaging system worked in reflectance mode and scanned all the samples line by line with an adjustable motor speed of 0.8 mm/s in our research. The distance between samples and camera lens was set to 60 cm. All the samples in the experimental set would be imaged with a single acquisition, thus, a total of one hundred and sixty hyperspectral images were got. It is noted

that the hyperspectral image directly acquired from the system is actually the signal intensity of the uncorrected radiance from the peach. In order to obtain the reflectance image, the hyperspectral reflectance image R for a spatial pixel (i) at a given wavelength was calculated by using the following equation (Lee et al., 2014):

Ri ¼



 RSi  RDi  100% RW i  RDi

ð1Þ

where RS, RD, and RW are the intensity values of identical pixels from the sample image, dark reference image, and white reference image, respectively. Ri is the corrected hyperspectral reflectance image. The dark reference image RD (with 0% reflectance) represents the dark response of the camera, and can be acquired by measuring a spectral image with the light source turned off completely and the camera lens covered completely with its non-reflective opaque black cap. The white reference image RW (with 99.9% reflectance) represents the highest reference intensity values, and can be acquired by measuring a spectral image of the Teflon white

B. Zhang et al. / Computers and Electronics in Agriculture 114 (2015) 14–24

board with a 99.9% reflectance. All the multivariate analysis, band math methods and image processing would be conducted on the hyperspectral reflectance images in this research.

3.2. Multivariate analysis and band math methods In our research, the common defects were detected by using hyperspectral imaging combined with multivariate analysis and band math. In order to reduce the number of the selected wavelengths, different methods were conducted for non-artificial defect and artificial defect detection. To select the discriminant wavelengths that carry the most important information for distinguishing the artificial defect (physical damage) from the sound tissues, a two-step multivariate analysis was conducted in the spectral domain of the hyperspectral images. For the non-artificial defect (scars, insect damage, indentation, and spots) detection, only two images at the selected pair of two characteristic wavelengths were used for constructing the band math equation. Spectral analysis for selecting discriminant wavelengths for physical damage detection was conducted in spectral domain. Both the average spectra of the physical damage and sound ROIs were collected by averaging the all the pixels in a rectangular region. Each region contains about 60–100 pixels. About five average spectra of both physical damage and sound ROIs were extracted for each peach (with physical damage) in the training set. A total of 210 average spectra were collected. While the average spectra were in the range of 326.7 nm to 1098 nm, only the spectra between 400 nm to 1000 nm were taken into account for the spectral analysis considering the single to noise ratios. All the 210 average spectra would be labeled as 0 (physical damage tissue class, 105 spectra) or 1 (sound tissue class, 105 spectra) according to their true classes. The 210 average spectra would be divided into two subsets: one subset containing 160 spectra (80 for physical damage tissue class and 80 for sound tissue class) were used for training, and the other subset containing 50 spectra (25 for physical damage tissue class and 25 for sound tissue class) were used for validation. The Monte Carlo-Uninformative Variable Elimination (MC-UVE) is a powerful multivariate analysis for analyzing complex multivariable problems (Li et al., 2014). MC-UVE uses the stability defined in UVE method to evaluate the reliability of each variable, but the stability values are obtained through the Monte Carlo method replacing the leave-one-out procedure in UVE. Moreover, instead of adding random noise variables to the original data matrix as in UVE method to estimate the cutoff threshold, the wavelengths to be selected are determined directly by their stability, which is more convenient (Cai et al., 2008). More details about MC-UVE can be found in reference Li et al., 2014; Cai et al., 2008; Wu et al., 2013. The successful projections algorithm (SPA), a forward selection method which uses simple operations in a vector space to minimize variable collinearity, is a novel variable selection strategy in hyperspectral image analysis for multivariate calibration (Wu et al., 2013). SPA employs a simple projection operation in a vector space to select subsets of variables with a minimum of colinearity. Generally, SPA comprises two phases. The first phase consists of projections carried out on the spectral matrix, which generate candidate subsets of variables with minimum colinearity. The second phase consists of evaluating candidate subsets of variables according to the RMSEV (root mean square error of variation) value obtained by applying the resulting MLR (multi-linear regression) model. Then, variable elimination procedures can be used to remove uninformative variables without significant loss of prediction capability (Galvao et al., 2008; Liu et al., 2014). More details about SPA method can be found in the above mentioned references. In our research, a combination of MC-UVE and SPA is used

17

for multivariate analysis in Vis/NIR spectral domain to select the discriminant wavelengths for physical damage detection. Band math methods, including addition, subtraction, multiplication and division calculation between two characteristic wavelengths, are efficient analytical methods in the spectral analysis and image processing. Band math methods could reduce the noise and non-uniform distribution of lightness on the fruits to some extent due to the calculation between intensity of the corresponding pixels in the same position and the influence caused by shape are similar in the same position. Band ration method was always used for the external quality related character detection, such as common defect detection on citrus and apples, in the previous studies (Yang et al., 2014; Li et al., 2011). In this paper, a combination of band subtraction and division were firstly conducted on two images for candidate non-artificial defect detection. Additionally, in order to distinguish the true defects from the stem, stem was also identified by using band math method in our research. The band math equation in our research would be constructed as follows:

RBM ði; jÞ ¼

Rk1 ði; jÞ  Rk2 ði; jÞ  255 Rk1 ði; jÞ

ð2Þ

where the Rk1 and Rk2 are two reflectance images at k1 and k2 , respectively. RBM is the result image after band math calculation. (i, j) represents the location of the calculated pixel. It is note that, if Rk1  Rk2 < 0, the value of Rk1  Rk2 would be coerced into 0 considering the characteristics of digital image data stored in computers. According to the Eq. (2), the greater difference between two corresponding pixels in the two images, the higher intensity value of that pixel would be, and the better discriminant between sound and defective tissues would be. Additionally, the similar influence (intensity) in each of the corresponding pixels and wavelengths caused by geometrical shape (spherical surface) are always similar, Eq. (2) can also be used for the uneven lightness correction. The even lightness distribution is very important and helpful for the subsequent image processing and defect classification. 3.3. Whole image processing algorithm for common defect detection In our study, the final common defect detection on peaches was developed by using hyperspectral imaging combined with multivariate analysis and band math methods. The flowchart of the whole detection algorithm is shown in Fig. 3. The whole detection algorithm mainly includes the following several parts: (1) Artificial defect (physical damage) detection, including: Spectra collection from the ROIs of sound and physical damage regions, multivariate analysis for the discriminant wavelength selection by using MC-UVE and SPA methods, image processing for the physical damage classification by using MNF transform based on images at the selected wavelengths; (2) Non-artificial defect (scars, insect damage, indentation, and spot) detection, including: Spectra collection from the ROIs of various defect regions, spectral analysis, a pair of two characteristic wavelengths selection, band math method development, image processing in the band math result image for non-artificial defect segmentation; (3) Stem identification, including: Spectra collection from the ROIs of stem regions, spectral analysis, band math method development, image processing for stem identification; (4) Combination common defect detection algorithm development. Specially, in order to make it better to understand for the general reader, the image processing algorithm would be concluded in Fig.11 in Section 4. Considering the image processing algorithm for artificial defect detection is easy to understand, only the image processing algorithm for non-artificial defect detection would be presented in detailed. The algorithm mainly contains two parts:

18

B. Zhang et al. / Computers and Electronics in Agriculture 114 (2015) 14–24

Hyperspectral image acquisition and calibration

Sub-objectives

Multivariate analysis

Artificial defect detection 774λ

Non-artificial defect detection 774λ

Stem identification 774λ

Spectra collection from ROIs

Spectra collection from ROIs

Spectra collection from ROIs

Discriminant wavelength selection by using MC-UVE and SPA

Spectral analysis, band selection and band math method development

Spectral analysis, band selection and band math method development

2λ

4λ Image Processing

1000λ

MNF transform based on DWs for physical damage detection

2λ

Band math and image processing for non-artificial defect detection

Band math and image processing for stem identification 8λ

Common defect detection Fig. 3. Flowchart of the whole detection algorithm.

Candidate defect detection, stem recognition and final defect classification. Some common image processing methods such as background removal, median filter, masking, band ratio methods would be combined and applied for defect detection. 4. Results and discussion 4.1. Average spectra collected for different peel condition Representative average spectra for each of the common defects (physical damage, scars, insect damage, indentation, and spot) and sound ROIs were extracted by using ENVI 4.6 and were shown in Fig. 4. From Fig. 4, the average spectra of physical damage and sound ROIs shared the similar trends and were very close in the full spectral range. This can be explain why it is very difficult to distinguish the physical damage from the sound tissues. Spectra of scars,

Scar 3200

Sound Spot Insect damage

Indentation Physcial damage

Stem

Rleative reflectance intensity

2800 2400 2000 1600 1200 800 400 0 400 450 500 550 600 650 700 750 800 850 900 950 1000

Wavelength/nm Fig. 4. Representative average spectra for the common defect, stem and sound ROIs.

insect damage, indentation, and spots have the similar curve variation tendencies expect the difference in the intensity values. The spectral curves of the non-artificial defects climbed up as the increasing of the wavelength from 500 nm to 925 nm, and they came to a peak at the 925 nm, then falling slowly from 925 nm to 1000 nm. The spectral character of the stem was different from the sound and defect tissues, it had a spectral peak at the 925 nm, which was different from that of sound and physical damage tissues, it had a spectral peak at the 650 nm and a valley at the 675 nm, which was different from that of non-artificial defects. The difference above mentioned provided basis for constructing band math methods for defect detection and stem identification.

4.2. Artificial defect detection According to the analysis for the spectral character of sound and physical damage in the Section 4.1, there is no obvious difference between spectral features of sound and physical damage. In order to remove the redundancy of hyperspectral image data, simplify the complexity of information, and select the discriminant wavelengths which carry the most important information for distinguishing the physical damage from sound tissues, a two-step multivariate analysis method (MC-UVE and SPA) was used for spectral analysis in spectral domain. The first step for discriminant wavelength selection was conducted by using MC-UVE method. Fig. 5(a) shows the stability of each variable in the spectral region of 400 nm to 1000 nm for the physical damage classification by using MC-UVE. In the Fig. 5(a), the black dot lines represents the cutoff, which is determined by a number N of the informative variables (Li et al., 2014). Variables whose stability lies between the two black dot lines will be eliminated, and the others whose stability lies out of the two black dot lines will be preliminarily selected as the discriminant wavelengths for the PLS calculation (Cai et al., 2008). For each variable number N, a PLS model is developed and used to predict the sample set. Therefore, the change relationship between the RMSEP values and variable number N is investigated and illustrated in Fig. 5(b). As shown in Fig. 5(b), the RMSEP is large

19

B. Zhang et al. / Computers and Electronics in Agriculture 114 (2015) 14–24

6

0.34

0.32

4

RMSEP

Stability

0.3

cutoff

2

0

0.28

0.26

-2

cutoff

N=440, RMSEP= 0.2119

0.24

-4

0.22

-6 400

0.2 474

550

627

704

783

826

941 1000

0

100

200

Wavelength (nm)

300

400

500

600

700

800

Number of variables

(a)

(b)

Fig. 5. Intermediate process results of discriminant wavelength selection by using MC-UVE method in the first step. (a) The stability of each wavelength for the physical damage classification. (b) The change of RMSEP with number of selected wavelengths for physical damage classification.

3000 First calibration object Selected variables

2500 Final number of selected variables: 4 (RMSE = 0.25711)

Rleative reflectance intensity

0.55 0.5

RMSE

0.45 0.4 0.35 0.3

2000

1500

1000

500

0.25 0.2

0

5

10

15

20

25

30

Number of variables included in the model

(a)

0

400

474

550

627

704

783

826

941 1000

Variable index

(b)

Fig. 6. Process results of discriminant wavelength selection by using SPA method in the second step. (a) The RMSE screen plot for the number of selected discriminant wavelengths. (b) The selected 4 wavelengths and their distribution in the calibration spectra.

at the beginning, and then decrease sharply with the increase of N. When N = 440, the lowest value of RMSEP is obtained. When N is larger than 440, the value of RMSEP is increasing with the increase of N. Therefore, four hundred and forty wavelengths were preliminarily selected as the discriminant wavelengths for distinguishing the physical damage from the sound tissues. After the first selection step, almost half of the wavelengths were eliminated as the uninformative variables. In the second step, SPA was used to refine the preliminarily selected discriminant wavelengths so as to make it possible to integrate the final selected wavelengths into a multispectral imaging system for online or real time inspection. Fig. 6(a) shows the RMSE (obtained by applying SPA) screen plot for the number of selected discriminant wavelengths increasing from 0 to 28. As shown in the Fig. 6(a), the RMSE values sharply decreased as the selected variables increasing from 1 to 2, the RMSE value increased when the selected wavelengths increase to 3. Considering the contradictoriness between

RMSE value and the number of the selected variables, the RMSE value climbed down to an optimal value (marked with an open square marker) when the number of selected wavelengths is 4. After the number of the selected wavelengths greater than 4, the descending trend was not obvious. The selected 4 discriminant wavelengths were marked with open square markers in the full region spectra in Fig. 6(b). After the two step multivariate analysis, four wavelengths (621, 713, 821, and 987 nm) were finally selected as the candidate discriminant wavelengths for distinguishing the physical damage from sound tissues. Their distribution in the calibration spectra were shown in the Fig. 6(b). The four selected wavelengths might be efficient for the distinguishing the two categories in the spectral domain because the multivariate analysis was conducted on the spectra collected from the ROIs of the sound and damaged tissues, whether they were still efficient in the spatial (image) domain was needed to be testified. In order to verify the detection performance

20

B. Zhang et al. / Computers and Electronics in Agriculture 114 (2015) 14–24

of the images at the four selected wavelengths, MNF transform was conducted on the images at the four discriminant wavelengths for physical damage detection. The MNF transform results and physical damage classification results were shown in Fig. 7. As shown in the pseudo RGB color images, the physical damages (marked with blue ellipses) are very difficult to detect even by human eyes due to the high similarity in color and texture between the damage and sound tissues. MNF1 images mainly illustrated the illumination and physical properties. MNF2 images mainly illustrated the shadows and the areas where the illumination was saturated. MNF3 images mainly illustrated the textural information. MNF4 images provided the best discriminant information for distinguishing damage regions from sound tissues. As shown in the classification images, the physical damage regions were successfully segmented from the sound tissues, but some sound tissues were falsely classified as physical damage regions. Considering the high variability of peach surface color and the soft sarcocarp, the classification results were satisfied. Actually, according the characteristic of the physical damage, the small areas which were falsely segmented would be removed by particle removal operation. It is note that the physical damages were produced 12 h before the hyperspectral image acquisition. The method and selected wavelengths were not so efficient for the detection of the damages produced less than 12 h. MNF transform on the four images is conducted by using ENVI 4.6, the transform needs about 5 s, the subsequent image processing for physical segmentation and classification needs 20–30 ms. The time consuming would largely reduce after we developing a real time multispectral imaging system due to the four images could be directly acquired.

4.3. Candidate non-artificial defect detection In order to reduce the impacts of uneven distribution of the lightness caused by the geometrical shape of peaches, and improve the efficient of defect detection and reduce the number of the selected wavelengths, band math method was constructed by using only two wavelengths for candidate non-artificial defect (including non-artificial defects and stems) detection. According to the representative average spectra for each of non-artificial defects, stem, artificial defect and sound tissues, two spectral characteristics, observable in Fig. 4, were identified as being potentially useful for distinguishing the candidate non-artificial defects from sound and physical damage tissues. The first spectral characteristic was the 726 nm spectral peak exhibited by the average spectra of

the sound and physical damage tissues. The other spectral characteristic was the 925 nm spectral peak exhibited by the average spectra of the stem, and various non-artificial defect tissues. The spectral reflectance values of the sound and physical damage tissues were higher than that of the stem, and various non-artificial defect tissues, and kept a fluctuation within a narrow range from 726 nm to 850 nm, and then decreased from 850 nm to 975 nm. However, the spectral reflectance values of the stem, and various non-artificial defect tissues climbed up from 675 nm to 925 nm, and then came to peaks about at the 925 nm. Therefore, a band math equation using the images at 925 nm and 726 nm was constructed as following:

R1 ði; jÞ ¼

R925 ði; jÞ  R726 ði; jÞ  255 R925 ði; jÞ

ð3Þ

where R1 was the band math result image for distinguishing stem and various non-artificial defects from sound and physical damage tissues. R925 and R726 were the images at the 925 and 726 nm wavelengths respectively. (i, j) represents the location of the calculated pixel. For the sound and physical damage tissues, due to the descending of the spectral reflectance values from 850 nm wavelength, R925 ði; jÞ  R726 ði; jÞ < 0, the R1(i, j) would be mandatorily pulled down to 0 (dark in the images) according to the rule made in the Section 3.2. For the stem and various non-artificial defect tissues, due to the increasing of the spectral reflectance values from 675 nm to 925 nm, R925 ði; jÞ  R726 ði; jÞ > 0, therefore, the higher the R1(i, j) value was, the greater the possibility of the (i, j) pixel being the part of a stem or non-artificial defect. The uneven lightness distribution on the peaches could also be corrected by the band math method. Fig. 8(a) is the monochromatic image extracted from the background removal hyperspectral image, Fig. 8(b) is the band math result image by using Equation (3), Fig. 8(c) is the spatial lightness profile for the red line transecting the defects in the monochromatic image in Fig. 8(a), and (d) is the spatial lightness profile for the red line transecting the defects in the band math result image in Fig. 8(b). As shown in Fig. 8(a) and (c), the lightness distribution is uneven on the surface of peach, the point in the edge area having a lower value than that of the point in the central area, even the lightness of the defects. The fact is even more apparent and intuition in Fig. 8(c), it can be used to explain how the spherical surface affects the distribution of the lightness. The edge area could be wrongly classified as the defects when conduct segmentation directly in monochromatic image with a simple threshold value 95. As shown in

RGB image

MNF1

MNF2

MNF3

MNF4

Physical damage

RGB image

MNF1

MNF2

MNF3

MNF4

Physical damage

Fig. 7. The MNF transform results and physical damage classification results.

B. Zhang et al. / Computers and Electronics in Agriculture 114 (2015) 14–24

(a)

(b)

Edge Intensity

21

Threshold=95

Background Defects Threshold=25

Defects

Sound tissues Pixel

Pixel

(c)

(d)

Fig. 8. Lightness correction results by using band math method. (a) Monochromatic image extracted from the background removal hyperspectral image. (b) Band math result image by using Eq. (3). (c) Spatial lightness profile for the red line transecting the defects in the monochromatic image in Fig. 5(a). (d) Spatial lightness profile for the red line transecting the defects in the band math result image in Fig. 5(b). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 8(b) and (d), the lightness correction result is encouraging, the defects were in higher intensity, and the sound tissue (including the sound tissues in the very edge area) is stretched down to very low intensity, this makes it more easier to segment the defects accurately by using a simple global threshold value of 25 in the band math result image than directly segmenting in the monochromatic image. In order to verify the defect detection performance of the band math method by using Equation (3), different size peaches with common defects in different regions were selected. The classification results for different non-artificial defect detection were shown in Fig. 9. The pseudo RGB color images are shown in Row (a). As shown in the Row (a), the tested peaches have different sizes and present different postures, the defects distribute randomly, some defects distribute at the edges of the peaches. In order to highlight the detection performance of the proposed method, results of defect segmented directly in monochromatic image at 726 nm are also shown in Row (b). It is important to note that the real boundary of each peach is added to the defect segmentation image to highlight the false segmentations in the edge regions due to the uneven distribution of the lightness. As shown in Row (b), some defects are not segmented accurately, and the edge parts of the peaches are wrongly segmented as defects due to their low intensity. Band math results are shown in Row (c). From Row (c), the defects have relative high intensity due to R925 ði; jÞ  R726 ði; jÞ > 0, but the sound tissues, including the sound tissues in the edge, are coerced into very low intensity. This makes it very easy to

segment the defects accurately only by using a simple threshold value of 25. The detection results by using band math method are shown in Row (d). According to the detection results, the Equation (3) was very efficient for distinguishing the stems and non-artificial defects from the sound and physical damage tissues. The time consuming is 20–40 ms, this can satisfy the requirement of the online inspection. It was clear that the defect detection performance based on Eq. (3) was very satisfied. However, it could not distinguish the stems from non-artificial defects due to their similar spectral trends from 726 nm to 925 nm. 4.4. Stem identification After applying the Eq. (3), stems and non-artificial defects were distinguished from sound and physical damage tissues. In order to distinguish the stems from non-artificial defects, another band math equation need to be constructed. The spectral peak and valley of stem at 650 nm and 675 nm were selected as the most important spectral characteristics that were potentially efficient for distinguishing the stems and non-artificial defects due to their difference trends between 650 nm and 675 nm wavelengths. The spectral reflectance values of stem had one peak at 650 nm and one valley at 675 nm. The spectral reflectance value at 650 nm was higher than that of 675 nm as well. However, for the nonartificial defects, due to the increasing of the spectral reflectance values from 650 nm, the spectral reflectance value at 650 nm

22

B. Zhang et al. / Computers and Electronics in Agriculture 114 (2015) 14–24

Fig. 9. The classification results for different non-artificial defect detection. (a) Pseudo color images for the peaches with various defects. (b) Defect segmentation directly in the monochromatic image at 726 nm. (c) Band math result images. (d) Detection results of common defects by using band math method.

was lower than or equal to that of 675 nm. Therefore, a band math equation using the images at 650 nm and 675 nm wavelengths was constructed as following:

R2 ði; jÞ ¼

R650 ði; jÞ  R675 ði; jÞ  255 R650 ði; jÞ

ð4Þ

where R2 was the band math result image for distinguishing stem from various non-artificial defects. R650 and R675 were the images at the 650 nm and 675 nm respectively. (i, j) represents the location of the calculated pixel. For the pixels of the stem, R650 ði; jÞ  R675 ði; jÞ > 0, and for pixels of the non-artificial defects, R650 ði; jÞ  R675 ði; jÞ  0. Therefore, the higher the R2(i, j) value was, the greater the possibility of the (i, j) pixel being the part of a stem.In order to verify the stem identification performance of the band math method by using Equation (4), different size peaches with common defects and stems in different regions were selected. The identification results for stem detection were shown in Fig. 10. It is note that the stems and non-artificial defects have been segmented in Section 4.3, the Eq. (4) is just applied to the pixels which have been segmented in the Section 4.3 for distinguishing the stems from true non-artificial defects. This would save time compared to applying Eq. (4) to all pixels in the images. The pseudo RGB color images are shown in Row (a). Band math results by using Equation (3) are shown in Row (b). As shown in the Row (b), the stems and non-artificial defects have relative high intensity due to R925 ði; jÞ  R726 ði; jÞ > 0, and the sound tissues are coerced into very low intensity. Both the stems and non-artificial defects are segmented by using a simple threshold value of 25, and the segmentation results are shown in Row (c). As shown in the Row (c), both the stems and non-artificial defects (even in the edge) are successfully segmented. Then, the Eq. (4) is applied to the pixels

segmented in the images in Row (c), and the pixels with an intensity value lower than 10 would be considered as defects, and would be removed from the images in Row (c), the stem identification results are shown in Row (d). It is worth noticing that not all the pixels belonging to the true stems are accurately identified as stem due to the various types of stem tissues. If a stem area identified is larger than one third of connected region which it overlapped in the images in Row (c), the entire corresponding connected region would be considered as the stem, and would be removed. Some false classification cases are also observed when the defect is connected with the stem such as the case of peach in the first line. The final defect classification results are shown in Row (e). According to the detection results, the Eq. (4) could be used to distinguish the stems from non-artificial defects. In order to make it more clear, the flowchart of the image processing algorithm for the detection of common defects is shown in Fig. 11. 4.5. Detection results of the whole algorithm All the 120 samples in the testing set were processed by the whole detection algorithm to verify the detection performance of the developed image processing algorithm by using hyperspectral imaging combined with multivariate analysis and band math methods. Table 1 shows the classification results by using the proposed algorithm. 35 samples of the 40 peaches with physical damage were classified as artificial defective peaches correctly, the accuracy was 87.5%. It was found that the physical damage of the 3 misclassified peaches in the artificial defect class was very slight, and the damage region kept the similar peel condition as the sound ones. This indicated that the proposed algorithm was

B. Zhang et al. / Computers and Electronics in Agriculture 114 (2015) 14–24

23

Fig. 10. The identification results for stem detection. (a) Pseudo color images for the peaches with various defects and stems. (b) Band math result images. (c) Segmentation results of common defects and stems by using band math method. (d) Stem identification results of stems by using Eq. (4). (e) Final defect detection results.

Images at 726 and 925 nm acquisition

Images at 650 and 675 nm acquisition Masking

Band math between two images by using Equation (3)

Band math between two images by using Equation (4)

Mask image acquisition from image at 925 nm

Segmentation on the band math result images

Background removal

Subtraction

Segmentation on Median Candidate defect the band math result region segmentation Filter and classification images Candidate defect segmentation

Final defect region segmentation and classification Stem recognition and final defect classification

Fig. 11. The overall view of the main image processing operation for common defect detection.

less efficient for the detection of very slight physical damage on peaches. The damage regions of the other 2 misclassified peaches in the artificial defect class in the MNF4 presented white, the physical damage regions were not segmented correctly. Considering the high variability of peach surface color and the soft sarcocarp, the classification results were satisfied. 58 samples of the 60 peaches with various non-artificial defects were classified as non-artificial defective peaches correctly, the accuracy was

96.7%. One misclassified peach in the non-artificial defect class was misclassified as sound peach due to the connection between the true defect and stem, the true defect was wrongly removed together with the stem. The other misclassified peach in the nonartificial defect class was misclassified as sound peach due to the true defect was wrongly identified as stem. 19 samples of the 20 peaches without defects were classified as sound peaches correctly, the accuracy was 95%. The misclassified peach in the

24

B. Zhang et al. / Computers and Electronics in Agriculture 114 (2015) 14–24

Table 1 Classification results.

References

Class

Peel condition

Artificial defect

Physical damage Scar Insect damage Indentation Spot No defects –

Non-artificial defect

Sound Total

Number

Correct

Incorrect

Accuracy (%)

40

35

5

87.5

60

58

2

96.7

20 120

19 112

1 8

95 93.3

sound class was misclassified as defective peach due to the stem was wrongly identified as defect. The overall classification accuracy of 93.3% indicated that the selected discriminant wavelengths and proposed method were suitable and efficient for the common defect detection. 5. Conclusions In our study, the common defects are divided into artificial defect (physical damage) and non-artificial defects (including scars, insect damage, indentation, and spot). In order to reduce the number of the selected wavelengths and increase the detection performance of different type defects, different methods were developed for detecting different type defects. For artificial defect detection, a combination of two step multivariate analysis was used for discriminant wavelengths selection. And, MNF was conducted on the images at the four selected wavelengths for image processing and physical damage segmentation. For non-artificial defect detection, a band math equation was constructed only by using two band images at the two characteristic spectra. Images at 925 nm and 726 nm were selected for constructing band math equation for non-artificial defect detection. In order to distinguish the stem from true defects, another band math equation was constructed. Additionally, the uneven lightness distribution on peaches caused by the spherical shape was also investigated in this paper, and the lightness was corrected by using the band math method. The lightness correction result was encouraging, and the defects could be easily segmented by using a single global threshold value in the corrected band math images. All the 120 samples in the testing set were processed by the whole detection algorithm, the overall detection accuracy with 93.3% indicated that the selected wavelengths and proposed method were suitable and efficient for the detection of common defects on peaches. Although the satisfied detection performance was got in laboratorial experiment, there would be more interesting and challenging work to be done to implement the algorithm into an automated inspection system to realize the fast whole surface detection required by industry. The limitation of our research is the static inspection in single view. Future work will be focused on high speed real time inspection by using multispectral imaging system combining with the proposed method. Acknowledgements This research was supported by the National Key Technology R&D Program (project no. 2014BAD21B01) and Young Scientist Fund of National Natural Science Foundation of China (project no. 31301236). The first author would also like to thank his grandmother Guirong Liu for her upbringing, and in loving memory of his grandfather Liugen Zhang.

Cai, W., Li, Y., Shao, X., 2008. A variable selection method based on uninformative variable elimination for multivariate calibration of near-infrared spectra. Chemometr. Intell. Lab. Syst. 90 (2), 188–194. Cho, B.-K., Chen, Y.-R., Kim, M.S., 2007. Multispectral detection of organic residues on poultry processing plant equipment based on hyperspectral reflectance imaging technique. Comput. Electron. Agric. 57 (2), 177–189. Elmasry, G., Kamruzzaman, M., Sun, D.W., Allen, P., 2012. Principles and applications of hyperspectral imaging in quality evaluation of agro-food products: a review. Crit. Rev. Food Sci. Nutr. 52 (11), 999–1023. ElMasry, G., Wang, N., Vigneault, C., Qiao, J., ElSayed, A., 2008. Early detection of apple bruises on different background colors using hyperspectral imaging. LWT-Food Sci. Technol. 41 (2), 337–345. Galvao, R.K.H., Araujo, M.C.U., Fragoso, W.D., Silva, E.C., Jose, G.E., Soares, S.F.C., Paiva, H.M., 2008. A variable elimination method to improve the parsimony of MLR models using the successive projections algorithm. Chemometr. Intell. Lab. Syst. 92 (1), 83–91. Haff, R.P., Saranwong, S., Thanapase, W., Janhiran, A., Kasemsumran, S., Kawano, S., 2013. Automatic image analysis and spot classification for detection of fruit fly infestation in hyperspectral images of mangoes. Postharvest Biol. Technol. 86, 23–28. Lee, W.-H., Kim, M.S., Lee, H., Delwiche, S.R., Bae, H., Kim, D.-Y., Cho, B.-K., 2014. Hyperspectral near-infrared imaging for the detection of physical damages of pear. J. Food Eng. 130, 1–7. Li, J., Huang, W., Chen, L., Fan, S., Zhang, B., Guo, Z., Zhao, C., 2014. Variable selection in visible and near-infrared spectral analysis for noninvasive determination of soluble solids content of ‘Ya’Pear. Food Anal. Methods, 1–12. Li, J.B., Rao, X.Q., Ying, Y.B., 2011. Detection of common defects on oranges using hyperspectral reflectance imaging. Comput. Electron. Agric. 78 (1), 38–48. Liu, D., Sun, D.-W., Zeng, X.-A., 2014. Recent advances in wavelength selection techniques for hyperspectral image processing in the food industry. Food Bioprocess Technol. 7 (2), 307–323. López-García, F., Andreu-García, G., Blasco, J., Aleixos, N., Valiente, J.-M., 2010. Automatic detection of skin defects in citrus fruits using a multivariate image analysis approach. Comput. Electron. Agric. 71 (2), 189–197. Luo, X., Takahashi, T., Kyo, K., Zhang, S., 2012. Wavelength selection in vis/NIR spectra for detection of bruises on apples by ROC analysis. J. Food Eng. 109 (3), 457–466. Nakariyakul, S., Casasent, D.P., 2011. Classification of internally damaged almond nuts using hyperspectral imagery. J. Food Eng. 103 (1), 62–67. Pathare, P.B., Opara, U.L., Al-Said, F.A.-J., 2013. Colour measurement and analysis in fresh and processed foods: a review. Food Bioprocess Technol. 6 (1), 36–60. Qin, J., Chao, K., Kim, M.S., Lu, R., Burks, T.F., 2013. Hyperspectral and multispectral imaging for evaluating food safety and quality. J. Food Eng. 118 (2), 157–171. Siedliska, A., Baranowski, P., Mazurek, W., 2014. Classification models of bruise and cultivar detection on the basis of hyperspectral imaging data. Comput. Electron. Agric. 106, 66–74. Wu, D., Sun, D.-W., 2013a. Application of visible and near infrared hyperspectral imaging for non-invasively measuring distribution of water-holding capacity in salmon flesh. Talanta 116, 266–276. Wu, D., Sun, D.-W., 2013b. Potential of time series-hyperspectral imaging (TS-HSI) for non-invasive determination of microbial spoilage of salmon flesh. Talanta 111, 39–46. Wu, D., Wang, S., Wang, N., Nie, P., He, Y., Sun, D.-W., Yao, J., 2013. Application of time series hyperspectral imaging (TS-HSI) for determining water distribution within beef and spectral kinetic analysis during dehydration. Food Bioprocess Technol. 6 (11), 2943–2958. Xiao-bo, Z., Jie-wen, Z., Yanxiao, L., Holmes, M., 2010. In-line detection of apple defects using three color cameras system. Comput. Electron. Agric. 70 (1), 129– 134. Xing, J., De Baerdemaeker, J., 2005. Bruise detection on ‘Jonagold’apples using hyperspectral imaging. Postharvest Biol. Technol. 37 (2), 152–162. Yang, C.-C., Kim, M.S., Millner, P., Chao, K., Cho, B.-K., Mo, C., Lee, H., Chan, D.E., 2014. Development of multispectral imaging algorithm for detection of frass on mature red tomatoes. Postharvest Biol. Technol. 93, 1–8. Yu, K., Zhao, Y., Li, X., Shao, Y., Zhu, F., He, Y., 2014a. Identification of crack features in fresh jujube using Vis/NIR hyperspectral imaging combined with image processing. Comput. Electron. Agric. 103, 1–10. Yu, K.-Q., Zhao, Y.-R., Liu, Z.-Y., Li, X.-L., Liu, F., He, Y., 2014b. Application of visible and near-infrared hyperspectral imaging for detection of defective features in loquat. Food Bioprocess Technol., 1–11 Zhang, B., Fan, S., Li, J., Huang, W., Zhao, C., Qian, M., Zheng, L., 2015a. Detection of early rottenness on apples by using hyperspectral imaging combined with spectral analysis and image processing. Food Anal. Methods, 1–12. Zhang, B., Huang, W., Gong, L., Li, J., Zhao, C., Liu, C., Huang, D., 2015b. Computer vision detection of defective apples using automatic lightness correction and weighted RVM classifier. J. Food Eng. 146, 143–151. Zhang, B., Huang, W., Li, J., Zhao, C., Fan, S., Wu, J., Liu, C., 2014a. Principles, developments and applications of computer vision for external quality inspection of fruits and vegetables: a review. Food Res. Int. 62, 326–343. Zhang, B., Li, J., Fan, S., Huang, W., Zhang, C., Wang, Q., Xiao, G., 2014b. Principles and applications of hyperspectral imaging technique in quality and safety inspection of fruits and vegetables. Spectrosc. Spect. Anal. 34 (10), 2743–2751.