- Email: [email protected]
Discrimination methods for biological contaminants in fresh-cut lettuce based on VNIR and NIR hyperspectral imaging
Accepted Manuscript Regular article Discrimination methods for biological contaminants in fresh-cut lettuce based on VNIR and NIR hyperspectral imaging Changyeun Mo, Giyoung Kim, Moon S. Kim, Jongguk Lim, Seung Hyun Lee, Hongsek Lee, Byoung-Kwan Cho PII: DOI: Reference:
S1350-4495(17)30065-8 http://dx.doi.org/10.1016/j.infrared.2017.05.003 INFPHY 2291
To appear in:
Infrared Physics & Technology
Received Date: Revised Date: Accepted Date:
1 February 2017 3 May 2017 7 May 2017
Please cite this article as: C. Mo, G. Kim, M.S. Kim, J. Lim, S. Hyun Lee, H. Lee, B-K. Cho, Discrimination methods for biological contaminants in fresh-cut lettuce based on VNIR and NIR hyperspectral imaging, Infrared Physics & Technology (2017), doi: http://dx.doi.org/10.1016/j.infrared.2017.05.003
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Discrimination methods for biological contaminants in fresh-cut lettuce based on VNIR and NIR hyperspectral imaging Changyeun Mo a, Giyoung Kim a, Moon S. Kim b, Jongguk Lim a, Seung Hyun Lee c, Hongsek Lee a, and Byoung-Kwan Cho c,* a
b
c
National Institute of Agricultural Sciences, Rural Development Administration, 310 Nonsaengmyeong-ro, Wansan-gu, Jeonju-si, Jeollabuk-do 54875, Republic of Korea Environmental Microbial and Food Safety Laboratory, BARC-East, Agricultural Research Service, US Department of Agriculture, 10300 Baltimore Avenue, Beltsville, MD 20705, USA Department of Biosystems Machinery Engineering, College of Agricultural and Life Science, Chungnam National University, 99 Daehak-ro, Yuseong-gu, Daejeon 34134, Republic of Korea
* Corresponding author ; E-mail address: [email protected] (B.K. Cho) Tel.: +82-42-821-6715; Fax: +82-42-823-6246.
Abstract The rapid detection of biological contaminants such as worms in fresh-cut vegetables is necessary to improve the efficiency of visual inspections carried out by workers. Multispectral imaging algorithms were developed using visible-near-infrared (VNIR) and near-infrared (NIR) hyperspectral imaging (HSI) techniques to detect worms in fresh-cut lettuce. The optimal wavebands that can detect worms in fresh-cut lettuce were investigated for each type of HSI using one-way ANOVA. Worm-detection imaging algorithms for VNIR and NIR imaging exhibited prediction accuracies of 97.00% (RI547/945) and 100.0% (RI1064/1176, SI1064-1176, RSI-I(1064-1173)/1064, and RSI-II(1064-1176)/(1064+1176)), respectively. The two HSI techniques revealed that spectral images with a pixel size of 1 × 1 mm or 2 × 2 mm had the best classification accuracy for worms. The results demonstrate that hyperspectral reflectance imaging techniques have the potential to detect worms in fresh-cut lettuce.
Future research relating to this work will focus on a real-time sorting system for lettuce that can simultaneously detect various defects such as browning, worms, and slugs. Keywords: VNIR hyperspectral imaging, NIR hyperspectral imaging, biological contaminant, discrimination, fresh-cut lettuce
1. Introduction The improvement in the standard of living and the rising interest among consumers to live a healthy life has led to increased consumption of high-quality and safe agro-food. In addition, the consumption of untreated fresh vegetables and fruits is being promoted. The Korean market for fresh-cut products has been increasing, and it was valued at $7000 million in 2012. Lettuce is one of the most important fresh-cut vegetables. When lettuce is harvested, pests such as cutworms, slugs, insects, and earthworms, and inanimate foreign materials such as gravel and soil are frequently found in it. These foreign materials are removed by washing and screening during postharvest handling. Screening is performed using only the visual skills of operators. Screening operators have difficulty in detecting worms that are small in size or of a color similar to that of lettuce. Therefore, development of technical methods to detect foreign living materials is required. Studies have been performed to detect foreign materials in agricultural products such as insects in bulk wheat using machine vision [1], insect fragments in wheat flour [2–4] and foreign objects in soybean [5] using near-infrared (NIR) spectroscopy, insects inside wheat kernels using NIR imaging [6], and insects in chocolate using multispectral imaging [7]. Machine vision techniques that use image processing detect color differences between agricultural products and defects such as insects [1]. The insect detection algorithm using RGB (Red, Green, and Blue) information showed a correct recognition rate of 92.5% for insects; however, they had difficulty in discriminating wheat and grass seeds that are similar in color to insects. NIR spectroscopy is used to determine the difference in the absorption characteristics of molecules constituting agricultural products and defects [2–5]. Previous studies show that live and dead internal insects can be detected using the NIR technique [2– 4]. NIR spectroscopy, which measures an integrated signal from an entire measurement area, has lower sensitivity than signals measured in a narrow area such as a defect area. Detection of small larvae in bulk wheat samples [4] and inspection for 75 insect fragments in 50 g of wheat flour as per the US Food and Drug Administration action level [2] had low accuracies. To overcome these drawbacks of the machine vision technique and NIR technique, hyperspectral imaging (HSI) and multispectral imaging technologies, which combine imaging and spectroscopic techniques, have been used in recent years to analyze quality [8] and defects in agricultural products [6–7, 9–11]. Spectral imaging methods are employed to measure the spatial distribution or positions of defects and foreign substances by combining
images at specific wavelengths. Furthermore, these techniques have been used to detect biological and surface contamination in agro-food using physical and chemical properties of objects [12–15]. Multispectral imaging may utilize an optimal combination of wavelengths determined using HSI. A multispectral imaging system that uses a multispectral camera can potentially have lower cost and better scan speeds than hyperspectral cameras, and therefore, have better applicability in real-time production systems [12, 16–18]. The two-wavelength subtraction images (1202–1300 nm) for detecting insects inside wheat kernels gives an enhanced view of the differences between infested and control kernels [6]. The two-waveband ratio has been used for detecting fecal contamination on poultry surface [19], and two-wavelength difference images have been used for detecting defects on Fuji apple surfaces [20]. Multispectral imaging based on the three-wavelength regions of 840–870 nm, 870–900 nm, and 900–930 nm showed a detection rate of 93% while inspecting the produce for insects inside chocolate [7]. Combinations of multispectral images improve the detection performance as compared to a single waveband image. The purpose of this study is to develop multispectral imaging methods that can detect worms in lettuce as the basis for the development of foreign substance screening technologies for fresh-cut lettuce. Multispectral imaging algorithms to discriminate worms in fresh-cut lettuce were developed using visible-NIR (VNIR) and NIR HSI techniques. The optimal wavebands that discriminate worms in fresh-cut lettuce were investigated for each type of HSI using one-way ANOVA. The developed algorithms will be employed as the basis for the development of a defect detection system for fresh-cut lettuce.
2. Materials and methods 2.1. Materials Iceberg lettuce produced in the southern part of Korea in 2015 was purchased from a lettuce-producing farm where worms were found. The primary types of worms found in iceberg lettuce are cutworms (Agrotis segetum); in this paper, they are hereafter referred to as worms. The lengths and widths of the worms were in the range of 19.0 mm to 40.1 mm and 1.9 mm to 5.1 mm, respectively. One hundred samples of lettuce that contained worms were used in the experiments. The samples were stored in a refrigerator at 4 °C before the
experiments. The hyperspectral images of worms on the surface of lettuce, which was cut into 5 cm × 5 cm samples, were obtained to develop a worm discrimination method. The samples were allowed to reach room temperature (20 °C) before cutting. Fifty calibration samples (set A) were used for algorithm development and 50 validation samples (set B) were used to test the algorithms.
2.2. Hyperspectral imaging system The following two HSI systems were employed in this study: a VNIR-HSI system with a resolution of 1004 spectral and 1002 spatial pixels in the wavelength range of 400–1000 nm and spectral resolution of 1004 × 1002 pixels, and an NIR-HSI system with a resolution of 640 spectral and 512 spatial pixels in the wavelength range of 980–1700 nm. The VNIR-HSI and NIR-HSI systems were constructed as shown in the schematics in Figs. 1 and 2. The VNIR-HSI system contained a low-light sensitive electron multiplying charge-coupled device camera (MegaLuca, Andor Technology Inc., Belfast, Northern Ireland), an imaging spectrograph (VNIR Hyperspec, HeadwallPhotonics Inc., Fitchburg, MA, USA), a Schneider-Kreuznach Xenoplan 1.4/23 C-mount lens (Schneider Optics, Hauppauge, NY, USA), and a pair of light sources. These light sources employed a pair of halogen-tungsten line lights (LS-F100HS-IR, Seokwang Inc., Hwasung, Korea) arranged toward the instantaneous field of view (IFOV) at off-nadir backward and forward angles of 15° for providing near-uniform excitation energy to a linear sample imaging area. The NIR-HSI system was composed of an InGaAs focal-plane-array (FPA) camera with a resolution of 640 × 512 pixels (Model Xeva-3035, Xenics, Leuven, Belgium), an imaging spectrograph (HyperspecTM NIR G4-249, Headwall Photonics, MA, USA), and a 25-mm zoom lens (Model OB-NIR35/2, Optec, Parabiago, Italy). In addition, the system included a computer for controlling the camera and acquiring images, four 150 W DC light sources with fiber optic bundles (LS-F100HS-IR, Seokwang Inc., Hwasung, Korea), a sample table, and a motorized uniaxial stage (BMS100-UFA, Aerotech, Pittsburgh, PA, USA) attached to the InGaAs FPA camera module and line light guide of the light sources. The FPA camera was thermo-electrically cooled to a temperature of −20 °C using a two-stage Peltier device. The light sources for NIR-HSI employed a pair of halogen-tungsten line lights (LS-F100HS-IR, Seokwang Inc., Hwasung, Korea) attached at 15° toward the IFOV. An imaging spectrograph with a 25-µm slit, a C-mount lens with focus adjustment, and an aperture diaphragm were
also attached. After passing through the 25 µm × 18 mm (width × length) aperture slit, light from a scanned line of the field-of-view (FOV) was dispersed by a dispersive grating and projected onto the FPA camera. A two-dimensional image was obtained with its spatial and spectral dimensions along the horizontal and vertical axes of the FPA, respectively. In the case of reflectance imaging, a spectral image amplified by the FPA camera was captured only in the wavelength range of 900–1700 nm.
Fig. 1. Schematic of the VNIR-HSI system.
Fig. 2. Schematic of the NIR hyperspectral reflectance imaging system.
2.3. Hyperspectral image data acquisition and analysis Ten individual sets of hyperspectral images were obtained using the two types of HSI systems to detect worms in the lettuce. Each set consisted of a 5 × 2 pieces cut leaf arrangement on a lined tray. VNIR images and NIR images were obtained using a line scanning method with an exposure time of 6 ms and 17 ms, respectively. The VNIR images with a resolution of 300 × 502 pixels per spectral band consisted of 125 spectral bands with a waveband interval of 8 nm. The resolution of the NIR images was 300 × 640 pixels, and each image consisted of 224 wavebands with a waveband interval 3.2 nm. White reference images were obtained for VNIR-HSI and NIR-HSI using a 99% diffuse reflectance standard (Spectralon™, SRT-99-120, Labsphere, NH, USA) to calibrate the intensity of the light source for each vertical pixel. Dark reference plate images for compensating the device noise were obtained without using a light source. The dark and white reference images were used to convert the raw reflectance images of worms in the lettuce into corrected reflectance images according to equation (1). Ireflectance (i) =
,
(1)
where Ireflectance, Ir, Id, and Iw denote the corrected reflectance image, the raw hyperspectral image, the dark reference image, and the white reference image, respectively, at the ith wavelength. The reflectance spectra of the pixels composed of worms and lettuce surfaces were extracted from the corrected VNIR and NIR hyperspectral images and used to calculate the average reflectance spectrum. Five waveband indexes were developed using a single-waveband reflectance and a combination of multiple waveband reflectances in the spectra to find an optimal combination of wavebands for discriminating between worms and lettuce. INDEX-1 was developed using one waveband. In addition, the following four types of indexes were developed using two wavebands: INDEX-2 using the ratio of two wavebands (A/B), INDEX-3 using the subtraction of two wavebands (A−B), INDEX-4 using a combination of two wavebands ((A−B)/A), and INDEX-5 using a different combination of two wavebands ((A−B)/(A+B)). A simple statistical comparison is sufficient for determining the significant differences between groups of interest [21]. The analysis of variance (ANOVA) method, which was reported as being capable of finding an optimal waveband for identifying bruised and healthy
areas of apple and pear [9, 22], was applied to determine the best pair of wavelengths for classifying worms in lettuce. The quality index values for the spectra of worms and lettuce were defined as 1 and 0, respectively. ANOVA was performed to determine the F-values between each of the five functions and the quality index values of the samples. The single waveband for INDEX-1, two-waveband ratio for INDEX-2, two-waveband subtraction for INDEX-3, and twowaveband ratio-subtraction functions for INDEX-4 and INDEX-5 were obtained for all possible combinations of one and two wavebands. ANOVA was performed to determine the combinations of wavebands that were the most highly classified for the quality index values of the sample spectra. The spectra extracted within a region of interest (ROI) from the worm and lettuce regions in Set A were used for calibration. Using these optimal wavebands, each of these five INDEX values were developed for use with pixel spectra. Then, a classification value (CV) as an optimal global threshold was determined at the highest classification accuracy. The validation of the developed INDEX values was conducted using five types of pixel spectra from Set B. Five imaging algorithms were developed to classify worms on lettuce surfaces using spectral images and CV of the five developed INDEX values. A single-waveband imaging (SWI) algorithm was developed using the single-waveband image identified by INDEX-1. A ratio imaging (RI) algorithm was developed using the ratio image (RI a/b) of two wavebands selected in INDEX-2 according to equation (2). RIa/b =
,
(2)
where Ia and Ib denote the corrected images at wavelengths a and b, respectively. A subtraction imaging (SI) algorithm was developed using the subtraction image (SI a-b) of the two wavebands identified by INDEX-3 according to equation (3). SIa-b =
,
(3)
where Ia and Ib denote the corrected images at wavelengths a and b, respectively. A ratio-subtraction imaging-I (RSI-I) algorithm was developed using a combination of the ratio and subtraction image (RSI-I(a-b)/a) of two wavebands chosen by INDEX-4 according to equation (4). RSI-I(a-b)/a =
,
(4)
where Ia and Ib denote the corrected images at wavelengths a and b, respectively.
A ratio-subtraction imaging-II (RSI-II) algorithm was developed using a combination of the ratio and subtraction image (RSI-II(a-b)/(a+b)) of two wavebands selected by INDEX-5 according to equation (5). RSI-IIa-b/a+b =
,
(5)
where Ia and Ib denote the corrected images at wavelengths a and b, respectively. The optimal pixel sizes of the discrimination images were investigated to enhance the performance of these worm-detection imaging algorithms. Imaging predictions for four pixel sizes were conducted using the algorithms for the calibration and validation samples. The four pixel sizes were a 1 × 1 pixel of 1 mm × 0.29 mm (Size-0), 1 × 4 pixels of 1 mm × 1.16 mm (Size-1), 2 × 7 pixels of 2 mm × 2.03 mm (Size-2), and 3 × 10 pixels of 3 mm × 2.97 mm (Size-3). For Size-1, if the number of pixels in the positive area of the discrimination images of 1 × 4 pixels of Size-0 was more than 4, then, the value of one pixel merged into 4 pixels was considered to be 1; otherwise, it was considered to be 0. For Size-2 and Size-3, if the number of pixels in the positive area of the discrimination images of 2 × 7 pixels and 3 × 10 pixels of Size-0 was more than 13 and 29, respectively, then, the values of one pixel merged into 14 pixels for Size-2 and into 30 pixels for Size-3 were considered to be 1; otherwise, the value of one pixel merged into 30 pixels was considered to be 0. The sensitivity of classification accuracy of worms refers to the number of positives in which the worms are correctly identified, and specificity of the classification accuracy for lettuce refers to the proportion of negatives in which the lettuce is correctly identified. MATLAB (version 7.0.4, the Mathworks, Natick, MA, USA) was used to extract and analyze the hyperspectral image data.
3. Results and discussion 3.1. Spectral characteristics of lettuce and foreign substances The representative average reflectance spectra of pixels consisting of each worm and lettuce image acquired using VNIR-HSI and NIR-HSI in the ranges of 400–1000 nm and 980–1700 nm, respectively, are shown in Fig. 3a and 3c. Fig. 3b for VNIR-HSI and Fig. 3d for NIR-HSI show the average, maximum, and minimum values of all pixel spectra extracted from corrected hyperspectral images of 50 worms and 50 lettuce samples.
The average VNIR reflectance spectrum of worms is lower than that of lettuce in the range of 400–1000 nm. The reflectance of worms increases in the range of 740–910 nm; however, the reflectance of lettuce is constant (Fig. 3a and 3b). Worm spectra in the range of 500–670 nm do not exhibit peaks; however, lettuce spectra exhibit broad peaks. A wavelength of approximately 681 nm corresponds to the wavelength absorbed by chlorophyll in the lettuce [24]. The NIR reflectance spectra of worms have higher slopes than those of lettuce in the ranges of 1000–1170 nm and 1230–1330 nm, as shown in Fig. 3c and 3d. The peaks at 1130 nm and 1330 nm are associated with the lipids of insects [3]. The average value of all pixel spectra of the 50 worm samples is larger than that of the 50 lettuce samples. Fig. 3b and 3d show the average spectra of pixels extracted using two types of hyperspectral images for 50 lettuce and 50 worm samples. The average value of pixel spectra for all types of hyperspectral images shows decreased pixel-to-pixel variation, and the overlap between the worm and lettuce spectra reduces. In the case of VNIR reflectance spectra, the reflectance of worms is lower than that of lettuce at all wavelengths in the range of 400–1000 nm. VNIR Reflectance
VNIR Spectrum
(b)
(a)
1.0
1.0
0.8
Reflectance
0.8
Reflectance
Lettuce Worms
Average of lettuce spectra Maximum of lettuce spectra Minimum of lettuce spectra Average of worm spectra Maximum of worm spectra Minimum of worm spectra
0.6
0.4
0.6
0.4
0.2
0.2
0.0
0.0 400
500
600
700
Wavelength (nm)
800
900
1000
400
500
600
700
Wavelength (nm)
800
900
1000
NIR Spectrum
(c)
(d) Average of lettuce spectra Maximum of lettuce spectra Minimum of lettuce spectra Average of worm spectra Maximum of worm spectra Minimum of worm spectra
0.8
Lettuce Worms
0.6
Reflectance
Reflectance
0.6
0.8
0.4
0.2
(e)
0.4
0.2
0.0
0.0
1000
1200
1400
1600
1000
Wavelength (nm)
1200
1400
1600
Wavelength (nm)
Fig. 3. (a) Pixel spectra with average, minimum, and maximum values and (b) average spectra of worms and lettuce acquired using VNIR-HSI; (c) pixel spectra with average, minimum, and maximum values and (d) average spectra of worms and lettuce acquired using NIR-HSI.
3.2. Algorithms for discriminating worms in lettuce Table 1 and Table 3 show the optimal wavebands and classification results for each index in terms of worm detection using pixel spectra for VNIR-HSI and NIR-HSI, respectively. Using pixel spectra for analysis can include deviation in spectra, rather than using average spectra.
3.2.1. VNIR-HSI 3.2.1.1. Spectrum index Worm detection indexes were developed using VNIR-HSI. First, single-waveband INDEX1 was developed for discriminating worms in lettuce. The F-value of each waveband obtained using ANOVA to distinguish between worms and lettuce for the single waveband using the calibration samples (Set A) is shown in Fig. 4a. The ANOVA results featured a peak with an F-value of 144,460.7 at a wavelength of 518 nm, which is associated with carotenoids [23]. The second peak in the F-value was observed at 725 nm, which is related to chlorophyll a [34]. A wavelength of 518 nm is included in the broad peak region of the spectra for lettuce and in the no-peak region of the spectra for worms. The lowest F-value was observed at 959 nm, which is close to the wavelength for water absorption [11], as shown in Fig. 3a. The classification accuracies for worms and lettuce at 518 nm for the validation sample
(Set B) were 96.17% and 97.5%, respectively. The threshold value with the best classification accuracy was 0.238, as shown in Table 1.
Table 1. Optimal wavebands and discrimination results of worm and lettuce using a singlewaveband INDEX and the combination of two-waveband INDEX values for VNIR spectra Waveband F-value A
CV1)
B
CA2) of calibration (%) CA2) of validation (%) Worm Lettuce Total Worm Lettuce Total
INDEX-13)
518
144,460.7 0.238 99.12
99.05 99.09 96.17 97.50 96.83
INDEX-24)
547
945 219,423.1 0.730 99.29
99.72 99.51 97.70 99.17 98.44
INDEX-35)
744
945 118,429.6 0.026 98.14
99.27 98.71
INDEX-46)
403
523 141,121.1 0.175 97.79
99.01 98.40 95.80 98.55 97.18
INDEX-57)
523
950 169,883.9 0.145 99.21
99.38 99.30 97.43 98.50 97.96
Note: 1)CV : Classification value, 2)CA : Classification Accuracy,
3)
917
99.29 96.73
INDEX-1 : Single waveband, 4) INDEX-2
: Two-waveband ratio (A/B), 5)INDEX-3 : Two-waveband subtraction (A− B),
6)
INDEX-4: Combination of
two wavebands ((A− B)/A), 7) INDEX-5: Combination of two wavebands ((A− B)/(A+B))
(a) 1.6 1.4
F-value (105)
1.2 1.0 0.8 0.6 0.4 0.2 0.0
400
500
600
700
800
900
1000
Wavelength (nm) Col 1 vs Col 2
Fig. 4. Results of one-way ANOVA for classifying worms and lettuce using (a) INDEX-1 of single waveband and (b) INDEX-2 of two-waveband ratio for the VNIR spectrum from 400–1000 nm.
The discrimination index for detecting worms on lettuce surfaces was developed using four combinations—ratio, subtraction, and combinations of ratio, subtraction, and addition—of two wavebands in the range from 400–1000 nm. INDEX-2 of the two-waveband ratio had a maximum F-value of 219,423.1 for the four combinations of two wavebands, as shown in Fig. 4b. The optimal wavelengths were 547 nm, which is related to the characteristics of carotenoids [23] and is close to the waveband with the highest F-value for a single waveband, and 945 nm, which is associated with the characteristics of water [12] and is close to the waveband with the lowest F-value for a single waveband. INDEX-2 for the calibration samples had a maximum classification accuracy of 99.29% for worms and 99.72% for lettuce, with a threshold value of 0.73. The validation results obtained using Set B demonstrated that INDEX-2 had a maximum classification accuracy of 97.70% for worms and 99.17% for lettuce, with a threshold value of 0.73. The classification performances of all two-waveband indexes improved as compared to that of the single-waveband index. Thus, worms can potentially be detected on lettuce surface using VNIR reflectance spectra.
3.2.1.2. VNIR imaging algorithm The imaging algorithms for worm detection were developed using hyperspectral VNIR images. Worm detection methods were investigated using single and multi-wavelength images with optimal results of ANOVA.
Single-waveband imaging algorithm Fig. 5 shows a sequence of representative images processed using single-waveband images to classify worms. A 657 nm image (Fig. 5a) was transformed into a masking image (Fig. 5b) binarized by a threshold reflectance value of 0.02. The masking image was created to remove the backgrounds and to select lettuce and worms. The 518 nm waveband image (Fig. 5c) was produced by applying the masking image and then transformed into the binary image (Fig. 5d) using a threshold reflectance value of 0.238. The enhanced image (Fig. 5e) used to reduce misclassification of the binary image was created using the pixel value modified with the value of the several adjacent pixels.
This threshold value yielded the best accuracy for
discriminating between the worm and lettuce when using the calibration samples (Set A). The black areas in the binary image represent worms.
Fig. 5. Illustration of image processing sequence using a single waveband: (a) grayscale I684 image; (b) masking images obtained by applying a reflectance intensity threshold of 0.02 to the I684 image; (c) grayscale and color images of I525 after masking; (d) resulting detection images after application of a threshold of 0.238; (e) enhanced image modified using pixel size.
Multispectral imaging algorithm Fig. 6 shows the sequence of images corresponding to the image processing method used to distinguish worm areas in lettuce samples using the dual-waveband index (INDEX-2547/945). The 657 nm image with the highest difference in intensity between the background and lettuce was selected as the masking image for the two wavebands (Fig. 6a). The binary image was created using the masking image following the same method as that for the SWI algorithm (Fig. 7b). A threshold value of 0.02 was obtained using the global threshold method. The ratio image for 547 nm and 945 nm (Fig. 6d) was created using the 547 nm and 945 nm images after application of the masking image (Fig. 6c). The ratio image was transformed into a binary image using the threshold value (Fig. 6e). The classification rate for worm and lettuce areas, obtained using the dual-waveband ratio, is shown in Fig. 6. A threshold value of 0.73 yielded the best classification accuracy for worm and lettuce areas using the calibration samples (Set A). If the pixel value of the ratio image was higher than the threshold value, the pixel was assigned a value of 1; otherwise, it was assigned a value of 0. The positive areas in the binary images represent worms.
Fig. 6. Illustration of image processing sequence for the RI algorithm using two wavebands: (a) grayscale I657 image; (b) masking image obtained by applying a reflectance threshold of 0.02 to the I657 image; (c) I547 and I945 images after masking; (d) the grayscale ratio image of I547/945; (e) resulting detection image after application of a threshold of 0.73; (f) enhanced image modified using pixel size.
Improvement of VNIR imaging algorithm using pixel size Imaging predictions for four kinds of pixel sizes for the calibration and validation samples were conducted using the five imaging algorithms for VNIR reflectance. The imaging prediction results are shown in Table 2. The images of worm detection on samples, obtained using the five algorithms for VNIR-HSI, are shown in Fig. 7. In the case of Size-0, the sensitivities and worm prediction accuracies of the SWI algorithm were above 99.9% and 90.0% for calibration and validation, respectively. For lettuce, the specificities and prediction accuracies of the SWI algorithm were 18.0% and 6.00% for calibration and validation, respectively. Low specificity and prediction accuracy is caused by lettuce pixels below 0.238 reflectance intensity in the 518 nm image associated with carotenoids, as shown in Fig. 3a. The best results were obtained using the dual-waveband RI algorithm images related to carotenoids and water with specificities of 52.0% and 56.0% for calibration and validation, respectively. The sensitivity and prediction accuracy of the RI algorithm were 98.0% and 77.0%, respectively. This algorithm exhibited better performance than the SWI algorithm. Low specificity was the reason that the lettuce area was recognized
as a worm area, resulting in false positive pixels. To overcome this problem, the size of a single pixel was modified to 1 × approximately 1 mm, 2 × approximately 2 mm, and 3 × approximately 3 mm, with the number of pixels modified to 1 × 4 pixels, 2 × 7 pixels, and 3 × 10 pixels, respectively. In the case of Size-1 with 1 × 1 mm pixels, the SWI algorithm had a sensitivity of more than 99.9% and a specificity of 50.0% for calibration, and a sensitivity of more than 90.0% and a specificity of 32.0% for validation. The maximum sensitivity of the RI and RSI-II algorithms was more than 99.9% for validation and the corresponding maximum specificity of the SI algorithm was more than 99.9%. The prediction accuracy of the SI algorithms surpassed that of all other algorithms. All dual-wavelength algorithms exhibited improved performance as compared to the SWI algorithm. The specificity of all algorithms for Size-1 was better than that for Size-0. Using a detection pixel size of approximately 1 mm decreased the detection error for lettuce. In the case of Size-2 with 2 × 2 mm pixels, the sensitivity of the SWI algorithm was 90.0% for validation, which was equal to the sensitivity of the SWI algorithm for Size-1; however, its specificity increased to 88.0%. The optimal sensitivity and prediction accuracy of the dualwavelength RI algorithm were 96.0% and 97%, respectively. The optimal specificities of the SI and RSI-I dual-wavelength algorithms were more than 99.9% for validation. Compared to the SWI algorithm, the specificities of the four dual-wavelength algorithms increased; however, their sensitivities decreased. The number of worm pixels detected using all algorithms decreased for Size-2. The prediction results for all algorithms obtained using approximately 3 × 3 mm pixel images (Size-3) indicated that specificities increased; however, sensitivities and prediction accuracies decreased. For worm detection, Size-1 with approximately 1 mm pixels may be more suitable than Size-2 with approximately 2 mm pixels because the rate of the pixels with a value of 1 in the worm areas for Size-1 was larger than that for Size-2. In addition, the detection accuracy for lettuce for Size-1 was slightly higher than that for Size-2. These results show that the RI algorithms with a pixel size of approximately 1 mm for VNIR reflectance HSI have the capacity to detect worms in lettuce. The classification accuracy of worm detection using VNIR reflectance HSI surpassed the 93% detection accuracy for insects in chocolate, obtained using VNIR multispectral images
of 840–870 nm, 870–900 nm, and 900–930 nm regions [7]. These results show that the RI algorithms for VNIR reflectance HSI exhibit the potential to discriminate between worms and lettuce.
Fig. 7. Prediction results for imaging algorithms of VNIR-HSI for each pixel size.
Table 2. The results of worm detection of four pixel sizes for five imaging algorithms of VNIR hyperspectral images CA1)
Algorithms
Wavebands
Size-0
Size-1
Size-2
Size-3
Pixel size of 1×1 pixel (1× 0.29 mm)
Pixel size of 1× 4 pixel (1×1.16mm)
Pixel size of 2×7 pixel (2×2.03mm)
Pixel size of 3× 10 pixel (3×2.9mm)
Calibration SE2)
Validation
Calibration SE2)
Validation
Calibration
SP3)
PA4)
SE2)
SP3)
PA4)
SP3)
PA4)
SE2)
SP3)
PA4)
SE2)
SP3)
90.0
32.0
61.0 100.0 88.0
Validation
Calibration
PA4)
SE2)
SP3)
PA4)
SE2)
90
90.0
88.0
89.0
93.0 100.0 100.0 100.0 96.0
98.0
SP3)
PA4)
Validation SE2)
SP3)
PA4)
72.0 100.0 86.0
70
96.0
85.0
96.0
80
SWI5)
518
100.0 18.0
59.0
90.0
6.0
48.0 100.0 50.0
75.0
RI6)
547/945
100.0 52.0
76.0
98.0
56.0
77.0 100.0
92.0 100.0 86.0
97.0
70.0 100.0 85.0
72.0
SI7)
744-945
100.0 32.0
66.0
90.0
28.0
59.0 100.0 100.0 100.0 90.0 100.0 95.0 100.0 100.0 100.0 90.0 100.0 95.0
62.0 100.0 81.0
60
52.0
96.0
2.0
49.0 100.0 90.0
95.0
69.0
98.0
48.0
73.0 100.0 80.0
90.0 100.0 82.0
RSI-I8) RSI-II9)
(403-523)/403 100.0 (523-950) /(523+950)
0
100.0 38.0
Note: 1) CA : classification accuracy, 2) SE : sensitivity, 8)
3)
80
SP : specificity,
waveband subtraction image, RSI-I: Ratio-Subtraction image-I,
9)
4)
98.0
90.0
90
100.0 98.0
99.0
91.0 100.0 96.0
98.0
90.0 100.0 95.0 90
90
PA : predictive accuracy, 5) SWI : single-waveband image,
RSI-II: Ratio-Subtraction image-II.
90
6)
60
100.0 82.0
76.0 100.0 88.0
100.0 82.0
52.0 100.0 76.0 70.0
96.0
RI : two-waveband ratio image,
7)
83.0
SI : two-
3.2.2. NIR-HSI 3.2.2.1. Spectrum index Worm detection indexes were developed using NIR-HSI in a manner similar to that for VNIRHSI. First, INDEX-1 was developed for a single waveband. The F-value of each waveband obtained using the calibration samples (Set A) is shown in Fig. 8a. The results of ANOVA showed an optimal wavelength of 1000 nm close to the 1020 nm peak associated with proteins [12, 25] because the protein content of worms is higher than that of lettuce. The F-value was 48,810.3. The second peak in the F-value was observed at 1208 nm, which is related to the 2nd harmonic C-H stretching of CH2 in lipids [12, 25]. The accuracies of classification results at 1000 nm for the validation sample (Set B) were 98.04% and 93.65% for worms and lettuce, respectively. The threshold value with best classification accuracy was 0.3791, as shown in Table 3.
Table 3. Optimal wavebands and discrimination results of worm and lettuce using a single waveband INDEX and the combination of two-waveband INDEXs for NIR spectra Waveband F-value A 3)
INDEX-1 INDEX-24) INDEX-35) INDEX-46) INDEX-57)
1000 1064 1064 1064 1064
1)
CA2) of calibration (%)
CA2) of validation (%)
CV
B 48,810.3 0.3791 1176 223,460.5 0.98 1176 234,199.1 0.0087 1173 199,087.1 0.03 1176 211,656.1 0.145
Worm
Lettuce
918
937
928
98.04
93.65
95.85
97.80 97.60 97.74 99.21
99.82 99.71 99.59 99.38
98.81 98.66 98.67 99.30
99.61 99.40
99.94 99.91
99.77 99.65
99.44 99.61
99.97 99.94
99.70 99.77
Note: 1)CV : Classification value, 2)CA : Classification Accuracy,
3)
Total Worm Lettuce Total
INDEX-1 : Single waveband,
waveband ratio (A/B), 5)INDEX-3 : Two-waveband subtraction (A− B),
6)
((A− B)/A), 7) INDEX-5: Combination of two wavebands ((A− B)/(A+B))
4)
INDEX-2 : Two-
INDEX-4: Combination of two wavebands
(a) 6
5
F-value (104)
4
3
2
1
0 1000
1200
1400
1600
Wavelength (nm)
Fig. 8. Results of one-way ANOVA for classifying worms and lettuce using (a) INDEX-1 of single waveband and (b) INDEX-3 of two-waveband subtraction for the NIR spectrum from 980–1700 nm.
The discrimination indexes for the dual waveband in the range of 980–1700 nm were developed in the same manner as that for VNIR-HSI. INDEX-3 of two-waveband subtraction in the four dual-waveband algorithms had a maximum Fvalue of 234,199.1 (Fig. 8b). The optimal wavebands were 1064 nm and 1176 nm, which are close to the peaks in the characteristics of protein and lipids [12, 25]. Similar to the F-value results for a single waveband, the F-values for the two-waveband ratio were higher in the combined region of 1000 nm–1100 nm, which is related to the characteristics of proteins, and 1160 nm–1190 nm, which is related to the characteristics of lipids [12, 25]. The classification accuracies obtained using the calibration samples (Set A) were 97.60% and 99.71% for worms and lettuce, respectively. The threshold value with the best classification accuracy was 0.0087, as shown in Table 3. INDEX-2 of the two-waveband ratio for the calibration samples had a maximum classification accuracy of 97.80% and 99.82% for worms and lettuce, respectively, with a threshold value of 0.98. The best wavebands for INDEX-2 were 1064 nm and 1176 nm with an F-value of 223,460.5. The results were validated using the validation samples (Set B). It was observed that INDEX-2 and INDEX-5 had maximum classification accuracies of 99.61% and 99.97% for worms and lettuce, with threshold values of 0.98 and 0.145, respectively. The best wavebands for these two indexes were
1064 nm and 1176 nm. Thus, worms can potentially be detected on lettuce surface using NIR reflectance spectra.
3.2.2.2. NIR imaging algorithms The single-waveband imaging algorithm for classifying worms on lettuce surface was developed using an image at 1000 nm that had the highest F-value according to ANOVA. The image processing used to develop the algorithm was similar to that for VNIR-HSI. A multispectral imaging algorithm to distinguish worm areas on lettuce samples was developed using a combination of ratio and subtraction of dual-waveband imaging. The image processing used to develop the algorithm was similar to that for VNIR-HSI. Imaging predictions for four pixel sizes were conducted for the calibration and validation samples using the five algorithms for NIR reflectance. The imaging prediction results are shown in Table 4. In the case of Size-0 with 1 × 1 pixel in NIR-HSI, the images of worm detection on samples obtained using the five algorithms are shown in Fig. 9. For worms, the sensitivity and prediction accuracy of the SWI algorithm were 90% and more than 99.9% for calibration and validation, respectively. For lettuce, the specificity and prediction accuracy of the SWI algorithm were 0.0% and 2.0% for calibration and validation, respectively. Low specificity and prediction accuracy is caused by pixels in the lettuce areas that are below 0.3791 reflectance intensity in the 1000 nm image in Fig. 3c, which is related to protein content. The dual-waveband subtraction image (SI) algorithm images related to protein and lipid contents had a maximum sensitivity, specificity, and prediction accuracy of more than 99.9%, 82.0%, and 91.0%, respectively, for the validation sample, which was a considerably better performance than that of the SWI algorithm. The reason for low discrimination accuracy for lettuce could be that lettuce was recognized as a worm owing to false positive pixels. For Size-1 with 1 × 1 mm pixels for NIR-HSI, the sensitivity of the SWI algorithm was more than 90.0% and more than 99.9% for calibration and validation, respectively, and its specificity was 0.0% and 2.0% for calibration and validation, respectively. The sensitivity and specificity of all dual-waveband SI algorithms were more than 99.9% for validation. All dual-waveband algorithms performed better than the SWI algorithm. The specificities of all algorithms for Size-1 were higher than that for Size-0. Using a pixel of 1 mm size decreased the detection error rate for lettuce. In the case of Size-2 with 2 × 2 mm pixels in NIR-HSI, the sensitivity of the SWI algorithm was 99.9% for validation, which was equal to that for Size-1, and its specificity was 16.0%, which was slightly more than that for Size-1. The four dual-waveband algorithms exhibited more than 99.9% discrimination of worms and lettuce for validation, which was similar to that for Size-1. The
specificities of the four dual-waveband algorithms for Size-2 were higher than that for Size-1. Likewise, the sensitivities of the four dual-waveband algorithms for Size-2 were equal to or more than that for Size-1. The number of worm pixels detected using all algorithms decreased for Size-2. For Size-3, the prediction results for all algorithms developed using NIR-HSI indicated an increase in the specificities; however, the sensitivities were the same for validation and decreased for calibration, as compared to those for Size-2. The number of worm pixels detected using all algorithms decreased for Size-3. Size-1 with a pixel size of 1 mm may be more suitable for worm detection than Size-2 with a pixel size of 2 mm because the number of pixels with a value of 1 in the worm areas for Size-1 were larger than that for Size-2. In addition, the detection accuracy for lettuce for Size-1 was slightly higher than that for Size-2. These results show that the four dual-waveband algorithms for NIR reflectance HSI for a pixel size of 1 mm exhibit the capacity to detect worms in lettuce. The classification performance for worm detection using NIR reflectance HSI exceeded the 92%– 93% detection accuracy for dead pupae and live larvae in wheat kernels, obtained using NIR reflectance spectroscopy [4]. These results indicate that the worm detection algorithms for NIR reflectance HSI have the potential to discriminate between worms and lettuce.
Fig. 9. Prediction results for imaging algorithms of NIR-HSI for each pixel size.
Table 4. The results of worm detection of four pixel sizes for five imaging algorithms of NIR hyperspectral images CA1)
Algorithms Wavebands
Size-0
Size-1
Size-2
Size-3
Pixel size of 1×1 pixel (1× 0.29 mm) Calibration Validation
Pixel size of 1× 4 pixel (1×1.16mm) Calibration Validation
Pixel size of 2×7 pixel (2×2.03mm) Calibration Validation
SE2)
SP3)
PA4)
SE2)
47.0 100.0
SP3)
PA4)
SE2)
SP3)
PA4)
SP3)
PA4)
Pixel size of 3× 10 pixel (3×2.9mm) Calibration Validation 4) 2) 3) 4) 2) SE SP PA SE SP3) PA
2.0
51.0
90.0
26.0
58.0 100.0 16.0
58.0
86.0
90.0
90.0 100.0 95.0 100.0 98.0
99.0
90.0 100.0 95.0 100.0 100.0 100.0 82.0 100.0 91.0
96.0 100.0 98.0
SP3)
PA4)
SE2)
SP3)
PA4)
SE2)
2.0
51.0
90.0
0.0
45.0 100.0
100.0 80.0
SE2)
65.0
SWI5)
1000
90
0.0
RI6)
1064/1176
90
70
SI7)
1064-1176
90
72.0
83.0 100.0 82.0
91.0
90.0 100.0 95.0 100.0 100.0 100.0 90.0 100.0 95.0 100.0 100.0 100.0 82.0 100.0 91.0
98.0 100.0 99.0
RSI-I8)
(1064-1173) /1064
90
70.0
82.0 100.0 78.0
89.0
90.0 100.0 95.0 100.0 98.0
99.0
90.0 100.0 95.0 100.0 100.0 100.0
100.0 92.0
92.0 100.0 96.0
RSI-II9)
(1064-1176) /(1064+1176)
90
68.0
81.0 100.0 76.0
88.0
90.0
99.0
90.0 100.0 95.0 100.0 100.0 100.0 82.0 100.0 91.0
96.0 100.0 98.0
Note: 1) CA : classification accuracy,
80
2)
98.0
SE : sensitivity, 3) SP : specificity,
4)
90
100.0 98.0
PA : predictive accuracy,
waveband subtraction image, 8) RSI-I: Ratio-Subtraction image-I, 9) RSI-II: Ratio-Subtraction image-II
5)
SWI : single-waveband image,
6)
80
42.0
60
100.0 30.0
RI : two-waveband ratio image, 7) SI: two-
5. Conclusions Nondestructive methods based on three types of HSI techniques that use VNIR and NIR were developed to detect worms in fresh-cut lettuce. A single wavelength index and multispectral indexes were developed for each HSI technique. The optimal wavebands for discriminating between worms and lettuce were investigated using one-way ANOVA. The worm detection indexes developed using VNIR and NIR spectra exhibited a prediction accuracy of 98.44% for INDEX-2547/945, and 99.77% for INDEX-21064/1176 and INDEX-5(10641176)/(1064+1176).
In addition, image algorithms were developed for the two HSI techniques
using images of four different pixel sizes to detect worms. The results showed that the spectral images with a pixel size of 1 × 1 mm or 2 × 2 mm had the best classification accuracy for worms. The RI547/945 algorithm for VNIR-HSI, and the four image algorithms, i.e., RI1064/1176, SI1064-1176, RSI-I(1064-1173)/1064, and RSI-II(1064-1176)/(1064+1176), for NIR-HSI had the highest prediction accuracies of 97.0% and 100%, respectively. The results of this study indicate that the NIR single and multispectral imaging algorithms used to detect worms in lettuce have the best discrimination accuracies for the two types of HSI techniques. Therefore, spectral information from a hyperspectral image can be applied to develop an economic real-time multispectral imaging system to detect worms in fresh-cut lettuce.
Acknowledgments This study was supported by “Research Program for Agricultural Science & Technology Development (Project Nos. PJ012487 and PJ00939901),” National Institute of Agricultural Science, Rural Development Administration, Republic of Korea.
Conflicts of Interest The authors declare no conflicts of interest.
Reference 1. I.Y. Zayas, P.W. Flinn, Detection of insects in bulk wheat samples with machine vision. Trans. ASAE 41(1998) 883-888.
2. J. Perez-Mendoza, J.E. Throne, F.E. Dowell, J.E. Baker, Detection of insect fragments in wheat flour by near-infrared spectroscopy. J. Stored Prod. Res. 39 (2003) 305–312. 3. F.E. Dowell, J.E. Throne, J.E. Baker. 1998. Automated nondestructive detection of internal insect infestation of wheat kernels by using near-infrared reflectance spectroscopy. J. Econ. Entomol. 91(1998) 899-904. 4. E.B. Maghirang, F.E. Dowell, J.E. Baker, J.E. Throne, Automated detection of single wheat kernels containing live or dead insects using near–infrared reflectance spectroscopy. Trans. ASAE. 46(2003) 1277–1282. 5. J.G. Lim, S.W. Kang, K.J. Lee, C.Y. Mo, J.Y. Son, Identification of Foreign Objects in Soybeans Using Near-infrared Spectroscopy, J. Food Eng. 15 (2011) 136-142. (In Kore an, with English abstract). 6. C. Ridgeway, J. Chambers, Detection of insects inside wheat kernels by NIR imaging. J. Near Infrared Spectrosc. 6 (1998) 115–119. 7. T. Ma, H. Kobori, N. Katayama, S. Tsuchikawa, Nondestructive inspection of insects in chocolate using near infrared multispectral imaging. J. Near Infrared Spectrosc. 24 (2016) 391-397. 8. C. Mo, J. Lim, K. Lee, S. Kang, M.S. Kim, G. Kim, B.K. Cho, Determination of germination quality of cucumber(cucumis sativus) seed by LED-induced hyperspectral reflectacne imaging. J. Biosyst. Eng. 38(2013) 318-326 9. B.K. Cho, I.S. Beak, N.G. Lee, C.H. Mo, Study on bruise detection of “Fuji” apple using hyperspectral reflectance imagery, J. Biosyst. Eng. 36 (2011) 484–490. 10. H. Lee, C.C. Yang, M.S. Kim, J. Lim, B.K. Cho, A. Lefcourt, K. Chao, C.D. Everard, A simple multispectral imaging algorithm for detection of defects on red delicious apples. J. Biosyst. Eng. 39 (2014) 142–149. 11. I. Murray, P.C. Williams, Chemical principles of near-infrared technology. In P. Williams & K. Norris (Eds.), Near-infrared technology in the agricultural and food industries (pp. 17–34). St. Paul, MN, USA: American Association of Cereal Chemists, 1987. 12. D. Wu, D.W. Sun, Advanced applications of hyperspectral imaging technology for food quality and safety analysis and assessment; A review—Part I; Fundamentals, Innov. Food Sci. Emerg. Technol. 19 (2013) 1–14.
13. M.S. Kim, Y.R. Chen, P.M. Mehl, Hyperspectral reflectance and fluorescence imaging system for food quality and safety, Trans. ASAE. 44 (2001) 721-729. 14. W. Jun, M.S. Kim, K. Lee, P. Millner, K. Chao, Assessment of bacterial biofilm on stainless steel by hyperspectral fluorescence imaging, Sens. Instrum. Food Qual. Saf. 3 (2009) 41–48. 15. L.M. Kandpal, B.K. Cho, A Review of the Applications of Spectroscopy for the Detection of Microbial Contaminations and Defects in Agro Foods. J. Biosyst. Eng. 39 (2014) 215–226. 16. M.S. Kim, A.M. Lefcourt, K. Chao, Y.R. Chen, I. Kim, D.E. Chan, Multispectral detection of fecal contamination on apples based on hyperspectral imagery; Part I. Application of visible and near-infrared reflectance imaging, Trans. ASAE. 45 (2002) 2027–2037. 17. H.Y. Lee, C. Everard, S. Kang, B.K. Cho, K. Chao, D. Chan, M.S. Kim, Multispectral fluorescence imaging for detection of bovine feces on Romaine lettuce and baby spinach leaves, Biosyst. Eng. 127 (2014) 125–134. 18. D.H. Jeong, M.S. Kim, H. Lee, H. Lee, B.K. Cho, Detection algorithm for cracks on the surface of tomatoes using multispectral VIS/NIR reflectance imagery, J. Biosyst. Eng. 38 (2013) 199–207. 19. B. Park, W.R. Windham, K.C. Lawrence, D. Smith, Contaminant classification of poultry hyperspectral imagery using a spectral angle mapper algorithm, Biosyst. Eng. 96 (2007) 323-333. 20. K.J. Lee, S.W. Kang, S.R. Delwiche, M.S. Kim, S.H. Noh Correlation analysis of hyperspectral imagery for multispectral wavelength selection for detection of defects on apples, Sens. Instrum. Food Qual. Saf. 2 (2008) 90–96. 21. M.H. Kutner, Applied Linear Statistical Models. McGraw-Hill Irwin, Boston, USA, 2005. 22. W.H. Lee, M.S. Kim, H. Lee, S.R. Delwiche, H. Bae, D.Y. Kim, B.K. Cho, Hyperspectral near-infrared imaging for the detection of physical damages of pear. J. Food Eng. 130(2014) 1-7. 23. M.N. Merzlyak, A.E. Solovchenko, A.A. Gitelson, Reflectance spectral features and non-destructive estimation of chlorophyll, carotenoid and anthocyanin content in apple fruit. Postharvest Biol. Technol. 27(2003) 197–211.
24. J. Gross, Pigments in Vegetables; Chlorophylls and Carotenoids, Van Nostrand Reinhold, New York, NY, USA, 1991. 25. H.I Chung, H.J. Kim, Near-Infrared Spectroscopy, Principles Analytical Sci. Tech. 12 (2000) 1-14.
Highlights
Visible-near-infrared and near-infrared hyperspectral imaging technology were employed for developing the method to detect worms on fresh-cut lettuce.
The optimal wavebands to discriminate worm and lettuce were investigated using the one-way ANOVA analysis.
The worm detection multispectral imaging algorithms resulted in the prediction accuracy of 97% for VNIR hyperspectral imaging and the prediction accuracy of 100.0% for NIR hyperspectral imaging .
The spectral images with the pixel size of 1 × 1 mm or 2 × 2 mm classification accuracy for detecting worms.
had the
best
S1350-4495(17)30065-8 http://dx.doi.org/10.1016/j.infrared.2017.05.003 INFPHY 2291
To appear in:
Infrared Physics & Technology
Received Date: Revised Date: Accepted Date:
1 February 2017 3 May 2017 7 May 2017
Please cite this article as: C. Mo, G. Kim, M.S. Kim, J. Lim, S. Hyun Lee, H. Lee, B-K. Cho, Discrimination methods for biological contaminants in fresh-cut lettuce based on VNIR and NIR hyperspectral imaging, Infrared Physics & Technology (2017), doi: http://dx.doi.org/10.1016/j.infrared.2017.05.003
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Discrimination methods for biological contaminants in fresh-cut lettuce based on VNIR and NIR hyperspectral imaging Changyeun Mo a, Giyoung Kim a, Moon S. Kim b, Jongguk Lim a, Seung Hyun Lee c, Hongsek Lee a, and Byoung-Kwan Cho c,* a
b
c
National Institute of Agricultural Sciences, Rural Development Administration, 310 Nonsaengmyeong-ro, Wansan-gu, Jeonju-si, Jeollabuk-do 54875, Republic of Korea Environmental Microbial and Food Safety Laboratory, BARC-East, Agricultural Research Service, US Department of Agriculture, 10300 Baltimore Avenue, Beltsville, MD 20705, USA Department of Biosystems Machinery Engineering, College of Agricultural and Life Science, Chungnam National University, 99 Daehak-ro, Yuseong-gu, Daejeon 34134, Republic of Korea
* Corresponding author ; E-mail address: [email protected] (B.K. Cho) Tel.: +82-42-821-6715; Fax: +82-42-823-6246.
Abstract The rapid detection of biological contaminants such as worms in fresh-cut vegetables is necessary to improve the efficiency of visual inspections carried out by workers. Multispectral imaging algorithms were developed using visible-near-infrared (VNIR) and near-infrared (NIR) hyperspectral imaging (HSI) techniques to detect worms in fresh-cut lettuce. The optimal wavebands that can detect worms in fresh-cut lettuce were investigated for each type of HSI using one-way ANOVA. Worm-detection imaging algorithms for VNIR and NIR imaging exhibited prediction accuracies of 97.00% (RI547/945) and 100.0% (RI1064/1176, SI1064-1176, RSI-I(1064-1173)/1064, and RSI-II(1064-1176)/(1064+1176)), respectively. The two HSI techniques revealed that spectral images with a pixel size of 1 × 1 mm or 2 × 2 mm had the best classification accuracy for worms. The results demonstrate that hyperspectral reflectance imaging techniques have the potential to detect worms in fresh-cut lettuce.
Future research relating to this work will focus on a real-time sorting system for lettuce that can simultaneously detect various defects such as browning, worms, and slugs. Keywords: VNIR hyperspectral imaging, NIR hyperspectral imaging, biological contaminant, discrimination, fresh-cut lettuce
1. Introduction The improvement in the standard of living and the rising interest among consumers to live a healthy life has led to increased consumption of high-quality and safe agro-food. In addition, the consumption of untreated fresh vegetables and fruits is being promoted. The Korean market for fresh-cut products has been increasing, and it was valued at $7000 million in 2012. Lettuce is one of the most important fresh-cut vegetables. When lettuce is harvested, pests such as cutworms, slugs, insects, and earthworms, and inanimate foreign materials such as gravel and soil are frequently found in it. These foreign materials are removed by washing and screening during postharvest handling. Screening is performed using only the visual skills of operators. Screening operators have difficulty in detecting worms that are small in size or of a color similar to that of lettuce. Therefore, development of technical methods to detect foreign living materials is required. Studies have been performed to detect foreign materials in agricultural products such as insects in bulk wheat using machine vision [1], insect fragments in wheat flour [2–4] and foreign objects in soybean [5] using near-infrared (NIR) spectroscopy, insects inside wheat kernels using NIR imaging [6], and insects in chocolate using multispectral imaging [7]. Machine vision techniques that use image processing detect color differences between agricultural products and defects such as insects [1]. The insect detection algorithm using RGB (Red, Green, and Blue) information showed a correct recognition rate of 92.5% for insects; however, they had difficulty in discriminating wheat and grass seeds that are similar in color to insects. NIR spectroscopy is used to determine the difference in the absorption characteristics of molecules constituting agricultural products and defects [2–5]. Previous studies show that live and dead internal insects can be detected using the NIR technique [2– 4]. NIR spectroscopy, which measures an integrated signal from an entire measurement area, has lower sensitivity than signals measured in a narrow area such as a defect area. Detection of small larvae in bulk wheat samples [4] and inspection for 75 insect fragments in 50 g of wheat flour as per the US Food and Drug Administration action level [2] had low accuracies. To overcome these drawbacks of the machine vision technique and NIR technique, hyperspectral imaging (HSI) and multispectral imaging technologies, which combine imaging and spectroscopic techniques, have been used in recent years to analyze quality [8] and defects in agricultural products [6–7, 9–11]. Spectral imaging methods are employed to measure the spatial distribution or positions of defects and foreign substances by combining
images at specific wavelengths. Furthermore, these techniques have been used to detect biological and surface contamination in agro-food using physical and chemical properties of objects [12–15]. Multispectral imaging may utilize an optimal combination of wavelengths determined using HSI. A multispectral imaging system that uses a multispectral camera can potentially have lower cost and better scan speeds than hyperspectral cameras, and therefore, have better applicability in real-time production systems [12, 16–18]. The two-wavelength subtraction images (1202–1300 nm) for detecting insects inside wheat kernels gives an enhanced view of the differences between infested and control kernels [6]. The two-waveband ratio has been used for detecting fecal contamination on poultry surface [19], and two-wavelength difference images have been used for detecting defects on Fuji apple surfaces [20]. Multispectral imaging based on the three-wavelength regions of 840–870 nm, 870–900 nm, and 900–930 nm showed a detection rate of 93% while inspecting the produce for insects inside chocolate [7]. Combinations of multispectral images improve the detection performance as compared to a single waveband image. The purpose of this study is to develop multispectral imaging methods that can detect worms in lettuce as the basis for the development of foreign substance screening technologies for fresh-cut lettuce. Multispectral imaging algorithms to discriminate worms in fresh-cut lettuce were developed using visible-NIR (VNIR) and NIR HSI techniques. The optimal wavebands that discriminate worms in fresh-cut lettuce were investigated for each type of HSI using one-way ANOVA. The developed algorithms will be employed as the basis for the development of a defect detection system for fresh-cut lettuce.
2. Materials and methods 2.1. Materials Iceberg lettuce produced in the southern part of Korea in 2015 was purchased from a lettuce-producing farm where worms were found. The primary types of worms found in iceberg lettuce are cutworms (Agrotis segetum); in this paper, they are hereafter referred to as worms. The lengths and widths of the worms were in the range of 19.0 mm to 40.1 mm and 1.9 mm to 5.1 mm, respectively. One hundred samples of lettuce that contained worms were used in the experiments. The samples were stored in a refrigerator at 4 °C before the
experiments. The hyperspectral images of worms on the surface of lettuce, which was cut into 5 cm × 5 cm samples, were obtained to develop a worm discrimination method. The samples were allowed to reach room temperature (20 °C) before cutting. Fifty calibration samples (set A) were used for algorithm development and 50 validation samples (set B) were used to test the algorithms.
2.2. Hyperspectral imaging system The following two HSI systems were employed in this study: a VNIR-HSI system with a resolution of 1004 spectral and 1002 spatial pixels in the wavelength range of 400–1000 nm and spectral resolution of 1004 × 1002 pixels, and an NIR-HSI system with a resolution of 640 spectral and 512 spatial pixels in the wavelength range of 980–1700 nm. The VNIR-HSI and NIR-HSI systems were constructed as shown in the schematics in Figs. 1 and 2. The VNIR-HSI system contained a low-light sensitive electron multiplying charge-coupled device camera (MegaLuca, Andor Technology Inc., Belfast, Northern Ireland), an imaging spectrograph (VNIR Hyperspec, HeadwallPhotonics Inc., Fitchburg, MA, USA), a Schneider-Kreuznach Xenoplan 1.4/23 C-mount lens (Schneider Optics, Hauppauge, NY, USA), and a pair of light sources. These light sources employed a pair of halogen-tungsten line lights (LS-F100HS-IR, Seokwang Inc., Hwasung, Korea) arranged toward the instantaneous field of view (IFOV) at off-nadir backward and forward angles of 15° for providing near-uniform excitation energy to a linear sample imaging area. The NIR-HSI system was composed of an InGaAs focal-plane-array (FPA) camera with a resolution of 640 × 512 pixels (Model Xeva-3035, Xenics, Leuven, Belgium), an imaging spectrograph (HyperspecTM NIR G4-249, Headwall Photonics, MA, USA), and a 25-mm zoom lens (Model OB-NIR35/2, Optec, Parabiago, Italy). In addition, the system included a computer for controlling the camera and acquiring images, four 150 W DC light sources with fiber optic bundles (LS-F100HS-IR, Seokwang Inc., Hwasung, Korea), a sample table, and a motorized uniaxial stage (BMS100-UFA, Aerotech, Pittsburgh, PA, USA) attached to the InGaAs FPA camera module and line light guide of the light sources. The FPA camera was thermo-electrically cooled to a temperature of −20 °C using a two-stage Peltier device. The light sources for NIR-HSI employed a pair of halogen-tungsten line lights (LS-F100HS-IR, Seokwang Inc., Hwasung, Korea) attached at 15° toward the IFOV. An imaging spectrograph with a 25-µm slit, a C-mount lens with focus adjustment, and an aperture diaphragm were
also attached. After passing through the 25 µm × 18 mm (width × length) aperture slit, light from a scanned line of the field-of-view (FOV) was dispersed by a dispersive grating and projected onto the FPA camera. A two-dimensional image was obtained with its spatial and spectral dimensions along the horizontal and vertical axes of the FPA, respectively. In the case of reflectance imaging, a spectral image amplified by the FPA camera was captured only in the wavelength range of 900–1700 nm.
Fig. 1. Schematic of the VNIR-HSI system.
Fig. 2. Schematic of the NIR hyperspectral reflectance imaging system.
2.3. Hyperspectral image data acquisition and analysis Ten individual sets of hyperspectral images were obtained using the two types of HSI systems to detect worms in the lettuce. Each set consisted of a 5 × 2 pieces cut leaf arrangement on a lined tray. VNIR images and NIR images were obtained using a line scanning method with an exposure time of 6 ms and 17 ms, respectively. The VNIR images with a resolution of 300 × 502 pixels per spectral band consisted of 125 spectral bands with a waveband interval of 8 nm. The resolution of the NIR images was 300 × 640 pixels, and each image consisted of 224 wavebands with a waveband interval 3.2 nm. White reference images were obtained for VNIR-HSI and NIR-HSI using a 99% diffuse reflectance standard (Spectralon™, SRT-99-120, Labsphere, NH, USA) to calibrate the intensity of the light source for each vertical pixel. Dark reference plate images for compensating the device noise were obtained without using a light source. The dark and white reference images were used to convert the raw reflectance images of worms in the lettuce into corrected reflectance images according to equation (1). Ireflectance (i) =
,
(1)
where Ireflectance, Ir, Id, and Iw denote the corrected reflectance image, the raw hyperspectral image, the dark reference image, and the white reference image, respectively, at the ith wavelength. The reflectance spectra of the pixels composed of worms and lettuce surfaces were extracted from the corrected VNIR and NIR hyperspectral images and used to calculate the average reflectance spectrum. Five waveband indexes were developed using a single-waveband reflectance and a combination of multiple waveband reflectances in the spectra to find an optimal combination of wavebands for discriminating between worms and lettuce. INDEX-1 was developed using one waveband. In addition, the following four types of indexes were developed using two wavebands: INDEX-2 using the ratio of two wavebands (A/B), INDEX-3 using the subtraction of two wavebands (A−B), INDEX-4 using a combination of two wavebands ((A−B)/A), and INDEX-5 using a different combination of two wavebands ((A−B)/(A+B)). A simple statistical comparison is sufficient for determining the significant differences between groups of interest [21]. The analysis of variance (ANOVA) method, which was reported as being capable of finding an optimal waveband for identifying bruised and healthy
areas of apple and pear [9, 22], was applied to determine the best pair of wavelengths for classifying worms in lettuce. The quality index values for the spectra of worms and lettuce were defined as 1 and 0, respectively. ANOVA was performed to determine the F-values between each of the five functions and the quality index values of the samples. The single waveband for INDEX-1, two-waveband ratio for INDEX-2, two-waveband subtraction for INDEX-3, and twowaveband ratio-subtraction functions for INDEX-4 and INDEX-5 were obtained for all possible combinations of one and two wavebands. ANOVA was performed to determine the combinations of wavebands that were the most highly classified for the quality index values of the sample spectra. The spectra extracted within a region of interest (ROI) from the worm and lettuce regions in Set A were used for calibration. Using these optimal wavebands, each of these five INDEX values were developed for use with pixel spectra. Then, a classification value (CV) as an optimal global threshold was determined at the highest classification accuracy. The validation of the developed INDEX values was conducted using five types of pixel spectra from Set B. Five imaging algorithms were developed to classify worms on lettuce surfaces using spectral images and CV of the five developed INDEX values. A single-waveband imaging (SWI) algorithm was developed using the single-waveband image identified by INDEX-1. A ratio imaging (RI) algorithm was developed using the ratio image (RI a/b) of two wavebands selected in INDEX-2 according to equation (2). RIa/b =
,
(2)
where Ia and Ib denote the corrected images at wavelengths a and b, respectively. A subtraction imaging (SI) algorithm was developed using the subtraction image (SI a-b) of the two wavebands identified by INDEX-3 according to equation (3). SIa-b =
,
(3)
where Ia and Ib denote the corrected images at wavelengths a and b, respectively. A ratio-subtraction imaging-I (RSI-I) algorithm was developed using a combination of the ratio and subtraction image (RSI-I(a-b)/a) of two wavebands chosen by INDEX-4 according to equation (4). RSI-I(a-b)/a =
,
(4)
where Ia and Ib denote the corrected images at wavelengths a and b, respectively.
A ratio-subtraction imaging-II (RSI-II) algorithm was developed using a combination of the ratio and subtraction image (RSI-II(a-b)/(a+b)) of two wavebands selected by INDEX-5 according to equation (5). RSI-IIa-b/a+b =
,
(5)
where Ia and Ib denote the corrected images at wavelengths a and b, respectively. The optimal pixel sizes of the discrimination images were investigated to enhance the performance of these worm-detection imaging algorithms. Imaging predictions for four pixel sizes were conducted using the algorithms for the calibration and validation samples. The four pixel sizes were a 1 × 1 pixel of 1 mm × 0.29 mm (Size-0), 1 × 4 pixels of 1 mm × 1.16 mm (Size-1), 2 × 7 pixels of 2 mm × 2.03 mm (Size-2), and 3 × 10 pixels of 3 mm × 2.97 mm (Size-3). For Size-1, if the number of pixels in the positive area of the discrimination images of 1 × 4 pixels of Size-0 was more than 4, then, the value of one pixel merged into 4 pixels was considered to be 1; otherwise, it was considered to be 0. For Size-2 and Size-3, if the number of pixels in the positive area of the discrimination images of 2 × 7 pixels and 3 × 10 pixels of Size-0 was more than 13 and 29, respectively, then, the values of one pixel merged into 14 pixels for Size-2 and into 30 pixels for Size-3 were considered to be 1; otherwise, the value of one pixel merged into 30 pixels was considered to be 0. The sensitivity of classification accuracy of worms refers to the number of positives in which the worms are correctly identified, and specificity of the classification accuracy for lettuce refers to the proportion of negatives in which the lettuce is correctly identified. MATLAB (version 7.0.4, the Mathworks, Natick, MA, USA) was used to extract and analyze the hyperspectral image data.
3. Results and discussion 3.1. Spectral characteristics of lettuce and foreign substances The representative average reflectance spectra of pixels consisting of each worm and lettuce image acquired using VNIR-HSI and NIR-HSI in the ranges of 400–1000 nm and 980–1700 nm, respectively, are shown in Fig. 3a and 3c. Fig. 3b for VNIR-HSI and Fig. 3d for NIR-HSI show the average, maximum, and minimum values of all pixel spectra extracted from corrected hyperspectral images of 50 worms and 50 lettuce samples.
The average VNIR reflectance spectrum of worms is lower than that of lettuce in the range of 400–1000 nm. The reflectance of worms increases in the range of 740–910 nm; however, the reflectance of lettuce is constant (Fig. 3a and 3b). Worm spectra in the range of 500–670 nm do not exhibit peaks; however, lettuce spectra exhibit broad peaks. A wavelength of approximately 681 nm corresponds to the wavelength absorbed by chlorophyll in the lettuce [24]. The NIR reflectance spectra of worms have higher slopes than those of lettuce in the ranges of 1000–1170 nm and 1230–1330 nm, as shown in Fig. 3c and 3d. The peaks at 1130 nm and 1330 nm are associated with the lipids of insects [3]. The average value of all pixel spectra of the 50 worm samples is larger than that of the 50 lettuce samples. Fig. 3b and 3d show the average spectra of pixels extracted using two types of hyperspectral images for 50 lettuce and 50 worm samples. The average value of pixel spectra for all types of hyperspectral images shows decreased pixel-to-pixel variation, and the overlap between the worm and lettuce spectra reduces. In the case of VNIR reflectance spectra, the reflectance of worms is lower than that of lettuce at all wavelengths in the range of 400–1000 nm. VNIR Reflectance
VNIR Spectrum
(b)
(a)
1.0
1.0
0.8
Reflectance
0.8
Reflectance
Lettuce Worms
Average of lettuce spectra Maximum of lettuce spectra Minimum of lettuce spectra Average of worm spectra Maximum of worm spectra Minimum of worm spectra
0.6
0.4
0.6
0.4
0.2
0.2
0.0
0.0 400
500
600
700
Wavelength (nm)
800
900
1000
400
500
600
700
Wavelength (nm)
800
900
1000
NIR Spectrum
(c)
(d) Average of lettuce spectra Maximum of lettuce spectra Minimum of lettuce spectra Average of worm spectra Maximum of worm spectra Minimum of worm spectra
0.8
Lettuce Worms
0.6
Reflectance
Reflectance
0.6
0.8
0.4
0.2
(e)
0.4
0.2
0.0
0.0
1000
1200
1400
1600
1000
Wavelength (nm)
1200
1400
1600
Wavelength (nm)
Fig. 3. (a) Pixel spectra with average, minimum, and maximum values and (b) average spectra of worms and lettuce acquired using VNIR-HSI; (c) pixel spectra with average, minimum, and maximum values and (d) average spectra of worms and lettuce acquired using NIR-HSI.
3.2. Algorithms for discriminating worms in lettuce Table 1 and Table 3 show the optimal wavebands and classification results for each index in terms of worm detection using pixel spectra for VNIR-HSI and NIR-HSI, respectively. Using pixel spectra for analysis can include deviation in spectra, rather than using average spectra.
3.2.1. VNIR-HSI 3.2.1.1. Spectrum index Worm detection indexes were developed using VNIR-HSI. First, single-waveband INDEX1 was developed for discriminating worms in lettuce. The F-value of each waveband obtained using ANOVA to distinguish between worms and lettuce for the single waveband using the calibration samples (Set A) is shown in Fig. 4a. The ANOVA results featured a peak with an F-value of 144,460.7 at a wavelength of 518 nm, which is associated with carotenoids [23]. The second peak in the F-value was observed at 725 nm, which is related to chlorophyll a [34]. A wavelength of 518 nm is included in the broad peak region of the spectra for lettuce and in the no-peak region of the spectra for worms. The lowest F-value was observed at 959 nm, which is close to the wavelength for water absorption [11], as shown in Fig. 3a. The classification accuracies for worms and lettuce at 518 nm for the validation sample
(Set B) were 96.17% and 97.5%, respectively. The threshold value with the best classification accuracy was 0.238, as shown in Table 1.
Table 1. Optimal wavebands and discrimination results of worm and lettuce using a singlewaveband INDEX and the combination of two-waveband INDEX values for VNIR spectra Waveband F-value A
CV1)
B
CA2) of calibration (%) CA2) of validation (%) Worm Lettuce Total Worm Lettuce Total
INDEX-13)
518
144,460.7 0.238 99.12
99.05 99.09 96.17 97.50 96.83
INDEX-24)
547
945 219,423.1 0.730 99.29
99.72 99.51 97.70 99.17 98.44
INDEX-35)
744
945 118,429.6 0.026 98.14
99.27 98.71
INDEX-46)
403
523 141,121.1 0.175 97.79
99.01 98.40 95.80 98.55 97.18
INDEX-57)
523
950 169,883.9 0.145 99.21
99.38 99.30 97.43 98.50 97.96
Note: 1)CV : Classification value, 2)CA : Classification Accuracy,
3)
917
99.29 96.73
INDEX-1 : Single waveband, 4) INDEX-2
: Two-waveband ratio (A/B), 5)INDEX-3 : Two-waveband subtraction (A− B),
6)
INDEX-4: Combination of
two wavebands ((A− B)/A), 7) INDEX-5: Combination of two wavebands ((A− B)/(A+B))
(a) 1.6 1.4
F-value (105)
1.2 1.0 0.8 0.6 0.4 0.2 0.0
400
500
600
700
800
900
1000
Wavelength (nm) Col 1 vs Col 2
Fig. 4. Results of one-way ANOVA for classifying worms and lettuce using (a) INDEX-1 of single waveband and (b) INDEX-2 of two-waveband ratio for the VNIR spectrum from 400–1000 nm.
The discrimination index for detecting worms on lettuce surfaces was developed using four combinations—ratio, subtraction, and combinations of ratio, subtraction, and addition—of two wavebands in the range from 400–1000 nm. INDEX-2 of the two-waveband ratio had a maximum F-value of 219,423.1 for the four combinations of two wavebands, as shown in Fig. 4b. The optimal wavelengths were 547 nm, which is related to the characteristics of carotenoids [23] and is close to the waveband with the highest F-value for a single waveband, and 945 nm, which is associated with the characteristics of water [12] and is close to the waveband with the lowest F-value for a single waveband. INDEX-2 for the calibration samples had a maximum classification accuracy of 99.29% for worms and 99.72% for lettuce, with a threshold value of 0.73. The validation results obtained using Set B demonstrated that INDEX-2 had a maximum classification accuracy of 97.70% for worms and 99.17% for lettuce, with a threshold value of 0.73. The classification performances of all two-waveband indexes improved as compared to that of the single-waveband index. Thus, worms can potentially be detected on lettuce surface using VNIR reflectance spectra.
3.2.1.2. VNIR imaging algorithm The imaging algorithms for worm detection were developed using hyperspectral VNIR images. Worm detection methods were investigated using single and multi-wavelength images with optimal results of ANOVA.
Single-waveband imaging algorithm Fig. 5 shows a sequence of representative images processed using single-waveband images to classify worms. A 657 nm image (Fig. 5a) was transformed into a masking image (Fig. 5b) binarized by a threshold reflectance value of 0.02. The masking image was created to remove the backgrounds and to select lettuce and worms. The 518 nm waveband image (Fig. 5c) was produced by applying the masking image and then transformed into the binary image (Fig. 5d) using a threshold reflectance value of 0.238. The enhanced image (Fig. 5e) used to reduce misclassification of the binary image was created using the pixel value modified with the value of the several adjacent pixels.
This threshold value yielded the best accuracy for
discriminating between the worm and lettuce when using the calibration samples (Set A). The black areas in the binary image represent worms.
Fig. 5. Illustration of image processing sequence using a single waveband: (a) grayscale I684 image; (b) masking images obtained by applying a reflectance intensity threshold of 0.02 to the I684 image; (c) grayscale and color images of I525 after masking; (d) resulting detection images after application of a threshold of 0.238; (e) enhanced image modified using pixel size.
Multispectral imaging algorithm Fig. 6 shows the sequence of images corresponding to the image processing method used to distinguish worm areas in lettuce samples using the dual-waveband index (INDEX-2547/945). The 657 nm image with the highest difference in intensity between the background and lettuce was selected as the masking image for the two wavebands (Fig. 6a). The binary image was created using the masking image following the same method as that for the SWI algorithm (Fig. 7b). A threshold value of 0.02 was obtained using the global threshold method. The ratio image for 547 nm and 945 nm (Fig. 6d) was created using the 547 nm and 945 nm images after application of the masking image (Fig. 6c). The ratio image was transformed into a binary image using the threshold value (Fig. 6e). The classification rate for worm and lettuce areas, obtained using the dual-waveband ratio, is shown in Fig. 6. A threshold value of 0.73 yielded the best classification accuracy for worm and lettuce areas using the calibration samples (Set A). If the pixel value of the ratio image was higher than the threshold value, the pixel was assigned a value of 1; otherwise, it was assigned a value of 0. The positive areas in the binary images represent worms.
Fig. 6. Illustration of image processing sequence for the RI algorithm using two wavebands: (a) grayscale I657 image; (b) masking image obtained by applying a reflectance threshold of 0.02 to the I657 image; (c) I547 and I945 images after masking; (d) the grayscale ratio image of I547/945; (e) resulting detection image after application of a threshold of 0.73; (f) enhanced image modified using pixel size.
Improvement of VNIR imaging algorithm using pixel size Imaging predictions for four kinds of pixel sizes for the calibration and validation samples were conducted using the five imaging algorithms for VNIR reflectance. The imaging prediction results are shown in Table 2. The images of worm detection on samples, obtained using the five algorithms for VNIR-HSI, are shown in Fig. 7. In the case of Size-0, the sensitivities and worm prediction accuracies of the SWI algorithm were above 99.9% and 90.0% for calibration and validation, respectively. For lettuce, the specificities and prediction accuracies of the SWI algorithm were 18.0% and 6.00% for calibration and validation, respectively. Low specificity and prediction accuracy is caused by lettuce pixels below 0.238 reflectance intensity in the 518 nm image associated with carotenoids, as shown in Fig. 3a. The best results were obtained using the dual-waveband RI algorithm images related to carotenoids and water with specificities of 52.0% and 56.0% for calibration and validation, respectively. The sensitivity and prediction accuracy of the RI algorithm were 98.0% and 77.0%, respectively. This algorithm exhibited better performance than the SWI algorithm. Low specificity was the reason that the lettuce area was recognized
as a worm area, resulting in false positive pixels. To overcome this problem, the size of a single pixel was modified to 1 × approximately 1 mm, 2 × approximately 2 mm, and 3 × approximately 3 mm, with the number of pixels modified to 1 × 4 pixels, 2 × 7 pixels, and 3 × 10 pixels, respectively. In the case of Size-1 with 1 × 1 mm pixels, the SWI algorithm had a sensitivity of more than 99.9% and a specificity of 50.0% for calibration, and a sensitivity of more than 90.0% and a specificity of 32.0% for validation. The maximum sensitivity of the RI and RSI-II algorithms was more than 99.9% for validation and the corresponding maximum specificity of the SI algorithm was more than 99.9%. The prediction accuracy of the SI algorithms surpassed that of all other algorithms. All dual-wavelength algorithms exhibited improved performance as compared to the SWI algorithm. The specificity of all algorithms for Size-1 was better than that for Size-0. Using a detection pixel size of approximately 1 mm decreased the detection error for lettuce. In the case of Size-2 with 2 × 2 mm pixels, the sensitivity of the SWI algorithm was 90.0% for validation, which was equal to the sensitivity of the SWI algorithm for Size-1; however, its specificity increased to 88.0%. The optimal sensitivity and prediction accuracy of the dualwavelength RI algorithm were 96.0% and 97%, respectively. The optimal specificities of the SI and RSI-I dual-wavelength algorithms were more than 99.9% for validation. Compared to the SWI algorithm, the specificities of the four dual-wavelength algorithms increased; however, their sensitivities decreased. The number of worm pixels detected using all algorithms decreased for Size-2. The prediction results for all algorithms obtained using approximately 3 × 3 mm pixel images (Size-3) indicated that specificities increased; however, sensitivities and prediction accuracies decreased. For worm detection, Size-1 with approximately 1 mm pixels may be more suitable than Size-2 with approximately 2 mm pixels because the rate of the pixels with a value of 1 in the worm areas for Size-1 was larger than that for Size-2. In addition, the detection accuracy for lettuce for Size-1 was slightly higher than that for Size-2. These results show that the RI algorithms with a pixel size of approximately 1 mm for VNIR reflectance HSI have the capacity to detect worms in lettuce. The classification accuracy of worm detection using VNIR reflectance HSI surpassed the 93% detection accuracy for insects in chocolate, obtained using VNIR multispectral images
of 840–870 nm, 870–900 nm, and 900–930 nm regions [7]. These results show that the RI algorithms for VNIR reflectance HSI exhibit the potential to discriminate between worms and lettuce.
Fig. 7. Prediction results for imaging algorithms of VNIR-HSI for each pixel size.
Table 2. The results of worm detection of four pixel sizes for five imaging algorithms of VNIR hyperspectral images CA1)
Algorithms
Wavebands
Size-0
Size-1
Size-2
Size-3
Pixel size of 1×1 pixel (1× 0.29 mm)
Pixel size of 1× 4 pixel (1×1.16mm)
Pixel size of 2×7 pixel (2×2.03mm)
Pixel size of 3× 10 pixel (3×2.9mm)
Calibration SE2)
Validation
Calibration SE2)
Validation
Calibration
SP3)
PA4)
SE2)
SP3)
PA4)
SP3)
PA4)
SE2)
SP3)
PA4)
SE2)
SP3)
90.0
32.0
61.0 100.0 88.0
Validation
Calibration
PA4)
SE2)
SP3)
PA4)
SE2)
90
90.0
88.0
89.0
93.0 100.0 100.0 100.0 96.0
98.0
SP3)
PA4)
Validation SE2)
SP3)
PA4)
72.0 100.0 86.0
70
96.0
85.0
96.0
80
SWI5)
518
100.0 18.0
59.0
90.0
6.0
48.0 100.0 50.0
75.0
RI6)
547/945
100.0 52.0
76.0
98.0
56.0
77.0 100.0
92.0 100.0 86.0
97.0
70.0 100.0 85.0
72.0
SI7)
744-945
100.0 32.0
66.0
90.0
28.0
59.0 100.0 100.0 100.0 90.0 100.0 95.0 100.0 100.0 100.0 90.0 100.0 95.0
62.0 100.0 81.0
60
52.0
96.0
2.0
49.0 100.0 90.0
95.0
69.0
98.0
48.0
73.0 100.0 80.0
90.0 100.0 82.0
RSI-I8) RSI-II9)
(403-523)/403 100.0 (523-950) /(523+950)
0
100.0 38.0
Note: 1) CA : classification accuracy, 2) SE : sensitivity, 8)
3)
80
SP : specificity,
waveband subtraction image, RSI-I: Ratio-Subtraction image-I,
9)
4)
98.0
90.0
90
100.0 98.0
99.0
91.0 100.0 96.0
98.0
90.0 100.0 95.0 90
90
PA : predictive accuracy, 5) SWI : single-waveband image,
RSI-II: Ratio-Subtraction image-II.
90
6)
60
100.0 82.0
76.0 100.0 88.0
100.0 82.0
52.0 100.0 76.0 70.0
96.0
RI : two-waveband ratio image,
7)
83.0
SI : two-
3.2.2. NIR-HSI 3.2.2.1. Spectrum index Worm detection indexes were developed using NIR-HSI in a manner similar to that for VNIRHSI. First, INDEX-1 was developed for a single waveband. The F-value of each waveband obtained using the calibration samples (Set A) is shown in Fig. 8a. The results of ANOVA showed an optimal wavelength of 1000 nm close to the 1020 nm peak associated with proteins [12, 25] because the protein content of worms is higher than that of lettuce. The F-value was 48,810.3. The second peak in the F-value was observed at 1208 nm, which is related to the 2nd harmonic C-H stretching of CH2 in lipids [12, 25]. The accuracies of classification results at 1000 nm for the validation sample (Set B) were 98.04% and 93.65% for worms and lettuce, respectively. The threshold value with best classification accuracy was 0.3791, as shown in Table 3.
Table 3. Optimal wavebands and discrimination results of worm and lettuce using a single waveband INDEX and the combination of two-waveband INDEXs for NIR spectra Waveband F-value A 3)
INDEX-1 INDEX-24) INDEX-35) INDEX-46) INDEX-57)
1000 1064 1064 1064 1064
1)
CA2) of calibration (%)
CA2) of validation (%)
CV
B 48,810.3 0.3791 1176 223,460.5 0.98 1176 234,199.1 0.0087 1173 199,087.1 0.03 1176 211,656.1 0.145
Worm
Lettuce
918
937
928
98.04
93.65
95.85
97.80 97.60 97.74 99.21
99.82 99.71 99.59 99.38
98.81 98.66 98.67 99.30
99.61 99.40
99.94 99.91
99.77 99.65
99.44 99.61
99.97 99.94
99.70 99.77
Note: 1)CV : Classification value, 2)CA : Classification Accuracy,
3)
Total Worm Lettuce Total
INDEX-1 : Single waveband,
waveband ratio (A/B), 5)INDEX-3 : Two-waveband subtraction (A− B),
6)
((A− B)/A), 7) INDEX-5: Combination of two wavebands ((A− B)/(A+B))
4)
INDEX-2 : Two-
INDEX-4: Combination of two wavebands
(a) 6
5
F-value (104)
4
3
2
1
0 1000
1200
1400
1600
Wavelength (nm)
Fig. 8. Results of one-way ANOVA for classifying worms and lettuce using (a) INDEX-1 of single waveband and (b) INDEX-3 of two-waveband subtraction for the NIR spectrum from 980–1700 nm.
The discrimination indexes for the dual waveband in the range of 980–1700 nm were developed in the same manner as that for VNIR-HSI. INDEX-3 of two-waveband subtraction in the four dual-waveband algorithms had a maximum Fvalue of 234,199.1 (Fig. 8b). The optimal wavebands were 1064 nm and 1176 nm, which are close to the peaks in the characteristics of protein and lipids [12, 25]. Similar to the F-value results for a single waveband, the F-values for the two-waveband ratio were higher in the combined region of 1000 nm–1100 nm, which is related to the characteristics of proteins, and 1160 nm–1190 nm, which is related to the characteristics of lipids [12, 25]. The classification accuracies obtained using the calibration samples (Set A) were 97.60% and 99.71% for worms and lettuce, respectively. The threshold value with the best classification accuracy was 0.0087, as shown in Table 3. INDEX-2 of the two-waveband ratio for the calibration samples had a maximum classification accuracy of 97.80% and 99.82% for worms and lettuce, respectively, with a threshold value of 0.98. The best wavebands for INDEX-2 were 1064 nm and 1176 nm with an F-value of 223,460.5. The results were validated using the validation samples (Set B). It was observed that INDEX-2 and INDEX-5 had maximum classification accuracies of 99.61% and 99.97% for worms and lettuce, with threshold values of 0.98 and 0.145, respectively. The best wavebands for these two indexes were
1064 nm and 1176 nm. Thus, worms can potentially be detected on lettuce surface using NIR reflectance spectra.
3.2.2.2. NIR imaging algorithms The single-waveband imaging algorithm for classifying worms on lettuce surface was developed using an image at 1000 nm that had the highest F-value according to ANOVA. The image processing used to develop the algorithm was similar to that for VNIR-HSI. A multispectral imaging algorithm to distinguish worm areas on lettuce samples was developed using a combination of ratio and subtraction of dual-waveband imaging. The image processing used to develop the algorithm was similar to that for VNIR-HSI. Imaging predictions for four pixel sizes were conducted for the calibration and validation samples using the five algorithms for NIR reflectance. The imaging prediction results are shown in Table 4. In the case of Size-0 with 1 × 1 pixel in NIR-HSI, the images of worm detection on samples obtained using the five algorithms are shown in Fig. 9. For worms, the sensitivity and prediction accuracy of the SWI algorithm were 90% and more than 99.9% for calibration and validation, respectively. For lettuce, the specificity and prediction accuracy of the SWI algorithm were 0.0% and 2.0% for calibration and validation, respectively. Low specificity and prediction accuracy is caused by pixels in the lettuce areas that are below 0.3791 reflectance intensity in the 1000 nm image in Fig. 3c, which is related to protein content. The dual-waveband subtraction image (SI) algorithm images related to protein and lipid contents had a maximum sensitivity, specificity, and prediction accuracy of more than 99.9%, 82.0%, and 91.0%, respectively, for the validation sample, which was a considerably better performance than that of the SWI algorithm. The reason for low discrimination accuracy for lettuce could be that lettuce was recognized as a worm owing to false positive pixels. For Size-1 with 1 × 1 mm pixels for NIR-HSI, the sensitivity of the SWI algorithm was more than 90.0% and more than 99.9% for calibration and validation, respectively, and its specificity was 0.0% and 2.0% for calibration and validation, respectively. The sensitivity and specificity of all dual-waveband SI algorithms were more than 99.9% for validation. All dual-waveband algorithms performed better than the SWI algorithm. The specificities of all algorithms for Size-1 were higher than that for Size-0. Using a pixel of 1 mm size decreased the detection error rate for lettuce. In the case of Size-2 with 2 × 2 mm pixels in NIR-HSI, the sensitivity of the SWI algorithm was 99.9% for validation, which was equal to that for Size-1, and its specificity was 16.0%, which was slightly more than that for Size-1. The four dual-waveband algorithms exhibited more than 99.9% discrimination of worms and lettuce for validation, which was similar to that for Size-1. The
specificities of the four dual-waveband algorithms for Size-2 were higher than that for Size-1. Likewise, the sensitivities of the four dual-waveband algorithms for Size-2 were equal to or more than that for Size-1. The number of worm pixels detected using all algorithms decreased for Size-2. For Size-3, the prediction results for all algorithms developed using NIR-HSI indicated an increase in the specificities; however, the sensitivities were the same for validation and decreased for calibration, as compared to those for Size-2. The number of worm pixels detected using all algorithms decreased for Size-3. Size-1 with a pixel size of 1 mm may be more suitable for worm detection than Size-2 with a pixel size of 2 mm because the number of pixels with a value of 1 in the worm areas for Size-1 were larger than that for Size-2. In addition, the detection accuracy for lettuce for Size-1 was slightly higher than that for Size-2. These results show that the four dual-waveband algorithms for NIR reflectance HSI for a pixel size of 1 mm exhibit the capacity to detect worms in lettuce. The classification performance for worm detection using NIR reflectance HSI exceeded the 92%– 93% detection accuracy for dead pupae and live larvae in wheat kernels, obtained using NIR reflectance spectroscopy [4]. These results indicate that the worm detection algorithms for NIR reflectance HSI have the potential to discriminate between worms and lettuce.
Fig. 9. Prediction results for imaging algorithms of NIR-HSI for each pixel size.
Table 4. The results of worm detection of four pixel sizes for five imaging algorithms of NIR hyperspectral images CA1)
Algorithms Wavebands
Size-0
Size-1
Size-2
Size-3
Pixel size of 1×1 pixel (1× 0.29 mm) Calibration Validation
Pixel size of 1× 4 pixel (1×1.16mm) Calibration Validation
Pixel size of 2×7 pixel (2×2.03mm) Calibration Validation
SE2)
SP3)
PA4)
SE2)
47.0 100.0
SP3)
PA4)
SE2)
SP3)
PA4)
SP3)
PA4)
Pixel size of 3× 10 pixel (3×2.9mm) Calibration Validation 4) 2) 3) 4) 2) SE SP PA SE SP3) PA
2.0
51.0
90.0
26.0
58.0 100.0 16.0
58.0
86.0
90.0
90.0 100.0 95.0 100.0 98.0
99.0
90.0 100.0 95.0 100.0 100.0 100.0 82.0 100.0 91.0
96.0 100.0 98.0
SP3)
PA4)
SE2)
SP3)
PA4)
SE2)
2.0
51.0
90.0
0.0
45.0 100.0
100.0 80.0
SE2)
65.0
SWI5)
1000
90
0.0
RI6)
1064/1176
90
70
SI7)
1064-1176
90
72.0
83.0 100.0 82.0
91.0
90.0 100.0 95.0 100.0 100.0 100.0 90.0 100.0 95.0 100.0 100.0 100.0 82.0 100.0 91.0
98.0 100.0 99.0
RSI-I8)
(1064-1173) /1064
90
70.0
82.0 100.0 78.0
89.0
90.0 100.0 95.0 100.0 98.0
99.0
90.0 100.0 95.0 100.0 100.0 100.0
100.0 92.0
92.0 100.0 96.0
RSI-II9)
(1064-1176) /(1064+1176)
90
68.0
81.0 100.0 76.0
88.0
90.0
99.0
90.0 100.0 95.0 100.0 100.0 100.0 82.0 100.0 91.0
96.0 100.0 98.0
Note: 1) CA : classification accuracy,
80
2)
98.0
SE : sensitivity, 3) SP : specificity,
4)
90
100.0 98.0
PA : predictive accuracy,
waveband subtraction image, 8) RSI-I: Ratio-Subtraction image-I, 9) RSI-II: Ratio-Subtraction image-II
5)
SWI : single-waveband image,
6)
80
42.0
60
100.0 30.0
RI : two-waveband ratio image, 7) SI: two-
5. Conclusions Nondestructive methods based on three types of HSI techniques that use VNIR and NIR were developed to detect worms in fresh-cut lettuce. A single wavelength index and multispectral indexes were developed for each HSI technique. The optimal wavebands for discriminating between worms and lettuce were investigated using one-way ANOVA. The worm detection indexes developed using VNIR and NIR spectra exhibited a prediction accuracy of 98.44% for INDEX-2547/945, and 99.77% for INDEX-21064/1176 and INDEX-5(10641176)/(1064+1176).
In addition, image algorithms were developed for the two HSI techniques
using images of four different pixel sizes to detect worms. The results showed that the spectral images with a pixel size of 1 × 1 mm or 2 × 2 mm had the best classification accuracy for worms. The RI547/945 algorithm for VNIR-HSI, and the four image algorithms, i.e., RI1064/1176, SI1064-1176, RSI-I(1064-1173)/1064, and RSI-II(1064-1176)/(1064+1176), for NIR-HSI had the highest prediction accuracies of 97.0% and 100%, respectively. The results of this study indicate that the NIR single and multispectral imaging algorithms used to detect worms in lettuce have the best discrimination accuracies for the two types of HSI techniques. Therefore, spectral information from a hyperspectral image can be applied to develop an economic real-time multispectral imaging system to detect worms in fresh-cut lettuce.
Acknowledgments This study was supported by “Research Program for Agricultural Science & Technology Development (Project Nos. PJ012487 and PJ00939901),” National Institute of Agricultural Science, Rural Development Administration, Republic of Korea.
Conflicts of Interest The authors declare no conflicts of interest.
Reference 1. I.Y. Zayas, P.W. Flinn, Detection of insects in bulk wheat samples with machine vision. Trans. ASAE 41(1998) 883-888.
2. J. Perez-Mendoza, J.E. Throne, F.E. Dowell, J.E. Baker, Detection of insect fragments in wheat flour by near-infrared spectroscopy. J. Stored Prod. Res. 39 (2003) 305–312. 3. F.E. Dowell, J.E. Throne, J.E. Baker. 1998. Automated nondestructive detection of internal insect infestation of wheat kernels by using near-infrared reflectance spectroscopy. J. Econ. Entomol. 91(1998) 899-904. 4. E.B. Maghirang, F.E. Dowell, J.E. Baker, J.E. Throne, Automated detection of single wheat kernels containing live or dead insects using near–infrared reflectance spectroscopy. Trans. ASAE. 46(2003) 1277–1282. 5. J.G. Lim, S.W. Kang, K.J. Lee, C.Y. Mo, J.Y. Son, Identification of Foreign Objects in Soybeans Using Near-infrared Spectroscopy, J. Food Eng. 15 (2011) 136-142. (In Kore an, with English abstract). 6. C. Ridgeway, J. Chambers, Detection of insects inside wheat kernels by NIR imaging. J. Near Infrared Spectrosc. 6 (1998) 115–119. 7. T. Ma, H. Kobori, N. Katayama, S. Tsuchikawa, Nondestructive inspection of insects in chocolate using near infrared multispectral imaging. J. Near Infrared Spectrosc. 24 (2016) 391-397. 8. C. Mo, J. Lim, K. Lee, S. Kang, M.S. Kim, G. Kim, B.K. Cho, Determination of germination quality of cucumber(cucumis sativus) seed by LED-induced hyperspectral reflectacne imaging. J. Biosyst. Eng. 38(2013) 318-326 9. B.K. Cho, I.S. Beak, N.G. Lee, C.H. Mo, Study on bruise detection of “Fuji” apple using hyperspectral reflectance imagery, J. Biosyst. Eng. 36 (2011) 484–490. 10. H. Lee, C.C. Yang, M.S. Kim, J. Lim, B.K. Cho, A. Lefcourt, K. Chao, C.D. Everard, A simple multispectral imaging algorithm for detection of defects on red delicious apples. J. Biosyst. Eng. 39 (2014) 142–149. 11. I. Murray, P.C. Williams, Chemical principles of near-infrared technology. In P. Williams & K. Norris (Eds.), Near-infrared technology in the agricultural and food industries (pp. 17–34). St. Paul, MN, USA: American Association of Cereal Chemists, 1987. 12. D. Wu, D.W. Sun, Advanced applications of hyperspectral imaging technology for food quality and safety analysis and assessment; A review—Part I; Fundamentals, Innov. Food Sci. Emerg. Technol. 19 (2013) 1–14.
13. M.S. Kim, Y.R. Chen, P.M. Mehl, Hyperspectral reflectance and fluorescence imaging system for food quality and safety, Trans. ASAE. 44 (2001) 721-729. 14. W. Jun, M.S. Kim, K. Lee, P. Millner, K. Chao, Assessment of bacterial biofilm on stainless steel by hyperspectral fluorescence imaging, Sens. Instrum. Food Qual. Saf. 3 (2009) 41–48. 15. L.M. Kandpal, B.K. Cho, A Review of the Applications of Spectroscopy for the Detection of Microbial Contaminations and Defects in Agro Foods. J. Biosyst. Eng. 39 (2014) 215–226. 16. M.S. Kim, A.M. Lefcourt, K. Chao, Y.R. Chen, I. Kim, D.E. Chan, Multispectral detection of fecal contamination on apples based on hyperspectral imagery; Part I. Application of visible and near-infrared reflectance imaging, Trans. ASAE. 45 (2002) 2027–2037. 17. H.Y. Lee, C. Everard, S. Kang, B.K. Cho, K. Chao, D. Chan, M.S. Kim, Multispectral fluorescence imaging for detection of bovine feces on Romaine lettuce and baby spinach leaves, Biosyst. Eng. 127 (2014) 125–134. 18. D.H. Jeong, M.S. Kim, H. Lee, H. Lee, B.K. Cho, Detection algorithm for cracks on the surface of tomatoes using multispectral VIS/NIR reflectance imagery, J. Biosyst. Eng. 38 (2013) 199–207. 19. B. Park, W.R. Windham, K.C. Lawrence, D. Smith, Contaminant classification of poultry hyperspectral imagery using a spectral angle mapper algorithm, Biosyst. Eng. 96 (2007) 323-333. 20. K.J. Lee, S.W. Kang, S.R. Delwiche, M.S. Kim, S.H. Noh Correlation analysis of hyperspectral imagery for multispectral wavelength selection for detection of defects on apples, Sens. Instrum. Food Qual. Saf. 2 (2008) 90–96. 21. M.H. Kutner, Applied Linear Statistical Models. McGraw-Hill Irwin, Boston, USA, 2005. 22. W.H. Lee, M.S. Kim, H. Lee, S.R. Delwiche, H. Bae, D.Y. Kim, B.K. Cho, Hyperspectral near-infrared imaging for the detection of physical damages of pear. J. Food Eng. 130(2014) 1-7. 23. M.N. Merzlyak, A.E. Solovchenko, A.A. Gitelson, Reflectance spectral features and non-destructive estimation of chlorophyll, carotenoid and anthocyanin content in apple fruit. Postharvest Biol. Technol. 27(2003) 197–211.
24. J. Gross, Pigments in Vegetables; Chlorophylls and Carotenoids, Van Nostrand Reinhold, New York, NY, USA, 1991. 25. H.I Chung, H.J. Kim, Near-Infrared Spectroscopy, Principles Analytical Sci. Tech. 12 (2000) 1-14.
Highlights
Visible-near-infrared and near-infrared hyperspectral imaging technology were employed for developing the method to detect worms on fresh-cut lettuce.
The optimal wavebands to discriminate worm and lettuce were investigated using the one-way ANOVA analysis.
The worm detection multispectral imaging algorithms resulted in the prediction accuracy of 97% for VNIR hyperspectral imaging and the prediction accuracy of 100.0% for NIR hyperspectral imaging .
The spectral images with the pixel size of 1 × 1 mm or 2 × 2 mm classification accuracy for detecting worms.
had the
best