- Email: [email protected]
Detection of citrus canker using hyperspectral reflectance imaging with spectral information divergence
Journal of Food Engineering 93 (2009) 183–191
Contents lists available at ScienceDirect
Journal of Food Engineering journal homepage: www.elsevier.com/locate/jfoodeng
Detection of citrus canker using hyperspectral reflectance imaging with spectral information divergence Jianwei Qin a,*, Thomas F. Burks a, Mark A. Ritenour b, W. Gordon Bonn c a
Department of Agricultural and Biological Engineering, P.O. Box 110570, University of Florida, Gainesville, FL 32611-0570, USA Department of Horticultural Sciences, 2199 South Rock Road, University of Florida, Fort Pierce, FL 34945-3138, USA c Division of Plant Industry, Florida Department of Agriculture and Consumer Services, Gainesville, FL 32614-7100, USA b
a r t i c l e
i n f o
Article history: Received 18 July 2008 Received in revised form 16 January 2009 Accepted 17 January 2009 Available online 30 January 2009 Keywords: Hyperspectral imaging Spectral similarity Citrus canker Disease detection Food safety
a b s t r a c t Citrus canker is one of the most devastating diseases that threaten marketability of citrus crops. This research was aimed to develop a hyperspectral imaging approach for detecting canker lesions on citrus fruit. A hyperspectral imaging system was developed for acquiring reflectance images from citrus samples in the spectral region from 450 to 930 nm. Ruby Red grapefruits with cankerous, normal and other common peel diseases including greasy spot, insect damage, melanose, scab, and wind scar were tested. Spectral information divergence (SID) classification method, which was based on quantifying the spectral similarities by using a predetermined canker reference spectrum, was performed on the hyperspectral images of the grapefruits for differentiating canker from normal fruit peels and other citrus surface conditions. The overall classification accuracy was 96.2% using an optimized SID threshold value of 0.008, which was determined under the condition that the errors of false negative and false positive were weighted equally. Considering the high economic impact of missing a cankerous fruit, zero false negative error was achieved by using a threshold value of 0.009, under which the classification accuracy was 95.2%. This research demonstrated that hyperspectral imaging technique coupled with the SID based image classification method could be used for discriminating citrus canker from other confounding diseases. Ó 2009 Elsevier Ltd. All rights reserved.
1. Introduction Citrus canker is a severe disease that can affect the peel of most commercial citrus varieties. The pathogen, usually characterized by conspicuous, erumpent lesions on leaves, stems, and fruit, could cause defoliation, blemished fruit, premature fruit drop, twig dieback, and tree decline (Schubert et al., 2001). Although there are various disorders that could infect citrus fruit, canker is considered as one of the most devastating diseases that threaten citrus crops. Since canker does not exist in all citrus-growing regions of the world, considerable effort has been made to prevent the spread of the disease to the unaffected areas where the climate is conducive for canker development. The Citrus Canker Eradication Program (CCEP), initialized by the Florida Department of Agriculture and Consumer Services – Division of Plant Industry (FDACS-DPI) and the United States Department of Agriculture – Animal and Plant Health Inspection Service (USDA-APHIS) in mid 1990s, was an attempt to alleviate the consequences that canker would have on the Florida citrus industry, as well as to keep other US citrusproducing areas (e.g., Texas and California) from being infected * Corresponding author. Tel.: +1 352 392 1864x233. E-mail address: qinj@ufl.edu (J. Qin). 0260-8774/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.jfoodeng.2009.01.014
and harmed by this disease. Unfortunately, there is no treatment or prevention at this time that could completely eradicate citrus canker in infected areas. The current emphasis is on minimizing the level of the disease and avoiding further infections. The presence of cankerous fruit in a shipment could result in the whole shipment being deemed unmarketable to some ‘canker-free’ areas, such as the European countries. Therefore it is crucial to detect citrus canker and eliminate the infected fruit in the packing houses. Manual removal of cankerous fruit is time-consuming and laborintensive. Technologies that can efficiently identify citrus canker would assure fruit quality and safety, which would enhance the competitiveness and profitability of the citrus industry. Optical sensing technologies (e.g., machine vision and spectroscopy) offer great potential for quality evaluation and safety inspection for food and agricultural products. A number of techniques have been studied to detect defects and diseases related to citrus. Gaffney (1973) obtained reflectance spectra of citrus fruit and some surface defects. Edwards and Sweet (1986) developed a method to assess damages due to citrus blight disease on citrus plants using reflectance spectra of the entire tree. Miller and Drouillard (2001) collected data from Florida grapefruit, orange, and tangerine varieties using a color vision system. They used various neural network classification strategies to detect blemish-related features for the
184
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
citrus fruit. Aleixos et al. (2002) developed a multispectral camera system that could acquire visible and near infrared images from the same scene, and used it on a real-time system for detecting defects on citrus surface. Pydipati et al. (2006) utilized the color cooccurrence method (CCM) to extract various textural features from the color RGB images of citrus leaves. The CCM texture statistics were used to identify diseased and normal citrus leaves using discriminant analysis. Most recently, Blasco et al. (2007) reported the application of near-infrared, ultraviolet, and fluorescence computer vision systems to identify common defects of citrus fruit. They proposed a fruit sorting algorithm that combines the different spectral information to classify fruit according to the type of the defect. These research works provided good references and resources for dealing with various problems associated with detection of citrus surface defects and diseases. However, to our knowledge no research has been conducted for detecting canker on citrus fruit. There is a need for studying the unique features of citrus canker and obtaining the fundamental information that will be useful for designing the canker detection system. In the recent years, hyperspectral imaging technique has been developed as a useful tool for inspecting internal and external attributes of food and agricultural products (Lu and Chen, 1998; Kim et al., 2001). The technique combines conventional spectroscopy and imaging techniques to acquire both spectral and spatial information from an object. Compared to traditional visible/nearinfrared spectroscopy that essentially relies on spot measurement, hyperspectral imaging has more advantages for evaluating whole surface of the individual food items. The technique has been applied for analyzing physical and/or chemical properties of a broad range of food and agricultural products. Examples include detecting contaminants on apples (Kim et al., 2002) and poultry carcasses (Park et al., 2007), diseases on poultry carcasses (Chao et al., 2007), bruises on apples (Lu, 2003) and cucumbers (Ariana et al., 2006), rottenness on citrus (Gómez-Sanchis et al., 2008), pits in tart cherries (Qin and Lu, 2005), and cracks in shell eggs (Lawrence et al., 2008). In many agricultural applications using hyperspectral imaging technique, it is required to identify a target pixel in an image scene covering the surface of the food items. Spectral matching algorithms are usually used to perform statistical comparisons between known reference spectra in an existing spectral library and unknown spectra extracted from hyperspectral images. Various spectral similarity measures have been developed for target detection and spectra classification. Frequent choices for such spectral metrics include spectral angle mapper (SAM), spectral correlation mapper (SCM), and Euclidean distance (ED). SAM, SCM, and ED calculate angle, correlation, and distance between two spectra (van der Meer, 2006), respectively, and use them as measures of discrimination. SAM has been widely used as a spectral similarity measure for target identification in using hyperspectral imagery, such as contaminant classification of broiler carcasses (Park et al., 2007) and yield estimation of grain sorghum fields (Yang et al., 2008). Chang (2000) proposed a new discrimination measure called spectral information divergence (SID) based on a stochastic approach called spectral information measure. The method models the spectrum of a pixel vector as a probability distribution, and many concepts in information theory can be readily applied to spectral characterization. Chang (2000) demonstrated that SID can characterize spectral variability more effectively than the common used SAM. van der Meer (2006) reported a study on comparing the performance of the various spectral matching algorithms for mineral identifications. The results showed that SID outperforms the classical spectral matching techniques including SAM, SCM, and ED. The objective of this research was to develop a hyperspectral detecting method to identify canker lesions on peel of citrus fruit
using a SID based classification algorithm. Specific objectives were to: Use a hyperspectral imaging system to measure the reflectance in the spectral region between 450 nm and 930 nm from grapefruits with cankerous, normal, and five other common diseased peel conditions (i.e., greasy spot, insect damage, melanose, scab, and wind scar); Develop SID based algorithms for hyperspectral image processing and classification to differentiate citrus canker lesions from normal and other diseased peel conditions. 2. Materials and methods 2.1. Hyperspectral imaging system A hyperspectral imaging system was developed for acquiring reflectance images from citrus samples, and its schematic diagram is shown in Fig. 1. It is a push broom, line-scan based imaging system that utilizes an electron-multiplying charge-coupled-device (EMCCD) imaging device (iXon, Andor Technology Inc., South Windsor, CT, USA). The EMCCD has 1004 1002 pixels and is thermoelectrically cooled to 80 °C via a double-stage Peltier device. An imaging spectrograph (ImSpector V10E, Spectral Imaging Ltd., Oulu, Finland) and a C-mount lens (Rainbow CCTV H6X8, International Space Optics, S.A., Irvine, CA, USA) are mounted to the EMCCD. The instantaneous field of view (IFOV) is limited to a thin line by the spectrograph aperture slit (30 lm), and the spectral resolution of the imaging spectrograph is 2.8 nm. Through the slit, light from the scanned IFOV line is dispersed by a prism-gratingprism device and projected onto the EMCCD. Therefore, for each line-scan, a two-dimensional (spatial and spectral) image is created with the spatial dimension along the horizontal axis and the spectral dimension along the vertical axis of the EMCCD. The reflectance light source consists of two 21 V, 150 W halogen lamps powered with a DC voltage regulated power supply (TechniQuip, Danville, CA, USA). The light is transmitted through optical fiber bundles toward line light distributors. Two line lights are arranged to illuminate the IFOV. A programmable, motorized positioning table (BiSlide-MN10, Velmex Inc., Bloomfield, NY, USA) moves citrus samples (five for each run) transversely through the line of the IFOV. One thousand seven hundred and forty line scans were performed for five fruit samples, and 400 pixels covering the scene of the fruit at each scan were saved, generating a 3-D hyperspectral image cube with the spatial dimension of 1740 400 for each band. The hyperspectral imaging system parameterization and datatransfer interface software were developed using a SDK (Software Development Kit) provided by the camera manufacturer on a Microsoft (MS) Visual Basic (Version 6.0) platform in the MS Windows operating system. Spectral calibration of the system was performed using an Hg–Ne spectral calibration lamp (Oriel Instruments, Stratford, CT, USA). Because of inefficiencies of the system at certain wavelength regions (e.g., low light output in the visible region less than 450 nm, and low quantum efficiency of the EMCCD in the NIR region beyond 930 nm), only the wavelength range between 450 nm and 930 nm (totaling 92 bands with a spectral resolution of 5.2 nm) was used in this investigation. 2.2. Citrus samples Grapefruit is one of the citrus varieties that are most susceptible to canker. Ruby Red grapefruits were used in this study. Fruit samples with normal surface, canker, and five common diseased peel conditions (i.e., greasy spot, insect damage, melanose, scab, and wind scar) were collected. Representative images for each peel
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
185
Fig. 1. Hyperspectral imaging system for acquiring reflectance images from citrus samples.
condition are shown in Fig. 2. The diseases on the fruit surface show different symptoms. Greasy spot, melanose, and scab are all caused by fungi, which generate surface blemishes that are formed by infection of immature fruit during the growing season. Greasy spot produces small necrotic specks, and the affected areas are colored in brown to black and exhibit greasy in appearance. Melanose is characterized by scattered raised pustules with dark brown to black in color. Scab appears as corky raised lesions usually with the color of light brown. Different from the fungal diseases, citrus canker is caused by bacteria, and it is featured with conspicuous dark lesions. Most circular in shape, canker lesions vary in size and they are superficial (up to 1 mm deep) on the fruit peel (Timmer et al., 2000). Diameter of the canker lesions tested in this study was approximately in the range of 2–9 mm. Insect damage is characterized by irregular grayish tracks on the fruit surface, which are generated by larvae of leafminers that burrow under the
epidermis of the fruit rind. Wind scar, which is caused by leaves, twigs, or thorns rubbing against the fruit, is a common physical injury on the fruit peel, and the scar tissue is generally gray. Fruit samples were hand picked from a grapefruit grove in Fort Pierce, Florida during the spring 2008 harvest season. Thirty samples for each peel condition were selected, hence a total of 210 grapefruits were tested in this study. All the grapefruits were washed and treated with chlorine and sodium o-phenylphenate (SOPP) at the Indian River Research and Education Center of University of Florida in Fort Pierce, Florida. The samples were then stored in an environmental control chamber maintained at 4 °C, and they were removed from cold storage about 2 h before imaging to allow them to reach room temperature. During image acquisition, the citrus samples were placed on the rubber cups that were fixed on the positioning table (Fig. 1) to make sure the diseased areas were on the top of each fruit.
Fig. 2. Typical normal and diseased peel conditions of grapefruit samples.
186
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
Ij ðXÞ ¼ log pj and Ij ðYÞ ¼ log qj
2.3. Hyperspectral data analysis 2.3.1. Image preprocessing To obtain the relative reflectance of the grapefruit samples, flatfield corrections were first performed on the original hyperspectral reflectance images using the following equation:
RðkÞ ¼
RsðkÞ RdðkÞ 100% RrðkÞ RdðkÞ
where R is the relative reflectance, Rs is the original sample image, Rr is the reference image obtained from a white diffuse reflectance panel (Spectralon SRT-99-100, Labsphere Inc., North Sutton, NH, USA), Rd is the dark current image acquired with the light source off and a cap covering the zoom lens, and k is the wavelength. A reflectance factor of 100% for the white panel was used in this study for simplification, although the actual reflectance value is around 98.5% in the wavelength range covered by the hyperspectral imaging system. The corrections using Eq. (1) were conducted for all the wavelengths in the region of 450 nm to 930 nm. The relative reflectance images were then multiplied by 10,000 so that the pixel value for the resultant images was in the range of 0 to 10,000, which was comparable to the range of the original data from the 14-bit EMCCD camera (0-16383), and could reduce rounding errors for further data analysis. After calculating the relative reflectance, a mask for the sample in each hyperspectral image was created to remove noisy background from fruit surface. To generate the mask template, a threshold value determined by visual inspection was used for one of the images in visible band (i.e., 650 nm) that has the best contrast between the fruit surface and the background. The mask was then applied to extract region of interest (ROI) for the other spectral images at each wavelength between 450 nm and 930 nm. Only the ROIs were subjected to further data analysis. To reduce the amount of data for computation, the hyperspectral image data were averaged by two neighboring pixels in both vertical and horizontal spatial dimensions at each wavelength, which reduced the size of the original 3-D image data from 1740 400 92 (92 bands) to 870 200 92, and resulted in a spatial resolution of 0.6 mm/pixel for both vertical and horizontal dimensions. The image preprocessing procedures described above were executed using programs developed in Matlab 7.0 (MathWorks, Natick, MA, USA). 2.3.2. Spectral information divergence (SID) The procedures for deriving spectral information divergence (SID) are briefly summarized in this section. More detailed information could be found in Chang (2000). For a given hyperspectral pixel X = (x1, x2, ..., xn, ..., xN)T, each component xn is a pixel representing its reflectance spectral signature in a single band image acquired at a certain wavelength kn. Owing to the inherent characteristic of the reflectance, all the components (i.e., xn’s) in X can be assumed nonnegative. From the theory of probability, X can be modeled as a random variable by defining a probability measure P as follows:
xj Pðfkj gÞ ¼ pj ¼ PN
n¼1 xn
The relative entropy of Y with respect to X, or information divergence, can be defined by:
DðXjjYÞ ¼
N X
pn Dn ðXjjYÞ ¼
n¼1
ð1Þ
ð2Þ
The vector P = (p1, p2, ..., pN)T is the probability vector for the pixel vector X. Thus any hyperspectral pixel X can be viewed as a single information source with its statistics governed by its probability vector P, and P can be used to describe the spectral variability of a pixel and its statistics of any order (mean, variance, third central moment, etc.). Assume that Y = (y1, y2, ..., yN)T is another pixel with probability vector Q = (q1, q2, ..., qN)T. Based on the information theory, the jth band self-information of X and Y can be defined as follows:
ð3Þ
¼
N X n¼1
pn log
pn qn
N X
pn ðIn ðYÞ In ðXÞÞ
n¼1
ð4Þ
The spectral information divergence can be defined as a spectral similarity measure between two pixel vectors X and Y as follows:
SIDðX; YÞ ¼ DðXjjYÞ þ DðYjjXÞ
ð5Þ
SID quantifies the spectral discrepancy by making use of the relative entropy to account for the spectral information provided by each pixel. The smaller the value of SID, the smaller the differences between two spectral pixels. 2.3.3. SID based classification algorithm In this study, reference spectrum of canker on the grapefruit was extracted from region of interest (ROI) hyperspectral image data using ENVI 4.3 (ITT Visual Information Solutions, Boulder, CO, USA). The ROI of canker was manually selected in a zoom window containing the infected peel areas using the circle drawing mode provided by ENVI ROI tools. A small circle was first drawn within a selected lesion, and the circle was then grown to merge the neighboring diseased pixels using a specified standard deviation away from the mean of the drawn region. The value of the standard deviation multiplier was determined by visual inspection of the increased ROI in such a way that all the pixels in the grown area were cankerous and the edge pixels were also avoided at the same time. This ‘grow’ ROI selection method is especially effective for selecting the diseased areas with irregular shapes. Ten different cankerous grapefruits were used for the ROI selection, and the mean reflectance spectrum of all the selected ROIs was used as the final reference spectrum of canker. After the reference spectrum of canker was obtained, rule images, within which each pixel represents the value of SID between the corresponding pixel in the hyperspectral images and the reference spectrum of canker, were generated through SID mapping using ENVI. The rule images have same spatial dimensions with the hyperspectral images, and they were used for the image classification. A simple thresholding method was then applied to the rule images in an attempt to segregate the canker lesions from the background of fruit surface and to differentiate canker from normal peels and other citrus diseases. Different thresholding values were used to evaluate the accuracies of canker identification. The performance of the SID based image classification method for canker detection was evaluated by all the grapefruit samples tested in this study. The 210 samples were divided into two classes: ‘Canker’ class including 30 samples with canker lesions, and ‘No Canker’ class including 30 normal samples and 150 samples with the other five diseased peel conditions. Due to the limited number of cankerous samples used in this study, the 10 samples used for establishing the reference spectrum of canker were also used for the testing. 3. Results and discussion 3.1. Hyperspectral reflectance images Representative single-band reflectance images of grapefruits with cankerous, normal and five other common diseased peel conditions at eight selected wavelengths from 450 nm to 930 nm are
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
shown in Fig. 3 to demonstrate general pattern of the hyperspectral images and differences among different spectral regions. The images at low and high wavelengths (i.e., less than 500 nm and beyond 900 nm) appeared fuzzy due to the relatively low efficiencies for both grating diffraction of the imaging spectrograph and the EMCCD camera within these wavelength ranges. The images at other bands from 500 nm to 900 nm showed relatively clear patterns in terms of both whole fruit surfaces and the diseased areas. Specular reflectance was observed around the center areas of some grapefruits (e.g., normal fruit at 600 nm as shown in Fig. 3), especially for the samples with normal peel conditions. The canker lesions on the peel of the grapefruit were observed at all wavelengths covered by the hyperspectral imaging system, and they were more distinctive from the fruit surface in the spectral region between 600 nm and 700 nm. The contrast between canker and the normal peel decreased below 500 nm and beyond 800 nm. Compared to the disease of canker, greasy spot, insect damage, melanose, scab, and wind scar exhibited different reflectance as well as textural features. For scab and wind scar, the diseased areas displayed as gray for all the wavelengths, whereas greasy spot, insect damage, and melanose showed black in certain wavelength ranges (e.g., greasy spot at 675 nm, insect damage and
187
melanose at 600 nm as shown in Fig. 3). This is similar to the canker lesions in terms of gray values. The contrasts between the normal peels and all the diseased regions decreased towards the longer wavelengths. Greasy spot, scab, and wind scar tended to disappear as the wavelength increased beyond 800 nm, while canker, insect damage, and melanose could still be observed at the long wavelengths. These observations suggest that the citrus canker detection may not be achieved using a single wavelength. A hyperspectral or multispectral imaging solution is necessary to discriminate canker from normal and other diseased peel conditions. Meanwhile, stem-ends and calyxes of the fruit samples, which is a persistent problem for automatic quality and safety inspection of fruits and vegetables using machine vision based technologies, appeared in the whole wavelength range. The reflectance of the stem-ends and calyxes changed with wavelengths. However, due to their different reflectance properties from canker, the stem-ends and calyxes demonstrated different contrasts with the canker lesions at different wavelengths. For example, the stem-end at 600 nm in Fig. 3 appeared brighter than the canker lesions due to its relatively high reflectance, while the brightness difference between the stem-end and the canker lesions was reduced at 800 nm owing to their similar reflectance properties at this
Fig. 3. Hyperspectral images of grapefruit samples with normal and diseased peel conditions.
188
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
wavelength. The contrast differences that existed among different wavelengths provided the possibility for separating the stem-ends and calyxes from the true diseased regions. 3.2. Reflectance spectra Fig. 4a shows mean reflectance spectrum and standard deviation obtained from the selected ROIs of the canker lesions. Also, typical reflectance spectra of the grapefruit samples with normal and diseased conditions over the wavelength range between 450 nm and 930 nm were shown in Fig. 4b to demonstrate the general spectral patterns. Each spectrum plot in Fig. 4b was extracted from the hyperspectral image using an average spatial window covering 5 5 pixels for each peel condition. A cankerous sample was also included for the purpose of comparisons. All the reflectance spectra were featured as two local minima around 500 nm and 675 nm due to the light absorption of carotenoid and chlorophyll a in the fruit peel, respectively. Chlorophyll is responsible for the green color of the citrus peel, while the yellow color of the fruit peel is related to carotenoid. The carotenoid and chlorophyll absorptions for greasy spot and canker were not as prominent as other peel conditions. Values of the reflectance for the diseased peel conditions were consistently lower than those from the normal fruit surface over the entire spectral region. Canker had the lowest reflectance except for the spectral region from
450 nm to 550 nm, in which greasy spot showed similar reflectance with canker. In the spectral region from 450 nm to 930 nm, the relative reflectance values of canker (mean spectrum in Fig. 4a) and normal peel were in the range of 12%–43% and 31%– 66%, respectively, while other peel conditions generally had values in between. For canker disease, once the bacterial pathogen invades the fruit peel, the region infected continually loses its moisture content through the season, and the lesions on the fruit surface usually become dark in appearance. This may be the reason for the low reflectance characteristics of the canker disease in visible and short-wavelength near-infrared region. The spectral differences between canker and normal as well as other diseased peel conditions provide a basis for detecting citrus canker using hyperspectral or multispectral imaging approach. 3.3. SID mapping of hyperspectral images Spectral information divergence mapping was applied for all the hyperspectral images of the grapefruit samples using the mean reflectance spectrum of canker in Fig. 4a as the reference. Values of SID between the canker reference spectrum and the spectra of seven peel conditions demonstrated in Fig. 4b were illustrated in Fig. 5a. The SID values with respect to canker reference for canker, normal, greasy spot, insect damage, melanose, scab, and wind scar
Fig. 4. (a) Mean spectrum and standard deviation from ROIs of canker, and (b) typical reflectance spectra of grapefruit samples with normal and diseased peel conditions.
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
189
Fig. 5. (a) Values of SID between reference spectrum of canker and spectra of seven tested peel conditions, and (b) rule images from SID mapping for the hyperspectral images of grapefruit samples using the reference spectrum of canker.
were 0.004, 0.022, 0.010, 0.013, 0.012, 0.016, and 0.017, respectively. As expected, the SID between canker and the canker reference had the lowest value, and the normal peel and the canker reference had the highest SID value. Values for other peel conditions were in between. It should be noticed that the SID algorithm is relatively insensitive to illumination effects and brightness shifts, which is similar to the SAM algorithm (Chang, 2000). It is the relative but not the absolute spectral discrepancies that determine the similarities between different spectra. For example, the absolute reflectance difference between scab and the canker reference was larger than that between wind scar and the canker reference (see Fig. 4b). However, the scab had the smaller SID value (0.016) than wind scar (0.017), indicating that scab was more similar to canker than wind scar in this case. Representative rule images from SID mapping for the hyperspectral images demonstrated in Fig. 3 are shown in Fig. 5b, in which each pixel represents the value of SID between the corre-
sponding spectrum extracted from that pixel and the reference spectrum of canker. The canker lesions showed distinctive dark due to their small SID values, while the stem-ends and the other diseases including greasy spot, insect damage, melanose, scab, and wind scar displayed as gray in different degrees. The normal areas on the fruit peels appeared lighter than the diseased areas because of their larger SID values, indicating their large spectral differences from canker. Similar patterns were observed for the rule images of other grapefruit samples. 3.4. Image classification Fig. 6 demonstrates primary procedures for hyperspectral image processing and classification for citrus canker identification using a grapefruit sample with both canker and wind scar on the fruit peel. ROIs of canker were first selected from the reference hyperspectral images of the selected cankerous samples. Mean
Fig. 6. Procedures of image processing and classification for citrus canker detection.
190
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
Fig. 7. Binary images after thresholding applied to the rule images of grapefruit samples.
reflectance spectrum, which was calculated from the spectra extracted from all the canker ROIs, was used as the reference spectrum of canker. SID mapping was then performed on the hyperspectral images of the test samples using the canker reference spectrum to obtain the rule images. Finally, a simple thresholding method was applied to the rule images to separate canker lesions from the fruit peel. The SID threshold value for segregating the canker lesions from the rule images was determined with an aim that all the image pixels in the unaffected regions including stem-ends and calyxes as well as the diseases other than canker on the fruit peel were converted to zero in the binary images (i.e., displayed as black if the pixel values were larger than the threshold). The binary images with remaining white pixels were classified as cankerous samples. As shown in Fig. 6, the fruit surface and the diseased areas of wind scar disappeared in the resultant binary image, and the white spots represent the canker lesions isolated from the fruit peel. The binary images after thresholding applied to the rule images of the grapefruit samples in Fig. 5b are shown in Fig. 7. The normal fruit surfaces and stem-ends as well as the diseases other than canker were converted to black. Only canker lesions were remained as white spots in the binary images. The canker lesions at both center and edge areas of the fruit were identified and the surrounding areas of the canker (e.g., the canker lesion at the center area) were excluded, showing the effectiveness of the SID based classification method for canker detection. Five SID threshold values from 0.005 to 0.009 with a step size of 0.001 were used to test the classification results. The value of 0.008 provided the best overall classification accuracy (balancing the errors of false negative and false positive), therefore it was chosen as the threshold for the classifier (illustrated as a horizontal line in Fig. 5a). The algorithms for hyperspectral image processing and classification described above were evaluated using all the grapefruit samples tested in this study. Table 1 summarizes the classification results for differentiating citrus canker from normal and other diseased peel conditions. As shown in the table, 96.7% of the cankerous grapefruits were correctly classified as ‘Canker’ class, and accuracy for classifying the 180 samples in ‘No Canker’ class was 96.1%. A total of eight samples was misclassified, and the overall classification accuracy for all the samples was 96.2%.
Table 1 Classification results for differentiating citrus canker from other peel conditions. Class
Peel condition
Canker (n = 30) No Canker (n = 180)
Canker Normal Greasy spot Insect damage Melanose Scab Wind scar Total
Number
Misclassified
Accuracy %
30 30 30 30 30 30 30
1 0 3 2 2 0 0
96.7 96.1
210
8
96.2
For the misclassified samples, one sample was from ‘Canker’ class, and the other seven were from ‘No Canker’ class, including three ‘Greasy Spot’ samples, two ‘Insect Damage’ samples, and two ‘Melanose’ samples. All ‘Normal’, ‘Scab’, and ‘Wind Scar’ samples were correctly classified. The reflectance properties for greasy spot, insect damage, and melanose were similar to those of canker (see Fig. 4b), thus the values of SID between these three diseases and canker reference spectrum were relatively small, as demonstrated in Fig. 5a. For the severe infected conditions, the SID values could be lower than the threshold value that was applied to the rule images. That may be the reason for the low classification accuracies for these three cases. It was also found that the misclassified sample in ‘Canker’ class had a single, light canker lesion that appeared in the early development stage, indicating that the classification method developed based on the SID mapping using the reference spectrum could be affected by different degrees of canker, which could cause changes for reflectance characteristics at different development stages over the harvest season. It should be noted that the best overall classification accuracy (96.2%) using the SID threshold value of 0.008 was obtained based on the assumption that the two types of errors (i.e., false negative and false positive) were weighted equally. However, from the economic standpoint, the error of missing a cankerous fruit (false negative) can significantly outweigh that of incorrect classification of a fruit without canker (false positive). Increasing the threshold generally can reduce the false negative error. For the five threshold values tested in this study, the value of 0.009 gave no error for the cankerous fruit classification, although the overall classification accuracy under this threshold was decreased to 95.2% due to the increased false positive error. Hence for the SID based classification method, we can adjust the two types of errors by decreasing or increasing the threshold value to meet the application needs. 4. Conclusion Hyperspectral reflectance imaging coupled with a spectral information divergence based image classification method provides a useful means for detecting canker lesions on citrus fruit. A hyperspectral imaging system was developed to measure the reflectance images from grapefruit samples with cankerous, normal and other common diseased peel conditions including greasy spot, insect damage, melanose, scab, and wind scar in the wavelength range between 450 nm and 930 nm. Reference spectrum of canker was obtained from the ROIs that were selected from the reference hyperspectral images of the cankerous samples. SID mapping was performed on the images of all the grapefruit samples to quantify the spectral similarities between the canker reference spectrum and the hyperspectral pixels. The rule images from SID mapping provided higher contrasts between canker lesions and normal peels as well as other diseases than the original reflectance images, indicating that SID mapping could extract useful features from hyperspectral images using particular reference spectrum. The simple thresholding method that was used for converting
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
the rule images to the binary images was able to segregate the canker lesions from the background of fruit surface. The SID based classifier could differentiate canker from normal fruit peels and other citrus diseases, and it also could avoid the negative effects of stem-ends and calyxes. Optimal SID threshold value was determined based on the assumption that the errors of false negative and false positive were equally weighted, by which the overall classification accuracy was achieved 96.2%. Since the economic impact of missing a cankerous fruit is much higher than that of misclassifying a fruit without canker, we can minimize the false negative error by properly selecting the threshold value with a cost of reducing the overall classification accuracy. Greasy spot, insect damage, and melanose had similar reflectance properties to canker, and the chances for misclassifying these three diseases were higher than other peel conditions. Canker lesions at different development stages could cause changes for reflectance characteristics, thus might affect the SID based classification method that utilizes the reference spectrum. Further work will be performed to investigate the changing patterns of the canker reflectance properties over the harvest season, and incorporate canker spectra at different development stages into the classification model for achieving better detection accuracies. Meanwhile, the full spectral information used by the SID mapping would limit its applications such as on-line disease inspection due to the large amount of hyperspectral data. Methods for optimal band selection and multispectral image classification will be developed in the future work to fulfill the goal of real-time citrus canker detection. Acknowledgements The authors gratefully acknowledge the financial support of Florida Fresh Packer Association and USDA Technical Assistance for Specialty Crops. The authors would also like to thank Dr. Moon Kim and Dr. Kuanglin Chao of USDA-ARS Environmental Microbial and Food Safety Laboratory, and Mr. Mike Zingaro and Mr. Greg Pugh of University of Florida, for their help in designing and building the hyperspectral imaging system. References Aleixos, N., Blasco, J., Navarrón, F., Moltó, E., 2002. Multispectral inspection of citrus in real-time using machine vision and digital signal processors. Computers and Electronics in Agriculture 33 (2), 121–137. Ariana, D.P., Lu, R., Guyer, D.E., 2006. Near-infrared hyperspectral reflectance imaging for detection of bruises on pickling cucumbers. Computers and Electronics in Agriculture 53 (1), 60–70.
191
Blasco, J., Aleixos, N., Gómez-Sanchis, J., Moltó, E., 2007. Citrus sorting by identification of the most common defects using multispectral computer vision. Journal of Food Engineering 83 (3), 384–393. Chang, C.-I., 2000. An information theoretic-based approach to spectral variability, similarity and discriminability for hyperspectral image analysis. IEEE Transactions on Information Theory 46 (5), 1927–1932. Chao, K., Yang, C.C., Chen, Y.R., Kim, M.S., Chan, D.E., 2007. Fast line-scan imaging system for broiler carcass inspection. Sensing and Instrumentation for Food Quality and Safety 1 (2), 62–71. Edwards, J.G., Sweet, C.H., 1986. Citrus blight assessment using a microcomputer, quantifying damage using an apple computer to solve reflectance spectra of entire trees. Florida Scientist 49 (1), 48–53. Gaffney, J.J., 1973. Reflectance properties of citrus fruits. Transactions of the ASAE 16 (2), 310–314. Gómez-Sanchis, J., Gómez-Chova, L., Aleixos, N., Camps-Valls, G., MontesinosHerrero, C., Moltó, E., Blasco, J., 2008. Hyperspectral system for early detection of rottenness caused by Penicillium digitatum in mandarins. Journal of Food Engineering 89 (1), 80–86. Kim, M.S., Chen, Y.R., Mehl, P.M., 2001. Hyperspectral reflectance and fluorescence imaging system for food quality and safety. Transactions of the ASAE 44 (3), 721–729. Kim, M.S., Lefcourt, A.M., Chao, K., Chen, Y.R., Kim, I., Chan, D.E., 2002. Multispectral detection of fecal contamination on apples based on hyperspectral imagery. Part I. Application of visible and near-infrared reflectance imaging. Transactions of the ASAE 45 (6), 2027–2037. Lawrence, K.C., Yoon, S.C., Heitschmidt, G.W., Jones, D.R., Park, B., 2008. Imaging system with modified-pressure chamber for crack detection in shell eggs. Sensing and Instrumentation for Food Quality and Safety 2 (3), 116–122. Lu, R., 2003. Detection of bruises on apples using near-infrared hyperspectral imaging. Transactions of the ASAE 46 (2), 523–530. Lu, R., Chen, Y.R. 1998. Hyperspectral imaging for safety inspection of food and agricultural products. In: Pathogen Detection and Remediation for Safe Eating, SPIE Proceedings, vol. 3544, pp. 121–133. Miller, W.M., Drouillard, G.P., 2001. Multiple feature analysis for machine vision grading of Florida citrus. Applied Engineering in Agriculture 17 (5), 627–633. Park, B., Windham, W.R., Lawrence, K.C., Smith, D.P., 2007. Contaminant classification of poultry hyperspectral imagery using a spectral angle mapper algorithm. Biosystems Engineering 96 (3), 323–333. Pydipati, R., Burks, T.F., Lee, W.S., 2006. Identification of citrus disease using color texture features and discriminant analysis. Computers and Electronics in Agriculture 52 (1–2), 49–59. Qin, J., Lu, R., 2005. Detection of pits in tart cherries by hyperspectral transmission imaging. Transactions of the ASAE 48 (5), 1963–1970. Schubert, T.S., Rizvi, S.A., Sun, X.A., Gottwald, T.R., Graham, J.H., Dixon, W.N., 2001. Meeting the challenge of eradicating citrus canker in Florida – Again. Plant Disease 85 (4), 340–356. Timmer, L.W., Garnsey, S.M., Graham, J.H., 2000. Compendium of Citrus Diseases, Second ed. The American Phytopathological Society, St. Paul, MN, USA. van der Meer, F., 2006. The effectiveness of spectral similarity measures for the analysis of hyperspectral imagery. International Journal of Applied Earth Observation and Geoinformation 8 (1), 3–17. Yang, C., Everitt, J.H., Bradford, J.M., 2008. Yield estimation from hyperspectral imagery using spectral angle mapper (SAM). Transactions of the ASABE 51 (2), 729–737.
Contents lists available at ScienceDirect
Journal of Food Engineering journal homepage: www.elsevier.com/locate/jfoodeng
Detection of citrus canker using hyperspectral reflectance imaging with spectral information divergence Jianwei Qin a,*, Thomas F. Burks a, Mark A. Ritenour b, W. Gordon Bonn c a
Department of Agricultural and Biological Engineering, P.O. Box 110570, University of Florida, Gainesville, FL 32611-0570, USA Department of Horticultural Sciences, 2199 South Rock Road, University of Florida, Fort Pierce, FL 34945-3138, USA c Division of Plant Industry, Florida Department of Agriculture and Consumer Services, Gainesville, FL 32614-7100, USA b
a r t i c l e
i n f o
Article history: Received 18 July 2008 Received in revised form 16 January 2009 Accepted 17 January 2009 Available online 30 January 2009 Keywords: Hyperspectral imaging Spectral similarity Citrus canker Disease detection Food safety
a b s t r a c t Citrus canker is one of the most devastating diseases that threaten marketability of citrus crops. This research was aimed to develop a hyperspectral imaging approach for detecting canker lesions on citrus fruit. A hyperspectral imaging system was developed for acquiring reflectance images from citrus samples in the spectral region from 450 to 930 nm. Ruby Red grapefruits with cankerous, normal and other common peel diseases including greasy spot, insect damage, melanose, scab, and wind scar were tested. Spectral information divergence (SID) classification method, which was based on quantifying the spectral similarities by using a predetermined canker reference spectrum, was performed on the hyperspectral images of the grapefruits for differentiating canker from normal fruit peels and other citrus surface conditions. The overall classification accuracy was 96.2% using an optimized SID threshold value of 0.008, which was determined under the condition that the errors of false negative and false positive were weighted equally. Considering the high economic impact of missing a cankerous fruit, zero false negative error was achieved by using a threshold value of 0.009, under which the classification accuracy was 95.2%. This research demonstrated that hyperspectral imaging technique coupled with the SID based image classification method could be used for discriminating citrus canker from other confounding diseases. Ó 2009 Elsevier Ltd. All rights reserved.
1. Introduction Citrus canker is a severe disease that can affect the peel of most commercial citrus varieties. The pathogen, usually characterized by conspicuous, erumpent lesions on leaves, stems, and fruit, could cause defoliation, blemished fruit, premature fruit drop, twig dieback, and tree decline (Schubert et al., 2001). Although there are various disorders that could infect citrus fruit, canker is considered as one of the most devastating diseases that threaten citrus crops. Since canker does not exist in all citrus-growing regions of the world, considerable effort has been made to prevent the spread of the disease to the unaffected areas where the climate is conducive for canker development. The Citrus Canker Eradication Program (CCEP), initialized by the Florida Department of Agriculture and Consumer Services – Division of Plant Industry (FDACS-DPI) and the United States Department of Agriculture – Animal and Plant Health Inspection Service (USDA-APHIS) in mid 1990s, was an attempt to alleviate the consequences that canker would have on the Florida citrus industry, as well as to keep other US citrusproducing areas (e.g., Texas and California) from being infected * Corresponding author. Tel.: +1 352 392 1864x233. E-mail address: qinj@ufl.edu (J. Qin). 0260-8774/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.jfoodeng.2009.01.014
and harmed by this disease. Unfortunately, there is no treatment or prevention at this time that could completely eradicate citrus canker in infected areas. The current emphasis is on minimizing the level of the disease and avoiding further infections. The presence of cankerous fruit in a shipment could result in the whole shipment being deemed unmarketable to some ‘canker-free’ areas, such as the European countries. Therefore it is crucial to detect citrus canker and eliminate the infected fruit in the packing houses. Manual removal of cankerous fruit is time-consuming and laborintensive. Technologies that can efficiently identify citrus canker would assure fruit quality and safety, which would enhance the competitiveness and profitability of the citrus industry. Optical sensing technologies (e.g., machine vision and spectroscopy) offer great potential for quality evaluation and safety inspection for food and agricultural products. A number of techniques have been studied to detect defects and diseases related to citrus. Gaffney (1973) obtained reflectance spectra of citrus fruit and some surface defects. Edwards and Sweet (1986) developed a method to assess damages due to citrus blight disease on citrus plants using reflectance spectra of the entire tree. Miller and Drouillard (2001) collected data from Florida grapefruit, orange, and tangerine varieties using a color vision system. They used various neural network classification strategies to detect blemish-related features for the
184
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
citrus fruit. Aleixos et al. (2002) developed a multispectral camera system that could acquire visible and near infrared images from the same scene, and used it on a real-time system for detecting defects on citrus surface. Pydipati et al. (2006) utilized the color cooccurrence method (CCM) to extract various textural features from the color RGB images of citrus leaves. The CCM texture statistics were used to identify diseased and normal citrus leaves using discriminant analysis. Most recently, Blasco et al. (2007) reported the application of near-infrared, ultraviolet, and fluorescence computer vision systems to identify common defects of citrus fruit. They proposed a fruit sorting algorithm that combines the different spectral information to classify fruit according to the type of the defect. These research works provided good references and resources for dealing with various problems associated with detection of citrus surface defects and diseases. However, to our knowledge no research has been conducted for detecting canker on citrus fruit. There is a need for studying the unique features of citrus canker and obtaining the fundamental information that will be useful for designing the canker detection system. In the recent years, hyperspectral imaging technique has been developed as a useful tool for inspecting internal and external attributes of food and agricultural products (Lu and Chen, 1998; Kim et al., 2001). The technique combines conventional spectroscopy and imaging techniques to acquire both spectral and spatial information from an object. Compared to traditional visible/nearinfrared spectroscopy that essentially relies on spot measurement, hyperspectral imaging has more advantages for evaluating whole surface of the individual food items. The technique has been applied for analyzing physical and/or chemical properties of a broad range of food and agricultural products. Examples include detecting contaminants on apples (Kim et al., 2002) and poultry carcasses (Park et al., 2007), diseases on poultry carcasses (Chao et al., 2007), bruises on apples (Lu, 2003) and cucumbers (Ariana et al., 2006), rottenness on citrus (Gómez-Sanchis et al., 2008), pits in tart cherries (Qin and Lu, 2005), and cracks in shell eggs (Lawrence et al., 2008). In many agricultural applications using hyperspectral imaging technique, it is required to identify a target pixel in an image scene covering the surface of the food items. Spectral matching algorithms are usually used to perform statistical comparisons between known reference spectra in an existing spectral library and unknown spectra extracted from hyperspectral images. Various spectral similarity measures have been developed for target detection and spectra classification. Frequent choices for such spectral metrics include spectral angle mapper (SAM), spectral correlation mapper (SCM), and Euclidean distance (ED). SAM, SCM, and ED calculate angle, correlation, and distance between two spectra (van der Meer, 2006), respectively, and use them as measures of discrimination. SAM has been widely used as a spectral similarity measure for target identification in using hyperspectral imagery, such as contaminant classification of broiler carcasses (Park et al., 2007) and yield estimation of grain sorghum fields (Yang et al., 2008). Chang (2000) proposed a new discrimination measure called spectral information divergence (SID) based on a stochastic approach called spectral information measure. The method models the spectrum of a pixel vector as a probability distribution, and many concepts in information theory can be readily applied to spectral characterization. Chang (2000) demonstrated that SID can characterize spectral variability more effectively than the common used SAM. van der Meer (2006) reported a study on comparing the performance of the various spectral matching algorithms for mineral identifications. The results showed that SID outperforms the classical spectral matching techniques including SAM, SCM, and ED. The objective of this research was to develop a hyperspectral detecting method to identify canker lesions on peel of citrus fruit
using a SID based classification algorithm. Specific objectives were to: Use a hyperspectral imaging system to measure the reflectance in the spectral region between 450 nm and 930 nm from grapefruits with cankerous, normal, and five other common diseased peel conditions (i.e., greasy spot, insect damage, melanose, scab, and wind scar); Develop SID based algorithms for hyperspectral image processing and classification to differentiate citrus canker lesions from normal and other diseased peel conditions. 2. Materials and methods 2.1. Hyperspectral imaging system A hyperspectral imaging system was developed for acquiring reflectance images from citrus samples, and its schematic diagram is shown in Fig. 1. It is a push broom, line-scan based imaging system that utilizes an electron-multiplying charge-coupled-device (EMCCD) imaging device (iXon, Andor Technology Inc., South Windsor, CT, USA). The EMCCD has 1004 1002 pixels and is thermoelectrically cooled to 80 °C via a double-stage Peltier device. An imaging spectrograph (ImSpector V10E, Spectral Imaging Ltd., Oulu, Finland) and a C-mount lens (Rainbow CCTV H6X8, International Space Optics, S.A., Irvine, CA, USA) are mounted to the EMCCD. The instantaneous field of view (IFOV) is limited to a thin line by the spectrograph aperture slit (30 lm), and the spectral resolution of the imaging spectrograph is 2.8 nm. Through the slit, light from the scanned IFOV line is dispersed by a prism-gratingprism device and projected onto the EMCCD. Therefore, for each line-scan, a two-dimensional (spatial and spectral) image is created with the spatial dimension along the horizontal axis and the spectral dimension along the vertical axis of the EMCCD. The reflectance light source consists of two 21 V, 150 W halogen lamps powered with a DC voltage regulated power supply (TechniQuip, Danville, CA, USA). The light is transmitted through optical fiber bundles toward line light distributors. Two line lights are arranged to illuminate the IFOV. A programmable, motorized positioning table (BiSlide-MN10, Velmex Inc., Bloomfield, NY, USA) moves citrus samples (five for each run) transversely through the line of the IFOV. One thousand seven hundred and forty line scans were performed for five fruit samples, and 400 pixels covering the scene of the fruit at each scan were saved, generating a 3-D hyperspectral image cube with the spatial dimension of 1740 400 for each band. The hyperspectral imaging system parameterization and datatransfer interface software were developed using a SDK (Software Development Kit) provided by the camera manufacturer on a Microsoft (MS) Visual Basic (Version 6.0) platform in the MS Windows operating system. Spectral calibration of the system was performed using an Hg–Ne spectral calibration lamp (Oriel Instruments, Stratford, CT, USA). Because of inefficiencies of the system at certain wavelength regions (e.g., low light output in the visible region less than 450 nm, and low quantum efficiency of the EMCCD in the NIR region beyond 930 nm), only the wavelength range between 450 nm and 930 nm (totaling 92 bands with a spectral resolution of 5.2 nm) was used in this investigation. 2.2. Citrus samples Grapefruit is one of the citrus varieties that are most susceptible to canker. Ruby Red grapefruits were used in this study. Fruit samples with normal surface, canker, and five common diseased peel conditions (i.e., greasy spot, insect damage, melanose, scab, and wind scar) were collected. Representative images for each peel
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
185
Fig. 1. Hyperspectral imaging system for acquiring reflectance images from citrus samples.
condition are shown in Fig. 2. The diseases on the fruit surface show different symptoms. Greasy spot, melanose, and scab are all caused by fungi, which generate surface blemishes that are formed by infection of immature fruit during the growing season. Greasy spot produces small necrotic specks, and the affected areas are colored in brown to black and exhibit greasy in appearance. Melanose is characterized by scattered raised pustules with dark brown to black in color. Scab appears as corky raised lesions usually with the color of light brown. Different from the fungal diseases, citrus canker is caused by bacteria, and it is featured with conspicuous dark lesions. Most circular in shape, canker lesions vary in size and they are superficial (up to 1 mm deep) on the fruit peel (Timmer et al., 2000). Diameter of the canker lesions tested in this study was approximately in the range of 2–9 mm. Insect damage is characterized by irregular grayish tracks on the fruit surface, which are generated by larvae of leafminers that burrow under the
epidermis of the fruit rind. Wind scar, which is caused by leaves, twigs, or thorns rubbing against the fruit, is a common physical injury on the fruit peel, and the scar tissue is generally gray. Fruit samples were hand picked from a grapefruit grove in Fort Pierce, Florida during the spring 2008 harvest season. Thirty samples for each peel condition were selected, hence a total of 210 grapefruits were tested in this study. All the grapefruits were washed and treated with chlorine and sodium o-phenylphenate (SOPP) at the Indian River Research and Education Center of University of Florida in Fort Pierce, Florida. The samples were then stored in an environmental control chamber maintained at 4 °C, and they were removed from cold storage about 2 h before imaging to allow them to reach room temperature. During image acquisition, the citrus samples were placed on the rubber cups that were fixed on the positioning table (Fig. 1) to make sure the diseased areas were on the top of each fruit.
Fig. 2. Typical normal and diseased peel conditions of grapefruit samples.
186
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
Ij ðXÞ ¼ log pj and Ij ðYÞ ¼ log qj
2.3. Hyperspectral data analysis 2.3.1. Image preprocessing To obtain the relative reflectance of the grapefruit samples, flatfield corrections were first performed on the original hyperspectral reflectance images using the following equation:
RðkÞ ¼
RsðkÞ RdðkÞ 100% RrðkÞ RdðkÞ
where R is the relative reflectance, Rs is the original sample image, Rr is the reference image obtained from a white diffuse reflectance panel (Spectralon SRT-99-100, Labsphere Inc., North Sutton, NH, USA), Rd is the dark current image acquired with the light source off and a cap covering the zoom lens, and k is the wavelength. A reflectance factor of 100% for the white panel was used in this study for simplification, although the actual reflectance value is around 98.5% in the wavelength range covered by the hyperspectral imaging system. The corrections using Eq. (1) were conducted for all the wavelengths in the region of 450 nm to 930 nm. The relative reflectance images were then multiplied by 10,000 so that the pixel value for the resultant images was in the range of 0 to 10,000, which was comparable to the range of the original data from the 14-bit EMCCD camera (0-16383), and could reduce rounding errors for further data analysis. After calculating the relative reflectance, a mask for the sample in each hyperspectral image was created to remove noisy background from fruit surface. To generate the mask template, a threshold value determined by visual inspection was used for one of the images in visible band (i.e., 650 nm) that has the best contrast between the fruit surface and the background. The mask was then applied to extract region of interest (ROI) for the other spectral images at each wavelength between 450 nm and 930 nm. Only the ROIs were subjected to further data analysis. To reduce the amount of data for computation, the hyperspectral image data were averaged by two neighboring pixels in both vertical and horizontal spatial dimensions at each wavelength, which reduced the size of the original 3-D image data from 1740 400 92 (92 bands) to 870 200 92, and resulted in a spatial resolution of 0.6 mm/pixel for both vertical and horizontal dimensions. The image preprocessing procedures described above were executed using programs developed in Matlab 7.0 (MathWorks, Natick, MA, USA). 2.3.2. Spectral information divergence (SID) The procedures for deriving spectral information divergence (SID) are briefly summarized in this section. More detailed information could be found in Chang (2000). For a given hyperspectral pixel X = (x1, x2, ..., xn, ..., xN)T, each component xn is a pixel representing its reflectance spectral signature in a single band image acquired at a certain wavelength kn. Owing to the inherent characteristic of the reflectance, all the components (i.e., xn’s) in X can be assumed nonnegative. From the theory of probability, X can be modeled as a random variable by defining a probability measure P as follows:
xj Pðfkj gÞ ¼ pj ¼ PN
n¼1 xn
The relative entropy of Y with respect to X, or information divergence, can be defined by:
DðXjjYÞ ¼
N X
pn Dn ðXjjYÞ ¼
n¼1
ð1Þ
ð2Þ
The vector P = (p1, p2, ..., pN)T is the probability vector for the pixel vector X. Thus any hyperspectral pixel X can be viewed as a single information source with its statistics governed by its probability vector P, and P can be used to describe the spectral variability of a pixel and its statistics of any order (mean, variance, third central moment, etc.). Assume that Y = (y1, y2, ..., yN)T is another pixel with probability vector Q = (q1, q2, ..., qN)T. Based on the information theory, the jth band self-information of X and Y can be defined as follows:
ð3Þ
¼
N X n¼1
pn log
pn qn
N X
pn ðIn ðYÞ In ðXÞÞ
n¼1
ð4Þ
The spectral information divergence can be defined as a spectral similarity measure between two pixel vectors X and Y as follows:
SIDðX; YÞ ¼ DðXjjYÞ þ DðYjjXÞ
ð5Þ
SID quantifies the spectral discrepancy by making use of the relative entropy to account for the spectral information provided by each pixel. The smaller the value of SID, the smaller the differences between two spectral pixels. 2.3.3. SID based classification algorithm In this study, reference spectrum of canker on the grapefruit was extracted from region of interest (ROI) hyperspectral image data using ENVI 4.3 (ITT Visual Information Solutions, Boulder, CO, USA). The ROI of canker was manually selected in a zoom window containing the infected peel areas using the circle drawing mode provided by ENVI ROI tools. A small circle was first drawn within a selected lesion, and the circle was then grown to merge the neighboring diseased pixels using a specified standard deviation away from the mean of the drawn region. The value of the standard deviation multiplier was determined by visual inspection of the increased ROI in such a way that all the pixels in the grown area were cankerous and the edge pixels were also avoided at the same time. This ‘grow’ ROI selection method is especially effective for selecting the diseased areas with irregular shapes. Ten different cankerous grapefruits were used for the ROI selection, and the mean reflectance spectrum of all the selected ROIs was used as the final reference spectrum of canker. After the reference spectrum of canker was obtained, rule images, within which each pixel represents the value of SID between the corresponding pixel in the hyperspectral images and the reference spectrum of canker, were generated through SID mapping using ENVI. The rule images have same spatial dimensions with the hyperspectral images, and they were used for the image classification. A simple thresholding method was then applied to the rule images in an attempt to segregate the canker lesions from the background of fruit surface and to differentiate canker from normal peels and other citrus diseases. Different thresholding values were used to evaluate the accuracies of canker identification. The performance of the SID based image classification method for canker detection was evaluated by all the grapefruit samples tested in this study. The 210 samples were divided into two classes: ‘Canker’ class including 30 samples with canker lesions, and ‘No Canker’ class including 30 normal samples and 150 samples with the other five diseased peel conditions. Due to the limited number of cankerous samples used in this study, the 10 samples used for establishing the reference spectrum of canker were also used for the testing. 3. Results and discussion 3.1. Hyperspectral reflectance images Representative single-band reflectance images of grapefruits with cankerous, normal and five other common diseased peel conditions at eight selected wavelengths from 450 nm to 930 nm are
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
shown in Fig. 3 to demonstrate general pattern of the hyperspectral images and differences among different spectral regions. The images at low and high wavelengths (i.e., less than 500 nm and beyond 900 nm) appeared fuzzy due to the relatively low efficiencies for both grating diffraction of the imaging spectrograph and the EMCCD camera within these wavelength ranges. The images at other bands from 500 nm to 900 nm showed relatively clear patterns in terms of both whole fruit surfaces and the diseased areas. Specular reflectance was observed around the center areas of some grapefruits (e.g., normal fruit at 600 nm as shown in Fig. 3), especially for the samples with normal peel conditions. The canker lesions on the peel of the grapefruit were observed at all wavelengths covered by the hyperspectral imaging system, and they were more distinctive from the fruit surface in the spectral region between 600 nm and 700 nm. The contrast between canker and the normal peel decreased below 500 nm and beyond 800 nm. Compared to the disease of canker, greasy spot, insect damage, melanose, scab, and wind scar exhibited different reflectance as well as textural features. For scab and wind scar, the diseased areas displayed as gray for all the wavelengths, whereas greasy spot, insect damage, and melanose showed black in certain wavelength ranges (e.g., greasy spot at 675 nm, insect damage and
187
melanose at 600 nm as shown in Fig. 3). This is similar to the canker lesions in terms of gray values. The contrasts between the normal peels and all the diseased regions decreased towards the longer wavelengths. Greasy spot, scab, and wind scar tended to disappear as the wavelength increased beyond 800 nm, while canker, insect damage, and melanose could still be observed at the long wavelengths. These observations suggest that the citrus canker detection may not be achieved using a single wavelength. A hyperspectral or multispectral imaging solution is necessary to discriminate canker from normal and other diseased peel conditions. Meanwhile, stem-ends and calyxes of the fruit samples, which is a persistent problem for automatic quality and safety inspection of fruits and vegetables using machine vision based technologies, appeared in the whole wavelength range. The reflectance of the stem-ends and calyxes changed with wavelengths. However, due to their different reflectance properties from canker, the stem-ends and calyxes demonstrated different contrasts with the canker lesions at different wavelengths. For example, the stem-end at 600 nm in Fig. 3 appeared brighter than the canker lesions due to its relatively high reflectance, while the brightness difference between the stem-end and the canker lesions was reduced at 800 nm owing to their similar reflectance properties at this
Fig. 3. Hyperspectral images of grapefruit samples with normal and diseased peel conditions.
188
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
wavelength. The contrast differences that existed among different wavelengths provided the possibility for separating the stem-ends and calyxes from the true diseased regions. 3.2. Reflectance spectra Fig. 4a shows mean reflectance spectrum and standard deviation obtained from the selected ROIs of the canker lesions. Also, typical reflectance spectra of the grapefruit samples with normal and diseased conditions over the wavelength range between 450 nm and 930 nm were shown in Fig. 4b to demonstrate the general spectral patterns. Each spectrum plot in Fig. 4b was extracted from the hyperspectral image using an average spatial window covering 5 5 pixels for each peel condition. A cankerous sample was also included for the purpose of comparisons. All the reflectance spectra were featured as two local minima around 500 nm and 675 nm due to the light absorption of carotenoid and chlorophyll a in the fruit peel, respectively. Chlorophyll is responsible for the green color of the citrus peel, while the yellow color of the fruit peel is related to carotenoid. The carotenoid and chlorophyll absorptions for greasy spot and canker were not as prominent as other peel conditions. Values of the reflectance for the diseased peel conditions were consistently lower than those from the normal fruit surface over the entire spectral region. Canker had the lowest reflectance except for the spectral region from
450 nm to 550 nm, in which greasy spot showed similar reflectance with canker. In the spectral region from 450 nm to 930 nm, the relative reflectance values of canker (mean spectrum in Fig. 4a) and normal peel were in the range of 12%–43% and 31%– 66%, respectively, while other peel conditions generally had values in between. For canker disease, once the bacterial pathogen invades the fruit peel, the region infected continually loses its moisture content through the season, and the lesions on the fruit surface usually become dark in appearance. This may be the reason for the low reflectance characteristics of the canker disease in visible and short-wavelength near-infrared region. The spectral differences between canker and normal as well as other diseased peel conditions provide a basis for detecting citrus canker using hyperspectral or multispectral imaging approach. 3.3. SID mapping of hyperspectral images Spectral information divergence mapping was applied for all the hyperspectral images of the grapefruit samples using the mean reflectance spectrum of canker in Fig. 4a as the reference. Values of SID between the canker reference spectrum and the spectra of seven peel conditions demonstrated in Fig. 4b were illustrated in Fig. 5a. The SID values with respect to canker reference for canker, normal, greasy spot, insect damage, melanose, scab, and wind scar
Fig. 4. (a) Mean spectrum and standard deviation from ROIs of canker, and (b) typical reflectance spectra of grapefruit samples with normal and diseased peel conditions.
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
189
Fig. 5. (a) Values of SID between reference spectrum of canker and spectra of seven tested peel conditions, and (b) rule images from SID mapping for the hyperspectral images of grapefruit samples using the reference spectrum of canker.
were 0.004, 0.022, 0.010, 0.013, 0.012, 0.016, and 0.017, respectively. As expected, the SID between canker and the canker reference had the lowest value, and the normal peel and the canker reference had the highest SID value. Values for other peel conditions were in between. It should be noticed that the SID algorithm is relatively insensitive to illumination effects and brightness shifts, which is similar to the SAM algorithm (Chang, 2000). It is the relative but not the absolute spectral discrepancies that determine the similarities between different spectra. For example, the absolute reflectance difference between scab and the canker reference was larger than that between wind scar and the canker reference (see Fig. 4b). However, the scab had the smaller SID value (0.016) than wind scar (0.017), indicating that scab was more similar to canker than wind scar in this case. Representative rule images from SID mapping for the hyperspectral images demonstrated in Fig. 3 are shown in Fig. 5b, in which each pixel represents the value of SID between the corre-
sponding spectrum extracted from that pixel and the reference spectrum of canker. The canker lesions showed distinctive dark due to their small SID values, while the stem-ends and the other diseases including greasy spot, insect damage, melanose, scab, and wind scar displayed as gray in different degrees. The normal areas on the fruit peels appeared lighter than the diseased areas because of their larger SID values, indicating their large spectral differences from canker. Similar patterns were observed for the rule images of other grapefruit samples. 3.4. Image classification Fig. 6 demonstrates primary procedures for hyperspectral image processing and classification for citrus canker identification using a grapefruit sample with both canker and wind scar on the fruit peel. ROIs of canker were first selected from the reference hyperspectral images of the selected cankerous samples. Mean
Fig. 6. Procedures of image processing and classification for citrus canker detection.
190
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
Fig. 7. Binary images after thresholding applied to the rule images of grapefruit samples.
reflectance spectrum, which was calculated from the spectra extracted from all the canker ROIs, was used as the reference spectrum of canker. SID mapping was then performed on the hyperspectral images of the test samples using the canker reference spectrum to obtain the rule images. Finally, a simple thresholding method was applied to the rule images to separate canker lesions from the fruit peel. The SID threshold value for segregating the canker lesions from the rule images was determined with an aim that all the image pixels in the unaffected regions including stem-ends and calyxes as well as the diseases other than canker on the fruit peel were converted to zero in the binary images (i.e., displayed as black if the pixel values were larger than the threshold). The binary images with remaining white pixels were classified as cankerous samples. As shown in Fig. 6, the fruit surface and the diseased areas of wind scar disappeared in the resultant binary image, and the white spots represent the canker lesions isolated from the fruit peel. The binary images after thresholding applied to the rule images of the grapefruit samples in Fig. 5b are shown in Fig. 7. The normal fruit surfaces and stem-ends as well as the diseases other than canker were converted to black. Only canker lesions were remained as white spots in the binary images. The canker lesions at both center and edge areas of the fruit were identified and the surrounding areas of the canker (e.g., the canker lesion at the center area) were excluded, showing the effectiveness of the SID based classification method for canker detection. Five SID threshold values from 0.005 to 0.009 with a step size of 0.001 were used to test the classification results. The value of 0.008 provided the best overall classification accuracy (balancing the errors of false negative and false positive), therefore it was chosen as the threshold for the classifier (illustrated as a horizontal line in Fig. 5a). The algorithms for hyperspectral image processing and classification described above were evaluated using all the grapefruit samples tested in this study. Table 1 summarizes the classification results for differentiating citrus canker from normal and other diseased peel conditions. As shown in the table, 96.7% of the cankerous grapefruits were correctly classified as ‘Canker’ class, and accuracy for classifying the 180 samples in ‘No Canker’ class was 96.1%. A total of eight samples was misclassified, and the overall classification accuracy for all the samples was 96.2%.
Table 1 Classification results for differentiating citrus canker from other peel conditions. Class
Peel condition
Canker (n = 30) No Canker (n = 180)
Canker Normal Greasy spot Insect damage Melanose Scab Wind scar Total
Number
Misclassified
Accuracy %
30 30 30 30 30 30 30
1 0 3 2 2 0 0
96.7 96.1
210
8
96.2
For the misclassified samples, one sample was from ‘Canker’ class, and the other seven were from ‘No Canker’ class, including three ‘Greasy Spot’ samples, two ‘Insect Damage’ samples, and two ‘Melanose’ samples. All ‘Normal’, ‘Scab’, and ‘Wind Scar’ samples were correctly classified. The reflectance properties for greasy spot, insect damage, and melanose were similar to those of canker (see Fig. 4b), thus the values of SID between these three diseases and canker reference spectrum were relatively small, as demonstrated in Fig. 5a. For the severe infected conditions, the SID values could be lower than the threshold value that was applied to the rule images. That may be the reason for the low classification accuracies for these three cases. It was also found that the misclassified sample in ‘Canker’ class had a single, light canker lesion that appeared in the early development stage, indicating that the classification method developed based on the SID mapping using the reference spectrum could be affected by different degrees of canker, which could cause changes for reflectance characteristics at different development stages over the harvest season. It should be noted that the best overall classification accuracy (96.2%) using the SID threshold value of 0.008 was obtained based on the assumption that the two types of errors (i.e., false negative and false positive) were weighted equally. However, from the economic standpoint, the error of missing a cankerous fruit (false negative) can significantly outweigh that of incorrect classification of a fruit without canker (false positive). Increasing the threshold generally can reduce the false negative error. For the five threshold values tested in this study, the value of 0.009 gave no error for the cankerous fruit classification, although the overall classification accuracy under this threshold was decreased to 95.2% due to the increased false positive error. Hence for the SID based classification method, we can adjust the two types of errors by decreasing or increasing the threshold value to meet the application needs. 4. Conclusion Hyperspectral reflectance imaging coupled with a spectral information divergence based image classification method provides a useful means for detecting canker lesions on citrus fruit. A hyperspectral imaging system was developed to measure the reflectance images from grapefruit samples with cankerous, normal and other common diseased peel conditions including greasy spot, insect damage, melanose, scab, and wind scar in the wavelength range between 450 nm and 930 nm. Reference spectrum of canker was obtained from the ROIs that were selected from the reference hyperspectral images of the cankerous samples. SID mapping was performed on the images of all the grapefruit samples to quantify the spectral similarities between the canker reference spectrum and the hyperspectral pixels. The rule images from SID mapping provided higher contrasts between canker lesions and normal peels as well as other diseases than the original reflectance images, indicating that SID mapping could extract useful features from hyperspectral images using particular reference spectrum. The simple thresholding method that was used for converting
J. Qin et al. / Journal of Food Engineering 93 (2009) 183–191
the rule images to the binary images was able to segregate the canker lesions from the background of fruit surface. The SID based classifier could differentiate canker from normal fruit peels and other citrus diseases, and it also could avoid the negative effects of stem-ends and calyxes. Optimal SID threshold value was determined based on the assumption that the errors of false negative and false positive were equally weighted, by which the overall classification accuracy was achieved 96.2%. Since the economic impact of missing a cankerous fruit is much higher than that of misclassifying a fruit without canker, we can minimize the false negative error by properly selecting the threshold value with a cost of reducing the overall classification accuracy. Greasy spot, insect damage, and melanose had similar reflectance properties to canker, and the chances for misclassifying these three diseases were higher than other peel conditions. Canker lesions at different development stages could cause changes for reflectance characteristics, thus might affect the SID based classification method that utilizes the reference spectrum. Further work will be performed to investigate the changing patterns of the canker reflectance properties over the harvest season, and incorporate canker spectra at different development stages into the classification model for achieving better detection accuracies. Meanwhile, the full spectral information used by the SID mapping would limit its applications such as on-line disease inspection due to the large amount of hyperspectral data. Methods for optimal band selection and multispectral image classification will be developed in the future work to fulfill the goal of real-time citrus canker detection. Acknowledgements The authors gratefully acknowledge the financial support of Florida Fresh Packer Association and USDA Technical Assistance for Specialty Crops. The authors would also like to thank Dr. Moon Kim and Dr. Kuanglin Chao of USDA-ARS Environmental Microbial and Food Safety Laboratory, and Mr. Mike Zingaro and Mr. Greg Pugh of University of Florida, for their help in designing and building the hyperspectral imaging system. References Aleixos, N., Blasco, J., Navarrón, F., Moltó, E., 2002. Multispectral inspection of citrus in real-time using machine vision and digital signal processors. Computers and Electronics in Agriculture 33 (2), 121–137. Ariana, D.P., Lu, R., Guyer, D.E., 2006. Near-infrared hyperspectral reflectance imaging for detection of bruises on pickling cucumbers. Computers and Electronics in Agriculture 53 (1), 60–70.
191
Blasco, J., Aleixos, N., Gómez-Sanchis, J., Moltó, E., 2007. Citrus sorting by identification of the most common defects using multispectral computer vision. Journal of Food Engineering 83 (3), 384–393. Chang, C.-I., 2000. An information theoretic-based approach to spectral variability, similarity and discriminability for hyperspectral image analysis. IEEE Transactions on Information Theory 46 (5), 1927–1932. Chao, K., Yang, C.C., Chen, Y.R., Kim, M.S., Chan, D.E., 2007. Fast line-scan imaging system for broiler carcass inspection. Sensing and Instrumentation for Food Quality and Safety 1 (2), 62–71. Edwards, J.G., Sweet, C.H., 1986. Citrus blight assessment using a microcomputer, quantifying damage using an apple computer to solve reflectance spectra of entire trees. Florida Scientist 49 (1), 48–53. Gaffney, J.J., 1973. Reflectance properties of citrus fruits. Transactions of the ASAE 16 (2), 310–314. Gómez-Sanchis, J., Gómez-Chova, L., Aleixos, N., Camps-Valls, G., MontesinosHerrero, C., Moltó, E., Blasco, J., 2008. Hyperspectral system for early detection of rottenness caused by Penicillium digitatum in mandarins. Journal of Food Engineering 89 (1), 80–86. Kim, M.S., Chen, Y.R., Mehl, P.M., 2001. Hyperspectral reflectance and fluorescence imaging system for food quality and safety. Transactions of the ASAE 44 (3), 721–729. Kim, M.S., Lefcourt, A.M., Chao, K., Chen, Y.R., Kim, I., Chan, D.E., 2002. Multispectral detection of fecal contamination on apples based on hyperspectral imagery. Part I. Application of visible and near-infrared reflectance imaging. Transactions of the ASAE 45 (6), 2027–2037. Lawrence, K.C., Yoon, S.C., Heitschmidt, G.W., Jones, D.R., Park, B., 2008. Imaging system with modified-pressure chamber for crack detection in shell eggs. Sensing and Instrumentation for Food Quality and Safety 2 (3), 116–122. Lu, R., 2003. Detection of bruises on apples using near-infrared hyperspectral imaging. Transactions of the ASAE 46 (2), 523–530. Lu, R., Chen, Y.R. 1998. Hyperspectral imaging for safety inspection of food and agricultural products. In: Pathogen Detection and Remediation for Safe Eating, SPIE Proceedings, vol. 3544, pp. 121–133. Miller, W.M., Drouillard, G.P., 2001. Multiple feature analysis for machine vision grading of Florida citrus. Applied Engineering in Agriculture 17 (5), 627–633. Park, B., Windham, W.R., Lawrence, K.C., Smith, D.P., 2007. Contaminant classification of poultry hyperspectral imagery using a spectral angle mapper algorithm. Biosystems Engineering 96 (3), 323–333. Pydipati, R., Burks, T.F., Lee, W.S., 2006. Identification of citrus disease using color texture features and discriminant analysis. Computers and Electronics in Agriculture 52 (1–2), 49–59. Qin, J., Lu, R., 2005. Detection of pits in tart cherries by hyperspectral transmission imaging. Transactions of the ASAE 48 (5), 1963–1970. Schubert, T.S., Rizvi, S.A., Sun, X.A., Gottwald, T.R., Graham, J.H., Dixon, W.N., 2001. Meeting the challenge of eradicating citrus canker in Florida – Again. Plant Disease 85 (4), 340–356. Timmer, L.W., Garnsey, S.M., Graham, J.H., 2000. Compendium of Citrus Diseases, Second ed. The American Phytopathological Society, St. Paul, MN, USA. van der Meer, F., 2006. The effectiveness of spectral similarity measures for the analysis of hyperspectral imagery. International Journal of Applied Earth Observation and Geoinformation 8 (1), 3–17. Yang, C., Everitt, J.H., Bradford, J.M., 2008. Yield estimation from hyperspectral imagery using spectral angle mapper (SAM). Transactions of the ASABE 51 (2), 729–737.