- Email: [email protected]
Visual discrimination of citrus HLB based on image features
Vibrational Spectroscopy 102 (2019) 103–111
Contents lists available at ScienceDirect
Vibrational Spectroscopy journal homepage: www.elsevier.com/locate/vibspec
Visual discrimination of citrus HLB based on image features ⁎
T
YanDe Liu , Huaichun Xiao, Hai Xu, Yu Rao, Xiaogang Jiang, Xudong Sun School of Mechanical and Electronical Engineering, East China Jiaotong University, NC, 330013, China
A R T I C LE I N FO
A B S T R A C T
Keywords: Citrus greening Visual discrimination GLCM Image texture feature PLS-DA
Citrus greening (Huanglongbing, HLB) has seriously threatened the healthy development of citrus industry in recent years, and it is great significance to diagnose citrus HLB timely and accurately. Hyperspectral technology has the characteristics of spectral analysis and image processing, which has shown great advantages in plant disease detection. The visual discrimination methods of citrus HLB based on features of images combined with hyperspectral imaging technology were discussed. Five types of citrus leaves, including the mild HLB, moderate HLB, serious HLB, malnourished and normal were studied and polymerase chain reaction (PCR) test was used to verify visual division. Effective spectral variables were selected by successive projections algorithm (SPA) and features of images were extracted by principal component analysis (PCA). Texture features of images based on grayscale co-occurrence matrix (GLCM) were used to develop the partial least squares discriminant analysis (PLS-DA) models. The influence of the number of textures on the models was discussed, and the effect of discrimination model was the best when a total of 36 variables which were obtained from 4 independent texture features were as input, which resulted in greater misjudgment rate of 3.12%, higher correlation coefficient for prediction (RP) of 0.98, lower root mean square error for prediction (RMSEP) of 0.32, and the number of principal component factors (PCs) of 16. The results highlighted that the image texture features based on GLCM combined with the PLS-DA models could realize the identification of citrus HLB, and provide the important reference value for the visual discrimination research of HLB.
1. Introduction Nowadays, the harm of citrus greening (Huanglongbing, HLB) for the citrus industry has increased. It is important to identify citrus HLB effectively for the development of citrus industry. Therefore, the detection of HLB has become a hot topic today. Hyperspectral imaging technology can provide spectral information of multi-target samples and represent external traits and internal variations, which has the advantages of both spectral analysis and image processing techniques for detecting the diseases of the leaves [1]. Cardinali et al proposed attenuated total reflection fourier transform infrared spectroscopy and induced classifier combined with partial least squares regression analysis to detect sweet orange leaf HLB and citrus variegated chlorosis. The results showed that the accuracy of identifying different symptom leaves was 93.8% [2]. Li Xiuhua et al. analyzed the image spectra of citrus orchards using airborne spectral imaging technology which showed that the reflectance rate and red edge position (REP) of the canopy of healthy and infected plants in different bands were different and the accuracy of separating HLB and healthy samples by simple REP threshold method was over 90% [3]. The external visual morphology and internal physiological function ⁎
of citrus leaves were changed after being infected by HLB pathogen. The former expression is the leaf becomes yellow, roll shape even fall off and so on. The latter showed a certain degree of increase or decrease in cell structure, interstitial space and pigment content, which resulted in the change of physical and chemical index content in citrus leaves [4]. The structure of chlorophyll molecules and the cytochrome gap were changed and chlorophyll content were decreased, the content of starch and soluble sugar were also changed, and the spectra of infected parts were different from those of healthy areas, which provided a theoretical basis for the diagnosis of citrus HLB by spectra and images [5,6]. Existing research was less on extracting texture features of citrus leaf images, so it was necessary to study the detection method of HLB based on the texture features of citrus leaves [7–9]. A similar unit with repeated and regular distribution of texture images was usually called image texture descriptor (texture feature). It summarizes the 3 dimensional distribution and correlation based on image gray level, and reflects the similarity or the same regularity of a large number of features of the sample comprehensively [10–12]. Compared to object geometry and grayscale features, texture features contain richer information to help judge categories visually [13].
Corresponding author. E-mail address: [email protected] (Y. Liu).
https://doi.org/10.1016/j.vibspec.2019.04.001 Received 11 December 2018; Received in revised form 5 March 2019; Accepted 2 April 2019 Available online 03 April 2019 0924-2031/ © 2019 Elsevier B.V. All rights reserved.
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
hyperspectral camera is composed of a charge-coupled device (Hamamatsu C8484-05G, Japan) with a resolution of 1344 × 1024 pix, an imaging spectrometer based on transmittance grating (ImSpector V10, Spectral Imaging Ltd., Finland) and a focusing lens (V23 Spectral Imaging Ltd). The samples were scanned by the hyperspectral camera using the method of line scanning to project the light of each band onto the camera chip, and the images were recorded in spatial and spectral dimensions. The samples were moved and scanned line by line by an electric translation platform under the online scanning camera. At the same time, the light source which was composed of 4 bromine tungsten lamps (ranging from 350 to 2,500 nm, total power ≥ 100 W) was used to irradiate the samples. The device structure is shown in Fig. 2. The whole device was controlled by the computer using the SpectVIEW software. The working principle is that the sample on the electric translation platform (or conveyor belt) is illuminated by the diffuse reflection source, and then the lens receives the reflected light which transmits it to the imaging spectrometer to obtain images. The sample moves with the motorized translation platform, so the continuous images and the real-time spectral information are obtained. All data are recorded by the SpectVIEW software, and finally a three-dimensional data cube which consists of the image and spectral information is obtained. The information inside or outside of the object, or other information required for sorting can be obtained by the data analysis, and full automated sorting may be realized by the subsequent control and development. The samples suitable for the system are limited to the size of 300 mm (length) × 300 mm (width) × 100 mm (height), and continuous measurement by small batches can be fulfilled.
In this study, 5 kinds of citrus leaves were used as the research object. Polymerase chain reaction (PCR) test was used to verify the feasibility of dividing the mild HLB, moderate HLB, serious HLB, deficiency and normal samples through vision. The variables were extracted by principal component analysis (PCA) and successive projections algorithm (SPA), and the sensitive wavelengths were selected for the analysis of image features. The methods of extracting image festures for the leaves based on gray level co-occurrence matrix (GLCM) were studied to explore the feasibility of detecting the citrus HLB based on image features. 2. Materials and methods 2.1. Sample preparation The changes in its leaves were most obvious after citrus infection with HLB, and citrus leaves were picked in a citrus orchard on Jiangxi Province in 2017, which were classified into the normal, HLB and malnourished visually. A total of 300 leaves, which contained 60 normal leaves, 180 HLB leaves and 60 malnourished leaves on each tree were picked from southeast to northwest. All leaves were pre-treated (washed, dried, flattened and labeled) and stored in a refrigerator at a temperature and humidity of 5 ℃ and 70%, respectively. Because of the different degrees of the HLB samples, the attempts were made to subdivide the leaves infected with HLB into mild, moderate and serious. The PCR tests were performed to verify the reliability of the visual initial division after all the hyperspectral images of the leaves were collected. The total number of failures screened out was 44, including 12 normal leaves, 12 leaves infected with mild, moderate and serious HLB and 20 malnourished leaves, and 256 leaves remained finally. Hyperspectral images were collected in the laboratory at 22 ℃ and humidity of 60%. The PCR tests were performed on all samples in conformity to national standards. OI1OI2 as Jagoueix et al and A2J5 as Hocquellet et al reported respectively were chosen as primers for tests and synthesized by with GenScript (Nanjing) Co., Ltd [14,15]. As shown in Fig. 1, the results of the former primer were more clear, and the results of infection with HLB were positive, while those of noninfection were negative. In Fig. 1, the labels of 17, 8 and 9 represented mild, moderate and serious infected leaves, respectively. They revealed bright bands successively and the light bands became heavier and heavier with the degree of severity [16].The labels of 11 and 12 also revealed bright bands, but the color of bright band was very light, which might be related to the lack of nutrients. The labels of 19 and 20 represented the normal leaves without bright band. M was used for DNA labeling and comparison in DNA gel electrophoresis to estimate the molecular weight of sample DNA. The remaining numbers represented the corresponding citrus tree labels, which were not relevant to the experiment. Table 1 below shows the specific categories of the leaves.
2.3. Image acquisition The hyperspectral system was preheated for about 30 min before the image collection. At the same time, the machine parameters were set by the SpecVIEW software, including the resolution of 1344 × 1024 pix, the spectral range of 367–978 nm, the spectral resolution of 2.8 nm, the exposure time of 10.9 ms, the moving speed of the electric translation platform of 0.4 cm/s, and the distance between objective lens by 25 cm. They were determined after the repeated tests since the latter three parameters have an effect on the distortion, size and sharpness of the image. After the parameters were set, the focusing operation was carried out and the process was that the camera lens was rotated to see 3 spectra of RGB in the focusing window after the lens was aligned with the dividing line between standard black and white plates. The corresponding focal length was the best when the sharpness of the spectra was the best, and the focusing work was finished. The sample was then tiled face up and placed on the same level with the standard whiteboard and the image acquisition was conducted. 2.4. Image calibration
2.2. Hyperspectral imaging system
After all the sample images were acquired, the black and white calibration was performed by the SpectraVIEW software to avoid the uneven distribution of the light intensity and the influence of the dark current on the images [17].The steps was to open the shutter to collect the images of the standard white board (IW) and the images of the black board reference (ID) were acquired after the shutter closed. Formula 1 displays the principle of the image calibration to switch the original image IS into calibrated image R.
The hyperspectral images of the samples were collected in the laboratory of College of Mechanical and Electrical Engineering, East China Jiaotong University. GaiaSorter hyperspectral imaging device was used in this study which includes an electric translation stage (or conveyor), the hyperspectral camera in the range of 367–978 nm, the diffuse light source, a computer and the controlling software etc. The
Fig. 1. The results of the PCR tests for citrus leaves.
104
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Table 1 The categories of the samples. The number of samples
Grade
The infection of the leaves
56 56 56 40 48
mild HLB Moderate HLB serious HLB Nutrient deficiency Normal
mild symptoms, PCR test positive Moderate symptoms, PCR test positive serious symptoms, PCR test positive No symptoms of HLB, PCR test negative No symptoms of HLB, PCR test negative
3. Results and discussion 3.1. Hyperspectral image features and spectral analysis of the citrus leaves The hyperspectral image of the leaves acquired was a three-dimensional data block containing a bunch of continuous two-dimensional images at a different wavelength. The image could be viewed as a spectrum I(λ) at each individual pixel(x、y)or a image I(x,y) at each individual wavelength λ. Visualization information of the spatial distribution was acquired at the pixel level of each image and used to analyze the distribution of HLB on the leaves. Each pixel containing the complete spectrum was used to characterize the stripping conditions. Therefore, hyperspectral image analysis and processing could be performed in the spectral or image domain [23]. Fig. 3 shows the concept of the hyperspectral images. The synthesized RGB true color image of the leaf was closest to the actual image of the leaf under the 3 channels of R: 639.84 nm, G: 548.98 nm and B: 460.5 nm. It was found that there were some differences in the original images of the HLB and normal leaves and their grayscale images at different wavelengths. The mesophyll of the HLB leaves was transparent and the distinction between veins and mesophyll was more obvious than normal leaves while the outline of normal leaves was clearer especially at 550 nm. At the same time, the average reflectance spectra of each pixel in the region of interest (ROI) were extracted and found to be apparently different at the 500–600 nm band [24]. Finally, the average reflectance spectra in the ROI of the middle rectangle on the left of the veins were selected for subsequent analysis. The representative spectra of mild, moderate, serious HLB, malnourished and normal leaves are shown in Fig. 4. It was found that the overall variation trend of spectral curves of the 5 types of leaves was the same. In the spectra, the distinct reflection band near 550 nm is assigned to chlorophyll which can be called “green peak”. The leaf reflection peak of HLB was higher than that of the malnourished and normal leaves, and the peak gradually decreased with the severity of disease. It might be that HLB pathogens hindered the absorption of water by the leaves, resulting in low chlorophyll content. The lower peak near 700 nm could depend on the number of organic molecules it contain in the leaves and be caused by the destruction of cell structure by pathogenic bacteria of HLB [24]. A steep slope in the range of 670–740 nm is close to a straight line and the reflectivity rised sharply, which can be called “red edge”. It has been increasingly and verified as an indicator of plant nutrition, growth, moisture, etc [25,26].
Fig. 2. The structure of the hyperspectral device.
R=
IS − ID IW − ID
(1)
Where R represents the calibrated image, IS represents the original image, ID represents the image of the black board reference, and IW represents the image of the standard white board. All data were analyzed by ENVI 4.5 software after the image calibration completed.
2.5. Modeling and dimension reduction method Partial least squares discriminant analysis (PLS-DA) is one of the multivariate statistical analysis methods based on PLS regression which is a bilinear modeling method and carried out by the spectral variables and classification value. The training and the credibility is usually conducted and tested according to the characteristics of the different samples. The advantage is to reduce the effects of the multicollinearity among the variables. The specific processes are as follows: (1) setting the classification value of the samples; (2) performing PLS analysis on the spectral data and the corresponding classification value to establish a regression model; (3) predicting the value YP of the unknown sample according to the model; and, (4) calculating the difference between the set value and the predicted value (δ = Y − YP) [18,19]. Due to the high dimensionality of the hyperspectral images, the noise components and irrelevant output frequency bands were separated from the images to reduce the spectral dimension. Principal component analysis (PCA) and successive projections algorithm (SPA) were selected as the dimensionality reduction methods in this study. The principle of PCA is to find a set of new orthogonal axes created by the average of the data and rotate them for maximizing the variance of the data. The number of bands corresponding to the principal component can be equal to the number of bands of the original images. However, only the first few unrelated principal components (PCs) contain most of the relevant information [20]. SPA, as a new variable selection strategy, was used in hyperspectral image analysis for multivariate calibration. Simple projection operations are used in vector space to select subsets of variables and minimize the collinearity of the variables. They can speed up the subsequent modeling based on feature image and simplify the model to improve robustness and accuracy [21,22].
3.2. Sensitive wavelength extraction and feature image analysis The SPA was used to select the sensitive wavelength variables based on the 256 data points of the average spectra in the ROI. The original variables were replaced by a few variables containing the characteristic information to simplify the subsequent model. The spectra were normalized during the operation of the SPA to facilitate subsequent data processing. Normalization is to perform data processing through a standardized algorithm and the variables can be reduced in the range of 0–1. Normalization can avoid data masking because of different dimensions, making the model more reliable and eliminating dimension effects. The normalization method is mature and widely used which is 105
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Fig. 3. The schematic diagram of the hyperspectral image in the spectral and spatial domains.
Fig. 4. Representative spectra of five kinds of leaves.
Fig. 5. The results of SPA.
suitable for a variety of data requirements. Z-score standard normalization algorithm was used in this study, by which the original data was normalized to a data set with a mean of 0 and variance of 1. The formulas are shown in 2–4.
According to the sensitive wavelengths selected by SPA, the grayscale image of the samples at the 410.29 nm were blurred and mixed with the background, so the remaining 9 wavelength points were selected for each sample. The hyperspectral image cubes of the samples at 256 data points were rich in information but difficult to process, so the image data were reduced by PCA, which could contain the information of the original data to the greatest extent and useful information was concentrated among at least a few PCs. PCA obtains new and unrelated variables by creating a new coordinate system that blocks the noise and reduces dimension. As shown in Formula 5, each PC image is a linear combination of each monochrome image.
Z=
μ=
σ=
x−μ σ
(2)
x1 + x2 +⋯+x n = n
1 n−1
n ∑i = 1 x i
n
(3)
n
∑ (xi − μ) i=1
(4)
p
PCimg =
Where x represents the variables of the original spectra, μ and σ represent the mean and standard deviation of the original data, and n represents the number of samples. The normalization method requires original data distribution to approximate the Gaussian distribution, otherwise the effect of the normalization will be worse [27]. Fig. 5 shows the results of the SPA and the selected wavelength points were 410.29、508.33、668.54、690.06、852.66、871.79、900.49、 943.53、957.87、974.61 nm.
∑ ωi Si i=1
(5)
Where wi represents the weight coefficient of wavelength, Si represents the image at the sensitive wavelength, and p represents the number of wavelengths. The weight values were calculated based on the image covariance matrix which represented the variance of each PC image [23]. The 256 variables of which the sum of the variance was kept constant were analyzed by PCA in ENVI 4.5 software and the PC images 106
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
the grayscale images of the HLB samples were darker than those of the normal leaves and the grayscale of the malnourished was the brightest in the PC1 images that contained the majority of the original data. However the grayscale of the PC1 images was high, which made it difficult to distinguish. In the PC2 images, the distinction degree between the mesophyll and veins of the HLB samples decreased with the severity of the disease gradually and the normal were the most blurred. The clarity of mesophyll tissue of the leaves decreased gradually for the first three kinds of the leaves in the PC3 images and the mild HLB was the clearest. While the absence of the normal might be due to the same color of the veins and mesophyll. Higher brightness of PC4 images might be caused by loud noise. Therefore, it was different to visually distinguish 5 kinds of leaves based on the first 4 PC images. So the PC contribution rate values were calculated to select the sensitive wavelengths. The weight coefficient is used to measure the significance of the indicator in the system, which reflects the impact on the results under certain preconditions. In this study, the weight coefficients of the first 4 PCs were estimated and analyzed, and the curve of the weight coefficients based on PCA could be drawn with 256 wavelength points in the range of 367–977 nm. 5 representative samples were selected randomly for all kinds of the leaves, and the maximum (minimum) values of the curve were analyzed to obtain the corresponding sensitive wavelengths. Table 2 shows the sensitive wavelengths selected by the above method. Therefore, the sensitive wavelengths were preferred to prepare for extracting texture features to classify 5 kinds of leaves visually [28,29]. Due to obvious differences in the grayscale images with a interval of 10 nm and above, 3 wavelengths of 700, 715 and 725 nm were selected between 700–725 nm and 870–890 nm were omitted. 14 sensitive wavelengths were finally extracted as shown in the Table. The PLS-DA model was established with the corresponding image texture features based on the sensitive wavelengths, which could help carry out visual identification and classification of citrus HLB.
Fig. 6. Cumulative contribution rates of the first 20 PCs images.
of the original band were obtained. The PC images with the order of the variance are PC1, PC2, and so on, which are the linear combination of original images and independent. The smaller is the variance, the larger of noises are contained in the PC images, and 20 PCs are usually selected. Fig. 6 shows the cumulative contribution rate of the first 20 PCs in the range of 367–977 nm. As shown in Fig. 6, the cumulative contribution rate of the first 2 PCs was 93%, and the cumulative contribution rate of the first 4 PCs reached 98%, which could explain most of the information of the original spectra. Therefore, the first 4 PCs grayscale images were analyzed for establishing the subsequent visual discrimination models. Because the sample spectra have a corresponding relationship with the images, the sensitive wavelengths need to be selected before the extraction of the texture features. Therefore, the sensitive wavelengths were selected by two ways in this study. On the one hand, the SPA was used to screen the sensitive wavelengths. On the other hand, the PCA was used to compress the variables of all the samples and the sensitive wavelengths were selected preferably. The corresponding grays images at the sensitive wavelengths were defined as the feature images for texture extraction. The feature images at the sensitive wavelengths selected by SPA were analyzed. Fig. 7 shows the grayscale images of 5 kinds of leaves including mild HLB, moderate HLB, serious HLB, malnourished and normal leaves at 508.33 nm. The highest transparency was on the leaves of the mild HLB, which decreased with the disease level gradually, so the discrimination between the mesophyll and veins also decreased. The transparency of normal mesophyll was the lowest. The edge profile of the HLB leaves decreased with the disease level gradually, but the mesophyll and veins of the nutritional deficiency were more blurred, which might be caused by the lack of the relevant nutrients in the leaves. Therefore, it was necessary to extract the grayscale texture features of the leaves deeply to visual discrimination and classification for HLB. As could be known from the above, the feature of the first 4 PCs images of samples was more specific, which reflected the majority information of 5 kind of samples. Fig. 8 shows the original images and the first 4 PCs grayscale images of the mild HLB, moderate HLB, serious HLB, malnourished and normal leaves and the PC grayscale images retained the majority of original image. Comparing the 4 PCs images,
3.3. Second-order statistic based on gray level co-occurrence matrix The image features are usually described by texture features, and the recent studies has proved that the statistical method based on gray level co-occurrence matrix (GLCM) has certain potential in extracting texture features. In this study, the texture features were obtained by calculating the secondary statistics of the GLCM. Since the shapes of the leaves were inconsistent and the capacity of the entire images were large, 8 texture features of the feature images were extracted from the same pixel size of the samples based on the GLCM, which was convenient to calculate and could improve the prediction accuracy and efficiency. The offset parameters were obtained from the directions of 0°, 45°, 90°, and 135° during the extraction. The characteristic indices were used to calculated the mean and standard deviation to suppress the directional component and make the relationship between the texture features and the direction reduce to zero [30–32]. The texture feature calculation formulas are shown in Table 3. 3.4. Visual discriminant model of citrus HLB based on GLCM 72 texture features at 9 sensitive wavelengths selected by SPA and 112 texture features at 14 sensitive wavelengths selected by PCA were used to establish the PLS-DA discriminant model of citrus HLB Fig. 7. Grayscale images of 5 kinds of leaves at 508.33 nm.
107
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Fig. 8. Original image and the first 4 PCs images by PCA for 5 kinds of leaves, (a)mild HLB. (b)moderate HLB. (c)serious HLB. (d)nutritional deficiency. (e)normal.
108
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Among the 8 characteristic parameters based on the GLCM, the parameters of mean, variance, consistency and difference had some correlations with the remaining parameters. Therefore, the PLS-DA model of HLB was built with the independent features of contrast, entropy, second moment, correlation and their combination respectively to explore the impact on performance of model with different texture features. As shown in Table 5, the recognition performance of model was more satisfactory based on the PCA weight coefficient method than SPA when single texture feature was as input. But the classification ability of model with SPA method was better than PCA when the texture features were combined as input, and the recognition ability of the combination was better than single texture feature because the useful information of single texture feature was limited. The recognition ability of the model was stronger with the texture features of second moment and correlation than with the texture features of contrast and entropy because the useful information was better represented by second moment and correlation. The recognition performance of PLS-DA model was best when the model based on SPA at the corresponding sensitive wavelengths with the combination of 4 independent texture features, resulting in lower misjudgment rate of 3.12%. The combination of 8 texture features were used to establish PLS-DA model, which could verify that the model effect was not affected by the number of texture parameters. Table 6 shows the results compared with the model with the combination of 4 texture features. As shown in Table 6, the classification effect of the model with the combination of 4 or 8 texture features by SPA was better than PCA, and the models were better with 4 independent textures than 8 textures. The classification effect of the former model was the best, which resulted in lower misjudgment rate of 3.12%, higher RP of 0.98, and lower RMSEP of 0.32. The latter model resulted in worse misjudgment rate of 28.13%. The numbers of PC factors are one of the main factors to evaluate the performance of the model. The cross validation method was used in the experiment to determine the optimal numbers of PC factors. As shown in Fig. 9(a), the RMSE of prediction and calibration decreased with the increase of the numbers of PCs. And the number of PCs was determined to be 16 because the RMSEP was the lowest. As shown in Fig. 9(b), a sample of serious HLB was misjudged as the moderate HLB and a malnourished sample was misjudged as serious HLB. The result of a moderate HLB sample was near the threshold of 1.5. The result of 2 serious HLB samples were near the threshold of 2.5 and 3.5, respectively, and the result of a normal sample was near the threshold of 4.5. These samples above were not misjudged, so the total misjudgment rate was 3.12%. The performance of discriminant model with texture features was evaluated by misjudgment rate of 64 predictive samples. The sensitive wavelengths were selected by two methods, and texture features were extracted from corresponding feature images to realize visual discrimination of non-destructive detection of citrus HLB. The performance of the model by SPA was superior to PCA, which could be due to the fact that the sensitive wavelengths with the main information of spectra but low weight coefficients were not selected. Comparing the full band, both methods could reduce dimension and optimize the model. The results showed that 9 sensitive bands selected by SPA combined with 4 texture features extracted from the GLCM were used to establish the model, resulting in the classification accuracy rate of above 95%. 9 sensitive wavelengths could be used to replace the full bands for visual expression of the image information and discrimination of citrus HLB.
Table 2 Sensitive wavelengths extracted by the weight coefficients based on PCA. category
number
Sensitive wavelength (/nm)
mild HLB moderate HLB serious HLB malnourished normal total
7 6 8 7 8 14
440、450、520、650、700、710、890 440、640、705、710、880、890 430、440、650、715、720、770、870、890 440、650、705、710、715、890、880 440、720、725、760、870、875、880、890 430、440、450、520、640、650、700、715、 725、760、770、870、880、890
Table 3 Texture features and calculation formulas. texture feature
calculation formulas
mean
Mean = ∑m, n = 0 mP (m, n)
variance
G G VAR = ∑m = 0 ∑n = 0 (m − t )2P (m, P (m, n) G G HOM = ∑m = 0 ∑n = 0 1 + (m − n)2
G
consistency
n)
contrast
CON = ∑m = 0 ∑n = 0 (m − n)2P (m, n)
difference
G G DIS = ∑m = 0 ∑n = 0 P (m, n)|m − n| G G ENT = − ∑m = 0 ∑n = 0 P (m, n) log P (m, n) G G ASM = ∑m = 0 ∑n = 0 P 2 (m, n) (m − Meanm )(n − Meann ) P (m, n) G G COR = ∑m = 0 ∑n = 0 VARm VARn
G
entropy second moment correlation
G
Table 4 Divisions of data sets. category
assignment
calibration set
prediction set
total
mild HLB moderate HLB serious HLB nutritional deficiency normal
1 2 3 4 5
42 42 42 30 36
14 14 14 10 12
56 56 56 40 58
Table 5 Results of PLS-DA model with different texture features. motheds
texture feature
Number of variable
PCs
RP
RMSEP
misjudgment rate/%
SPA
second moment contrast correlation entropy second moment,correlation contrast, entropy independent feature combination
9 9 9 9 18
2 1 1 4 8
0.34 0.29 0.80 0.79 0.86
1.28 1.31 0.84 0.84 0.69
53.12% 65.63% 59.37% 50% 21.88%
18 36
7 16
0.75 0.98
0.90 0.32
43.75% 3.12%
second moment contrast correlation entropy second moment,correlation contrast, entropy independent feature combination
14 14 14 14 28
3 1 1 13 17
0.65 0.58 0.83 0.85 0.87
1.0 1.09 0.77 0.67 0.66
46.88% 53.12% 50% 34.37% 28.13%
28 56
14 14
0.87 0.87
0.70 0.63
46.88% 28.13%
PCA
4. Conclusion
respectively. The samples were divided into two parts, the calibration and prediction sets (ratio of 3:1). The former contained 192 leaves including 126 HLB leaves, 30 malnourished leaves and 36 normal leaves, and the latter contained 64 leaves. The divisions of data sets are shown in Table 4.
In this study, the hyperspectral images of citrus leaves were transformed into the first 4 PC images by PCA. The 14 sensitive bands with high weight coefficients were further obtained by analyzing the PC images, and the SPA method was used to select 9 sensitive wavelengths 109
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Table 6 Results of PLS-DA model with the texture features of different numbers. screening methods
The number of feature
The number of variables
PCs
RC
RP
RMSEC
RMSEP
misjudgment rate/%
SPA
4 8
36 72
16 19
0.98 0.96
0.98 0.90
0.28 0.38
0.32 0.62
3.12% 18.75%
PCA
4 8
56 112
14 18
0.94 0.92
0.87 0.83
0.46 0.55
0.63 0.75
28.13% 28.13%
Fig. 9. The results of best PLS-DA model, (a) the optimal PCs number. (b) the scatter diagram of model.
based on the spectral domain. The texture features of mean, variance, consistency, contrast, difference, entropy, second-order moment and similarity on the corresponding feature images were counted, and 4 independent feature parameters were selected to establish the PLS-DA classification recognition model of HLB. The PLS-DA model based on the combination of 4 independent texture features from the images at the sensitive wavelengths by SPA had better effect by comparing the performance of all the model, which resulted in lower misjudgment rate of 3.12%. The results proved that the feature bands could replace full bands to describe the image information of the samples at which the image information could be used for visual description and realizing the identification of HLB.
[6] M.L. Adams, W.D. Philpot, W.A. Norvel, Yellowness index: an application of spectral second derivatives to estimate chlorosis of leaves in stressed vegetation, Int. J. Remote Sens. 20 (1999) 3663–3675. [7] G.f. Tang, Z.f. Xiao, Q. Liu, H. Liu, A novel airport detection method via line segment classification and texture classification, IEEE Geosci. Remote Sens. Lett. 12 (2015) 2408–2412. [8] R. Rastghalam, H. Pourghassem, Breast cancer detection using MRF-based probable texture feature and decision-level fusion-based classification using HMM on thermography images, Pattern Recognit. 51 (2016) 176–186. [9] S. Murala, Q.M.J.W. Jonathan, Spherical symmetric 3D local ternary patterns for natural, texture and biomedical image indexing and retrieval, Neurocomputing 149 (2015) 1502–1514. [10] H. Murano, Y. Takata, T. Isoi, Origin of the soil texture classification system used in Japan, Soil Sci. Plant Nutr. 61 (2015) 688–697. [11] J.R. Jensen, K. Lulla, Introductory digital image processing: a remote sensing perspective, Geocarto Int. 2 (1987) 65-65. [12] P. Liu, Research on Ovean Oil Spill Detection and Recognition Using SAR Data, Ocean University of China, Shandong, 2012, https://doi.org/10.7666/d.y2158798. [13] J.Z. Lu, Plant Leaves Diseases Detection Using Spectral Imaging Technology, Zhejiang University, Hangzhou, 2016. [14] T. Li, Ke Chong, Detection of the bearing rate of Liberobacter asiaticum, in citrus psylla and its host plant Murraya panciculata by Nested PCR, Acta Phytophylacica Sin. 29 (2002) 31–35. [15] A. Hocquellet, P. Toorawa, J.M. Bové, M. Garnie, Detection and identification of the two Candidatus liberobacter species associated with citrus huanglongbing by pcr amplification of ribosomal protein genes of theβ operon, Mol. Cell. Probes 13 (1999) 373–379. [16] I. Michael, Doing it faster and smarter (Lesson 6 of Matrix Algebra), Spectrosc. Europe 14 (2002) 24–26. [17] L. Yu, Y.S. Hong, Y. Zhou, Q. Zhu, L. Xu, J.Y. Li, Y. Nie, Wavelength variable selection methods for estimation of soil organic matter content using hyperspectral technique, Trans. Chin. Soc. Agric. Eng. (Trans. CSAE) 32 (2016) 95–102. [18] Z. Cheng, L.Q. Zhang, H.Y. Liu, A.S. Chu, Successive projections algorithm and its application to selecting the wheat near-infrared spectral variables, Spectrosc. Spectr. Anal. 30 (2010) 949–952. [19] B. Chen, X.L. Meng, H. Wang, Application of successive projections algorithm in optimizing near infrared spectroscopic calibration model, Fenxi ceshi xuebao 26 (2007) 66–69. [20] P. Baranowski, W. Mazurek, J. Wozniak, U. Majewska, Detection of early bruises in apples using hyperspectral data and thermal imaging, J. Food Eng. 110 (2012) 345–355. [21] X. Zhou, Implementation and comparison of two classifiers based on LS-SVM algorithm, Comput. Knowl. Technol. 7 (2011) 7281–7283. [22] Z. Yang, H.Q. Ren, Z.H. Jiang, Discrimination of wood biological decay by NIR and partial least squares discriminant analysis (PLS-DA), Spectrosc. Spectr. Anal. 28 (2008) 793–796. [23] B.H. Zhang, S.X. Fan, J.B. Li, W.Q. Huang, C.J. Zhao, M. Qian, L. Zheng, Detection of early rottenness on apples by using hyperspectral imaging combined with spectral analysis and image processing, Food Anal. Methods 8 (2015) 2075–2086.
Conflicts of interest There are no conflicts to declare. Acknowledgments The research was funded by: National Natural Science Foundation of China (No. 31760344), Center of the Technology and Equipment of the Intelligent Management for the Southern Mountain Orchard Collaborative Innovation (No. 2014-60). Jiangxi Advantage Science and Technology Innovation Team Construction Project (No. 20153BCB24002). References [1] A.R. Mishra, D. Karimi, R. Ehsani, W.S. Lee, Identification of citrus greening (HLB) using a VIS-NIR spectroscopy technique, Trans. ASABE 55 (2012) 711–720. [2] M.C. do Brasil Cardinali, P.R.V. Boas, D.M.B.P. Milori, E.J. Ferreira, M.F. e Silva, M.A. Machado, B.S. Bellete, Infrared spectroscopy: a potential tool in huanglongbing and citrus variegated chlorosis diagnosis, Talanta 91 (2012) 1–6. [3] X.H. Li, M.Z. Li, S.L. Won, R. Ehsani, Ashish, R. Mishra, Visible-NIR spectral feature of citrus greening disease, Spectrosc. Spectr. Anal. 34 (2014) 1553–1559. [4] L.Q. Lei, C.Y. Guan, Advances in the application of agricultural spectral digital techniques in crop information monitoring, Crop Res. 25 (2012) 626–629. [5] Y. Wu, H.N. Su, A.J. Huang, Y. Zhou, Z.A. Li, J.X. Liu, C.Y. Zhou, Effect of Candidatus Liberibacter asiaticus infection on carbohydrate metabolism in Citrus sinensis, Sci. Agric. Sin. 48 (2015) 63–72.
110
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Mach. 49 (2018) 226–232. [29] Z.X. Zheng, L. Qi, X. Ma, X.Y. Zhu, W.J. Wang, Grading method of rice leaf blast using hyperspectral imaging technology, Trans. Chin. Soc. Agric. Eng. 29 (2013) 138–144. [30] P.P. Jiao, Y.Z. Guo, L.J. Liu, X. Wei, Implementation of gray level co-occurrence matrix texture feature extraction using matlab, Comput. Technol. Dev. 22 (2012) 169–171. [31] Ma, H.B. Pu, D.W. Sun, W.H. Gao, J.H. Qu, K.Y. Ma, Application of vis–NIR hyperspectral imaging in classification between fresh and frozen-thawed pork Longissimus dorsi muscles, Int. J. Refrig. 50 (2015) 10–18. [32] N. Shao, H. Zhou, L.J. Jiang, Y.D. Bao, Y. He, Using reflectance and gray-level texture for water content prediction in grape vines, Trans. ASABE 60 (2017) 207–213.
[24] X.H. Li, W.S. Lee, M.Z. Li, R. Ehsani, A.R. Mishra, C.H. Yang, R.L. Mangan, Spectral difference analysis and airborne imaging classification for citrus greening infected trees, Comput. Electron. Agric. 83 (2012) 32–46. [25] Y.W. Huang, J.H. Wang, X.Y. Li, J.S. Jacqueline, D.H. Han, Research on fast discrimination between Panax ginseng and Panax quinque folium based on near infrared spectroscopy, Spectrosc. Spectr. Anal. 30 (2010) 2954–2957. [26] Z.J. Shan, Z. Feng, G. Zhou, X.L. Deng, Cloning and sequencing of Hang longbing pathogen in shatianyou pomelo, J. Zhongkai Univ. Agric. Technol. 18 (2015) 45–48. [27] X. Peng, S.W. Ye, Z. Shao, H.X. Cai, J. Shi, Y. Tan, J. Li, Recognition of space debris based on spectra and probabilistic neural network, J. Changchun Univ. Technol. 36 (2015) 395–400. [28] Y. Lei, D.J. Han, Q.D. Zeng, D.J. He, Grading method of disease severity of wheat stripe rust based on hyperspectral imaging technology, Trans. Chin. Soc. Agric.
111
Contents lists available at ScienceDirect
Vibrational Spectroscopy journal homepage: www.elsevier.com/locate/vibspec
Visual discrimination of citrus HLB based on image features ⁎
T
YanDe Liu , Huaichun Xiao, Hai Xu, Yu Rao, Xiaogang Jiang, Xudong Sun School of Mechanical and Electronical Engineering, East China Jiaotong University, NC, 330013, China
A R T I C LE I N FO
A B S T R A C T
Keywords: Citrus greening Visual discrimination GLCM Image texture feature PLS-DA
Citrus greening (Huanglongbing, HLB) has seriously threatened the healthy development of citrus industry in recent years, and it is great significance to diagnose citrus HLB timely and accurately. Hyperspectral technology has the characteristics of spectral analysis and image processing, which has shown great advantages in plant disease detection. The visual discrimination methods of citrus HLB based on features of images combined with hyperspectral imaging technology were discussed. Five types of citrus leaves, including the mild HLB, moderate HLB, serious HLB, malnourished and normal were studied and polymerase chain reaction (PCR) test was used to verify visual division. Effective spectral variables were selected by successive projections algorithm (SPA) and features of images were extracted by principal component analysis (PCA). Texture features of images based on grayscale co-occurrence matrix (GLCM) were used to develop the partial least squares discriminant analysis (PLS-DA) models. The influence of the number of textures on the models was discussed, and the effect of discrimination model was the best when a total of 36 variables which were obtained from 4 independent texture features were as input, which resulted in greater misjudgment rate of 3.12%, higher correlation coefficient for prediction (RP) of 0.98, lower root mean square error for prediction (RMSEP) of 0.32, and the number of principal component factors (PCs) of 16. The results highlighted that the image texture features based on GLCM combined with the PLS-DA models could realize the identification of citrus HLB, and provide the important reference value for the visual discrimination research of HLB.
1. Introduction Nowadays, the harm of citrus greening (Huanglongbing, HLB) for the citrus industry has increased. It is important to identify citrus HLB effectively for the development of citrus industry. Therefore, the detection of HLB has become a hot topic today. Hyperspectral imaging technology can provide spectral information of multi-target samples and represent external traits and internal variations, which has the advantages of both spectral analysis and image processing techniques for detecting the diseases of the leaves [1]. Cardinali et al proposed attenuated total reflection fourier transform infrared spectroscopy and induced classifier combined with partial least squares regression analysis to detect sweet orange leaf HLB and citrus variegated chlorosis. The results showed that the accuracy of identifying different symptom leaves was 93.8% [2]. Li Xiuhua et al. analyzed the image spectra of citrus orchards using airborne spectral imaging technology which showed that the reflectance rate and red edge position (REP) of the canopy of healthy and infected plants in different bands were different and the accuracy of separating HLB and healthy samples by simple REP threshold method was over 90% [3]. The external visual morphology and internal physiological function ⁎
of citrus leaves were changed after being infected by HLB pathogen. The former expression is the leaf becomes yellow, roll shape even fall off and so on. The latter showed a certain degree of increase or decrease in cell structure, interstitial space and pigment content, which resulted in the change of physical and chemical index content in citrus leaves [4]. The structure of chlorophyll molecules and the cytochrome gap were changed and chlorophyll content were decreased, the content of starch and soluble sugar were also changed, and the spectra of infected parts were different from those of healthy areas, which provided a theoretical basis for the diagnosis of citrus HLB by spectra and images [5,6]. Existing research was less on extracting texture features of citrus leaf images, so it was necessary to study the detection method of HLB based on the texture features of citrus leaves [7–9]. A similar unit with repeated and regular distribution of texture images was usually called image texture descriptor (texture feature). It summarizes the 3 dimensional distribution and correlation based on image gray level, and reflects the similarity or the same regularity of a large number of features of the sample comprehensively [10–12]. Compared to object geometry and grayscale features, texture features contain richer information to help judge categories visually [13].
Corresponding author. E-mail address: [email protected] (Y. Liu).
https://doi.org/10.1016/j.vibspec.2019.04.001 Received 11 December 2018; Received in revised form 5 March 2019; Accepted 2 April 2019 Available online 03 April 2019 0924-2031/ © 2019 Elsevier B.V. All rights reserved.
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
hyperspectral camera is composed of a charge-coupled device (Hamamatsu C8484-05G, Japan) with a resolution of 1344 × 1024 pix, an imaging spectrometer based on transmittance grating (ImSpector V10, Spectral Imaging Ltd., Finland) and a focusing lens (V23 Spectral Imaging Ltd). The samples were scanned by the hyperspectral camera using the method of line scanning to project the light of each band onto the camera chip, and the images were recorded in spatial and spectral dimensions. The samples were moved and scanned line by line by an electric translation platform under the online scanning camera. At the same time, the light source which was composed of 4 bromine tungsten lamps (ranging from 350 to 2,500 nm, total power ≥ 100 W) was used to irradiate the samples. The device structure is shown in Fig. 2. The whole device was controlled by the computer using the SpectVIEW software. The working principle is that the sample on the electric translation platform (or conveyor belt) is illuminated by the diffuse reflection source, and then the lens receives the reflected light which transmits it to the imaging spectrometer to obtain images. The sample moves with the motorized translation platform, so the continuous images and the real-time spectral information are obtained. All data are recorded by the SpectVIEW software, and finally a three-dimensional data cube which consists of the image and spectral information is obtained. The information inside or outside of the object, or other information required for sorting can be obtained by the data analysis, and full automated sorting may be realized by the subsequent control and development. The samples suitable for the system are limited to the size of 300 mm (length) × 300 mm (width) × 100 mm (height), and continuous measurement by small batches can be fulfilled.
In this study, 5 kinds of citrus leaves were used as the research object. Polymerase chain reaction (PCR) test was used to verify the feasibility of dividing the mild HLB, moderate HLB, serious HLB, deficiency and normal samples through vision. The variables were extracted by principal component analysis (PCA) and successive projections algorithm (SPA), and the sensitive wavelengths were selected for the analysis of image features. The methods of extracting image festures for the leaves based on gray level co-occurrence matrix (GLCM) were studied to explore the feasibility of detecting the citrus HLB based on image features. 2. Materials and methods 2.1. Sample preparation The changes in its leaves were most obvious after citrus infection with HLB, and citrus leaves were picked in a citrus orchard on Jiangxi Province in 2017, which were classified into the normal, HLB and malnourished visually. A total of 300 leaves, which contained 60 normal leaves, 180 HLB leaves and 60 malnourished leaves on each tree were picked from southeast to northwest. All leaves were pre-treated (washed, dried, flattened and labeled) and stored in a refrigerator at a temperature and humidity of 5 ℃ and 70%, respectively. Because of the different degrees of the HLB samples, the attempts were made to subdivide the leaves infected with HLB into mild, moderate and serious. The PCR tests were performed to verify the reliability of the visual initial division after all the hyperspectral images of the leaves were collected. The total number of failures screened out was 44, including 12 normal leaves, 12 leaves infected with mild, moderate and serious HLB and 20 malnourished leaves, and 256 leaves remained finally. Hyperspectral images were collected in the laboratory at 22 ℃ and humidity of 60%. The PCR tests were performed on all samples in conformity to national standards. OI1OI2 as Jagoueix et al and A2J5 as Hocquellet et al reported respectively were chosen as primers for tests and synthesized by with GenScript (Nanjing) Co., Ltd [14,15]. As shown in Fig. 1, the results of the former primer were more clear, and the results of infection with HLB were positive, while those of noninfection were negative. In Fig. 1, the labels of 17, 8 and 9 represented mild, moderate and serious infected leaves, respectively. They revealed bright bands successively and the light bands became heavier and heavier with the degree of severity [16].The labels of 11 and 12 also revealed bright bands, but the color of bright band was very light, which might be related to the lack of nutrients. The labels of 19 and 20 represented the normal leaves without bright band. M was used for DNA labeling and comparison in DNA gel electrophoresis to estimate the molecular weight of sample DNA. The remaining numbers represented the corresponding citrus tree labels, which were not relevant to the experiment. Table 1 below shows the specific categories of the leaves.
2.3. Image acquisition The hyperspectral system was preheated for about 30 min before the image collection. At the same time, the machine parameters were set by the SpecVIEW software, including the resolution of 1344 × 1024 pix, the spectral range of 367–978 nm, the spectral resolution of 2.8 nm, the exposure time of 10.9 ms, the moving speed of the electric translation platform of 0.4 cm/s, and the distance between objective lens by 25 cm. They were determined after the repeated tests since the latter three parameters have an effect on the distortion, size and sharpness of the image. After the parameters were set, the focusing operation was carried out and the process was that the camera lens was rotated to see 3 spectra of RGB in the focusing window after the lens was aligned with the dividing line between standard black and white plates. The corresponding focal length was the best when the sharpness of the spectra was the best, and the focusing work was finished. The sample was then tiled face up and placed on the same level with the standard whiteboard and the image acquisition was conducted. 2.4. Image calibration
2.2. Hyperspectral imaging system
After all the sample images were acquired, the black and white calibration was performed by the SpectraVIEW software to avoid the uneven distribution of the light intensity and the influence of the dark current on the images [17].The steps was to open the shutter to collect the images of the standard white board (IW) and the images of the black board reference (ID) were acquired after the shutter closed. Formula 1 displays the principle of the image calibration to switch the original image IS into calibrated image R.
The hyperspectral images of the samples were collected in the laboratory of College of Mechanical and Electrical Engineering, East China Jiaotong University. GaiaSorter hyperspectral imaging device was used in this study which includes an electric translation stage (or conveyor), the hyperspectral camera in the range of 367–978 nm, the diffuse light source, a computer and the controlling software etc. The
Fig. 1. The results of the PCR tests for citrus leaves.
104
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Table 1 The categories of the samples. The number of samples
Grade
The infection of the leaves
56 56 56 40 48
mild HLB Moderate HLB serious HLB Nutrient deficiency Normal
mild symptoms, PCR test positive Moderate symptoms, PCR test positive serious symptoms, PCR test positive No symptoms of HLB, PCR test negative No symptoms of HLB, PCR test negative
3. Results and discussion 3.1. Hyperspectral image features and spectral analysis of the citrus leaves The hyperspectral image of the leaves acquired was a three-dimensional data block containing a bunch of continuous two-dimensional images at a different wavelength. The image could be viewed as a spectrum I(λ) at each individual pixel(x、y)or a image I(x,y) at each individual wavelength λ. Visualization information of the spatial distribution was acquired at the pixel level of each image and used to analyze the distribution of HLB on the leaves. Each pixel containing the complete spectrum was used to characterize the stripping conditions. Therefore, hyperspectral image analysis and processing could be performed in the spectral or image domain [23]. Fig. 3 shows the concept of the hyperspectral images. The synthesized RGB true color image of the leaf was closest to the actual image of the leaf under the 3 channels of R: 639.84 nm, G: 548.98 nm and B: 460.5 nm. It was found that there were some differences in the original images of the HLB and normal leaves and their grayscale images at different wavelengths. The mesophyll of the HLB leaves was transparent and the distinction between veins and mesophyll was more obvious than normal leaves while the outline of normal leaves was clearer especially at 550 nm. At the same time, the average reflectance spectra of each pixel in the region of interest (ROI) were extracted and found to be apparently different at the 500–600 nm band [24]. Finally, the average reflectance spectra in the ROI of the middle rectangle on the left of the veins were selected for subsequent analysis. The representative spectra of mild, moderate, serious HLB, malnourished and normal leaves are shown in Fig. 4. It was found that the overall variation trend of spectral curves of the 5 types of leaves was the same. In the spectra, the distinct reflection band near 550 nm is assigned to chlorophyll which can be called “green peak”. The leaf reflection peak of HLB was higher than that of the malnourished and normal leaves, and the peak gradually decreased with the severity of disease. It might be that HLB pathogens hindered the absorption of water by the leaves, resulting in low chlorophyll content. The lower peak near 700 nm could depend on the number of organic molecules it contain in the leaves and be caused by the destruction of cell structure by pathogenic bacteria of HLB [24]. A steep slope in the range of 670–740 nm is close to a straight line and the reflectivity rised sharply, which can be called “red edge”. It has been increasingly and verified as an indicator of plant nutrition, growth, moisture, etc [25,26].
Fig. 2. The structure of the hyperspectral device.
R=
IS − ID IW − ID
(1)
Where R represents the calibrated image, IS represents the original image, ID represents the image of the black board reference, and IW represents the image of the standard white board. All data were analyzed by ENVI 4.5 software after the image calibration completed.
2.5. Modeling and dimension reduction method Partial least squares discriminant analysis (PLS-DA) is one of the multivariate statistical analysis methods based on PLS regression which is a bilinear modeling method and carried out by the spectral variables and classification value. The training and the credibility is usually conducted and tested according to the characteristics of the different samples. The advantage is to reduce the effects of the multicollinearity among the variables. The specific processes are as follows: (1) setting the classification value of the samples; (2) performing PLS analysis on the spectral data and the corresponding classification value to establish a regression model; (3) predicting the value YP of the unknown sample according to the model; and, (4) calculating the difference between the set value and the predicted value (δ = Y − YP) [18,19]. Due to the high dimensionality of the hyperspectral images, the noise components and irrelevant output frequency bands were separated from the images to reduce the spectral dimension. Principal component analysis (PCA) and successive projections algorithm (SPA) were selected as the dimensionality reduction methods in this study. The principle of PCA is to find a set of new orthogonal axes created by the average of the data and rotate them for maximizing the variance of the data. The number of bands corresponding to the principal component can be equal to the number of bands of the original images. However, only the first few unrelated principal components (PCs) contain most of the relevant information [20]. SPA, as a new variable selection strategy, was used in hyperspectral image analysis for multivariate calibration. Simple projection operations are used in vector space to select subsets of variables and minimize the collinearity of the variables. They can speed up the subsequent modeling based on feature image and simplify the model to improve robustness and accuracy [21,22].
3.2. Sensitive wavelength extraction and feature image analysis The SPA was used to select the sensitive wavelength variables based on the 256 data points of the average spectra in the ROI. The original variables were replaced by a few variables containing the characteristic information to simplify the subsequent model. The spectra were normalized during the operation of the SPA to facilitate subsequent data processing. Normalization is to perform data processing through a standardized algorithm and the variables can be reduced in the range of 0–1. Normalization can avoid data masking because of different dimensions, making the model more reliable and eliminating dimension effects. The normalization method is mature and widely used which is 105
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Fig. 3. The schematic diagram of the hyperspectral image in the spectral and spatial domains.
Fig. 4. Representative spectra of five kinds of leaves.
Fig. 5. The results of SPA.
suitable for a variety of data requirements. Z-score standard normalization algorithm was used in this study, by which the original data was normalized to a data set with a mean of 0 and variance of 1. The formulas are shown in 2–4.
According to the sensitive wavelengths selected by SPA, the grayscale image of the samples at the 410.29 nm were blurred and mixed with the background, so the remaining 9 wavelength points were selected for each sample. The hyperspectral image cubes of the samples at 256 data points were rich in information but difficult to process, so the image data were reduced by PCA, which could contain the information of the original data to the greatest extent and useful information was concentrated among at least a few PCs. PCA obtains new and unrelated variables by creating a new coordinate system that blocks the noise and reduces dimension. As shown in Formula 5, each PC image is a linear combination of each monochrome image.
Z=
μ=
σ=
x−μ σ
(2)
x1 + x2 +⋯+x n = n
1 n−1
n ∑i = 1 x i
n
(3)
n
∑ (xi − μ) i=1
(4)
p
PCimg =
Where x represents the variables of the original spectra, μ and σ represent the mean and standard deviation of the original data, and n represents the number of samples. The normalization method requires original data distribution to approximate the Gaussian distribution, otherwise the effect of the normalization will be worse [27]. Fig. 5 shows the results of the SPA and the selected wavelength points were 410.29、508.33、668.54、690.06、852.66、871.79、900.49、 943.53、957.87、974.61 nm.
∑ ωi Si i=1
(5)
Where wi represents the weight coefficient of wavelength, Si represents the image at the sensitive wavelength, and p represents the number of wavelengths. The weight values were calculated based on the image covariance matrix which represented the variance of each PC image [23]. The 256 variables of which the sum of the variance was kept constant were analyzed by PCA in ENVI 4.5 software and the PC images 106
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
the grayscale images of the HLB samples were darker than those of the normal leaves and the grayscale of the malnourished was the brightest in the PC1 images that contained the majority of the original data. However the grayscale of the PC1 images was high, which made it difficult to distinguish. In the PC2 images, the distinction degree between the mesophyll and veins of the HLB samples decreased with the severity of the disease gradually and the normal were the most blurred. The clarity of mesophyll tissue of the leaves decreased gradually for the first three kinds of the leaves in the PC3 images and the mild HLB was the clearest. While the absence of the normal might be due to the same color of the veins and mesophyll. Higher brightness of PC4 images might be caused by loud noise. Therefore, it was different to visually distinguish 5 kinds of leaves based on the first 4 PC images. So the PC contribution rate values were calculated to select the sensitive wavelengths. The weight coefficient is used to measure the significance of the indicator in the system, which reflects the impact on the results under certain preconditions. In this study, the weight coefficients of the first 4 PCs were estimated and analyzed, and the curve of the weight coefficients based on PCA could be drawn with 256 wavelength points in the range of 367–977 nm. 5 representative samples were selected randomly for all kinds of the leaves, and the maximum (minimum) values of the curve were analyzed to obtain the corresponding sensitive wavelengths. Table 2 shows the sensitive wavelengths selected by the above method. Therefore, the sensitive wavelengths were preferred to prepare for extracting texture features to classify 5 kinds of leaves visually [28,29]. Due to obvious differences in the grayscale images with a interval of 10 nm and above, 3 wavelengths of 700, 715 and 725 nm were selected between 700–725 nm and 870–890 nm were omitted. 14 sensitive wavelengths were finally extracted as shown in the Table. The PLS-DA model was established with the corresponding image texture features based on the sensitive wavelengths, which could help carry out visual identification and classification of citrus HLB.
Fig. 6. Cumulative contribution rates of the first 20 PCs images.
of the original band were obtained. The PC images with the order of the variance are PC1, PC2, and so on, which are the linear combination of original images and independent. The smaller is the variance, the larger of noises are contained in the PC images, and 20 PCs are usually selected. Fig. 6 shows the cumulative contribution rate of the first 20 PCs in the range of 367–977 nm. As shown in Fig. 6, the cumulative contribution rate of the first 2 PCs was 93%, and the cumulative contribution rate of the first 4 PCs reached 98%, which could explain most of the information of the original spectra. Therefore, the first 4 PCs grayscale images were analyzed for establishing the subsequent visual discrimination models. Because the sample spectra have a corresponding relationship with the images, the sensitive wavelengths need to be selected before the extraction of the texture features. Therefore, the sensitive wavelengths were selected by two ways in this study. On the one hand, the SPA was used to screen the sensitive wavelengths. On the other hand, the PCA was used to compress the variables of all the samples and the sensitive wavelengths were selected preferably. The corresponding grays images at the sensitive wavelengths were defined as the feature images for texture extraction. The feature images at the sensitive wavelengths selected by SPA were analyzed. Fig. 7 shows the grayscale images of 5 kinds of leaves including mild HLB, moderate HLB, serious HLB, malnourished and normal leaves at 508.33 nm. The highest transparency was on the leaves of the mild HLB, which decreased with the disease level gradually, so the discrimination between the mesophyll and veins also decreased. The transparency of normal mesophyll was the lowest. The edge profile of the HLB leaves decreased with the disease level gradually, but the mesophyll and veins of the nutritional deficiency were more blurred, which might be caused by the lack of the relevant nutrients in the leaves. Therefore, it was necessary to extract the grayscale texture features of the leaves deeply to visual discrimination and classification for HLB. As could be known from the above, the feature of the first 4 PCs images of samples was more specific, which reflected the majority information of 5 kind of samples. Fig. 8 shows the original images and the first 4 PCs grayscale images of the mild HLB, moderate HLB, serious HLB, malnourished and normal leaves and the PC grayscale images retained the majority of original image. Comparing the 4 PCs images,
3.3. Second-order statistic based on gray level co-occurrence matrix The image features are usually described by texture features, and the recent studies has proved that the statistical method based on gray level co-occurrence matrix (GLCM) has certain potential in extracting texture features. In this study, the texture features were obtained by calculating the secondary statistics of the GLCM. Since the shapes of the leaves were inconsistent and the capacity of the entire images were large, 8 texture features of the feature images were extracted from the same pixel size of the samples based on the GLCM, which was convenient to calculate and could improve the prediction accuracy and efficiency. The offset parameters were obtained from the directions of 0°, 45°, 90°, and 135° during the extraction. The characteristic indices were used to calculated the mean and standard deviation to suppress the directional component and make the relationship between the texture features and the direction reduce to zero [30–32]. The texture feature calculation formulas are shown in Table 3. 3.4. Visual discriminant model of citrus HLB based on GLCM 72 texture features at 9 sensitive wavelengths selected by SPA and 112 texture features at 14 sensitive wavelengths selected by PCA were used to establish the PLS-DA discriminant model of citrus HLB Fig. 7. Grayscale images of 5 kinds of leaves at 508.33 nm.
107
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Fig. 8. Original image and the first 4 PCs images by PCA for 5 kinds of leaves, (a)mild HLB. (b)moderate HLB. (c)serious HLB. (d)nutritional deficiency. (e)normal.
108
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Among the 8 characteristic parameters based on the GLCM, the parameters of mean, variance, consistency and difference had some correlations with the remaining parameters. Therefore, the PLS-DA model of HLB was built with the independent features of contrast, entropy, second moment, correlation and their combination respectively to explore the impact on performance of model with different texture features. As shown in Table 5, the recognition performance of model was more satisfactory based on the PCA weight coefficient method than SPA when single texture feature was as input. But the classification ability of model with SPA method was better than PCA when the texture features were combined as input, and the recognition ability of the combination was better than single texture feature because the useful information of single texture feature was limited. The recognition ability of the model was stronger with the texture features of second moment and correlation than with the texture features of contrast and entropy because the useful information was better represented by second moment and correlation. The recognition performance of PLS-DA model was best when the model based on SPA at the corresponding sensitive wavelengths with the combination of 4 independent texture features, resulting in lower misjudgment rate of 3.12%. The combination of 8 texture features were used to establish PLS-DA model, which could verify that the model effect was not affected by the number of texture parameters. Table 6 shows the results compared with the model with the combination of 4 texture features. As shown in Table 6, the classification effect of the model with the combination of 4 or 8 texture features by SPA was better than PCA, and the models were better with 4 independent textures than 8 textures. The classification effect of the former model was the best, which resulted in lower misjudgment rate of 3.12%, higher RP of 0.98, and lower RMSEP of 0.32. The latter model resulted in worse misjudgment rate of 28.13%. The numbers of PC factors are one of the main factors to evaluate the performance of the model. The cross validation method was used in the experiment to determine the optimal numbers of PC factors. As shown in Fig. 9(a), the RMSE of prediction and calibration decreased with the increase of the numbers of PCs. And the number of PCs was determined to be 16 because the RMSEP was the lowest. As shown in Fig. 9(b), a sample of serious HLB was misjudged as the moderate HLB and a malnourished sample was misjudged as serious HLB. The result of a moderate HLB sample was near the threshold of 1.5. The result of 2 serious HLB samples were near the threshold of 2.5 and 3.5, respectively, and the result of a normal sample was near the threshold of 4.5. These samples above were not misjudged, so the total misjudgment rate was 3.12%. The performance of discriminant model with texture features was evaluated by misjudgment rate of 64 predictive samples. The sensitive wavelengths were selected by two methods, and texture features were extracted from corresponding feature images to realize visual discrimination of non-destructive detection of citrus HLB. The performance of the model by SPA was superior to PCA, which could be due to the fact that the sensitive wavelengths with the main information of spectra but low weight coefficients were not selected. Comparing the full band, both methods could reduce dimension and optimize the model. The results showed that 9 sensitive bands selected by SPA combined with 4 texture features extracted from the GLCM were used to establish the model, resulting in the classification accuracy rate of above 95%. 9 sensitive wavelengths could be used to replace the full bands for visual expression of the image information and discrimination of citrus HLB.
Table 2 Sensitive wavelengths extracted by the weight coefficients based on PCA. category
number
Sensitive wavelength (/nm)
mild HLB moderate HLB serious HLB malnourished normal total
7 6 8 7 8 14
440、450、520、650、700、710、890 440、640、705、710、880、890 430、440、650、715、720、770、870、890 440、650、705、710、715、890、880 440、720、725、760、870、875、880、890 430、440、450、520、640、650、700、715、 725、760、770、870、880、890
Table 3 Texture features and calculation formulas. texture feature
calculation formulas
mean
Mean = ∑m, n = 0 mP (m, n)
variance
G G VAR = ∑m = 0 ∑n = 0 (m − t )2P (m, P (m, n) G G HOM = ∑m = 0 ∑n = 0 1 + (m − n)2
G
consistency
n)
contrast
CON = ∑m = 0 ∑n = 0 (m − n)2P (m, n)
difference
G G DIS = ∑m = 0 ∑n = 0 P (m, n)|m − n| G G ENT = − ∑m = 0 ∑n = 0 P (m, n) log P (m, n) G G ASM = ∑m = 0 ∑n = 0 P 2 (m, n) (m − Meanm )(n − Meann ) P (m, n) G G COR = ∑m = 0 ∑n = 0 VARm VARn
G
entropy second moment correlation
G
Table 4 Divisions of data sets. category
assignment
calibration set
prediction set
total
mild HLB moderate HLB serious HLB nutritional deficiency normal
1 2 3 4 5
42 42 42 30 36
14 14 14 10 12
56 56 56 40 58
Table 5 Results of PLS-DA model with different texture features. motheds
texture feature
Number of variable
PCs
RP
RMSEP
misjudgment rate/%
SPA
second moment contrast correlation entropy second moment,correlation contrast, entropy independent feature combination
9 9 9 9 18
2 1 1 4 8
0.34 0.29 0.80 0.79 0.86
1.28 1.31 0.84 0.84 0.69
53.12% 65.63% 59.37% 50% 21.88%
18 36
7 16
0.75 0.98
0.90 0.32
43.75% 3.12%
second moment contrast correlation entropy second moment,correlation contrast, entropy independent feature combination
14 14 14 14 28
3 1 1 13 17
0.65 0.58 0.83 0.85 0.87
1.0 1.09 0.77 0.67 0.66
46.88% 53.12% 50% 34.37% 28.13%
28 56
14 14
0.87 0.87
0.70 0.63
46.88% 28.13%
PCA
4. Conclusion
respectively. The samples were divided into two parts, the calibration and prediction sets (ratio of 3:1). The former contained 192 leaves including 126 HLB leaves, 30 malnourished leaves and 36 normal leaves, and the latter contained 64 leaves. The divisions of data sets are shown in Table 4.
In this study, the hyperspectral images of citrus leaves were transformed into the first 4 PC images by PCA. The 14 sensitive bands with high weight coefficients were further obtained by analyzing the PC images, and the SPA method was used to select 9 sensitive wavelengths 109
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Table 6 Results of PLS-DA model with the texture features of different numbers. screening methods
The number of feature
The number of variables
PCs
RC
RP
RMSEC
RMSEP
misjudgment rate/%
SPA
4 8
36 72
16 19
0.98 0.96
0.98 0.90
0.28 0.38
0.32 0.62
3.12% 18.75%
PCA
4 8
56 112
14 18
0.94 0.92
0.87 0.83
0.46 0.55
0.63 0.75
28.13% 28.13%
Fig. 9. The results of best PLS-DA model, (a) the optimal PCs number. (b) the scatter diagram of model.
based on the spectral domain. The texture features of mean, variance, consistency, contrast, difference, entropy, second-order moment and similarity on the corresponding feature images were counted, and 4 independent feature parameters were selected to establish the PLS-DA classification recognition model of HLB. The PLS-DA model based on the combination of 4 independent texture features from the images at the sensitive wavelengths by SPA had better effect by comparing the performance of all the model, which resulted in lower misjudgment rate of 3.12%. The results proved that the feature bands could replace full bands to describe the image information of the samples at which the image information could be used for visual description and realizing the identification of HLB.
[6] M.L. Adams, W.D. Philpot, W.A. Norvel, Yellowness index: an application of spectral second derivatives to estimate chlorosis of leaves in stressed vegetation, Int. J. Remote Sens. 20 (1999) 3663–3675. [7] G.f. Tang, Z.f. Xiao, Q. Liu, H. Liu, A novel airport detection method via line segment classification and texture classification, IEEE Geosci. Remote Sens. Lett. 12 (2015) 2408–2412. [8] R. Rastghalam, H. Pourghassem, Breast cancer detection using MRF-based probable texture feature and decision-level fusion-based classification using HMM on thermography images, Pattern Recognit. 51 (2016) 176–186. [9] S. Murala, Q.M.J.W. Jonathan, Spherical symmetric 3D local ternary patterns for natural, texture and biomedical image indexing and retrieval, Neurocomputing 149 (2015) 1502–1514. [10] H. Murano, Y. Takata, T. Isoi, Origin of the soil texture classification system used in Japan, Soil Sci. Plant Nutr. 61 (2015) 688–697. [11] J.R. Jensen, K. Lulla, Introductory digital image processing: a remote sensing perspective, Geocarto Int. 2 (1987) 65-65. [12] P. Liu, Research on Ovean Oil Spill Detection and Recognition Using SAR Data, Ocean University of China, Shandong, 2012, https://doi.org/10.7666/d.y2158798. [13] J.Z. Lu, Plant Leaves Diseases Detection Using Spectral Imaging Technology, Zhejiang University, Hangzhou, 2016. [14] T. Li, Ke Chong, Detection of the bearing rate of Liberobacter asiaticum, in citrus psylla and its host plant Murraya panciculata by Nested PCR, Acta Phytophylacica Sin. 29 (2002) 31–35. [15] A. Hocquellet, P. Toorawa, J.M. Bové, M. Garnie, Detection and identification of the two Candidatus liberobacter species associated with citrus huanglongbing by pcr amplification of ribosomal protein genes of theβ operon, Mol. Cell. Probes 13 (1999) 373–379. [16] I. Michael, Doing it faster and smarter (Lesson 6 of Matrix Algebra), Spectrosc. Europe 14 (2002) 24–26. [17] L. Yu, Y.S. Hong, Y. Zhou, Q. Zhu, L. Xu, J.Y. Li, Y. Nie, Wavelength variable selection methods for estimation of soil organic matter content using hyperspectral technique, Trans. Chin. Soc. Agric. Eng. (Trans. CSAE) 32 (2016) 95–102. [18] Z. Cheng, L.Q. Zhang, H.Y. Liu, A.S. Chu, Successive projections algorithm and its application to selecting the wheat near-infrared spectral variables, Spectrosc. Spectr. Anal. 30 (2010) 949–952. [19] B. Chen, X.L. Meng, H. Wang, Application of successive projections algorithm in optimizing near infrared spectroscopic calibration model, Fenxi ceshi xuebao 26 (2007) 66–69. [20] P. Baranowski, W. Mazurek, J. Wozniak, U. Majewska, Detection of early bruises in apples using hyperspectral data and thermal imaging, J. Food Eng. 110 (2012) 345–355. [21] X. Zhou, Implementation and comparison of two classifiers based on LS-SVM algorithm, Comput. Knowl. Technol. 7 (2011) 7281–7283. [22] Z. Yang, H.Q. Ren, Z.H. Jiang, Discrimination of wood biological decay by NIR and partial least squares discriminant analysis (PLS-DA), Spectrosc. Spectr. Anal. 28 (2008) 793–796. [23] B.H. Zhang, S.X. Fan, J.B. Li, W.Q. Huang, C.J. Zhao, M. Qian, L. Zheng, Detection of early rottenness on apples by using hyperspectral imaging combined with spectral analysis and image processing, Food Anal. Methods 8 (2015) 2075–2086.
Conflicts of interest There are no conflicts to declare. Acknowledgments The research was funded by: National Natural Science Foundation of China (No. 31760344), Center of the Technology and Equipment of the Intelligent Management for the Southern Mountain Orchard Collaborative Innovation (No. 2014-60). Jiangxi Advantage Science and Technology Innovation Team Construction Project (No. 20153BCB24002). References [1] A.R. Mishra, D. Karimi, R. Ehsani, W.S. Lee, Identification of citrus greening (HLB) using a VIS-NIR spectroscopy technique, Trans. ASABE 55 (2012) 711–720. [2] M.C. do Brasil Cardinali, P.R.V. Boas, D.M.B.P. Milori, E.J. Ferreira, M.F. e Silva, M.A. Machado, B.S. Bellete, Infrared spectroscopy: a potential tool in huanglongbing and citrus variegated chlorosis diagnosis, Talanta 91 (2012) 1–6. [3] X.H. Li, M.Z. Li, S.L. Won, R. Ehsani, Ashish, R. Mishra, Visible-NIR spectral feature of citrus greening disease, Spectrosc. Spectr. Anal. 34 (2014) 1553–1559. [4] L.Q. Lei, C.Y. Guan, Advances in the application of agricultural spectral digital techniques in crop information monitoring, Crop Res. 25 (2012) 626–629. [5] Y. Wu, H.N. Su, A.J. Huang, Y. Zhou, Z.A. Li, J.X. Liu, C.Y. Zhou, Effect of Candidatus Liberibacter asiaticus infection on carbohydrate metabolism in Citrus sinensis, Sci. Agric. Sin. 48 (2015) 63–72.
110
Vibrational Spectroscopy 102 (2019) 103–111
Y. Liu, et al.
Mach. 49 (2018) 226–232. [29] Z.X. Zheng, L. Qi, X. Ma, X.Y. Zhu, W.J. Wang, Grading method of rice leaf blast using hyperspectral imaging technology, Trans. Chin. Soc. Agric. Eng. 29 (2013) 138–144. [30] P.P. Jiao, Y.Z. Guo, L.J. Liu, X. Wei, Implementation of gray level co-occurrence matrix texture feature extraction using matlab, Comput. Technol. Dev. 22 (2012) 169–171. [31] Ma, H.B. Pu, D.W. Sun, W.H. Gao, J.H. Qu, K.Y. Ma, Application of vis–NIR hyperspectral imaging in classification between fresh and frozen-thawed pork Longissimus dorsi muscles, Int. J. Refrig. 50 (2015) 10–18. [32] N. Shao, H. Zhou, L.J. Jiang, Y.D. Bao, Y. He, Using reflectance and gray-level texture for water content prediction in grape vines, Trans. ASABE 60 (2017) 207–213.
[24] X.H. Li, W.S. Lee, M.Z. Li, R. Ehsani, A.R. Mishra, C.H. Yang, R.L. Mangan, Spectral difference analysis and airborne imaging classification for citrus greening infected trees, Comput. Electron. Agric. 83 (2012) 32–46. [25] Y.W. Huang, J.H. Wang, X.Y. Li, J.S. Jacqueline, D.H. Han, Research on fast discrimination between Panax ginseng and Panax quinque folium based on near infrared spectroscopy, Spectrosc. Spectr. Anal. 30 (2010) 2954–2957. [26] Z.J. Shan, Z. Feng, G. Zhou, X.L. Deng, Cloning and sequencing of Hang longbing pathogen in shatianyou pomelo, J. Zhongkai Univ. Agric. Technol. 18 (2015) 45–48. [27] X. Peng, S.W. Ye, Z. Shao, H.X. Cai, J. Shi, Y. Tan, J. Li, Recognition of space debris based on spectra and probabilistic neural network, J. Changchun Univ. Technol. 36 (2015) 395–400. [28] Y. Lei, D.J. Han, Q.D. Zeng, D.J. He, Grading method of disease severity of wheat stripe rust based on hyperspectral imaging technology, Trans. Chin. Soc. Agric.
111