- Email: [email protected]
LeafSpec: An accurate and portable hyperspectral corn leaf imager
Computers and Electronics in Agriculture 169 (2020) 105209
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag
LeafSpec: An accurate and portable hyperspectral corn leaf imager a
a,⁎
a
b
a
T a
Liangju Wang , Jian Jin , Zhihang Song , Jialei Wang , Libo Zhang , Tanzeel U. Rehman , Dongdong Maa, Neal R. Carpenterc, Mitchell R. Tuinstrac a b c
Department of Agricultural and Biological Engineering, Purdue University, West Lafayette, IN 47907, United States School of Mechanical Engineering, Purdue University, West Lafayette, IN 47907, United States Department of Agronomy, Purdue University, West Lafayette, IN 47907, United States
A R T I C LE I N FO
A B S T R A C T
Keywords: Handheld sensor Crop leaf scanner Plant phenotyping Hyperspectral imaging LeafSpec
Hyperspectral imaging (HSI) technology has been widely applied in industry and academia for plant phenotyping. However, most HSI systems are large and expensive, making it challenging to benefit more people. The overall goal of this study was to develop a portable and low-cost hyperspectral imaging handheld device (named LeafSpec) with even improved measurement quality compared with traditional HSI systems for crop leaves imaging. The hardware of LeafSpec device was comprised of a push-broom hyperspectral camera (HSC), leaf scanner with an encoder system for leaf position information, a lightbox as an intensive and uniform beam lighting source, and an ARM-based microcontroller. In each scanning, a smooth and clear hyperspectral image of the entire leaf was obtained by quickly sliding LeafSpec across the leaf from the beginning to tip. Each measurement was geo-referenced by sending processed data to a smartphone and combining it with the GPS location and time information before uploading to a Geography Information System (GIS) with Digital Ag Map Services and internet connection. After calibration, the HSC’s imaging results were highly consistent with a commercialized hyperspectral camera. In the field test in the summer of 2018, LeafSpec was able to detect the difference between two nitrogen treatments of corn plants in each genotype, as well as the differences between three genotypes in high nitrogen treatment, and the difference between two genotypes in low nitrogen treatment before it was visible to human eyes. In the greenhouse test in the spring of 2019, LeafSpec predicted nitrogen content and relative water content with R2 of 0.880 and 0.771, RMSE of 0.265 and 0.049, respectively. Generally, LeafSpec is an easy-to-use and low-cost crop phenotyping sensor with improved measurement accuracy, which could benefit more people in plant science research and agriculture production.
1. Introduction The world population will reach 9 billion by 2050, and food production must increase by 70% to feed more people (Godfray et al., 2010). To ensure that high yielding and stress-tolerant plants can be selected rapidly and efficiently, connecting the genotype with the phenotype robustly is more important than before (Li et al., 2014). Hyperspectral Imaging (HSI) technology has been explored and applied in plant phenotyping as both spatial and spectral information are obtained in a high-throughput and non-invasive way (Gowen et al., 2007; Li et al., 2014, 2013). HSI was originally defined and used in remote sensing in the 1980s’ (Goetz et al., 1985; Goetz, 2009). A typical field HSI system configuration involves mounting a hyperspectral camera (HSC) on an unmanned aerial vehicle (UAV) or satellite to image the crop canopy at field level with high-throughput under ambient light
(Adão et al., 2017; Bareth et al., 2015; Zarco-Tejada, González-Dugo, & Berni, 2012; Goetz, 2009). Alternatively, a ground vehicle or gantry is used to maneuver the HSC to image crop plants in closer proximity than UAV or satellite (Gutiérrez et al., 2018; Gutiérrez et al., 2019; Suomalainen et al., 2014; Vigneau et al., 2011; Virlet et al., 2017; Ravi et al., 2018). The ground vehicle or gantry helps obtain higher resolution images of plants from the side view or the top view and makes it possible to analysis plants at the individual plant level. To reduce the affection of ambient light, artificial light could be installed on the imaging system. In greenhouse applications, the plant is placed on an imaging platform manually or automatically with conveyor and then imaged by HSC (Ge et al., 2016; Pandey et al., 2017; “Purdue University Controlled Environment Phenotyping Facility,” n.d.). Another imaging method for greenhouse applications is mounting an HSC on a large scale gantry system to image still plants (Zhang et al., 2016). When the
⁎ Corresponding author at: Department of Agricultural and Biological Engineering, Purdue University, 225 South University Street, West Lafayette, IN 47907, United States. E-mail address: [email protected] (J. Jin).
https://doi.org/10.1016/j.compag.2019.105209 Received 3 March 2019; Received in revised form 18 October 2019; Accepted 31 December 2019 0168-1699/ © 2020 Elsevier B.V. All rights reserved.
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 1. Handheld hyperspectral imaging scanner (LeafSpec). (a) LeafSpec’s photograph. (b) LeafSpec’s brief structure.
In recent years, some customized HSCs have been developed. A recent study conducted by Sigernes et al. (2018) discussed the basic mechanism of developing a small, lightweight, and cheap push broom HSC for handheld to airborne operations. A compact HSC equipped on a motorized selfie stick to perform remote sensing was developed (Chen et al., 2018). These customized HSCs were not designed to image a leaf sample within ten centimeters, and they were also not equipped with high dynamic range imaging sensors. The development of a new portable hyperspectral corn leaf imager, LeafSpec, with a customized new HSC is discussed in this paper. A nonprovisional patent on the hardware and software design of LeafSpec was filed by Purdue research Foundation. LeafSpec has been successfully tested in both Purdue’s research farm fields and greenhouses, and has been commercialized from 2018.
operation space in a greenhouse is enough for a large-scale imaging system and a plant moving system, the plant can be imaged at the plant level and even at leaf level. In an indoor laboratory, either part of a plant, or a small plant like Arabidopsis can be imaged while placed on the imaging system platform with uniform light. Although HSI has demonstrated the capability of predicting plant physiological features, the signal quality is compromised with various noise factors. For remote sensing in the field, the HSI signal is severely impacted by daylight change and weather conditions. For the greenhouse imaging towers, although the ambient light can be blocked by the imaging tower, the artificial imaging lights still have complicated interactions with the 3D plant canopies. The plant leaves are often at different distances from the camera and lights, with various angles of the leaf surfaces. This imaging method causes non-uniform lighting intensities on the leaf surfaces, which is difficult to calibrate with current referencing technologies. According to the PROSAIL model, the spectrum reflected from the leaf surface is significantly impacted by the leaf angle resulting in inconsistency in colors detected (Jacquemoud et al., 2009). These noise factors in remote-sensing or medium-distancesensing do not exist for touch-based imaging sensors. For example, a handheld imaging system with an enclosed imaging chamber and the fixed configuration between light, camera, and object could guarantee the fixed imaging distance, angle, and lighting intensity on the object. Portable spectrometer device, which obtains the spectrum of a spot on the leaf, is applied in plant phenotyping to estimate fresh leaf status in the field. One of these portable spectrometers, ASD FieldSpec 4 (Malvern Panalytical Ltd., Malvern, UK) covers the full visible-near infrared-shortwave infrared (VSWIR; 400–2500 nm) and is widely used for phenotyping of leaf physiological and biochemical traits in plant phenotyping research (Cotrozzi et al., 2018; Ge et al., 2019; Yendrek et al., 2017; M. Yuan et al., 2016). However, the measurement at one small local spot on the leaf does not represent the whole leaf well due to the significant variance of leaf color or spectrum across the leaf (Ciganda, Gitelson, & Schepers, 2008; Z. Yuan et al., 2016). It has also been reported that the variance and distribution of color on the leaf contains useful information for improved phenotyping quality. Gray spots on the leaf caused by a type of fungal disease of corn can be diagnosed with leaf’s color distribution (Bubeck et al., 1993; Ward et al., 1999). The N-P-K deficiencies of rice can be identified with about 90% accuracy with the leaf images (Chen et al., 2014). Leaf color distributions are visibly different among different nutrient deficiencies (Kumar and Sharma, 2013). Therefore, taking the entire leaf image with a handheld touch-based HSC has the potential to enable to development of new algorithms on nutrition and stress distribution analysis, which could provide higher measurement quality. In terms of price, volume and weight, existing HSC on the market does not satisfy the requirement of a handheld device (Li et al., 2014).
2. LeafSpec device development The LeafSpec device took the hyperspectral image of the entire leaf by smoothly sliding the leaf through its imaging chamber. The imaging process took within three to five seconds for each leaf scanning, and the video of the process was available (https://www.youtube.com/watch? v=mPvhV1J8BG8). The raw HS images and the imaging processing results (plant physiological features predictions) were stored in the memory of the device, which could be easily downloaded after the data collection. If the user has a smartphone, a smartphone APP was developed and available to view the images and prediction results on the spot in real-time. Although the device was self-contained, the data could optionally be uploaded by the smartphone APP together with the geo-location and time of the measurement to Purdue’s Ag Geography Information System (GIS). The GIS provides various digital Ag map viewing functions for the phenotyping measurements in real-time when the measurements were being collected. The design of the device, including the general structural design and specific design details of each part in both hardware and software, is included in the non-provisional patent application (Patent No. PCT/ US2018/027953, 2018). 2.1. Hardware configuration of LeafSpec The total weight of LeafSpec (Fig. 1(a)) was about 2 kg with length, width, and height of 230 mm, 150 mm and 300 mm, respectively. The hardware of LeafSpec consisted of four main components (Fig. 1(b)): an HSC, a scanning mechanism, a lightbox, and an ARM® based microcontroller. The lightbox generated a uniform full spectrum beam light. The scanning mechanism equipped with a roller-encoder system was used to detect the leaf motion in a non-invasive way accurately. The HSC collected spectra and spatial information of the leaf. The 2
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 2. LeafSpec’s hyperspectral camera. (a) Camera’s structure. (b) Camera’s photograph.
458 nm to 1650 nm was selected for the first version of the device to filter out the second-order diffraction pattern of less than 440 nm.
microcontroller obtained and analyzed data from the camera and other sensors, and then transferred the result data to a smartphone. 2.2. Hyperspectral camera
2.2.2. Slit and lens Slit plays a pivotal role in the final spectral resolution by determining the amount of light to be imaged. A slit (S50RH, Thorlabs Inc., Newton, NJ, USA) was selected for this device. Its physical dimensions were 50-μm-wide with ± 1 μm tolerance and 3-mm-long. The collimating lens was a convex lens to parallel the light exiting through the slit on the grating. The focusing lens was also convex, but its purpose was to focus the light exiting through the grating on the image sensor. An achromatic doublet lens consisting of two components cemented together was optimized to correct for on-axis spherical and chromatic aberrations (Hecht, 2016). Two achromatic doublets (#63720, Edmund Optics Inc., Barrington, NJ, USA) were selected as a collimating lens and a focusing lens, respectively. The focal length of these two achromatic doublets was 30 mm with ± 2% tolerance and diameter was 10 mm. To make the device smaller and decrease the distance between the collecting lens and imaging sample, a wide-angle lens was required. A lens (ZWO-LENS-2.5, High Point Scientific, Montague, NJ, USA) with 2.5 mm focus length and f/1.2 aperture was mounted at the front of the slit with CS-mount as the collecting lens in the HSC.
The structure of LeafSpec’s hyperspectral camera is presented in Fig. 2. It was consisted of collecting lens, slit, filter, collating lens, grating, focusing lens, and imaging sensor. The light was collected and focused on the slit by the collecting lens. The collating lens collated a line beam passing through the slit. The collated light was spread by the grating and then focused on the imaging sensor by the focusing lens. A detailed explanation of different components is provided in the sections below. 2.2.1. Diffraction grating and filter One of the essential parts in the HSC was the diffraction grating, whose function was to separate polychromatic light into constituent monochromatic lights showing on different positions of an image plane (Palmer and Loewen, 2005). Transmission grating offered low alignment sensitivity, which decreased the requirements for machining tolerance (Heilmann et al., 2009; Yang and Wang, 2018). A visible transmission grating (GT13-03, Thorlabs Inc., Newton, NJ, USA) was selected for the first version of this device. Its physical dimensions were 12.7 mm × 12.7 mm (Length × Width) and had 300 grooves per millimeter with a groove angle of 17.5°. The transmission efficiency of the selected grating was in a range of 38%-74% for 450 nm to 900 nm spectral range. To obtain the spectra from 450 nm to 900 nm, the angle of the diffracted monochromatic light was calculated to be 11.35° by using Eq. (1).
nλ = d (sinθ + sinθ')
2.2.3. Imaging sensor For HSI, quantum efficiency, dark noise, and saturation capacity were important specifications for determining the imaging quality. A monochrome camera (BFLY-U3-05S2M-CS, FLIR Integrated Imaging Solutions Inc., Richmond, BC, Canada), based on Sony CCD sensor ICX693, was selected as the imaging sensor. Its specifications are listed in Table 1. Its power consumption was less than 3 W and could be triggered with general purpose input output (GPIO) of a controller to take images. Its USB 3.1 port worked not only as a power supply port but also as a data port to transfer the image to the controller.
(1)
where n is the diffraction order, λ is the wavelength of the diffracted monochromatic light, d is groove spacing of the grating, θ is the groove angle and θ' is the diffracted angle of monochromatic light. A long-pass filter (62–975, Edmund Optics Inc., Barrington, NJ, USA) with over 91% transmission efficiency in the spectral range 3
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
and the surface inside was machined to a low roughness level with the mirror-like surface to reflect most of the light. The aluminum material was also a good thermal conductor to dissipate the heat generated by the halogen bulbs.
Table 1 Specifications of the imaging sensor. Parameters
Corresponding settings
Camera model Resolution Frame rate Chroma Sensor name Sensor size (mm) Pixel size Shutter type ADC Quantum efficiency (% at 525 nm) Temporal dark noise (e-) Saturation capacity (e-) Dynamic range (dB) Exposure range Interface Power requirements Dimensions (mm) Mass (g)
BFLY-U3-05S2M-CS 808 × 608 50 FPS Mono Sony ICX693 4.85 × 3.65 6.0 µm Global 12 bit 81 9.51 22,074 66.87 0.031 ms–31.9 s USB 3.1 Gen 1 5–24 V via GPIO or 5 V via USB3 29 × 29 × 30 36
2.5. Microcontroller To control the cost and improve the development efficiency, an ARM board was adopted for LeafSpec’s microcontroller. The controller acquired images from the camera through the USB port, position information from the encoder, and other switch signals, and then processed them. It then saved and sent the processed results to a smartphone through a Bluetooth® adapter. Due to the image’s high speed acquiring and processing requirements, ODROID-XU4 (Hardkernel Co., Ltd, Gyeonggi, South Korea) was selected since it had more processing power than the commonly used ARM board Raspberry Pi. The board with a dimension of 83 × 58 × 20 mm was based on Samsung Exynos5422 with 2 GHz (GHz) Cortex™-A15 quad-core and 1.2 GHz Cortex™-A7 quad-core CPUs. It contained 2 gigabytes (GB) LPDDR3 RAM at 933 MHz (MHz), supported eMMC5.0 flash storage and had 3 USB ports and a 30-pin GPIO port. Armbian, a Debian based operating system with Linux kernel 4.9 LTS, was running on the board. The programming language was Python 2.X.
2.3. Leaf scanning mechanism As LeafSpec’s HSC was a push-broom type camera, it imaged only one line of the leaf sample in one shot (Li et al., 2013). To image an entire leaf, the HSC scanned the leaf by moving along the leaf’s midrib direction. The scanning motion was synchronized with the frame acquisition rate of the camera to ensure a smooth integrated image. Imaging of each line was triggered at a specified distance.
2.6. Power supply A suitable battery was important for LeafSpec because of the halogens’ high-power consumption but the device needed to be lightweighted for people to carry. A portable power bank (YB12011000USB, TalentCell, Guangdong, China) with 12 V/11,000 mAh capacity was selected as the power supply. A stable voltage for bulbs during working time to ensure a steady spectrum was a key factor for HSI. A 12–5 V with 3 A and a 12–12 V with 2 A voltage regulators were equipped for the controller and bulbs in the power supply system. To reduce the volume of LeafSpec, the power bank was installed in LeafSpec after its shell was removed. When the inside battery was out of power or consumed, another back-up power bank could be connected to LeafSpec.
2.3.1. Leaf motion detection method A displacement sensor can be installed to acquire the position information of LeafSpec along the leaf during scanning. The sensed information can then be used to determine when to trigger the camera to capture an image. To avoid damaging the leaf, the sensor either should not press on the leaf too firmly or not make contact with the leaf. An optical flow sensor used in a computer’s mouse could measure the movement without touching the leaf surface; however, any change in the spacing between sensor and leaf during leaf scanning may cause low positioning accuracy. The variation in leaf width caused the movement to be undetectable. A rotary encoder was conventionally applied in industrial applications because of its accuracy and high-resolution in most cases. The challenge in the development of this device was to rotate the encoder along a leaf smoothly without damaging the leaf or missing any movement. To solve this problem, the leaf movement measuring mechanism was designed as Fig. 1(b) shown. A roller connected to an encoder and covered with rubber was pressed on the leaf. The encoder driven by the roller outputs the movement signal to the microcontroller; this provides the position information used to trigger the camera.
2.7. Data flow and management The structure of data flow between different components in LeafSpec system is presented in Fig. 3. The microcontroller acquired images from the HSC, and position information from encoder. A list of line images was constructed into one hyperspectral image, then segmented and processed to get leaf parameters. The leaf’s morphological
2.3.2. Seal method To avoid any damage to the leaf, a flexible sealing method was developed for LeafSpec. Black soft and dense cloth strips were attached around the imaging platform (Fig. 1(a)). The height of the sample room was limited within 5 mm so that the force produced by the deflection of the sealing cloth pressed itself on the leaf. Ambient lighting was blocked off, as shown in Fig. 1(a). 2.4. Light source A steady broad-spectrum light source was required for HSI. Halogen bulb was a low-cost but stable option and selected as the light source for LeafSpec. To generate uniform light on a narrow and long window, a white strip made with white Teflon material was placed to reflect and diffuse the light. The light box was machined with aluminum material,
Fig. 3. The data flow and communication in LeafSpec system. 4
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 6. The color checkerboard. Checker panels in red rectangle were imaging samples. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 4. The spectral lines image of krypton light source by LeafSpec’s HSC. Table 2 Parameters for the hyperspectral imaging test.
Table 3 The wavelength and column indices of spectral lines.
Parameters
MSV
LeafSpec’s HSC
Camera model Spectrograph Frame rate (FPS) Exposure time (ms) Spectral resolution Spatial resolution Spectral range (nm) Scan speed (mm/s)
acA780-75gm SpecIM V10H 20 15 582 782 362–1230 5.08
BFLY-U3-05S2M-CS Customized 20 50 676 500 450–900 5.08
Note: MSV, a commercialized hyperspectral camera. HSC, hyperspectral camera.
Peak index
Wavelength (nm)
Column (pixel)
Fitting error (nm)
1 2 3 4 5 6 7 8 9 10 11
557.029 587.096 760.155 769.454 785.482 810.436 819.006 829.811 850.794 877.675 892.869
158 204 460 475 500 538 550 566 598 639 662
−0.441 0.624 −0.811 −0.140 0.428 0.649 0.017 −0.214 −0.082 0.027 −0.057
3. Calibration and tests of the new sensor device 3.1. Spectra calibration of LeafSpec’s HSC For any hyperspectral imager, the imaging result needs to be calibrated with a wavelength standard to precisely label each band with the actual wavelength. A frame (Fig. 4) obtained by the image sensor contained the spatial and spectral information of one-line on the object through the slit. The spectral information of the frame was distributed along the X direction (columns), while the spatial information was distributed along the Y direction (rows). A calibration light source (KR1, Ocean Optics Inc, Largo, FL, USA) with a krypton bulb produced spectral lines from 427 nm to 893 nm. The range covered the spectra calibration requirements of LeafSpec’s HSC whose wavelength range was 450 nm to 900 nm. After imaging the krypton light source (Fig. 4), the corresponding wavelengths of these bright lines in the image were obtained according to the known spectrum of krypton light. The wavelength corresponding to each pixel of the camera with its column index was calculated using the column indices of the bright lines and their corresponding wavelengths (Eq. (2)). Fig. 5. The Hyperspectral imaging test system for LeafSpec.
λp = I + C1 P + C2 P 2 features, including width, length, and area of the leaf, were calculated based on the position information and the image. LeafSpec also provided application interface for any plant physiological feature predicting model based on the leaf’s hyperspectral image or averaged spectrum. The device stored all raw data and results locally on an SD memory card. A selected list of results including the NDVI heatmap, spectrum, and plant physiological parameters were transferred to a smartphone APP through Bluetooth®. The user can choose to upload these leaf parameters and GPS information from smartphone to a GIS server through an internet connection. The raw data could also be downloaded directly from LeafSpec’s storage to PC for further analysis.
(2)
where P is the column index, λp is the wavelength of the column P, I is the wavelength of the column 0, C1 and C2 are the coefficient of the polynomial. 3.2. Hyperspectral imaging performance test The imaging quality of LeafSpec’s HSC was examined by comparing it with a popular commercialized HSC MSV (MSV-101-W, Middleton Spectral Vision, Middleton, WI, USA). The MSV consisted of an imaging sensor (acA780-75gm, Basler AG, Ahrensburg, Germany) and a spectrograph (V10H, SpecIM Spectral Imaging Ltd, Oulu, Finland). The key parameters of these two HSC are compared in Table 2. The two cameras 5
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 7. The RGB images of a color checkerboard. (a) RGB image from a commercialized hyperspectral camera MSV. (b) RGB image from LeafSpec’s hyperspectral camera.
Fig. 8. Comparison of spectra from LeafSpec’s hyperspectral camera and MSV. LC, LeafSpec’s hyperspectral camera. MSV, a commercialized hyperspectral camera. The curve is the median of all pixel’s intensity at each wavelength. The outliers are between lower and upper quartiles of all pixel’s intensity at each wavelength. To display variance clearly, the lower and upper quartiles were multiplied by 10.
6
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
watered, 16 plants with high N and drought-stressed, 11 plants with low N and well-watered, and 16 plants with low N and droughtstressed. The high and low N treatments were achieved with 120 ppm and 25 ppm nitrogen fertilizers, respectively. The well-watered plants were irrigated as needed, whereas, we stopped irrigating the droughtstressed plants a few days before imaging. When irrigating, we guaranteed that each pot achieved saturation. The experiment was carried out in the greenhouse on March 19, 2019, from 12:30 PM to 2:30 PM. Corn plants were at V7-V8 stage. LeafSpec was used to scan the top collared leaf of each plant from the start to the tip. The smartphone, Huawei® Mate 20 (Huawei Technologies Co., Ltd., Shenzhen, Guangdong China) with Android 9.0 operating system, was connected with LeafSpec through Bluetooth® for result visualization. A piece of leaf sample of each plant was cut off and weighed to obtain leaf fresh weight (FW) after measured by LeafSpec. Then the sample was immediately hydrated until it was completely turgid under normal room lighting and temperature. Leaf sample was then weighed to obtain the fully turgid weight (TW) after its surface water was removed lightly and quickly with filter paper. Finally, the leaf sample was fully dried with the dry oven (persistently at 60 °C) for approximately 24 h to obtain the dry weight (DW). RWC was computed by using the relationship (Eq. (3)):
Table 4 Comparative performance of LeafSpec’s HSC and MSV on the imaging stage. Color of Checkerboard
Intensity Std. LeafSpec’s HSC
Intensity Std. MSV
Intensity Mean Absolute Distance
Intensity Maximum Absolute Distance
Intensity Mean Relative Distance
Cyan Magenta Yellow Red Green Orange yellow Yellow green Purple Moderate red Purplish blue Average
11.742 17.777 19.654 14.579 12.408 18.339 16.365 13.924 18.345 14.336 15.747
25.617 35.846 39.374 29.826 26.288 33.389 30.702 25.226 30.880 27.372 30.452
26.304 41.202 30.991 53.108 45.277 38.358 43.784 58.563 34.271 62.386 43.424
107.271 217.858 283.175 271.982 264.053 314.277 286.847 325.895 290.450 256.334 261.814
0.040 0.029 0.023 0.078 0.044 0.039 0.033 0.090 0.038 0.060 0.047
Note: HSC, hyperspectral camera. MSV, a commercialized hyperspectral camera. Std., standard deviation. Intensity mean absolute distance is the mean value of the coupled spectra’s difference. Intensity mean relative distance is the mean value of the ratio between the coupled spectra’s difference and MSV’s spectrum.
were tested using a linear hyperspectral scanning stage (MRC-920-044, Middleton Spectral Vision, Middleton, WI, USA) equipped with halogen light source (Fig. 5). A color checkerboard (MSCCC, X-Rite Inc., Grand Rapids, MI, USA) was fixed on the sampling stage as the imaging object (Fig. 6). When the color checkerboard was linearly moved by the stage, the two HSCs imaged the color panels inside the red rectangle (see Fig. 6 for reference) simultaneously. A PVC white strip was imaged by both HSCs as white reference to calibrate the hyperspectral images. After calibration, RGB images were extracted from hyperspectral data. The spectrum of each checker panel was obtained with the median intensities of all pixels inside that panel. Finally, the image quality and spectral quality were compared between the two cameras.
RWC (%) =
FW − DW × 100% TW − DW
(3)
Nitrogen content of all collared leaves of each plant was analyzed in the Agronomy Department at Purdue University to analyze the nitrogen content using a FlashEA 1112 Nitrogen and Carbon Analyzer (Thermo Fisher Scientific, Waltham, MA USA). We obtained the averaged spectrum of each leaf sample with LeafSpec. The partial least square regression (PLSR) model was applied to these spectra to predict N content and RWC of each plant. Forty-one samples were used to calibrate the model, and 18 samples were used to test the model. Leave-one-out cross-validation method was introduced to limit problems like overfitting.
3.3. Corn leaf imaging test in the field A field test with corn plants was conducted on July 17, 2018, in Agronomy Center for Research and Education field #9D of Purdue University (40°28′14.3″N, 86°59′40.2″W, 4540 US-52, West Lafayette, IN 47906). Three genotypes including B73xMo17, CML550xPHP02, and P1105AM (annotated as A, B, and C, respectively hereafter) were planted in 24 plots. Twelve plots were treated as high (H) nitrogen group (250 kg/ha), and the remaining twelve were treated as low (L) nitrogen group (0 kg/ha). For each genotype and treatment combination, there were four replicate plots. Eight plants were randomly selected from each plot, so in total 192 corn plants were imaged at the V8 stage. LeafSpec was used to scan the top collared leaf of each plant from the start to the tip. The smartphone, Huawei® Mate 9 (Huawei Technologies Co., Ltd., Shenzhen, Guangdong China) with Android 8.0 operating system, was connected with LeafSpec through Bluetooth® for result visualization. The hyperspectral raw data was saved on the local storage of the LeafSpec. The NDVI heatmaps and leaf parameters of each leaf were sent to the smartphone. Finally, these parameters were uploaded to the cloud GIS database with GPS location and timestamp information. In this study, all NDVI values were calculated with equation NDVI = (b850 − b680)/(b850 + b680) , in which b850 and b680 were the intensities at wavelength 850 nm and 680 nm, respectively.
4. Testing results and discussions 4.1. Spectra calibration result With Eq. (2) and Table 3, the wavelengths regression function is shown in Eq. (4) for the tested LeafSpec’s HSC.
λp = 448.97591 + 0.68443P − 0.00002P 2
(4)
The fitting error of each bright line was within 0.811 nm (Table 3) and was considered to be accurate enough for most projects of plant phenotyping (Adão et al., 2017; Li et al., 2013). 4.2. Results of hyperspectral imaging performance test RGB images (Fig. 7) were respectively extracted from hyperspectral images of MSV and LeafSpec’s HSC. Each sample owned a couple of spectra from MSV and LeafSpec’s HSC, respectively. A set of samples’ coupled spectra comparing the performance of MSV and LeafSpe’s HSC were shown in Fig. 8. The standard deviation of each spectrum, the mean and maximum absolute distance between each coupled spectra, and the mean relative distance between each coupled spectra were given in Table 4. For each color checker panel, the shape of the coupled spectra was highly consistent with each other between 450 nm and 900 nm, with the mean relative distance of less than 0.09. However, the difference between the coupled spectra was large in the wavelength range of 450–500 nm and 875–900 nm compared to the range of 500–875 nm. It was caused by the 450 nm filter, which did not completely block the short-wavelength light in the range of 450–500 nm. The outliers’ area of LeafSpec’s HSC was much smaller than MSV’s, and
3.4. Corn leaf imaging test in the greenhouse A total of 59 corn plants (genotype: Hybrid B73 × Mo17) were grown in the Purdue Lilly Greenhouse (40°25′19.7″N, 86°55′7.8″W) for this study. The temperature in the greenhouse was 23–29 °C, and supplemental lighting was on 12 h a day. The plants were under four different treatments: 16 plants with high nitrogen (N) and well7
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 9. The NDVI heatmaps and spectra of top collar leaves. (a) Genotype A with high nitrogen treatment. (b) Genotype B with high nitrogen treatment. (c) Genotype C with high nitrogen treatment. (d) Genotype A with low nitrogen treatment. (e) Genotype B with low nitrogen treatment. (f) Genotype C with low nitrogen genotype. (g) Colormap bar. (h) Averaged spectrum of each leaf, HA for leaf (a), HB for leaf (b), HC for leaf (c), LA for leaf (d), LB for leaf (e), and LC for leaf (f). Fig. 10. The effect of nitrogen treatment and genotype on the averaged NDVI. A, genotype A; B, genotype B; C, genotype C; H, high nitrogen treated; L, low nitrogen treated. Graphs show (a) Nitrogen treatment effect within each genotype (by one-way ANOVA with Bonferroni correction); (b) Genotype effect within each nitrogen treatment (by Tukey multiple comparisons following two-way ANOVA). Bar graph data shows mean ± SE. (****P < 0.0001, ***P < 0.001, **P < 0.001, *P < 0.5).
8
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 11. Georeferencing NDVI values from LeafSpec. (a) GIS map. (b) Test field after zoom-in (the green rectangles showing the plot boundaries were added manually, each rectangle including three neighboring plots). (c) Colormap for NDVI values on the GIS map (NDVI values were multiplied by 1000). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 12. The relationships between (a) N measured values and N predicted values, (b) RWC measured values and RWC predicted values. RMSEC: root mean square error for calibration. RMSEP: root mean square error for prediction. R2 (Cal): R2 value in calibration. R2 (Pred): R2 value in test.
highest, and the NDVI of genotype B was the lowest within low nitrogen treatment, while the difference between genotype A and C within high nitrogen treatment was not significant (Fig. 10(b)). By combining phenotyping measurements with geo-location information, LeafSpec generates geo-referenced plant health data, which can be viewed with Purdue’s GIS system. The system provides plant health map viewing functions at both farm and regional levels. For example, the NDVI measurements by LeafSpec in one of the 2018 field tests can be viewed on the map from the GIS shown in Fig. 11. Most of the measurements fell within the range of the row of plants. For customers with higher GPS accuracy requirement, a real-time kinematic GPS module could be added for the improvement.
spectra of LeafSpe’s HSC showed a smaller standard deviation which was averaged to 15.747 compared 30.452 for MSV. So, LeafSpec’s HSC owned a higher ratio of mean to standard deviation of each spectrum. 4.3. Qualitative results in the field Spectral indices calculated based on the spectrum are useful to estimate plant health status (Xue and Su, 2017). In this study, NDVI was selected to make a difference between different N treatments and genotypes. NDVI heatmaps (generated from the hyperspectral images in the field) and spectra of different genotypes and nitrogen treatments were shown in Fig. 9. The middle part of each leaf had higher NDVI values compared to the ends. Within the same genotype, leaves from high nitrogen treated plants generally showed higher NDVI values. The leaf veins had relatively lower NDVI value for all plants. The spectra in green and red bands showed the difference between different genotypes and nitrogen treatments. Fig. 10 shows the effect of nitrogen treatment and genotype on the averaged NDVI values of the collard leaves. Both nitrogen treatment and genotype affected the averaged NDVI. Their interaction was also significant. (p < 0.0001 for nitrogen treatment, p < 0.0001 for genotype, p < 0.0001 for nitrogen treatment × genotype, by two-way ANOVA). Specifically, the NDVI of high nitrogen treated leaves was higher than the NDVI of low nitrogen within each genotype (Fig. 10(a)). The NDVI of genotype B was lower than that of the other two genotypes no matter in which nitrogen treatment. The NDVI of genotype C was the
4.4. Modeling results in the greenhouse PLSR model was trained to predict N content with the averaged spectral data obtained by LeafSpec. The model with latent variables (LVs) of 3 achieved good performance both in calibration and test, with R2 of 0.874 and 0.880, RMSE of 0.246 and 0.265, respectively. As illustrated in Fig. 12(a), a linear regression model was fitted between the predicted and measured N content values. There was a strong correlation between LeafSpec predictions and measured N content values. Similarly, the PLSR model was also applied to predict RWC. The model with LVs = 8 achieved good performance both in calibration and test, with R2 of 0.791 and 0.771, RMSE of 0.051 and 0.049, respectively. Fig. 12(b) showed the relation between the predicted and measured 9
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
RWC values. LeafSpec worked to measure N content and RWC for corn plant by hyperspectral imaging the plant’s recently collared leaf with high performance. To improve LeafSpec’s performance, the analysis of spectra distribution across the leaf will be reported in a separate paper.
org/10.1127/pfg/2015/0256. Bubeck, D.M., Goodman, M.M., Beavis, W.D., Grant, D., 1993. Quantitative trait loci controlling resistance to gray leaf spot in maize. Crop Sci. 33 (4), 838–847. https:// doi.org/10.2135/cropsci1993.0011183X003300040041x. Chen, J., Cai, F., He, R., He, S., 2018. Experimental demonstration of remote and compact imaging spectrometer based on mobile devices. Sensors 18 (7), 1989. https://doi. org/10.3390/s18071989. Chen, L., Lin, L., Cai, G., Sun, Y., Huang, T., Wang, K., Deng, J., 2014. Identification of nitrogen, phosphorus, and potassium deficiencies in rice based on static scanning technology and hierarchical identification method. PLoS ONE 9 (11), e113200. https://doi.org/10.1371/journal.pone.0113200. Ciganda, V., Gitelson, A., Schepers, J., 2008. Vertical profile and temporal variation of chlorophyll in maize canopy: quantitative “crop vigor” indicator by means of reflectance-based techniques. Agronomy J. 100 (5), 1409–1417. https://doi.org/10. 2134/agronj2007.0322. Cotrozzi, L., Townsend, P.A., Pellegrini, E., Nali, C., Couture, J.J., 2018. Reflectance spectroscopy: a novel approach to better understand and monitor the impact of air pollution on Mediterranean plants. Environ. Sci. Pollut. Res. 25 (9), 8249–8267. https://doi.org/10.1007/s11356-017-9568-2. Ge, Y., Atefi, A., Zhang, H., Miao, C., Ramamurthy, R.K., Sigmon, B., Schnable, J.C., 2019. High-throughput analysis of leaf physiological and chemical traits with VIS–NIR–SWIR spectroscopy: a case study with a maize diversity panel. Plant Methods 15 (1), 66. https://doi.org/10.1186/s13007-019-0450-8. Ge, Y., Bai, G., Stoerger, V., Schnable, J.C., 2016. Temporal dynamics of maize plant growth, water use, and leaf water content using automated high throughput RGB and hyperspectral imaging. Comput. Electron. Agric. 127. https://doi.org/10.1016/j. compag.2016.07.028. Godfray, H.C.J., Beddington, J.R., Crute, I.R., Haddad, L., Lawrence, D., Muir, J.F., et al., 2010. Food security: the challenge of feeding 9 billion people. Science (New York, N.Y.) 327 (5967), 812–818. https://doi.org/10.1126/science.1185383. Goetz, A.F.H., 2009. Three decades of hyperspectral remote sensing of the Earth: a personal view. Remote Sensing Environ. 113, S5–S16. https://doi.org/10.1016/j.rse. 2007.12.014. Goetz, A., Vane, G., Solomon, J., Rock, B., 1985. Imaging spectrometry for earth remote sensing. Science 228, 1147–1153. https://doi.org/10.1126/science.228.4704.1147. Gowen, A.A., O’Donnell, C.P., Cullen, P.J., Downey, G., Frias, J.M., 2007. Hyperspectral imaging – an emerging process analytical tool for food quality and safety control. Trends Food Sci. Technol. 18 (12), 590–598. https://doi.org/10.1016/j.tifs.2007.06. 001. Gutiérrez, S., Fernández-Novales, J., Diago, M.P., Tardaguila, J., 2018. On-the-go hyperspectral imaging under field conditions and machine learning for the classification of grapevine varieties. Front. Plant Sci. 9, 1102. https://doi.org/10.3389/fpls.2018. 01102. Gutiérrez, S., Wendel, A., Underwood, J., 2019. Ground based hyperspectral imaging for extensive mango yield estimation. Comput. Electron. Agric. 157, 126–135. https:// doi.org/10.1016/J.COMPAG.2018.12.041. Hecht, E., 2016. Optics, fifth ed. Pearson. Heilmann, R.K., Ahn, M., Bautz, M.W., Foster, R., Huenemoerder, D.P., Marshall, H.L., et al., 2009. Development of a critical-angle transmission grating spectrometer for the International X-Ray Observatory. https://doi.org/https://doi.org/10.1117/12. 825394. Jacquemoud, S., Verhoef, W., Baret, F., Bacour, C., Zarco-Tejada, P.J., Asner, G.P., François, C., 2009. PROSPECT + SAIL models: a review of use for vegetation characterization. Remote Sensing Environ. 113, S56–S66. https://doi.org/10.1016/J. RSE.2008.01.026. Kumar, P., Sharma, M.K., 2013. Nutrient Deficiencies of Field Crops: Guide to Diagnosis and Management. https://doi.org/10.1079/9781780642789.0000. Li, L., Zhang, Q., Huang, D., 2014. A review of imaging techniques for plant phenotyping. Sensors (Switzerland) 14 (11), 20078–20111. https://doi.org/10.3390/s141120078. Li, Q., He, X., Wang, Y., Liu, H., Xu, D., Guo, F., 2013. Review of spectral imaging technology in biomedical engineering: achievements and challenges. J. Biomed. Optics 18 (10), 100901. https://doi.org/10.1117/1.JBO.18.10.100901. Palmer, C.A., Loewen, E.G., 2005. Diffraction Grating Handbook. Newport Corporation New York. Pandey, P., Ge, Y., Stoerger, V., Schnable, J.C., 2017. High throughput in vivo analysis of plant leaf chemical properties using hyperspectral imaging. Front. Plant Sci. 8 (August), 1–12. https://doi.org/10.3389/fpls.2017.01348. Purdue University Controlled Environment Phenotyping Facility, n.d. Retrieved September 5, 2019, from https://ag.purdue.edu/cepf/. Ravi, R., Lin, Y.-J., Shamseldin, T., Elbahnasawy, M., Masjedi, A., Crawford, M., Habib, A., 2018. Wheel-based Lidar data for plant height and canopy cover evaluation to aid biomass prediction. In: IGARSS. IEEE, pp. 3242–3245. https://doi.org/10.1109/ IGARSS.2018.8518673. Sigernes, F., Syrjäsuo, M., Storvold, R., Fortuna, J., Grøtte, M.E., Johansen, T.A., 2018. Do it yourself hyperspectral imager for handheld to airborne operations. Optics Exp. 26 (5), 6021. https://doi.org/10.1364/OE.26.006021. Suomalainen, J., Anders, N., Iqbal, S., Roerink, G., Franke, J., Wenting, P., Kooistra, L., 2014. A lightweight hyperspectral mapping system and photogrammetric processing chain for unmanned aerial vehicles. Remote Sensing 6 (11), 11013–11030. https:// doi.org/10.3390/rs61111013. Vigneau, N., Ecarnot, M., Rabatel, G., Roumet, P., 2011. Potential of field hyperspectral imaging as a non destructive method to assess leaf nitrogen content in Wheat. Field Crops Res. 122 (1), 25–31. https://doi.org/10.1016/j.fcr.2011.02.003. Virlet, N., Sabermanesh, K., Sadeghi-Tehran, P., Hawkesford, M.J., 2017. Field Scanalyzer: an automated robotic field phenotyping platform for detailed crop monitoring. Funct. Plant Biol. 44 (1), 143–153. https://doi.org/10.1071/FP16163.
5. Conclusions A new hyperspectral corn leaf imager, named LeafSpec, was developed for more accurate plant phenotyping in both greenhouse and field. A push-broom hyperspectral camera was customized for this device for optimized performance in crop leaf phenotyping applications. According to the imaging tests data, the camera’s imaging results were highly consistent with a commercially available hyperspectral camera. Due to the touch-based sliding imaging mode of LeafSpec, the imaging data quality no longer suffered from the major noise factors such as the changing ambient light, imaging distance, imaging angle, ununiform lighting intensity, etc. The LeafSpec device was successfully tested in Purdue’s research farm field in the summer of 2018. The results significantly segregated the measured plants from different nitrogen treatments and different genotypes within low nitrogen treatment. The NDVI value of genotype B was lowest within both nitrogen treatments. Digital ag maps such as field nitrogen map were automatically generated immediately after the measurements were collected. The LeafSpec was tested to measure N content and RWC of corn plants in the greenhouse in the spring of 2019. With PLSR, LeafSpec predicted N content and RWC with R2 of 0.880 and 0.771, RMSE of 0.265 and 0.049, respectively when compared with ground truth measurements. Based on the preliminary results, it can be concluded that LeafSpec is an easy-to-use and low-cost crop phenotyping sensor with improved measurement accuracy. The LeafSpec could benefit more people in plant science research and agriculture production. 6. Future work By imaging an entire corn leaf, LeafSpec enables us to analyze the distribution of nutrition and stress across the leaf. Preliminary results showed that the distribution information could help to improve the performance of our current plant physiological features predictions models. The Purdue plant phenotyping sensor team is also working on the development of new handheld hyperspectral imagers for other species such as soybean, wheat, rice and so on. The progresses on these projects will be reported in future papers. Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgements The authors would like to acknowledge Indiana Soybean Alliance for funding this project. We also thank Shelby Gruss in the Department of Agronomy at Purdue University for her help in cultivating the corn plants and ground truth data collection. References Adão, T., Hruška, J., Pádua, L., Bessa, J., Peres, E., Morais, R., Sousa, J.J., 2017. Hyperspectral imaging: a review on UAV-based sensors, data processing and applications for agriculture and forestry. Remote Sensing 9 (11). https://doi.org/10.3390/ rs9111110. Bareth, G., Aasen, H., Bendig, J., Gnyp, M.L., Bolten, A., Jung, A., Soukkamäki, J., 2015. Low-weight and UAV-based hyperspectral full-frame cameras for monitoring crops: spectral comparison with portable spectroradiometer measurements. Photogrammetrie - Fernerkundung - Geoinformation 2015 (1), 69–79. https://doi.
10
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Yuan, M., Couture, J.J., Townsend, P.A., Ruark, M.D., Bland, W.L., 2016a. Spectroscopic determination of leaf nitrogen concentration and mass per area in sweet corn and snap bean. Agronomy J. 108 (6), 2519–2526. https://doi.org/10.2134/agronj2016. 05.0260. Yuan, Z., Cao, Q., Zhang, K., Ata-Ul-Karim, S.T., Tian, Y., Zhu, Y., Liu, X., 2016b. Optimal leaf positions for SPAD meter measurement in rice. Front. Plant Sci. 7, 719. https:// doi.org/10.3389/fpls.2016.00719. Zarco-Tejada, P.J., González-Dugo, V., Berni, J.A.J., 2012. Fluorescence, temperature and narrow-band indices acquired from a UAV platform for water stress detection using a micro-hyperspectral imager and a thermal camera. Remote Sensing Environ. 117, 322–337. https://doi.org/10.1016/J.RSE.2011.10.007. Zhang, Y., Xiao, Y., Zhuang, Z., Zhou, L., Liu, F., He, Y., 2016. Development of a near ground remote sensing system. Sensors 16 (5). https://doi.org/10.3390/s16050648.
Ward, J.M.J., Stromberg, E.L., Nowell, D.C., Nutter, F.W., 1999. Gray Leaf Spot: a disease of global importance in maize production. 884 Plant Disease 83 (10), 884–895. https://doi.org/10.1094/PDIS.1999.83.10.884. Xue, J., Su, B., 2017. Significant remote sensing vegetation indices: a review of developments and applications. J. Sensors 2017, 1–17. https://doi.org/10.1155/2017/ 1353691. Yang, Q., Wang, W., 2018. Compact orthogonal-dispersion device using a prism and a transmission grating. J. Eur. Optical Soc. 14 (1). https://doi.org/10.1186/s41476018-0077-9. Yendrek, C.R., Tomaz, T., Montes, C.M., Cao, Y., Morse, A.M., Brown, P.J., Ainsworth, E.A., 2017. High-throughput phenotyping of maize leaf physiological and biochemical traits using hyperspectral reflectance. Plant Physiol. 173 (1), 614–626. https:// doi.org/10.1104/pp.16.01447.
11
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag
LeafSpec: An accurate and portable hyperspectral corn leaf imager a
a,⁎
a
b
a
T a
Liangju Wang , Jian Jin , Zhihang Song , Jialei Wang , Libo Zhang , Tanzeel U. Rehman , Dongdong Maa, Neal R. Carpenterc, Mitchell R. Tuinstrac a b c
Department of Agricultural and Biological Engineering, Purdue University, West Lafayette, IN 47907, United States School of Mechanical Engineering, Purdue University, West Lafayette, IN 47907, United States Department of Agronomy, Purdue University, West Lafayette, IN 47907, United States
A R T I C LE I N FO
A B S T R A C T
Keywords: Handheld sensor Crop leaf scanner Plant phenotyping Hyperspectral imaging LeafSpec
Hyperspectral imaging (HSI) technology has been widely applied in industry and academia for plant phenotyping. However, most HSI systems are large and expensive, making it challenging to benefit more people. The overall goal of this study was to develop a portable and low-cost hyperspectral imaging handheld device (named LeafSpec) with even improved measurement quality compared with traditional HSI systems for crop leaves imaging. The hardware of LeafSpec device was comprised of a push-broom hyperspectral camera (HSC), leaf scanner with an encoder system for leaf position information, a lightbox as an intensive and uniform beam lighting source, and an ARM-based microcontroller. In each scanning, a smooth and clear hyperspectral image of the entire leaf was obtained by quickly sliding LeafSpec across the leaf from the beginning to tip. Each measurement was geo-referenced by sending processed data to a smartphone and combining it with the GPS location and time information before uploading to a Geography Information System (GIS) with Digital Ag Map Services and internet connection. After calibration, the HSC’s imaging results were highly consistent with a commercialized hyperspectral camera. In the field test in the summer of 2018, LeafSpec was able to detect the difference between two nitrogen treatments of corn plants in each genotype, as well as the differences between three genotypes in high nitrogen treatment, and the difference between two genotypes in low nitrogen treatment before it was visible to human eyes. In the greenhouse test in the spring of 2019, LeafSpec predicted nitrogen content and relative water content with R2 of 0.880 and 0.771, RMSE of 0.265 and 0.049, respectively. Generally, LeafSpec is an easy-to-use and low-cost crop phenotyping sensor with improved measurement accuracy, which could benefit more people in plant science research and agriculture production.
1. Introduction The world population will reach 9 billion by 2050, and food production must increase by 70% to feed more people (Godfray et al., 2010). To ensure that high yielding and stress-tolerant plants can be selected rapidly and efficiently, connecting the genotype with the phenotype robustly is more important than before (Li et al., 2014). Hyperspectral Imaging (HSI) technology has been explored and applied in plant phenotyping as both spatial and spectral information are obtained in a high-throughput and non-invasive way (Gowen et al., 2007; Li et al., 2014, 2013). HSI was originally defined and used in remote sensing in the 1980s’ (Goetz et al., 1985; Goetz, 2009). A typical field HSI system configuration involves mounting a hyperspectral camera (HSC) on an unmanned aerial vehicle (UAV) or satellite to image the crop canopy at field level with high-throughput under ambient light
(Adão et al., 2017; Bareth et al., 2015; Zarco-Tejada, González-Dugo, & Berni, 2012; Goetz, 2009). Alternatively, a ground vehicle or gantry is used to maneuver the HSC to image crop plants in closer proximity than UAV or satellite (Gutiérrez et al., 2018; Gutiérrez et al., 2019; Suomalainen et al., 2014; Vigneau et al., 2011; Virlet et al., 2017; Ravi et al., 2018). The ground vehicle or gantry helps obtain higher resolution images of plants from the side view or the top view and makes it possible to analysis plants at the individual plant level. To reduce the affection of ambient light, artificial light could be installed on the imaging system. In greenhouse applications, the plant is placed on an imaging platform manually or automatically with conveyor and then imaged by HSC (Ge et al., 2016; Pandey et al., 2017; “Purdue University Controlled Environment Phenotyping Facility,” n.d.). Another imaging method for greenhouse applications is mounting an HSC on a large scale gantry system to image still plants (Zhang et al., 2016). When the
⁎ Corresponding author at: Department of Agricultural and Biological Engineering, Purdue University, 225 South University Street, West Lafayette, IN 47907, United States. E-mail address: [email protected] (J. Jin).
https://doi.org/10.1016/j.compag.2019.105209 Received 3 March 2019; Received in revised form 18 October 2019; Accepted 31 December 2019 0168-1699/ © 2020 Elsevier B.V. All rights reserved.
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 1. Handheld hyperspectral imaging scanner (LeafSpec). (a) LeafSpec’s photograph. (b) LeafSpec’s brief structure.
In recent years, some customized HSCs have been developed. A recent study conducted by Sigernes et al. (2018) discussed the basic mechanism of developing a small, lightweight, and cheap push broom HSC for handheld to airborne operations. A compact HSC equipped on a motorized selfie stick to perform remote sensing was developed (Chen et al., 2018). These customized HSCs were not designed to image a leaf sample within ten centimeters, and they were also not equipped with high dynamic range imaging sensors. The development of a new portable hyperspectral corn leaf imager, LeafSpec, with a customized new HSC is discussed in this paper. A nonprovisional patent on the hardware and software design of LeafSpec was filed by Purdue research Foundation. LeafSpec has been successfully tested in both Purdue’s research farm fields and greenhouses, and has been commercialized from 2018.
operation space in a greenhouse is enough for a large-scale imaging system and a plant moving system, the plant can be imaged at the plant level and even at leaf level. In an indoor laboratory, either part of a plant, or a small plant like Arabidopsis can be imaged while placed on the imaging system platform with uniform light. Although HSI has demonstrated the capability of predicting plant physiological features, the signal quality is compromised with various noise factors. For remote sensing in the field, the HSI signal is severely impacted by daylight change and weather conditions. For the greenhouse imaging towers, although the ambient light can be blocked by the imaging tower, the artificial imaging lights still have complicated interactions with the 3D plant canopies. The plant leaves are often at different distances from the camera and lights, with various angles of the leaf surfaces. This imaging method causes non-uniform lighting intensities on the leaf surfaces, which is difficult to calibrate with current referencing technologies. According to the PROSAIL model, the spectrum reflected from the leaf surface is significantly impacted by the leaf angle resulting in inconsistency in colors detected (Jacquemoud et al., 2009). These noise factors in remote-sensing or medium-distancesensing do not exist for touch-based imaging sensors. For example, a handheld imaging system with an enclosed imaging chamber and the fixed configuration between light, camera, and object could guarantee the fixed imaging distance, angle, and lighting intensity on the object. Portable spectrometer device, which obtains the spectrum of a spot on the leaf, is applied in plant phenotyping to estimate fresh leaf status in the field. One of these portable spectrometers, ASD FieldSpec 4 (Malvern Panalytical Ltd., Malvern, UK) covers the full visible-near infrared-shortwave infrared (VSWIR; 400–2500 nm) and is widely used for phenotyping of leaf physiological and biochemical traits in plant phenotyping research (Cotrozzi et al., 2018; Ge et al., 2019; Yendrek et al., 2017; M. Yuan et al., 2016). However, the measurement at one small local spot on the leaf does not represent the whole leaf well due to the significant variance of leaf color or spectrum across the leaf (Ciganda, Gitelson, & Schepers, 2008; Z. Yuan et al., 2016). It has also been reported that the variance and distribution of color on the leaf contains useful information for improved phenotyping quality. Gray spots on the leaf caused by a type of fungal disease of corn can be diagnosed with leaf’s color distribution (Bubeck et al., 1993; Ward et al., 1999). The N-P-K deficiencies of rice can be identified with about 90% accuracy with the leaf images (Chen et al., 2014). Leaf color distributions are visibly different among different nutrient deficiencies (Kumar and Sharma, 2013). Therefore, taking the entire leaf image with a handheld touch-based HSC has the potential to enable to development of new algorithms on nutrition and stress distribution analysis, which could provide higher measurement quality. In terms of price, volume and weight, existing HSC on the market does not satisfy the requirement of a handheld device (Li et al., 2014).
2. LeafSpec device development The LeafSpec device took the hyperspectral image of the entire leaf by smoothly sliding the leaf through its imaging chamber. The imaging process took within three to five seconds for each leaf scanning, and the video of the process was available (https://www.youtube.com/watch? v=mPvhV1J8BG8). The raw HS images and the imaging processing results (plant physiological features predictions) were stored in the memory of the device, which could be easily downloaded after the data collection. If the user has a smartphone, a smartphone APP was developed and available to view the images and prediction results on the spot in real-time. Although the device was self-contained, the data could optionally be uploaded by the smartphone APP together with the geo-location and time of the measurement to Purdue’s Ag Geography Information System (GIS). The GIS provides various digital Ag map viewing functions for the phenotyping measurements in real-time when the measurements were being collected. The design of the device, including the general structural design and specific design details of each part in both hardware and software, is included in the non-provisional patent application (Patent No. PCT/ US2018/027953, 2018). 2.1. Hardware configuration of LeafSpec The total weight of LeafSpec (Fig. 1(a)) was about 2 kg with length, width, and height of 230 mm, 150 mm and 300 mm, respectively. The hardware of LeafSpec consisted of four main components (Fig. 1(b)): an HSC, a scanning mechanism, a lightbox, and an ARM® based microcontroller. The lightbox generated a uniform full spectrum beam light. The scanning mechanism equipped with a roller-encoder system was used to detect the leaf motion in a non-invasive way accurately. The HSC collected spectra and spatial information of the leaf. The 2
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 2. LeafSpec’s hyperspectral camera. (a) Camera’s structure. (b) Camera’s photograph.
458 nm to 1650 nm was selected for the first version of the device to filter out the second-order diffraction pattern of less than 440 nm.
microcontroller obtained and analyzed data from the camera and other sensors, and then transferred the result data to a smartphone. 2.2. Hyperspectral camera
2.2.2. Slit and lens Slit plays a pivotal role in the final spectral resolution by determining the amount of light to be imaged. A slit (S50RH, Thorlabs Inc., Newton, NJ, USA) was selected for this device. Its physical dimensions were 50-μm-wide with ± 1 μm tolerance and 3-mm-long. The collimating lens was a convex lens to parallel the light exiting through the slit on the grating. The focusing lens was also convex, but its purpose was to focus the light exiting through the grating on the image sensor. An achromatic doublet lens consisting of two components cemented together was optimized to correct for on-axis spherical and chromatic aberrations (Hecht, 2016). Two achromatic doublets (#63720, Edmund Optics Inc., Barrington, NJ, USA) were selected as a collimating lens and a focusing lens, respectively. The focal length of these two achromatic doublets was 30 mm with ± 2% tolerance and diameter was 10 mm. To make the device smaller and decrease the distance between the collecting lens and imaging sample, a wide-angle lens was required. A lens (ZWO-LENS-2.5, High Point Scientific, Montague, NJ, USA) with 2.5 mm focus length and f/1.2 aperture was mounted at the front of the slit with CS-mount as the collecting lens in the HSC.
The structure of LeafSpec’s hyperspectral camera is presented in Fig. 2. It was consisted of collecting lens, slit, filter, collating lens, grating, focusing lens, and imaging sensor. The light was collected and focused on the slit by the collecting lens. The collating lens collated a line beam passing through the slit. The collated light was spread by the grating and then focused on the imaging sensor by the focusing lens. A detailed explanation of different components is provided in the sections below. 2.2.1. Diffraction grating and filter One of the essential parts in the HSC was the diffraction grating, whose function was to separate polychromatic light into constituent monochromatic lights showing on different positions of an image plane (Palmer and Loewen, 2005). Transmission grating offered low alignment sensitivity, which decreased the requirements for machining tolerance (Heilmann et al., 2009; Yang and Wang, 2018). A visible transmission grating (GT13-03, Thorlabs Inc., Newton, NJ, USA) was selected for the first version of this device. Its physical dimensions were 12.7 mm × 12.7 mm (Length × Width) and had 300 grooves per millimeter with a groove angle of 17.5°. The transmission efficiency of the selected grating was in a range of 38%-74% for 450 nm to 900 nm spectral range. To obtain the spectra from 450 nm to 900 nm, the angle of the diffracted monochromatic light was calculated to be 11.35° by using Eq. (1).
nλ = d (sinθ + sinθ')
2.2.3. Imaging sensor For HSI, quantum efficiency, dark noise, and saturation capacity were important specifications for determining the imaging quality. A monochrome camera (BFLY-U3-05S2M-CS, FLIR Integrated Imaging Solutions Inc., Richmond, BC, Canada), based on Sony CCD sensor ICX693, was selected as the imaging sensor. Its specifications are listed in Table 1. Its power consumption was less than 3 W and could be triggered with general purpose input output (GPIO) of a controller to take images. Its USB 3.1 port worked not only as a power supply port but also as a data port to transfer the image to the controller.
(1)
where n is the diffraction order, λ is the wavelength of the diffracted monochromatic light, d is groove spacing of the grating, θ is the groove angle and θ' is the diffracted angle of monochromatic light. A long-pass filter (62–975, Edmund Optics Inc., Barrington, NJ, USA) with over 91% transmission efficiency in the spectral range 3
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
and the surface inside was machined to a low roughness level with the mirror-like surface to reflect most of the light. The aluminum material was also a good thermal conductor to dissipate the heat generated by the halogen bulbs.
Table 1 Specifications of the imaging sensor. Parameters
Corresponding settings
Camera model Resolution Frame rate Chroma Sensor name Sensor size (mm) Pixel size Shutter type ADC Quantum efficiency (% at 525 nm) Temporal dark noise (e-) Saturation capacity (e-) Dynamic range (dB) Exposure range Interface Power requirements Dimensions (mm) Mass (g)
BFLY-U3-05S2M-CS 808 × 608 50 FPS Mono Sony ICX693 4.85 × 3.65 6.0 µm Global 12 bit 81 9.51 22,074 66.87 0.031 ms–31.9 s USB 3.1 Gen 1 5–24 V via GPIO or 5 V via USB3 29 × 29 × 30 36
2.5. Microcontroller To control the cost and improve the development efficiency, an ARM board was adopted for LeafSpec’s microcontroller. The controller acquired images from the camera through the USB port, position information from the encoder, and other switch signals, and then processed them. It then saved and sent the processed results to a smartphone through a Bluetooth® adapter. Due to the image’s high speed acquiring and processing requirements, ODROID-XU4 (Hardkernel Co., Ltd, Gyeonggi, South Korea) was selected since it had more processing power than the commonly used ARM board Raspberry Pi. The board with a dimension of 83 × 58 × 20 mm was based on Samsung Exynos5422 with 2 GHz (GHz) Cortex™-A15 quad-core and 1.2 GHz Cortex™-A7 quad-core CPUs. It contained 2 gigabytes (GB) LPDDR3 RAM at 933 MHz (MHz), supported eMMC5.0 flash storage and had 3 USB ports and a 30-pin GPIO port. Armbian, a Debian based operating system with Linux kernel 4.9 LTS, was running on the board. The programming language was Python 2.X.
2.3. Leaf scanning mechanism As LeafSpec’s HSC was a push-broom type camera, it imaged only one line of the leaf sample in one shot (Li et al., 2013). To image an entire leaf, the HSC scanned the leaf by moving along the leaf’s midrib direction. The scanning motion was synchronized with the frame acquisition rate of the camera to ensure a smooth integrated image. Imaging of each line was triggered at a specified distance.
2.6. Power supply A suitable battery was important for LeafSpec because of the halogens’ high-power consumption but the device needed to be lightweighted for people to carry. A portable power bank (YB12011000USB, TalentCell, Guangdong, China) with 12 V/11,000 mAh capacity was selected as the power supply. A stable voltage for bulbs during working time to ensure a steady spectrum was a key factor for HSI. A 12–5 V with 3 A and a 12–12 V with 2 A voltage regulators were equipped for the controller and bulbs in the power supply system. To reduce the volume of LeafSpec, the power bank was installed in LeafSpec after its shell was removed. When the inside battery was out of power or consumed, another back-up power bank could be connected to LeafSpec.
2.3.1. Leaf motion detection method A displacement sensor can be installed to acquire the position information of LeafSpec along the leaf during scanning. The sensed information can then be used to determine when to trigger the camera to capture an image. To avoid damaging the leaf, the sensor either should not press on the leaf too firmly or not make contact with the leaf. An optical flow sensor used in a computer’s mouse could measure the movement without touching the leaf surface; however, any change in the spacing between sensor and leaf during leaf scanning may cause low positioning accuracy. The variation in leaf width caused the movement to be undetectable. A rotary encoder was conventionally applied in industrial applications because of its accuracy and high-resolution in most cases. The challenge in the development of this device was to rotate the encoder along a leaf smoothly without damaging the leaf or missing any movement. To solve this problem, the leaf movement measuring mechanism was designed as Fig. 1(b) shown. A roller connected to an encoder and covered with rubber was pressed on the leaf. The encoder driven by the roller outputs the movement signal to the microcontroller; this provides the position information used to trigger the camera.
2.7. Data flow and management The structure of data flow between different components in LeafSpec system is presented in Fig. 3. The microcontroller acquired images from the HSC, and position information from encoder. A list of line images was constructed into one hyperspectral image, then segmented and processed to get leaf parameters. The leaf’s morphological
2.3.2. Seal method To avoid any damage to the leaf, a flexible sealing method was developed for LeafSpec. Black soft and dense cloth strips were attached around the imaging platform (Fig. 1(a)). The height of the sample room was limited within 5 mm so that the force produced by the deflection of the sealing cloth pressed itself on the leaf. Ambient lighting was blocked off, as shown in Fig. 1(a). 2.4. Light source A steady broad-spectrum light source was required for HSI. Halogen bulb was a low-cost but stable option and selected as the light source for LeafSpec. To generate uniform light on a narrow and long window, a white strip made with white Teflon material was placed to reflect and diffuse the light. The light box was machined with aluminum material,
Fig. 3. The data flow and communication in LeafSpec system. 4
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 6. The color checkerboard. Checker panels in red rectangle were imaging samples. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 4. The spectral lines image of krypton light source by LeafSpec’s HSC. Table 2 Parameters for the hyperspectral imaging test.
Table 3 The wavelength and column indices of spectral lines.
Parameters
MSV
LeafSpec’s HSC
Camera model Spectrograph Frame rate (FPS) Exposure time (ms) Spectral resolution Spatial resolution Spectral range (nm) Scan speed (mm/s)
acA780-75gm SpecIM V10H 20 15 582 782 362–1230 5.08
BFLY-U3-05S2M-CS Customized 20 50 676 500 450–900 5.08
Note: MSV, a commercialized hyperspectral camera. HSC, hyperspectral camera.
Peak index
Wavelength (nm)
Column (pixel)
Fitting error (nm)
1 2 3 4 5 6 7 8 9 10 11
557.029 587.096 760.155 769.454 785.482 810.436 819.006 829.811 850.794 877.675 892.869
158 204 460 475 500 538 550 566 598 639 662
−0.441 0.624 −0.811 −0.140 0.428 0.649 0.017 −0.214 −0.082 0.027 −0.057
3. Calibration and tests of the new sensor device 3.1. Spectra calibration of LeafSpec’s HSC For any hyperspectral imager, the imaging result needs to be calibrated with a wavelength standard to precisely label each band with the actual wavelength. A frame (Fig. 4) obtained by the image sensor contained the spatial and spectral information of one-line on the object through the slit. The spectral information of the frame was distributed along the X direction (columns), while the spatial information was distributed along the Y direction (rows). A calibration light source (KR1, Ocean Optics Inc, Largo, FL, USA) with a krypton bulb produced spectral lines from 427 nm to 893 nm. The range covered the spectra calibration requirements of LeafSpec’s HSC whose wavelength range was 450 nm to 900 nm. After imaging the krypton light source (Fig. 4), the corresponding wavelengths of these bright lines in the image were obtained according to the known spectrum of krypton light. The wavelength corresponding to each pixel of the camera with its column index was calculated using the column indices of the bright lines and their corresponding wavelengths (Eq. (2)). Fig. 5. The Hyperspectral imaging test system for LeafSpec.
λp = I + C1 P + C2 P 2 features, including width, length, and area of the leaf, were calculated based on the position information and the image. LeafSpec also provided application interface for any plant physiological feature predicting model based on the leaf’s hyperspectral image or averaged spectrum. The device stored all raw data and results locally on an SD memory card. A selected list of results including the NDVI heatmap, spectrum, and plant physiological parameters were transferred to a smartphone APP through Bluetooth®. The user can choose to upload these leaf parameters and GPS information from smartphone to a GIS server through an internet connection. The raw data could also be downloaded directly from LeafSpec’s storage to PC for further analysis.
(2)
where P is the column index, λp is the wavelength of the column P, I is the wavelength of the column 0, C1 and C2 are the coefficient of the polynomial. 3.2. Hyperspectral imaging performance test The imaging quality of LeafSpec’s HSC was examined by comparing it with a popular commercialized HSC MSV (MSV-101-W, Middleton Spectral Vision, Middleton, WI, USA). The MSV consisted of an imaging sensor (acA780-75gm, Basler AG, Ahrensburg, Germany) and a spectrograph (V10H, SpecIM Spectral Imaging Ltd, Oulu, Finland). The key parameters of these two HSC are compared in Table 2. The two cameras 5
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 7. The RGB images of a color checkerboard. (a) RGB image from a commercialized hyperspectral camera MSV. (b) RGB image from LeafSpec’s hyperspectral camera.
Fig. 8. Comparison of spectra from LeafSpec’s hyperspectral camera and MSV. LC, LeafSpec’s hyperspectral camera. MSV, a commercialized hyperspectral camera. The curve is the median of all pixel’s intensity at each wavelength. The outliers are between lower and upper quartiles of all pixel’s intensity at each wavelength. To display variance clearly, the lower and upper quartiles were multiplied by 10.
6
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
watered, 16 plants with high N and drought-stressed, 11 plants with low N and well-watered, and 16 plants with low N and droughtstressed. The high and low N treatments were achieved with 120 ppm and 25 ppm nitrogen fertilizers, respectively. The well-watered plants were irrigated as needed, whereas, we stopped irrigating the droughtstressed plants a few days before imaging. When irrigating, we guaranteed that each pot achieved saturation. The experiment was carried out in the greenhouse on March 19, 2019, from 12:30 PM to 2:30 PM. Corn plants were at V7-V8 stage. LeafSpec was used to scan the top collared leaf of each plant from the start to the tip. The smartphone, Huawei® Mate 20 (Huawei Technologies Co., Ltd., Shenzhen, Guangdong China) with Android 9.0 operating system, was connected with LeafSpec through Bluetooth® for result visualization. A piece of leaf sample of each plant was cut off and weighed to obtain leaf fresh weight (FW) after measured by LeafSpec. Then the sample was immediately hydrated until it was completely turgid under normal room lighting and temperature. Leaf sample was then weighed to obtain the fully turgid weight (TW) after its surface water was removed lightly and quickly with filter paper. Finally, the leaf sample was fully dried with the dry oven (persistently at 60 °C) for approximately 24 h to obtain the dry weight (DW). RWC was computed by using the relationship (Eq. (3)):
Table 4 Comparative performance of LeafSpec’s HSC and MSV on the imaging stage. Color of Checkerboard
Intensity Std. LeafSpec’s HSC
Intensity Std. MSV
Intensity Mean Absolute Distance
Intensity Maximum Absolute Distance
Intensity Mean Relative Distance
Cyan Magenta Yellow Red Green Orange yellow Yellow green Purple Moderate red Purplish blue Average
11.742 17.777 19.654 14.579 12.408 18.339 16.365 13.924 18.345 14.336 15.747
25.617 35.846 39.374 29.826 26.288 33.389 30.702 25.226 30.880 27.372 30.452
26.304 41.202 30.991 53.108 45.277 38.358 43.784 58.563 34.271 62.386 43.424
107.271 217.858 283.175 271.982 264.053 314.277 286.847 325.895 290.450 256.334 261.814
0.040 0.029 0.023 0.078 0.044 0.039 0.033 0.090 0.038 0.060 0.047
Note: HSC, hyperspectral camera. MSV, a commercialized hyperspectral camera. Std., standard deviation. Intensity mean absolute distance is the mean value of the coupled spectra’s difference. Intensity mean relative distance is the mean value of the ratio between the coupled spectra’s difference and MSV’s spectrum.
were tested using a linear hyperspectral scanning stage (MRC-920-044, Middleton Spectral Vision, Middleton, WI, USA) equipped with halogen light source (Fig. 5). A color checkerboard (MSCCC, X-Rite Inc., Grand Rapids, MI, USA) was fixed on the sampling stage as the imaging object (Fig. 6). When the color checkerboard was linearly moved by the stage, the two HSCs imaged the color panels inside the red rectangle (see Fig. 6 for reference) simultaneously. A PVC white strip was imaged by both HSCs as white reference to calibrate the hyperspectral images. After calibration, RGB images were extracted from hyperspectral data. The spectrum of each checker panel was obtained with the median intensities of all pixels inside that panel. Finally, the image quality and spectral quality were compared between the two cameras.
RWC (%) =
FW − DW × 100% TW − DW
(3)
Nitrogen content of all collared leaves of each plant was analyzed in the Agronomy Department at Purdue University to analyze the nitrogen content using a FlashEA 1112 Nitrogen and Carbon Analyzer (Thermo Fisher Scientific, Waltham, MA USA). We obtained the averaged spectrum of each leaf sample with LeafSpec. The partial least square regression (PLSR) model was applied to these spectra to predict N content and RWC of each plant. Forty-one samples were used to calibrate the model, and 18 samples were used to test the model. Leave-one-out cross-validation method was introduced to limit problems like overfitting.
3.3. Corn leaf imaging test in the field A field test with corn plants was conducted on July 17, 2018, in Agronomy Center for Research and Education field #9D of Purdue University (40°28′14.3″N, 86°59′40.2″W, 4540 US-52, West Lafayette, IN 47906). Three genotypes including B73xMo17, CML550xPHP02, and P1105AM (annotated as A, B, and C, respectively hereafter) were planted in 24 plots. Twelve plots were treated as high (H) nitrogen group (250 kg/ha), and the remaining twelve were treated as low (L) nitrogen group (0 kg/ha). For each genotype and treatment combination, there were four replicate plots. Eight plants were randomly selected from each plot, so in total 192 corn plants were imaged at the V8 stage. LeafSpec was used to scan the top collared leaf of each plant from the start to the tip. The smartphone, Huawei® Mate 9 (Huawei Technologies Co., Ltd., Shenzhen, Guangdong China) with Android 8.0 operating system, was connected with LeafSpec through Bluetooth® for result visualization. The hyperspectral raw data was saved on the local storage of the LeafSpec. The NDVI heatmaps and leaf parameters of each leaf were sent to the smartphone. Finally, these parameters were uploaded to the cloud GIS database with GPS location and timestamp information. In this study, all NDVI values were calculated with equation NDVI = (b850 − b680)/(b850 + b680) , in which b850 and b680 were the intensities at wavelength 850 nm and 680 nm, respectively.
4. Testing results and discussions 4.1. Spectra calibration result With Eq. (2) and Table 3, the wavelengths regression function is shown in Eq. (4) for the tested LeafSpec’s HSC.
λp = 448.97591 + 0.68443P − 0.00002P 2
(4)
The fitting error of each bright line was within 0.811 nm (Table 3) and was considered to be accurate enough for most projects of plant phenotyping (Adão et al., 2017; Li et al., 2013). 4.2. Results of hyperspectral imaging performance test RGB images (Fig. 7) were respectively extracted from hyperspectral images of MSV and LeafSpec’s HSC. Each sample owned a couple of spectra from MSV and LeafSpec’s HSC, respectively. A set of samples’ coupled spectra comparing the performance of MSV and LeafSpe’s HSC were shown in Fig. 8. The standard deviation of each spectrum, the mean and maximum absolute distance between each coupled spectra, and the mean relative distance between each coupled spectra were given in Table 4. For each color checker panel, the shape of the coupled spectra was highly consistent with each other between 450 nm and 900 nm, with the mean relative distance of less than 0.09. However, the difference between the coupled spectra was large in the wavelength range of 450–500 nm and 875–900 nm compared to the range of 500–875 nm. It was caused by the 450 nm filter, which did not completely block the short-wavelength light in the range of 450–500 nm. The outliers’ area of LeafSpec’s HSC was much smaller than MSV’s, and
3.4. Corn leaf imaging test in the greenhouse A total of 59 corn plants (genotype: Hybrid B73 × Mo17) were grown in the Purdue Lilly Greenhouse (40°25′19.7″N, 86°55′7.8″W) for this study. The temperature in the greenhouse was 23–29 °C, and supplemental lighting was on 12 h a day. The plants were under four different treatments: 16 plants with high nitrogen (N) and well7
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 9. The NDVI heatmaps and spectra of top collar leaves. (a) Genotype A with high nitrogen treatment. (b) Genotype B with high nitrogen treatment. (c) Genotype C with high nitrogen treatment. (d) Genotype A with low nitrogen treatment. (e) Genotype B with low nitrogen treatment. (f) Genotype C with low nitrogen genotype. (g) Colormap bar. (h) Averaged spectrum of each leaf, HA for leaf (a), HB for leaf (b), HC for leaf (c), LA for leaf (d), LB for leaf (e), and LC for leaf (f). Fig. 10. The effect of nitrogen treatment and genotype on the averaged NDVI. A, genotype A; B, genotype B; C, genotype C; H, high nitrogen treated; L, low nitrogen treated. Graphs show (a) Nitrogen treatment effect within each genotype (by one-way ANOVA with Bonferroni correction); (b) Genotype effect within each nitrogen treatment (by Tukey multiple comparisons following two-way ANOVA). Bar graph data shows mean ± SE. (****P < 0.0001, ***P < 0.001, **P < 0.001, *P < 0.5).
8
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Fig. 11. Georeferencing NDVI values from LeafSpec. (a) GIS map. (b) Test field after zoom-in (the green rectangles showing the plot boundaries were added manually, each rectangle including three neighboring plots). (c) Colormap for NDVI values on the GIS map (NDVI values were multiplied by 1000). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 12. The relationships between (a) N measured values and N predicted values, (b) RWC measured values and RWC predicted values. RMSEC: root mean square error for calibration. RMSEP: root mean square error for prediction. R2 (Cal): R2 value in calibration. R2 (Pred): R2 value in test.
highest, and the NDVI of genotype B was the lowest within low nitrogen treatment, while the difference between genotype A and C within high nitrogen treatment was not significant (Fig. 10(b)). By combining phenotyping measurements with geo-location information, LeafSpec generates geo-referenced plant health data, which can be viewed with Purdue’s GIS system. The system provides plant health map viewing functions at both farm and regional levels. For example, the NDVI measurements by LeafSpec in one of the 2018 field tests can be viewed on the map from the GIS shown in Fig. 11. Most of the measurements fell within the range of the row of plants. For customers with higher GPS accuracy requirement, a real-time kinematic GPS module could be added for the improvement.
spectra of LeafSpe’s HSC showed a smaller standard deviation which was averaged to 15.747 compared 30.452 for MSV. So, LeafSpec’s HSC owned a higher ratio of mean to standard deviation of each spectrum. 4.3. Qualitative results in the field Spectral indices calculated based on the spectrum are useful to estimate plant health status (Xue and Su, 2017). In this study, NDVI was selected to make a difference between different N treatments and genotypes. NDVI heatmaps (generated from the hyperspectral images in the field) and spectra of different genotypes and nitrogen treatments were shown in Fig. 9. The middle part of each leaf had higher NDVI values compared to the ends. Within the same genotype, leaves from high nitrogen treated plants generally showed higher NDVI values. The leaf veins had relatively lower NDVI value for all plants. The spectra in green and red bands showed the difference between different genotypes and nitrogen treatments. Fig. 10 shows the effect of nitrogen treatment and genotype on the averaged NDVI values of the collard leaves. Both nitrogen treatment and genotype affected the averaged NDVI. Their interaction was also significant. (p < 0.0001 for nitrogen treatment, p < 0.0001 for genotype, p < 0.0001 for nitrogen treatment × genotype, by two-way ANOVA). Specifically, the NDVI of high nitrogen treated leaves was higher than the NDVI of low nitrogen within each genotype (Fig. 10(a)). The NDVI of genotype B was lower than that of the other two genotypes no matter in which nitrogen treatment. The NDVI of genotype C was the
4.4. Modeling results in the greenhouse PLSR model was trained to predict N content with the averaged spectral data obtained by LeafSpec. The model with latent variables (LVs) of 3 achieved good performance both in calibration and test, with R2 of 0.874 and 0.880, RMSE of 0.246 and 0.265, respectively. As illustrated in Fig. 12(a), a linear regression model was fitted between the predicted and measured N content values. There was a strong correlation between LeafSpec predictions and measured N content values. Similarly, the PLSR model was also applied to predict RWC. The model with LVs = 8 achieved good performance both in calibration and test, with R2 of 0.791 and 0.771, RMSE of 0.051 and 0.049, respectively. Fig. 12(b) showed the relation between the predicted and measured 9
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
RWC values. LeafSpec worked to measure N content and RWC for corn plant by hyperspectral imaging the plant’s recently collared leaf with high performance. To improve LeafSpec’s performance, the analysis of spectra distribution across the leaf will be reported in a separate paper.
org/10.1127/pfg/2015/0256. Bubeck, D.M., Goodman, M.M., Beavis, W.D., Grant, D., 1993. Quantitative trait loci controlling resistance to gray leaf spot in maize. Crop Sci. 33 (4), 838–847. https:// doi.org/10.2135/cropsci1993.0011183X003300040041x. Chen, J., Cai, F., He, R., He, S., 2018. Experimental demonstration of remote and compact imaging spectrometer based on mobile devices. Sensors 18 (7), 1989. https://doi. org/10.3390/s18071989. Chen, L., Lin, L., Cai, G., Sun, Y., Huang, T., Wang, K., Deng, J., 2014. Identification of nitrogen, phosphorus, and potassium deficiencies in rice based on static scanning technology and hierarchical identification method. PLoS ONE 9 (11), e113200. https://doi.org/10.1371/journal.pone.0113200. Ciganda, V., Gitelson, A., Schepers, J., 2008. Vertical profile and temporal variation of chlorophyll in maize canopy: quantitative “crop vigor” indicator by means of reflectance-based techniques. Agronomy J. 100 (5), 1409–1417. https://doi.org/10. 2134/agronj2007.0322. Cotrozzi, L., Townsend, P.A., Pellegrini, E., Nali, C., Couture, J.J., 2018. Reflectance spectroscopy: a novel approach to better understand and monitor the impact of air pollution on Mediterranean plants. Environ. Sci. Pollut. Res. 25 (9), 8249–8267. https://doi.org/10.1007/s11356-017-9568-2. Ge, Y., Atefi, A., Zhang, H., Miao, C., Ramamurthy, R.K., Sigmon, B., Schnable, J.C., 2019. High-throughput analysis of leaf physiological and chemical traits with VIS–NIR–SWIR spectroscopy: a case study with a maize diversity panel. Plant Methods 15 (1), 66. https://doi.org/10.1186/s13007-019-0450-8. Ge, Y., Bai, G., Stoerger, V., Schnable, J.C., 2016. Temporal dynamics of maize plant growth, water use, and leaf water content using automated high throughput RGB and hyperspectral imaging. Comput. Electron. Agric. 127. https://doi.org/10.1016/j. compag.2016.07.028. Godfray, H.C.J., Beddington, J.R., Crute, I.R., Haddad, L., Lawrence, D., Muir, J.F., et al., 2010. Food security: the challenge of feeding 9 billion people. Science (New York, N.Y.) 327 (5967), 812–818. https://doi.org/10.1126/science.1185383. Goetz, A.F.H., 2009. Three decades of hyperspectral remote sensing of the Earth: a personal view. Remote Sensing Environ. 113, S5–S16. https://doi.org/10.1016/j.rse. 2007.12.014. Goetz, A., Vane, G., Solomon, J., Rock, B., 1985. Imaging spectrometry for earth remote sensing. Science 228, 1147–1153. https://doi.org/10.1126/science.228.4704.1147. Gowen, A.A., O’Donnell, C.P., Cullen, P.J., Downey, G., Frias, J.M., 2007. Hyperspectral imaging – an emerging process analytical tool for food quality and safety control. Trends Food Sci. Technol. 18 (12), 590–598. https://doi.org/10.1016/j.tifs.2007.06. 001. Gutiérrez, S., Fernández-Novales, J., Diago, M.P., Tardaguila, J., 2018. On-the-go hyperspectral imaging under field conditions and machine learning for the classification of grapevine varieties. Front. Plant Sci. 9, 1102. https://doi.org/10.3389/fpls.2018. 01102. Gutiérrez, S., Wendel, A., Underwood, J., 2019. Ground based hyperspectral imaging for extensive mango yield estimation. Comput. Electron. Agric. 157, 126–135. https:// doi.org/10.1016/J.COMPAG.2018.12.041. Hecht, E., 2016. Optics, fifth ed. Pearson. Heilmann, R.K., Ahn, M., Bautz, M.W., Foster, R., Huenemoerder, D.P., Marshall, H.L., et al., 2009. Development of a critical-angle transmission grating spectrometer for the International X-Ray Observatory. https://doi.org/https://doi.org/10.1117/12. 825394. Jacquemoud, S., Verhoef, W., Baret, F., Bacour, C., Zarco-Tejada, P.J., Asner, G.P., François, C., 2009. PROSPECT + SAIL models: a review of use for vegetation characterization. Remote Sensing Environ. 113, S56–S66. https://doi.org/10.1016/J. RSE.2008.01.026. Kumar, P., Sharma, M.K., 2013. Nutrient Deficiencies of Field Crops: Guide to Diagnosis and Management. https://doi.org/10.1079/9781780642789.0000. Li, L., Zhang, Q., Huang, D., 2014. A review of imaging techniques for plant phenotyping. Sensors (Switzerland) 14 (11), 20078–20111. https://doi.org/10.3390/s141120078. Li, Q., He, X., Wang, Y., Liu, H., Xu, D., Guo, F., 2013. Review of spectral imaging technology in biomedical engineering: achievements and challenges. J. Biomed. Optics 18 (10), 100901. https://doi.org/10.1117/1.JBO.18.10.100901. Palmer, C.A., Loewen, E.G., 2005. Diffraction Grating Handbook. Newport Corporation New York. Pandey, P., Ge, Y., Stoerger, V., Schnable, J.C., 2017. High throughput in vivo analysis of plant leaf chemical properties using hyperspectral imaging. Front. Plant Sci. 8 (August), 1–12. https://doi.org/10.3389/fpls.2017.01348. Purdue University Controlled Environment Phenotyping Facility, n.d. Retrieved September 5, 2019, from https://ag.purdue.edu/cepf/. Ravi, R., Lin, Y.-J., Shamseldin, T., Elbahnasawy, M., Masjedi, A., Crawford, M., Habib, A., 2018. Wheel-based Lidar data for plant height and canopy cover evaluation to aid biomass prediction. In: IGARSS. IEEE, pp. 3242–3245. https://doi.org/10.1109/ IGARSS.2018.8518673. Sigernes, F., Syrjäsuo, M., Storvold, R., Fortuna, J., Grøtte, M.E., Johansen, T.A., 2018. Do it yourself hyperspectral imager for handheld to airborne operations. Optics Exp. 26 (5), 6021. https://doi.org/10.1364/OE.26.006021. Suomalainen, J., Anders, N., Iqbal, S., Roerink, G., Franke, J., Wenting, P., Kooistra, L., 2014. A lightweight hyperspectral mapping system and photogrammetric processing chain for unmanned aerial vehicles. Remote Sensing 6 (11), 11013–11030. https:// doi.org/10.3390/rs61111013. Vigneau, N., Ecarnot, M., Rabatel, G., Roumet, P., 2011. Potential of field hyperspectral imaging as a non destructive method to assess leaf nitrogen content in Wheat. Field Crops Res. 122 (1), 25–31. https://doi.org/10.1016/j.fcr.2011.02.003. Virlet, N., Sabermanesh, K., Sadeghi-Tehran, P., Hawkesford, M.J., 2017. Field Scanalyzer: an automated robotic field phenotyping platform for detailed crop monitoring. Funct. Plant Biol. 44 (1), 143–153. https://doi.org/10.1071/FP16163.
5. Conclusions A new hyperspectral corn leaf imager, named LeafSpec, was developed for more accurate plant phenotyping in both greenhouse and field. A push-broom hyperspectral camera was customized for this device for optimized performance in crop leaf phenotyping applications. According to the imaging tests data, the camera’s imaging results were highly consistent with a commercially available hyperspectral camera. Due to the touch-based sliding imaging mode of LeafSpec, the imaging data quality no longer suffered from the major noise factors such as the changing ambient light, imaging distance, imaging angle, ununiform lighting intensity, etc. The LeafSpec device was successfully tested in Purdue’s research farm field in the summer of 2018. The results significantly segregated the measured plants from different nitrogen treatments and different genotypes within low nitrogen treatment. The NDVI value of genotype B was lowest within both nitrogen treatments. Digital ag maps such as field nitrogen map were automatically generated immediately after the measurements were collected. The LeafSpec was tested to measure N content and RWC of corn plants in the greenhouse in the spring of 2019. With PLSR, LeafSpec predicted N content and RWC with R2 of 0.880 and 0.771, RMSE of 0.265 and 0.049, respectively when compared with ground truth measurements. Based on the preliminary results, it can be concluded that LeafSpec is an easy-to-use and low-cost crop phenotyping sensor with improved measurement accuracy. The LeafSpec could benefit more people in plant science research and agriculture production. 6. Future work By imaging an entire corn leaf, LeafSpec enables us to analyze the distribution of nutrition and stress across the leaf. Preliminary results showed that the distribution information could help to improve the performance of our current plant physiological features predictions models. The Purdue plant phenotyping sensor team is also working on the development of new handheld hyperspectral imagers for other species such as soybean, wheat, rice and so on. The progresses on these projects will be reported in future papers. Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgements The authors would like to acknowledge Indiana Soybean Alliance for funding this project. We also thank Shelby Gruss in the Department of Agronomy at Purdue University for her help in cultivating the corn plants and ground truth data collection. References Adão, T., Hruška, J., Pádua, L., Bessa, J., Peres, E., Morais, R., Sousa, J.J., 2017. Hyperspectral imaging: a review on UAV-based sensors, data processing and applications for agriculture and forestry. Remote Sensing 9 (11). https://doi.org/10.3390/ rs9111110. Bareth, G., Aasen, H., Bendig, J., Gnyp, M.L., Bolten, A., Jung, A., Soukkamäki, J., 2015. Low-weight and UAV-based hyperspectral full-frame cameras for monitoring crops: spectral comparison with portable spectroradiometer measurements. Photogrammetrie - Fernerkundung - Geoinformation 2015 (1), 69–79. https://doi.
10
Computers and Electronics in Agriculture 169 (2020) 105209
L. Wang, et al.
Yuan, M., Couture, J.J., Townsend, P.A., Ruark, M.D., Bland, W.L., 2016a. Spectroscopic determination of leaf nitrogen concentration and mass per area in sweet corn and snap bean. Agronomy J. 108 (6), 2519–2526. https://doi.org/10.2134/agronj2016. 05.0260. Yuan, Z., Cao, Q., Zhang, K., Ata-Ul-Karim, S.T., Tian, Y., Zhu, Y., Liu, X., 2016b. Optimal leaf positions for SPAD meter measurement in rice. Front. Plant Sci. 7, 719. https:// doi.org/10.3389/fpls.2016.00719. Zarco-Tejada, P.J., González-Dugo, V., Berni, J.A.J., 2012. Fluorescence, temperature and narrow-band indices acquired from a UAV platform for water stress detection using a micro-hyperspectral imager and a thermal camera. Remote Sensing Environ. 117, 322–337. https://doi.org/10.1016/J.RSE.2011.10.007. Zhang, Y., Xiao, Y., Zhuang, Z., Zhou, L., Liu, F., He, Y., 2016. Development of a near ground remote sensing system. Sensors 16 (5). https://doi.org/10.3390/s16050648.
Ward, J.M.J., Stromberg, E.L., Nowell, D.C., Nutter, F.W., 1999. Gray Leaf Spot: a disease of global importance in maize production. 884 Plant Disease 83 (10), 884–895. https://doi.org/10.1094/PDIS.1999.83.10.884. Xue, J., Su, B., 2017. Significant remote sensing vegetation indices: a review of developments and applications. J. Sensors 2017, 1–17. https://doi.org/10.1155/2017/ 1353691. Yang, Q., Wang, W., 2018. Compact orthogonal-dispersion device using a prism and a transmission grating. J. Eur. Optical Soc. 14 (1). https://doi.org/10.1186/s41476018-0077-9. Yendrek, C.R., Tomaz, T., Montes, C.M., Cao, Y., Morse, A.M., Brown, P.J., Ainsworth, E.A., 2017. High-throughput phenotyping of maize leaf physiological and biochemical traits using hyperspectral reflectance. Plant Physiol. 173 (1), 614–626. https:// doi.org/10.1104/pp.16.01447.
11