- Email: [email protected]
Development of an android app to estimate chlorophyll content of corn leaves based on contact imaging
Computers and Electronics in Agriculture 116 (2015) 211–220
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag
Development of an android app to estimate chlorophyll content of corn leaves based on contact imaging q Farshad Vesali a, Mahmoud Omid a,⇑, Amy Kaleita b, Hossein Mobli a a b
Department of Agricultural Machinery Engineering, Faculty of Agricultural Engineering and Technology, University of Tehran, Karaj, Iran Department of Agricultural and Biosystems Engineering, Iowa State University, Ames, IA, USA
a r t i c l e
i n f o
Article history: Received 17 March 2015 Received in revised form 12 June 2015 Accepted 12 June 2015
Keywords: Smartphones Chlorophyll content App SPAD Contact image
a b s t r a c t In this study a new android app for smartphones to estimate chlorophyll content of a corn leaf is presented. Contact imaging was used for image acquisition from the corn leaves which captures the light passing through the leaf directly by a smartphone’s camera. This approach would eliminate the needs for background segmentation and other pre-processing tasks. To estimate SPAD (Soil Plant Analysis Development) values, various features were extracted from each image. Then, superior features were extracted by stepwise regression and sensitivity analysis. The selected features were finally used use as inputs to the linear (regression) and neural network models. Performance of the models was evaluated using the images taken from a corn field located in West of Ames, IA, USA, with Minolta SPAD 502 Chlorophyll Meter. The R2 and RMSE values for the linear model were 0.74 and 6.2. The corresponding values for the neural network model were 0.82 and 5.10, respectively. Finally, these models were successfully implemented on an app named SmartSPAD on the smartphone. After installing the developed app on the smartphone, the performance of the models were evaluated again using a new independent set of data collected by SmartSPAD directly from maize plants inside a greenhouse. The SmartSPAD estimation compared well with the corresponding SPAD meter values (R2 = 0.88 and 0.72, and RMSE = 4.03 and 5.96 for neural network and linear model, respectively). The developed app can be considered as a low cost alternative for estimating the chlorophyll content especially when there is a demand for high availability. Ó 2015 Elsevier B.V. All rights reserved.
1. Introduction Determining the chlorophyll content of plants gives valuable information relevant to plant health and crop management. Chlorophyll is the main pigment in leaves and it is responsible for leaf greenness. Leaf colour is an indicator of plant health and also it can indicate plant nutrient status (Yadav et al., 2010; Muñoz-Huerta et al., 2013). For example, there is significant correlation between chlorophyll and nitrogen content of leaf tissues, thus by measuring chlorophyll content, nitrogen status can be assessed (Evans, 1989; Tewari et al., 2013). On the other hand, excess nutrients like nitrogen in an agricultural environment is a leading cause of water quality impairment (Turner and Rabalais, 1991). Therefore, managing and balancing agricultural nutrients
q Financial support by: Department of Agricultural Machinery Engineering, Faculty of Agricultural Engineering and Technology, University of Tehran, Karaj, Iran. ⇑ Corresponding author. Tel.: +98 912 3631832; fax: +98 2632808138. E-mail address: [email protected] (M. Omid).
http://dx.doi.org/10.1016/j.compag.2015.06.012 0168-1699/Ó 2015 Elsevier B.V. All rights reserved.
use has economic benefits in addition to reducing the risk of water and environment pollution (Daughtry et al., 2000; Sawyer et al., 2004). Destructive methods like Kjeldahl tissue analysis to determine nutrients status, in addition to their high costs, cannot be used as a way for variable rate fertilising (i.e., real-time application) because of time lag between collecting tissue sampling and obtaining results (Piekielek et al., 1995; Muñoz-Huerta et al., 2013). Chlorophyll meters (CMs) have been used by many researchers as a non-destructive method to measure the chlorophyll content and estimate the nitrogen value of agricultural crops (Richardson et al., 2002; Chang and Robison, 2003; Murillo-Amador et al., 2004; Scharf et al., 2006; Uddling et al., 2007; Miao et al., 2009). CMs use two wavebands to assess chlorophyll content, infrared light centred 930 nm and red light centred 650 nm (Blackmer et al., 1994). In recent years, additional non-destructive techniques based on spectral and hyperspectral reflectance have been investigated for estimating the chlorophyll content of plants. These methods were developed to achieve special purposes such as, real time and accurate nutrients status reporting (Feng et al., 2008; Fitzgerald et al.,
212
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
2010; Tian et al., 2011). Because chlorophyll content affects visual features of leaves, using digital cameras or in other words RGB (red, green, blue) imaging as a low cost instrument in the visible range has also been used in nitrogen status estimation (Dutta Gupta et al., 2013; Lee and Lee, 2013; Wang et al., 2013). Standard cameras are significantly lower in cost than other imaging systems and chlorophyll meters (Rorie et al., 2011a), which cost about $1500US, but there are some challenges in using them for this purpose; for example, different ambient lighting conditions or shadows on leaves will affect the images. To overcome these issues Tewari et al. (2013) used an experimental setup with a cover and an artificial light to estimate nitrogen content of a rice paddy crop. In addition, utilising independent indices extracted from images can be useful in reducing the effect of variation in light. Wang et al. (2013) used GMR (G–R), G/R, NGI (normalised green index), NRI (normalised red index) and Hue indices for estimating biomass, N content and leaf area index (LAI). They mentioned that in the GMR index, the colour of the plant canopy is sharply different from the background and it is feasible to set a threshold to segment rice plant from background. Moreover, GMR and G/R indices had a better correlation than the other indices to estimate biomass, N content and LAI. It seems that combinations of component values have better correlation with N content of plants. Karcher and Richardson (2003) found that the green value in RGB colour space cannot exactly represent how green the vegetation will appear, and that red and blue values also may change the appearance of the green colour of turfgrass. They introduce the dark green colour index (DGCI) based on HSB (hue, saturation, and brightness) colour space, and after calibrating HSB values, the DGCI showed a good correlation with N content. Further studies also reported the capability of DGCI to estimate N content of plants (Rorie et al., 2011a,b). In order to use any of the above strategies or similar, computer processing is required. With the advent of smartphones, the camera and the processor exist in the same device, opening new opportunities for image capture and data generation. Gong et al. (2013) developed and evaluated an app for android smartphones that can estimate the citrus yield two weeks before harvest time. They use a phone-implemented image processing technique for identifying fruits in an image of a tree by segmenting and clustering. Confalonieri et al. (2013) developed an app called PocketLAI, which used smartphone images to estimate LAI, one of the principal indices for assessing crop water requirements and photosynthetic primary production. The authors note that their approach is an inexpensive and highly portable alternative to commercially available LAI devices. Recently, developments in smartphones especially in their processors with built in sensors like cameras have brought us an opportunity that in addition to using their sensors as measurement tools, computation and analysis can be done on them without any additional attachment. Yet to date, no standalone android apps for measuring leaf chlorophyll content have been developed. In this study we designed and implemented an app for android smartphones named SmartSPAD to estimate the SPAD value of corn plants. In order to increase practicality and accuracy in real conditions, a new method of imaging is introduced. Overall performance of the app is compared with Minolta SPAD-502 chlorophyll meter.
2. Materials and methods 2.1. Data collection The data were collected from maize (Zea mays) plots at the Iowa State University Field Extension Education Laboratory, Ames, IA (USA) during the 2014 growing season. Various levels of
nitrogen deficiency were induced by using different fertiliser treatments: 0, 56, 112, 168, and 224 kg ha1 (0, 50, 100, 150, 200, pounds per acre). Each treatment was applied on two of the 55 m 27 m plots since 2011, and N treatment was replicated in 6 rows. Both plots of each treatment were corn–corn rotation; the east plot had no tillage and the west plot was ploughed by chisel in falls of 2012 and 2013. A set of validation data were collected from maize plants inside a greenhouse located in Iowa State University (Agronomy Department). These plants were fertilised by nitrogen at different levels and leaf images with corresponding SPAD observations were collected among them randomly. 2.2. Image acquisition and SPAD determination To take images from the plant leaves, a LG E975 smartphone with CCD sensor camera was used. To avoid or reduce effects of ambient conditions on images, a new method of smartphone imaging is presented which we refer to as contact imaging. In this method, unlike standard picture-taking, leaves are held to the camera lens of the smartphone and the camera captures the light passing through the leaf (Fig. 1). Compared to standard image capturing, this method of contact imaging has several advantages including: No interference from the background: One of the main steps in using image processing techniques in leaf imaging is segmenting the background. A variety of methods exist for distinguishing the target from the background, but the possibility of misclassification always exists (Teimouri et al., 2014), and even a small misclassification rate can affect the results. With contact imaging, however, there is no need to remove the background, and the entire image can be used as an input data. No variation in the distance between leaf and sensor: Generally, for robust use of cameras in either the lab or field, predefined distances are assumed; to keep this distance between the target and the sensor constant during image capture, frames or other setups have been used. Contact imaging eliminates the need for any other attachments. No differences in image focus or blur: The nature of contact images overcome this problem. Because in this method there is no space between leaf and camera, there is essentially no difference between a focused image and an unfocused image. Lower influence of different ambient conditions: Cloudy or sunny lighting conditions, shadows, and wind conditions can affect the success of using cameras in field conditions. Sun position also affects the reflectance of vegetation indices of plants (de Souza et al., 2010). But in contact imaging, only low lighting condition has some influence on the resulting images; as described below, we reduce this effect by adding luminance factor to the features used in computing the SPAD estimates. Other field conditions like shadow, sun position, and wind, have no effect on contact images. An additional benefit of contact imaging is that the effects of camera-to-camera variation are minimal. Because of the nature of contact imaging and the image processing described below, differences in sensor size or camera focusing algorithms will not have an effect on the results of the contact image. About 480 contact images in RGB colour space were captured by the smartphone and were transferred to a desktop computer for further analysis and model development. Images were intentionally taken from all plots under different light conditions (clear and cloudy sky) and at different times of the day. During image acquisition, field meteorological conditions including solar radiation, air temperature and relative humidity were varied from 200
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
213
of colour spaces, other combination indices suggested by the literature were calculated, including GMR (difference between green component and red component), GDR (green divided by red), VI (vegetation index), DGCI (dark green colour index), normalised red (NRI) and normalised green (NGI) (Kawashima and Nakatani, 1998; Gonzalez et al., 2009; Li et al., 2010; Rorie et al., 2011b; Lee and Lee, 2013; Tewari et al., 2013; Wang et al., 2013); their forms are given by Eqs. (8)–(13).
C ¼ max ðR; G; BÞ min ðR; G; BÞ 8 GB > < 60 C ; maxðR; G; BÞ ¼ R Hue ¼ 60 2 þ BR ; maxðR; G; BÞ ¼ G C > : 60 4 þ GB ; maxðR; G; BÞ ¼ B C ( Saturation ¼
Fig. 1. The way of contact imaging by smartphone from corn leaves.
to 750 (w m2), 20 to 30 (°C) and 65 to 95 (percent) respectively during the measurement periods according to The Weather Channel (http://www.weather.com). Fig. 2 shows five sample pictures that were taken by the contact imaging method from different N treatment plots. To determine the chlorophyll content of corn leaves, a SPAD-502 (Minolta Osaka Co., Ltd., Japan) was used. The SPAD value has been found to be well correlated with leaf chlorophyll content extracted through organic solvent method (Yadav et al., 2010). Chlorophyll meters calculate the SPAD value by the ratio of transmitted red light and infrared light, and this SPAD value has exponential relation (Eq. (1)) with chlorophyll content of plants (Markwell et al., 1995).
Chl ¼ 10M
^0:265
ð1Þ 2
where Chl is the chlorophyll content in unit of lmol m and M is the SPAD value measured by Minolta SPAD 502. After taking each image, a corresponding SPAD value was recorded from the same corn leaf position that at which the camera image had been captured. The range and average of SPAD values from different fertilisation plots are shown in Fig. 3. In our study the SPAD meter data were assumed to be perfect measures of chlorophyll content for the purpose of calibrating and validating the contact image model. 2.3. Feature extraction Most mobile phones take images in RGB colour space with 8 bit depth, which means for each component, the value is integer number between 0 and 255. Our images were in 8-bit RGB mode, and they were saved in jpeg (joint photographic experts group) format. For each image Hue, saturation, and brightness from HSB colour space, and Y (relative luminance), Cb (difference between the blue component and a reference value) and Cr (difference between the blue component and a reference value) from YCbCr colour space were extracted by Eqs. (2)–(7). In addition to these main channels
0;
V¼0
C ; maxðR;G;BÞ
V–0
ð2Þ
ð3Þ
Brightness ¼ maxðR; G; BÞ=255
ð4Þ
Y ¼ 0:257 R þ 0:504 G þ 0:098 B þ 16
ð5Þ
Cb ¼ 0:148 R 0:291 G þ 0:439 B þ 128
ð6Þ
Cr ¼ 0:439 R 0:368 G 0:071 B þ 128
ð7Þ
GMR ¼ G R
ð8Þ
GDR ¼ G=R
ð9Þ
VI ¼
GR GþR
DGCI ¼ ½ðHue=60 1Þ þ ð1 SÞ þ ð1 BÞ=3
ð10Þ ð11Þ
NRI ¼
R RþGþB
ð12Þ
NGI ¼
G RþGþB
ð13Þ
where R, G and B in the above equations denote the average values of the red, green and blue components of each contact image, respectively. For all indices including, R, G, B and indices calculated by Eqs. (2)–(13), the coefficient of determination (R2) and root mean square error (RMSE) of estimated SPAD values were obtained by a Matlab 8.2 program (MathWorks, Inc, Natick, MA, USA) installed on desktop computer. In order to find a better and more accurate combination of colour space components for estimating SPAD value, statistical and artificial intelligence models were used. Input variables for these models included: mean of Red, Green, and Blue, from RGB colour space, mean of Hue, Saturation and Brightness from HSB colour space and mean of Y, Cb, and Cr from YCbCr colour space. In addition to means, standard deviations for each feature were included in the features dataset. To account for variations in the amount of light during image capturing, an additional variable, luminance factor, was also included. Hiscocks and Eng (2011) measured luminance for regular digital cameras with the following equation: 2
L¼
Nf KtS
ð14Þ
where N is the maximum pixel value in an image, f is the relative aperture or F number, t is shutter speed or exposure time, S is ISO or sensor light sensitivity, and K is the camera constant. The
214
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
a
b
c
d
e
Fig. 2. Contact images from corn leaves from (a) 0 kg ha1, (b) 56 kg ha1, (c) 112 kg ha1, (d) 168 kg ha1 and (e) 224 kg ha1, N treatment captured by smartphone.
70
SPAD Value
60 51.53
50
54.33
42.28
40 30
52.47
Mean
27.97
20 10 0
Zero kg/ha
56 kg/ha
112 kg/ha
168 kg/ha
224 kg/ha
Max Min Average Fig. 3. Maximum, minimum and average of SPAD value in different nitrogen treatment in corn field.
constant K is calculated through calibration of the camera, but the other factors are stored in the EXIF (Exchangeable image file). This file is attached to each image and is accessible both during and after taking images. F number is the ratio of the lens’s focal length to the diameter of the camera’s entrance pupil; both of these two parameters are fixed in most smartphones, so this can be treated as a constant. The maximum pixel value of an image (N) in regular images can be useful because regular images contain different elements so to that different intensities of light will show up in image components, but in contact imaging the most image has nearly the same value for all pixels. In the other words the pixel standard deviation in contact images are low (Table 1), therefor maximum pixel value in contact images does not represent whiter pixels as in regular images. Thus for contact imaging, variations in luminance from image to image depend only on differences in sensor sensitivity and exposure time. The modified luminance factor used in this study is therefore:
LF ¼
Table 1 Range values (minimum and maximum) of mean and standard deviation of all extracted features. For all but LF, values represent the range across all images of mean and standard deviation of pixel values in an image. For LF, each image has but a single value, so range and standard deviation are over the whole image dataset.
1 tS
ð15Þ
Use of luminance factor also ensures minimal differences between cameras on other android phones, because any differences in exposure time or ISO speed will be accessible through the EXIF file data. Overall minimum and maximum values of mean and standard deviation of all 19 features that were extracted from contact images are shown in Table 1. 2.4. SPAD value estimation 2.4.1. Linear modelling A multivariable linear model was developed to estimate SPAD value of corn leaves. Altogether, 19 features were extracted from each image to develop a linear model: mean and standard deviation of R, G, B, H, S, Br, Y, Cb, and Cr, plus LF. Since the main goal was developing a model to implement in smartphones, the number of variables should be reduced in order to reduce processing time.
Red Green Blue Hue Saturation Brightness Y Cb Cr LF
Standard Deviation
Min
Max
Min
Max
0.17 128.62 0 9.99 0.69 0.50 98.02 54.06 66.13 0.0075
157.28 190.75 48.46 20.81 1 0.75 145.31 91.35 139.19 1.22
0.49 2.20 0 0.04 0 0.009 1.77 0.92 0.58 0.2852
22.57 17.49 29.72 1.15 0.18 0.07 13.08 12.57 8.42
Furthermore, with more input variables, the possibility of overfitting the data increases. Therefore a feature selection method was applied to select the best features to estimate chlorophyll content (SPAD value) of leaves. Numerous statistical methods to find superior features exist, such as LASSO, stepwise regression, wrapper methods, sensitivity analysis, and correlation-based feature selection (Saeys et al., 2007; Mollazade et al., 2012; Lee and Lee, 2013; Teimouri et al., 2014). In this study to develop a linear model we used stepwise regression to choose the most reliable and effective features. In this method, terms are added to or removed from the multivariable model based on their statistical significance level. To accomplish stepwise regression between variables, MATLAB software installed on a desktop computer was used. Stepwise feature selection was developed on 70% of the image dataset, and the remaining data were used to evaluate the performance of developed model.
2.4.2. Neural network modelling All 19 extracted features (mean and standard deviation of R, G, B, H, S, Br, Y, Cb, and Cr, plus LF) from contact images were used again to create and train predictive neural networks. Between different kinds of neural networks, perceptron is the most common and popular topology in both prediction and classification (Heaton, 2008; Omid et al., 2010) and is used here. To avoid overfitting and enable faster computation, feature selection was applied among the 19 inputs for the neural network too. Here sensitivity analysis was used to rank parameters in order of influence so that the most influential parameters could be determined. Sensitivity analysis attempts to provide a ranking of the model inputs based on their relative contributions to model output variability and uncertainty. For calculating the sensitivities, each input (feature vector) was shifted slightly and the corresponding change in the output was computed (Teimouri et al., 2014). The number of hidden layers was minimised based on the observation that one hidden layer neural network can approximate any function that contains a continuous mapping from one finite space to another.
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
The number of nodes in each hidden layer were chosen by trial and error and the rule-of-thumb that the number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer (Heaton, 2008). Initially a MLP (multi-layer perceptron) neural network was trained with all features. Then by sensitivity analysis the best features were selected, and a MLP with new architecture based on the selected input features was trained and evaluated using the same 70% and 30% of contact images respectively as the linear model development and evaluation. The learning algorithm in both neural networks was Levenberg Marquardt. This algorithm is one of the fastest optimisation backpropagation (BP) algorithms and is recommended as a first-choice supervised algorithm to update weights and biases in neural networks (Demuth and Beale, 1993). 2.4.3. Performance analysis To evaluate performance of the proposed methods (linear model as well as neural networks), coefficient of determination (R2) and RSME were computed for the portion of the dataset not used during the training process (Taghadomi-Saberi et al., 2013). In this study we used form of R2 which measures the coefficient of determination of models (Ramsey and Schafer, 2012) rather than of a linear trend line fitted through the data. 2.5. Implementing and characterizing the app The SmartSPAD application oversees taking contact images with the smartphone’s camera and storing them in the smartphone memory. To avoid crashing due to large image file sizes, each image is resized by bicubic interpolation to VGA dimension (640 480), or near this dimension based on its original aspect ratio. After invoking an image, the SmartSPAD application examines the image based on a uniformity characteristic to ensure the image is a contact image. Contact images of a plant leaf are highly uniform, and thus low uniformity suggests the image may be non-contact. Uniformity U can be extracted from image by the following equation (Gonzalez et al., 2009):
U¼
m 1 X
p2 ðBrightnessi Þ
ð16Þ
i¼0
where p(Brightnessi) is the histogram of the brightness levels in image, and m is the number of possible brightness levels. Using our image database, we determined a minimum threshold of uniformity corresponding to contact images. If image uniformity is below the minimum threshold, SmartSPAD suggests that the operator take another picture. When the image become ready to process, depending on which model is selected to estimate the SPAD value, features will extracted and the SPAD value calculated through the model. The calculated value of SPAD will show on screen and operator can save the value (Fig. 4). To start another estimation, the user may take another image and the process will repeat. During android application development stage, ‘‘activities1’’ are used in order to do something by smartphones. These allow user to interact with the phone and to do particular task. Therefore, depending on the type of application and its intended function, some number of activities must be created (Meier, 2012). Specifically, the developed SmartSPAD application contains four main activities including the main menu, SPAD estimation, data, and information. The menu activity is simply a connector activity and is merely used to select the other activities. To operate the app appropriately on any
1 An Activity is an application component that provides a screen with which users can interact in order to do something.
215
other phones with different screen size and resolution, an adaptive layout for each activity was designed. As android app development is based on Java programing language, for each activity there must be a class2 file that contains the commands to operate that activity. In addition to having a class file for each activity, there are four additional classes including the image processing class which converts images to other colour spaces and extracts input features for further processing by neural network class and linear model class, to compute the SPAD value through them and database class for saving the SPAD values. Programing to develop SmartSPAD application was done on Android Studio 64 bit version 8 which included the accessory development kit (ADK) and software development kit (SDK) for android. Platform of android API 19 (KitKat or Android 4.4.2) was chosen as the target platform but for compatibility with other mobile phones, using it on API 14 (Ice-cream sandwich) is possible. It should be noted that about 88 percent of all android phones are operated with this version or later (Statista, 2014). After finishing the app development, SmartSPAD was successfully installed on the mobile phone. Finally, in order to validate the performance of the installed app on the smartphone, about 60 new images were taken from maize plants cultivated inside a greenhouse, in which the light condition was a combination of day light and artificial light. These images were captured independently through the developed app, and processed on the phone using the app itself. SmartSPAD estimates were then downloaded for comparison to corresponding SPAD meter measurements for each image. None of these images were used during the app development. During this stage of app evaluation, criterions used for performance evaluation were R2 and RMSE as before.
3. Results and discussion 3.1. Correlation of different indices with SPAD values The mean value of R, G, B and all indices explained in Section 2.3 were compared with SPAD obtained by Minolta SPAD-502. As previously reported (Yadav et al., 2010; Tewari et al., 2013), B was poorly fitted to chlorophyll content, while R showed a good agreement with chlorophyll content. But in this study, G component which others have reported having good relation with chlorophyll content, was poorly fitted to SPAD value. It seems in our contact imaging of corn leaves, the Green and Blue components are almost fixed (having relatively very low variation from one image to the next) and only the red is fitted with SPAD value (Fig. 5). Among the other indices, Hue index had the strongest linear relationship with SPAD value (R2 = 0.57, RMSE = 7.7) (Fig. 6.). All other indices, including DGCI that has been used as one of the main indices for estimation chlorophyll content (Karcher and Richardson, 2003; Rorie et al., 2011a,b; Lee and Lee, 2013), were not strongly fitted to SPAD value (Table 2). 3.2. Estimation of SPAD values by linear model From the 19 features, the stepwise algorithm with a p = 0.01 significance level threshold selected 4 superior features as input variables for linear model. The selected features were luminance factor, mean of hue, standard deviation of hue, and standard deviation of Cr. Coefficient and p values of each feature are shown in Table 3. The resulting expression for estimation SPAD values (with R2 = 0.74 and RMSE 6.20) is:
2 A class, in the context of Java, is a template that is used to create objects, and to define object data types and methods.
216
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
Fig. 4. Flowchart shows the procedure of SmartSPAD app to estimate SPAD value.
80
R² = 0.04
70
R² = 0.11
SPAD Value
60 50 40 30
R² = 0.56 20 10
0
20
40
60
80
100
120
140
160
180
200
Mean of R, G, or B Fig. 5. Correlation between mean of Red (d), Green ( ) and Blue ( ) indices with SPAD value.
SPAD ¼ 22:48 LF þ 2:06 Huemean þ 8:72 Huestd þ 0:73 Crstd þ 14:62 ð19Þ
The p-values for luminance factor and hue component were almost zero, demonstrating that these are the two most important features. Overfitting is evident in the all-features linear model.
Despite the higher value for R2 in training mode, in test mode this value is lower than for the stepwise linear model which has just four features (Fig. 7). In both train mode and test mode, the linear model with 4 selected features had lower RMSE and higher R2 than did the best single index, which was hue. Since the range of actual and
217
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
80
R² = 0.59 70
SPAD Value
60 50 40 30 20 10
8
10
12
14
16
18
20
22
Mean of Hue Fig. 6. Correlation between mean of Hue index with SPAD value.
Table 2 Regression parameters (slope and intercept), coefficient of determination and root mean square error of different indices correlated to SPAD value. Hue index is the more correlated to SPAD value than other indices (bold row). Indices
Red Green Blue Hue Saturation Brightness Y Cb Cr GMR GDR VI DGCI NRI NGI
Regression parameters
RMSE 2
Slope
Intercept
R
0.30 0.63 0.43 4.63 66.0 159.91 1.05 1.42 0.65 0.28 0.02 47.30 296.7 95.69 93.73
71.99 55.11 44.77 22.30 110.85 54.86 171.1 53.55 115.39 24.75 45.81 30.67 82.34 78.02 15.36
0.56 0.09 0.04 0.59 0.04 0.11 0.39 0.37 0.57 0.57 0.003 0.47 0.20 0.48 0.43
7.72 11.0 11.42 7.46 11.44 11.01 9.07 9.28 7.79 7.65 11.63 8.30 10.42 8.18 8.60
estimated SPAD is almost the same as in Yadav et al. (2010), the RMSE values can be directly compared. They used a linear model with different input variables; in the best situation the RMSE value was 6.6 in training, higher than our linear model RMSE (5.8 and 6.2 for train and test mode respectively). Their images were taken by a scanner in a fixed lab condition. 3.3. Estimation of SPAD values by neural network model At first a neural network having all 19 features as inputs and one hidden layer which contains 14 nodes, was trained on the same training dataset (70% of data) as was used to develop the linear model, and its performance was evaluated with remaining
(test) data. Sensitivity analysis selected superior features by feature sensitivity value; in other words the cause and effect relationship between the inputs and output of the network were extracted, and features ranked based on their importance in SPAD estimation. The input channels that produce low sensitivity values can be considered insignificant and most often can be removed from the network. In this case the four highest ranked features including hue, luminance factor, standard deviation of Cr and standard deviation of hue were selected for designing the new neural network structure (Fig. 8). Three of those features were also selected by the stepwise regression. Accordingly, another MLP neural network was trained using the four selected features with four neurons in one hidden layer (Fig. 9) Based on performance evaluation by RMSE and R2, the multi-layer perceptron networks gave better results than the linear model with both all features and four selected features as an inputs (Table 4). As with the linear model, overfitting problem also showed its effect in train mode; the error value in train mode in the primary neural network is lower than in test mode. The neural network with selected features, however, estimated SPAD value accurately in test mode (Table 4 and Fig. 10). Neural network model showed a strong capability (RMSE = 5.1 and R2 = 0.82) to estimate SPAD value using just four superior features. In addition to reducing overfitting problem by using fewer features, less computation is needed to calculate the estimated SPAD value. Liu et al. (2010) reported that BP neural networks can estimate chlorophyll content of rice better than a linear model. Their inputs were spectral features and they achieve 0.71 and 0.90 for R2 value with linear model and neural network respectively. Although the R2 of their neural network model is better than in this study, in addition to using a spectrometer to gather data from field experiments, which is more expensive than a smartphone, their spectral measurements were under cloudless or near cloudless conditions, whereas the measurements in this study were made under a range of conditions. 3.4. Implementation of the models on android smartphone In the SmartSPAD application, to avoid memory leak as was mentioned, contact images were resized before doing any processing on them. Since the means of extracted features were used as an input variables and bicubic interpolation is based on averaging of 4 by 4 neighbourhood pixels, the mean values will not change after resizing. Small changes in standard deviations of different features were observed which was predictable. Before extracting features to process through linear model or neural network, the application examines the uniformity characteristic of each image in order to determine if the picture has the quality of a contact image. In entire our image database, the lowest value for the uniformity was 0.022 (in comparison to 0.0058 for an example regular image). Therefore the SmartSPAD offers the operator to take another picture if the computed uniformity for an image if this value is less than 0.015. To estimate SPAD value from contact images, while the neural network gave better results, the linear model was also implemented in the Beta version of the SmartSPAD application for
Table 3 Coefficients of linear model with superior features and compares its performance with linear model with all feature in train and test mode. Coefficients
Linear model with 4 selected features p value Linear model with all features
Train mode
Test mode
LF
Hue
STD of Hue
STD of Cr
Intercept
R2
RMSE
R2
RMSE
22.48 5.81e47
2.06 1.53e35
8.72 4.08e04
0.73 3.4e03
14.62 –
0.75
5.8
0.74
6.2
0.77
5.69
0.68
6.9
218
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
80
80
RMSE= 6.2
RMSE= 6.9
60
Actual SPAD value
Actual SPAD Value
60
40
R 2 = 0.68 20
0
40
R 2 = 0.74 20
0
20
40
60
80
0
0
20
40
60
Estimated SPAD Value
Estimated SPAD Value
a
b
80
Fig. 7. Correlation in test mode between actual SPAD value and the SPAD values estimated by (a) linear model with all features, (b) linear model with 4 superior features.
Table 4 Performance evaluation of neural networks.
4.5 4
Structure
Sensitivity
3.5 3
Neural net with all features Neural net with 4 features
2.5
19-14-1 4-4-1
Train mode
Test mode
RMSE
R2
RMSE
R2
4.1 4.82
0.87 0.82
6.16 5.10
0.74 0.82
2 1.5 1 0.5 Lum Fact. Red STD Red Green STD G Blue STD Blue Hue STD Hue Sat. STD S Bright. STD B Y STD Y Cb STD Cb Cr STD Cr
0
Fig. 8. Sensitivity of different features in the neural network output (SPAD value).
further evaluation. Processing time per contact image for both the neural network and the linear model was less than 2 s on a quad core 1.5 GHz Snapdragon smartphone. It should be noted that in this application, just two parallel process were used so it can be easily run in any dual core smartphones as well.
In the SmartSPAD app, three main activities (pages) do the task of capturing image, measuring the approximate SPAD value and storing the data (Fig. 11). The menu activity contains menu lists of capturing or browsing images, previous records and information about the application (Fig. 11A). SPAD estimation activity gets the contact image and estimates the SPAD value by neural network or linear model, and if operator wants to save the value, it can be saved (Fig. 11B). Data activity (page) contains list of data, and the operator can get the specific average value of the data generated by each method (Fig. 11C). In addition the data can be exported as a CSV file and the file shared with other applications like email, to send the exported file via email to any other devices. 3.5. Evaluation of SmartSPAD app For the validation (greenhouse) data, every SPAD value estimated by the SmartSPAD application installed on the phone, was
Fig. 9. BP neural network flowchart with 4-4-1 structure.
219
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
80
80
RMSE= 6.16
RMSE= 5.10 60
Actual SPAD value
Actual SPAD value
60
40
20
0
20
40
60
R 2 = 0.82
20
R 2 = 0.74
0
40
80
0
0
20
40
60
Estimated SPAD value
Estimated SPAD value
a
b
80
Fig. 10. Correlation in test mode between actual SPAD value and SPAD value estimated by (a) neural network with all features, (b) neural network with 4 superior features.
a
b
c
Fig. 11. SmartSPAD application developed on android smartphones. (a) Menu activity. (b) SPAD estimation activity or calculation page. (c) Data activity.
recorded using the embedded data logging section in the app. At the end of the app calculation, results were transferred to a PC to be compared with the measured SPAD values by Minolta SPAD 502 (Fig. 12). The values of R2 and RMSE (R2 were 0.88 and 0.72, and RMSE were 4.03 and 5.96 for neural network and linear model, respectively) showed that the app had almost the same performance on the greenhouse data as on the development data. 4. Conclusion An app called SmartSPAD was developed for android smartphones to estimate the SPAD (soil Plant Analysis Development) value of corn plants. Overall, the proposed method of imaging
(contact image by smartphone) showed a strong agreement to estimate chlorophyll content (SPAD value) of corn leaves in real field conditions. Using data from the camera and derived from the contact images, a linear model and a neural network model were developed and tested on independent data. Hue and luminance factor were the most important features in these contact images. The R2 and RMSE value for validation data were about 0.74 and 6.2, and 0.82 and 5.10 for the linear model and neural network respectively. The results indicate this approach can be used to efficiently replicate the performance of a higher-cost SPAD meter. Implementing this method on a smartphone can be a practical alternative to measure chlorophyll content as low cost device. In addition to the lower cost, SmartSPAD has several other
220
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
80
Actual SPAD Value
60
40
20
R 2 = 0.74 R 2 = 0.88
0
0
20
40
60
80
Estimated SPAD value Fig. 12. Evaluation of SmartSPAD for both (N) neural network and () linear models against actual SPAD values, on images from maize plants inside a greenhouse.
advantages, like storing a large number of chlorophyll content values, or potentially logging the GPS data at same time, while Minolta SPAD 502 users encounter limitation in memory and no capability to store location position. Also, in comparison with other proposed devices which usually include digital cameras and computers, SmartSPAD is a stand-alone approach and because luminance factor is computed from the camera data, SmartSPAD compensates for different lighting conditions, requiring less pre-processing and giving accurate results. Acknowledgment The financial support provided by the Research Department of University of Tehran, Iran, under Grant No. 1305051.6.29 is duly acknowledged. References Blackmer, T.M., Schepers, J.S., Varvel, G.E., 1994. Light reflectance compared with other nitrogen stress measurements in corn leaves. Agron. J. 86, 934–938. Chang, S.X., Robison, D.J., 2003. Nondestructive and rapid estimation of hardwood foliar nitrogen status using the SPAD-502 chlorophyll meter. For. Ecol. Manage. 181, 331–338. Confalonieri, R., Foi, M., Casa, R., Aquaro, S., Tona, E., Peterle, M., Boldini, A., De Carli, G., Ferrari, A., Finotto, G., Guarneri, T., Manzoni, V., Movedi, E., Nisoli, A., Paleari, L., Radici, I., Suardi, M., Veronesi, D., Bregaglio, S., Cappelli, G., Chiodini, M.E., Dominoni, P., Francone, C., Frasso, N., Stella, T., Acutis, M., 2013. Development of an app for estimating leaf area index using a smartphone. Trueness and precision determination and comparison with other indirect methods. Comput. Electron. Agric. 96, 67–74. Daughtry, C.S.T., Walthall, C.L., Kim, M.S., de Colstoun, E.B., McMurtrey, J.E., 2000. Estimating corn leaf chlorophyll concentration from leaf and canopy reflectance. Remote Sens. Environ. 74, 229–239. de Souza, E.G., Scharf, P.C., Sudduth, K.A., 2010. Sun Position and Cloud Effects on Reflectance and Vegetation Indices of Corn. Agron. J. 102, 734. Demuth, H., Beale, M., 1993. Neural network toolbox for use with MATLAB. Dutta Gupta, S., Ibaraki, Y., Pattanayak, A.K., 2013. Development of a digital image analysis method for real-time estimation of chlorophyll content in micropropagated potato plants. Plant Biotechnol. Rep. 7, 91–97. Evans, J., 1989. Partitioning of nitrogen between and within leaves grown under different irradiances. Funct. Plant Biol. 16, 533–548. Feng, W., Yao, X., Zhu, Y., Tian, Y.C., Cao, W.X., 2008. Monitoring leaf nitrogen status with hyperspectral reflectance in wheat. Eur. J. Agron. 28, 394–404. Fitzgerald, G., Rodriguez, D., O’Leary, G., 2010. Measuring and predicting canopy nitrogen nutrition in wheat using a spectral index—the canopy chlorophyll content index (CCCI). Field Crops Res. 116, 318–324. Gong, A., Yu, J., He, Y., Qiu, Z., 2013. Citrus yield estimation based on images processed by an android mobile phone. Biosyst. Eng. 115, 162–170. Gonzalez, R.C., Woods, R.E., Eddins, S.L., 2009. Digital Image Processing using MATLAB, second ed. Gatesmark Pub., S.I..
Heaton, J., 2008. Introduction to Neural Networks with Java. Heaton Research Inc.. Hiscocks, P.D., Eng, P., 2011. Measuring Luminance with a Digital Camera. Syscomp Electronic Design Limited. Karcher, D.E., Richardson, M.D., 2003. Quantifying turfgrass color using digital image analysis. Crop Sci. 43, 943. Kawashima, S., Nakatani, M., 1998. An algorithm for estimating chlorophyll content in leaves using a video camera. Ann. Bot. 81, 49–54. Lee, K.-J., Lee, B.-W., 2013. Estimation of rice growth and nitrogen nutrition status using color digital camera image analysis. Eur. J. Agron. 48, 57–65. Li, Y., Chen, D., Walker, C.N., Angus, J.F., 2010. Estimating the nitrogen status of crops using a digital camera. Field Crops Res. 118, 221–227. Liu, M., Liu, X., Li, M., Fang, M., Chi, W., 2010. Neural-network model for estimating leaf chlorophyll concentration in rice under stress from heavy metals using four spectral indices. Biosyst. Eng. 106, 223–233. Markwell, J., Osterman, J., Mitchell, J., 1995. Calibration of the Minolta SPAD-502 leaf chlorophyll meter. Photosynth. Res. 46, 467–472. Meier, R., 2012. Professional Android 4 Application Development. John Wiley & Sons. Miao, Y., Mulla, D., Randall, G., Vetsch, J., Vintila, R., 2009. Combining chlorophyll meter readings and high spatial resolution remote sensing images for in-season site-specific nitrogen management of corn. Precision Agric. 10, 45–62. Mollazade, K., Omid, M., Arefi, A., 2012. Comparing data mining classifiers for grading raisins based on visual features. Comput. Electron. Agric. 84, 124–131. Muñoz-Huerta, R.F., Guevara-Gonzalez, R.G., Contreras-Medina, L.M., TorresPacheco, I., Prado-Olivarez, J., Ocampo-Velazquez, R.V., 2013. A review of methods for sensing the nitrogen status in plants: advantages, disadvantages and recent advances. Sensors 13, 10823–10843. Murillo-Amador, B., Ávila-Serrano, N.Y., García-Hernández, J.L., López-Aguilar, R., Troyo-Diéguez, E., Kaya, C., 2004. Relationship between a nondestructive and an extraction method for measuring chlorophyll contents in cowpea leaves. Beziehung zwischen den mit einer nicht destruktiven und den mit einer Extraktionsmethode ermittelten Chlorophyllgehalten in Cowpea-Blättern. J. Plant Nutr. Soil Sci. 167, 363–364. Omid, M., Mahmoudi, A., Omid, M.H., 2010. Development of pistachio sorting system using principal component analysis (PCA) assisted artificial neural network (ANN) of impact acoustics. Expert Syst. Appl. 37, 7205–7212. Piekielek, W.P., Fox, R.H., Toth, J.D., Macneal, K.E., 1995. Use of a chlorophyll meter at the early dent stage of corn to evaluate nitrogen sufficiency. Agron. J. 87, 403. Ramsey, F., Schafer, D., 2012. The Statistical Sleuth: A Course in Methods of Data Analysis. Cengage Learning. Richardson, A.D., Duigan, S.P., Berlyn, G.P., 2002. An evaluation of noninvasive methods to estimate foliar chlorophyll content. New Phytol. 153, 185–194. Rorie, R.L., Purcell, L.C., Karcher, D.E., King, C.A., 2011a. The assessment of leaf nitrogen in corn from digital images. Crop Sci. 51, 2174–2180. Rorie, R.L., Purcell, L.C., Mozaffari, M., Karcher, D.E., King, C.A., Marsh, M.C., Longer, D.E., 2011b. Association of ‘‘greenness’’ in corn with yield and leaf nitrogen concentration. Agron. J. 103, 529. Saeys, Y., Inza, I., Larrañaga, P., 2007. A review of feature selection techniques in bioinformatics. Bioinformatics 23, 2507–2517. Sawyer, J.E., Barker, D.W., Lundvall, J.P., 2004. Using Chlorophyll Meter Readings to Determine n Application Rates for Corn. North Central Extension-Industry Soil Fertility, Des Moines, IA. Scharf, P.C., Brouder, S.M., Hoeft, R.G., 2006. Chlorophyll meter readings can predict nitrogen need and yield response of corn in the north-central USA. Agron. J. 98, 655. Statista, 2014. Android devices distribution platform versions 2014 | Statistic, Statista. Taghadomi-Saberi, S., Omid, M., Emam-Djomeh, Z., Ahmadi, H., 2013. Development of an intelligent system to determine sour cherry’s antioxidant activity and anthocyanin content during ripening. Int. J. Food Prop. 17, 1169–1181. Teimouri, N., Omid, M., Mollazade, K., Rajabipour, A., 2014. A novel artificial neural networks assisted segmentation algorithm for discriminating almond nut and shell from background and shadow. Comput. Electron. Agric. 105, 34–43. Tewari, V.K., Aruda, A.K., Kumar, S.P., Pandey, V., Chandel, N.S., 2013. Estimation of plant nitrogen content using digital image processing. Agric. Eng. Int.: CIGR J. 15, 78–86. Tian, Y.C., Yao, X., Yang, J., Cao, W.X., Hannaway, D.B., Zhu, Y., 2011. Assessing newly developed and published vegetation indices for estimating rice leaf nitrogen concentration with ground- and space-based hyperspectral reflectance. Field Crops Res. 120, 299–310. Turner, R.E., Rabalais, N.N., 1991. Changes in mississippi river water quality this century. Bioscience 41, 140–147. Uddling, J., Gelang-Alfredsson, J., Piikki, K., Pleijel, H., 2007. Evaluating the relationship between leaf chlorophyll concentration and SPAD-502 chlorophyll meter readings. Photosynth. Res. 91, 37–46. Wang, Y., Wang, D., Zhang, G., Wang, J., 2013. Estimating nitrogen status of rice using the image segmentation of G–R thresholding method. Field Crops Res. 149, 33–39. Yadav, S., Ibaraki, Y., Dutta Gupta, S., 2010. Estimation of the chlorophyll content of micropropagated potato plants using RGB based image analysis. Plant Cell Tissue Organ Cult. 100, 183–188.
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag
Development of an android app to estimate chlorophyll content of corn leaves based on contact imaging q Farshad Vesali a, Mahmoud Omid a,⇑, Amy Kaleita b, Hossein Mobli a a b
Department of Agricultural Machinery Engineering, Faculty of Agricultural Engineering and Technology, University of Tehran, Karaj, Iran Department of Agricultural and Biosystems Engineering, Iowa State University, Ames, IA, USA
a r t i c l e
i n f o
Article history: Received 17 March 2015 Received in revised form 12 June 2015 Accepted 12 June 2015
Keywords: Smartphones Chlorophyll content App SPAD Contact image
a b s t r a c t In this study a new android app for smartphones to estimate chlorophyll content of a corn leaf is presented. Contact imaging was used for image acquisition from the corn leaves which captures the light passing through the leaf directly by a smartphone’s camera. This approach would eliminate the needs for background segmentation and other pre-processing tasks. To estimate SPAD (Soil Plant Analysis Development) values, various features were extracted from each image. Then, superior features were extracted by stepwise regression and sensitivity analysis. The selected features were finally used use as inputs to the linear (regression) and neural network models. Performance of the models was evaluated using the images taken from a corn field located in West of Ames, IA, USA, with Minolta SPAD 502 Chlorophyll Meter. The R2 and RMSE values for the linear model were 0.74 and 6.2. The corresponding values for the neural network model were 0.82 and 5.10, respectively. Finally, these models were successfully implemented on an app named SmartSPAD on the smartphone. After installing the developed app on the smartphone, the performance of the models were evaluated again using a new independent set of data collected by SmartSPAD directly from maize plants inside a greenhouse. The SmartSPAD estimation compared well with the corresponding SPAD meter values (R2 = 0.88 and 0.72, and RMSE = 4.03 and 5.96 for neural network and linear model, respectively). The developed app can be considered as a low cost alternative for estimating the chlorophyll content especially when there is a demand for high availability. Ó 2015 Elsevier B.V. All rights reserved.
1. Introduction Determining the chlorophyll content of plants gives valuable information relevant to plant health and crop management. Chlorophyll is the main pigment in leaves and it is responsible for leaf greenness. Leaf colour is an indicator of plant health and also it can indicate plant nutrient status (Yadav et al., 2010; Muñoz-Huerta et al., 2013). For example, there is significant correlation between chlorophyll and nitrogen content of leaf tissues, thus by measuring chlorophyll content, nitrogen status can be assessed (Evans, 1989; Tewari et al., 2013). On the other hand, excess nutrients like nitrogen in an agricultural environment is a leading cause of water quality impairment (Turner and Rabalais, 1991). Therefore, managing and balancing agricultural nutrients
q Financial support by: Department of Agricultural Machinery Engineering, Faculty of Agricultural Engineering and Technology, University of Tehran, Karaj, Iran. ⇑ Corresponding author. Tel.: +98 912 3631832; fax: +98 2632808138. E-mail address: [email protected] (M. Omid).
http://dx.doi.org/10.1016/j.compag.2015.06.012 0168-1699/Ó 2015 Elsevier B.V. All rights reserved.
use has economic benefits in addition to reducing the risk of water and environment pollution (Daughtry et al., 2000; Sawyer et al., 2004). Destructive methods like Kjeldahl tissue analysis to determine nutrients status, in addition to their high costs, cannot be used as a way for variable rate fertilising (i.e., real-time application) because of time lag between collecting tissue sampling and obtaining results (Piekielek et al., 1995; Muñoz-Huerta et al., 2013). Chlorophyll meters (CMs) have been used by many researchers as a non-destructive method to measure the chlorophyll content and estimate the nitrogen value of agricultural crops (Richardson et al., 2002; Chang and Robison, 2003; Murillo-Amador et al., 2004; Scharf et al., 2006; Uddling et al., 2007; Miao et al., 2009). CMs use two wavebands to assess chlorophyll content, infrared light centred 930 nm and red light centred 650 nm (Blackmer et al., 1994). In recent years, additional non-destructive techniques based on spectral and hyperspectral reflectance have been investigated for estimating the chlorophyll content of plants. These methods were developed to achieve special purposes such as, real time and accurate nutrients status reporting (Feng et al., 2008; Fitzgerald et al.,
212
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
2010; Tian et al., 2011). Because chlorophyll content affects visual features of leaves, using digital cameras or in other words RGB (red, green, blue) imaging as a low cost instrument in the visible range has also been used in nitrogen status estimation (Dutta Gupta et al., 2013; Lee and Lee, 2013; Wang et al., 2013). Standard cameras are significantly lower in cost than other imaging systems and chlorophyll meters (Rorie et al., 2011a), which cost about $1500US, but there are some challenges in using them for this purpose; for example, different ambient lighting conditions or shadows on leaves will affect the images. To overcome these issues Tewari et al. (2013) used an experimental setup with a cover and an artificial light to estimate nitrogen content of a rice paddy crop. In addition, utilising independent indices extracted from images can be useful in reducing the effect of variation in light. Wang et al. (2013) used GMR (G–R), G/R, NGI (normalised green index), NRI (normalised red index) and Hue indices for estimating biomass, N content and leaf area index (LAI). They mentioned that in the GMR index, the colour of the plant canopy is sharply different from the background and it is feasible to set a threshold to segment rice plant from background. Moreover, GMR and G/R indices had a better correlation than the other indices to estimate biomass, N content and LAI. It seems that combinations of component values have better correlation with N content of plants. Karcher and Richardson (2003) found that the green value in RGB colour space cannot exactly represent how green the vegetation will appear, and that red and blue values also may change the appearance of the green colour of turfgrass. They introduce the dark green colour index (DGCI) based on HSB (hue, saturation, and brightness) colour space, and after calibrating HSB values, the DGCI showed a good correlation with N content. Further studies also reported the capability of DGCI to estimate N content of plants (Rorie et al., 2011a,b). In order to use any of the above strategies or similar, computer processing is required. With the advent of smartphones, the camera and the processor exist in the same device, opening new opportunities for image capture and data generation. Gong et al. (2013) developed and evaluated an app for android smartphones that can estimate the citrus yield two weeks before harvest time. They use a phone-implemented image processing technique for identifying fruits in an image of a tree by segmenting and clustering. Confalonieri et al. (2013) developed an app called PocketLAI, which used smartphone images to estimate LAI, one of the principal indices for assessing crop water requirements and photosynthetic primary production. The authors note that their approach is an inexpensive and highly portable alternative to commercially available LAI devices. Recently, developments in smartphones especially in their processors with built in sensors like cameras have brought us an opportunity that in addition to using their sensors as measurement tools, computation and analysis can be done on them without any additional attachment. Yet to date, no standalone android apps for measuring leaf chlorophyll content have been developed. In this study we designed and implemented an app for android smartphones named SmartSPAD to estimate the SPAD value of corn plants. In order to increase practicality and accuracy in real conditions, a new method of imaging is introduced. Overall performance of the app is compared with Minolta SPAD-502 chlorophyll meter.
2. Materials and methods 2.1. Data collection The data were collected from maize (Zea mays) plots at the Iowa State University Field Extension Education Laboratory, Ames, IA (USA) during the 2014 growing season. Various levels of
nitrogen deficiency were induced by using different fertiliser treatments: 0, 56, 112, 168, and 224 kg ha1 (0, 50, 100, 150, 200, pounds per acre). Each treatment was applied on two of the 55 m 27 m plots since 2011, and N treatment was replicated in 6 rows. Both plots of each treatment were corn–corn rotation; the east plot had no tillage and the west plot was ploughed by chisel in falls of 2012 and 2013. A set of validation data were collected from maize plants inside a greenhouse located in Iowa State University (Agronomy Department). These plants were fertilised by nitrogen at different levels and leaf images with corresponding SPAD observations were collected among them randomly. 2.2. Image acquisition and SPAD determination To take images from the plant leaves, a LG E975 smartphone with CCD sensor camera was used. To avoid or reduce effects of ambient conditions on images, a new method of smartphone imaging is presented which we refer to as contact imaging. In this method, unlike standard picture-taking, leaves are held to the camera lens of the smartphone and the camera captures the light passing through the leaf (Fig. 1). Compared to standard image capturing, this method of contact imaging has several advantages including: No interference from the background: One of the main steps in using image processing techniques in leaf imaging is segmenting the background. A variety of methods exist for distinguishing the target from the background, but the possibility of misclassification always exists (Teimouri et al., 2014), and even a small misclassification rate can affect the results. With contact imaging, however, there is no need to remove the background, and the entire image can be used as an input data. No variation in the distance between leaf and sensor: Generally, for robust use of cameras in either the lab or field, predefined distances are assumed; to keep this distance between the target and the sensor constant during image capture, frames or other setups have been used. Contact imaging eliminates the need for any other attachments. No differences in image focus or blur: The nature of contact images overcome this problem. Because in this method there is no space between leaf and camera, there is essentially no difference between a focused image and an unfocused image. Lower influence of different ambient conditions: Cloudy or sunny lighting conditions, shadows, and wind conditions can affect the success of using cameras in field conditions. Sun position also affects the reflectance of vegetation indices of plants (de Souza et al., 2010). But in contact imaging, only low lighting condition has some influence on the resulting images; as described below, we reduce this effect by adding luminance factor to the features used in computing the SPAD estimates. Other field conditions like shadow, sun position, and wind, have no effect on contact images. An additional benefit of contact imaging is that the effects of camera-to-camera variation are minimal. Because of the nature of contact imaging and the image processing described below, differences in sensor size or camera focusing algorithms will not have an effect on the results of the contact image. About 480 contact images in RGB colour space were captured by the smartphone and were transferred to a desktop computer for further analysis and model development. Images were intentionally taken from all plots under different light conditions (clear and cloudy sky) and at different times of the day. During image acquisition, field meteorological conditions including solar radiation, air temperature and relative humidity were varied from 200
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
213
of colour spaces, other combination indices suggested by the literature were calculated, including GMR (difference between green component and red component), GDR (green divided by red), VI (vegetation index), DGCI (dark green colour index), normalised red (NRI) and normalised green (NGI) (Kawashima and Nakatani, 1998; Gonzalez et al., 2009; Li et al., 2010; Rorie et al., 2011b; Lee and Lee, 2013; Tewari et al., 2013; Wang et al., 2013); their forms are given by Eqs. (8)–(13).
C ¼ max ðR; G; BÞ min ðR; G; BÞ 8 GB > < 60 C ; maxðR; G; BÞ ¼ R Hue ¼ 60 2 þ BR ; maxðR; G; BÞ ¼ G C > : 60 4 þ GB ; maxðR; G; BÞ ¼ B C ( Saturation ¼
Fig. 1. The way of contact imaging by smartphone from corn leaves.
to 750 (w m2), 20 to 30 (°C) and 65 to 95 (percent) respectively during the measurement periods according to The Weather Channel (http://www.weather.com). Fig. 2 shows five sample pictures that were taken by the contact imaging method from different N treatment plots. To determine the chlorophyll content of corn leaves, a SPAD-502 (Minolta Osaka Co., Ltd., Japan) was used. The SPAD value has been found to be well correlated with leaf chlorophyll content extracted through organic solvent method (Yadav et al., 2010). Chlorophyll meters calculate the SPAD value by the ratio of transmitted red light and infrared light, and this SPAD value has exponential relation (Eq. (1)) with chlorophyll content of plants (Markwell et al., 1995).
Chl ¼ 10M
^0:265
ð1Þ 2
where Chl is the chlorophyll content in unit of lmol m and M is the SPAD value measured by Minolta SPAD 502. After taking each image, a corresponding SPAD value was recorded from the same corn leaf position that at which the camera image had been captured. The range and average of SPAD values from different fertilisation plots are shown in Fig. 3. In our study the SPAD meter data were assumed to be perfect measures of chlorophyll content for the purpose of calibrating and validating the contact image model. 2.3. Feature extraction Most mobile phones take images in RGB colour space with 8 bit depth, which means for each component, the value is integer number between 0 and 255. Our images were in 8-bit RGB mode, and they were saved in jpeg (joint photographic experts group) format. For each image Hue, saturation, and brightness from HSB colour space, and Y (relative luminance), Cb (difference between the blue component and a reference value) and Cr (difference between the blue component and a reference value) from YCbCr colour space were extracted by Eqs. (2)–(7). In addition to these main channels
0;
V¼0
C ; maxðR;G;BÞ
V–0
ð2Þ
ð3Þ
Brightness ¼ maxðR; G; BÞ=255
ð4Þ
Y ¼ 0:257 R þ 0:504 G þ 0:098 B þ 16
ð5Þ
Cb ¼ 0:148 R 0:291 G þ 0:439 B þ 128
ð6Þ
Cr ¼ 0:439 R 0:368 G 0:071 B þ 128
ð7Þ
GMR ¼ G R
ð8Þ
GDR ¼ G=R
ð9Þ
VI ¼
GR GþR
DGCI ¼ ½ðHue=60 1Þ þ ð1 SÞ þ ð1 BÞ=3
ð10Þ ð11Þ
NRI ¼
R RþGþB
ð12Þ
NGI ¼
G RþGþB
ð13Þ
where R, G and B in the above equations denote the average values of the red, green and blue components of each contact image, respectively. For all indices including, R, G, B and indices calculated by Eqs. (2)–(13), the coefficient of determination (R2) and root mean square error (RMSE) of estimated SPAD values were obtained by a Matlab 8.2 program (MathWorks, Inc, Natick, MA, USA) installed on desktop computer. In order to find a better and more accurate combination of colour space components for estimating SPAD value, statistical and artificial intelligence models were used. Input variables for these models included: mean of Red, Green, and Blue, from RGB colour space, mean of Hue, Saturation and Brightness from HSB colour space and mean of Y, Cb, and Cr from YCbCr colour space. In addition to means, standard deviations for each feature were included in the features dataset. To account for variations in the amount of light during image capturing, an additional variable, luminance factor, was also included. Hiscocks and Eng (2011) measured luminance for regular digital cameras with the following equation: 2
L¼
Nf KtS
ð14Þ
where N is the maximum pixel value in an image, f is the relative aperture or F number, t is shutter speed or exposure time, S is ISO or sensor light sensitivity, and K is the camera constant. The
214
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
a
b
c
d
e
Fig. 2. Contact images from corn leaves from (a) 0 kg ha1, (b) 56 kg ha1, (c) 112 kg ha1, (d) 168 kg ha1 and (e) 224 kg ha1, N treatment captured by smartphone.
70
SPAD Value
60 51.53
50
54.33
42.28
40 30
52.47
Mean
27.97
20 10 0
Zero kg/ha
56 kg/ha
112 kg/ha
168 kg/ha
224 kg/ha
Max Min Average Fig. 3. Maximum, minimum and average of SPAD value in different nitrogen treatment in corn field.
constant K is calculated through calibration of the camera, but the other factors are stored in the EXIF (Exchangeable image file). This file is attached to each image and is accessible both during and after taking images. F number is the ratio of the lens’s focal length to the diameter of the camera’s entrance pupil; both of these two parameters are fixed in most smartphones, so this can be treated as a constant. The maximum pixel value of an image (N) in regular images can be useful because regular images contain different elements so to that different intensities of light will show up in image components, but in contact imaging the most image has nearly the same value for all pixels. In the other words the pixel standard deviation in contact images are low (Table 1), therefor maximum pixel value in contact images does not represent whiter pixels as in regular images. Thus for contact imaging, variations in luminance from image to image depend only on differences in sensor sensitivity and exposure time. The modified luminance factor used in this study is therefore:
LF ¼
Table 1 Range values (minimum and maximum) of mean and standard deviation of all extracted features. For all but LF, values represent the range across all images of mean and standard deviation of pixel values in an image. For LF, each image has but a single value, so range and standard deviation are over the whole image dataset.
1 tS
ð15Þ
Use of luminance factor also ensures minimal differences between cameras on other android phones, because any differences in exposure time or ISO speed will be accessible through the EXIF file data. Overall minimum and maximum values of mean and standard deviation of all 19 features that were extracted from contact images are shown in Table 1. 2.4. SPAD value estimation 2.4.1. Linear modelling A multivariable linear model was developed to estimate SPAD value of corn leaves. Altogether, 19 features were extracted from each image to develop a linear model: mean and standard deviation of R, G, B, H, S, Br, Y, Cb, and Cr, plus LF. Since the main goal was developing a model to implement in smartphones, the number of variables should be reduced in order to reduce processing time.
Red Green Blue Hue Saturation Brightness Y Cb Cr LF
Standard Deviation
Min
Max
Min
Max
0.17 128.62 0 9.99 0.69 0.50 98.02 54.06 66.13 0.0075
157.28 190.75 48.46 20.81 1 0.75 145.31 91.35 139.19 1.22
0.49 2.20 0 0.04 0 0.009 1.77 0.92 0.58 0.2852
22.57 17.49 29.72 1.15 0.18 0.07 13.08 12.57 8.42
Furthermore, with more input variables, the possibility of overfitting the data increases. Therefore a feature selection method was applied to select the best features to estimate chlorophyll content (SPAD value) of leaves. Numerous statistical methods to find superior features exist, such as LASSO, stepwise regression, wrapper methods, sensitivity analysis, and correlation-based feature selection (Saeys et al., 2007; Mollazade et al., 2012; Lee and Lee, 2013; Teimouri et al., 2014). In this study to develop a linear model we used stepwise regression to choose the most reliable and effective features. In this method, terms are added to or removed from the multivariable model based on their statistical significance level. To accomplish stepwise regression between variables, MATLAB software installed on a desktop computer was used. Stepwise feature selection was developed on 70% of the image dataset, and the remaining data were used to evaluate the performance of developed model.
2.4.2. Neural network modelling All 19 extracted features (mean and standard deviation of R, G, B, H, S, Br, Y, Cb, and Cr, plus LF) from contact images were used again to create and train predictive neural networks. Between different kinds of neural networks, perceptron is the most common and popular topology in both prediction and classification (Heaton, 2008; Omid et al., 2010) and is used here. To avoid overfitting and enable faster computation, feature selection was applied among the 19 inputs for the neural network too. Here sensitivity analysis was used to rank parameters in order of influence so that the most influential parameters could be determined. Sensitivity analysis attempts to provide a ranking of the model inputs based on their relative contributions to model output variability and uncertainty. For calculating the sensitivities, each input (feature vector) was shifted slightly and the corresponding change in the output was computed (Teimouri et al., 2014). The number of hidden layers was minimised based on the observation that one hidden layer neural network can approximate any function that contains a continuous mapping from one finite space to another.
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
The number of nodes in each hidden layer were chosen by trial and error and the rule-of-thumb that the number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer (Heaton, 2008). Initially a MLP (multi-layer perceptron) neural network was trained with all features. Then by sensitivity analysis the best features were selected, and a MLP with new architecture based on the selected input features was trained and evaluated using the same 70% and 30% of contact images respectively as the linear model development and evaluation. The learning algorithm in both neural networks was Levenberg Marquardt. This algorithm is one of the fastest optimisation backpropagation (BP) algorithms and is recommended as a first-choice supervised algorithm to update weights and biases in neural networks (Demuth and Beale, 1993). 2.4.3. Performance analysis To evaluate performance of the proposed methods (linear model as well as neural networks), coefficient of determination (R2) and RSME were computed for the portion of the dataset not used during the training process (Taghadomi-Saberi et al., 2013). In this study we used form of R2 which measures the coefficient of determination of models (Ramsey and Schafer, 2012) rather than of a linear trend line fitted through the data. 2.5. Implementing and characterizing the app The SmartSPAD application oversees taking contact images with the smartphone’s camera and storing them in the smartphone memory. To avoid crashing due to large image file sizes, each image is resized by bicubic interpolation to VGA dimension (640 480), or near this dimension based on its original aspect ratio. After invoking an image, the SmartSPAD application examines the image based on a uniformity characteristic to ensure the image is a contact image. Contact images of a plant leaf are highly uniform, and thus low uniformity suggests the image may be non-contact. Uniformity U can be extracted from image by the following equation (Gonzalez et al., 2009):
U¼
m 1 X
p2 ðBrightnessi Þ
ð16Þ
i¼0
where p(Brightnessi) is the histogram of the brightness levels in image, and m is the number of possible brightness levels. Using our image database, we determined a minimum threshold of uniformity corresponding to contact images. If image uniformity is below the minimum threshold, SmartSPAD suggests that the operator take another picture. When the image become ready to process, depending on which model is selected to estimate the SPAD value, features will extracted and the SPAD value calculated through the model. The calculated value of SPAD will show on screen and operator can save the value (Fig. 4). To start another estimation, the user may take another image and the process will repeat. During android application development stage, ‘‘activities1’’ are used in order to do something by smartphones. These allow user to interact with the phone and to do particular task. Therefore, depending on the type of application and its intended function, some number of activities must be created (Meier, 2012). Specifically, the developed SmartSPAD application contains four main activities including the main menu, SPAD estimation, data, and information. The menu activity is simply a connector activity and is merely used to select the other activities. To operate the app appropriately on any
1 An Activity is an application component that provides a screen with which users can interact in order to do something.
215
other phones with different screen size and resolution, an adaptive layout for each activity was designed. As android app development is based on Java programing language, for each activity there must be a class2 file that contains the commands to operate that activity. In addition to having a class file for each activity, there are four additional classes including the image processing class which converts images to other colour spaces and extracts input features for further processing by neural network class and linear model class, to compute the SPAD value through them and database class for saving the SPAD values. Programing to develop SmartSPAD application was done on Android Studio 64 bit version 8 which included the accessory development kit (ADK) and software development kit (SDK) for android. Platform of android API 19 (KitKat or Android 4.4.2) was chosen as the target platform but for compatibility with other mobile phones, using it on API 14 (Ice-cream sandwich) is possible. It should be noted that about 88 percent of all android phones are operated with this version or later (Statista, 2014). After finishing the app development, SmartSPAD was successfully installed on the mobile phone. Finally, in order to validate the performance of the installed app on the smartphone, about 60 new images were taken from maize plants cultivated inside a greenhouse, in which the light condition was a combination of day light and artificial light. These images were captured independently through the developed app, and processed on the phone using the app itself. SmartSPAD estimates were then downloaded for comparison to corresponding SPAD meter measurements for each image. None of these images were used during the app development. During this stage of app evaluation, criterions used for performance evaluation were R2 and RMSE as before.
3. Results and discussion 3.1. Correlation of different indices with SPAD values The mean value of R, G, B and all indices explained in Section 2.3 were compared with SPAD obtained by Minolta SPAD-502. As previously reported (Yadav et al., 2010; Tewari et al., 2013), B was poorly fitted to chlorophyll content, while R showed a good agreement with chlorophyll content. But in this study, G component which others have reported having good relation with chlorophyll content, was poorly fitted to SPAD value. It seems in our contact imaging of corn leaves, the Green and Blue components are almost fixed (having relatively very low variation from one image to the next) and only the red is fitted with SPAD value (Fig. 5). Among the other indices, Hue index had the strongest linear relationship with SPAD value (R2 = 0.57, RMSE = 7.7) (Fig. 6.). All other indices, including DGCI that has been used as one of the main indices for estimation chlorophyll content (Karcher and Richardson, 2003; Rorie et al., 2011a,b; Lee and Lee, 2013), were not strongly fitted to SPAD value (Table 2). 3.2. Estimation of SPAD values by linear model From the 19 features, the stepwise algorithm with a p = 0.01 significance level threshold selected 4 superior features as input variables for linear model. The selected features were luminance factor, mean of hue, standard deviation of hue, and standard deviation of Cr. Coefficient and p values of each feature are shown in Table 3. The resulting expression for estimation SPAD values (with R2 = 0.74 and RMSE 6.20) is:
2 A class, in the context of Java, is a template that is used to create objects, and to define object data types and methods.
216
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
Fig. 4. Flowchart shows the procedure of SmartSPAD app to estimate SPAD value.
80
R² = 0.04
70
R² = 0.11
SPAD Value
60 50 40 30
R² = 0.56 20 10
0
20
40
60
80
100
120
140
160
180
200
Mean of R, G, or B Fig. 5. Correlation between mean of Red (d), Green ( ) and Blue ( ) indices with SPAD value.
SPAD ¼ 22:48 LF þ 2:06 Huemean þ 8:72 Huestd þ 0:73 Crstd þ 14:62 ð19Þ
The p-values for luminance factor and hue component were almost zero, demonstrating that these are the two most important features. Overfitting is evident in the all-features linear model.
Despite the higher value for R2 in training mode, in test mode this value is lower than for the stepwise linear model which has just four features (Fig. 7). In both train mode and test mode, the linear model with 4 selected features had lower RMSE and higher R2 than did the best single index, which was hue. Since the range of actual and
217
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
80
R² = 0.59 70
SPAD Value
60 50 40 30 20 10
8
10
12
14
16
18
20
22
Mean of Hue Fig. 6. Correlation between mean of Hue index with SPAD value.
Table 2 Regression parameters (slope and intercept), coefficient of determination and root mean square error of different indices correlated to SPAD value. Hue index is the more correlated to SPAD value than other indices (bold row). Indices
Red Green Blue Hue Saturation Brightness Y Cb Cr GMR GDR VI DGCI NRI NGI
Regression parameters
RMSE 2
Slope
Intercept
R
0.30 0.63 0.43 4.63 66.0 159.91 1.05 1.42 0.65 0.28 0.02 47.30 296.7 95.69 93.73
71.99 55.11 44.77 22.30 110.85 54.86 171.1 53.55 115.39 24.75 45.81 30.67 82.34 78.02 15.36
0.56 0.09 0.04 0.59 0.04 0.11 0.39 0.37 0.57 0.57 0.003 0.47 0.20 0.48 0.43
7.72 11.0 11.42 7.46 11.44 11.01 9.07 9.28 7.79 7.65 11.63 8.30 10.42 8.18 8.60
estimated SPAD is almost the same as in Yadav et al. (2010), the RMSE values can be directly compared. They used a linear model with different input variables; in the best situation the RMSE value was 6.6 in training, higher than our linear model RMSE (5.8 and 6.2 for train and test mode respectively). Their images were taken by a scanner in a fixed lab condition. 3.3. Estimation of SPAD values by neural network model At first a neural network having all 19 features as inputs and one hidden layer which contains 14 nodes, was trained on the same training dataset (70% of data) as was used to develop the linear model, and its performance was evaluated with remaining
(test) data. Sensitivity analysis selected superior features by feature sensitivity value; in other words the cause and effect relationship between the inputs and output of the network were extracted, and features ranked based on their importance in SPAD estimation. The input channels that produce low sensitivity values can be considered insignificant and most often can be removed from the network. In this case the four highest ranked features including hue, luminance factor, standard deviation of Cr and standard deviation of hue were selected for designing the new neural network structure (Fig. 8). Three of those features were also selected by the stepwise regression. Accordingly, another MLP neural network was trained using the four selected features with four neurons in one hidden layer (Fig. 9) Based on performance evaluation by RMSE and R2, the multi-layer perceptron networks gave better results than the linear model with both all features and four selected features as an inputs (Table 4). As with the linear model, overfitting problem also showed its effect in train mode; the error value in train mode in the primary neural network is lower than in test mode. The neural network with selected features, however, estimated SPAD value accurately in test mode (Table 4 and Fig. 10). Neural network model showed a strong capability (RMSE = 5.1 and R2 = 0.82) to estimate SPAD value using just four superior features. In addition to reducing overfitting problem by using fewer features, less computation is needed to calculate the estimated SPAD value. Liu et al. (2010) reported that BP neural networks can estimate chlorophyll content of rice better than a linear model. Their inputs were spectral features and they achieve 0.71 and 0.90 for R2 value with linear model and neural network respectively. Although the R2 of their neural network model is better than in this study, in addition to using a spectrometer to gather data from field experiments, which is more expensive than a smartphone, their spectral measurements were under cloudless or near cloudless conditions, whereas the measurements in this study were made under a range of conditions. 3.4. Implementation of the models on android smartphone In the SmartSPAD application, to avoid memory leak as was mentioned, contact images were resized before doing any processing on them. Since the means of extracted features were used as an input variables and bicubic interpolation is based on averaging of 4 by 4 neighbourhood pixels, the mean values will not change after resizing. Small changes in standard deviations of different features were observed which was predictable. Before extracting features to process through linear model or neural network, the application examines the uniformity characteristic of each image in order to determine if the picture has the quality of a contact image. In entire our image database, the lowest value for the uniformity was 0.022 (in comparison to 0.0058 for an example regular image). Therefore the SmartSPAD offers the operator to take another picture if the computed uniformity for an image if this value is less than 0.015. To estimate SPAD value from contact images, while the neural network gave better results, the linear model was also implemented in the Beta version of the SmartSPAD application for
Table 3 Coefficients of linear model with superior features and compares its performance with linear model with all feature in train and test mode. Coefficients
Linear model with 4 selected features p value Linear model with all features
Train mode
Test mode
LF
Hue
STD of Hue
STD of Cr
Intercept
R2
RMSE
R2
RMSE
22.48 5.81e47
2.06 1.53e35
8.72 4.08e04
0.73 3.4e03
14.62 –
0.75
5.8
0.74
6.2
0.77
5.69
0.68
6.9
218
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
80
80
RMSE= 6.2
RMSE= 6.9
60
Actual SPAD value
Actual SPAD Value
60
40
R 2 = 0.68 20
0
40
R 2 = 0.74 20
0
20
40
60
80
0
0
20
40
60
Estimated SPAD Value
Estimated SPAD Value
a
b
80
Fig. 7. Correlation in test mode between actual SPAD value and the SPAD values estimated by (a) linear model with all features, (b) linear model with 4 superior features.
Table 4 Performance evaluation of neural networks.
4.5 4
Structure
Sensitivity
3.5 3
Neural net with all features Neural net with 4 features
2.5
19-14-1 4-4-1
Train mode
Test mode
RMSE
R2
RMSE
R2
4.1 4.82
0.87 0.82
6.16 5.10
0.74 0.82
2 1.5 1 0.5 Lum Fact. Red STD Red Green STD G Blue STD Blue Hue STD Hue Sat. STD S Bright. STD B Y STD Y Cb STD Cb Cr STD Cr
0
Fig. 8. Sensitivity of different features in the neural network output (SPAD value).
further evaluation. Processing time per contact image for both the neural network and the linear model was less than 2 s on a quad core 1.5 GHz Snapdragon smartphone. It should be noted that in this application, just two parallel process were used so it can be easily run in any dual core smartphones as well.
In the SmartSPAD app, three main activities (pages) do the task of capturing image, measuring the approximate SPAD value and storing the data (Fig. 11). The menu activity contains menu lists of capturing or browsing images, previous records and information about the application (Fig. 11A). SPAD estimation activity gets the contact image and estimates the SPAD value by neural network or linear model, and if operator wants to save the value, it can be saved (Fig. 11B). Data activity (page) contains list of data, and the operator can get the specific average value of the data generated by each method (Fig. 11C). In addition the data can be exported as a CSV file and the file shared with other applications like email, to send the exported file via email to any other devices. 3.5. Evaluation of SmartSPAD app For the validation (greenhouse) data, every SPAD value estimated by the SmartSPAD application installed on the phone, was
Fig. 9. BP neural network flowchart with 4-4-1 structure.
219
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
80
80
RMSE= 6.16
RMSE= 5.10 60
Actual SPAD value
Actual SPAD value
60
40
20
0
20
40
60
R 2 = 0.82
20
R 2 = 0.74
0
40
80
0
0
20
40
60
Estimated SPAD value
Estimated SPAD value
a
b
80
Fig. 10. Correlation in test mode between actual SPAD value and SPAD value estimated by (a) neural network with all features, (b) neural network with 4 superior features.
a
b
c
Fig. 11. SmartSPAD application developed on android smartphones. (a) Menu activity. (b) SPAD estimation activity or calculation page. (c) Data activity.
recorded using the embedded data logging section in the app. At the end of the app calculation, results were transferred to a PC to be compared with the measured SPAD values by Minolta SPAD 502 (Fig. 12). The values of R2 and RMSE (R2 were 0.88 and 0.72, and RMSE were 4.03 and 5.96 for neural network and linear model, respectively) showed that the app had almost the same performance on the greenhouse data as on the development data. 4. Conclusion An app called SmartSPAD was developed for android smartphones to estimate the SPAD (soil Plant Analysis Development) value of corn plants. Overall, the proposed method of imaging
(contact image by smartphone) showed a strong agreement to estimate chlorophyll content (SPAD value) of corn leaves in real field conditions. Using data from the camera and derived from the contact images, a linear model and a neural network model were developed and tested on independent data. Hue and luminance factor were the most important features in these contact images. The R2 and RMSE value for validation data were about 0.74 and 6.2, and 0.82 and 5.10 for the linear model and neural network respectively. The results indicate this approach can be used to efficiently replicate the performance of a higher-cost SPAD meter. Implementing this method on a smartphone can be a practical alternative to measure chlorophyll content as low cost device. In addition to the lower cost, SmartSPAD has several other
220
F. Vesali et al. / Computers and Electronics in Agriculture 116 (2015) 211–220
80
Actual SPAD Value
60
40
20
R 2 = 0.74 R 2 = 0.88
0
0
20
40
60
80
Estimated SPAD value Fig. 12. Evaluation of SmartSPAD for both (N) neural network and () linear models against actual SPAD values, on images from maize plants inside a greenhouse.
advantages, like storing a large number of chlorophyll content values, or potentially logging the GPS data at same time, while Minolta SPAD 502 users encounter limitation in memory and no capability to store location position. Also, in comparison with other proposed devices which usually include digital cameras and computers, SmartSPAD is a stand-alone approach and because luminance factor is computed from the camera data, SmartSPAD compensates for different lighting conditions, requiring less pre-processing and giving accurate results. Acknowledgment The financial support provided by the Research Department of University of Tehran, Iran, under Grant No. 1305051.6.29 is duly acknowledged. References Blackmer, T.M., Schepers, J.S., Varvel, G.E., 1994. Light reflectance compared with other nitrogen stress measurements in corn leaves. Agron. J. 86, 934–938. Chang, S.X., Robison, D.J., 2003. Nondestructive and rapid estimation of hardwood foliar nitrogen status using the SPAD-502 chlorophyll meter. For. Ecol. Manage. 181, 331–338. Confalonieri, R., Foi, M., Casa, R., Aquaro, S., Tona, E., Peterle, M., Boldini, A., De Carli, G., Ferrari, A., Finotto, G., Guarneri, T., Manzoni, V., Movedi, E., Nisoli, A., Paleari, L., Radici, I., Suardi, M., Veronesi, D., Bregaglio, S., Cappelli, G., Chiodini, M.E., Dominoni, P., Francone, C., Frasso, N., Stella, T., Acutis, M., 2013. Development of an app for estimating leaf area index using a smartphone. Trueness and precision determination and comparison with other indirect methods. Comput. Electron. Agric. 96, 67–74. Daughtry, C.S.T., Walthall, C.L., Kim, M.S., de Colstoun, E.B., McMurtrey, J.E., 2000. Estimating corn leaf chlorophyll concentration from leaf and canopy reflectance. Remote Sens. Environ. 74, 229–239. de Souza, E.G., Scharf, P.C., Sudduth, K.A., 2010. Sun Position and Cloud Effects on Reflectance and Vegetation Indices of Corn. Agron. J. 102, 734. Demuth, H., Beale, M., 1993. Neural network toolbox for use with MATLAB. Dutta Gupta, S., Ibaraki, Y., Pattanayak, A.K., 2013. Development of a digital image analysis method for real-time estimation of chlorophyll content in micropropagated potato plants. Plant Biotechnol. Rep. 7, 91–97. Evans, J., 1989. Partitioning of nitrogen between and within leaves grown under different irradiances. Funct. Plant Biol. 16, 533–548. Feng, W., Yao, X., Zhu, Y., Tian, Y.C., Cao, W.X., 2008. Monitoring leaf nitrogen status with hyperspectral reflectance in wheat. Eur. J. Agron. 28, 394–404. Fitzgerald, G., Rodriguez, D., O’Leary, G., 2010. Measuring and predicting canopy nitrogen nutrition in wheat using a spectral index—the canopy chlorophyll content index (CCCI). Field Crops Res. 116, 318–324. Gong, A., Yu, J., He, Y., Qiu, Z., 2013. Citrus yield estimation based on images processed by an android mobile phone. Biosyst. Eng. 115, 162–170. Gonzalez, R.C., Woods, R.E., Eddins, S.L., 2009. Digital Image Processing using MATLAB, second ed. Gatesmark Pub., S.I..
Heaton, J., 2008. Introduction to Neural Networks with Java. Heaton Research Inc.. Hiscocks, P.D., Eng, P., 2011. Measuring Luminance with a Digital Camera. Syscomp Electronic Design Limited. Karcher, D.E., Richardson, M.D., 2003. Quantifying turfgrass color using digital image analysis. Crop Sci. 43, 943. Kawashima, S., Nakatani, M., 1998. An algorithm for estimating chlorophyll content in leaves using a video camera. Ann. Bot. 81, 49–54. Lee, K.-J., Lee, B.-W., 2013. Estimation of rice growth and nitrogen nutrition status using color digital camera image analysis. Eur. J. Agron. 48, 57–65. Li, Y., Chen, D., Walker, C.N., Angus, J.F., 2010. Estimating the nitrogen status of crops using a digital camera. Field Crops Res. 118, 221–227. Liu, M., Liu, X., Li, M., Fang, M., Chi, W., 2010. Neural-network model for estimating leaf chlorophyll concentration in rice under stress from heavy metals using four spectral indices. Biosyst. Eng. 106, 223–233. Markwell, J., Osterman, J., Mitchell, J., 1995. Calibration of the Minolta SPAD-502 leaf chlorophyll meter. Photosynth. Res. 46, 467–472. Meier, R., 2012. Professional Android 4 Application Development. John Wiley & Sons. Miao, Y., Mulla, D., Randall, G., Vetsch, J., Vintila, R., 2009. Combining chlorophyll meter readings and high spatial resolution remote sensing images for in-season site-specific nitrogen management of corn. Precision Agric. 10, 45–62. Mollazade, K., Omid, M., Arefi, A., 2012. Comparing data mining classifiers for grading raisins based on visual features. Comput. Electron. Agric. 84, 124–131. Muñoz-Huerta, R.F., Guevara-Gonzalez, R.G., Contreras-Medina, L.M., TorresPacheco, I., Prado-Olivarez, J., Ocampo-Velazquez, R.V., 2013. A review of methods for sensing the nitrogen status in plants: advantages, disadvantages and recent advances. Sensors 13, 10823–10843. Murillo-Amador, B., Ávila-Serrano, N.Y., García-Hernández, J.L., López-Aguilar, R., Troyo-Diéguez, E., Kaya, C., 2004. Relationship between a nondestructive and an extraction method for measuring chlorophyll contents in cowpea leaves. Beziehung zwischen den mit einer nicht destruktiven und den mit einer Extraktionsmethode ermittelten Chlorophyllgehalten in Cowpea-Blättern. J. Plant Nutr. Soil Sci. 167, 363–364. Omid, M., Mahmoudi, A., Omid, M.H., 2010. Development of pistachio sorting system using principal component analysis (PCA) assisted artificial neural network (ANN) of impact acoustics. Expert Syst. Appl. 37, 7205–7212. Piekielek, W.P., Fox, R.H., Toth, J.D., Macneal, K.E., 1995. Use of a chlorophyll meter at the early dent stage of corn to evaluate nitrogen sufficiency. Agron. J. 87, 403. Ramsey, F., Schafer, D., 2012. The Statistical Sleuth: A Course in Methods of Data Analysis. Cengage Learning. Richardson, A.D., Duigan, S.P., Berlyn, G.P., 2002. An evaluation of noninvasive methods to estimate foliar chlorophyll content. New Phytol. 153, 185–194. Rorie, R.L., Purcell, L.C., Karcher, D.E., King, C.A., 2011a. The assessment of leaf nitrogen in corn from digital images. Crop Sci. 51, 2174–2180. Rorie, R.L., Purcell, L.C., Mozaffari, M., Karcher, D.E., King, C.A., Marsh, M.C., Longer, D.E., 2011b. Association of ‘‘greenness’’ in corn with yield and leaf nitrogen concentration. Agron. J. 103, 529. Saeys, Y., Inza, I., Larrañaga, P., 2007. A review of feature selection techniques in bioinformatics. Bioinformatics 23, 2507–2517. Sawyer, J.E., Barker, D.W., Lundvall, J.P., 2004. Using Chlorophyll Meter Readings to Determine n Application Rates for Corn. North Central Extension-Industry Soil Fertility, Des Moines, IA. Scharf, P.C., Brouder, S.M., Hoeft, R.G., 2006. Chlorophyll meter readings can predict nitrogen need and yield response of corn in the north-central USA. Agron. J. 98, 655. Statista, 2014. Android devices distribution platform versions 2014 | Statistic, Statista. Taghadomi-Saberi, S., Omid, M., Emam-Djomeh, Z., Ahmadi, H., 2013. Development of an intelligent system to determine sour cherry’s antioxidant activity and anthocyanin content during ripening. Int. J. Food Prop. 17, 1169–1181. Teimouri, N., Omid, M., Mollazade, K., Rajabipour, A., 2014. A novel artificial neural networks assisted segmentation algorithm for discriminating almond nut and shell from background and shadow. Comput. Electron. Agric. 105, 34–43. Tewari, V.K., Aruda, A.K., Kumar, S.P., Pandey, V., Chandel, N.S., 2013. Estimation of plant nitrogen content using digital image processing. Agric. Eng. Int.: CIGR J. 15, 78–86. Tian, Y.C., Yao, X., Yang, J., Cao, W.X., Hannaway, D.B., Zhu, Y., 2011. Assessing newly developed and published vegetation indices for estimating rice leaf nitrogen concentration with ground- and space-based hyperspectral reflectance. Field Crops Res. 120, 299–310. Turner, R.E., Rabalais, N.N., 1991. Changes in mississippi river water quality this century. Bioscience 41, 140–147. Uddling, J., Gelang-Alfredsson, J., Piikki, K., Pleijel, H., 2007. Evaluating the relationship between leaf chlorophyll concentration and SPAD-502 chlorophyll meter readings. Photosynth. Res. 91, 37–46. Wang, Y., Wang, D., Zhang, G., Wang, J., 2013. Estimating nitrogen status of rice using the image segmentation of G–R thresholding method. Field Crops Res. 149, 33–39. Yadav, S., Ibaraki, Y., Dutta Gupta, S., 2010. Estimation of the chlorophyll content of micropropagated potato plants using RGB based image analysis. Plant Cell Tissue Organ Cult. 100, 183–188.