Robust image processing algorithm for computational resource limited smart apple sunburn sensing system

Journal Pre-proofs Robust image processing algorithm for computational resource limited smart apple sunburn sensing system Guobin Shi, Rakesh Ranjan, Lav R. Khot PII: DOI: Reference:

S2214-3173(19)30067-8 https://doi.org/10.1016/j.inpa.2019.09.007 INPA 222

To appear in:

Information Processing in Agriculture

Received Date: Revised Date: Accepted Date:

8 March 2019 6 August 2019 27 September 2019

Please cite this article as: G. Shi, R. Ranjan, L.R. Khot, Robust image processing algorithm for computational resource limited smart apple sunburn sensing system, Information Processing in Agriculture (2019), doi: https:// doi.org/10.1016/j.inpa.2019.09.007

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

© 2019 China Agricultural University. Production and hosting by Elsevier B.V. on behalf of KeAi

Robust image processing algorithm for computational resource limited smart apple sunburn sensing system Guobin Shi a, b, ×, Rakesh Ranjana, ×, Lav R. Khota,* bDepartment

of Biological Systems Engineering, Center for Precision and Automated Agricultural Systems, Washington State University, Prosser, WA, USA

aHeilongjiang

Bayi Agricultural University, Daqing, 163319, People’s Republic of China

* Corresponding author: E-mail address: [email protected], Phone: 509-786-9302. × Equal contribution authors

Abstract Heat and light stress causes sunburn to the maturing apple fruits and results in crop production and quality losses. Typically, when the fruit surface temperature (FST) rises above critical limits for a prolonged duration, the fruit may suffer several physiological disorders including sunburn. To manage apple sunburn, monitoring FST is critical and our group at Washington State University is developing a noncontact smart proximal sensing system that integrates infrared and visible imaging sensors for real time FST monitoring. Pertinent system needs to perform in-field imagery data analysis onboard a single board computer with processing unit that has limited computational resources. Therefore, key objective of this study was to develop a novel image processing algorithm optimized to use available resources of a single board computer. Algorithm logic flow includes color space transformation, k-means++ classification and morphological operators prior to fruit segmentation and FST estimation. The developed algorithm demonstrated the segmentation accuracy of 57.78% (missing error = 12.09% and segmentation error = 0.13%). This aided successful apple FST estimation that was 10–18 ℃ warmer than ambient air temperature. Moreover, algorithm reduced the imagery data processing time cost of the smart sensing system from 87 s to 44 s using image compression approach. Keywords: Apple sunburn; Fruit surface temperature; Thermal-RGB imaging; On-board image processing; Kmeans++ clustering; Smart sensing systems

1

1. Introduction Washington State is the biggest producer of fresh market apples with 64% of total apples produced in the United States [1]. However, about 10–40% of annual apple crop yield loss has been reported due to sunburn [2]. Excessive heat and high intensity sunlight during summer causes sunburn disorder. It adversely affects the quality, appearance and yield of the produce. Previous studies have reported that fruit surface temperature (FST) can be used as a reliable indicator for assessing sunburn susceptibility in apples. When the fruit surface temperature surpasses the threshold limits (45–52 °C) for a definite time span, fruit may get sunburn [3–4]. Sunburn necrosis, sunburn browning, and photo-oxidative sunburn are three major categories of apple sunburn [5–6]. Excessive heating of the fruit peel and underlying tissues can surpass the tolerance limit during the ripening period which causes sunburn necrosis disorder. Sunburn necrosis can occur, when FST approaches above 52 ± 1 °C (mean ± std. dev.) for about 10 minutes. Sunburn browning can be caused by the combined effect of high FST and ultraviolet (UV)-B radiation of sun [7]. Sudden exposure of a shaded apple to sunlight can cause photo-oxidative sunburn and may propagate to necrosis disorder [4]. Apart from FST, sunburn susceptibility in apples depends upon several other factors including cultivars and water stress in crop [8–9]. Modern high-density orchards use dwarfing rootstocks and training systems which further increase the crop sunburn susceptibility. Traditional methods to monitor the sunburn event are based on weather reports or growers experiences which can often be unreliable. Existing technologies for sunburn protection involve the use of overhead sprinkler irrigation for evaporative cooling, protectant sprays and shade nets. Above technologies can be used either independently or in combinations [10–11]. Evaporative cooling is a widely adopted method of sunburn management but inaccurate sunburn event prediction may make them ineffective [12–13]. Moreover, unsupervised evaporative cooling may result in excess water and energy consumption and may also stimulate water-borne diseases and food safety risks. Accurate and real-time FST information is thus critical to actuate suitable sunburn protection measures.

In prior efforts, surface temperature probe, thermocouples, infrared thermometers,

thermal imagers and microclimate-based FST models have been explored for FST estimation [8,

2

14–15]. Infrared thermal imaging techniques have been used for several applications in agriculture including detection of apple bruises [16], crop water stress monitoring [17] and canopy temperature estimation [18]. Moreover, thermal imaging has also demonstrated a great potential for accurate FST assessment and has produced reliable FST estimation results compared to other measurement techniques [15, 19–20]. However, thermal imagers fail in distinguishing objects at similar temperature, which limits its unsupervised applications needing real-time decision-making [15]. In terms of apple sunburn monitoring, our group has developed and tested a sensing system prototype that uses thermal-RGB sensing for FST estimation [15]. A smart monitoring system with automated image acquisition, processing and real time apple FST estimation capability in varied light conditions, typical in agricultural farms and orchards, could help growers in real-time apple sunburn detection and subsequent actuation of prevention techniques. Field level implementation of such system requires it to operate with limited computing resources. In terms of apple FST estimation through real-time imagery data processing on such limited resource system, accurate segmentation of target object from the image has been a major challenge [21]. Most of the existing segmentation algorithms typically rely on color-based models for crop sensing including extraction of projected shoot area and geometric parameters. These models utilize various color spaces viz.; Red(R)-Green(G)-Blue(B) (RGB), hue(H)-saturation(S)-value(V) (HSV), International Commission on Illumination: luminance-color (CIELAB) and luminance-chrominance (YCbCr) for object segmentation. Similarly, mathematical manipulation of the acquired spectral bands such as R−G, R−B, G−B and normalized difference index (NDI) has been used for object segmentation [22–23]. A combination of color and texture features has been explored as well during artificial neural network based segmentation of fruits from canopy [24–26]). Above discussed methodologies for post- or real-time image processing describe the viability of these algorithms for image segmentation. Nevertheless, there are several shortcomings associated with these approaches. For instance, the RGB based color model is a device dependent and pixel values alter significantly as one uses different imaging sensors. Due to chrominance and luminance mixing, such models tend to become unreliable for illumination color change or shading conditions [27]. Hence, the RGB model does not perform well for segmentation with a complex background. To overcome above

3

limitations, other color spaces like HSV, YCbCr and CIELAB based models have been employed for segmentation. Such models are closer to the human perception. The RGB and HSV are non-uniform color models that produce distinct differences for the same numerical distance and are not very effective for color segmentation. Furthermore, YCbCr tends to have poor outcomes around edges and results in false color segmentation. CIELAB (denoted by L*, a*, and b* coordinates) [hereafter termed as LAB] is a uniform and device-independent color space. It consists of a wide range of color shades. Moreover, CIELAB is less vulnerable with curvature, shadows, and glossiness compared to other color spaces [28] and is appropriate for color comparison [29]. An artificial neural network is also a well-tested alternative for accurate segmentation [25]. However, this technique involves a supervise learning that needs a large training dataset and big computing memory footprint. Overall, the major drawbacks with most of the above-mentioned segmentation techniques are: lack of needed accuracy in complex background scenarios, costprohibitive commercial software/hardware requirements and cumbersome equipment involvement in the system. Hence, these techniques are not very suitable for practical and large-scale smart applications implemented on a single board or limited computing resource environments. To address pertinent technological gaps discussed above, the specific objective of this study were to: 1) Develop and implement a novel image processing algorithm onboard the single board computer for estimating apple fruit surface temperature using a smart in-field sensing system. 2) Compare the performance of the proposed algorithm with commonly existing image processing methods.

2. Materials and methods 2.1 Sunburn monitoring system A smart sensing system was developed for real-time sunburn monitoring in apples. A commercial thermal-RGB imager was integrated with a data acquisition, control hardware and

4

associated software for apple FST monitoring. Following sections describe aspects of the involved hardware required to build a smart sensing system. 2.1.1 Thermal-RGB imager A thermal-RGB imaging sensor (FLIR Duo R, FLIR Systems, Inc., Oregon, USA) was selected for apple FST monitoring. This sensor consists of a focal plane array-based uncooled microbolometer infrared sensor of resolution 160×120 pixels and a visible (RGB) sensor of resolution 1920×1080 pixels. This sensor can operate in temperature range of 0 to + 50 °C and a measurement accuracy of ± 5 °C. It can be remotely triggered using pulse-width modulation (PWM) control. In this study, PWM-enabled general-purpose input/output (GPIO) pins of the ‘Raspberry Pi®’ were utilized for automatic triggering of the sensor. The universal serial bus (USB) port available with this sensor can be used to access image data stored on-board the sensor SD card. 2.1.2 Data acquisition and processing module A single board computer (Raspberry Pi®, V3B, Raspberry Pi® foundation, Cambridge, UK) was used for real-time data acquisition, and imagery data processing. Raspberry Pi® operates with ARM Cortex-A53, 1.2 GHz processor and supports ARM GNU/Linux/Windows 10 internet of things (IOT) platform. GPIO pins available on the computer can be used for PWM as well as serial port control. Moreover, four USB ports can be utilized for connection, communication and power supply to the peripheral devices. It works on Linux operating system (OS) and can run ‘PythonTM’ with an open source computer vision (OpenCV) library for system control and machine vision applications. 2.2 Onboard image processing algorithm A customized image processing algorithm was developed in ‘OpenCV-Python’ on the Raspberry Pi® platform for automated and accurate FST estimation. The radiometric image obtained from thermal-RGB imager is a combined thermal and RGB image (Fig. 1). In the implemented algorithm (Fig. 2), a RGB image was used to segment the apple from canopy. Two different color spaces (LAB and RGB) were individually adopted in the algorithm to test the precision of segmentation. The size of the thermal image was then enlarged by 9 times to make it suitable for

5

overlapping with the RGB image. Additionally, classification techniques and morphological operators were implemented in the algorithm to improve its versatility for automated image processing. The resulting segmented RGB image was overlapped with the thermal image to get pixelated temperature of the segmented apple and finally apple FST was retrieved.

Fig. 1. Image fusion for fruit surface temperature estimation

6

Fig. 2. Image analysis algorithm process flow for fruit surface temperature estimation using thermal-RGB imagery data

2.2.1 Color space transformation As preliminary test, RGB and CIELAB color space were used for fruit segmentation. A sample apple image was downloaded from the web (source: www.centuryfarmorchards.com) For RGB based segmentation, red (R) channel was subtracted with green (G) channel (R−G) followed by the Otsu thresholding technique. In LAB color space based segmentation, only a* (=red/green) and b*(green/blue) color components were used for segmentation [21]. The obtained segmented image from both approaches was visually analyzed to ascertain the suitability of these color spaces.

7

2.2.2. Fruit-background classification Color and spatial features based unsupervised or supervised classification technique can be used to overcome the discussed limitations of color segmentation in automated image processing [31]. Clustering is an unsupervised learning approach that groups the alike data points in a cluster. This technique does not need labeled data points. Moreover, clustering also eliminates the requirement of manual thresholding for appropriate segmentation. The modified k-means++ unsupervised classification algorithm was implemented for automated image processing. The original k-means algorithm separates a dataset S {δ1,δ2,…,δN} into k-clusters {ε1,ε2,…,εk}, based on minimum distance between each data point and its nearest centroid to have minimum sum of square error (SSE) (equation 1). K SSE=∑i = 1∑δ

dist2(Mi, δ)

∈ ε

(1)

Where, Mi {M1, M2, …, Mk } is the set of cluster mean, and [𝑑𝑖𝑠𝑡2(𝑀𝑖, 𝛿)] is the Euclidean distance that gives an approximate measure of the relative perceptual distance between two colors. However, the k-means algorithm has been reported to be very sensitive with selection of initial centroid [32]. In case of random selection of initial centroids, k-means often leads to the local minimum solution instead of the global minimum solution which does not guarantee the unique clustering results. Hence, a modified k-means++ technique was adopted to overcome above limitations. This algorithm also selects a random centroid at first; however, it keeps updating this initial seed values with maximum distance value from all existing centroids, until the last centroid is found [33–34]. Number of clusters (k) plays a critical role in clustering efficiency. In the developed algorithm, apples, leaves, branches, soil and other background objects carry different colors and hence such objects were categorized in five different clusters ranging from 2 to 5 for effective segmentation. Influence of k-value on classification accuracy and image segmentation was tested on 45 randomly selected apple images acquired from web (Google Images, Google Inc., California, USA) containing fruit, leaves, and branches. k-means++ clustering was performed to have each image grouped in 2, 3, 4 and 5 clusters. A method proposed by Meyer and Neto (2008) [35] was adopted for estimation of classification accuracy (%) (equation 2).

8

Classification accuracy =

∑𝑛 = 𝑚𝐴(𝑛) ∩ 𝐵(𝑛) 𝑛=1 ∑𝑛 = 𝑚𝐴(𝑛) ∪ 𝐵(𝑛) 𝑛=1

× 100

(2)

Where, n is apple blobs in foreground, m is number of blobs in foreground, A(n) is the real area of foreground, B(n) is the area of foreground from proposed algorithm. After identification of appropriate k-value for various images, next step was to segment the apple from rest of the canopy. To eliminate influence of backgrounds of obtained clustered images on the processing, it was replaced with white color. Different combinations of RGB pixels (R ― G, R ― B,

R―G

R ― B,

G ― B,

R―G

G ― B) obtained after classification were then tested to achieve best segmentation

results as suggested in previous studies [30, 36–37]. Therein, R, G, and B represent the average value of color components in a set of the clusters. These combinations were tested for all 45 images to develop appropriate segmentation equations (equations 3 and 4). R―G >0 εi(max

| |) R―G G―B

(3) (4)

2.2.3 Morphological operations Various morphological operators were implemented in the algorithm to improve the accuracy of the segmentation by removing various noises. Each of the classified images was first converted into greyscale and then thresholding of 0.9 was applied to convert the image into a binary image. Smaller blobs, sets of connected pixels with area 32 times smaller than the maximum area, were then removed from the image to eliminate small noise. Additionally, a hole-filling operator was implemented to fill the small holes in the image [21]. Region matching between the binary image and the original RGB image was then performed to extract a RGB apple blob. The resulting image was converted into a greyscale image. A 3 × 3 kernel blur filter was applied on the greyscale image to eliminate the effect of the local minimum in the segmented apple and an erosion operator was implemented to separate the slightly connected apples [38]. Adaptive thresholding [39] was applied on the greyscale image to convert it to a binary image. A distance transformation [40] was then applied on the binary image with a threshold of 0.3×Dmax to extract a precise foreground area of objects; where, Dmax is the maximum distance transform value

9

to the edge. Finally, a marker-controlled watershed algorithm [41] that labels the edge of apples and background was implemented so that a bunch of apples that forms an amalgamated blob can be separated into individual apples for precise overlapping and FST estimation.

2.2.4 Performance evaluation of segmentation algorithm The performance of the developed segmentation algorithm, henceforth termed as algorithm-3, was evaluated by comparing the calculated apple area with actual area of apple in the image. Firstly, out of 45 images acquired for estimation of classification accuracy, 10 were randomly selected. Actual foreground (apple) and background (objects other than apple) area of selected images, were then calculated with the help of an image-editing tool (Adobe Photoshop CS6, Adobe Inc., California, USA ). Additionally, the developed image processing algorithm was implemented on sample images to compute apple blob area. Finally, equations 5 and 6 were used to calculate percent segmentation error (SE, %) and missing error (ME, %). SE =

Sbg ∩ Sout

× 100

Sfg ∪ Sbg Sfg ― Sfg ∩ Sout

ME =

Sfg

× 100

(5) (6)

Where, 𝑆𝑓𝑔 is the actual foreground region of apple in an image, 𝑆𝑏𝑔 is the actual background region in an image, and 𝑆𝑜𝑢𝑡 is foreground region. In addition, two widely used algorithms [R−G>40 (hereafter termed as ‘algorithm-1’) and 2R−G−B using adaptive threshold (hereafter termed as ‘algorithm-2’)] [36] were tested for segmentation of images. The SE and ME estimated for algorithm-1, algorithm-2 was compared with that of the algorithm-3.

2.2.5 Thermal-RGB image overlapping A pixelated offset between the overlapped thermal and RGB image can exist due to eccentricity between the two sensors. Hence, horizontal and vertical image translation was performed on such images. Due to optical dispersion, it is necessary for sensor optics to focus on a minimum cluster of at least 3 × 3 pixels with one pixel on either side of the central pixel for acceptable measurement

10

accuracy [42]. At minimal 10 pixels are required to capture temperature in range of 40–50 °C at 1 °C temperature interval and two additional pixels are required on each side. The distance between the apple and sensor was restricted to 1 m for further experiments because beyond 1 m, number of pixels on a segmented apple was found to be less than 12 pixels (Fig. 3).

Fig. 3. The thermal and corresponding RGB image acquired from 0.5 m, 1 m, 1.5 m and 2 m

2.2.6 Calibration of thermal-RGB imager A blackbody calibrator (model: BB701, Omega Engineering, Inc. USA) with target plate diameter of 64 mm, operational temperature of −18 to 149 °C and target plate emissivity of 0.95 was used for blackbody calibration. The target plate temperature was first set at 35 °C and images were acquired from 0.5 m, 1 m, 1.5 m. A similar process was repeated for target plate temperatures of 37, 39, 41, 43, 45, 47 and 49 °C. A FLIR® Tool software (FLIR® Systems, Inc., Oregon, USA) was used to extract the target plate temperatures in the images. The temperature data at five different points on the target plate was averaged to get a mean target plate temperature. A linear regression analysis was performed between actual and measured temperature data and a calibration equation was established individually for three image acquisition distances. Furthermore, calculated was the corresponding root mean square error (RMSE) and the coefficient of determination (R2) for each calibration equation.

2.2.7 Apple fruit surface temperature

11

Thermal sensors acquire and estimate emitted infrared radiation (IR) from an object following Stefan-Boltzmann's law [19]. In this study, output image data from the thermal-RGB imager was in ‘radiometric JPEG’ format. The acquired image contained radiance data and image metadata in exchangeable image file format [43–44] with embedded raw temperature data at each pixel. Such file format allows the alteration of radiometric parameters, like emissivity and reflected temperature in post-processing of an image. An algorithm was developed in PythonTM for apple FST extraction using equations 7–9 [44]. Equations 7–9 were further simplified as equation 10 to calculate the apple FST at an emissivity value of 1.

Rref =

R1

(

B Tref

(

R2 × e

Robj =

Tobj =

Tobj =

))

(7)

―O

―F

(S ― (1 ― Em) × Rref)

(8)

Em

(9)

B ln

((

R1

R2 × (Robj + O))

)

+F

B ln

(

R1 (R2 × (S + O))

)

+F

(10)

Where, Rref is radiance of the reflected objects, R1 is Planck R1 constant, R2 is Planck R2 constant, B is Planck B constant, F is Planck F constant, O is Planck O (offset) constant, Tref is reflected temperature in Kelvins, S is raw matrix values from thermal image, Em is emissivity of object. Robj is radiance amount of the measured object and Tobj is object temperature in Kelvin. The IR attenuation factors used for this study have been summarized in table 1. Distance between the sensor and target object significantly affects the accuracy of the measurement. Similarly, ambient temperature, relative humidity (RH), reflected temperature and emissivity can cause fluctuation in the fruit temperature [19]. Ideal apple emissivity value of 0.95 [20, 45] was first used for FST estimation. Additionally, apple FST was calculated for emissivity value of 1.00 for further

12

analysis. Table 1: IR attenuation factors considered during apple fruit surface temperature estimation

Factor

Value

Emissivity, dimensionless

0.95 and 1

Object Distance, m*

0.5~1

Reflected Apparent Temperature, °C

25.0

Atmospheric Temperature, °C

25.0

Relative Humidity, %

50.0

*object distance according to the experiment

Additionally, the estimated pixel temperature was further refine using calibration equation. In previous studies, sunburn in apples have been reported above FST of 40 °C. Therefore, pixel temperatures (𝑇𝑃𝑖𝑥) above 40 °C were grouped separately so that outlier effect of pixels at lower temperature can be eliminated. Moreover, the operating temperature range of thermal-RGB imager was up to 50 °C therefore, data above 50 °C was considered as 50 °C in mean FST calculation. Moreover, when the maximum value of 𝑇𝑃𝑖𝑥 [max (𝑇𝑃𝑖𝑥)] was below 40 °C, max (𝑇𝑃𝑖𝑥) was considered as the mean FST because temperatures below 40 °C do not contribute to apple sunburn. The mean FST (Tmean) was calculated using equations 11 and 12. ∑50

T i = 40 i

× Ni

if (max (TPix) > 40)

Tmean =

if (max (Tpix) < 40)

Tmean = max (Tpix)

∑50 Ni i = 40

(11) (12)

Where, 𝑁𝑖 is number of pixels at 𝑇𝑖°C temperature in the segmented object and Tpix is individual pixel temperature.

2.3 System performance evaluation The thermal-RGB imagery data was acquired in Pullman, Washington, USA (latitude 46.7298°N, longitude 17.1817°W) on June 25, 2018. The developed imaging system was used to acquire images of Fuji apples [(mean height=8.5 cm, mean width=8 cm)] in an outdoor environment. The imaging sensor was positioned 1 m away from the fruit and such that the sun-exposed surface of the apple

13

can be captured with minimal backlighting effect. Image acquisition was started at 15:10 hour and 30 images were captured until 16:00 hour. Additionally, weather data was collected during image acquisition to investigate the effect of various environmental parameters (e.g. wind speed, ambient temperature, humidity etc.) on apple FST. The acquired data was downloaded from an open field station (WSU Agricultural weather network (AgWeatherNet, http://weather.wsu.edu/index.php) of the Washington State University. Evaluated was also the processing time of the smart sensing system for real time temperature estimation. The processing time of the sensing system was first tested for the originally acquired full sized images. Acquired images were then compressed to 44%, 25% and 11% of the original size and such data was analyzed to understand the effect of data compression on processing time. Furthermore, the developed algorithm was run on a computer with a Windows OS (Windows 10, Intel® Core™ i5 processor, 2.5 GHZ, 8 GB RAM, Microsoft, WA, USA) to compare the computing run time performance.

3. Results and discussion 3.1. Fruit-background classification Fig. 4 depicts the classification accuracy using k-means++ for all 45 images with k-value of 2, 3, 4 and 5. Classification results revealed that 2 clusters produce was reliable in segmentation for an image with a green background (containing only leaves) whereas the segmentation accuracy deteriorates for combinations of soil, grass, sky and plants (leaves, branches) in the background. Classification with 3 clusters produced best results in most of the cases as most of the images had leaves and branches as the background. Wang et al., (2015) [21] has reported similar results. Hence, k = 3 was opted for classification in subsequent algorithms. Classification accuracy for (k=) 4 and 5 clusters was 42.22% and 22.22%, respectively. Five clusters divided the image into five classes however, most of the sample images had 3 to 4 objects that resulted in over-segmentation and reduced segmentation accuracy.

14

Fig. 4. Classification accuracy (CA) for thermal RGB images (n =45) with respect to optimum number of cluster (k)

3.2. Segmentation using developed algorithm The LAB demonstrated a slightly better segmentation around apple edges compared to RGB color space (Fig. 5). Therefore, the former was used in the development of the segmentation algorithm. Classified images [Fig. 6(b)] consisted of some noise [Fig. 6(c)]. Therefore, several apple pixels were lost on the output binary image [Fig. 6(d)]. Morphological operations help remove the noises from the image and filled the remaining hole to convert the apple into a complete blob. Fig. 6(e) depicts the extracted apple blob after region matching of the binary image with the original image. Adapting thresholding on the filtered and eroded greyscale image resulted in the binary image [Fig. 6(f)]. Furthermore, distance transformation on the obtained binary image extracted the foreground area of the apple [Fig. 6(g)]. Pixels between the background and contours of the apple were defined as an unknown region [dark blue region in Fig. 6(h)]. Finally, a marker-controlled watershed implemented on the labeled image successfully separated the apple clusters into the individual apples [Fig. 6(i)]. Fig. 7 depicts the segmentation output of the proposed algorithm for some of the sample images with different background and foreground conditions.

15

(a)

(b)

(c)

(d)

Fig. 5. Effect of color conversion on target object segmentation: (a) Input image in RGB color space (source: www.centuryfarmorchards.com), (b) Resultant segmented apple after R−G based Otsu’s thresholding (red markers represent the segmentation error at edges), (c) RGB to LAB color converted image, and (d) Resultant segmented apple after RGB to LAB color conversion.

Fig. 6. Extracted output image at different segmentation stages: (a) RGB sample image, (c) RGB to LAB color space conversion, (c) k-means++ classification, (d) Binary image, (e) Apple blob extraction after holes and noise removal and region matching, (f) Binary image after blur filtering and adaptive thresholding, (g) Foreground extraction of apple blob, (h) Overlapping of foreground and background with blue line representing unknown region, and (i) Final output image using marker controlled watershed algorithm

16

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 7. Representative sample images (a, b and c) and respective fruit segmentation (d, e and f) using developed algorithm-3

Average missing error of algorithm-1, algorithm-2 and the algorithm-3 was 12.55%, 40.30% and 12.09%, respectively. Overall, the developed algorithm-3 achieved almost the same mean missing error as algorithm-1 but evidently lesser segmentation error compared to algorithm-2 (table 2). Additionally, the algorithm-3 achieved the least segmentation error for all sample images compared to the standard algorithms. The highest missing error (24.84%) by the algorithm-3 was reported for image 6. This image consisted of several branches and portion of soil in the background which appears red in color at the specific incident angle of the sunrays which might have led to an incorrect segmentation and lowest segmentation accuracy. Si et al. (2015) [37], reported similar inaccurate segmentation due to variable color characteristics of an acquired image. The apples with a higher percent of leaf shadows reported higher missing error compared to lower or no leaf shadows. Furthermore, image acquisition from proximity demonstrated better segmentation because of higher visual area of the targeted apples. As an advancement to this study, the suitability of hue color component from HSV color space for fruit segmentation can be explored as suggested by GarcíaLamont et al. (2018) [46] Table. 2. The fruit segmentation performance for three different algorithms tested in this study

Missing error (%)

Segmentation error (%)

17

Sample

Algorithm-1

Algorithm-2

Algorithm-

Algorithm-1

Algorithm-2

Algorithm-

3

image

3

1

6.42

28.62

4.19

1.73

28.19

0.30

2

7.29

30.69

7.55

4.00

5.00

0.03

3

19.52

56.62

18.94

0.5

3.59

0.02

4

8.60

54.47

7.83

1.32

0.55

0.01

5

11.03

36.30

7.64

0.16

0.12

0.05

6

15.25

37.14

24.84

3.43

10.06

0.02

7

7.78

34.93

9.27

0.05

2.05

0.05

8

25.24

40.27

15.75

0.63

4.85

0.53

9

9.69

44.97

7.78

0.14

3.61

0.07

10

14.71

38.96

17.08

0.25

1.99

0.19

Average

12.55

40.30

12.09

1.23

6.00

0.13

3.3. Fruit surface temperature estimation Table 3 indicates the blackbody calibration equations for various imaging distances. The linear regression analysis, of target plate temperature [actual temperature (Ta)] and measured temperature (Tm) data from the blackbody, had R2 and RMSE of 0.99 and 0.91, respectively, for a 0.5 m image acquisition distance. Additionally, R2 and RMSE for the images acquired from a distance of 1 and 1.5 m was 0.99 and 1.29, and 0.99 and 1.58, respectively. Images obtained from 1.5 m distance had less than 12 pixels at the region of interest and could have adversely affected the thermal measurement accuracy. Hence, 1.5 m imaging distance was not selected for FST determination. Additionally, images acquired from 0.5 m provided the best resolution and the least RMSE but had a relatively smaller field of view (FOV). Smaller FOV reduces the number of apple in the frame and hence may adversely affect the reliability of measured temperature for the sunburn management. Therefore, imaging distance of 1 m was selected for sunburn monitoring. Table 3. Calibration equation of thermal RGB imagers for varied image acquisition distances

Imaging distance (m)

Calibration equation

R2

RMSE

18

0.5

Ta = 0.938×Tm +3.3329

0.99

0.91

1

Ta = 0.9808×Tm +1.9833

0.99

1.29

1.5

Ta = 1.0185×Tm + 0.7879

0.99

1.58

During actual field trial, the maximum FST was 44 °C at 15:45 hours and minimum FST was 35 °C at 15:35 hours (Fig. 8). The atmospheric temperature at the time of the experiment was between 26–27 °C. A temperature difference of 10–18 °C was recorded between apple FST and air temperature. Such temperature difference was found in accordance to the study by Rackso & Schreder (2012) [5]. Thus, localized microclimate data obtained from an in-field weather station may provide more accurate estimation of local weather conditions. The estimated FST at emissivity of 0.95 was 2 ℃ higher than at 1.00. The reported emissivity of an apple is around 0.95 [47], therefore FST achieved with 0.95 can be reliably adopted for decision making. After achieving a maximum FST at 15:45 hours, a sudden drop was recorded in the apple FST. Reduction in solar intensity due to cloud cover and an increase in wind speed between 15:40 and 15:50 hours might have caused this phenomenon. Reduction in solar intensity declines the energy input rate into an apple while higher wind speed enhances the energy withdrawal rate from an apple. Such reduction in apple energy would have resulted in lower FST as inferred by Li et al. (2014) [20].

19

Fig. 8. Estimated FST for two emissivity values and corresponding air temperature data during field imagery acquisition

3.4. Data processing time The average processing time for the originally acquired images was 87 s for the Linux OS based Raspberry Pi ® single board computer and 19 s for the Windows OS based PC. The time cost on the single board computer for compressed images with sizes 44%, 25% and 11% of the original images were 42 s, 40 s and 12 s, respectively. Pertinent processing times on Windows OS based PC were 10 s, 7 s and 3 s, respectively. The compression of an image significantly reduced the processing time but at the cost of reduced resolution. An image compression of 44% was found to be adequate for this study as it reduced the processing time by 50% without hampering the reliability of estimation. 4. Conclusion The following are the conclusions from this study: 1. The proposed on-board image processing algorithm demonstrated the best apple

segmentation performance among tested algorithms (i.e., algorithm-1 and algorithm-2) with the highest segmentation accuracy of 57.78% for three clusters (k=3) using k-means++

20

classifier. The customized algorithm had missing and segmentation errors of 12.09% and 0.13%, respectively 2. FST estimates at emissivity of 0.95 was 2 °C higher than emissivity of 1.00. Overall, FST

estimation time for the developed smart sensing system was 87 s. Image processing time was reduced to 44 s with 44% image compression and was found adequate in FST estimation. In our future trials, we plan to integrate the proposed FST estimation algorithm with open field and microclimate-based apple FST models to further improve the reliability of the smart sensing system.

Acknowledgements This project was funded in part by NSF/USDA NIFA Cyber Physical Systems and USDA NIFA WNP0745. The author extends their gratitude to Dr. Sindhuja Sankaran, Mr. Abhilash K. Chandel, Dr. Rajeev R. Sinha, Dr. Haitham Bahlol, Mr. Jake Schrader, Dr. Carlos Zuniga, Dr. Sanaz Jarolmasjed, Mr. Chongyuan Zhang and Ms. Afef Marzougui of Washington State University for their kind assistance in this study.

References [1] USDA, NAAS. Statistics of Fruits, Tree Nuts, and Horticultural Specialties. link: https://www.nass.usda.gov/Publications/Ag_Statistics/2016/Chapter05.pdf. 2016. [2] Schrader L, Sun J, Zhang J, Felicetti D, Tian J. Heat and light-induced apple skin disorders: causes and prevention. In XXVII International Horticultural Congress-IHC2006: International Symposium on Enhancing Economic and Environmental. Seoul, Korea; 2008. 772:51–58. [3] Piskolczi M, Varga C, Racsko J. A Review of the Meteorological Causes of Sunburn Injury on the Surface of Apple Fruit (Malus Domestica Borkh). J. Fruit Ornam. Plant Res. 2004;12. [4] Felicetti DA, Schrader LE. Photooxidative Sunburn of Apples: Characterization of a Third Type of Apple Sunburn. Int J Fruit Sci. 2008;8:160–172. [5] Racsko J, Schrader LE. Sunburn of Apple Fruit: Historical Background, Recent Advances and Future Perspectives. Crit Rev Plant Sci. 2012;31:455–504. [6] Mendoza MC. Evaporative cooling : sunburn suppression benefits and influence on microbial survival. Master thesis. Pullman; Washington State University. 2015. [7] Schrader LE, Zhang J, Duplaga WK. Two Types of Sunburn in Apple Caused by High Fruit Surface (Peel) Temperature. Plant Health Prog. 2001;10:1–5. [8] Schrader L, Zhang J, Sun J. Environmental stresses that cause sunburn of apple. Acta Hortic.

21

2003;618:397–405. [9] Makeredza B, Schmeisser M, Lötze E, Steyn W. Water stress increases sunburn in ‘Cripps’ Pink’ apple. HortScience. 2013;48(4):444–447. [10] Parchomchuk P, Meheriuk M. Orchard Cooling with Pulsed Overtree Irrigation to Prevent Solar Injury and Improve Fruit Quality of 'Jonagold' Apples. Hortscience. 1996;31:802–804. [11] Glenn DM, Prado E, Erez A, McFerson J, Puterka GJ. A reflective, processed-kaolin particle film affects fruit temperature, radiation reflection, and solar injury in apple. J Am Soc Hortic Sci. 2002;127(2):188–193. [12] McCaskill MR, Lexie M, Ian G, Steve G, Debra LP. How hail netting reduces apple fruit surface temperature: A microclimate and modelling study. Agric For Meteorol. 2016;226:148–160. [13] Han JW, Zhao CJ, Qian JP, Ruiz-Garcia L, Zhang X. Numerical modeling of forced-air cooling of palletized apple: Integral evaluation of cooling efficiency. Int J Refrig. 2018;89:131–141. [14] Liu HJ, Kang Y. Regulating Field Microclimate using Sprinkler Misting under Hot-dry Windy Conditions. Biosyst Eng. 2006;95:349–358. [15] Chandel AK, Khot LR, Osroosh Y, Peters TR. Thermal-RGB imager derived in-field apple surface temperature estimates for sunburn management. Agric For Meteorol. 2018;253:132–140. [16] Oerke EC, Steiner U, Dehne H, Lindenthal M, Thermal imaging of cucumber leaves affected by downy mildew and environmental conditions. J Exp Bot. 2006;57:2121–2132. [17] Go´mez-Cando´n D, Virlet N, LabbГ© S, Jolivot A, Regnard JL. Field phenotyping of water stress at tree scale by UAV-sensed imagery: new insights for thermal acquisition and calibration. Precis Agric. 2016;17(6):786–800. [18] Leinonen I, Jones HG. Combining thermal and visible imagery for estimating canopy temperature and identifying plant stress. J Exp Bot. 2004;55:1423–1431. [19] Bulanon DM, Burks TF, Alchanatis V. Study on temporal variation in citrus canopy using thermal imaging for citrus fruit detection. Biosyst Eng. 2008;101:161–171. [20] Li L, Peters T, Zhang Q, Zhang J, Huang D. Modeling apple surface temperature dynamics based on weather data. Sensors (Basel). 2014;14:20217–20234. [21] Wang D, Song H, Yu X, Zhang W, Qu W, Xu Y. An improved contour symmetry axes extraction algorithm and its application in the location of picking points of apples. Span J Agric Res. 2015;13:0205. [22] Sural S, Qian G, Pramanik S. Segmentation and histogram generation using the HSV color space for image retrieval. In: Proceedings. International Conference on Image Processing, Rochester, USA; 2002. p. 589–592. [23] Stajnko D, Lakota M, Hočevar M. Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging. Comput Electron Agric. 2004;42(1):31–42. [24] Kavdir I, Guyer DE. Apple Sorting Using Artificial Neural Networks and Spectral Imaging. Trans ASAE 2002;45(6):1995–2005. [25] Inkyu S, Zongyuan G, Feras D, Ben U, Tristan P, Chris M. DeepFruits: A Fruit Detection System Using Deep Neural Networks. Sensors (Basel). 2016;16(8):1222. [26] Lal S, Behera SK, Sethy PK, Rath AK. Identification and counting of mature apple fruit based on BP feed forward neural network, In: Third International Conference on Sensing, Signal Processing and Security (ICSSS). Chennai, India; 2017.361–368.

22

[27] Shaik KB, Ganesan P, Kalist V, Sathish BS, Jenitha JMM. Comparative Study of Skin Color Detection and Segmentation in HSV and YCbCr Color Space. Procedia Comput Sci. 2015;57:41– 48. [28] Mendoza F, Dejmek P, Aguilera JM. Calibrated color measurements of agricultural foods using image analysis. Postharvest Biol Technol. 2006;41:285–295. [29] Kaur A, Kranthi B. Comparison between YCbCr color space and CIELab color space for skin color segmentation. Int J Appl Inf Syst. 2012;3:30–33. [30] Schwarz M, Cowan W, Beatty J. An experimental comparison of RGB, YIQ, LAB, HSV, and opponent color models. ACM Trans Graph. 1987;6:123–158. [31] Cheng HD, Jiang XH, Sun Y, Wang J. Color image segmentation: advances and prospects. Pattern Recognit. 2001;34:2259–2281. [32] Alnaji KW, Ashour WM. A Novel Clustering Algorithm using k-means (CUK). Methods. 2011;25(1):25–30. [33] Zahra S, Ghazanfar MA, Khalid A, Azam MA, Naeem U, Prugel-Bennett A. Novel centroid selection approaches for KMeans-clustering based recommender systems. Inf Sci. 2015;320:156– 189. [34] Peng K, Leung VCM, Huang Q. Clustering Approach Based on Mini Batch Kmeans for Intrusion Detection System Over Big Data. IEEE Access. 6(2018);11897–11906. [35] Meyer GE, Neto JC. Verification of color vegetation indices for automated crop imaging applications. Comput Electron Agric. 2008;63:282-293. [36] Zhou R, Damerow L, Sun Y, Blanke M. Using colour features of cv. 'Gala' apple fruits in an orchard in image processing to predict yield. Precis Agric. 2012;13:568-580. [37] Si Y, Liu G, Feng J. Location of apples in trees using stereoscopic vision. Comput Electron Agric. 2015;112:68–74. [38] Rossl C, Kobbelt L, Seidel, H. P. Extraction of feature lines on triangulated surfaces using morphological operators. In Proceedings of the AAAI Symposium on Smart Graphics. 2000; 71– 75. [39] Navon, E, Miller, O, Averbuch, A. Color image segmentation based on adaptive local thresholds. Image Vis Comput. 2005;23(1):69-85. [40] Cuisenaire, O. Distance transformations: fast algorithms and applications to medical image processing. Ph. D. thesis, Université catholique de Louvain, Belgium. 1999. [41] Veta M, Huisman A, Viergever M. A, van Diest P. J, Pluim J. P. Marker-controlled watershed segmentation of nuclei in H&E stained breast cancer biopsy images. In 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 2011; 618-621. [42] Phillips LA. Infrared thermography on the spot: Because size does matter. link: http://www.lenphillips.com/images/FEATURES/SPOT_SIZE.pdf. 2013. [43] Felice FD, Mazzini A, Stefano GD, Romeo G. Drone high resolution infrared imaging of the Lusi mud eruption. Mar Pet Geol. 2018;98:38–51. [44] Harvey P. enhancement: extract binary data from FLIR radiometric jpg. ExifTool Forum. Link: http://u88.n24.queensu.ca/exiftool/forum/index.php?topic=4898.60. 2013 [45] Saudreau M, Sinoquet H, Santin O, Marquier A, Adam B, Longuenesse JJ, Guilioni L, Chelle M. A 3D model for simulating the spatial and temporal distribution of temperature within ellipsoidal

23

fruit. Agric For Meteorol. 2007;147:1–15. [46] García-Lamont, F, Cervantes, J and López-Chau, A. Human mimic color perception for segmentation of color images using a three-layered self-organizing map previously trained to classify color chromaticity. Neural Comput Appl. 2018; 30(3):871-889 [47] Hellebrand HJ, Beuche H, Linke M. Determination of Thermal Emissivity and Surface Temperature Distribution of Horticultural Products. In: Sixth International Symposium on Fruit, Nut and Vegetable Production Engineering, Potsdam, Germany; 2001.p.1-4.

24

Highlights 

Smart sensing system with on-board fruit surface temperature (FST) estimation algorithm



Customized clustering algorithm-based fruit segmentation approach



Successful apple FST estimation on-board the single board computer



Algorithm time cost further reduced by image compression

25