- Email: [email protected]
A leaf-wall-to-spray-device distance and leaf-wall-density-based automatic route-planning spray algorithm for vineyards
Crop Protection 111 (2018) 33–41
Contents lists available at ScienceDirect
Crop Protection journal homepage: www.elsevier.com/locate/cropro
A leaf-wall-to-spray-device distance and leaf-wall-density-based automatic route-planning spray algorithm for vineyards
T
Guandong Gaoa, Ke Xiaob,c, Yuejin Mac,∗ a
Department of Information Management, The National Police University for Criminal Justice, Baoding, Hebei Province 071000, China College of Information Science & Technology, Hebei Agricultural University, Baoding, Hebei Province 071001, China c College of Mechanical and Electrical Engineering, Hebei Agricultural University, Baoding, Hebei Province 071001, China b
A R T I C LE I N FO
A B S T R A C T
Keywords: LW–SD distance Leaf wall density Spray route Grapevine
In this study, to achieve precision spraying of pesticide in vineyards and reduce pesticide waste and pollution, algorithms were proposed for estimating parameters such as the average distance between the leaf wall and the spray device (LW–SD distance) and the leaf-wall density by integrating colour and depth images acquired by Microsoft Kinect. First, the colour video images were segmented using a morphological image segmentation technique to accurately separate the leaf walls from the remaining images. Then, based on the depth images, algorithms for estimating the average LW–SD distance and the leaf wall density were established to estimate the spray control parameters. A spline area-based algorithm for estimating the average LW–SD distance was proposed to accurately estimate the LW–SD distance. Finally, an algorithm for estimating the route deviation, as well as an algorithm for correcting and planning the spray route, was established to guide and maintain the spray device on the optimal path. The experimental results demonstrate that the spray parameter estimation algorithms produce relatively small errors in the estimation of the average LW–SD distance and the leaf wall density. In addition, the differences between the spray distance and route deviation estimated using the route planning algorithm and the measurements were also relatively small, demonstrating the accuracy of the algorithm.
1. Introduction Grapevines are prone to pests and diseases (Nita et al., 2007; Chuche and Thiéry, 2009), of which powdery mildew and grey mould severely affect the growth and development of grapes (HéCtor et al., 2008; Valdésgómez et al., 2011; Calonnec et al., 2013; Guilpart et al., 2017). Pesticides used to manage crop pest and diseases have adverse effects on human health and the environment. Therefore, a major challenge facing researchers is the reduction of pesticide usage while maintaining crop yields. Hence, in recent years, precision spraying (Maghsoudi et al., 2015; Malnersic et al., 2016; Qiu et al., 2016) has increasingly garnered more attention. The aim of studying precision spraying technology is to acquire accurate information related to the target (fruit trees, when used in orchards). Two common indices reflect the crop size, namely, the leaf area index (LAI) (Magney et al., 2016; Pearse et al., 2016; Woodgate et al., 2016; Gonsamo et al., 2017) and the leaf wall area (LWA) (Gil et al., 2011; Walklate et al., 2011; Escolà et al., 2013; Walklate and Cross, 2013). The LAI is defined as one half the total leaf surface areas per unit ground area. It is a forest canopy variable that is closely related to forest
∗
Corresponding author. E-mail address: [email protected] (Y. Ma).
https://doi.org/10.1016/j.cropro.2018.04.015 Received 29 December 2017; Received in revised form 9 April 2018; Accepted 16 April 2018 0261-2194/ © 2018 Elsevier Ltd. All rights reserved.
growth and health (Pearse et al., 2016). The Leaf Wall Area model only uses two parameters, tree height and row distance (Pergher and Petris, 2008). Thus, row distance and canopy height are the key to the calculation of the LWA. It can reasonably be used as the basis for research. Currently, several target detection technologies exist, including light detection and ranging (LiDAR)-based, ultrasonic sensor-based, infrared (IR) light-based and machine-vision-based target detection technologies. In terms of LiDAR-based target detection, Duga et al. (2016) proposed a three-dimensional (3D) spray drift model based on computational fluid dynamics that considers the canopy wind speed, the amount of movement of the sprayer, and the airborne spray drift distance. This model was used to reduce the spray drift distance; however, its use increased near-ground fertilizer deposition. Because they are generated from point light sources, scanning laser could be influenced by moving targets such as fluttering leaves. Moreover, due to their high cost, laser beam scanners are unsuitable for extensive popularization. In ultrasonic sensor-based target detection, to establish an orchard-canopy planar-target density model, essential for calculating canopy densities, Li et al. (2016) proposed a target density-detection system based on an ultrasonic sensor and employed a time-domain
Crop Protection 111 (2018) 33–41
G. Gao et al.
Fig. 1. The algorithm flow diagram of the precision spraying system.
2. Materials and methods
energy analysis method to analyse the ultrasonic signals. Ultrasonic sensors may cause errors in the determination of distances and densities to fruit trees, when the detection distance is too close or there is fog. In terms of IR light-based target detection, Narváez et al. (2016) combined an IR sensor with a LiDAR system. They first classified the canopies based on the corresponding LiDAR measurements and then used the IR sensor to match these points. Near-IR sensors have a small and nonuniform transmission range, resulting in increased processing difficulty and errors. In terms of machine-vision-based target detection, Bargoti et al. (2015) presented a method for identifying individual trees in an apple orchard using ground-based LiDAR and image data and evaluated it by conducting an experiment in an apple orchard where apples were planted in two different trellis structures, namely, a standard vertical trellis structure and an updated Guttingen V-trellis structure. This method had a few shortcomings: the camera was incapable of acquiring distance information, and the leaf wall detection was easily affected by the complex background. In this study, to address the aforementioned lack of distance information associated with machine-vision-based target detection technologies, an approach involving the use of integrated Microsoft Kinect sensors was presented to measure distances, and a precision spray algorithm capable of automatically estimating spray parameters and planning routes was also established. To determine the precise location of the spray target, the system used two Kinect devices to acquire colour and depth images on its left side and at its front and employs a leaf wall segmentation technique to accurately extract the leaf walls. In addition, algorithms were also established to estimate the leaf wall parameters including density and average distance between the leaf wall and the spray device (LW–SD) based on the information acquired from the left angle of view (AOV). Finally, to plan the route for the spray device, a spline-based algorithm for estimating the LW–SD distance based on the information acquired from the forward AOV was established. The spline-based LW–SD distance was estimated as the characteristic parameter to correct and plan the spray route, as well as to plan and predict the optimal spray route to further achieve precision spray control and reduce the amount of pesticide wasted and pollution dispersed.
2.1. Experimental materials The experimental spray system used two Kinect devices (model number: XBOX 360 Kinect 1.0), with one being fixed to the front and one to the left side of the spray platform to acquire video images of the forward route and the leaf wall on the left side, respectively. The forward speed was about 1.1 m/s (or 4 km/h). It also comprised a portable computer, a portable power source, hydraulic valves, movable spray arms, air conduits, air-assisted sprayers and a spray adjustment and control device. And it can adjust the spray distance by outputting a control signal through the processing software. The spray arm consisted of upper, middle and lower nozzles at a height of 60 cm, 125 cm, and 200 cm. It could be moved in and out at range of 0–50 cm in horizontal direction. The nozzles were fan-shaped that the spray range is 1.3 m. The experimental time was before 9a.m. and after 3p.m. in sunny day or the whole daytime in cloudy day, which was an “operation window”. It is because that Kinect has daytime light contamination problem for outdoor usage (George et al., 2013; Sharon et al., 2015). When the ambient light is too strong, the depth images will be turned into blank. Two vineyards (vineyards 1 and 2) in Hebei Province, China (northern China) were selected as the experimental sites. The grapevines in the two vineyards were all 5 years of age. The Muscat grape variety was planted in vineyard 1 with a plant spacing of 1 m, a row spacing of 3 m and the height of 2.2 m. The Kyoho grape variety was planted in vineyard 2 with a plant spacing of 1.5 m and a row spacing of 4 m and the height of 2.3 m. And all the row spacing is within the measurement range of Kinect. In the experiments, 30 sampling positions were chosen randomly, which the distance between each other is greater than 5 m. 2.2. Overview of the method The whole method consisted of two components, namely, a spray parameter estimation component, which calculated the leaf wall spray 34
Crop Protection 111 (2018) 33–41
G. Gao et al.
of segmenting the leaf wall into an image separate from the remaining area.
control parameters, and a spray route planning component, which planned the spray route. Fig. 1 shows the overall algorithm flowchart. 2.3. Spray parameter estimation algorithms
2.3.2. Average leaf-wall-to-spray-device distance estimation algorithm The average leaf-wall-to-spray-device distance is used as an index parameter that the Kinect device on the left side to control the spray distances by adjusting the spray arm. Therefore, calculation of the average distance between the Kinect and the grapevines is necessary. Through a derivation based on basic principles, an equation for estimating the average LW–SD distance was established. In a depth image, each pixel represents the distance between the scene and the Kinect. Thus, the average distance can be determined using the following equation:
To achieve intelligent precision spraying of grapevines in vineyards, the identification of the leaf walls and determination of the distance between each leaf wall and the spray system were needed to determine a spray route that could lead to the optimal spray distance and minimize the dose of pesticide sprayed and the energy consumed. 2.3.1. Leaf wall segmentation algorithm Most studies using machine-vision-based leaf wall detection use image processing techniques to analyse and process each frame in a video sequence and detect the leaf walls by image erosion (Narváez et al., 2016). In this approach, a next row of fruit trees will be misidentified as a leaf wall when used as the background, resulting in an inaccurate estimation of the distance. To segment the leaf wall, a colour image was first processed using a colour segmentation algorithm. A subtraction operation was performed with the pixel values of the green and red layers of the RGB image. The specific equation is as follows:
Ibin (x , y ) =
⎧ 255, ⎨ 0, ⎩
N
Dav =
N
∑i = 1 i
, Ii ∈ ILWA (2)
where Dav is the average distance between the grapevines and the Kinect; Ii is the distance of each pixel of the leaf wall; and N is the number of pixels of the leaf wall, i ∈ [1, N ]. When calculating the average LW–SD distance, each acquired image was divided into three (upper, middle and lower) areas based on the spray areas of the three nozzles on the spray arm. The distance between each of the upper, middle and lower sections of a grapevine leaf wall and the Kinect was calculated using Equation (2). Fig. 3 shows the three areas of an image in which the leaf wall is segmented from the background, corresponding to the locations and spray areas of the three nozzles.
if IG (x , y ) − IR (x , y ) ≥ 0 if IG (x , y ) − IR (x , y ) < 0
∑i = 1 Ii
(1)
where IG(x,y) and IR(x,y) are the pixel values of the green and red layers of the colour image at (x,y), respectively, and Ibin(x,y) is the image obtained from binarization. When segmenting the leaf wall in a left-side Kinect image, the distant grapevine background in the low-density part of the leaf wall, i.e., the background consisting of green leaves in a distance, must be removed. Therefore, the image obtained from binarization and the corresponding depth images were used for logical And operations to remove background. Moreover, the gaps between the leaves and the shadows in a leaf wall can easily affect the extraction of its outline. Therefore, the small gaps and voids in each image need to be filled. In this study, a dilation algorithm, specifically, a closed morphological operation algorithm (Li et al., 2010), was employed to fill the small gaps using the structure elements of a 5 × 5 square matrix. Then, the algorithm further fills the relatively large voids within 300 pixels based on four-connected pixels. Fig. 2 shows the leaf wall segmentation process and results. Experimental results showed that this algorithm was relatively highly capable
2.3.3. Spline-based algorithm for estimating the LW–SD distance For the spray route planning, to effectively estimate the vertical LW–SD distance when the spray device was advancing, the derivation of distance information from the depth images acquired by the forward Kinect and prediction of the vertical distance between the leaf wall in the front and the spray device were necessary. Therefore, a spline areabased algorithm was proposed to estimate the average LW–SD distance. Two vertical spline areas with a width of 20 pixels were marked at the two vertical edges of each leaf wall image, and the average of the distances between the spray device and the pixels of the leaf wall within these two splines was calculated and used as the average distance between the leaf wall and the Kinect on the spray device to reduce the single-point estimation error. The equation is as follows:
Fig. 2. The process and results of LWA segmentation. 35
Crop Protection 111 (2018) 33–41
G. Gao et al.
estimation by the total area of the leaf wall (including the leaf wall and the voids inside the leaf wall):
LWAdens =
sl sLWA
(4)
where LWAdens is the leaf wall density, Sl is the area of the image where the leaves are located, and SLWA is the section of the leaf wall within the rectangular box. In fact, Sl represents all the pixels that represent the leaves in the depth image, and SLWA represents the total number of pixels of the leaf wall. LWAdens is the percentage of the leaf wall covered by leaves. According to Equation (4), when the leaf wall density is low, the spray pressure of the nozzles is low, and the dose of pesticide sprayed per unit area is small. 2.4. Spray route planning algorithm Spray route planning mainly relies on the processing of the forward video images. The advancing route of the spray device was estimated and planned based on the LW–SD distance, the optimal spray distance for the nozzles and the current location of the spray device. The optimal spray results were obtained by correcting the route deviation.
Fig. 3. Division of spraying area and LWA. N
Disav =
∑i = 1 Di (x , y ) N
, Di (x , y ) ∈ ILWA (x , y )
(3) 2.4.1. Vertical spray distance estimation To plan the spray route, estimation of the spray distance, which can be obtained by subtracting the width of the spray device from the vertical LW–SD (mainly the Kinect) distance, is necessary. Therefore, the vertical LW–SD distance within the horizontal AOV of the Kinect was calculated. Fig. 5 shows the specific algorithm for estimating the spray distance. The left-side straight line signifies the leaf wall. Point O signifies the location of the Kinect camera lens. The rectangular box signifies the field of view (FOV) of the Kinect. Point A is the median point of the leaf wall spline. Point B is the point of intersection between the vertical line that goes through point O and the horizontal line extending from median point A. Point C is the centre of the FOV of the camera. The horizontal AOV of the Kinect is 54°. Thus, the angle between the centre and the edge of an image acquired by the Kinect, ∠AOC, is approximately half of its horizontal FOV. Here, we establish that ∠AOC ≈27°. Because ∠BOC = 90°, we have
Furthermore, x ∈ [1, Sw] ∪ [W , W − Sw], y ∈ [1, H ] Where Di(x,y) is the pixel value of the depth image at (x,y), i.e., the distance; N is the number of pixels; ILWA is the binarized segmented leaf wall image; SW is the width of the spline areas; W and H are the width and height of the image; and Disav is the average LW–SD distance. To avoid issues that may arise due to the potential blank strips at the edges of a depth image, the outer boundary of each spline area was set at a horizontal distance of 10 pixels from the closest edge of the image. In Fig. 4, the red areas signify the spline areas used for estimating the average LW–SD distance. The areas that do not contain any part of the leaf wall in each spline will be removed before calculation. 2.3.4. Leaf wall density estimation algorithm Plant growth is inhomogeneous, resulting in a relatively large regional difference in the leaf wall density. When spraying the low-density sections of a leaf wall, if the dose of pesticide sprayed per unit area remains unchanged, the result will be pesticide waste and pollution drift. Therefore, leaf wall density is proposed as an index for the side video images, which can facilitate the automatic adjustment of the flow to control the sprayed dose. Due to the current lack of a relevant equation, a leaf wall density estimation equation was established based on the basic principle of density calculation. The leaf wall density was estimated by dividing the area of the section of the leaf wall used for
∠AOB = ∠BOC - ∠AOC = 90 – 27 = 63°
(5)
Because the length of the line AO is known from Equation (3) (i.e., Disav), the length of line BO can be determined using the following equation:
DisLWA = BO = AO × cos 63° = Disav × cos 63°
(6)
The distance between the leaf wall and each nozzle can then be determined by subtracting half of the width of the spray device from the result obtained from Equation (6), i.e.,
Disp = DisLWA − Wp/2
(7)
where WP is the width of the spray device, and DisP is the spray distance between each nozzle and the leaf wall. 2.4.2. Route deviation estimation algorithm To optimize the spray path, calculation of the difference between the route and the advancing direction between the grapevine rows and the optimal spray distance is necessary to correct the route. Therefore, first, identification of the boundaries between the leaf walls and the ground is necessary to determine the route range between the adjacent grapevine rows. The method of least squares was employed to linearly fit the edge of each leaf wall in each post-leaf-wall-segmentation image to determine the boundaries. Hence, the edge is first detected using the following equation:
Fig. 4. Spline areas for estimating the LW-SD distance. 36
Crop Protection 111 (2018) 33–41
G. Gao et al.
between leaf walls and the ground, based on the datasets of the coordinates of the points on the edges of the left and right walls, is necessary. After determining the boundaries between the leaf walls and the ground through fitting, the deviation between the current route and the optimal route could be calculated. This deviation was the difference between the average LW–SD distance and the spray distance on the left or right side calculated using Equation (7).
Δpe = (DispL + DispR)/2 − DispL
(12)
or
Δpe = DispR − (DispL + DispR)/2
(13)
where DisPL and DisPR are the left and right spray distances, respectively, obtained using Equation (7), and Δpe is the route deviation. In addition, the calculation of the current advancing route of the spray device and comparison of it with the planned route were necessary. The current advancing route in an FOV image was calculated based on the width of the spray device with the centre of the bottom of the FOV as the reference. 2.4.3. Route correction and planning line generation algorithm To generate a route correction planning line, the turning angle of the spray device must be calculated first. Let us set Δx as the deviation between the current route and the planned route at a certain point p(x, y) on a line extended from the centre of the spray device in the image coordinate system (because the advancing route is primarily corrected in the horizontal direction, the deviation is only the difference along the x-axis). Thus, the turning angle can be calculated using the following equation:
α= tan−1
xl2 + (y1 − L cos α )2 = (L cos a − Wl /2)2
(8)
x r2 + (yr − L cos α )2 = (L cos α + Wl /2)2
Each fitted straight boundary line is obtained by calculating the coefficients a and b based on the dataset of the coordinates of the points on the leaf wall edge. The optimal estimates of the coefficients a and b can be determined using the following equations:
N (∑ x i yi ) − (∑ x i )(∑ yi ) N (∑ x i2) − (∑ x i )2
(10)
bˆ =
(∑ x i2)(∑ yi ) − (∑ x i )(∑ x i yi ) N (∑ x i2) − (∑ x i )2
(11)
(16)
where (xl,yl), (xr,yr) are the coordinate points of the left and right wheels in the image, respectively; L is the wheelbase of the spray vehicle; and Wl is the front wheel tread. The map obtained from the calculation using the above equations is a bird's eye view of the advancing route. However, in reality, a
(9)
aˆ =
(15)
The equation for the advancing trajectory of the wheels on the right side of the spray vehicle is as follows:
where Ig is the edge detection result, ILWA is the binary segmented leaf wall image, B is a suitable structure element (in this study, B is a 3 × 3 matrix, each of whose components is 1), and Θ is a morphological erosion operation. After the edge of each leaf wall has been detected, the coordinates of all the points on the edge were used to form a dataset for linear fitting. A straight boundary line between each leaf wall and the ground was fitted using the method of least squares, which satisfies the following equation:
y = ax + b
(14)
Then, the turning route of the spray device is further calculated. Fig. 6 shows a schematic diagram of the turning route of the spray vehicle. Thus, the equation for the advancing trajectory of the wheels on the left side of the spray vehicle is as follows:
Fig. 5. The diagram for computing the spraying distance.
Ig = ILWA − (ILWA ΘB)
Δx y
where xi,yi∈Ig are the coordinates of a point on the leaf-wall edge. Because leaf walls and the edge lines are mainly located in the lower half of an image, yi∈[H/2,H], where H is the image height, and N is the number of edge points. Because leaf walls are generally located on the left and right sides of the FOV, determination of the boundaries
Fig. 6. Rotation diagram of the spraying vehicle. 37
Crop Protection 111 (2018) 33–41
G. Gao et al.
In Table 1, the mean values of the differences between the measured and estimated values are relatively small. The smallest value is 0.4606 cm, and the largest one is 0.8956 cm. The RMSEs between the estimated and measured average distances between the upper, middle and lower sections of the leaf wall and the spray device are 1.2767, 1.8256 and 1.8822, respectively, demonstrating that the differences between the measured and estimated values are relatively small and that the average LW–SD distances calculated based on the depth images acquired by the Kinect are in good agreement with the actual distances; i.e., the estimated values are accurate. 3.1.2. Spline-based LW–SD distance estimation The experiment was conducted in vineyard 2. First, the spray system was allowed to stand still in the space between the different grape trellises. The spray distance (DisLWA.) was calculated using Equation (7) based on the data acquired by the Kinect. The vertical distance was measured using a tape by three times at each location. And the average value used as the measured spray distance. During the experiment, 30 sample points were randomly selected, which the spray system was at left, centre and right of the route. Fig. 9 shows the results. Fig. 9(a) shows the histogram of the estimated spray distances on the left and right sides of the spray device. The red columns corresponding to the left deviation of the route are relatively short, whereas the blue columns are relatively long. This indicates that the estimated spray distances on the left side are relatively short, and the estimated spray distances on the right side are relatively long. Therefore, a relatively even distribution of the estimated distances can be observed in Fig. 9(a). Fig. 9(b) and (c) shows the broken lines connecting the estimated and measured distances on the left and right sides, respectively, of the spray device and their differences. From these figures, the differences between the estimated and measured values are relatively small, indicating that the estimated values are relatively accurate. Moreover, the relatively large number of yellow areas indicates that the estimated values are more frequently greater than the measured values. It is because the estimated distance was average value, which could be affected by sags and crests of leaf wall. Table 2 summarizes the mean values of the 30 groups of experimental data, as well as the mean values of the measured and algorithmestimated values and the standard deviations. From Table 2, the mean values of the differences between the estimated and measured distances on the left and right sides are small as 1.5 and 2.4 cm, respectively, and the RMSEs are also small (3.9 and 3.1, respectively). This indicates that the estimated values are relatively accurate.
Fig. 7. Planned curve of corrected route.
trajectory with the coordinates of the Kinect camera as the coordinates of the central point should be displayed in each image photographed by the Kinect. Therefore, a route correction planning line should be generated and displayed in the colour image. Fig. 7 shows the planned route in the vineyard. The green dotted line signifies the route correction planning line. Based on the route correction planning line, the spray device can be guided to remain at the centre of the route when advancing. The route planned by the algorithm is the optimal spray route. 3. Results 3.1. LW–SD distance estimation results 3.1.1. Average LW–SD distance estimation results An experiment was conducted in vineyard 1 to validate the estimation of the average LW–SD distance. The average LW–SD distance was calculated using the 30 group data acquired by the left side Kinect for comparison with the estimated values in Fig. 8. Fig. 8(a) shows the lines connecting the average distances between the spray device and the upper, middle and lower sections of the leaf wall estimated using the algorithm established in this study. The points on the broken lines are the average distances estimated between the sample points on the leaf wall and the spray device. As demonstrated in Fig. 8(a), similar patterns can be observed in the distribution of the average distances; in addition, compared to the average distances between the lower sections, the upper and middle sections are relatively shorter. This is because the leaves in this section grew luxuriantly, took up a relatively large space and were thus closer to the spray device. Fig. 8(b), (c) and (d) shows the broken lines connecting the estimated and measured average distances between the spray device and the upper, middle and lower sections of the leaf wall and their differences, respectively. In each of these plots, the red broken line signifies the estimated values, and the blue broken line signifies the measured values; in addition, the differences between the estimated and measured values are also marked with yellow and blue areas, which are relatively small, suggesting that the estimated values are relatively accurate. Moreover, the relatively large of yellow areas indicates that the measured values are more frequently greater than the estimated values. This is because some protruding leaves on the leaf wall surface affected the average estimated values, resulting in relatively lower estimated values. Furthermore, the mean values of the estimated and measured average distances between the 30 sample pointed on the leaf wall and the spray device, as well as the mean values of the differences and the root-mean-square errors (RMSEs) between the estimated and measured values, were also calculated. Table 1 shows the results.
3.2. Leaf wall density estimation results Estimation of the leaf wall density can facilitate better control of the sprayed dose of pesticide in section 2.3.4. Fig. 10 shows the leaf wall densities at 30 sample points estimated from the top, middle and bottom areas of the images, which were collected from vineyard 1. The red, blue and yellow line connects the top, middle and bottom densities. In Fig. 10, the densities in the middle and bottom sections of the leaf wall are the highest, whereas the densities in the top section are the lowest. The mean values of the 30 groups of density data in the different sections were calculated. The results show that the mean densities in the top, middle and bottom sections are 0.2481, 0.6800 and 0.6720, respectively. The relatively low densities in the top section are a result of containing a sky background partly. Consequently, the leaf wall densities are relatively high in the middle and bottom sections and relatively low in the top section. 3.3. Route deviation estimation results To validate the estimated route deviations, route deviation data at 30 sample points were collected from vineyard 2. Fig. 11 shows the experimental results. The red dotted line signifies the measured values, 38
Crop Protection 111 (2018) 33–41
G. Gao et al.
Fig. 8. The average LW-SD distance in the top, middle and bottom.
4. Discussion
the blue solid line signifies the algorithm-estimated values, and the black solid line signifies each result of the subtraction of a measured value from the corresponding estimated value (a positive value indicates that the spray device was slightly to the left of the route; a negative value is to the right). The curve depicting the differences between the measured and estimated values fluctuates to a relatively small extent around the 0-axis. In addition, the variance of the differences between the estimated and measured values is 2.29 cm. Because the optimal spray distance range for the nozzles of the spray device is 30–40 cm, the estimation error will basically have no significant impact. Thus, the experimental results confirmed that the route deviations estimated using the algorithm established in this study were accurate.
The main objectives of this study were to establish algorithms for estimating control parameters for precision spraying and route planning based on Kinect video processing techniques to optimally control the spraying of pesticide in vineyards and plan the optimal spray route in order to achieve precision spraying, thus preventing the waste of pesticide and generation of pollution. Two Kinect 1.0 devices, which face to the left side and the front to acquire colour and depth image data, were used to estimate the spray distance and leaf wall density and plan the optimal spray route. The algorithm first segmented the leaf wall using morphological image segmentation technique. Based on the depth image data, an algorithm for estimating the average LW–SD distance, a spline area-based algorithm for estimating the LW–SD distance and an algorithm for estimating the leaf wall density were established for
Table 1 Means and errors of estimated and measured distances.
Means (cm) Mean of errors (cm) RMSE
Estimated distance at top region
Measured distance at top region
Estimated distance at middle region
Measured distance at middle region
Estimated distance at bottom region
Measured distance at bottom region
47.0523 0.8377
47.89
28.9294 0.4606
29.39
27.2844
28.18 0.8956
1.2767
1.8256
1.8822
39
Crop Protection 111 (2018) 33–41
G. Gao et al.
Fig. 9. The comparison of the LW-SD distances between measured and estimated values on the left and right sides. Table 2 The data on the left and right distances between the spraying system and LWA.
Means (cm) Mean of errors (cm) RMSE
Left side of the spraying system
Right side of the spraying system
Measured value
Estimated value
Measured value
Estimated value
76.57 1.5
78.07
73.06 2.4
75.49
3.9
3.1
Fig. 11. The line chart of route deviation.
differences between the estimated and measured route deviations was 2.29. This demonstrated that the differences between the measured and estimated values were relatively small. The algorithms proposed in this study are capable of accurately estimating the spray control parameters. And the planned spray route can be accurately maintained at the centre of the route. In the future work, to overcome Kinect's light contamination shortage, we plan to use 3D ToF depth sensor - Pico Monstar, which the measurement range is 6 m. And it is very robust against ambient light that can be used outdoors. Fig. 10. Leaf wall density in the top, middle and bottom.
References
automatically adjusting the spray arms and nozzles of the spray device. Furthermore, an algorithm for estimating the route deviation and planning the route were also established to keep the spray device remaining on the optimal spray route. To validate the proposed algorithms, a field experiment was conducted in vineyards. The statistical analysis of the key measured and estimated values showed that the RMSEs between the average LW–SD distances estimated using the algorithm based on the upper, middle and lower areas of the images taken from the left AOV and the measurements were 1.2767, 1.8256 and 1.8822, respectively; the RMSEs between the spray distances on the left and right sides estimated based on the forward AOV and the measurements were 3.9 and 3.1, respectively; and the variance of the
Bargoti, S., Underwood, J.P., Nieto, J.I., Sukkarieh, S., 2015. A pipeline for trunk detection in trellis structured apple orchards. J. Field Robot. 32 (8), 1075–1094. Calonnec, A., Burie, J.B., Langlais, M., Guyader, S., Saint-Jean, S., 2013. Impacts of plant growth and architecture on pathogen processes and their consequences for epidemic behaviour. Eur. J. Plant Pathol. 135 (3), 479–497. Chuche, J., Thiéry, D., 2009. Cold winter temperatures condition the egg-hatching dynamics of a grape disease vector. Naturwissenschaften 96 (7), 827–834. Duga, A.T., Delele, M.A., Ruysen, K., Dekeyser, D., Nuyttens, D., 2016. Development and validation of a 3D CFD model of drift and its application to air-assisted orchard sprayers. Biosyst. Eng. 154, 62–75. Escolà, A., Rosell-Polo, J.R., Planas, S., Gil, E., Pomar, J., 2013. Variable rate sprayer. Part 1 – orchard prototype: design, implementation and validation. Comput. Electron. Agric. 95 (1), 122–135. Gil, E., Llorens, J., Landers, A., Llop, J., Giralt, L., 2011. Field validation of Dosaviña, a decision support system to determine the optimal volume rate for pesticide application in vineyards. Eur. J. Agron. 35 (1), 33–46.
40
Crop Protection 111 (2018) 33–41
G. Gao et al.
LiDAR and thermal images fusion for ground-based 3D characterisation of fruit trees. Biosyst. Eng. 151, 479–494. Nita, M., Ellis, M.A., Wilson, L.L., Madden, L.V., 2007. Effects of Application of Fungicide during the Dormant Period on Phomopsis Cane and Leaf Spot of Grape Disease Intensity and Inoculum Production, vol 90. pp. 1195–1200 9. Pergher, G., Petris, R., 2008. Pesticide dose adjustment in vineyard spraying and potential for dose reduction. Agric. Eng. Int. CIGR Ejournal X, 1–9. Pearse, G.D., Watt, M.S., Morgenroth, J., 2016. Comparison of optical LAI measurements under diffuse and clear skies after correcting for scattered radiation. Agric. For. Meteorol. 221, 61–70. Qiu, W., Sun, C., Lv, X., Ding, W., Feng, X., 2016. Effect of air-assisted spray application rate on spray droplet deposition distribution on fruit tree canopies. Appl. Eng. Agric. 32 (6), 739–749. Sharon, N., Goldberger, J., Alchanatis, V., 2015. Obstacle detection in a greenhouse environment using the Kinect sensor. Comput. Electron. Agric. 113, 104–115. Valdésgómez, H., Gary, C., Cartolaro, P., Lolascaneo, M., Calonnec, A., 2011. Powdery mildew development is positively influenced by grapevine vegetative growth induced by different soil management strategies. Crop Protect. 30 (9), 1168–1177. Walklate, P.J., Cross, J.V., Pergher, G., 2011. Support system for efficient dosage of orchard and vineyard spraying products. Comput. Electron. Agric. 75 (2), 355–362. Walklate, P.J., Cross, J.V., 2013. Regulated dose adjustment of commercial orchard spraying products. Crop Protect. 54 (12), 65–73. Woodgate, W., Armston, J.D., Disney, M., Jones, S.D., Suarez, L., Hill, M.J., Wilkes, P., Soto-Berelov, M., 2016. Quantifying the impact of woody material on leaf area index estimation from hemispherical photography using 3D canopy simulations. Agric. For. Meteorol. 226–227, 1–12.
George, A., Goulden, M.L., Rusu, R.B., 2013. Rapid characterization of vegetation structure with a Microsoft Kinect sensor. Sensors 13 (2), 2384–2398. Gonsamo, A., Walter, J.M., Jing, M.C., Pellikka, P., Schleppi, P., 2017. A robust leaf area index algorithm accounting for the expected errors in gap fraction observations. Agric. For. Meteorol. 248, 197–204. Guilpart, N., Roux, S., Gary, C., Metay, A., 2017. The trade-off between grape yield and grapevine susceptibility to powdery mildew and grey mould depends on inter-annual variations in water stress. Agric. For. Meteorol. 234–235, 203–211. HéCtor, V., Marc, F., Jean, R., AgnèS, C., Christian, G., 2008. Grey mould incidence is reduced on grapevines with lower vegetative and reproductive growth. Crop Protect. 27 (8), 1174–1186. Li, H., Zhai, C., Weckler, P., 2016. A canopy density model for planar orchard target detection based on ultrasonic sensors. Sensors 17 (1), 31. Li, M., Shi, Y., Wang, X., Yuan, H., 2010. Target recognition for the automatically targeting variable rate sprayer. IFIP Int. Fed. Inf. Process. 346, 20–28. Maghsoudi, H., Minaei, S., Ghobadian, B., Masoudi, H., 2015. Ultrasonic sensing of pistachio canopy for low-volume precision spraying. Comput. Electron. Agric. 112, 149–160. Magney, T.S., Eitel, J.U.H., Griffin, K.L., Boelman, N.T., Greaves, H.E., Prager, C.M., Logan, B.A., Zheng, G., Ma, L.X., Forting, E.A., Oliver, R.Y., Vierling, L.A., 2016. LiDAR canopy radiation model reveals patterns of photosynthetic partitioning in an Arctic shrub. Agric. For. Meteorol. 221, 78–93. Malnersic, A., Dular, M., Sirok, B., Oberti, R., Hocevar, M., 2016. Close-range air-assisted precision spot-spraying for robotic applications: aerodynamics and spray coverage analysis. Biosyst. Eng. 146, 216–226. Narváez, F.J.Y., Pedregal, J.S.D., Prieto, P.A., Torres-Torriti, M., Cheein, F.A.A., 2016.
41
Contents lists available at ScienceDirect
Crop Protection journal homepage: www.elsevier.com/locate/cropro
A leaf-wall-to-spray-device distance and leaf-wall-density-based automatic route-planning spray algorithm for vineyards
T
Guandong Gaoa, Ke Xiaob,c, Yuejin Mac,∗ a
Department of Information Management, The National Police University for Criminal Justice, Baoding, Hebei Province 071000, China College of Information Science & Technology, Hebei Agricultural University, Baoding, Hebei Province 071001, China c College of Mechanical and Electrical Engineering, Hebei Agricultural University, Baoding, Hebei Province 071001, China b
A R T I C LE I N FO
A B S T R A C T
Keywords: LW–SD distance Leaf wall density Spray route Grapevine
In this study, to achieve precision spraying of pesticide in vineyards and reduce pesticide waste and pollution, algorithms were proposed for estimating parameters such as the average distance between the leaf wall and the spray device (LW–SD distance) and the leaf-wall density by integrating colour and depth images acquired by Microsoft Kinect. First, the colour video images were segmented using a morphological image segmentation technique to accurately separate the leaf walls from the remaining images. Then, based on the depth images, algorithms for estimating the average LW–SD distance and the leaf wall density were established to estimate the spray control parameters. A spline area-based algorithm for estimating the average LW–SD distance was proposed to accurately estimate the LW–SD distance. Finally, an algorithm for estimating the route deviation, as well as an algorithm for correcting and planning the spray route, was established to guide and maintain the spray device on the optimal path. The experimental results demonstrate that the spray parameter estimation algorithms produce relatively small errors in the estimation of the average LW–SD distance and the leaf wall density. In addition, the differences between the spray distance and route deviation estimated using the route planning algorithm and the measurements were also relatively small, demonstrating the accuracy of the algorithm.
1. Introduction Grapevines are prone to pests and diseases (Nita et al., 2007; Chuche and Thiéry, 2009), of which powdery mildew and grey mould severely affect the growth and development of grapes (HéCtor et al., 2008; Valdésgómez et al., 2011; Calonnec et al., 2013; Guilpart et al., 2017). Pesticides used to manage crop pest and diseases have adverse effects on human health and the environment. Therefore, a major challenge facing researchers is the reduction of pesticide usage while maintaining crop yields. Hence, in recent years, precision spraying (Maghsoudi et al., 2015; Malnersic et al., 2016; Qiu et al., 2016) has increasingly garnered more attention. The aim of studying precision spraying technology is to acquire accurate information related to the target (fruit trees, when used in orchards). Two common indices reflect the crop size, namely, the leaf area index (LAI) (Magney et al., 2016; Pearse et al., 2016; Woodgate et al., 2016; Gonsamo et al., 2017) and the leaf wall area (LWA) (Gil et al., 2011; Walklate et al., 2011; Escolà et al., 2013; Walklate and Cross, 2013). The LAI is defined as one half the total leaf surface areas per unit ground area. It is a forest canopy variable that is closely related to forest
∗
Corresponding author. E-mail address: [email protected] (Y. Ma).
https://doi.org/10.1016/j.cropro.2018.04.015 Received 29 December 2017; Received in revised form 9 April 2018; Accepted 16 April 2018 0261-2194/ © 2018 Elsevier Ltd. All rights reserved.
growth and health (Pearse et al., 2016). The Leaf Wall Area model only uses two parameters, tree height and row distance (Pergher and Petris, 2008). Thus, row distance and canopy height are the key to the calculation of the LWA. It can reasonably be used as the basis for research. Currently, several target detection technologies exist, including light detection and ranging (LiDAR)-based, ultrasonic sensor-based, infrared (IR) light-based and machine-vision-based target detection technologies. In terms of LiDAR-based target detection, Duga et al. (2016) proposed a three-dimensional (3D) spray drift model based on computational fluid dynamics that considers the canopy wind speed, the amount of movement of the sprayer, and the airborne spray drift distance. This model was used to reduce the spray drift distance; however, its use increased near-ground fertilizer deposition. Because they are generated from point light sources, scanning laser could be influenced by moving targets such as fluttering leaves. Moreover, due to their high cost, laser beam scanners are unsuitable for extensive popularization. In ultrasonic sensor-based target detection, to establish an orchard-canopy planar-target density model, essential for calculating canopy densities, Li et al. (2016) proposed a target density-detection system based on an ultrasonic sensor and employed a time-domain
Crop Protection 111 (2018) 33–41
G. Gao et al.
Fig. 1. The algorithm flow diagram of the precision spraying system.
2. Materials and methods
energy analysis method to analyse the ultrasonic signals. Ultrasonic sensors may cause errors in the determination of distances and densities to fruit trees, when the detection distance is too close or there is fog. In terms of IR light-based target detection, Narváez et al. (2016) combined an IR sensor with a LiDAR system. They first classified the canopies based on the corresponding LiDAR measurements and then used the IR sensor to match these points. Near-IR sensors have a small and nonuniform transmission range, resulting in increased processing difficulty and errors. In terms of machine-vision-based target detection, Bargoti et al. (2015) presented a method for identifying individual trees in an apple orchard using ground-based LiDAR and image data and evaluated it by conducting an experiment in an apple orchard where apples were planted in two different trellis structures, namely, a standard vertical trellis structure and an updated Guttingen V-trellis structure. This method had a few shortcomings: the camera was incapable of acquiring distance information, and the leaf wall detection was easily affected by the complex background. In this study, to address the aforementioned lack of distance information associated with machine-vision-based target detection technologies, an approach involving the use of integrated Microsoft Kinect sensors was presented to measure distances, and a precision spray algorithm capable of automatically estimating spray parameters and planning routes was also established. To determine the precise location of the spray target, the system used two Kinect devices to acquire colour and depth images on its left side and at its front and employs a leaf wall segmentation technique to accurately extract the leaf walls. In addition, algorithms were also established to estimate the leaf wall parameters including density and average distance between the leaf wall and the spray device (LW–SD) based on the information acquired from the left angle of view (AOV). Finally, to plan the route for the spray device, a spline-based algorithm for estimating the LW–SD distance based on the information acquired from the forward AOV was established. The spline-based LW–SD distance was estimated as the characteristic parameter to correct and plan the spray route, as well as to plan and predict the optimal spray route to further achieve precision spray control and reduce the amount of pesticide wasted and pollution dispersed.
2.1. Experimental materials The experimental spray system used two Kinect devices (model number: XBOX 360 Kinect 1.0), with one being fixed to the front and one to the left side of the spray platform to acquire video images of the forward route and the leaf wall on the left side, respectively. The forward speed was about 1.1 m/s (or 4 km/h). It also comprised a portable computer, a portable power source, hydraulic valves, movable spray arms, air conduits, air-assisted sprayers and a spray adjustment and control device. And it can adjust the spray distance by outputting a control signal through the processing software. The spray arm consisted of upper, middle and lower nozzles at a height of 60 cm, 125 cm, and 200 cm. It could be moved in and out at range of 0–50 cm in horizontal direction. The nozzles were fan-shaped that the spray range is 1.3 m. The experimental time was before 9a.m. and after 3p.m. in sunny day or the whole daytime in cloudy day, which was an “operation window”. It is because that Kinect has daytime light contamination problem for outdoor usage (George et al., 2013; Sharon et al., 2015). When the ambient light is too strong, the depth images will be turned into blank. Two vineyards (vineyards 1 and 2) in Hebei Province, China (northern China) were selected as the experimental sites. The grapevines in the two vineyards were all 5 years of age. The Muscat grape variety was planted in vineyard 1 with a plant spacing of 1 m, a row spacing of 3 m and the height of 2.2 m. The Kyoho grape variety was planted in vineyard 2 with a plant spacing of 1.5 m and a row spacing of 4 m and the height of 2.3 m. And all the row spacing is within the measurement range of Kinect. In the experiments, 30 sampling positions were chosen randomly, which the distance between each other is greater than 5 m. 2.2. Overview of the method The whole method consisted of two components, namely, a spray parameter estimation component, which calculated the leaf wall spray 34
Crop Protection 111 (2018) 33–41
G. Gao et al.
of segmenting the leaf wall into an image separate from the remaining area.
control parameters, and a spray route planning component, which planned the spray route. Fig. 1 shows the overall algorithm flowchart. 2.3. Spray parameter estimation algorithms
2.3.2. Average leaf-wall-to-spray-device distance estimation algorithm The average leaf-wall-to-spray-device distance is used as an index parameter that the Kinect device on the left side to control the spray distances by adjusting the spray arm. Therefore, calculation of the average distance between the Kinect and the grapevines is necessary. Through a derivation based on basic principles, an equation for estimating the average LW–SD distance was established. In a depth image, each pixel represents the distance between the scene and the Kinect. Thus, the average distance can be determined using the following equation:
To achieve intelligent precision spraying of grapevines in vineyards, the identification of the leaf walls and determination of the distance between each leaf wall and the spray system were needed to determine a spray route that could lead to the optimal spray distance and minimize the dose of pesticide sprayed and the energy consumed. 2.3.1. Leaf wall segmentation algorithm Most studies using machine-vision-based leaf wall detection use image processing techniques to analyse and process each frame in a video sequence and detect the leaf walls by image erosion (Narváez et al., 2016). In this approach, a next row of fruit trees will be misidentified as a leaf wall when used as the background, resulting in an inaccurate estimation of the distance. To segment the leaf wall, a colour image was first processed using a colour segmentation algorithm. A subtraction operation was performed with the pixel values of the green and red layers of the RGB image. The specific equation is as follows:
Ibin (x , y ) =
⎧ 255, ⎨ 0, ⎩
N
Dav =
N
∑i = 1 i
, Ii ∈ ILWA (2)
where Dav is the average distance between the grapevines and the Kinect; Ii is the distance of each pixel of the leaf wall; and N is the number of pixels of the leaf wall, i ∈ [1, N ]. When calculating the average LW–SD distance, each acquired image was divided into three (upper, middle and lower) areas based on the spray areas of the three nozzles on the spray arm. The distance between each of the upper, middle and lower sections of a grapevine leaf wall and the Kinect was calculated using Equation (2). Fig. 3 shows the three areas of an image in which the leaf wall is segmented from the background, corresponding to the locations and spray areas of the three nozzles.
if IG (x , y ) − IR (x , y ) ≥ 0 if IG (x , y ) − IR (x , y ) < 0
∑i = 1 Ii
(1)
where IG(x,y) and IR(x,y) are the pixel values of the green and red layers of the colour image at (x,y), respectively, and Ibin(x,y) is the image obtained from binarization. When segmenting the leaf wall in a left-side Kinect image, the distant grapevine background in the low-density part of the leaf wall, i.e., the background consisting of green leaves in a distance, must be removed. Therefore, the image obtained from binarization and the corresponding depth images were used for logical And operations to remove background. Moreover, the gaps between the leaves and the shadows in a leaf wall can easily affect the extraction of its outline. Therefore, the small gaps and voids in each image need to be filled. In this study, a dilation algorithm, specifically, a closed morphological operation algorithm (Li et al., 2010), was employed to fill the small gaps using the structure elements of a 5 × 5 square matrix. Then, the algorithm further fills the relatively large voids within 300 pixels based on four-connected pixels. Fig. 2 shows the leaf wall segmentation process and results. Experimental results showed that this algorithm was relatively highly capable
2.3.3. Spline-based algorithm for estimating the LW–SD distance For the spray route planning, to effectively estimate the vertical LW–SD distance when the spray device was advancing, the derivation of distance information from the depth images acquired by the forward Kinect and prediction of the vertical distance between the leaf wall in the front and the spray device were necessary. Therefore, a spline areabased algorithm was proposed to estimate the average LW–SD distance. Two vertical spline areas with a width of 20 pixels were marked at the two vertical edges of each leaf wall image, and the average of the distances between the spray device and the pixels of the leaf wall within these two splines was calculated and used as the average distance between the leaf wall and the Kinect on the spray device to reduce the single-point estimation error. The equation is as follows:
Fig. 2. The process and results of LWA segmentation. 35
Crop Protection 111 (2018) 33–41
G. Gao et al.
estimation by the total area of the leaf wall (including the leaf wall and the voids inside the leaf wall):
LWAdens =
sl sLWA
(4)
where LWAdens is the leaf wall density, Sl is the area of the image where the leaves are located, and SLWA is the section of the leaf wall within the rectangular box. In fact, Sl represents all the pixels that represent the leaves in the depth image, and SLWA represents the total number of pixels of the leaf wall. LWAdens is the percentage of the leaf wall covered by leaves. According to Equation (4), when the leaf wall density is low, the spray pressure of the nozzles is low, and the dose of pesticide sprayed per unit area is small. 2.4. Spray route planning algorithm Spray route planning mainly relies on the processing of the forward video images. The advancing route of the spray device was estimated and planned based on the LW–SD distance, the optimal spray distance for the nozzles and the current location of the spray device. The optimal spray results were obtained by correcting the route deviation.
Fig. 3. Division of spraying area and LWA. N
Disav =
∑i = 1 Di (x , y ) N
, Di (x , y ) ∈ ILWA (x , y )
(3) 2.4.1. Vertical spray distance estimation To plan the spray route, estimation of the spray distance, which can be obtained by subtracting the width of the spray device from the vertical LW–SD (mainly the Kinect) distance, is necessary. Therefore, the vertical LW–SD distance within the horizontal AOV of the Kinect was calculated. Fig. 5 shows the specific algorithm for estimating the spray distance. The left-side straight line signifies the leaf wall. Point O signifies the location of the Kinect camera lens. The rectangular box signifies the field of view (FOV) of the Kinect. Point A is the median point of the leaf wall spline. Point B is the point of intersection between the vertical line that goes through point O and the horizontal line extending from median point A. Point C is the centre of the FOV of the camera. The horizontal AOV of the Kinect is 54°. Thus, the angle between the centre and the edge of an image acquired by the Kinect, ∠AOC, is approximately half of its horizontal FOV. Here, we establish that ∠AOC ≈27°. Because ∠BOC = 90°, we have
Furthermore, x ∈ [1, Sw] ∪ [W , W − Sw], y ∈ [1, H ] Where Di(x,y) is the pixel value of the depth image at (x,y), i.e., the distance; N is the number of pixels; ILWA is the binarized segmented leaf wall image; SW is the width of the spline areas; W and H are the width and height of the image; and Disav is the average LW–SD distance. To avoid issues that may arise due to the potential blank strips at the edges of a depth image, the outer boundary of each spline area was set at a horizontal distance of 10 pixels from the closest edge of the image. In Fig. 4, the red areas signify the spline areas used for estimating the average LW–SD distance. The areas that do not contain any part of the leaf wall in each spline will be removed before calculation. 2.3.4. Leaf wall density estimation algorithm Plant growth is inhomogeneous, resulting in a relatively large regional difference in the leaf wall density. When spraying the low-density sections of a leaf wall, if the dose of pesticide sprayed per unit area remains unchanged, the result will be pesticide waste and pollution drift. Therefore, leaf wall density is proposed as an index for the side video images, which can facilitate the automatic adjustment of the flow to control the sprayed dose. Due to the current lack of a relevant equation, a leaf wall density estimation equation was established based on the basic principle of density calculation. The leaf wall density was estimated by dividing the area of the section of the leaf wall used for
∠AOB = ∠BOC - ∠AOC = 90 – 27 = 63°
(5)
Because the length of the line AO is known from Equation (3) (i.e., Disav), the length of line BO can be determined using the following equation:
DisLWA = BO = AO × cos 63° = Disav × cos 63°
(6)
The distance between the leaf wall and each nozzle can then be determined by subtracting half of the width of the spray device from the result obtained from Equation (6), i.e.,
Disp = DisLWA − Wp/2
(7)
where WP is the width of the spray device, and DisP is the spray distance between each nozzle and the leaf wall. 2.4.2. Route deviation estimation algorithm To optimize the spray path, calculation of the difference between the route and the advancing direction between the grapevine rows and the optimal spray distance is necessary to correct the route. Therefore, first, identification of the boundaries between the leaf walls and the ground is necessary to determine the route range between the adjacent grapevine rows. The method of least squares was employed to linearly fit the edge of each leaf wall in each post-leaf-wall-segmentation image to determine the boundaries. Hence, the edge is first detected using the following equation:
Fig. 4. Spline areas for estimating the LW-SD distance. 36
Crop Protection 111 (2018) 33–41
G. Gao et al.
between leaf walls and the ground, based on the datasets of the coordinates of the points on the edges of the left and right walls, is necessary. After determining the boundaries between the leaf walls and the ground through fitting, the deviation between the current route and the optimal route could be calculated. This deviation was the difference between the average LW–SD distance and the spray distance on the left or right side calculated using Equation (7).
Δpe = (DispL + DispR)/2 − DispL
(12)
or
Δpe = DispR − (DispL + DispR)/2
(13)
where DisPL and DisPR are the left and right spray distances, respectively, obtained using Equation (7), and Δpe is the route deviation. In addition, the calculation of the current advancing route of the spray device and comparison of it with the planned route were necessary. The current advancing route in an FOV image was calculated based on the width of the spray device with the centre of the bottom of the FOV as the reference. 2.4.3. Route correction and planning line generation algorithm To generate a route correction planning line, the turning angle of the spray device must be calculated first. Let us set Δx as the deviation between the current route and the planned route at a certain point p(x, y) on a line extended from the centre of the spray device in the image coordinate system (because the advancing route is primarily corrected in the horizontal direction, the deviation is only the difference along the x-axis). Thus, the turning angle can be calculated using the following equation:
α= tan−1
xl2 + (y1 − L cos α )2 = (L cos a − Wl /2)2
(8)
x r2 + (yr − L cos α )2 = (L cos α + Wl /2)2
Each fitted straight boundary line is obtained by calculating the coefficients a and b based on the dataset of the coordinates of the points on the leaf wall edge. The optimal estimates of the coefficients a and b can be determined using the following equations:
N (∑ x i yi ) − (∑ x i )(∑ yi ) N (∑ x i2) − (∑ x i )2
(10)
bˆ =
(∑ x i2)(∑ yi ) − (∑ x i )(∑ x i yi ) N (∑ x i2) − (∑ x i )2
(11)
(16)
where (xl,yl), (xr,yr) are the coordinate points of the left and right wheels in the image, respectively; L is the wheelbase of the spray vehicle; and Wl is the front wheel tread. The map obtained from the calculation using the above equations is a bird's eye view of the advancing route. However, in reality, a
(9)
aˆ =
(15)
The equation for the advancing trajectory of the wheels on the right side of the spray vehicle is as follows:
where Ig is the edge detection result, ILWA is the binary segmented leaf wall image, B is a suitable structure element (in this study, B is a 3 × 3 matrix, each of whose components is 1), and Θ is a morphological erosion operation. After the edge of each leaf wall has been detected, the coordinates of all the points on the edge were used to form a dataset for linear fitting. A straight boundary line between each leaf wall and the ground was fitted using the method of least squares, which satisfies the following equation:
y = ax + b
(14)
Then, the turning route of the spray device is further calculated. Fig. 6 shows a schematic diagram of the turning route of the spray vehicle. Thus, the equation for the advancing trajectory of the wheels on the left side of the spray vehicle is as follows:
Fig. 5. The diagram for computing the spraying distance.
Ig = ILWA − (ILWA ΘB)
Δx y
where xi,yi∈Ig are the coordinates of a point on the leaf-wall edge. Because leaf walls and the edge lines are mainly located in the lower half of an image, yi∈[H/2,H], where H is the image height, and N is the number of edge points. Because leaf walls are generally located on the left and right sides of the FOV, determination of the boundaries
Fig. 6. Rotation diagram of the spraying vehicle. 37
Crop Protection 111 (2018) 33–41
G. Gao et al.
In Table 1, the mean values of the differences between the measured and estimated values are relatively small. The smallest value is 0.4606 cm, and the largest one is 0.8956 cm. The RMSEs between the estimated and measured average distances between the upper, middle and lower sections of the leaf wall and the spray device are 1.2767, 1.8256 and 1.8822, respectively, demonstrating that the differences between the measured and estimated values are relatively small and that the average LW–SD distances calculated based on the depth images acquired by the Kinect are in good agreement with the actual distances; i.e., the estimated values are accurate. 3.1.2. Spline-based LW–SD distance estimation The experiment was conducted in vineyard 2. First, the spray system was allowed to stand still in the space between the different grape trellises. The spray distance (DisLWA.) was calculated using Equation (7) based on the data acquired by the Kinect. The vertical distance was measured using a tape by three times at each location. And the average value used as the measured spray distance. During the experiment, 30 sample points were randomly selected, which the spray system was at left, centre and right of the route. Fig. 9 shows the results. Fig. 9(a) shows the histogram of the estimated spray distances on the left and right sides of the spray device. The red columns corresponding to the left deviation of the route are relatively short, whereas the blue columns are relatively long. This indicates that the estimated spray distances on the left side are relatively short, and the estimated spray distances on the right side are relatively long. Therefore, a relatively even distribution of the estimated distances can be observed in Fig. 9(a). Fig. 9(b) and (c) shows the broken lines connecting the estimated and measured distances on the left and right sides, respectively, of the spray device and their differences. From these figures, the differences between the estimated and measured values are relatively small, indicating that the estimated values are relatively accurate. Moreover, the relatively large number of yellow areas indicates that the estimated values are more frequently greater than the measured values. It is because the estimated distance was average value, which could be affected by sags and crests of leaf wall. Table 2 summarizes the mean values of the 30 groups of experimental data, as well as the mean values of the measured and algorithmestimated values and the standard deviations. From Table 2, the mean values of the differences between the estimated and measured distances on the left and right sides are small as 1.5 and 2.4 cm, respectively, and the RMSEs are also small (3.9 and 3.1, respectively). This indicates that the estimated values are relatively accurate.
Fig. 7. Planned curve of corrected route.
trajectory with the coordinates of the Kinect camera as the coordinates of the central point should be displayed in each image photographed by the Kinect. Therefore, a route correction planning line should be generated and displayed in the colour image. Fig. 7 shows the planned route in the vineyard. The green dotted line signifies the route correction planning line. Based on the route correction planning line, the spray device can be guided to remain at the centre of the route when advancing. The route planned by the algorithm is the optimal spray route. 3. Results 3.1. LW–SD distance estimation results 3.1.1. Average LW–SD distance estimation results An experiment was conducted in vineyard 1 to validate the estimation of the average LW–SD distance. The average LW–SD distance was calculated using the 30 group data acquired by the left side Kinect for comparison with the estimated values in Fig. 8. Fig. 8(a) shows the lines connecting the average distances between the spray device and the upper, middle and lower sections of the leaf wall estimated using the algorithm established in this study. The points on the broken lines are the average distances estimated between the sample points on the leaf wall and the spray device. As demonstrated in Fig. 8(a), similar patterns can be observed in the distribution of the average distances; in addition, compared to the average distances between the lower sections, the upper and middle sections are relatively shorter. This is because the leaves in this section grew luxuriantly, took up a relatively large space and were thus closer to the spray device. Fig. 8(b), (c) and (d) shows the broken lines connecting the estimated and measured average distances between the spray device and the upper, middle and lower sections of the leaf wall and their differences, respectively. In each of these plots, the red broken line signifies the estimated values, and the blue broken line signifies the measured values; in addition, the differences between the estimated and measured values are also marked with yellow and blue areas, which are relatively small, suggesting that the estimated values are relatively accurate. Moreover, the relatively large of yellow areas indicates that the measured values are more frequently greater than the estimated values. This is because some protruding leaves on the leaf wall surface affected the average estimated values, resulting in relatively lower estimated values. Furthermore, the mean values of the estimated and measured average distances between the 30 sample pointed on the leaf wall and the spray device, as well as the mean values of the differences and the root-mean-square errors (RMSEs) between the estimated and measured values, were also calculated. Table 1 shows the results.
3.2. Leaf wall density estimation results Estimation of the leaf wall density can facilitate better control of the sprayed dose of pesticide in section 2.3.4. Fig. 10 shows the leaf wall densities at 30 sample points estimated from the top, middle and bottom areas of the images, which were collected from vineyard 1. The red, blue and yellow line connects the top, middle and bottom densities. In Fig. 10, the densities in the middle and bottom sections of the leaf wall are the highest, whereas the densities in the top section are the lowest. The mean values of the 30 groups of density data in the different sections were calculated. The results show that the mean densities in the top, middle and bottom sections are 0.2481, 0.6800 and 0.6720, respectively. The relatively low densities in the top section are a result of containing a sky background partly. Consequently, the leaf wall densities are relatively high in the middle and bottom sections and relatively low in the top section. 3.3. Route deviation estimation results To validate the estimated route deviations, route deviation data at 30 sample points were collected from vineyard 2. Fig. 11 shows the experimental results. The red dotted line signifies the measured values, 38
Crop Protection 111 (2018) 33–41
G. Gao et al.
Fig. 8. The average LW-SD distance in the top, middle and bottom.
4. Discussion
the blue solid line signifies the algorithm-estimated values, and the black solid line signifies each result of the subtraction of a measured value from the corresponding estimated value (a positive value indicates that the spray device was slightly to the left of the route; a negative value is to the right). The curve depicting the differences between the measured and estimated values fluctuates to a relatively small extent around the 0-axis. In addition, the variance of the differences between the estimated and measured values is 2.29 cm. Because the optimal spray distance range for the nozzles of the spray device is 30–40 cm, the estimation error will basically have no significant impact. Thus, the experimental results confirmed that the route deviations estimated using the algorithm established in this study were accurate.
The main objectives of this study were to establish algorithms for estimating control parameters for precision spraying and route planning based on Kinect video processing techniques to optimally control the spraying of pesticide in vineyards and plan the optimal spray route in order to achieve precision spraying, thus preventing the waste of pesticide and generation of pollution. Two Kinect 1.0 devices, which face to the left side and the front to acquire colour and depth image data, were used to estimate the spray distance and leaf wall density and plan the optimal spray route. The algorithm first segmented the leaf wall using morphological image segmentation technique. Based on the depth image data, an algorithm for estimating the average LW–SD distance, a spline area-based algorithm for estimating the LW–SD distance and an algorithm for estimating the leaf wall density were established for
Table 1 Means and errors of estimated and measured distances.
Means (cm) Mean of errors (cm) RMSE
Estimated distance at top region
Measured distance at top region
Estimated distance at middle region
Measured distance at middle region
Estimated distance at bottom region
Measured distance at bottom region
47.0523 0.8377
47.89
28.9294 0.4606
29.39
27.2844
28.18 0.8956
1.2767
1.8256
1.8822
39
Crop Protection 111 (2018) 33–41
G. Gao et al.
Fig. 9. The comparison of the LW-SD distances between measured and estimated values on the left and right sides. Table 2 The data on the left and right distances between the spraying system and LWA.
Means (cm) Mean of errors (cm) RMSE
Left side of the spraying system
Right side of the spraying system
Measured value
Estimated value
Measured value
Estimated value
76.57 1.5
78.07
73.06 2.4
75.49
3.9
3.1
Fig. 11. The line chart of route deviation.
differences between the estimated and measured route deviations was 2.29. This demonstrated that the differences between the measured and estimated values were relatively small. The algorithms proposed in this study are capable of accurately estimating the spray control parameters. And the planned spray route can be accurately maintained at the centre of the route. In the future work, to overcome Kinect's light contamination shortage, we plan to use 3D ToF depth sensor - Pico Monstar, which the measurement range is 6 m. And it is very robust against ambient light that can be used outdoors. Fig. 10. Leaf wall density in the top, middle and bottom.
References
automatically adjusting the spray arms and nozzles of the spray device. Furthermore, an algorithm for estimating the route deviation and planning the route were also established to keep the spray device remaining on the optimal spray route. To validate the proposed algorithms, a field experiment was conducted in vineyards. The statistical analysis of the key measured and estimated values showed that the RMSEs between the average LW–SD distances estimated using the algorithm based on the upper, middle and lower areas of the images taken from the left AOV and the measurements were 1.2767, 1.8256 and 1.8822, respectively; the RMSEs between the spray distances on the left and right sides estimated based on the forward AOV and the measurements were 3.9 and 3.1, respectively; and the variance of the
Bargoti, S., Underwood, J.P., Nieto, J.I., Sukkarieh, S., 2015. A pipeline for trunk detection in trellis structured apple orchards. J. Field Robot. 32 (8), 1075–1094. Calonnec, A., Burie, J.B., Langlais, M., Guyader, S., Saint-Jean, S., 2013. Impacts of plant growth and architecture on pathogen processes and their consequences for epidemic behaviour. Eur. J. Plant Pathol. 135 (3), 479–497. Chuche, J., Thiéry, D., 2009. Cold winter temperatures condition the egg-hatching dynamics of a grape disease vector. Naturwissenschaften 96 (7), 827–834. Duga, A.T., Delele, M.A., Ruysen, K., Dekeyser, D., Nuyttens, D., 2016. Development and validation of a 3D CFD model of drift and its application to air-assisted orchard sprayers. Biosyst. Eng. 154, 62–75. Escolà, A., Rosell-Polo, J.R., Planas, S., Gil, E., Pomar, J., 2013. Variable rate sprayer. Part 1 – orchard prototype: design, implementation and validation. Comput. Electron. Agric. 95 (1), 122–135. Gil, E., Llorens, J., Landers, A., Llop, J., Giralt, L., 2011. Field validation of Dosaviña, a decision support system to determine the optimal volume rate for pesticide application in vineyards. Eur. J. Agron. 35 (1), 33–46.
40
Crop Protection 111 (2018) 33–41
G. Gao et al.
LiDAR and thermal images fusion for ground-based 3D characterisation of fruit trees. Biosyst. Eng. 151, 479–494. Nita, M., Ellis, M.A., Wilson, L.L., Madden, L.V., 2007. Effects of Application of Fungicide during the Dormant Period on Phomopsis Cane and Leaf Spot of Grape Disease Intensity and Inoculum Production, vol 90. pp. 1195–1200 9. Pergher, G., Petris, R., 2008. Pesticide dose adjustment in vineyard spraying and potential for dose reduction. Agric. Eng. Int. CIGR Ejournal X, 1–9. Pearse, G.D., Watt, M.S., Morgenroth, J., 2016. Comparison of optical LAI measurements under diffuse and clear skies after correcting for scattered radiation. Agric. For. Meteorol. 221, 61–70. Qiu, W., Sun, C., Lv, X., Ding, W., Feng, X., 2016. Effect of air-assisted spray application rate on spray droplet deposition distribution on fruit tree canopies. Appl. Eng. Agric. 32 (6), 739–749. Sharon, N., Goldberger, J., Alchanatis, V., 2015. Obstacle detection in a greenhouse environment using the Kinect sensor. Comput. Electron. Agric. 113, 104–115. Valdésgómez, H., Gary, C., Cartolaro, P., Lolascaneo, M., Calonnec, A., 2011. Powdery mildew development is positively influenced by grapevine vegetative growth induced by different soil management strategies. Crop Protect. 30 (9), 1168–1177. Walklate, P.J., Cross, J.V., Pergher, G., 2011. Support system for efficient dosage of orchard and vineyard spraying products. Comput. Electron. Agric. 75 (2), 355–362. Walklate, P.J., Cross, J.V., 2013. Regulated dose adjustment of commercial orchard spraying products. Crop Protect. 54 (12), 65–73. Woodgate, W., Armston, J.D., Disney, M., Jones, S.D., Suarez, L., Hill, M.J., Wilkes, P., Soto-Berelov, M., 2016. Quantifying the impact of woody material on leaf area index estimation from hemispherical photography using 3D canopy simulations. Agric. For. Meteorol. 226–227, 1–12.
George, A., Goulden, M.L., Rusu, R.B., 2013. Rapid characterization of vegetation structure with a Microsoft Kinect sensor. Sensors 13 (2), 2384–2398. Gonsamo, A., Walter, J.M., Jing, M.C., Pellikka, P., Schleppi, P., 2017. A robust leaf area index algorithm accounting for the expected errors in gap fraction observations. Agric. For. Meteorol. 248, 197–204. Guilpart, N., Roux, S., Gary, C., Metay, A., 2017. The trade-off between grape yield and grapevine susceptibility to powdery mildew and grey mould depends on inter-annual variations in water stress. Agric. For. Meteorol. 234–235, 203–211. HéCtor, V., Marc, F., Jean, R., AgnèS, C., Christian, G., 2008. Grey mould incidence is reduced on grapevines with lower vegetative and reproductive growth. Crop Protect. 27 (8), 1174–1186. Li, H., Zhai, C., Weckler, P., 2016. A canopy density model for planar orchard target detection based on ultrasonic sensors. Sensors 17 (1), 31. Li, M., Shi, Y., Wang, X., Yuan, H., 2010. Target recognition for the automatically targeting variable rate sprayer. IFIP Int. Fed. Inf. Process. 346, 20–28. Maghsoudi, H., Minaei, S., Ghobadian, B., Masoudi, H., 2015. Ultrasonic sensing of pistachio canopy for low-volume precision spraying. Comput. Electron. Agric. 112, 149–160. Magney, T.S., Eitel, J.U.H., Griffin, K.L., Boelman, N.T., Greaves, H.E., Prager, C.M., Logan, B.A., Zheng, G., Ma, L.X., Forting, E.A., Oliver, R.Y., Vierling, L.A., 2016. LiDAR canopy radiation model reveals patterns of photosynthetic partitioning in an Arctic shrub. Agric. For. Meteorol. 221, 78–93. Malnersic, A., Dular, M., Sirok, B., Oberti, R., Hocevar, M., 2016. Close-range air-assisted precision spot-spraying for robotic applications: aerodynamics and spray coverage analysis. Biosyst. Eng. 146, 216–226. Narváez, F.J.Y., Pedregal, J.S.D., Prieto, P.A., Torres-Torriti, M., Cheein, F.A.A., 2016.
41