- Email: [email protected]
Identification of pruning branches in tall spindle apple trees for automated pruning
Computers and Electronics in Agriculture 103 (2014) 127–135
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag
Identification of pruning branches in tall spindle apple trees for automated pruning Manoj Karkee a,b,⇑, Bikram Adhikari a,b, Suraj Amatya a,b, Qin Zhang b a b
Biological Systems Engineering Department, Washington State University, United States Center for Precision and Automated Agricultural Systems, Washington State University, United States
a r t i c l e
i n f o
Article history: Received 27 September 2012 Received in revised form 22 February 2014 Accepted 26 February 2014
Keywords: Apple tree pruning Selective pruning Pruning algorithm Pruning branch identification Machine vision 3D camera
a b s t r a c t Pruning is a labor intensive operation that constitutes a significant component of total apple production cost. As growers are adapting simpler, narrower, more accessible and productive (SNAP) tree architectures such as the tall spindle fruiting wall system, new opportunities have emerged to reduce pruning cost and labor through automated pruning. This work focused on identification of pruning branches on apple trees in a tall spindle architecture. A time-of-flight-of-light-based three dimensional (ToF 3D) camera was used to construct 3D skeletons of apple trees. Pruning branches were identified in the reconstructed trees using a simplified two-step pruning rule; (i) maintain specified branch spacing and (ii) maintain specified branch length. Performance of the algorithm was optimized using a training sample of 10 trees to achieve human worker’s pruning level. With a selected branch spacing (28 cm) and branch length (20 cm), the algorithm achieved 19.5% branch removal with the training dataset and 19.8% of branch removal with the validation dataset (10 trees) compared to 22% average branch removal by workers. Root Mean Square Deviation (RMSD) between human and algorithm in number of branches identified for pruning was 10% for the training dataset and 13% for the validation dataset. The algorithm and the human pruning resulted in similar average branch spacing. The algorithm maintained an average spacing of 35.7 cm for validation set whereas the average spacing for three workers was 33.7 cm. RMSD in branch spacing between the algorithm and the workers was found to be 13%. The algorithm removed 85% of long branches whereas the overlapping branch removal was only 69%. With some additional work to improve the performance in terms of overlapping branch removal, it is expected that this work will provide a good foundation for automated pruning of tall spindle apple trees in the future. Ó 2014 Elsevier B.V. All rights reserved.
1. Introduction Pruning refers to removing old and/or damaged parts of trees, especially unproductive shoots and branches. It is essential to improve plant health and appearance, to control plant size and to increase fruit quality and yield. Pruning is labor intensive and could cost 20% or more of the annual pre-harvest production cost for crops such as cherries (Galinato et al., 2010), apples (Gallardo et al., 2010) and pears (Galinato and Gallardo, 2011). Automated pruning of fruit trees has a potential to reduce labor demand and cost, which will help to increase the long-term sustainability of fruit production. Researchers have been working on mechanization of fruit tree pruning for several decades. Early research work in pruning ⇑ Corresponding author at: Biological Systems Engineering Department, Center for Precision and Automated Agricultural Systems, IAREC-Washington State University, 24106 N Bunn Rd, Prosser, WA 99350, United States. Tel.: +1 509 786 9208; fax: +1 509 786 9321. E-mail address: [email protected] (M. Karkee). http://dx.doi.org/10.1016/j.compag.2014.02.013 0168-1699/Ó 2014 Elsevier B.V. All rights reserved.
focused on mass pruning in which a cutting tool was run over rows in orchards while maintaining a certain distance from the tree canopy center (Moore, 1958; Davies and Crocker, 1994; Gautz et al., 2002). These mass mechanical pruning systems have limited ability to ensure the quality of pruning and therefore may lead to reduced quality and yield of fruit (Moore-Gough and Gough, 2007; Jensen, 1980; Carbonneau, 1979; Sevilla, 1983; Velazquez and Gonzalez, 2010). Therefore, an automatic machine for selective pruning, which can identify the branches on the tree that need to be pruned, is essential to reduce manual labor while maintaining or improving the quality and productivity of fruit crops. Apple orchards are evolving towards simpler, narrower, more accessible and productive (SNAP) systems that are friendlier to mechanization and automation. These architectures represent various training systems that create very narrow two-dimensional canopies in which the majority of branches and fruit are visible and accessible to machines. These systems could increase profitability by improving yield and/or reducing labor cost (Robinson, 2008). One simplified architecture that lends itself better than
128
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
conventional orchard systems to mechanization is the tall spindle fruiting wall system. In this architecture, trees are planted at smaller intra-plant spacing compared to classical architectures and the branches are not allowed to grow multiple secondary and tertiary branches. This simpler tree architecture opens up an opportunity for automated pruning. A proper sensing technique is essential to identify unwanted branches in fruit trees for automated, selective pruning. Machine vision technology can be used to obtain such information nondestructively. In the past, some researchers have investigated the use of 2D machine vision systems to locate grapevines to guide an automated pruning device. Naugle et al. (1989), Ochs and Gunkel (1993), McFarlane et al. (1997), and Gao and Lu (2006) used a single 2D camera and image processing techniques for identification of pruning points in grapevines. However, extracting three-dimensional (3D) information of objects from a single two-dimensional (2D) image is difficult. It is essential to accurately represent objects and scenes in 3D to support the development of automatic pruning machines. Researchers have been attracted by 3D technologies such as stereo vision and structured light for general use in 3D sensing and object reconstruction. However, only limited studies have been reported in their application to 3D reconstruction and pruning of fruit trees. One such study was conducted in a laboratory setting using multiple images of a dormant apple tree (Tabb et al., 2009). A shape-from-silhouette method was used to reconstruct the shape of the tree from the images captured from multiple orientations. Though the method showed promise for generating a 3D tree structure for pruning, the method is complex and has challenges for practical use in the orchard environment. Recently, a 3D reconstruction method based on a low cost video gaming sensor was explored by Wang and Zhang (2013) for potential application in apple tree pruning. This preliminary study indicated that a high resolution 3D structure of fruit trees can be generated using RGB-D cameras (Wang and Zhang, 2013). However, the technique requires further work on understanding its accuracy and robustness in outdoor orchard environment. In the past several years, time-of-flight-of-light-based cameras (called 3D camera in the following text) have also been used for 3D reconstruction of plant structures. These 3D cameras have shown to be more accurate than stereo vision systems for such reconstruction (Beder et al., 2007). Lindner et al. (2007) also used a 3D camera with high resolution color images to present an efficient way of identifying false mappings due to hidden surfaces in 3D reconstruction. Nakarmi and Tang (2012) used a 3D camera for interplant space sensing in natural illumination conditions. These studies showed that 3D camera-based systems can be used to construct skeletons of apple trees in field environments, which then can be used to identify pruning branches for automated pruning. The main goal of this research was to detect and identify branches that need to be pruned in apple trees using a 3D machine vision system, which is the first step towards automated pruning of apple trees. Specific objectives were to:
Pacific Rose cultivar apple trees in their fourth year. These trees were planted in high density tall spindle fruiting wall architecture (Fig. 1). The intra-plant spacing was 0.46 m and between-row spacing was 2.44 m. Maximum canopy height was 2.74 m. A tree in this architecture consisted of a single central trunk with lateral branching. The images were captured in the late dormant season on April 05, 2012. 2.2. Branch identification A machine vision system was used to develop 3D structure of apple trees and identify branches. Four different steps were involved in processing the images captured by the 3D camera to identify branches; (i) image acquisition, (ii) image pre-processing, (iii) skeletonization, (iv) and skeleton analysis. Descriptions of each of these steps are provided below. Additional details of equipment, methods and algorithms used in these steps can be found in Adhikari and Karkee (2011) and Adhikari (2012). 2.2.1. Image acquisition A 3D camera (CamCube 3.0, PMD Technologies, Siegen, Germany) was used to capture 3D information of the apple trees (Fig. 2). The camera generated 3D coordinates of objects with reference to the imaging plane of the camera. The imaging system was mounted on a pan-and-tilt system (PTU-D48 E, FLIR Motion Control Systems, CA, USA). Using the pan-and-tilt system, a total of 20 different images at different positions were captured for each of the 20 trees randomly selected for this study. The pan-and-tilt system was rotated vertically at the rate of 1.285° per second between [30° and 30°] with reference to horizontal. The camera captured image frames at two-second intervals (i.e. at 2.57° interval). The platform was located approximately 1.27 m away from the center of the row of trees being imaged. 2.2.2. Image preprocessing First, the noise in 3D images was filtered out using two different parameters; (i) the flag and (ii) the amplitude data. The time-offlight (TOF) 3D camera provides a parameter called flag to indicate the quality of distance estimation at each pixel. Any pixel whose
Develop an algorithm to detect and identify pruning branches in apple trees in tall spindle orchard architecture. Optimize input parameters of the pruning branch identification method to achieve desired level of pruning and compare the results with actual human pruning. 2. Materials and methods 2.1. Study site This study was conducted on a test plot in a commercial apple orchard near Prosser, WA. The test plot consisted of four rows of
Fig. 1. Image of the tall spindle fruiting wall apple orchard in Prosser, WA that was used in the study.
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
129
3D Camera
Pan-Tilt System
(a)
(b)
Fig. 2. Imaging platform, (a) user interface computer and a 3D camera mounted on a pan-and-tilt system, and (b) imaging system in a tall spindle apple orchard.
corresponding flag value was not zero was identified as a noisy pixel. Amplitude, on the other hand, signifies the signal strength of the active illumination. This parameter provides an additional quality measure for distances measured by the camera. Very low amplitude value indicated a low accuracy of the distance measurement. In addition, the distance to the objects from the camera was used to filter out trees and other objects located outside the desired range of distances to the objects. The entire preprocessing operation changed pixels to background if, (i) the measurement flag was non-zero, (ii) the signal amplitude was less than 2000 (unitless quantity obtained empirically), or (iii) the object distance from the camera was outside the range of 0.3 m and 2.2 m. The images captured from the camera were radially distorted. This distortion was removed using the following two equations (Nakarmi and Tang, 2012):
xi ¼
f xw p zw þ c
ð1Þ
yi ¼
f yw p zw þ c
ð2Þ
where (xi, yi) are the corrected image coordinates to the captured image coordinates (xw, yw), zw is the depth axis value in the image coordinates, f is the focal length, p is the pixel-pitch of the camera (a parameter that describes the distance between dots on imaging plane), and c is the distance from the front face of the camera to the imaging area. Among these parameters, the value of c was measured to be 10 cm, whereas values of f and p were specified camera specifications. After background removal and radial distortion correction, different point clouds obtained for a single tree were registered together. Since the camera system was mounted on top of a panand-tilt system rotating at a fixed rate, a precise transformation matrix could be obtained between consecutive frames captured by the 3D camera. Each point cloud was captured when the pan-and-tilt system was rotated by an angle of {Hx, Hy, Hz} = {0°, 2.57°, 0°} and translated by {tx, ty, tz} = {0, 0, 0}, where H and t are rotation and translation along each of the three axes. Each point in a point cloud of a given frame, when transformed using a 3D transformation (Eq. (3)), was mapped to the new coordinate system of the next frame. This 3D transformation was performed recursively between consecutive frames to obtain the registered point cloud of the complete tree.
pt 0k ¼ pt k1 transform3dðhx ; hy ; hz ; t x ; ty ; t z Þ
ð3Þ
where, transform3d is a three dimensional transformation function for rigid body transformation (Eq. (4)).
transform3dðhx ;hy ; hz ; t x ;t y ;t z Þ 2 cos hx coshy cos hx sin hy sin hz sin hx coshz coshx sin hy cos hz þ sin hx sin hz 6 sin h cos h cosh sin h sin h þ cosh cosh sin h sin h cosh cos h sin h x y x y z x z x y z x z 6 ¼6 4 sinhy cos hy sin hz cos hy cos hz 0 0 0
tx
3
ty 7 7 7 tz 5 1 ð4Þ
2.2.3. Skeletonization Skeletonization is a way to reorganize 3D points into a set of fewer points that contain the required information about the object. Skeletal representation requires less memory and computation time compared to raw point cloud. A good skeletonization algorithm should hold various properties including centeredness satisfaction, connectivity preservation, and topology preservation (She et al., 2009; Adhikari and Karkee, 2011). In this work, the medial axis thinning algorithm (Lee and Kashyap, 1994) was used to generate 3D skeleton of apple trees from corresponding point clouds. The algorithm performed recursive thinning operations over the surface of the point cloud until the cloud was only one point (voxel) wide. Before applying the skeletonization algorithm, 3D images were dilated to fill unwanted holes and surfaces of the point cloud were smoothed by the 3D Gaussian filter to minimize spurs in the skeletons. 2.2.4. Skeleton analysis On the skeletonized 3D point cloud, all points connected to each other were labeled to identify individual connected objects. This connected object labeling separated the main tree from objects that were not connected to the tree. Then, the shortest connected paths were calculated between all junction points in the main tree. The longest of such paths was identified as the trunk of the tree. Each point on the trunk that had more than two connected neighboring points was defined as a junction/branching point. Junction points were sorted in ascending order based on the height (y-coordinate value). Distance between consecutive junction points represented the branch spacing in the trunk. The length of the branch was obtained by finding the longest path on the skeleton originating from each branching point. This longest path was represented by a set of points corresponding to the branch originating from the branching point on the trunk. The distance between end points of
130
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
branches originating from two consecutive branching points represented the branch spacing at the end of two consecutive branches. In skeletonization process, thinning algorithm extracted only one voxel wide skeletons. Therefore, it was possible to use unit of voxel to measure branch length and spacing. However, the voxel used in this work was 1 mm (deep) 2 mm (wide) 10 mm (tall), which was needed to optimize the data size and computational time while providing results with sufficient resolution. Therefore, a given branch length or branch spacing in voxels would mean different physical length or spacing based on the specific orientation of tree trunk and branches. To simplify the unit conversion, it was assumed that tree trunks were oriented vertically allowing measurement of branch spacing in the vertical direction. It was also assumed that branches were located in the vertical plane with an orientation of 45° to the horizontal plane. Because trees in the tall spindle architecture are grown vertically with branches allowed to grow only in a thin vertical wall, these assumptions were considered acceptable to demonstrate the potential of pruning branch identification algorithm. These assumptions allowed one-to-one conversion from physical units of branch length and spacing to voxels and vice versa.
algorithm over the total number of branches on the tree (Adhikari and Karkee, 2011). This parameter may vary from 0% to 100% indicating what percentage of the actual branches was detected by the algorithm.
BIA ¼
A field experiment was conducted to understand human pruning operation in tall spindle apple trees. Three workers (called worker1, worker2 and worker3 respectively) were selected from the orchard field crew. Twenty trees randomly selected from the test plot for imaging were used for the pruning trials as well. These three workers individually tagged pruning branches on each tree using three different colored markers. To keep tagging independent between workers, each tag was removed from the tree before another worker tagged the pruning points on the same tree. Color images of each tagged tree were captured. Finally, all tags were put together in each test tree to observe the consistency in pruning. Sample images (Fig. 3) show red, blue and green tags tagged by worker1, worker2 and worker3 respectively. Pruning branches identified by workers as well as the total number of branches were counted for each tree. All parameters measured or estimated to evaluate the algorithm are described below. 2.4. Performance evaluation measures 2.4.1. Branch identification accuracy (BIA) A measure called branch identification accuracy (Eq. (5)) was defined as the percentage of actual branches detected by the
End point of a branch
Start point of a branch
Fig. 3. Pruning branches in two sample images/trees tagged by three different workers (worker1 – red; worker2 – blue; worker3 – green). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
ð5Þ
where nb, total number of branches detected by the algorithm. nfp, no. of false positive branches (branches identified by the algorithm but not actual branches). nB, total number of actual branches on the tree. 2.4.2. Pruning Branch Proportion (PBP) Pruning Branch Proportion was defined as the percentage of branches selected for pruning over the total number of branches on the tree. This formulation can be applied to assess outcome of pruning by human as well as the algorithm. When used for calculating PBP for algorithm, pruning branches include false positive branches as well. Total number of branches (denominator of Eq. (6)) is always actual number of branches in the tree.
PBP ¼ 2.3. Study of manual pruning
nb nfp 100% nB
npa 100% nB
ð6Þ
where npa, no. of branches selected for pruning (including false positive branches when used for algorithm). nB, total number of actual branches on the tree. 2.4.3. Net Pruning Branch Proportion (NPBP) Net Pruning Branch Proportion was defined as the percentage of the number of actual branches selected by the algorithm over the total number of branches on the tree (Eq. (7)). Number of actual pruning branches was obtained by excluding all the pruning branches that were selected for pruning but were not actual branches. This measure was applicable to only the algorithm.
NPBP ¼
npa npfp 100% nB
ð7Þ
where npa, no. of pruning branches detected by the algorithm. npfp, false positive branch detected for pruning by the algorithm. nB, total no. of actual branches on the tree. 2.4.4. Other measures Percentage of false negative branches (actual branches in the tree but not detected by the algorithm) and false positive branches were calculated to assess the performance of branch identification algorithm. To assess the performance of pruning branch identification algorithm, number of branches identified for pruning by the algorithm and workers were compared. In addition, average branch spacing achieved by the algorithm was calculated and compared with the average spacing achieved by manual pruning. For each tree, branch spacing was computed as the distance between branches at their start point. To compute branch spacing on pruning branches tagged by human workers, each tagged branch was identified manually in the branch database of the corresponding tree. Finally, percentage removals of long and overlapping branches were estimated to further evaluate the performance of the branch identification algorithm. These two measures provided performance assessment of the algorithm independent of the results of manual pruning. Reference counts for long and overlapping branches were estimated from the color images captured during the data collection. 2.5. Pruning branch identification and pruning level optimization Field observation and analysis of the trees tagged by three workers (Section 2.3) showed that the workers used some level
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
of their subjective knowledge to detect unproductive, old or diseased branches for pruning. However, it was observed that there was a good agreement in the general strategy the workers used to select the pruning branches. First, long branches were selected by all three workers. Closely spaced branches were also selected, even though the three workers did not always choose the same branch for pruning. Similarly, in cases of overlapping branches, two different workers sometimes chose different branches for pruning but leading to the same outcome. These pruning approaches of the workers were in agreement with the opinion of experienced growers and horticulturists working in this area. This observation and evaluation was used to develop a pruning rule for automated identification of pruning branches as follows. Criterion (I): removing branches based on start point and/or end point spacing. Criterion (II): removing branches based on length. This pruning rule ensures that the vegetative structure of the tree is maintained by removing old wood and maintaining a certain spacing between branches. Overlapping between branches will also be avoided by maintaining certain spacing between branch endpoints, which ensures sufficient light interception for fruit to develop better color and texture. The three threshold values for branch length, end-point branch spacing, and start-point branch spacing required by these criteria were the inputs to the system, which would be provided by end users. Branches on the left and right sides of the tree were separated before pruning branches were identified. Then if two branches on the same side of the tree had start-point spacing or end-point spacing less than the threshold value, the branch with lower coordinate value was identified as the pruning branch. For branches that were not removed based on criterion (I), if their length was longer than the threshold length, they were also identified as pruning branches. In this study, different branch lengths and branch spacing were evaluated to achieve a desired pruning level. Branch length was varied from 14 to 56 cm, which was equal to 50 to 200 voxels. Similarly, start-point spacing was varied from 10 to 40 cm whereas end-point spacing was kept constant at 10 cm. The desired pruning level was estimated based on the average of manual pruning by three workers. First, a set of 10 trees were randomly selected as training dataset for the optimization algorithm. Remaining 10 trees were used to validate the performance of the algorithm after optimization. The objective function used was the sum of squares of the deviation between percentage branches removed by the algorithm and the workers (Eq. (8)). With this objective function, goal of the parameter optimization method was to minimize the objective function within the predefined range of parameter values.
f ¼
X
ðPBPWi PBPAi Þ2
ð8Þ
where i varies from 1 to 10 to represent the trees used to optimize the algorithm. PBP stands for Pruning Branch Proportion. Subscript W stands for workers and A stands for the algorithm. Two-step optimization process was used. In the first step, PBP (which includes false positives when calculated for the algorithm) was used in the objective function to narrow down the potential combinations of parameters to achieve desired pruning level. In the next step, NPBP (which does not include false positives and is a better performance indicator for the algorithm) was used to make the final selection of optimal parameters. The performances of the branch identification and pruning branch identification algorithms were evaluated using the performance measures described above. Analysis was performed in individual tree and worker level and Root Mean Square Deviations and/ or averages were calculated when appropriate. Different software
131
tools were used to carry out different tasks of this study. Matlab (version 7.12.0, Mathworks, MA) was used for image acquisition, image preprocessing, and identification of pruning branches. Fiji (Version 1.46j, RSB National Institute of Mental Health, MD) was used for skeletonization and to obtain junction points in the skeleton. 3. Results and discussions 3.1. Branch identification Actual branches in the tree, and total number of branches and actual branches detected by the algorithm were plotted in Fig. 4a. Number of branches detected by the algorithm (including false positives) as compared to the total actual branches on the trees was higher in nine of the trees and was lower in another eleven trees. Number of actual branches detected by the algorithm (true positives) was lower than total branches on tree in all cases. Branch identification accuracy (BIA) was also calculated for each tree as the ratio of actual branches identified by the algorithm to the total number of branches in the tree. On average, the algorithm achieved a branch identification accuracy of 77%. Even though the branch identification accuracy varied substantially (caused by the variability in false positives), the algorithm achieved 80% or higher identification accuracy for more than 60% of the trees evaluated in this study (Fig. 4b). The average branch identification accuracy was particularly skewed by the effect of a few trees (such as T17 and T19) with a dramatic error. On average, false positive and false negative on branch identification was 24% and 23% respectively. In general, the false positive detection was caused by the noise in the 3D cloud data that appeared to be small branches to the algorithm. The most critical reason for the false negatives, on the other hand, was that many short branches on the trees were washed away during image preprocessing and were not identified. In future, 3D reconstruction and branch identification algorithm will be improved to minimize false negative and false positive cases and increase branch identification accuracy. However, short branches, which were the primary reason of false positives and false negatives in this study, are not considered crucial in pruning apple trees because short branches do not cause much canopy overlap. 3.2. Human pruning evaluation In Table 1, number of pruning branches identified by individual workers and corresponding Pruning Branch Proportion (PBP) for all trees are shown. Number of pruning branches and therefore PBP varied substantially between workers. On average, worker1 pruned 28% of the branches with the standard deviation of 8.8% (Table 1). Worker3 was more conservative and consistent than the other two in selecting pruning branches, who selected 16% of the branches on average with a standard deviation of 7.3%. The average PBP of all the workers was 22%, which was consistent with recommendations of experienced horticulturists and apple growers in Pacific Northwest region (unpublished personal communication). Therefore, 22% PBP was set as the benchmark for the pruning branch identification algorithm used in this study. The variability between workers was partly caused by some level of subjective knowledge that workers used in pruning including identification of old, unproductive and diseased branches. It was also observed that there was some level of subjectivity in defining desired branch spacing and branch length even through the strategy of pruning was the same; to maintain certain branch spacing and limit branch length. This variability between individual workers, however, is not desirable and an automated method has the
132
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
35
Number of Branches
30 25 20 15 10 5 0 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15 T16 T17 T18 T19 T20
Tree (Number) Total Branches on Tree
Total Branches Detected
Actual Branches Detected
(a)
(b) Fig. 4. Accuracy of the branch identification algorithm; (a) actual branches on the tree and identified branches; and (b) distribution of branch identification accuracy (BIA). In Fig a, ‘total branches on tree’ represents the number of actual branches in individual trees, ‘Total Branches Detected’ represents the number of branches identified by the algorithm (including false positives) and ‘actual branches detected’ represents the number of actual branches identified by the algorithm.
Table 1 Number of actual branches, number of actual branches identified by the algorithm and number of pruning branches identified by three workers for 20 trees studied in this research. Tree #
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15 T16 T17 T18 T19 T20 Average Std. Dev.
Number of branches on tree
23 20 23 17 14 22 24 19 21 27 26 27 17 25 24 25 23 18 25 19
Number of pruning branches
Pruning Branch Proportion (PBP) (%)
W1
W2
W3
Average of workers
W1
W2
W3
Average of workers
8 9 9 7 7 7 6 3 5 6 7 8 3 6 7 8 5 5 7 5
4 3 5 6 3 5 5 4 1 3 3 4 2 5 2 3 2 1 5 4
7 5 7 8 6 5 3 5 5 6 8 8 3 4 3 3 2 1 4 2
6.3 5.7 7.0 7.0 5.3 5.7 4.7 4.0 3.7 5.0 6.0 6.7 2.7 5.0 4.0 4.7 3.0 2.3 5.3 3.7
34.8 45.0 39.1 41.2 50.0 31.8 25.0 15.8 23.8 22.2 26.9 29.6 17.6 24.0 29.2 32.0 21.7 27.8 28.0 26.3
17.4 15.0 21.7 35.3 21.4 22.7 20.8 21.1 4.8 11.1 11.5 14.8 11.8 20.0 8.3 12.0 8.7 5.6 20.0 21.1
30.4 25.0 30.4 47.1 42.9 22.7 12.5 26.3 23.8 22.2 30.8 29.6 17.6 16.0 12.5 12.0 8.7 5.6 16.0 10.5
27.5 28.3 30.4 41.2 38.1 25.8 19.4 21.1 17.5 18.5 23.1 24.7 15.7 20.0 16.7 18.7 13.0 13.0 21.3 19.3
28.2 8.8
16.0 7.3
22.1 10.8
22.0 7.5
133
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
Fig. 5. Surface plot of the objective function used to optimize pruning parameters (start-point branch spacing and branch length). The surface has a concave structure with an apparent global minimum within the parameter ranges evaluated.
Fig. 6. Net Pruning Branch Proportion (NPBP) for different combinations of branch lengths and start-point branch spacing values.
potential to reduce such variability and improve the consistency of pruning. 3.3. Pruning branch identification and optimization for desired pruning level A training dataset of 10 trees was used first to optimize the algorithm to match average Pruning Branch Proportion (PBP) achieved by three workers. This first step of two step optimization process narrowed down the selection for optimal combinations of pruning parameters. The plot (Fig. 5) of the objective function (Eq. (8)) developed with varying branch length and start-point branch spacing parameters formed a concave surface. PBP for branch length of 14 cm was high for all start-point spacing values as many branches were selected for pruning based on the short branch
length threshold. Therefore, values of the objective function, which represented the sum of squares of the deviation of PBP between human and algorithm, was very high in these cases. Similarly, for start-point spacing of 40 cm, the PBP of the algorithm was high for all branch lengths leading to high values of objective function. Minimum value of the objective function, which is the optimum result, was found with the combination of 28 cm of branch length and 20 cm of start-point branch spacing. However, some other combinations including 42 cm branch length and 20 cm branch spacing, and 56 cm branch length and 20 cm branch spacing also achieved smaller objective function values. Objective function used to develop the parameter space (Fig. 5) was defined based on PBP, which includes false positive branches. On average, about 11.0% of the selected branches were false positive for the selected branch combinations. The second step of the optimization was performed using Net Pruning Branch Proportion (NPBP), which excludes any false positive branches. Six most promising combinations of parameters identified in previous step was evaluated in this step (Fig. 6). The mean NPBP for training dataset with branch length of 28 cm and start-point spacing of 20 cm was 19.5% with a standard deviation of 7.2%. These rates were closet to the average PBP of the three workers, which was 22%. Other selected combinations achieved much lower NPBPs compared to the benchmark of 22%. Based on this result, branch length of 28 cm and branch spacing of 20 cm was considered to be optimal for achieving similar pruning results compared to that of human workers evaluated in this study. The number of actual pruning branches selected by the algorithm with optimal parameter combination was close to human pruning in most of the trees (Table 2). However, there were substantial differences in some other trees. Root Mean Square Deviation (RMSD) of NPBP was found to be 13% for validation dataset. One reason for this deviation was the variability between individual trees. In addition, workers applied some level of subjective judgment that led to pruning differences between workers, and workers and algorithm. Despite larger RMSD, the overall percentage branch removal with the algorithm was close to the desired branch removal percentage. Because overall percentage wood (branch) removal is a critical measure for farmers to ensure the extent of pruning in orchards, this result is encouraging and provides a basis for further research and development in automated apple tree pruning. Resulting branch spacing was also estimated (Table 3) for the optimal parameter combination of 28 cm branch length and 20 cm start-point branch spacing. On average, workers resulted in a branch spacing of 33.7 cm whereas the algorithm achieved the average actual branch spacing of 35.7 cm. Root Mean Square
Table 2 Performance of pruning branch identification algorithm with the optimal parameter combination of 28 cm branch length and 20 cm start-point branch spacing for the validation dataset of 10 trees. Tree #
T2 T3 T5 T7 T8 T11 T13 T15 T17 T18 Average Std. Dev.
No. of Actual Branches in Tree
20 23 14 24 19 26 17 23 23 18
No. of pruning branches Workers (average)
Algorithm
5.7 7.0 5.3 4.7 4.0 6.0 2.7 4.0 3.0 2.3
2 5 2 4 4 7 3 10 2 3
PBP for workers (%)
NPBP for algorithm (%)
Deviation (%)
28.3 30.4 38.1 19.4 21.1 23.1 15.7 17.4 13.0 13.0
10.0 21.7 14.3 16.7 21.1 26.9 17.6 43.5 8.7 16.7
18.3 8.7 23.8 2.8 0.0 3.8 2.0 26.1 4.3 3.7
22.0 7.4
19.7 9.0
RMSD = 13.1
134
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
Table 3 Average start-point branch spacing after manual and automated pruning for 10 trees used for validating optimal pruning branch identification algorithm. Tree
Start-point branch spacing (cm)
Deviation (%)
Average of workers
Algorithm
T2 T3 T5 T7 T11 T12 T13 T15 T17 T18
53.0 40.2 51.6 22.2 21.8 18.3 31.5 20.1 50.4 27.7
54.3 38.7 55.7 23.1 27.2 18.6 38.1 24.6 47.0 29.7
2.4 3.6 8.1 4.0 25.1 1.8 21.1 22.7 6.7 7.0
Average
33.7
35.7
RMSD = 13.4%
Table 4 Percentage of long and overlapping branches removed by the algorithm for 10 trees used for validating pruning branch identification algorithm. Trees
T2 T3 T5 T7 T8 T11 T13 T15 T17 T18 Total % Removal
Long branches
Overlapping branches
Total
Removed
Total
Removed
0 3 2 4 2 4 0 2 0 3 20
0 2 2 4 1 4 0 2 0 2 17
2 5 1 4 3 3 3 5 3 3 32
1 2 0 3 2 3 2 4 2 3 22
85.0%
69.0%
Deviation (RMSD) of branch spacing across 10 validation trees was 13%. Some short branches were lost during tree reconstruction, which resulted in a generally higher average branch spacing by the algorithm in a tree compared to that by the workers. However, this difference will have only minimal effect on the pruning outcome because short branches will have a minimal effect on canopy density and light distribution. The algorithm identified most of the branches longer than 28 cm for pruning (Table 4). Cumulative removal percentage for long branches (total number of long branches removed for 10 trees over total number of actual branches) in validation trees was 85%. Reasonably high accuracy in identifying long branches for pruning was achieved because most of the long branches were detected by the branch identification algorithm. The pruning branch identification algorithm removed 69% of overlapping branches in 10 validation trees (Table 4). The overlapping branch identification error was very high in one of the 10 validation trees (T3; only 2 out of 5 overlapping branches were identified), partly due to the false negative branch identification. Percentage removal of overlapping branches increased to 74% when T3 was neglected from the calculation. Improvement in branch identification algorithm will help improve these performance measures of the pruning branch identification algorithm.
4. Conclusion The main objective of this research was to develop a machine vision system to identify branches in tall spindle apple trees that need to be pruned to achieve a desired level of pruning. A timeof-flight-of-light-based 3D camera was used to acquire 3D images and a thinning algorithm was used to compute skeletons of apple
trees. A set of pruning rules was developed based on observation of manual pruning, which was used to identify pruning branches in the skeletonized trees. System parameters were optimized to minimize the difference in Pruning Branch Proportion between the algorithm and average of three workers. It was found that 28 cm of maximum branch length and 20 cm of minimum branch spacing was optimal to match the pruning level of three workers. With these parameters, the algorithm suggested pruning about 20% of branches on average whereas the average Pruning Branch Proportion of the workers was 22%. The algorithm and the human pruning resulted in similar average branch spacing. The algorithm maintained an average of 35.7 cm whereas the average spacing for three workers was 33.7 cm. Root Mean Square Deviation (RMSD) in branch spacing between the algorithm and the workers was found to be 13%. The algorithm was reasonably successful in removing long branches (longer than the given threshold) achieving a removal rate of 85%. The algorithm also removed about 70% of overlapping branches. This research is the first step towards automated pruning of tall spindle apple trees. It was observed that there was some variability between trees and the way workers prune a tree. However, even with such variability, the results from this work showed promise and established a background for further research and development on this critically important technology. In this work, only branch length and branch spacing were used as the pruning criteria. In the future, more criteria could be defined to accurately detect unproductive, deceased or dead branches for pruning. A pan-and-tilt system was used for capturing 3D images in this work. The use of the pan-and-tilt system increased the field of view of the camera but the camera location remained the same. To improve reliability and accuracy of 3D reconstruction, which leads to better performance of pruning branch identification algorithm, multiple perspective views of an object from different camera positions will be useful. Improvement of pruning branch identification algorithm to remove dead branches and branches with large diameter is another area for further research. Future work toward automated pruning will also include localization of pruning points in the pruning branches and developing an end-effector and robotic system for automated pruning.
Acknowledgments This research was partially supported by Washington State University (WSU) Seed Grant, WSU Irrigated Agriculture Research and Extension Center, and WSU Center for Precision and Automated Agricultural Systems and WSU Agricultural Research Center (ARC). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the view of Washington State University. We also would like to acknowledge Mr. Patrick Scharf, Mr. Mark DeKleine, Dr. Ajay Sharda and Dr. Peter Larbi for their support and help during fabrication of experiment platform and field experiments.
References Adhikari, B., 2012. Identification of pruning branches in Tall Spindle apple trees for automated pruning. Master Thesis, Washington State University, Pullman, WA, USA. Adhikari, B., Karkee, M., 2011. 3D Reconstruction of Apple Trees for Mechanical Pruning. ASABE Paper No. 111161. St. Joseph, Mich.: ASABE. Beder, C., Bartczak, B., Koch, R., 2007. A Comparison of PMD-Cameras and StereoVision for the Task of Surface Reconstruction using Patchlets. In: Proc. CVPR. Carbonneau, A., 1979. La taille mechanique de la vigne: des experiences austrialieness. Progress Agricole et Viticole (abstract in English, paper in French). Davies, F.S., Crocker, T.E., 1994. Pruning Blueberries in Florida. Florida Cooperative Extension.
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135 Galinato, S., Gallardo, R.K., 2011. 2010 Estimated Cost of Producing Pears in North Central Washington (FS031E). (accessed 07.01.13). Galinato, S., Gallardo, R.K., Taylor, M. 2010. 2009 Cost Estimates of Establishing and Producing Sweet Cherries in Washington (FS022E). (accessed 07.01.13). Gallardo, R.K., Taylor, M., Hinman, H., 2010. 2009 Cost Estimates of Establishing and Producing Gala Apples in Washington (FS005E). (accessed 07.01.13). Gao, M., Lu, T.F., 2006. Image Processing and Analysis for Autonomous Grapevine Pruning. Mechatronics and Automation. In: Proceedings of the 2006 IEEE International Conference on, pp. 922–927. Gautz, L.D., Bittenbender, H.C., Mauri, S., 2002. Effect of Mechanized Pruning on Coffee Regrowth and Fruit Maturity Timing. An ASAE Meeting Presentation, paper no. 021110. Jensen, F.L., 1980. Effects of mechanical pruning on grapes. J. Calif. Agric. 34 (7), 33– 34. Lee, T.C., Kashyap, R.L., 1994. Building skeleton models via 3-D medial surface/axis thinning algorithms. CVGIP 56 (6), 463–478. Lindner, M., Kilb, A., Hartmann, K., 2007. Data-fusion of pmd-based distanceinformation and high-resolution RGB-images. IEEE Sym. Signals Circuits & Systems (ISSCS), 121–124. McFarlane, N.J.B., Tissryre, B., Sinfort, C., Tillett, R.D., Sevila, F., 1997. Image analysis for pruning of long wood grape vines. J. Agric. Eng. Res. 66 (2), 111–119. Moore, P.W., 1958. Mechanical Pruning for Citrus, California Agriculture. (accessed 02.07.12).
135
Moore-Gough, C., Gough, R.E., 2007. Pruning Fruit Trees in Montana, A self Learning Resource from MSU Extension. (accessed 02.07.12). Nakarmi, A., Tang, L., 2012. Automatic inter-plant spacing sensing at early growth stages using a 3D vision sensor. Comput. Electron. Agric. 82, 23–31. Naugle, J.A., Rehkugler, G.E., Throop, J.A., 1989. Grapevine cordon following using digital image processing. Trans. ASAE I (32), 309–315. Ochs, E.S., Gunkel, W.W., 1993. Robotic grape pruner field performance simulation. Paper no. 933528. St. Joseph, MI: ASAE. Sevilla, F., 1983. Etude de la faisabilte d une mechanization de la vigne en France. Progress Agricole et Viticole (abstract in English, paper in French). Robinson, T.L., 2008. The evolution towards more competitive apple orchard systems in the USA. Acta Hort. 772, 491–500. She, F.H., Chen, R.H., Gao, W.M., Hodgson, P.D., Kong, L.X., Hong, H.Y., 2009. Improved 3D thinning algorithms for skeleton extraction, digital image computing: techniques and applications. IEEE Comput. Soc., 14–18. Tabb, A., Resort, G.S., Casino Reno, N., June 2009. Three-dimensional reconstruction of fruit trees by a shape from silhouette method. ASABE Paper No. 096138, St. Joseph, Mich.: ASABE. Velazquez, M.B., Gonzalez, E.F., 2010. The influence of mechanical pruning in cost reduction, production of fruit, and biomass waste in citrus orchards. Trans ASABE 26 (4), 531–540. Wang, Q., Zhang, Q., 2013. Three-Dimensional Reconstruction of a Dormant Tree Using RGB-D Cameras. ASABE Paper No. 131593521, St. Joseph, Mich.: ASABE.
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag
Identification of pruning branches in tall spindle apple trees for automated pruning Manoj Karkee a,b,⇑, Bikram Adhikari a,b, Suraj Amatya a,b, Qin Zhang b a b
Biological Systems Engineering Department, Washington State University, United States Center for Precision and Automated Agricultural Systems, Washington State University, United States
a r t i c l e
i n f o
Article history: Received 27 September 2012 Received in revised form 22 February 2014 Accepted 26 February 2014
Keywords: Apple tree pruning Selective pruning Pruning algorithm Pruning branch identification Machine vision 3D camera
a b s t r a c t Pruning is a labor intensive operation that constitutes a significant component of total apple production cost. As growers are adapting simpler, narrower, more accessible and productive (SNAP) tree architectures such as the tall spindle fruiting wall system, new opportunities have emerged to reduce pruning cost and labor through automated pruning. This work focused on identification of pruning branches on apple trees in a tall spindle architecture. A time-of-flight-of-light-based three dimensional (ToF 3D) camera was used to construct 3D skeletons of apple trees. Pruning branches were identified in the reconstructed trees using a simplified two-step pruning rule; (i) maintain specified branch spacing and (ii) maintain specified branch length. Performance of the algorithm was optimized using a training sample of 10 trees to achieve human worker’s pruning level. With a selected branch spacing (28 cm) and branch length (20 cm), the algorithm achieved 19.5% branch removal with the training dataset and 19.8% of branch removal with the validation dataset (10 trees) compared to 22% average branch removal by workers. Root Mean Square Deviation (RMSD) between human and algorithm in number of branches identified for pruning was 10% for the training dataset and 13% for the validation dataset. The algorithm and the human pruning resulted in similar average branch spacing. The algorithm maintained an average spacing of 35.7 cm for validation set whereas the average spacing for three workers was 33.7 cm. RMSD in branch spacing between the algorithm and the workers was found to be 13%. The algorithm removed 85% of long branches whereas the overlapping branch removal was only 69%. With some additional work to improve the performance in terms of overlapping branch removal, it is expected that this work will provide a good foundation for automated pruning of tall spindle apple trees in the future. Ó 2014 Elsevier B.V. All rights reserved.
1. Introduction Pruning refers to removing old and/or damaged parts of trees, especially unproductive shoots and branches. It is essential to improve plant health and appearance, to control plant size and to increase fruit quality and yield. Pruning is labor intensive and could cost 20% or more of the annual pre-harvest production cost for crops such as cherries (Galinato et al., 2010), apples (Gallardo et al., 2010) and pears (Galinato and Gallardo, 2011). Automated pruning of fruit trees has a potential to reduce labor demand and cost, which will help to increase the long-term sustainability of fruit production. Researchers have been working on mechanization of fruit tree pruning for several decades. Early research work in pruning ⇑ Corresponding author at: Biological Systems Engineering Department, Center for Precision and Automated Agricultural Systems, IAREC-Washington State University, 24106 N Bunn Rd, Prosser, WA 99350, United States. Tel.: +1 509 786 9208; fax: +1 509 786 9321. E-mail address: [email protected] (M. Karkee). http://dx.doi.org/10.1016/j.compag.2014.02.013 0168-1699/Ó 2014 Elsevier B.V. All rights reserved.
focused on mass pruning in which a cutting tool was run over rows in orchards while maintaining a certain distance from the tree canopy center (Moore, 1958; Davies and Crocker, 1994; Gautz et al., 2002). These mass mechanical pruning systems have limited ability to ensure the quality of pruning and therefore may lead to reduced quality and yield of fruit (Moore-Gough and Gough, 2007; Jensen, 1980; Carbonneau, 1979; Sevilla, 1983; Velazquez and Gonzalez, 2010). Therefore, an automatic machine for selective pruning, which can identify the branches on the tree that need to be pruned, is essential to reduce manual labor while maintaining or improving the quality and productivity of fruit crops. Apple orchards are evolving towards simpler, narrower, more accessible and productive (SNAP) systems that are friendlier to mechanization and automation. These architectures represent various training systems that create very narrow two-dimensional canopies in which the majority of branches and fruit are visible and accessible to machines. These systems could increase profitability by improving yield and/or reducing labor cost (Robinson, 2008). One simplified architecture that lends itself better than
128
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
conventional orchard systems to mechanization is the tall spindle fruiting wall system. In this architecture, trees are planted at smaller intra-plant spacing compared to classical architectures and the branches are not allowed to grow multiple secondary and tertiary branches. This simpler tree architecture opens up an opportunity for automated pruning. A proper sensing technique is essential to identify unwanted branches in fruit trees for automated, selective pruning. Machine vision technology can be used to obtain such information nondestructively. In the past, some researchers have investigated the use of 2D machine vision systems to locate grapevines to guide an automated pruning device. Naugle et al. (1989), Ochs and Gunkel (1993), McFarlane et al. (1997), and Gao and Lu (2006) used a single 2D camera and image processing techniques for identification of pruning points in grapevines. However, extracting three-dimensional (3D) information of objects from a single two-dimensional (2D) image is difficult. It is essential to accurately represent objects and scenes in 3D to support the development of automatic pruning machines. Researchers have been attracted by 3D technologies such as stereo vision and structured light for general use in 3D sensing and object reconstruction. However, only limited studies have been reported in their application to 3D reconstruction and pruning of fruit trees. One such study was conducted in a laboratory setting using multiple images of a dormant apple tree (Tabb et al., 2009). A shape-from-silhouette method was used to reconstruct the shape of the tree from the images captured from multiple orientations. Though the method showed promise for generating a 3D tree structure for pruning, the method is complex and has challenges for practical use in the orchard environment. Recently, a 3D reconstruction method based on a low cost video gaming sensor was explored by Wang and Zhang (2013) for potential application in apple tree pruning. This preliminary study indicated that a high resolution 3D structure of fruit trees can be generated using RGB-D cameras (Wang and Zhang, 2013). However, the technique requires further work on understanding its accuracy and robustness in outdoor orchard environment. In the past several years, time-of-flight-of-light-based cameras (called 3D camera in the following text) have also been used for 3D reconstruction of plant structures. These 3D cameras have shown to be more accurate than stereo vision systems for such reconstruction (Beder et al., 2007). Lindner et al. (2007) also used a 3D camera with high resolution color images to present an efficient way of identifying false mappings due to hidden surfaces in 3D reconstruction. Nakarmi and Tang (2012) used a 3D camera for interplant space sensing in natural illumination conditions. These studies showed that 3D camera-based systems can be used to construct skeletons of apple trees in field environments, which then can be used to identify pruning branches for automated pruning. The main goal of this research was to detect and identify branches that need to be pruned in apple trees using a 3D machine vision system, which is the first step towards automated pruning of apple trees. Specific objectives were to:
Pacific Rose cultivar apple trees in their fourth year. These trees were planted in high density tall spindle fruiting wall architecture (Fig. 1). The intra-plant spacing was 0.46 m and between-row spacing was 2.44 m. Maximum canopy height was 2.74 m. A tree in this architecture consisted of a single central trunk with lateral branching. The images were captured in the late dormant season on April 05, 2012. 2.2. Branch identification A machine vision system was used to develop 3D structure of apple trees and identify branches. Four different steps were involved in processing the images captured by the 3D camera to identify branches; (i) image acquisition, (ii) image pre-processing, (iii) skeletonization, (iv) and skeleton analysis. Descriptions of each of these steps are provided below. Additional details of equipment, methods and algorithms used in these steps can be found in Adhikari and Karkee (2011) and Adhikari (2012). 2.2.1. Image acquisition A 3D camera (CamCube 3.0, PMD Technologies, Siegen, Germany) was used to capture 3D information of the apple trees (Fig. 2). The camera generated 3D coordinates of objects with reference to the imaging plane of the camera. The imaging system was mounted on a pan-and-tilt system (PTU-D48 E, FLIR Motion Control Systems, CA, USA). Using the pan-and-tilt system, a total of 20 different images at different positions were captured for each of the 20 trees randomly selected for this study. The pan-and-tilt system was rotated vertically at the rate of 1.285° per second between [30° and 30°] with reference to horizontal. The camera captured image frames at two-second intervals (i.e. at 2.57° interval). The platform was located approximately 1.27 m away from the center of the row of trees being imaged. 2.2.2. Image preprocessing First, the noise in 3D images was filtered out using two different parameters; (i) the flag and (ii) the amplitude data. The time-offlight (TOF) 3D camera provides a parameter called flag to indicate the quality of distance estimation at each pixel. Any pixel whose
Develop an algorithm to detect and identify pruning branches in apple trees in tall spindle orchard architecture. Optimize input parameters of the pruning branch identification method to achieve desired level of pruning and compare the results with actual human pruning. 2. Materials and methods 2.1. Study site This study was conducted on a test plot in a commercial apple orchard near Prosser, WA. The test plot consisted of four rows of
Fig. 1. Image of the tall spindle fruiting wall apple orchard in Prosser, WA that was used in the study.
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
129
3D Camera
Pan-Tilt System
(a)
(b)
Fig. 2. Imaging platform, (a) user interface computer and a 3D camera mounted on a pan-and-tilt system, and (b) imaging system in a tall spindle apple orchard.
corresponding flag value was not zero was identified as a noisy pixel. Amplitude, on the other hand, signifies the signal strength of the active illumination. This parameter provides an additional quality measure for distances measured by the camera. Very low amplitude value indicated a low accuracy of the distance measurement. In addition, the distance to the objects from the camera was used to filter out trees and other objects located outside the desired range of distances to the objects. The entire preprocessing operation changed pixels to background if, (i) the measurement flag was non-zero, (ii) the signal amplitude was less than 2000 (unitless quantity obtained empirically), or (iii) the object distance from the camera was outside the range of 0.3 m and 2.2 m. The images captured from the camera were radially distorted. This distortion was removed using the following two equations (Nakarmi and Tang, 2012):
xi ¼
f xw p zw þ c
ð1Þ
yi ¼
f yw p zw þ c
ð2Þ
where (xi, yi) are the corrected image coordinates to the captured image coordinates (xw, yw), zw is the depth axis value in the image coordinates, f is the focal length, p is the pixel-pitch of the camera (a parameter that describes the distance between dots on imaging plane), and c is the distance from the front face of the camera to the imaging area. Among these parameters, the value of c was measured to be 10 cm, whereas values of f and p were specified camera specifications. After background removal and radial distortion correction, different point clouds obtained for a single tree were registered together. Since the camera system was mounted on top of a panand-tilt system rotating at a fixed rate, a precise transformation matrix could be obtained between consecutive frames captured by the 3D camera. Each point cloud was captured when the pan-and-tilt system was rotated by an angle of {Hx, Hy, Hz} = {0°, 2.57°, 0°} and translated by {tx, ty, tz} = {0, 0, 0}, where H and t are rotation and translation along each of the three axes. Each point in a point cloud of a given frame, when transformed using a 3D transformation (Eq. (3)), was mapped to the new coordinate system of the next frame. This 3D transformation was performed recursively between consecutive frames to obtain the registered point cloud of the complete tree.
pt 0k ¼ pt k1 transform3dðhx ; hy ; hz ; t x ; ty ; t z Þ
ð3Þ
where, transform3d is a three dimensional transformation function for rigid body transformation (Eq. (4)).
transform3dðhx ;hy ; hz ; t x ;t y ;t z Þ 2 cos hx coshy cos hx sin hy sin hz sin hx coshz coshx sin hy cos hz þ sin hx sin hz 6 sin h cos h cosh sin h sin h þ cosh cosh sin h sin h cosh cos h sin h x y x y z x z x y z x z 6 ¼6 4 sinhy cos hy sin hz cos hy cos hz 0 0 0
tx
3
ty 7 7 7 tz 5 1 ð4Þ
2.2.3. Skeletonization Skeletonization is a way to reorganize 3D points into a set of fewer points that contain the required information about the object. Skeletal representation requires less memory and computation time compared to raw point cloud. A good skeletonization algorithm should hold various properties including centeredness satisfaction, connectivity preservation, and topology preservation (She et al., 2009; Adhikari and Karkee, 2011). In this work, the medial axis thinning algorithm (Lee and Kashyap, 1994) was used to generate 3D skeleton of apple trees from corresponding point clouds. The algorithm performed recursive thinning operations over the surface of the point cloud until the cloud was only one point (voxel) wide. Before applying the skeletonization algorithm, 3D images were dilated to fill unwanted holes and surfaces of the point cloud were smoothed by the 3D Gaussian filter to minimize spurs in the skeletons. 2.2.4. Skeleton analysis On the skeletonized 3D point cloud, all points connected to each other were labeled to identify individual connected objects. This connected object labeling separated the main tree from objects that were not connected to the tree. Then, the shortest connected paths were calculated between all junction points in the main tree. The longest of such paths was identified as the trunk of the tree. Each point on the trunk that had more than two connected neighboring points was defined as a junction/branching point. Junction points were sorted in ascending order based on the height (y-coordinate value). Distance between consecutive junction points represented the branch spacing in the trunk. The length of the branch was obtained by finding the longest path on the skeleton originating from each branching point. This longest path was represented by a set of points corresponding to the branch originating from the branching point on the trunk. The distance between end points of
130
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
branches originating from two consecutive branching points represented the branch spacing at the end of two consecutive branches. In skeletonization process, thinning algorithm extracted only one voxel wide skeletons. Therefore, it was possible to use unit of voxel to measure branch length and spacing. However, the voxel used in this work was 1 mm (deep) 2 mm (wide) 10 mm (tall), which was needed to optimize the data size and computational time while providing results with sufficient resolution. Therefore, a given branch length or branch spacing in voxels would mean different physical length or spacing based on the specific orientation of tree trunk and branches. To simplify the unit conversion, it was assumed that tree trunks were oriented vertically allowing measurement of branch spacing in the vertical direction. It was also assumed that branches were located in the vertical plane with an orientation of 45° to the horizontal plane. Because trees in the tall spindle architecture are grown vertically with branches allowed to grow only in a thin vertical wall, these assumptions were considered acceptable to demonstrate the potential of pruning branch identification algorithm. These assumptions allowed one-to-one conversion from physical units of branch length and spacing to voxels and vice versa.
algorithm over the total number of branches on the tree (Adhikari and Karkee, 2011). This parameter may vary from 0% to 100% indicating what percentage of the actual branches was detected by the algorithm.
BIA ¼
A field experiment was conducted to understand human pruning operation in tall spindle apple trees. Three workers (called worker1, worker2 and worker3 respectively) were selected from the orchard field crew. Twenty trees randomly selected from the test plot for imaging were used for the pruning trials as well. These three workers individually tagged pruning branches on each tree using three different colored markers. To keep tagging independent between workers, each tag was removed from the tree before another worker tagged the pruning points on the same tree. Color images of each tagged tree were captured. Finally, all tags were put together in each test tree to observe the consistency in pruning. Sample images (Fig. 3) show red, blue and green tags tagged by worker1, worker2 and worker3 respectively. Pruning branches identified by workers as well as the total number of branches were counted for each tree. All parameters measured or estimated to evaluate the algorithm are described below. 2.4. Performance evaluation measures 2.4.1. Branch identification accuracy (BIA) A measure called branch identification accuracy (Eq. (5)) was defined as the percentage of actual branches detected by the
End point of a branch
Start point of a branch
Fig. 3. Pruning branches in two sample images/trees tagged by three different workers (worker1 – red; worker2 – blue; worker3 – green). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
ð5Þ
where nb, total number of branches detected by the algorithm. nfp, no. of false positive branches (branches identified by the algorithm but not actual branches). nB, total number of actual branches on the tree. 2.4.2. Pruning Branch Proportion (PBP) Pruning Branch Proportion was defined as the percentage of branches selected for pruning over the total number of branches on the tree. This formulation can be applied to assess outcome of pruning by human as well as the algorithm. When used for calculating PBP for algorithm, pruning branches include false positive branches as well. Total number of branches (denominator of Eq. (6)) is always actual number of branches in the tree.
PBP ¼ 2.3. Study of manual pruning
nb nfp 100% nB
npa 100% nB
ð6Þ
where npa, no. of branches selected for pruning (including false positive branches when used for algorithm). nB, total number of actual branches on the tree. 2.4.3. Net Pruning Branch Proportion (NPBP) Net Pruning Branch Proportion was defined as the percentage of the number of actual branches selected by the algorithm over the total number of branches on the tree (Eq. (7)). Number of actual pruning branches was obtained by excluding all the pruning branches that were selected for pruning but were not actual branches. This measure was applicable to only the algorithm.
NPBP ¼
npa npfp 100% nB
ð7Þ
where npa, no. of pruning branches detected by the algorithm. npfp, false positive branch detected for pruning by the algorithm. nB, total no. of actual branches on the tree. 2.4.4. Other measures Percentage of false negative branches (actual branches in the tree but not detected by the algorithm) and false positive branches were calculated to assess the performance of branch identification algorithm. To assess the performance of pruning branch identification algorithm, number of branches identified for pruning by the algorithm and workers were compared. In addition, average branch spacing achieved by the algorithm was calculated and compared with the average spacing achieved by manual pruning. For each tree, branch spacing was computed as the distance between branches at their start point. To compute branch spacing on pruning branches tagged by human workers, each tagged branch was identified manually in the branch database of the corresponding tree. Finally, percentage removals of long and overlapping branches were estimated to further evaluate the performance of the branch identification algorithm. These two measures provided performance assessment of the algorithm independent of the results of manual pruning. Reference counts for long and overlapping branches were estimated from the color images captured during the data collection. 2.5. Pruning branch identification and pruning level optimization Field observation and analysis of the trees tagged by three workers (Section 2.3) showed that the workers used some level
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
of their subjective knowledge to detect unproductive, old or diseased branches for pruning. However, it was observed that there was a good agreement in the general strategy the workers used to select the pruning branches. First, long branches were selected by all three workers. Closely spaced branches were also selected, even though the three workers did not always choose the same branch for pruning. Similarly, in cases of overlapping branches, two different workers sometimes chose different branches for pruning but leading to the same outcome. These pruning approaches of the workers were in agreement with the opinion of experienced growers and horticulturists working in this area. This observation and evaluation was used to develop a pruning rule for automated identification of pruning branches as follows. Criterion (I): removing branches based on start point and/or end point spacing. Criterion (II): removing branches based on length. This pruning rule ensures that the vegetative structure of the tree is maintained by removing old wood and maintaining a certain spacing between branches. Overlapping between branches will also be avoided by maintaining certain spacing between branch endpoints, which ensures sufficient light interception for fruit to develop better color and texture. The three threshold values for branch length, end-point branch spacing, and start-point branch spacing required by these criteria were the inputs to the system, which would be provided by end users. Branches on the left and right sides of the tree were separated before pruning branches were identified. Then if two branches on the same side of the tree had start-point spacing or end-point spacing less than the threshold value, the branch with lower coordinate value was identified as the pruning branch. For branches that were not removed based on criterion (I), if their length was longer than the threshold length, they were also identified as pruning branches. In this study, different branch lengths and branch spacing were evaluated to achieve a desired pruning level. Branch length was varied from 14 to 56 cm, which was equal to 50 to 200 voxels. Similarly, start-point spacing was varied from 10 to 40 cm whereas end-point spacing was kept constant at 10 cm. The desired pruning level was estimated based on the average of manual pruning by three workers. First, a set of 10 trees were randomly selected as training dataset for the optimization algorithm. Remaining 10 trees were used to validate the performance of the algorithm after optimization. The objective function used was the sum of squares of the deviation between percentage branches removed by the algorithm and the workers (Eq. (8)). With this objective function, goal of the parameter optimization method was to minimize the objective function within the predefined range of parameter values.
f ¼
X
ðPBPWi PBPAi Þ2
ð8Þ
where i varies from 1 to 10 to represent the trees used to optimize the algorithm. PBP stands for Pruning Branch Proportion. Subscript W stands for workers and A stands for the algorithm. Two-step optimization process was used. In the first step, PBP (which includes false positives when calculated for the algorithm) was used in the objective function to narrow down the potential combinations of parameters to achieve desired pruning level. In the next step, NPBP (which does not include false positives and is a better performance indicator for the algorithm) was used to make the final selection of optimal parameters. The performances of the branch identification and pruning branch identification algorithms were evaluated using the performance measures described above. Analysis was performed in individual tree and worker level and Root Mean Square Deviations and/ or averages were calculated when appropriate. Different software
131
tools were used to carry out different tasks of this study. Matlab (version 7.12.0, Mathworks, MA) was used for image acquisition, image preprocessing, and identification of pruning branches. Fiji (Version 1.46j, RSB National Institute of Mental Health, MD) was used for skeletonization and to obtain junction points in the skeleton. 3. Results and discussions 3.1. Branch identification Actual branches in the tree, and total number of branches and actual branches detected by the algorithm were plotted in Fig. 4a. Number of branches detected by the algorithm (including false positives) as compared to the total actual branches on the trees was higher in nine of the trees and was lower in another eleven trees. Number of actual branches detected by the algorithm (true positives) was lower than total branches on tree in all cases. Branch identification accuracy (BIA) was also calculated for each tree as the ratio of actual branches identified by the algorithm to the total number of branches in the tree. On average, the algorithm achieved a branch identification accuracy of 77%. Even though the branch identification accuracy varied substantially (caused by the variability in false positives), the algorithm achieved 80% or higher identification accuracy for more than 60% of the trees evaluated in this study (Fig. 4b). The average branch identification accuracy was particularly skewed by the effect of a few trees (such as T17 and T19) with a dramatic error. On average, false positive and false negative on branch identification was 24% and 23% respectively. In general, the false positive detection was caused by the noise in the 3D cloud data that appeared to be small branches to the algorithm. The most critical reason for the false negatives, on the other hand, was that many short branches on the trees were washed away during image preprocessing and were not identified. In future, 3D reconstruction and branch identification algorithm will be improved to minimize false negative and false positive cases and increase branch identification accuracy. However, short branches, which were the primary reason of false positives and false negatives in this study, are not considered crucial in pruning apple trees because short branches do not cause much canopy overlap. 3.2. Human pruning evaluation In Table 1, number of pruning branches identified by individual workers and corresponding Pruning Branch Proportion (PBP) for all trees are shown. Number of pruning branches and therefore PBP varied substantially between workers. On average, worker1 pruned 28% of the branches with the standard deviation of 8.8% (Table 1). Worker3 was more conservative and consistent than the other two in selecting pruning branches, who selected 16% of the branches on average with a standard deviation of 7.3%. The average PBP of all the workers was 22%, which was consistent with recommendations of experienced horticulturists and apple growers in Pacific Northwest region (unpublished personal communication). Therefore, 22% PBP was set as the benchmark for the pruning branch identification algorithm used in this study. The variability between workers was partly caused by some level of subjective knowledge that workers used in pruning including identification of old, unproductive and diseased branches. It was also observed that there was some level of subjectivity in defining desired branch spacing and branch length even through the strategy of pruning was the same; to maintain certain branch spacing and limit branch length. This variability between individual workers, however, is not desirable and an automated method has the
132
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
35
Number of Branches
30 25 20 15 10 5 0 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15 T16 T17 T18 T19 T20
Tree (Number) Total Branches on Tree
Total Branches Detected
Actual Branches Detected
(a)
(b) Fig. 4. Accuracy of the branch identification algorithm; (a) actual branches on the tree and identified branches; and (b) distribution of branch identification accuracy (BIA). In Fig a, ‘total branches on tree’ represents the number of actual branches in individual trees, ‘Total Branches Detected’ represents the number of branches identified by the algorithm (including false positives) and ‘actual branches detected’ represents the number of actual branches identified by the algorithm.
Table 1 Number of actual branches, number of actual branches identified by the algorithm and number of pruning branches identified by three workers for 20 trees studied in this research. Tree #
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15 T16 T17 T18 T19 T20 Average Std. Dev.
Number of branches on tree
23 20 23 17 14 22 24 19 21 27 26 27 17 25 24 25 23 18 25 19
Number of pruning branches
Pruning Branch Proportion (PBP) (%)
W1
W2
W3
Average of workers
W1
W2
W3
Average of workers
8 9 9 7 7 7 6 3 5 6 7 8 3 6 7 8 5 5 7 5
4 3 5 6 3 5 5 4 1 3 3 4 2 5 2 3 2 1 5 4
7 5 7 8 6 5 3 5 5 6 8 8 3 4 3 3 2 1 4 2
6.3 5.7 7.0 7.0 5.3 5.7 4.7 4.0 3.7 5.0 6.0 6.7 2.7 5.0 4.0 4.7 3.0 2.3 5.3 3.7
34.8 45.0 39.1 41.2 50.0 31.8 25.0 15.8 23.8 22.2 26.9 29.6 17.6 24.0 29.2 32.0 21.7 27.8 28.0 26.3
17.4 15.0 21.7 35.3 21.4 22.7 20.8 21.1 4.8 11.1 11.5 14.8 11.8 20.0 8.3 12.0 8.7 5.6 20.0 21.1
30.4 25.0 30.4 47.1 42.9 22.7 12.5 26.3 23.8 22.2 30.8 29.6 17.6 16.0 12.5 12.0 8.7 5.6 16.0 10.5
27.5 28.3 30.4 41.2 38.1 25.8 19.4 21.1 17.5 18.5 23.1 24.7 15.7 20.0 16.7 18.7 13.0 13.0 21.3 19.3
28.2 8.8
16.0 7.3
22.1 10.8
22.0 7.5
133
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
Fig. 5. Surface plot of the objective function used to optimize pruning parameters (start-point branch spacing and branch length). The surface has a concave structure with an apparent global minimum within the parameter ranges evaluated.
Fig. 6. Net Pruning Branch Proportion (NPBP) for different combinations of branch lengths and start-point branch spacing values.
potential to reduce such variability and improve the consistency of pruning. 3.3. Pruning branch identification and optimization for desired pruning level A training dataset of 10 trees was used first to optimize the algorithm to match average Pruning Branch Proportion (PBP) achieved by three workers. This first step of two step optimization process narrowed down the selection for optimal combinations of pruning parameters. The plot (Fig. 5) of the objective function (Eq. (8)) developed with varying branch length and start-point branch spacing parameters formed a concave surface. PBP for branch length of 14 cm was high for all start-point spacing values as many branches were selected for pruning based on the short branch
length threshold. Therefore, values of the objective function, which represented the sum of squares of the deviation of PBP between human and algorithm, was very high in these cases. Similarly, for start-point spacing of 40 cm, the PBP of the algorithm was high for all branch lengths leading to high values of objective function. Minimum value of the objective function, which is the optimum result, was found with the combination of 28 cm of branch length and 20 cm of start-point branch spacing. However, some other combinations including 42 cm branch length and 20 cm branch spacing, and 56 cm branch length and 20 cm branch spacing also achieved smaller objective function values. Objective function used to develop the parameter space (Fig. 5) was defined based on PBP, which includes false positive branches. On average, about 11.0% of the selected branches were false positive for the selected branch combinations. The second step of the optimization was performed using Net Pruning Branch Proportion (NPBP), which excludes any false positive branches. Six most promising combinations of parameters identified in previous step was evaluated in this step (Fig. 6). The mean NPBP for training dataset with branch length of 28 cm and start-point spacing of 20 cm was 19.5% with a standard deviation of 7.2%. These rates were closet to the average PBP of the three workers, which was 22%. Other selected combinations achieved much lower NPBPs compared to the benchmark of 22%. Based on this result, branch length of 28 cm and branch spacing of 20 cm was considered to be optimal for achieving similar pruning results compared to that of human workers evaluated in this study. The number of actual pruning branches selected by the algorithm with optimal parameter combination was close to human pruning in most of the trees (Table 2). However, there were substantial differences in some other trees. Root Mean Square Deviation (RMSD) of NPBP was found to be 13% for validation dataset. One reason for this deviation was the variability between individual trees. In addition, workers applied some level of subjective judgment that led to pruning differences between workers, and workers and algorithm. Despite larger RMSD, the overall percentage branch removal with the algorithm was close to the desired branch removal percentage. Because overall percentage wood (branch) removal is a critical measure for farmers to ensure the extent of pruning in orchards, this result is encouraging and provides a basis for further research and development in automated apple tree pruning. Resulting branch spacing was also estimated (Table 3) for the optimal parameter combination of 28 cm branch length and 20 cm start-point branch spacing. On average, workers resulted in a branch spacing of 33.7 cm whereas the algorithm achieved the average actual branch spacing of 35.7 cm. Root Mean Square
Table 2 Performance of pruning branch identification algorithm with the optimal parameter combination of 28 cm branch length and 20 cm start-point branch spacing for the validation dataset of 10 trees. Tree #
T2 T3 T5 T7 T8 T11 T13 T15 T17 T18 Average Std. Dev.
No. of Actual Branches in Tree
20 23 14 24 19 26 17 23 23 18
No. of pruning branches Workers (average)
Algorithm
5.7 7.0 5.3 4.7 4.0 6.0 2.7 4.0 3.0 2.3
2 5 2 4 4 7 3 10 2 3
PBP for workers (%)
NPBP for algorithm (%)
Deviation (%)
28.3 30.4 38.1 19.4 21.1 23.1 15.7 17.4 13.0 13.0
10.0 21.7 14.3 16.7 21.1 26.9 17.6 43.5 8.7 16.7
18.3 8.7 23.8 2.8 0.0 3.8 2.0 26.1 4.3 3.7
22.0 7.4
19.7 9.0
RMSD = 13.1
134
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135
Table 3 Average start-point branch spacing after manual and automated pruning for 10 trees used for validating optimal pruning branch identification algorithm. Tree
Start-point branch spacing (cm)
Deviation (%)
Average of workers
Algorithm
T2 T3 T5 T7 T11 T12 T13 T15 T17 T18
53.0 40.2 51.6 22.2 21.8 18.3 31.5 20.1 50.4 27.7
54.3 38.7 55.7 23.1 27.2 18.6 38.1 24.6 47.0 29.7
2.4 3.6 8.1 4.0 25.1 1.8 21.1 22.7 6.7 7.0
Average
33.7
35.7
RMSD = 13.4%
Table 4 Percentage of long and overlapping branches removed by the algorithm for 10 trees used for validating pruning branch identification algorithm. Trees
T2 T3 T5 T7 T8 T11 T13 T15 T17 T18 Total % Removal
Long branches
Overlapping branches
Total
Removed
Total
Removed
0 3 2 4 2 4 0 2 0 3 20
0 2 2 4 1 4 0 2 0 2 17
2 5 1 4 3 3 3 5 3 3 32
1 2 0 3 2 3 2 4 2 3 22
85.0%
69.0%
Deviation (RMSD) of branch spacing across 10 validation trees was 13%. Some short branches were lost during tree reconstruction, which resulted in a generally higher average branch spacing by the algorithm in a tree compared to that by the workers. However, this difference will have only minimal effect on the pruning outcome because short branches will have a minimal effect on canopy density and light distribution. The algorithm identified most of the branches longer than 28 cm for pruning (Table 4). Cumulative removal percentage for long branches (total number of long branches removed for 10 trees over total number of actual branches) in validation trees was 85%. Reasonably high accuracy in identifying long branches for pruning was achieved because most of the long branches were detected by the branch identification algorithm. The pruning branch identification algorithm removed 69% of overlapping branches in 10 validation trees (Table 4). The overlapping branch identification error was very high in one of the 10 validation trees (T3; only 2 out of 5 overlapping branches were identified), partly due to the false negative branch identification. Percentage removal of overlapping branches increased to 74% when T3 was neglected from the calculation. Improvement in branch identification algorithm will help improve these performance measures of the pruning branch identification algorithm.
4. Conclusion The main objective of this research was to develop a machine vision system to identify branches in tall spindle apple trees that need to be pruned to achieve a desired level of pruning. A timeof-flight-of-light-based 3D camera was used to acquire 3D images and a thinning algorithm was used to compute skeletons of apple
trees. A set of pruning rules was developed based on observation of manual pruning, which was used to identify pruning branches in the skeletonized trees. System parameters were optimized to minimize the difference in Pruning Branch Proportion between the algorithm and average of three workers. It was found that 28 cm of maximum branch length and 20 cm of minimum branch spacing was optimal to match the pruning level of three workers. With these parameters, the algorithm suggested pruning about 20% of branches on average whereas the average Pruning Branch Proportion of the workers was 22%. The algorithm and the human pruning resulted in similar average branch spacing. The algorithm maintained an average of 35.7 cm whereas the average spacing for three workers was 33.7 cm. Root Mean Square Deviation (RMSD) in branch spacing between the algorithm and the workers was found to be 13%. The algorithm was reasonably successful in removing long branches (longer than the given threshold) achieving a removal rate of 85%. The algorithm also removed about 70% of overlapping branches. This research is the first step towards automated pruning of tall spindle apple trees. It was observed that there was some variability between trees and the way workers prune a tree. However, even with such variability, the results from this work showed promise and established a background for further research and development on this critically important technology. In this work, only branch length and branch spacing were used as the pruning criteria. In the future, more criteria could be defined to accurately detect unproductive, deceased or dead branches for pruning. A pan-and-tilt system was used for capturing 3D images in this work. The use of the pan-and-tilt system increased the field of view of the camera but the camera location remained the same. To improve reliability and accuracy of 3D reconstruction, which leads to better performance of pruning branch identification algorithm, multiple perspective views of an object from different camera positions will be useful. Improvement of pruning branch identification algorithm to remove dead branches and branches with large diameter is another area for further research. Future work toward automated pruning will also include localization of pruning points in the pruning branches and developing an end-effector and robotic system for automated pruning.
Acknowledgments This research was partially supported by Washington State University (WSU) Seed Grant, WSU Irrigated Agriculture Research and Extension Center, and WSU Center for Precision and Automated Agricultural Systems and WSU Agricultural Research Center (ARC). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the view of Washington State University. We also would like to acknowledge Mr. Patrick Scharf, Mr. Mark DeKleine, Dr. Ajay Sharda and Dr. Peter Larbi for their support and help during fabrication of experiment platform and field experiments.
References Adhikari, B., 2012. Identification of pruning branches in Tall Spindle apple trees for automated pruning. Master Thesis, Washington State University, Pullman, WA, USA. Adhikari, B., Karkee, M., 2011. 3D Reconstruction of Apple Trees for Mechanical Pruning. ASABE Paper No. 111161. St. Joseph, Mich.: ASABE. Beder, C., Bartczak, B., Koch, R., 2007. A Comparison of PMD-Cameras and StereoVision for the Task of Surface Reconstruction using Patchlets. In: Proc. CVPR. Carbonneau, A., 1979. La taille mechanique de la vigne: des experiences austrialieness. Progress Agricole et Viticole (abstract in English, paper in French). Davies, F.S., Crocker, T.E., 1994. Pruning Blueberries in Florida. Florida Cooperative Extension.
M. Karkee et al. / Computers and Electronics in Agriculture 103 (2014) 127–135 Galinato, S., Gallardo, R.K., 2011. 2010 Estimated Cost of Producing Pears in North Central Washington (FS031E).
135
Moore-Gough, C., Gough, R.E., 2007. Pruning Fruit Trees in Montana, A self Learning Resource from MSU Extension.