- Email: [email protected]
Computer Vision for Identifying Weeds in Crops
Copyright ® IF AC Bio-Robotics, Information Technology and Intelligent Control for Bio-Production Systems, Sakai, Osaka, Japan, 2000
COMPUTER VISION FOR IDENTIFYING WEEDS IN CROPS
Thomas Rath 1 and Jochen Hemmini
Ilnst. for Horticultural and Agricultural Eng., University of Hannover, Germany 21MAG, Inst. of Agricultural and Environmental Eng., Wagertingen, The Netherlands
Abstract: The methods of digital image analysis were used to develop an identification system for weeds in crops (cabbage and carrots) . Colour CCD-cameras mounted on a tractor rack were used to record images under practically orientated conditions. Depending on the species, the growth stage and the method of calculation, between 51 and 95 % of the crop plants were classified correctly. Problems still exist in separating and locating single plants in scenes where many plants have grown together. Copyright @20001FAC Keywords: computer vision, image analysis, agriCUlture
a maize crop from multispectral images. Tian,
1. INTRODUCTION
el
aI., (1999) presented the development of a precision sprayer for site-specific weed management.
Modern production methods in horticulture are characterised by an increasing amount of automation combined with the use of horticulturespecific information . A fundamental requirement is the use of computer based "Visual analysis. Onc goal in this area is to identify and count weeds for plarming herbicide applications or physical weed control (see also Woebbecke, et aI. , 1995) . Numerous research activities have been registered in the last few years in the field of automated weed identification. For example Sokefeld, et al., (1996) showed a system for identifying 22 different weed species. However, the images were taken under laboratory conditions. Lee, et aI. , (1997) developed a real-time robotic weed control system for selective spraying in-row weeds using a machine vision system and a chemical precision application system. Dzinaj. et aI. , (1998) presented a multiple sensor system with photodiodes, CCD line sensors and other sensor systems for distinguishing cropplants (maize) from weeds . Meyer, et aI., (1998) investigated the discrimination of two species of grasses and broad leaves. Chapron, et al., (1999) showed a method for identifying vegetal species in
The following article shows the potential of using image processing techniques as a plant identification system not only under labomtory conditions but also in natural scenes.
2. MATERIAL AND METHODS
2.1 Crops and Field Set-up
The metllOds of digital image analysis were used to develop an identification system for weeds in crops in open field situations. Two crops with contrasting leaf shapes were used to develop and test the algoritluns: cabbage (Brassica oleracea L. conv. capitata var. capitata) with simple round to ellipse shaped leaves and carrots (Daucus carota L.) witl1 feathery leaves. The carrots were sown directly on the open field, whereas the cabbage was planted (both in rows, but two different investigations). The field area for every species was about 100 1112
187
l .3 identifi cation I'rocess
Neither mechanical nor chemica l treatment was applied to the crops The main weed species which emerge d were LmlliulII alllple:'(Jcaule. Galison ga ciltala . .-lgropvroJ1 repens and ClrsiulII an·ense .
The following image processing and statistical methods were used for the identification process (for detailed information about the methods see Hemmi ng, 2000 and Rath, 1997)
l .l Hardwa re and Eqlllpll lent
Image Foregro und-bac kground separation and space -colour HSI the to conversion lding. thresho annelmultich 2 Segmentation (leaves) and object separation Approach based on mathematICal Illorpholog: (erosion, dilation , opening , closing). 3. Feature extraction: For each separate regIOn 8 different shape features and 3 colour features are calculated (area, area/contour length, length/w idth. Circularity, convexity, max. diameter. roundness. spikes, hue (colour), saturation (colour), intensity (colour ». 4. Feature selectio n and classific ation Objects are classified with statistical models using standardised Gaussian distributions and the Student's distribution for feature weighting. 5. Total plant classification: In order to recognise not only single leaves but total plants a nonhierarchical cluster analysis is used . 6 Crop row identification Infonna tion about the crop rows are extracted by using the Houghtransformation. I.
A tractor add-on Wlit was constructed with the following compon ents (sce figure I) : I Three CCD-co lour-ca meras were mounted on a steel-tube-rack at the rear of a tractor. 2. To avoid natural lighting the Wlit was cO\'ered with lightpro of PE-film and equippe d with three 400 W metal halogen lamps, each switched to a different phase of the current. to balance the light variatio n 3. The camera s were mounte d in such a way that there was partial overlap in adjacent images (see figure 2) The overlap ping in the driving direction was achie\'e d by a preCise triggering and multipi exing of the shots using tractor speed measur ing sensors. 4. To store and analyse the pictures conventional personal comput ers were used. The image process ing algorith ms were developed by using the HALCO N image process ing software system (Mvtec, 1998).
A softwar e-system was built including the above mentioned methods and tasks. The aim of this system was to analyse the images automatically and to classity differen t species. The analysis program had a special mode that pemlits to select objects manually identified with a mousep ointer on the screen Tllis objects are used to train the analysis program for the automa ted classification of I1C\\ unknown objects .
l.4 Experimental Set-up The illyestlgatlons were carned out by recorJlI1g the whole field with cabbage crops 5 times (carrots 3 times) at different growing stages by passing the field with the rack. The analysis of the images \\as carried out off-line in a second step in the office. Different analysis models have been imestig ated (for detailed results see Hemmi ng, 2000)
Fig. 1. TIle mounte d camera system with lighting
POS'IlJOr. ~-arlle,iI
1
overlappi ng caused by camera • pos i ti on on the rack
The first pictures of the field were used to extract a training data set with the special manually mode. described above. In the in\'estigations presented here the selected objects are split into four different training classes (one class is the crop and three classification classes for the weeds). The number of training objects per class differed from 5 and 25 . \Vith these data the discrimination algorill uns were trained and applied on the rest of llle images by simulating a field passage willl llle tractor and the
,'. overlappi ng by tnggenng and multiplex ing picture .3.3 _ _ _ _ _ _ _ ;X"SI[ 10 Tl
(amera )
Fig. 2. Combin ation of overlap ping images
rack.
188
2 Binaris3tion Separation
~ ;d ' ; , , ; : -
J"""""J"~
I Recording
5 Ro" detection
\
____
Class ification Implo\,ement
~
3
\
Table I summarises tlle classification errors for cabbage and carrot identification for different growing stages. The minimum and ma:ximum \'alues result from different model and algorithm parameters. Depending on growth stage and method of calculation between 59 % and 95 % of the cabbage plants were classified correctly (an 3\'Crage of 87 %) Problems still exist in separating ,md locating single plants in scenes where many plants h,l\c grown together. This has Cl ncgati\c effect especially in crops \\ith finc-stmctured Ica\es like carrots. Between 51 % and 90 % of the carrot plants were classified correctly (an average of 69 %) Future work has to consider this problem and ma~' be new methods are necessary.
... .;.
Featllree~1ractlon
I e.. •
',~
~ . CLtssll;CilIIUIl
,': _
CIlIstamg
Fig. 3. Schematic yie\\ of the identification process First tlle green plants are selected, then different leaves are separated and classified into the given classes, tllen the leaves are combined into plants, tlle crop rows are identified step by step and - to improve the classification results - if plants outside the rows were classifIed as crops, they were new classified as weeds.
The usage of the row information for tlle classification process could improve the type 2 error (weed classified as crop) up to 2 % , but it was paid by an increase of the type 1 error (crop not identified) depending on model parameters between I and 9 %. The metllOd was very robust in tlle case tllat only a few plants were missing (because of diseases, pests or other failures) or misclassified. When too many plants were lost or misclassified the row detection was incomplete and useless for the identification process.
3. RESULTS AND DISCUSSION Figure 4 shows a typical scene recorded with a camera and the analysed and interpreted image . Witll constant light conditions tlle separation of plants and backgroWld worked very well with the used binarisation procedure. Witll the help of morphological image processing operators it was possible - limited to certain degree - to segment tlle single components. Even plants barely visible with the eyes could be separated. It has to be taken into consideration tllat plants or part of plants are not always green. Especially the stalks of the cabbage leaves sometimes appeared whitish and for this reason were not separated. The used clustering algorithms solve this problem (see figure 4) by grabbing the leaves together and rebuilding the stems. But if too few leaves of one plant are detected the results are not very well
Compared to other studies tlle plant identification system presented here can be understood as an improvement, especially considering tllat tlle experiments were carried out under practically orientated conditions. TIle methods could be used also for identifying special weeds. Presently the methods are transferred to practical applications in sugar beet. Table 1 Classification results for different crops and growing stages crop
age (days) 21 24
cabbage
error type * 1 2 I 2
27 32 34
2 1 2 I 2
25 2 carrots
36 58
wrong identified leaves
Fig 4. Image (original and after identification)
2 1 2
muumWll error (%) 13 .28 5.40 6.26 5.74 18.30 4.43 10.07 6.37 5.16 1365 26 .35 15.96 23.30 902 18.25 44.14
• error type I : not identified crop error Iype 2 : weed classified as crop
189
m a :xi mlml error (%) 34.7~
7.62 1484 7.95 23.79 lU8 19 .7 1 1072 15 .72 20 .16 40.86 37.46 4854 22 .38 3705 76.77
With the computerised individual plant identification and localisation, field machinery can be developed to perform automatic treatments such as weeding, thinning and applying pesticides or other substances quantitatively and precisely. This could lead to the reduction of chemical waste, crop damage and envirorunental pollution.
Acknowledgement Financial support through the Deutsche Forschungsgemeinschaft is gratefully acknowledged.
4. REFERENCES Chapron, M., M. Requena-Esteso, P . Boisssard and L Assemat (1999). A method for recognizing vegetal species from multi spectral images. In : Precision Agriculture '99. rd European Conference on Precision Agriculture, Odense, Denmark, 11.-15 , July 1999, Pan 1, 239-247 , Sheffield Academic Press . Dzinaj , T. , S. KJeine-H6rstkamp, A.Linz, A. Ruckelshausen, O. B6ttger, M . Kemper, 1.Marquering, J. Naescher, D. Trautz and E. Wisserodt (1998) . Multi-Sensor-System zur Unterscheidung von Nutzpflanzen und Beikrautern. z. far. Pjl.krankheiten und Pjl.schutz, SonderheJt, 233-242 . Hemming, 1. (2000). Computer vision for identifying weeds in crops. Gartenbautechnische lnformationen , Heft 50, Institute for Horticultural Engineering, University of Hannover, Hannover. Lee, W.S. , D . C. Slaughter and D. K. Giles (1997). Robotic weed control system for tomatoes using machine vision system and precision chemical application. ASAE-Paper, 97-3093 , 1-15. Meyer, G. E. , T. Mehta, M. F. Kocher, D. A. Mortensen and A. Sanlal (1998). Textural imaging and discriminant analysis for distinguishing weeds for spot spraying. Transactions of the ASAE, 41 (4), 1189-1197. Mvtec (1998) . HALCON C++ Version 5.01. MVTec Software GmbH, Munich. Rath, T. (1997). Methoden zur computerbildana1yIischen Pflanzenidentifikation am Beispiel dendrologischer Bestirrunungen. Gartenbautechnische lnformationen , Heft 42, Institute for Horticultural Engineering, University of Hannover, Hannover. S6kefeld, M. , R. Gerhards and W . Kiihbauch (1996) . Automatische Erkennung von Unkrautem im Keimblattstadiwn mit digitaler Bildverarbeitung. KTBL-Arbeirspapier. Kuratoriwn fur Teclmik und Bauwesen in der Landwirtschaft e.v.. Dannstadt. Tian, L.. 1. F. Reid and 1. W. HunU11el (1999). Development of a precision sprayer for sitespecific weed management. Transactions of the AS.4E, 42 (4), 893-900. Woebbecke, D . M ., G.E. Meyer, D.A. Mortensen and K. von Bargen (1995) . Shape features for identifying young weeds using image analysis. Transactions of the ASAE, 38 (1), 271-281.
190
COMPUTER VISION FOR IDENTIFYING WEEDS IN CROPS
Thomas Rath 1 and Jochen Hemmini
Ilnst. for Horticultural and Agricultural Eng., University of Hannover, Germany 21MAG, Inst. of Agricultural and Environmental Eng., Wagertingen, The Netherlands
Abstract: The methods of digital image analysis were used to develop an identification system for weeds in crops (cabbage and carrots) . Colour CCD-cameras mounted on a tractor rack were used to record images under practically orientated conditions. Depending on the species, the growth stage and the method of calculation, between 51 and 95 % of the crop plants were classified correctly. Problems still exist in separating and locating single plants in scenes where many plants have grown together. Copyright @20001FAC Keywords: computer vision, image analysis, agriCUlture
a maize crop from multispectral images. Tian,
1. INTRODUCTION
el
aI., (1999) presented the development of a precision sprayer for site-specific weed management.
Modern production methods in horticulture are characterised by an increasing amount of automation combined with the use of horticulturespecific information . A fundamental requirement is the use of computer based "Visual analysis. Onc goal in this area is to identify and count weeds for plarming herbicide applications or physical weed control (see also Woebbecke, et aI. , 1995) . Numerous research activities have been registered in the last few years in the field of automated weed identification. For example Sokefeld, et al., (1996) showed a system for identifying 22 different weed species. However, the images were taken under laboratory conditions. Lee, et aI. , (1997) developed a real-time robotic weed control system for selective spraying in-row weeds using a machine vision system and a chemical precision application system. Dzinaj. et aI. , (1998) presented a multiple sensor system with photodiodes, CCD line sensors and other sensor systems for distinguishing cropplants (maize) from weeds . Meyer, et aI., (1998) investigated the discrimination of two species of grasses and broad leaves. Chapron, et al., (1999) showed a method for identifying vegetal species in
The following article shows the potential of using image processing techniques as a plant identification system not only under labomtory conditions but also in natural scenes.
2. MATERIAL AND METHODS
2.1 Crops and Field Set-up
The metllOds of digital image analysis were used to develop an identification system for weeds in crops in open field situations. Two crops with contrasting leaf shapes were used to develop and test the algoritluns: cabbage (Brassica oleracea L. conv. capitata var. capitata) with simple round to ellipse shaped leaves and carrots (Daucus carota L.) witl1 feathery leaves. The carrots were sown directly on the open field, whereas the cabbage was planted (both in rows, but two different investigations). The field area for every species was about 100 1112
187
l .3 identifi cation I'rocess
Neither mechanical nor chemica l treatment was applied to the crops The main weed species which emerge d were LmlliulII alllple:'(Jcaule. Galison ga ciltala . .-lgropvroJ1 repens and ClrsiulII an·ense .
The following image processing and statistical methods were used for the identification process (for detailed information about the methods see Hemmi ng, 2000 and Rath, 1997)
l .l Hardwa re and Eqlllpll lent
Image Foregro und-bac kground separation and space -colour HSI the to conversion lding. thresho annelmultich 2 Segmentation (leaves) and object separation Approach based on mathematICal Illorpholog: (erosion, dilation , opening , closing). 3. Feature extraction: For each separate regIOn 8 different shape features and 3 colour features are calculated (area, area/contour length, length/w idth. Circularity, convexity, max. diameter. roundness. spikes, hue (colour), saturation (colour), intensity (colour ». 4. Feature selectio n and classific ation Objects are classified with statistical models using standardised Gaussian distributions and the Student's distribution for feature weighting. 5. Total plant classification: In order to recognise not only single leaves but total plants a nonhierarchical cluster analysis is used . 6 Crop row identification Infonna tion about the crop rows are extracted by using the Houghtransformation. I.
A tractor add-on Wlit was constructed with the following compon ents (sce figure I) : I Three CCD-co lour-ca meras were mounted on a steel-tube-rack at the rear of a tractor. 2. To avoid natural lighting the Wlit was cO\'ered with lightpro of PE-film and equippe d with three 400 W metal halogen lamps, each switched to a different phase of the current. to balance the light variatio n 3. The camera s were mounte d in such a way that there was partial overlap in adjacent images (see figure 2) The overlap ping in the driving direction was achie\'e d by a preCise triggering and multipi exing of the shots using tractor speed measur ing sensors. 4. To store and analyse the pictures conventional personal comput ers were used. The image process ing algorith ms were developed by using the HALCO N image process ing software system (Mvtec, 1998).
A softwar e-system was built including the above mentioned methods and tasks. The aim of this system was to analyse the images automatically and to classity differen t species. The analysis program had a special mode that pemlits to select objects manually identified with a mousep ointer on the screen Tllis objects are used to train the analysis program for the automa ted classification of I1C\\ unknown objects .
l.4 Experimental Set-up The illyestlgatlons were carned out by recorJlI1g the whole field with cabbage crops 5 times (carrots 3 times) at different growing stages by passing the field with the rack. The analysis of the images \\as carried out off-line in a second step in the office. Different analysis models have been imestig ated (for detailed results see Hemmi ng, 2000)
Fig. 1. TIle mounte d camera system with lighting
POS'IlJOr. ~-arlle,iI
1
overlappi ng caused by camera • pos i ti on on the rack
The first pictures of the field were used to extract a training data set with the special manually mode. described above. In the in\'estigations presented here the selected objects are split into four different training classes (one class is the crop and three classification classes for the weeds). The number of training objects per class differed from 5 and 25 . \Vith these data the discrimination algorill uns were trained and applied on the rest of llle images by simulating a field passage willl llle tractor and the
,'. overlappi ng by tnggenng and multiplex ing picture .3.3 _ _ _ _ _ _ _ ;X"SI[ 10 Tl
(amera )
Fig. 2. Combin ation of overlap ping images
rack.
188
2 Binaris3tion Separation
~ ;d ' ; , , ; : -
J"""""J"~
I Recording
5 Ro" detection
\
____
Class ification Implo\,ement
~
3
\
Table I summarises tlle classification errors for cabbage and carrot identification for different growing stages. The minimum and ma:ximum \'alues result from different model and algorithm parameters. Depending on growth stage and method of calculation between 59 % and 95 % of the cabbage plants were classified correctly (an 3\'Crage of 87 %) Problems still exist in separating ,md locating single plants in scenes where many plants h,l\c grown together. This has Cl ncgati\c effect especially in crops \\ith finc-stmctured Ica\es like carrots. Between 51 % and 90 % of the carrot plants were classified correctly (an average of 69 %) Future work has to consider this problem and ma~' be new methods are necessary.
... .;.
Featllree~1ractlon
I e.. •
',~
~ . CLtssll;CilIIUIl
,': _
CIlIstamg
Fig. 3. Schematic yie\\ of the identification process First tlle green plants are selected, then different leaves are separated and classified into the given classes, tllen the leaves are combined into plants, tlle crop rows are identified step by step and - to improve the classification results - if plants outside the rows were classifIed as crops, they were new classified as weeds.
The usage of the row information for tlle classification process could improve the type 2 error (weed classified as crop) up to 2 % , but it was paid by an increase of the type 1 error (crop not identified) depending on model parameters between I and 9 %. The metllOd was very robust in tlle case tllat only a few plants were missing (because of diseases, pests or other failures) or misclassified. When too many plants were lost or misclassified the row detection was incomplete and useless for the identification process.
3. RESULTS AND DISCUSSION Figure 4 shows a typical scene recorded with a camera and the analysed and interpreted image . Witll constant light conditions tlle separation of plants and backgroWld worked very well with the used binarisation procedure. Witll the help of morphological image processing operators it was possible - limited to certain degree - to segment tlle single components. Even plants barely visible with the eyes could be separated. It has to be taken into consideration tllat plants or part of plants are not always green. Especially the stalks of the cabbage leaves sometimes appeared whitish and for this reason were not separated. The used clustering algorithms solve this problem (see figure 4) by grabbing the leaves together and rebuilding the stems. But if too few leaves of one plant are detected the results are not very well
Compared to other studies tlle plant identification system presented here can be understood as an improvement, especially considering tllat tlle experiments were carried out under practically orientated conditions. TIle methods could be used also for identifying special weeds. Presently the methods are transferred to practical applications in sugar beet. Table 1 Classification results for different crops and growing stages crop
age (days) 21 24
cabbage
error type * 1 2 I 2
27 32 34
2 1 2 I 2
25 2 carrots
36 58
wrong identified leaves
Fig 4. Image (original and after identification)
2 1 2
muumWll error (%) 13 .28 5.40 6.26 5.74 18.30 4.43 10.07 6.37 5.16 1365 26 .35 15.96 23.30 902 18.25 44.14
• error type I : not identified crop error Iype 2 : weed classified as crop
189
m a :xi mlml error (%) 34.7~
7.62 1484 7.95 23.79 lU8 19 .7 1 1072 15 .72 20 .16 40.86 37.46 4854 22 .38 3705 76.77
With the computerised individual plant identification and localisation, field machinery can be developed to perform automatic treatments such as weeding, thinning and applying pesticides or other substances quantitatively and precisely. This could lead to the reduction of chemical waste, crop damage and envirorunental pollution.
Acknowledgement Financial support through the Deutsche Forschungsgemeinschaft is gratefully acknowledged.
4. REFERENCES Chapron, M., M. Requena-Esteso, P . Boisssard and L Assemat (1999). A method for recognizing vegetal species from multi spectral images. In : Precision Agriculture '99. rd European Conference on Precision Agriculture, Odense, Denmark, 11.-15 , July 1999, Pan 1, 239-247 , Sheffield Academic Press . Dzinaj , T. , S. KJeine-H6rstkamp, A.Linz, A. Ruckelshausen, O. B6ttger, M . Kemper, 1.Marquering, J. Naescher, D. Trautz and E. Wisserodt (1998) . Multi-Sensor-System zur Unterscheidung von Nutzpflanzen und Beikrautern. z. far. Pjl.krankheiten und Pjl.schutz, SonderheJt, 233-242 . Hemming, 1. (2000). Computer vision for identifying weeds in crops. Gartenbautechnische lnformationen , Heft 50, Institute for Horticultural Engineering, University of Hannover, Hannover. Lee, W.S. , D . C. Slaughter and D. K. Giles (1997). Robotic weed control system for tomatoes using machine vision system and precision chemical application. ASAE-Paper, 97-3093 , 1-15. Meyer, G. E. , T. Mehta, M. F. Kocher, D. A. Mortensen and A. Sanlal (1998). Textural imaging and discriminant analysis for distinguishing weeds for spot spraying. Transactions of the ASAE, 41 (4), 1189-1197. Mvtec (1998) . HALCON C++ Version 5.01. MVTec Software GmbH, Munich. Rath, T. (1997). Methoden zur computerbildana1yIischen Pflanzenidentifikation am Beispiel dendrologischer Bestirrunungen. Gartenbautechnische lnformationen , Heft 42, Institute for Horticultural Engineering, University of Hannover, Hannover. S6kefeld, M. , R. Gerhards and W . Kiihbauch (1996) . Automatische Erkennung von Unkrautem im Keimblattstadiwn mit digitaler Bildverarbeitung. KTBL-Arbeirspapier. Kuratoriwn fur Teclmik und Bauwesen in der Landwirtschaft e.v.. Dannstadt. Tian, L.. 1. F. Reid and 1. W. HunU11el (1999). Development of a precision sprayer for sitespecific weed management. Transactions of the AS.4E, 42 (4), 893-900. Woebbecke, D . M ., G.E. Meyer, D.A. Mortensen and K. von Bargen (1995) . Shape features for identifying young weeds using image analysis. Transactions of the ASAE, 38 (1), 271-281.
190