Automated weed control in organic row crops using hyperspectral species identification and thermal micro-dosing

Crop Protection 41 (2012) 96e105

Contents lists available at SciVerse ScienceDirect

Crop Protection journal homepage: www.elsevier.com/locate/cropro

Automated weed control in organic row crops using hyperspectral species identification and thermal micro-dosing Yun Zhang*, Erik S. Staab, David C. Slaughter, D. Ken Giles, Daniel Downey Department of Biological and Agricultural Engineering, University of California, One Shields Avenue, Davis, CA 95616, United States

a r t i c l e i n f o

a b s t r a c t

Article history: Received 25 June 2011 Received in revised form 3 May 2012 Accepted 8 May 2012

A hyperspectral imaging system was coupled to a precision, pulsed-jet, micro-dosing system to selectively deliver high-temperature, organic, food-grade oil for intra-row weed control in early growth tomatoes. The imaging system, based upon a multispectral Bayesian classifier, successfully discriminated the species of 95.9%, on average, of plant canopy for tomato, Solanum nigrum L. and Amaranthus retroflexus L. using canopy reflectance in the 384e810 nm range. Food-grade oil, heated to approximately 160
C, was applied to weeds using a pressurized micro-dosing pulsed-jet. The target application rate was approximately 0.85 mg/cm2 in 10-ms-pulsed doses while traveling at a ground speed of 0.04 m/s. In an outdoor test, approximately 95.8% of S. nigrum and 93.8% of A. retroflexus were controlled 15 days post thermal treatment, while only 2.4% of the tomato plants received significant damage. Application coverage assessments of leaf surfaces immediately after the heated oil application found that tomato viability was retained if 50% or less of the leaf surface was inadvertently dosed, while 100% mortality was achieved for S. nigrum and A. retroflexus if more than 90% of their respective leaf surfaces were covered with heated oil. Ó 2012 Elsevier Ltd. All rights reserved.

Keywords: Micro-spray Machine vision Plant discrimination Heated oil Precision agriculture Tomatoes

1. Introduction Weed infestation has been estimated to cause a reduction of 12% in potential agricultural yields in the U.S., representing a $33 billion loss in annual crop production (Pimentel et al., 2001). Weeds emerging within the seedline during the early growth stage (before or during crop establishment) are more harmful due to competition for nutrients, water and sunlight and more critical to remove. Currently, herbicide application (preemergence, postemergence or a combination), mechanical cultivation and hand hoeing have been the common methods used for weed control in row crops (Slaughter et al., 2008a). While selective and non-selective herbicides are still widely used in agricultural production, weeds remain a perennial problem and public concern about potential environmental costs and risks to food safety regarding the use of herbicides have driven a growing interest in organic production. Mechanical cultivation systems are generally effective in removing weeds growing between crop rows but not within the seed-line. As a result, the need for hand labor continues for weed removal within close proximity of crop plants and within the crop row. However, manual hand hoeing can be over five times the cost of conventional

* Corresponding author. Tel.: þ1 530 754 9776; fax: þ1 530 752 2640. E-mail address: [email protected] (Y. Zhang). 0261-2194/$ e see front matter Ó 2012 Elsevier Ltd. All rights reserved. doi:10.1016/j.cropro.2012.05.007

mechanical cultivation (Chandler and Cooke, 1992) and is not completely effective. Heterogeneous distribution patterns of weed populations in agricultural fields have inspired the concept of site-specific weed management, where herbicide application is spatially varied to suit the local weed problem (Christensen et al., 2009). Over the last two decades, rapid development and implementation of new technologies (such as RTK GPS, optical imaging and spectroscopy) for precision agriculture have encouraged extensive studies on automated weed control in a row crop environment. As a result, researchers have investigated several concepts for automated weed control systems for a variety of crops using machine vision for plant species recognition and robotic actuation mechanisms for in-row weed removal. For example, Åstrand and Baerveldt (2002, 2004 and 2005) developed and tested a mobile platform for field scale autonomous weed control in sugar beets (Beta vulgaris L.). The system incorporated two vision systems: one (gray-level) for guidance along the row and the second (color-based) for identifying crop among weeds plants. They integrated spatial context information with plant shape and color features to improve species identification accuracy for fields with low to moderate weed density (no or minor leaf occlusion). Based on correct recognition of the weeds within the seedline, the vision system actuated a mechanical wheel that rotated perpendicular to the row to control the weeds. The

Y. Zhang et al. / Crop Protection 41 (2012) 96e105

results of a field test demonstrated that the system successfully removed 41e53% of the weeds and 99% of the sugar beets were not removed. Lamm et al. (2002) developed a real-time robotic weed control system that was capable of distinguishing weeds from cotton (Gossypium hirsutum L.) plants and then applying chemical spray to only the targeted weeds. A unique shape-based machine vision algorithm was developed using minimum distance from pixels within an object to the edge of that object for discriminating between broad cotton leaves and grass-liked weeds. A mixture of water with blue dye was used as the spray liquid to evaluate the system performance by visual inspection. In 14 commercial field tests, the system traveled at a constant speed of 0.45 m/s and correctly identified and did not spray 78.7% of cotton plants while 88.8% of the weeds were automatically treated. Tillett et al. (2007) developed a mechanical weed control system for transplanted cabbages (Brassica oleracea L.) using machine vision for system guidance and plant localization in the crop row. The machine was based on a steerage hoe fitted with two novel shallow cultivation modules acting within the seed-line of early growth row crops. Each module featured a hydraulically driven disc with an interior section cut away to allow crop plants to pass undamaged. Field trials found that under normal growing conditions weed control was estimated between 62% and 87% within a 0.25 m radius around crop plants. They reported an average ground speed of the system at 0.5 m/s, which was substantially faster that achieved by manual hand hoeing. While these automated weed control systems are all based upon shape or color information of machine vision images for plant species recognition, Williams and Hunt (2002) indicated that differentiation of individual plant species is challenging because, among other reasons, all green plants have similar color characteristics and high spectral resolution data is probably a more appropriate for identifying and mapping individual plant species with a high level of accuracy and precision. Slaughter et al. (2008a) also commented that the use of spectral reflectance features for discriminating plant species had significant advantages over the shape- or color-based methods due to its robustness to partial leaf occlusion and less computational intensity. There have been many research attempts and successful applications conducted using plant foliage reflectance for weed detection in the field environment (Zwiggelaar, 1998; Brown and Noble, 2005; Christensen et al., 2009; López-Granados, 2011). Integrating narrow-banded contiguous spectral fingerprints with spatial registry of each pixel in an image array, hyperspectral imaging (HIS, Goetz et al., 1985) is promising for plant identification and localization in the context of automated intra-row weed control. In the last decade, there have been a number of studies applying HSI in plant species identification. Borregaard et al. (2000) conducted a laboratory-based, crop-weed discrimination study for sugar beets and potatoes (Solanum tuberosum L.) using controlled lighting. They developed a crop-weed classifier, with a combined-three-species weed class, and successfully recognized 89% of sugar beets, using narrowband reflectance at 694 nm and 970 nm, and 94% of potatoes using 856 nm and 686 nm reflectance. Feyaerts and van Gool (2001) developed a hyperspectral vision system which successfully distinguished 80% of sugar beet and 91% of weed using six wavelengths extracted from the 435e1000 nm spectral range under real-field conditions. Slaughter et al. (2008b) applied line imaging spectroscopy for precision weed control in direct seeded lettuce (Lactuca sativa L.). The classifier used reflectance values from a small spatial area, 3 mm diameter, in order to allow the method to be robust to occlusion and to eliminate the need to identify leaf boundaries. Field hyperspectral data (384e810 nm) collected from 150 plants established a 21-

97

waveband classifier that recognized 90% of plant canopy in a cross-validation analysis. Slaughter et al. (2008b) also commented that if an average weed classification accuracy of approximately 88% using a multiwaveband model could be translated into a similar level of weed control, the model could be adapted for crop/weed sensing using an automated weed control machine. Similar criteria were identified by Vargas et al. (1996) indicating that weed control efficacy above 85% would have superior performance to weed control achieved with many hand labor crews. Ongoing developments for large scale alternative weed control strategies include biodegradable materials (e.g., Tworkoski, 2002) for spray applications, site-specific mechanical cultivation techniques (e.g., Nørremark et al., 2008) and micro-liter scale herbicide applications (e.g., Giles et al., 2004). Several researchers have evaluated the use of essential oils (containing volatile aromatics) from natural plant products for organic weed control. Early work by Tworkoski (2002) found that plant-derived oils from red thyme (Thymus vulgaris), summer savory (Satureja hortensis), cinnamon (Cinnamomum zeylanicum) and clove (Syzgium aromaticum) were the most phytotoxic to Chenopodium album L. (common lambsquarter), Ambrosia artemisiifolia L. (common ragweed) and Sorghum halepense L. (johnsongrass) at concentrations of 5e10% (v/v) when applied with two adjuvants (nonionic surfactant and paraffinic oil blend at 0.2% (v/v)). Applications were made with hand-held sprayers to the point of runoff; weed mortality was observed within 1 h to 1 d after application. The barriers for the use and commercialization of essential oils as herbicides (Isman, 2000) include their scarcity and the requirement for standardization and quality control of the material. While essential oils can be phytotoxic and efficacious against pests, their selectivity has not been well documented. Clove oil, for example, can control broadleaf weeds at high concentrations. However its cost makes broadcast applications prohibitive, even in high-value vegetable production systems (Boyd and Brennan, 2006). Mechanical methods of removing weeds within the seedline, including mechanical knives and rotating hoe, are suited for use as robotic weed control actuator (Slaughter et al., 2008a). However, the performance of mechanical weed removal is constrained by the spacing between weeds and crop plants, the height of plants (determining the depth of cut) and operation timing (the time between the motion of the mechanical tool is initiated and the tool actually cuts into the row) (Garrett, 1966a, 1996b). Åstrand and Baerveldt (2002) utilized the rotating hoe as the weed control actuator and they noted that the weeds not removed by the robot in the field test were mostly (31%) proximate to crop plants. Thermal weed control, besides mechanical methods, is an organic alternative for selective plant removal in automated weed control systems. The primary advantage of this method over mechanical cutting is that it minimizes soil disturbance and its incorporation with the “micro-spray” technique, i.e., pulsed jet micro-dosing application (Giles et al., 2004), increases the spatial resolution of weed control. In Lamm et al.’s (2002) work discussed previously, the selective micro-dosing application achieved spatial accuracy in the order of centimeter. The critical factor for efficacious thermal weed control is the application temperature. Levitt (1980) demonstrated that higher temperatures are more effective for thermal weed control. Lee et al. (1999) developed a prototype of micro-spray system for precision weed control in tomato seedlines identified by machine vision. The system consisted of an array of eight nozzles (four in series), each responsible for spraying weeds in a target grid of 0.63 cm  1.25 cm. Based upon the detection of weed foliage, a spray map was generated by the machine vision system and used to direct a micro-controller which triggered specific solenoid valves to deliver liquid to the corresponding nozzle. A buffer zone, one

98

Y. Zhang et al. / Crop Protection 41 (2012) 96e105

grid cell, was placed around the boundaries of the identified crop plants in the spray map to prevent inadvertent spray due to splash and to compensate for some plant recognition errors. A concept of targeted thermal micro-dosing for agricultural applications was tested prior to the present study based upon initial work by Lee et al. (1999), Lamm et al. (2002) and Giles et al. (2004). Initial bench-top studies from 2004 established the feasibility of using food-grade oil (canola) for thermal treatment of several weeds, Echinochloa crus-galli L. (barnyardgrass), Portulaca oleracea L. (purslane), Amaranthus retroflexus L. (redroot pigweed) and Solanum nigrum L. (black nightshade), common in Californian processing tomato fields. In these tests, the pulsed-jet micro-dosing concept was evaluated for selective high-temperature oil applications based on pulse time for the liquid emission, pressure of the delivery system and oil temperature. The preliminary results showed that efficacy of weed control using heated oil decreased with temperature and that a minimum of 150
C was required for reliable efficacy (weed mortality) based on a reasonable application volume per leaf size area. In addition, Giles et al. (2005) recommended that higher temperatures were required to achieve greater than 75% weed mortality for a variety of weed species found in field-grown processing tomatoes. To the authors’ knowledge, there has been no prior research investigating the potential of a thermal micro-dosing application system coupled with a hyperspectral imaging system for plant species identification and automated weed control in row crops. Although a number of studies have reported success of applying machine vision techniques both on-line and off-line for plant recognition (most of the work is reviewed by Zwiggelaar (1998), Lamb and Brown (2001), Scotford and Miller (2005), Brown and Noble (2005), Slaughter et al. (2008a)), few studies have demonstrated the performances of the developed machine vision systems for weed control practices under real-world conditions. The general objective of this study was to develop a new system for organic intrarow weed control in high-valued crops using hyperspectral imaging for species identification and a pulsed-jet thermal micro-dosing for weed control. The integrated weed mapping and thermal treatment system described in this work was evaluated through the following experimental objectives: (1) to investigate the use of a hyperspectral vision identification system and a multispectral image processing technique for accurately identifying and mapping weeds within the seedline at the early growth stage using tomato (Lycopersicon esculentum Mill.) as the target crop; (2) to evaluate weed control performance of the system when coupled with an automated pulsed-jet micro-dose application system for organic crops using high-temperature, food-grade oil; (3) to establish the performance of the automated weed control system under outdoor conditions and identify application coverage requirements for weed control.

Fig. 1. Indoor heated oil application system using hyperspectral imaging for species identification and weed mapping (Note the original prototype thermal micro-dosing application system, located on the right of the image, contained a single reservoir oil heater, distribution system, pump and original micro-dosing manifold).

leaking; (3) the manifold delivery system tended to drip oil due to an inadequate length to diameter ratio of the application orifices/ nozzles when applying high-temperature oil; (4) recirculated foodgrade oil degraded over time due to oxidation (Sebben et al., 1998), resulting in clogging of the nozzles and gel formation within the solenoid valves, leading to inadequate application performance of the heated oil and high maintenance requirements. Based on these initial design challenges the micro-spray manifold and heated oil reservoir system were re-designed to provide a robust, drip-free system that was able to maintain the thermal capacity of the heated oil through the distribution system prior to application onto weed plants and minimize oil degradation due to oxidation. A conceptual rendition of the upgraded heated oil microdosing application system is shown in Fig. 2. The new design for heated oil applications utilized pressurized nitrogen gas, providing a pressure head, to deliver the heated oil to the spray manifold through an internally pressurized chamber. Fig. 3 shows the interior food-grade oil chamber that was submersed within an outer heated reservoir (nominal volume of 19 L). Oil (Bio Flo FG32, f BioBlend Renewable Resources, Joilet, IL) placed in the outer reservoir was heated with an immersion heater (Model SETFAA, 120 V, 1650 W, Charmglow, Columbus, GA) and the reservoir was insulated to minimize heat dissipation (Fig. 3b). Food-grade oil was contained within the pressurized inner chamber (nominal capacity of 3 L) submersed within the outer heated reservoir. A third oil containment chamber was used to capture inadvertent splashing from the unpressurized heated exterior reservoir. The interior (submersed) pressure chamber was

2. Materials and methods 2.1. Thermal weed treatment system The thermal micro-spray system described here was developed based on the previous bench-top studies. The prototype system for pressurized heated-oil micro-dosing applications is shown in Fig. 1 as interfaced with the hyperspectral imaging system for discrimination between tomato and weeds. Initial studies were conducted indoors to evaluate the thermal food-grade oil treatment for weed control using this system. However, the heated oil application device was determined to be inadequate for a number of reasons: (1) it was found that the heated oil lost most thermal energy during circulation through the micro-dosing manifold and transport back to the main containment reservoir; (2) robust fittings for handling high temperatures were required to prevent the connections from

Fig. 2. Schematic diagram of micro-dosing application system for weed control using heated food-grade oil.

Y. Zhang et al. / Crop Protection 41 (2012) 96e105

99

cylinder provided a constant pressure of 275 kPa (single stage regulator) within the headspace volume of the submersed interior chamber to deliver heated oil from the interior reservoir to the manifold/nozzles for site-specific heated oil application. The micro-dosing manifold, for delivering heated oil to weeds, was fabricated from brass with 0.28 mm inner diameter stainless steel dosing tubes pressed and glued with high-temperature epoxy into the manifold. Length to diameter ratio (L/D) for all tubes was 45. The internal chamber of the dosing manifold provided heated oil, under pressure, to a bank of eight nozzles where each nozzle contained 4 dosing tubes. The flow through each nozzle was independently controlled by a bank of eight solenoid valves (12 Vdc, 7 W, 700 kPa, 2.4 mm dia. Orifice, Model 359014, Kip, Inc., Farmington, CT). At the dosing manifold, two electric resistance heaters (Model DI-5575-KEP, Therm-Coil MFG, 120 V, 100 W, West Newton, PA) was embedded in the manifold to boost the temperature of food-grade oil as it passed through the dosing manifold and nozzles prior to application. For both of the prototype and the upgraded thermal microdosing application systems, each nozzle (consisting of four oil application tubes) was capable of applying heated oil to a rectangular area approximately 0.64 cm  1.27 cm and delivered 85 mg/ cm2 (standard deviation of 19 mg/cm2) of heated food-grade oil based on a ground travel speed of 0.04 m/s and 10 ms valve opening time. The travel speed of the system was constrained by the hyperspectral image transfer rate from camera to computer. Unless otherwise noted, the average oil temperature during application was approximately 160
C. 2.2. Hyperspectral species identification system

Fig. 3. Thermal application system showing (a) pressurized interior chamber for foodgrade oil and (b) outer reservoir used to heat the interior chamber.

fabricated with 6061 aluminum and sealed by TIG (gas tungsten arc) welds. All connection fittings were stainless steel (nominal 6.35 mm) for transporting heated food-grade oil to the spray manifold for micro-dosing applications. Fins were added to the interior oil chamber to increase heat transfer rates to the foodgrade oil volume (Fig. 3a). There was no fluid contact, or mixing, between the food-grade oil in the interior chamber and the heating oil located in the outer reservoir. This design eliminated both oil recirculation and oxidation. The outer reservoir heating oil (flashpoint higher than 204
C) was 75% biodegradable, non-toxic and had no expected adverse effects on humans and the environment including fish and wildlife. Food-grade canola oil is generally recognized as safe (GRAS) as defined by the US FDA. Physical density and kinematic viscosity of the canola oil applied to the weeds were reported as 0.91 g/ml and 170 mm/s2 respectively at room temperature and 0.85 g/ml and less than 20 mm/s2 respectively at 110
C, with a flash point of approximately 275
C (Przybylski, 2009). The oil jet stream left the nozzles at approximately 4.5 m/s, similar as in Lee et al. (1999)’s prototype system. A pressurized nitrogen (0.3 m3, 13,700 kPa)

A prototype of hyperspectral imaging system was developed to identify processing tomatoes versus weed species (Fig. 1). The core component of the system was a 12-bit, temperature-controlled, monochrome area camera (Photometrics CoolSNAPcf, Roper ScientificÒ Photometrics, Tucson, AZ) equipped with a hyperspectral spectrograph (ImSpector V8_4_102, Spectral Imaging Ltd., Oulu, Finland), which had a nominal spectral wavelength range of 384e810 nm and half band-width of 5 nm. To improve spectral uniformity and signal to noise ratio, a blue filter (KB-12, BþW, Jos. Schneider Optische Werke GmbH., Bad Kreuznach, Germany) was placed on the camera lens (12 mm F/1.2, C61215, Cosmicar/Pentax, Hoya Corp., Tokyo, Japan). The camera height was adjusted, resulting a field of view of 2.5 mm along and 10.8 cm across the seedline. Scene illumination was provided by two tungsten-halogen bulbs (Ushio EYF/FG, 12 Vdc, 75 W, SP12) collimated through a light duct (76 cm in length with rectangular dimensions of 20.3 cm  7.6 cm transitioning to 20.3 cm  5.1 cm) and directed onto the plant leaf foliage at a 15
angle to vertical direction to avoid glare. A line imaging configuration was used where the camera lineby-line sequentially scanned the surface of the seedline perpendicular to the travel direction as the system moved along the row. A 4  4 binning was operated in the camera before data transfer to reduce noise and minimize image transfer time. The resulted hyperspectral images had a size of 330 pixels in the spatial dimension (perpendicular to the seedline) and 260 pixels in the spectral dimension. The system was calibrated (SPECIM, 2003) for dark signal noise and responses to an optically-flat reference plate (whose reflectance intensity was nearly uniform in the studied spectral range) before acquiring canopy reflectance of the crop row. 2.3. System integration and operation The hyperspectral imaging system was mounted in an enclosed chamber, with a black interior and white exterior with rubber sheet

100

Y. Zhang et al. / Crop Protection 41 (2012) 96e105

draped against the ground along the circumference, to prevent ambient light from entering the system area (Fig. 4). For initial indoor tests, the original prototype was configured as shown in Fig. 1. In the outdoor test, the hyperspectral imaging chamber and the upgraded thermal micro-dosing system were mounted on a cultivation sled (Fig. 4). The spray manifold used for thermal micro-dosing application was positioned approximately 52 cm behind the camera. The two systems were integrated and synchronized with a ground-driven encoder wheel location stamp (i.e. by odometry). For indoor tests, plants were placed on trays of an in-house developed transportation frame (not shown), which traveled at a continuous speed of 0.04 m/s and the weed control system was stationary. The transportation frame simulated the motion of a tractor driven along field beds. In the outdoor test, the cultivator sled was pulled by a tractor (John Deere 6430, 70 kW PTO, Deere & Company, Moline, IL, equipped with transmission that allowed for slow speed transport) traveling continuously at approximately the same speed as indoor tests (Fig. 4). All electrical power requirements for the camera, illumination, heating components and computer system were provided by a gas-powered generator (120 V, 2000 W, Model EU 2000ia, Honda Corp., Alpharetta, GA). 2.4. Thermal weed treatment Two weed species, S. nigrum and A. retroflexus, common in processing tomato fields in California’s Central Valley (UC IPM, 2009) were grown to evaluate the performances of the hyperspectral weed detection system and thermal micro-dosing system. Six tomato cultivars (SEMINIS APT 410, Campbell CXD 179, BOS Halley 3155, SEMINIS Hypeel 108, Heinz 8892 and Heinz 9780) were used for indoor thermal application tests; cultivar “Heinz 8892” was used for outdoor tests. All plants were grown from seed to the 2nd-true leaf juvenile stage in small pots stationed in a controlled environment facility at the University of California, Davis. The environmental settings (temperature, humidity, illumination, etc.) were controlled automatically to simulate the springtime natural growing conditions typical of a processing tomato field in the Valley (UC IPM, 2007). Thermal treatment application tests were conducted both indoors and outdoors to assess the system performance for species identification and weed control. Initial “treat-all-green” tests were performed indoors to ensure the system was correctly identifying and dosing green plant material. In these tests, only S. nigrum and A. retroflexus plants, excluding the ones reserved for control (Table 1), were scanned by the hyperspectral camera and treated with heated food-grade oil. Plants were mapped in real-time using a green foliage segmentation algorithm, modified red ratio vegetation index (MRVI; Biller, 1998), which calculated the ratio of the reflectance intensities at 555 nm (green) and 665 nm (red). Independent of species, this algorithm correctly identified more than 95% of the plant foliage pixels from the background using a threshold of 1.4. Initial “treat-by-species” tests were evaluated indoors. In these tests, the tomato (6 cultivars) and weed plants (Table 2), except the ones selected for control, were intermingled by species and the system was set to detected weeds and apply thermal treatment exclusively to the identified weed foliage (i.e., not to tomato foliage) on the fly. Zhang and Slaughter (2011) demonstrated that the development of tomato cultivar-specific classifiers was unlikely to be of much benefit in this context). Therefore, an abridgedspectrophotometer Bayesian classifier (Slaughter et al., 2008b) was pre-trained using hyperspectral canopy reflectance to optimize the discrimination between tomato and weeds. A feature vector using fifteen wavebands uniformly spaced over the 384e810 nm

range was determined to be the simplest model producing the minimum total error rate without overfitting using cross-validation analysis. A pixel-based weed map was then generated based upon the discriminant results of the Bayesian classifier. To translate the weed map into a spray control map (plant-based), the predominant object classification (weed, tomato, soil) in each 1.27 cm linear region of the hyperspectral image, which corresponded to the 1.27 cm size of a nozzle target zone, determined the spray decision for that zone. Thus, if the majority of pixels in a nozzle zone were classified as weeds, then that zone was targeted (i.e., not to the ground or tomato plants) for heated-oil spray at the appropriate time during travel. System performance for outdoor application, based on the “treat-by-species” mode, was evaluated in August 2008. Approximately one-third of the plants were randomly selected for each species as the calibration set to train the hyperspectral imaging classifier while the remaining plants were identified as the validation set (Table 4). A 13-waveband Bayesian classifier was developed using stepwise discriminant analysis (Zhang and Slaughter, 2011). This model was the simplest to achieve the optimum classification rate for the calibration plant spectra using 20-fold cross-validation analysis (Duda et al., 2001) and was slightly more efficient than the abridged feature selection method. For the heated-oil test, eight plants of each species were randomly selected from the calibration set of the plant classifier and used as thermal treatment control (untreated) plants (Table 5). The remaining plants, including the other plants in the calibration set and the entire validation set (Table 5), were placed outdoors along a row (concrete base) to simulate a weedy tomato seed-line for the thermal dosing application. In the row, the tomato plants were spaced 45 cm apart and in between, two plants of each weed species were placed in random order, resulting a 15 cm interval between plants (Fig. 5c). A visual assessment was performed for all treated plants immediately after the oil application. The extent of oil coverage as a percentage of plant foliage area, i.e., less than 10%, 10e25%, 25e50%, 50e75%, 75e90% and over 90%, were determined for each treated plant. The heated-oil coverage for control plants was defined as 0. After treatment, all plants (control and dosingtreated) of the indoor tests were returned to the controlled growth chambers; while all plants of the outdoor test were exposed to ambient conditions and irrigation scheduling typical of

Fig. 4. Hyperspectral imaging/heated oil dosing application system mounted for field transport.

Y. Zhang et al. / Crop Protection 41 (2012) 96e105

101

Table 1 Weed mortality and viable biomass 15 days post treatment of indoor “treat-all-green” tests. Plant species

Heated-oil treatment

Surface coverage (%)

Number of plants

15-day mortality (%)

Viable biomass (g)a

Solanum nigrum L.

Control (untreated) Treated by coverage

0 75e90 90e100 e 0 25e50 75e90 90e100 

4 21 29 50 14 6 38 58 102

0.0 81.0 82.7 82.0 0.0 33.3 78.9 93.1 84.3

0.30 0.02 0.03 0.03 1.05 0.14 0.09 0.02 0.05

Amaranthus retroflexus L.

All treated Control (untreated) Treated by coverage

All treated

a b b a b b b

Entries with identical lettering within each species were not significantly different at a ¼ 0.05 level using the Tukey multiple means test. Harmonic means of cell size (7 and 14 respectively for S. nigrum and A. retroflexus) were used in the analysis due to unequal sample numbers for each treatment (control and treated with each defined spray coverage). a

Table 2 Weed mortality and viable biomass 15 days post treatment of indoor “treat-by-species” tests. Plant species Tomato

a

Solanum nigrum L.

Amaranthus retroflexus L.

Heated-oil treatment

Surface coverage (%)

Number of plants

15-day mortality (%)

Viable biomass (g)b

Control (untreated) All treated Control (untreated) Treated by coverage

0

206 271 30 11 28 44 145 228 29 9 8 58 121 196

0.0 0.0 0.0 0.0 7.7 38.6 72.4 54.4 0.0 0.0 12.5 46.6 76.0 61.2

1.58 1.59 0.92 0.40 0.17 0.11 0.02 0.07 0.80 0.85 0.38 0.31 0.02 0.16

All treated Control (untreated) Treated by coverage

All treated

0e10 0 25e50 50e75 75e90 90e100 e 0 25e50 50e75 75e90 90e100 e

a a a b c cd d a a b b c

a

Population of treated tomato was composed of 46 SEMINIS APT 410, 46 Campbell CXD 179, 40 Halley 3155, 56 SEMINIS Hypeel 108, 45 Heinz 8892 and 38 Heinz 9780 plants. b Entries with identical lettering within each species were not significantly different at a ¼ 0.05 level using the Tukey multiple means test. Harmonic means of cell size (234, 26 and 17 respectively for tomato, S. nigrum and A. retroflexus) were used in the analysis due to unequal sample numbers for each treatment (control and treated with each defined spray coverage).

a processing tomato field. Efficacy of all heated-oil tests (indoors and outdoors) was evaluated using a visual rating of weed control relative to the control plants (Giles et al., 2004). Manual counting of viable and dead plants was conducted 5, 10 and 15 days post application. Weed mortality (on a percentage basis) was determined according to assessments of weed symptoms for the cumulative 15-day span. All viable plants (control and dosingtreated) were harvested 15 days post the tests and biomass yield was evaluated by measuring the dry mass (oven dried at 105
C for 24 h) of each viable plant. 3. Results and discussion 3.1. Indoor thermal weed treatment Fifteen-day cumulative weed mortality and viable biomass yield of the indoor “treat-all-green” tests are shown in Table 1. It should Table 3 Pixel based classification performance of the hyperspectral species classifier. Actual plant species Classified plant species (%, with number of pixels below)

Tomato Solanum nigrum L. Amaranthus retroflexus L.

Tomato

Solanum nigrum L.

Amaranthus retroflexus L.

94.9% 66,220 6.0% 6285 1.2% 1375

4.6% 3225 93.5% 97,388 0.2% 275

0.5% 333 0.5% 498 98.6% 113,289

be noted that the 15-day mortality entry (on a percentage basis) in Table 1 indicates the percentage of plants within that grouping of heated-oil coverage that had died. In general, the hyperspectral imaging system correctly identified and mapped the majority of the weed canopy and the thermal micro-dosing treatment was efficacious for weed control. Fifteen-day mortalities for S. nigrum and A. retroflexus were 82.0% and 84.3%, respectively. All S. nigrum plants and 94.1% of A. retroflexus plants treated with thermal microdosing had heated-oil coverage of over 75% of their leaf surface. At this level of surface area coverage, an average of about 82% of all treated plants (S. nigrum and A. retroflexus combined) died 15 days post treatment. These results were also verified by the viable dry mass data. Weeds still living 15-day post treatment with greater than 75% of their leaf surface treated by heated oil also had significantly (a ¼ 0.05) decreased dry mass compared to the control plants. The thermal treatment resulted in significantly (a ¼ 0.05) lower biomass yields for both weeds compared to the untreated control. In general, weed mortality was positively correlated with the spray coverage while biomass yield was negatively correlated with it. Increased heated-oil coverage of leaf surface impacted plant viability and reduced dry mass of treated plants. On average, the proportion of the leaf surface treated with heated-oil was below the 95% foliage mapping accuracy of the green foliage segmentation algorithm used by the hyperspectral imaging system. In the case of A. retroflexus, about 6% of the plants received heated-oil coverage of less than 50%. It is probably because the plant size of several individuals exceeded the view of the hyperspectral camera (10.8 cm) and the range of the spray manifold (10.2 cm) in the direction perpendicular to the plant row. Another

102

Y. Zhang et al. / Crop Protection 41 (2012) 96e105

Table 4 Calibration versus validation spray performance from outdoor “treat-by-species” tests. Plant species

Classification data

Number of plants

Heated-oil surface coverage (%)

15-day mortality (%)

Viable biomass (g)

Tomato

Calibration Validation Calibration Validation Calibration Validation

28 63 42 62 32 72

12.2 9.9 96.5 91.7 94.4 89.8

5.0 1.6 94.1 96.8 100.0 91.7

1.09 1.32 0.01 0.01 0.00 0.05

Solanum nigrum L. Amaranthus retroflexus L.

b b b b b b

a

a a a a a a

a Entries with identical lettering within each species were not significantly different at a ¼ 0.05 level using the Tukey multiple means test. Harmonic means of cell size (36, 30 and 44 respectively for tomato, S. nigrum and A. retroflexus) were used in the analysis due to unequal sample numbers for each classification set.

possible reason was that leaf movement (typically deflection and drooping) occurred during the micro-dosing treatment as a result of the heat and or kinetic energy transfer to the leaf by the jet of oil. Fig. 5a shows an example of a S. nigrum leaf deflected when the jet stream of heated oil hit the surface during an indoor test. For the indoor “treat-by-species” tests, the majority of tomato foliage (above 90%) was correctly identified by the hyperspectral imaging system and the system correctly avoided treating nearly all tomato foliage (Table 2). Less than 10% of tomato leaf area was mistreated with a small dose of heated oil. This is possibly due to hyperspectral misclassification (approximately 6.6% overall) and occasional minor timing errors associated with image capture and dosing synchronization. However, these inadvertent thermal treatments did not impact the viability of tomato plants. No mortality of tomato occurred due to these misapplications and the 15-day biomass yield of the tomato plants that were mistreated was not significantly (a ¼ 0.05) lower than that of the control plants (Table 2). Although the performance of the hyperspectral species identification system degraded slightly in mapping weed foliage compared to discrimination of tomato, the majority of weeds (82.9% of S. nigrum and 91.3% of A. retroflexus) received heated oil coverage to over 75% of the leaf surface and the remaining plants had above 25% of the foliage treated by thermal dosing (Table 2). On average 54.4% of S. nigrum and 61.2% of A. retroflexus plants were dead 15 days after the heated oil application. The viability of the living weeds was severely impacted by the treatment with their biomass yields significantly (a ¼ 0.05) reduced when compared to the control plants. As expected, there was a positive correlation of the proportion of the foliage treated to the 15-day mortality rate, and a negative correlation to viable biomass. These results were consistent with those observed in the indoor “treat-all-green” tests. In general, spray coverage to more than 90% of leaf surface killed

72.4% of S. nigrum and 76% of A. retroflexus and dramatically reduced the viable biomass of plants not killed. The reduction in weed viability was less severe if the surface coverage fell below 90%. Less than half of the treated weed plants were dead 15-days post treatment when 50e90% of the surface was treated and all weeds survived when 50% or less of the foliage was treated by thermal dosing. However, 50% or more heated oil coverage was adequate to impair weed viability, resulting in a significant (a ¼ 0.05) decrease in dry mass 15 days post treatment. 3.2. Outdoor thermal weed treatment The 13-waveband stepwise Bayesian classifier calibrated before the outdoor application test showed very good species identification performance on the training plants. On a plant basis, the system correctly mapped nearly all (100%) plant species in the training set. On a pixel basis, the classifier successfully discriminated approximately 94.9% of tomato, 93.5% of S. nigrum and 98.6% of A. retroflexus (Table 3). The validation performance of the hyperspectral species identification system was evaluated using the inspection results of the thermal treatment test and compared to the calibration performance in Table 4. The system showed no consistent bias of validation set in favor of the calibration set for all three species. The 13-wavebands selected for the Bayesian classifier did not appear to overfit the classifier training data. Viable biomass assessment also confirmed that the dry mass of living plants in the validation set was not statistically (a ¼ 0.05) different from that of plants in the calibration set. The results are also consistent with the viable plant count analysis. Lethalities observed in the calibration set for all three species were virtually comparable to those of the validation plants.

Table 5 Weed mortality and viable biomass 15 days post treatment of outdoor “treat-by-species” test. Plant species

Heated-oil treatment

Heated-oil surface coverage (%)

Number of plants

15-day mortality (%)

Viable biomass (g)a

Tomato

Control (untreated) Treated by coverage

0

8 63 10 8 2 83 8 5 3 88 96 8 1 4 9 82 96

0.0 0.0 0.0 0.0 100.0 2.4 0.0 40.0 66.7 100.0 95.8 0.0 0.0 0.0 88.9 100.0 93.8

1.13 1.43 0.87 0.78 0.00 1.27 1.18 0.09 0.04 0.00 0.01 1.76 1.14 0.58 0.04 0.00 0.04

Solanum nigrum L.

All treated Control (untreated) Treated by coverage

Amaranthus retroflexus L.

All treated Control (untreated) Treated by coverage

All treated

0e10 10e25 25e50 50e75 e 0 50e75 75e90 90e100 e 0 0e10 25e50 75e90 90e100 e

a a a a b a b bc c a b c d d

Entries with identical lettering within each species were not significantly different at a ¼ 0.05 level using the Tukey multiple means test. Harmonic means of cell size (6 for both tomato and S. nigrum; and 3 for A. retroflexus) were used in the analysis due to unequal sample numbers for each treatment (control and treated with each defined spray coverage). a

Y. Zhang et al. / Crop Protection 41 (2012) 96e105

Fig. 5. Thermal micro-dosing spray application in indoor and outdoor tests: (a) S. nigrum leaf deflected during an indoor test due to heat and or kinetic energy transfer by the oil jet; (b) A. retroflexus leaf surface wilting/drooping effect observable immediately after heated oil application in outdoor test; (c) Heated-oil treated S. nigrum plant with a tomato plant identified and unsprayed by the system immediately after outdoor application.

103

The overall species mapping performance of the hyperspectral imaging system and the weed control performance of the thermal micro-dosing system were established in the outdoor test (Table 5). Although minor mis-spray occurred on many tomato plants in the spray trial, the plant viability was retained after 15 days and the dry mass of the mistreated plants did not significantly (a ¼ 0.05) deviate from that of the control. Surface coverage of up to 50% did not kill juvenile (2nd-true-leaf stage) tomato plants and its impact on biomass reduction was minor. Only two (2.4%) tomato plants in the study received hot oil treatment to more than 50% of the foliage, and neither survived 15 days post treatment. Overall, 95.8% of the S. nigrum and 93.8% of the A. retroflexus were controlled using the thermal treatment system and the resulting biomass yields were significant less than those of the control. Results from coverage estimates found that 100% of the plants died if over 90% of the leaf surface was treated. Although viability was retained if less than 90% of the leaf surface was treated, the mean biomass of weeds alive 15-days post treatment was significantly (a ¼ 0.05) less than untreated controls. This indicated some potential benefit to thermal treatment in terms of reduced weed vigor even when complete control was not achieved. None of the five A. retroflexus plants receiving heated-oil spray of less than 50% of their foliage surface were killed, indicating that spray coverage of lower than 50% is not sufficient for this type of thermal micro-dosing weed control. This level of performance meets the 88% target control rate suggested by Slaughter et al. (2008b) for achieving automated weed control of good practical potential. In total, 8.3% of S. nigrum and 14.6% of A. retroflexus plants received less than 90% surface coverage. One possible reason that the surface coverage was slightly lower than the mapping accuracy of the hyperspectral imaging system (Table 3) might be that the impinging jet stream imparted kinetic energy to the leaf surface, and in combination with the thermal effects and may deflect and/or re-orient the original surface imaged prior to nozzle actuation. Since the spray map was not dynamically updated in real-time, spray was applied to the place where the leaf was located at the time of image acquisition, but may spray off-target due to subsequent motion of the foliage. An example of the re-orientation effect is shown in Fig. 5a; readily observable is the immediate wilting of leaves and deformation of the plant when compared to the original form and location that was presented to the hyperspectral imaging and identification system (Fig. 5b). Heated oil coverage of less than 75% was probably due to a centering alignment issue associated with the 10.2 cm maximum spray zone width, i.e., some plants were not perfectly centered when manually placed along the row before the test. Portions of the misaligned plant foliage were out of the range of both the camera view (5.4 cm half-way width) and the spray manifold (5.1 cm halfway width). Thus, those portions of the leaf foliage may have been missed because it was outside the physical range of application. This problem could be eliminated by increasing the field of view of the imaging system and by extending the width of the micro-spray boom. As a whole, the results of the outdoor test demonstrated that this technology of integrating thermal micro-dosing of hightemperature food-grade oil for weed control with hyperspectral imaging for plant species identification has the good potential to be translated into real-world applications to improve the weed control efficacy in row crops. 4. Conclusions Weed control within early growth tomatoes was accomplished using an automated machine vision identification and thermal

104

Y. Zhang et al. / Crop Protection 41 (2012) 96e105

treatment application system. This study described a hyperspectral imaging system that was trained to identify tomato and two weed species, S. nigrum and A. retroflexus, during thermal, non-chemical, applications within the seed-line of early growth processing tomatoes. The results from this study lead to the following conclusions: (1) A hyperspectral imaging system, using a multivariate Bayesian discriminant analysis, overall correctly identified about 95.9% of plant canopy for tomatoes, S. nigrum and A. retroflexus. The 275 kPa pressurized thermal application system accurately delivered 85 mg/cm2 per 10 ms pulse duration of food-grade oil preheated to 160
C to targeted weed foliage based on the automatic species identification and mapping by the vision system. (2) Outdoor test results found that the spray map generated by the image processing technique correctly treated over 91% for all three species without bias of validation set used for testing the target heated-oil applications for weed control in favor of the calibration set used to train the species classifier. The vision/ thermal application system was able to traverse a simulated processing tomato seed-line at 0.04 m/s and control approximately 95.8% of S. nigrum and 93.8% of A. retroflexus 15-day post thermal treatment; while only 2.4% of tomato was damaged to the extent of non-viability due to inadvertent spray. (3) Heated-oil deposition assessments found that tomato plants survived and S. nigrum and A. retroflexus plants remained viable as well if less than 50% of the leaf surface area was treated. Complete weed control (100% lethality) was achieved if over 90% of the leaf surface was treated. Before adapting this technology for use in agricultural production, future work is needed to assess the application efficiencies of the hyperspectral vision/heated oil application system under realfield conditions. The ground speed of the current prototype system was principally constrained by the transfer rate of the image frames from the camera to the computer. Modern GigE-based, spectroscopy-grade cameras, should make it possible to improve the data transfer rate nearly 2 orders of magnitude and modern parallel graphics processors (for which hyperspectral imaging is well suited) can dramatically minimize overall computer processing time. These improvements can eliminate the current travel speed constraints due to the hyperspectral weed sensing system of the tested prototype, allowing travel speeds to approach commercially acceptable levels, limited only by mass transfer rates of the thermal fluid application system. The hyperspectral imaging classifier used for species identification in this study was trained with plants grown in a controlled environment. Successful field application requires investigation of the spectral discrimination technique for robust weed detection under natural variability in plant canopy reflectance due to varying growing conditions. Finally, the potential yield savings and cost of automated weed detection and control may need to be evaluated and compared to the conventional methods for commercializing this technology.

Acknowledgments Funding for this research was provided in part by the California Tomato Research Institute, the Agricultural Experiment Station of the University of California, the U.S. Department of Agriculture Buy California Specialty Crops Research Program, and the California Department of Food and Agriculture Buy California Initiative. The authors wish to thank Burt Vannucci, Rassam Zarghami and David Salzer of UC Davis for their technical assistance.

References Åstrand, B., Baerveldt, A.-J., 2002. An Agricultural mobile robot with vision-based perception for mechanical weed control. Auton. Robots 13, 21e35. Åstrand, B., Baerveldt, A.-J., 2004. Plant recognition and localization using context information. In: Proc. of the IEEE Conference on Mechatronics and Robotics e Autonomous Machines in Agriculture, pp. 1191e1196. 13e15 September, Aachen, Germany. Åstrand, B., Baerveldt, A.-J., 2005. A vision based row-following system for agricultural field machinery. Mechatronics 15, 251e269. Biller, R.H., 1998. Reduced input of herbicides by use of optoelectronic sensors. J. Agric. Eng. Res. 71, 357e362. Borregaard, T., Nielsen, H., Norgaard, L., Have, H., 2000. Crop-weed discrimination by line imaging spectroscopy. J. Agr. Eng. Res. 75, 389e400. Boyd, N.S., Brennan, E.B., 2006. Burning nettle, common purslane, and rye response to a clove oil herbicide. Weed Technol. 20, 646e650. Brown, R.B., Noble, S.D., 2005. Site-specific weed management: sensing requirementsewhat do we need to see? Weed Sci. 53, 252e258. Chandler, J.M., Cooke, F.T., 1992. Economics of cotton losses caused by weeds. In: McWhorter, C.G., Abernathy, J.R. (Eds.), Weeds of Cotton: Characterization and Control. The Cotton Foundation, Memphis, TN, pp. 85e116. Christensen, S., Søgaard, H.T., Kudsk, P., Nørremark, M., Lund, I., Nadimi, E.S., Jørgensen, R., 2009. Site-specific weed control technologies. Weed Res. 49, 233e241. Duda, R.O., Hart, P.E., Stork, D.G., 2001. Pattern Classification, second ed. John Wiley and Sons, Inc., New York. Feyaerts, F., van Gool, L., 2001. Multi-spectral vision system for weed detection. Pattern Recogn. Lett. 22, 667e674. Garrett, R.E., 1966a. Development of a synchronous thinner. J. Am. Soc. Sugar Beet 14, 206e213. Garrett, R.E., 1966b. Device designed for synchronous thinning of plants. Agr. Eng. 47, 652e653. Giles, D.K., Downey, D., Slaughter, D.C., Brevis-Acuna, J.C., Lanini, W.T., 2004. Herbicide micro-dosing for weed control in field-grown processing tomatoes. Appl. Eng. Agric. 20, 735e743. Giles, D.K., Lanini, W.T., Slaughter, D.C., 2005. Precision Weed Control for Organic and Conventional Specialty Crops. In: Buy California Crop Block Grant Program Final Report. California Department of Food and Agriculture, Sacramento, CA, p. 9. Goetz, A.F.H., Vane, G., Solomon, J., Rock, B.N., 1985. Imaging spectrometry for Earth remote sensing. Science 228, 1147e1153. Isman, M.B., 2000. Plant essential oils for pest and disease management. Crop Prot. 19, 603e608. Lamb, D.W., Brown, R.B., 2001. Remote-sensing and mapping of weeds in crops. J. Agr. Eng. Res. 78, 117e125. Lamm, R.D., Slaughter, D.C., Giles, D.K., 2002. Precision weed control system for cotton. Trans. ASAE 45, 231e238. Lee, W.S., Slaughter, D.C., Giles, D.K., 1999. Robotic weed control system for tomatoes. Precis. Agric. 1, 95e113. Levitt, J., 1980. Responses of Plants to Environmental Stresses. Chilling, Freezing, and High Temperature Stresses, second ed. Academic Press, New York. López-Granados, F., 2011. Weed detection for site-specific weed management: mapping and real-time approaches. Weed Res. 51, 1e11. Nørremark, M., Griepentrog, H.W., Nielsen, J., Søgaard, H.T., 2008. The development and assessment of the accuracy of an autonomous GPS-based system for intra-row mechanical weed control in row crops. Biosyst. Eng. 101, 396e410. Pimentel, D., McNair, S., Janecka, J., Wightman, J., Simmonds, C., O’Connell, C., Wong, E., Russel, L., Zern, J., Aquino, T., Tsomondo, T., 2001. Economic and environmental threats of alien plant, animal, and microbe invasions. Agr. Ecosyst. Environ. 84, 1e20. Przybylski, R., 2009. Canola Oil: Physical and Chemical Properties, Canola Council of Canada. http://www.canola-council.org/oil_tech.aspx (accessed 05.09.). Scotford, I.M., Miller, P.C.H., 2005. Applications of spectral reflectance techniques in Northern European cereal production: a review. Biosyst. Eng. 90, 235e250. Slaughter, D.C., Giles, D.K., Downey, D., 2008a. Autonomous robotic weed control systems: a review. Comput. Electron. Agr 61, 63e78. Slaughter, D.C., Giles, D.K., Fennimore, S.A., Smith, R.F., 2008b. Multispectral machine vision identification of lettuce and weed seedlings for automated weed control. Weed Technol. 22, 378e384. Sebben, E., Slaughter, D.C., Singh, P.R., 1998. Optical assessment of corn oil deterioration during frying. J. Food Process. Preserv. 22, 265e282. SPECIM, 2003. Imspector e Imaging Spectrograph User Manual, ver. 2.2. Spectral Imaging Ltd., Oulu, Finland. Tillett, N.D., Hague, T., Grundy, A.C., Dedousis, A.P., 2007. Mechanical within-row weed control for transplanted crops using computer vision. Biosyst. Eng. 99, 171e178. Tworkoski, T., 2002. Herbicide effects of essential oils. Weed Sci. 50, 425e431. UC IPM, 2007. California Weather Data. University of California, Davis. http://www. ipm.ucdavis.edu/WEATHER/wxretrieve.html (accessed 02.07.). UC IPM, 2009. Pest Management Guidelines: Tomatoes. University of California, Davis. http://www.ipm.ucdavis.edu/PMG/selectnewpest.tomatoes.html (accessed 02.10.).

Y. Zhang et al. / Crop Protection 41 (2012) 96e105 Vargas, R., Fischer, W.B., Kempen, H.M., Wright, S.D., 1996. Cotton weed management. In: Johnson, M.S., Kerby, T.A., Hake, K.D. (Eds.), Cotton Production. UC DANR, Oakland, CA. Williams, A.P., Hunt, E.R., 2002. Estimation of leafy spurge cover from hyperspectral imagery using mixture tuned matched filtering. Remote Sens. Environ. 82, 446e456.

105

Zhang, Y., Slaughter, D.C., 2011. Hyperspectral species mapping for automatic weed control in tomato under thermal environmental stress. Comput. Electron. Agr 77, 95e104. Zwiggelaar, R., 1998. A review of spectral properties of plants and their potential use for crop/weed discrimination in row-crops. Crop Prot. 17, 189e206.