- Email: [email protected]
Leader–follower system using two robot tractors to improve work efficiency
Computers and Electronics in Agriculture 121 (2016) 269–281
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag
Original papers
Leader–follower system using two robot tractors to improve work efficiency Chi Zhang, Noboru Noguchi ⇑, Liangliang Yang Laboratory of Vehicle Robotics, Graduate School of Agriculture, Hokkaido University, Kita-9, Nishi-9, Kita-ku, Sapporo 060-8589, Japan
a r t i c l e
i n f o
Article history: Received 15 August 2015 Received in revised form 26 December 2015 Accepted 27 December 2015
Keywords: Leader–follower system Robot tractor Fault tolerant Work efficiency RTK-GPS
a b s t r a c t Two robot tractors were used in a leader–follower system for agricultural field work. Each of the robots is fully independent and can conduct field work alone. They can also work together to form a certain spatial arrangement during the operation. During the headland turn, to make the best use of headland, the two robots coordinate to turn to next path and do not keep the spatial arrangement. Each robot is simplified as a rectangular zone, and the two robots cooperate and coordinate to turn to the next path without collision. This system is designed for practical application, and the system gains the ability to tolerate most of the disturbances in a real field. Fault tolerant methods in accordance with agricultural work are illustrated to solve the common disturbances from the GPS and the IMU. Field experiments were conducted to determine the effectiveness of the system. The results of the experiments showed that the two robot tractors can work safely together to complete the field work. The average lateral error of the navigation system of the robots was less than 0.04 m, and the efficiency of the leader–follower system was improved by 95.1% compared with that of a conventional single robot. Ó 2016 Elsevier B.V. All rights reserved.
1. Introduction The coordination of plural robots has undergone rapid development in recent years. Cooperation of simple robots has many advantages such as flexibility, large application domain and capability to accomplish complex missions that cannot be accomplished by a single unit. For the use of two robots, most researchers have focused on a master–slave robot system. Kosuge and Ishikawa (1994) developed a task-oriented control algorithm for coordinated motion control of a single-master multi-slave manipulator system that consists of a master arm with six degrees of freedom and slave arms that each have six or more degrees of freedom, and they showed that the system has improved load capacity, rigidity and dexterity. Wojtara et al. (2005) reported a master–slave system consisting of a data glove and a hydraulic robot hand for removal of landmines and visible obstacles. Cheng et al. (2011) developed a network of autonomous mobile robots to accomplish the sweep coverage. They proposed a decentralized control algorithm for the robots. Robotic technology is also widely used for field tasks in production agriculture. In the agriculture sector, robot tractors have been developed for conducting field work in order to reduce work strength and work time of farmers. ⇑ Corresponding author. E-mail address: [email protected] (N. Noguchi). http://dx.doi.org/10.1016/j.compag.2015.12.015 0168-1699/Ó 2016 Elsevier B.V. All rights reserved.
Researchers have also developed master–slave robot systems to conduct field work. Noguchi et al. (2004) proposed GOTO and FOLLOW algorithms for a master–slave system. The master controls the slave to follow a parallel path at a given distance and angle from the master or to go to a certain point along any path as long as it does not collide with the master. Zhang et al. (2010) developed an intelligent master–slave system that enables a semiautonomous agricultural vehicle (slave) to follow a master with a given lateral and longitudinal offset. They used a state space dynamic model and a proportional-derivative controller with state feedback and disturbance feed-forward for the tractor. However, the slave robots in both Noguchi’s system and Zhang’s system have less decision-making capability than that of a master robot. Fault tolerance is important for robots operating in dangerous or complex environments since robot failures are likely to occur. Yang et al. (2011) developed a fault-tolerant flocking algorithm for a group of autonomous mobile robots. The algorithm includes three parts: ranking assignment, failure detector and flocking algorithm. In their study, the robots group moves in formation and preserves the formation. Zajac (2014) used a parallelizing particle filtering-based approach combined with the negative loglikelihood test for a fault detection task. It is aimed to improve the efficiency, estimation error and execution time of online setup. Portugal and Rocha (2013) reported two distributed solutions that make decisions using a Bayesian-based mathematical formalism to
270
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
coordinate a team of mobile robots. One of the advantages of this framework is that it leads to a scalable and fault-tolerant solution. Agricultural field work is needed to be done quickly in some situations. For example, sowing or harvesting before the rainy season. One robot is not enough to finish the field work in a limited time. In this study, we developed a system with two robot tractors (called a leader–follower system) to conduct field work. Unlike the master– slave robot system, each of the robot tractors in the leader– follower system is fully independent and can conduct field work alone. The two robots can also work together by forming a spatial arrangement during the operation to reduce total work time. In addition, to make the best use of headland, the two robot tractors do not need to keep a certain arrangement during a headland turn. Each robot uses its own turn method and the trajectory of the robots may collide. It is one of the differences between the current article and the previous articles (Noguchi et al., 2004; Zhang et al., 2010; Cheng et al., 2011). In this article, we focus on the usefulness of the leader–follower system that is fault-tolerant and has improved efficiency compared with a conventional single robot. For example, for a robotic combine harvester harvesting in the field, an on-the-go unloading system with a transport that moves the harvested products to collection positions helps improve harvesting efficiency since the harvester does not have to stop. Another example is that the two robots doing tillage together to extend the work width to reduce the total work time. In general, experiment differs from simulation due to the existence of disturbance which from interior or exterior of the system. The system should gain the ability to tolerate most of the disturbances in order to work safely. It is challenge to solve the problem of external disturbances including humans or animals in the field and other vehicles. Researchers have made many efforts to solve this problem. Monta et al. (1999) used four ultrasonic sensors and four infrared sensors in a grape harvesting robot for detecting humans. Kise et al. (2005) used a laser scanner in front of a robot tractor to detect and locate other vehicles working in the same field. Yang and Noguchi (2012) used an omnidirectional stereo vision sensor in a robot tractor to detect humans. As for touching sensors, Kondo et al. (2004) used a bumper switch as a robot safety system mounted in front of a robot tractor, and the robot stops if something touches the bumper. In the present study, a laser scanner and a bumper switch were used as safety sensors to ensure safety of the robots in the field. The reasons that use laser scanner are the large detection range (30 m, 270°) and high frequency (40 Hz). If the laser fails to detect the objects, the bumper switch is used to ensure the robot to be stopped. Two robots can coordinate in several ways and each of them has a different efficiency than that of a conventional single robot. The improved efficiency of common coordination method, such as the two robots kept a constant formation during operation is discussed in this study. For headland turning, the two robots coordinate and turn together if they are in a safe condition. A skipping path turn method is proposed for reducing waiting time of the two robots in order to increase the efficiency of the system. The rest of this article is structured as follows. The equipment, control design and fault-tolerant method used in the system are presented in Section 2. Results of the experiments using the leader–follower system are presented in Section 3. Conclusions are given in Section 4.
2. Method
tors, as shown in Fig. 1. The robots were both half-track crawlertyped tractors that are commercially available. The length, width, and height of EG453 are 3.41 m, 1.54 m and 2.38 m, respectively, and the power is 39.0 kW. The specifications of each tractor include steering control, a switch for forward and backward movements, easy-change transmission, a switch for power take off, hitch functions, engine speed set, engine stop and brake. Each robot was equipped with an RTK-GPS, an IMU, a PDA, a laser scanner, a PC and a Bluetooth. The RTK-GPS and IMU were used for navigation, the PDA received the correction signal for RTK-GPS from a GPS Earth Observation Network (GEONET) service of Japan, the Bluetooth was used to communicate with the other robot, and the PC was used as a controller. The tractor’s ECU communicates with a PC through a CAN-BUS. The output of RTK-GPS can be simplified as Gi ðt; x; y; f Þ; i 2 f1; 2g, includes current time (Gi t), position (Gi x; Gi y) and flag (Gi f ), where i represents the number of robot. The latitude and longitude are transformed to UTM coordinates since the UTM system allows the coordinate numbering system to be tied directly to the metric system. Normally, the flags of RTK-GPS include ‘‘fix”, ‘‘standalone” and ‘‘float”. When the flag is ‘‘fix”, the accuracy of RTK-GPS is 0.02 m to 0.02 m in the horizontal plane, which is sufficient to navigate the robot. However, the accuracies of the ‘‘float” status and ‘‘standalone” status can exceed 0.30 m and 1 m, respectively, much worse than the accuracy of the ‘‘fix” status. The data of IMU Ii ða; b; hÞ (roll, pitch, and yaw) are used to provide current rotation of the robot and also help to modify the GPS position during headland turn. The position and posture data were sent to the control PC at 10 Hz. The effective distance of Bluetooth is 200 m, which is sufficient for the cooperation system. The controller of the cooperation system guarantees the safety of the two robots based on Gi and Ii , but it cannot ensure safety for exterior objects such as people and animals in the field. A laser scanner is needed to ensure that the robot does not collide with the other robot or other objects. The detectable range of the laser scanner is 30 m with a 270° horizontal view, and it can provide data at frequencies up to 40 Hz, which is satisfactory for most situations. 2.2. Controller design Fig. 2 shows the structure of the communication method. The controller of the whole system can be divided into two parts: the robot’s navigator and the robot’s server/client. 2.2.1. Navigator of a robot The navigator of a robot was used to perform all functions of robot control including path planning, steer control, parse command code, transmitting control parameters to the tractor’s ECU, receiving feedback information from the tractor’s ECU, reading commands from the client/server and sending feedback information to the server/client. Fig. 3 illustrates a block diagram of control flow of the robot’s navigator. Firstly, the navigator read the navigation map and obtained position and posture data from the RTK GPS and IMU. Secondly, the steering angle (Dd) was calculated on the basis of lateral error and heading error bearing from the predetermined path by Eq. (1). Also, control parameters for the implement carried by the robot tractor, such as PTO (on/off) and hitch (up/down), were decoded from the codes embedded in the navigation map. In addition, the navigator read the command from the client/server of the leader–follower system.
2.1. Equipment
Dd ¼ ðku Du þ kd dÞ
ROBOT-1 (abbreviated as RT1, EG453, YANMAR Co., Ltd., Japan) and ROBOT-2 (abbreviated as RT2) were used as unmanned trac-
where d is the lateral error; Du is the heading error; ku and kd are control gains.
ð1Þ
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
271
Fig. 1. Platform of robot tractors.
Fig. 2. Communication structure of leader–follower system.
Fig. 3. Block diagram of control flow of navigator.
The lateral error, d, is the shortest distance from the current position of the robot to the predetermined path. The heading error, Du, is calculated from the relative angle between the desired angle and actual heading angle in the ground coordinates, where the desired angle is defined by the vector that is spanned by the orthogonal projection of the current robot position on the map trail and the other point that distance to the projection with a look ahead distance (Yang and Noguchi, 2014a). Finally, the steering angle and control commands were sent to the tractor’s ECU by the CAN-BUS. At the same time, the navigator
read the feedback information from the tractor’s ECU and sent it back to the server/client. 2.2.2. Server/client of the leader–follower system The server/client was used for cooperation and coordination of the two robots. For an easy understanding, suppose that the client was on RT1 and the server was on RT2. The client read RT1’s information through memory and sent it to the server through Bluetooth, while the server read RT2’s information through memory and received RT1’s status through Bluetooth. After calculation,
272
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 4. Flow chart of server/client.
Table 1 Format of communication message in leader–follower system. Command message format 1. 2. 3. 4. 5. 6. 7.
GPS time GPS status UTM_Easting UTM_Northing Yaw Work flag Turn status
8. 9. 10. 11. 12. 13. 14.
Danger status Engine speed Velocity PTO status Hitch status Start/stop Checksum
Fig. 6. Safety zone of robot.
Fig. 5. Method of processing IMU data.
the server sent messages to RT2’s navigator through memory and to RT1 through Bluetooth. The client received the message and transmitted it to RT1’s navigator, as shown in Fig. 4. The communication message of the leader–follower system is shown in Table 1. The calculation includes formation control, turning cooperation and safety evaluation. The frequencies of both the server and client were 5 Hz. 2.3. Fault tolerant Unlike a laboratory experiment, an experiment using a robot in the field is challenging because there are various kinds of disturbances. Disturbances can be divided into two categories: internal disturbances and external disturbances. Internal disturbances are from the system, such as GPS error, IMU error and transmission delay, and external disturbances are disturbances from the environment, such as uneven ground and animals or people in the field. Before conducting an operation in the field, the system should gain the ability to tolerate most of the disturbances.
Fig. 7. Simulation of routine of the two robots when they turn to adjacent path.
2.3.1. Internal disturbances Suppose the error of the GPS was eGi , the error of the IMU was eIi , and the error of transmission was eTi . For the leader–follower system, Gi and Ii were the data from the sensors. Define Gui and Iui as the real position and posture data that were used in the system, respectively.
Gui ¼ Gi þ f ðeGi ; eTi Þ
ð2Þ
Iui ¼ Ii þ f ðeIi ; eTi Þ
ð3Þ
where f ðeGi ; eTi Þ is the error caused by the GPS and transmission delay, and f ðeIi ; eTi Þ is the error caused by the IMU and transmission delay.
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
273
Fig. 8. Simulation of the two robots when they turn to adjacent path.
eGi can be caused by low accuracy status of the GPS. If Gi f is not ‘‘fix”, the accuracy of the RTK-GPS is more than 0.3 m. In this case, the robot stops and waits to ensure the accuracy. The server/client needs position and posture data for both robots to coordinate the two robots, and the current position of RT-i is defined as
Another factor that leads to GPS error is transmission delay. As mentioned in Section 2.2, both the GPS and IMU data were sent to the navigator, and then the server/client obtained these data from the navigator. There is a transmission delay in the radio transmission, and the predicted current position of RT-i is
8 R 0 < Gui x ¼ Gi x þ T 1 v it cosðIit hÞdt T0 T0 T1 : Gu y ¼ G y þ R T 01 v sinðI hÞdt iT0 it it iT1 T0
8 R < Gui x ¼ Gi x þ T 1 v it cosðIit hÞdt T0 T0 T1 : Gu y ¼ Gi y þ R T 1 v i sinðIi hÞdt t t iT1 T0 T0
ð4Þ
where GuiT1 is the current position data used in the system, GiT0 is the
last position data that Gi f is ‘‘fix” solution, v it is the velocity from the tractor’s ECU at the time of t, T 0 is the last time that Gi f is fix, T 01 is the estimated delay of the system, and Iit h is the yaw angle at the time of t.
ð5Þ
where T 1 is current time. Unlike the GPS, which receives signal from satellites, the two IMU from the two robots are independent from each other. In addition, the output of IMU (Ii ) do not include time. It is difficult to compare them at a given time because the clocks of the two PC
274
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 9. Simulation of velocity data for the two robots when they turn to adjacent path.
1
a ¼ ðI Ti:h I i:h Þ I Ti:h I i:G
ð8Þ
Delays of the system include sensor transmission delay (include GPS/IMU sensor delay, Bluetooth transmission delay), delay of the navigator, delay of the server/client, and delay of the tractor’s ECU. Eq. (5) was used to deal with sensor transmission delay. The navigator read the data from the sensors, and exchange data with server/client, as shown in Fig. 3. The frequency of both the navigator and server/client are 5 Hz, which is enough in this study. The delay of the tractor’s ECU (at the speed of 4.0 km/h) was 1.8 s.
Fig. 10. Trajectory of the two robots when they turn to adjacent path.
are not synchronized. Thus, the transmission delay of the IMU was ignored and the GPS position was used for fusing with the IMU data to improve the accuracy of yaw angle. Fig. 5 shows the method to modify IMU data. Given a period of time, T p , m sets of data of the IMU and GPS were stored as training examples, simplified as ðIi:hj ; Ii:Gj Þ; j 2 ½1; m, where Ii:Gj is the direction angle calculated by the GPS position.
Ii:Gj ¼ tan1
Gi yT j Gi yT j1 Gi xT j Gi xT j1
ð6Þ
where ðGi xT j ; Gi yT j Þ is the position data at the time of T j , (Gi xT j1 ; Gi yT j1 ) is the position data at the time of T j1 . Least squares regression (LSR) was used as the learning algorithm, as shown in Eq. (7), where a is a weights matrix calculated by Eq. (8) and ðI i:h ; I i:G Þ are the related inputs matrix.
I ui:h ¼ aI i:h
ð7Þ
2.3.2. External disturbances The external disturbances include uneven ground, humans or animals in the field, other vehicles or objects in the field. In general, the field is simplified as a two-dimensional plane, and the robot’s routine covers the whole plane to complete the work. However, the real field is three-dimensional considering the inclination of the robot. Thus, the position data from the RTK-GPS should be transformed by using posture data from the IMU. Position correction of the robot’s inclinations is done using the following equation:
0 1 3 3 2 m Gi x Gi X c B C 7 6 7 6 E ¼ G G Y y @n A 4 i c5 4 i 5 h Gi Z c Gi z 2
0 1 cosIi:b sinIi:h cosIi:a sinIi:b sinIi:h þ sinIi:a cosIi:h cosIi:a cosIi:h þ sinIi:a sinIi:b sinIi:h B C E ¼ @ cosIi:a sinIi:h þ sinIi:a sinIi:b cosIi:h cosIi:b cosIi:h cosIi:a sinIi:b cosIi:h sinIi:a sinIi:h A sinIi:a cosIi:b sinIi:b cosIi:a cosIi:b
ð9Þ
where Gi X c ; Gi Y c and Gi Z c are corrected positions of the GPS antenna; m and n are distance from the center of form of tractor to the GPS antenna, h is the height of the antenna from the surface of the ground in a vertical direction; and E is the transform matrix of the vehicle coordinate system to the X–Y coordinate system. A laser scanner was previously developed in our laboratory (Yang and Noguchi, 2014b). In that study, two laser scanners were used to cover all of the area surrounding the robot. The area
275
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 11. Experiment of the two robots when they turn to adjacent path.
surrounding the robot was divided into five areas: left area, right area, front area, rear area and center area. The wavelet transform (WT) method was used to detect obstacles in the front and rear areas, and the range-based obstacle detection (ROD) method was used to detect obstacles in the right, left and center areas. The distance error in that study was 0.1 m. In this article, the results obtained by using the laser scanner are not discussed.
A navigation map with six paths for each robot was made for the simulation, and each robot turned to the adjacent path. The total number of paths of the leader–follower system is 12. RT1’s navigation map differs from RT2’s navigation map. According to practical experience, when the robots conducted rotary tillage operation, the average velocity of the robots was 3.0 km/h and the maximum velocity was 3.5 km/h. The longitudinal distance between two robots, which means the distance between two robots in the path direction, was 12 m, which is calculated by Eq. (10).
3. Simulation and experiment results
dRT1betRT2 ¼ RT1 lrear þ RT2 lfront þ Before conducting the experiment, simulation was carried out to check the status of the leader–follower system in order to determine whether the two robots will collide in some situations.
1 v set t Em 2
stop
ð10Þ
where dRT1betRT2 is the set longitudinal distance between two robots, RT1 lrear is the distance from the center to rear of RT1, RT2 lfront is
276
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 12. Trajectory during headland turn when they turn to adjacent path.
the distance from center to front of RT2. v set is the set velocity of robots, and t Em stop is the stopping time of the robots. In this study, we took RT1 lrear and RT2 lfront as 3 m (large than real values, as shown in Fig. 6). v set was replaced by v max of tractor (not v max of robot), which is 15 km/h, t Em stop was 2.5 s. That is how 12 m was calculated. In addition, the minimum longitudinal distance between two robots was 8 m (3 m + 3 m + 0.5 ⁄ 1 m/s ⁄ 2.5 s). Thus, the longitudinal distance between two robots should be more than 8 m. The safety zone of each robot was simplified as a rectangle. The length and width of the robot were 4.4 m and 2.1 m, respectively, which is based on the size of the tractor and the implement carried by the robot tractor. Normally, the size of safety zone is 0.2 m larger than that of the robot. The length and width of safety zones were 4.6 m and 2.3 m, respectively, as shown in Fig. 6. The status (whether they are intersect or not) of two robots can be known by judging two safety zones. 3.1. Simulation results Fig. 7 shows the routine of the two robots. Hereafter, if the dashed line and dotted line appears in the same figure, the dashed line indicates RT1 and the dotted line indicates RT2. The robots turned to the adjacent path. The width of each path was 2 m and the width of each path set was thus 4 m. The path set, is the adjacent path of two robots which share the same path order, as shown in Fig. 7. The length of the path was 138 m. Each robot started from the related start point, and changed velocity according to the longitudinal distance between two robots. During headland turn
operation, the simulation software simulated the trajectory of the robot based on the distance between current path and next path (in this case, the distance is 4.0 m) and the robot followed the trajectory. The distance between the two robots and the distance between the two safety zones are shown in Fig. 8(a). When the work flag is 0, the robot is conducting a headland turn operation, and when the work flag is 1, the robot is conducting a work operation. We can judge whether the two robots were safe by judging these two distances. Fig. 8(b) shows the status of the two robots’ safety zones when the distance between the robots or between the safety zones is minimum. According to Fig. 8(b), the minimum distance between the two robots was 5.14 m, and the distance between the two safety zones was 1.51 m. The minimum distance between the two safety zones was 1.37 m, and the distance between the two robots was 5.53 m. The dashed line and dotted line indicates the trajectory of RT1 and RT2, respectively. The dashed rectangle and dotted rectangle indicates the safety zones of RT1 and RT2, respectively. Fig. 9 shows velocity data for the two robots. According to the velocity data, RT1 stopped and waited for 23.8 s, and RT2 stopped and waited for 25.4 s. It took 23.1 min for the two robots to complete the work. There is a difference between RT1’s waiting time and RT2’s waiting time, and the difference can be explained by the velocity data for RT2. At the beginning of each path, RT2 speeded up to catch up with RT1 since the initial longitudinal distance between the two robots was 1.2 m more than the setting value, which was 12 m. After a while, the two robots reached a balanced condition and maintained that condition. Thus, the time difference does not influence the results for efficiency. The distance between the two safety zones, which means the space distance between the two robots, was more than 1.3 m, and it can thus be concluded that it is safe for the two robots to work in this simulation. In conclusion, the leader–follower system can work without collision according to the results of simulation. And the experiment can be taken to testify the availability of the system. 3.2. Field experiment results 3.2.1. Turn to adjacent path experiment For an easy comparison with the results of simulation, the setting parameters (include velocity, path width, path length, path sequence, set longitudinal distance) in the field experiment were same as those in the simulation. Fig. 10 shows the trajectories of the two robots in the field experiment. The latitude and longitude of start point of RT1 is (43.07412111, 141.3359952). Each robot turned to the adjacent path. Considering the overlap of the implement (rotary cultivator), the width of path should be less than the width of implement. The width of each path was 2.0 m. Thus, the width between each path set was 4.0 m, the robot need to go backward during headland turn to ensure turn radius to enter the next path. Fig. 11(a) shows the distance between the two robots and the two safety zones. In this experiment, the minimum distance between the two robots was 4.09 m, and the distance between the two safety zones was 1.24 m. The minimum distance between the two safety zones was 0.8 m, and the distance between the two robots was 4.14 m. The status of the two robots’ safety zones when the distance between the robots or between the safety zones are shown in Fig. 11(b). Fig. 12 shows the status of safety zones embedded with time during turning. The x-axis is UTM-easting, the y-axis is UTMnorthing, and the z-axis is time. The rectangle was drawn every 2 s. It shows the two robots’ status during the headland turn coordination. The results for the 3-D figure show that there are
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
277
Fig. 13. Experiment of the two robots when they conduct skipping path turn method.
no rectangular zones that intersect during headland turn, which means it is a safe experiment. The minimum distance between two safety zones (0.8 m) is smaller than that shows by the simulation (1.37 m). There are two possible reasons for the difference. Firstly, there is a difference between the turning trajectory of simulation and experiment (the turning trajectory is different according to the field condition). Secondly, as mentioned in Section 2.3.1, the IMU of two robots are not synchronized, and the LSR method was used to modify the IMU data. However, the LSR method is more effective for a work operation than for a headland turn operation. From Fig. 11(b), it can be seen that only a small error of the IMU can lead to a large difference in distance between the two safety zones (a little rotates
of the rectangles will have a big effect on the distance). We can conclude that it is safe for the two robots to work in this experiment. The average error of distance between the two robots when they were working on the path was 0.13 m, and the RMS of this distance was 0.15 m. The average error of distance between the two safety zones when the two robots were working on the path was 0.14 m, and the RMS of this distance was 0.16 m. The accuracy of distance between the two robots is influenced by the GPS; however, the accuracy of distance between the two safety zones is influenced not only by the GPS but also by the IMU. That is the reason the distance error between two safety zones is larger than the distance error between two robots.
278
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 14. Trajectory during headland turn when they conduct skipping path turn method.
3.2.2. Skipping path turn experiment The waiting time of the two robots influences the work efficiency of the leader–follower system. One way to reduce the waiting time is to increase the longitudinal distance between the two robots, and another way is to use the skipping path turn method (The path order is A-1 ? D-4 ? B-2 ? E-5 ? C-3 ? F-6). It is easy to understand that increasing the longitudinal distance between the two robots decrease the waiting time. Another field experiment was taken to demonstrate that whether skipping path turn method will decrease the waiting time. The setting parameters in this experiment were the same as those in the experiment for turning to the adjacent path. The distance between the two robots and two safety zones are shown in Fig. 13(a). When the two robots skipped one path to enter the next path, the minimum distance between the two robots was 4.36 m, and the distance between the two safety zones was 1.72 m. The minimum distance between the two safety zones was 1.70 m, and the distance between the two robots was 4.38 m. When the two robots skipped two paths to enter the next path, the minimum distance between the two robots was 4.63 m, and the distance between the two safety zones was 1.38 m. The minimum distance between the two safety zones was 0.59 m, and the distance between the two robots was 5.26 m, as shown in Fig. 13(b). The distance between the two safety zones was 1.1 m less than that of a skip one path turn, but it was also a safe experiment. The trajectory of the two safety zones of the robots during a headland turn is shown in Fig. 14. The results for the 3-D figure show that there was no intersection of safety zones during the headland turn, which indicating that it is a safe experiment. The average errors of distances between the two robots and two safety zones were 0.13 m and 0.18 m, respectively. The RMS errors of distances between the two robots and two safety zones were 0.16 m and 0.22 m, respectively. These are acceptable values. According to Fig. 13(a), the distance between the two robots was unusual at 450 s and 700 s after the start of the experiment. To evaluate distance performance, data for distance must be combined with velocity data for the two robots, which are shown in Fig. 16. At 450 s after the start of the experiment, RT1 stopped since
Fig. 15. Experiment 1 of velocity data for the two robots.
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
279
Fig. 16. Experiment 2 of velocity data for the two robots.
Table 2 Waiting time of two robots during each turn.
Skip 0 path Skip one path Skip two paths
Waiting time of RT1 (s)
Waiting time of RT2 (s)
Total turning time (s)
Total time (min)
22.2
23.8
87.0
24.6
18.2
19.0
68.0
20.4
5.2
5.2
64.6
the accuracy of the GPS was not good. RT2 continued to work and the distance between two robots decreased. When the longitudinal distance between the two robots was less than 8.0 m, RT2 stopped and waited for RT1. When the accuracy of the GPS of RT1 was good enough for navigation, RT1 started working again. On the other hand, at 700 s, RT2’s GPS data were not good enough for navigation and it stopped working. RT1 continued to work and the distance between the two robots increased. When the longitudinal distance between the two robots was more than 16 m, RT1 stopped working and waited for RT2. After RT2 had started working again, RT1 also restarted. Thus, the leader–follower system can work well even if the accuracy of GPS data is not so high. 3.3. Efficiency and accuracy of the leader–follower system Fig. 15 shows velocity data for the two robots of experiment 1 (Turn to adjacent path experiment). According to the results of the experiment, RT1 stopped and waited for 22.2 s during each headland turn, 1.6 s shorter than simulation, RT2 stopped and waited for 23.8 s during each headland turn, 1.6 s shorter than simulation. The total work time (from point A to point D, as shown in Fig. 15) in the experiment was 24.6 min, almost the same as that shown by the simulation. The efficiency was 82.1 percent higher than that using a single robot, for which the total work time was 44.8 min. The total turning time, which is the time that both two robots need to complete a headland turn sequence (from point B to point C, as shown in Fig. 15), in the field experiment was 87.0 s, and the
total turning time in the simulation was 93.2 s. To evaluate the turning time, we need to combine it with velocity data for the two robots. In the simulation, the velocity of the robot tractor was 2.2 km/h; however, the real velocity of the robot tractor changed depending on the turning conditions. That is the reason why the actual turning time and actual waiting time of the robots were not the same as the simulation results. Also, the difference in velocity influenced the distances between the two robots and two safety zones. The difference between distances in the simulation and experiment might be caused by the trajectory difference between simulation and experiment, the IMU and the difference in velocity. As for the experiment 2 (skipping path turn experiment), shown in Fig. 16, when the two robots skipped one path to turn to the next path, the waiting times of RT1 and RT2 were 18.2 s and 19.0 s, respectively. When the two robots skipped two paths to turn to the next path, the waiting times of both RT1 and RT2 were 5.2 s. Table 2 shows the waiting times of the two robots using different turning methods. The total turning time by skipping one path was 68.0 s, while the total turning time by skipping two paths was 64.6 s. The total working time in the skipping path turn experiment was 20.4 min, 4.2 min less than that in the turn to the adjacent path experiment. Work efficiency was 95.1 percent higher than that using a single robot, which was 39.8 min (the single robot also used the skipping path turn method). The efficiency is limited by the length of field and the waiting time of the two robots. Fig. 17 shows the accuracy of each robot of the two experiments. Lateral error was used to evaluate the robot’s performance. Take the experiment 1 as example. The average lateral error of RT1 was 0.04 m, and the RMS of the lateral error of RT1 was 0.07 m. The average lateral error of RT2 was 0.03 m, 0.01 m less than that of RT1. The RMS of the lateral error of RT2 was 0.05 m. The average lateral error of both experiments was 0.03 m. The lateral errors of the two robots are shown in Table 3. The average lateral errors of both RT1 and RT2 were less than 0.05 m, and the RMS values were less than 0.07 m, which is high enough for agricultural field. In addition, the path width was 0.1 m less than the width of the implement carried by the robot taking into account overlapping. In the experiment, the path width was 2.0 m and the implement width was 2.1 m. From the lateral error results, we can see the
280
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 17. Lateral error of the two robots.
4. Summary
Table 3 Comparison of lateral error. Experiment 1 (turn to adjacent path experiment)
Average RMS Max Min
Experiment 2 (skipping path turn experiment)
RT1 (m)
RT2 (m)
RT1 (m)
RT2 (m)
0.04 0.07 0.13 0.35
0.03 0.05 0.22 0.15
0.02 0.07 0.15 0.22
0.03 0.06 0.11 0.25
range of RT1’s lateral error is 0.2 m, and most of the lateral errors are narrowed in 0.15 m; the range of RT2’s lateral error is 0.15 m, and most of the lateral errors are narrowed in 0.10 m. Thus, the overlap of the path (0.1 m) can cover most of the conditions, which means the two robots can cover the whole work field.
This article showed the usefulness of the leader–follower system for field work. Two robots that individually track a desired path were used for cooperation and coordination in one field. To ensure safety of the leader–follower system, each robot tractor was simplified as a rectangular zone. By judging the two safety zones, it can be ensured that the two robots do not collide with each other. A laser scanner and a bumper switch were used to avoid collisions with other objects or humans. Methods to solve the problem of interior/exterior disturbances were discussed in Section 2. The results of a field experiment showed that it is possible for the two robots to work safely in one field. According to the results of the experiment, the minimum distance between the two robots was 4.09 m, and the minimum distance between the two safety
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
zones was 0.59 m. It was a small value but is acceptable. When the robot turned to the adjacent path, the waiting time on each path of the system was 22.2 s. However, when the robot skipped one path to enter the next path, the waiting time was 18.2 s in the headland turn process. When the robot skipped two paths to enter the next path, the waiting time was 5.2 s. The total time to finish an experimental field by one robot tractor was 39.8 min; by using the skipping path turn method, the total time to finish the same field was 20.4 min, the efficiency of the system was 95.1 percent higher than that of a conventional single robot. The efficiency of the leader–follower system compared with that of a conventional single robot was limited by field length, set longitudinal distance between the two robots and set velocity. Assuming that set velocities were the same, the set longitudinal distance between the two robots clearly affected the waiting time of the system. Thus, the leader– follower system would be more effective in a large field. The method of leader–follower system can be also used in multi-robot system. In the future, we are going to develop multirobot system that several robot tractors can cooperate and coordinate to work in the same field to reduce total work time and improve work efficiency. References Cheng, T.M., Savkin, A.V., Javed, F., 2011. Decentralized control of a group of mobile robots for deployment in sweep coverage. Robot. Autonomous Syst. 59, 497–507.
281
Kise, M., Zhang, Q., Noguchi, N., 2005. An obstacle identification algorithm for a laser range finder-based obstacle detector. Trans. ASAE 48, 1269–1278. Kondo, N., Monta, M., Noguchi, N., 2004. Agricultural Robotics (1). Corona Publishing Co., Ltd. Kosuge, K., Ishikawa, J., 1994. Task-oriented control of single-master multi-slave manipulator system. Rob. Autonomous Syst. 12 (1994), 95–105. Monta, M., Kondo, N., Nakatsuka, K., 1999. Man-machine cooperative system for agricultural robot (Part 2): human sensing system in working environment of manipulator. J. Jpn. Soc. Agric. Mach. 61, 91–100. Noguchi, N., Will, J., Reid, J., Zhang, Q., 2004. Development of a master–slave robot system for farm operations. Comput. Electron. Agric. 44, 1–19. Portugal, D., Rocha, R.P., 2013. Distributed multi-robot patrol: a scalable and faulttolerant framework. Robot. Autonomous Syst. 61, 1572–1587. Wojtara, T., Nonami, K., Shao, H., Yuasa, R., Amano, S., Waterman, D., Nobumoto, Y., 2005. Hydraulic master–slave land mine clearance robot hand controlled by pulse modulation. Mechatronics 15, 589–609. Yang, L., Noguchi, N., 2012. Human detection for a robot tractor using omnidirectional stereo vision. Comput. Electron. Agric. 89, 116–125. Yang, L., Noguchi, N., 2014a. Development of a wheel-type robot tractor and its utilization. In: Proceedings of the 19th World Congress of the IFAC. Cape Town, South Africa. pp. 11571–11576. http://dx.doi.org/10.3182/20140824-6-ZA1003.00952. Yang, L., Noguchi, N., 2014b. An active safety system using two laser scanners for a robot tractor. In: Proceedings of the 19th World Congress of the IFAC. Cape Town, South Africa. pp. 11577–11582. http://dx.doi.org/10.3182/20140824-6ZA-1003.00953. Yang, Y., Souissi, S., Defago, X., Takizawa, M., 2011. Fault-tolerant flocking for a group of autonomous mobile robots. J. Syst. Software 84, 29–36. Zajac, M., 2014. Online fault detection of a mobile robot with a parallelized particle filter. Neurocomputing 126, 151–165. Zhang, X., Geimer, M., Noack, P.O., Grandl, L., 2010. Development of an intelligent mastereslave system between agricultural vehicles. Proceedings of the 2010 IEEE Intelligent Vehicles Symposium. Institute of Electrical and Electronics Engineers, San Diego, CA, USA, pp. 250–255.
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag
Original papers
Leader–follower system using two robot tractors to improve work efficiency Chi Zhang, Noboru Noguchi ⇑, Liangliang Yang Laboratory of Vehicle Robotics, Graduate School of Agriculture, Hokkaido University, Kita-9, Nishi-9, Kita-ku, Sapporo 060-8589, Japan
a r t i c l e
i n f o
Article history: Received 15 August 2015 Received in revised form 26 December 2015 Accepted 27 December 2015
Keywords: Leader–follower system Robot tractor Fault tolerant Work efficiency RTK-GPS
a b s t r a c t Two robot tractors were used in a leader–follower system for agricultural field work. Each of the robots is fully independent and can conduct field work alone. They can also work together to form a certain spatial arrangement during the operation. During the headland turn, to make the best use of headland, the two robots coordinate to turn to next path and do not keep the spatial arrangement. Each robot is simplified as a rectangular zone, and the two robots cooperate and coordinate to turn to the next path without collision. This system is designed for practical application, and the system gains the ability to tolerate most of the disturbances in a real field. Fault tolerant methods in accordance with agricultural work are illustrated to solve the common disturbances from the GPS and the IMU. Field experiments were conducted to determine the effectiveness of the system. The results of the experiments showed that the two robot tractors can work safely together to complete the field work. The average lateral error of the navigation system of the robots was less than 0.04 m, and the efficiency of the leader–follower system was improved by 95.1% compared with that of a conventional single robot. Ó 2016 Elsevier B.V. All rights reserved.
1. Introduction The coordination of plural robots has undergone rapid development in recent years. Cooperation of simple robots has many advantages such as flexibility, large application domain and capability to accomplish complex missions that cannot be accomplished by a single unit. For the use of two robots, most researchers have focused on a master–slave robot system. Kosuge and Ishikawa (1994) developed a task-oriented control algorithm for coordinated motion control of a single-master multi-slave manipulator system that consists of a master arm with six degrees of freedom and slave arms that each have six or more degrees of freedom, and they showed that the system has improved load capacity, rigidity and dexterity. Wojtara et al. (2005) reported a master–slave system consisting of a data glove and a hydraulic robot hand for removal of landmines and visible obstacles. Cheng et al. (2011) developed a network of autonomous mobile robots to accomplish the sweep coverage. They proposed a decentralized control algorithm for the robots. Robotic technology is also widely used for field tasks in production agriculture. In the agriculture sector, robot tractors have been developed for conducting field work in order to reduce work strength and work time of farmers. ⇑ Corresponding author. E-mail address: [email protected] (N. Noguchi). http://dx.doi.org/10.1016/j.compag.2015.12.015 0168-1699/Ó 2016 Elsevier B.V. All rights reserved.
Researchers have also developed master–slave robot systems to conduct field work. Noguchi et al. (2004) proposed GOTO and FOLLOW algorithms for a master–slave system. The master controls the slave to follow a parallel path at a given distance and angle from the master or to go to a certain point along any path as long as it does not collide with the master. Zhang et al. (2010) developed an intelligent master–slave system that enables a semiautonomous agricultural vehicle (slave) to follow a master with a given lateral and longitudinal offset. They used a state space dynamic model and a proportional-derivative controller with state feedback and disturbance feed-forward for the tractor. However, the slave robots in both Noguchi’s system and Zhang’s system have less decision-making capability than that of a master robot. Fault tolerance is important for robots operating in dangerous or complex environments since robot failures are likely to occur. Yang et al. (2011) developed a fault-tolerant flocking algorithm for a group of autonomous mobile robots. The algorithm includes three parts: ranking assignment, failure detector and flocking algorithm. In their study, the robots group moves in formation and preserves the formation. Zajac (2014) used a parallelizing particle filtering-based approach combined with the negative loglikelihood test for a fault detection task. It is aimed to improve the efficiency, estimation error and execution time of online setup. Portugal and Rocha (2013) reported two distributed solutions that make decisions using a Bayesian-based mathematical formalism to
270
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
coordinate a team of mobile robots. One of the advantages of this framework is that it leads to a scalable and fault-tolerant solution. Agricultural field work is needed to be done quickly in some situations. For example, sowing or harvesting before the rainy season. One robot is not enough to finish the field work in a limited time. In this study, we developed a system with two robot tractors (called a leader–follower system) to conduct field work. Unlike the master– slave robot system, each of the robot tractors in the leader– follower system is fully independent and can conduct field work alone. The two robots can also work together by forming a spatial arrangement during the operation to reduce total work time. In addition, to make the best use of headland, the two robot tractors do not need to keep a certain arrangement during a headland turn. Each robot uses its own turn method and the trajectory of the robots may collide. It is one of the differences between the current article and the previous articles (Noguchi et al., 2004; Zhang et al., 2010; Cheng et al., 2011). In this article, we focus on the usefulness of the leader–follower system that is fault-tolerant and has improved efficiency compared with a conventional single robot. For example, for a robotic combine harvester harvesting in the field, an on-the-go unloading system with a transport that moves the harvested products to collection positions helps improve harvesting efficiency since the harvester does not have to stop. Another example is that the two robots doing tillage together to extend the work width to reduce the total work time. In general, experiment differs from simulation due to the existence of disturbance which from interior or exterior of the system. The system should gain the ability to tolerate most of the disturbances in order to work safely. It is challenge to solve the problem of external disturbances including humans or animals in the field and other vehicles. Researchers have made many efforts to solve this problem. Monta et al. (1999) used four ultrasonic sensors and four infrared sensors in a grape harvesting robot for detecting humans. Kise et al. (2005) used a laser scanner in front of a robot tractor to detect and locate other vehicles working in the same field. Yang and Noguchi (2012) used an omnidirectional stereo vision sensor in a robot tractor to detect humans. As for touching sensors, Kondo et al. (2004) used a bumper switch as a robot safety system mounted in front of a robot tractor, and the robot stops if something touches the bumper. In the present study, a laser scanner and a bumper switch were used as safety sensors to ensure safety of the robots in the field. The reasons that use laser scanner are the large detection range (30 m, 270°) and high frequency (40 Hz). If the laser fails to detect the objects, the bumper switch is used to ensure the robot to be stopped. Two robots can coordinate in several ways and each of them has a different efficiency than that of a conventional single robot. The improved efficiency of common coordination method, such as the two robots kept a constant formation during operation is discussed in this study. For headland turning, the two robots coordinate and turn together if they are in a safe condition. A skipping path turn method is proposed for reducing waiting time of the two robots in order to increase the efficiency of the system. The rest of this article is structured as follows. The equipment, control design and fault-tolerant method used in the system are presented in Section 2. Results of the experiments using the leader–follower system are presented in Section 3. Conclusions are given in Section 4.
2. Method
tors, as shown in Fig. 1. The robots were both half-track crawlertyped tractors that are commercially available. The length, width, and height of EG453 are 3.41 m, 1.54 m and 2.38 m, respectively, and the power is 39.0 kW. The specifications of each tractor include steering control, a switch for forward and backward movements, easy-change transmission, a switch for power take off, hitch functions, engine speed set, engine stop and brake. Each robot was equipped with an RTK-GPS, an IMU, a PDA, a laser scanner, a PC and a Bluetooth. The RTK-GPS and IMU were used for navigation, the PDA received the correction signal for RTK-GPS from a GPS Earth Observation Network (GEONET) service of Japan, the Bluetooth was used to communicate with the other robot, and the PC was used as a controller. The tractor’s ECU communicates with a PC through a CAN-BUS. The output of RTK-GPS can be simplified as Gi ðt; x; y; f Þ; i 2 f1; 2g, includes current time (Gi t), position (Gi x; Gi y) and flag (Gi f ), where i represents the number of robot. The latitude and longitude are transformed to UTM coordinates since the UTM system allows the coordinate numbering system to be tied directly to the metric system. Normally, the flags of RTK-GPS include ‘‘fix”, ‘‘standalone” and ‘‘float”. When the flag is ‘‘fix”, the accuracy of RTK-GPS is 0.02 m to 0.02 m in the horizontal plane, which is sufficient to navigate the robot. However, the accuracies of the ‘‘float” status and ‘‘standalone” status can exceed 0.30 m and 1 m, respectively, much worse than the accuracy of the ‘‘fix” status. The data of IMU Ii ða; b; hÞ (roll, pitch, and yaw) are used to provide current rotation of the robot and also help to modify the GPS position during headland turn. The position and posture data were sent to the control PC at 10 Hz. The effective distance of Bluetooth is 200 m, which is sufficient for the cooperation system. The controller of the cooperation system guarantees the safety of the two robots based on Gi and Ii , but it cannot ensure safety for exterior objects such as people and animals in the field. A laser scanner is needed to ensure that the robot does not collide with the other robot or other objects. The detectable range of the laser scanner is 30 m with a 270° horizontal view, and it can provide data at frequencies up to 40 Hz, which is satisfactory for most situations. 2.2. Controller design Fig. 2 shows the structure of the communication method. The controller of the whole system can be divided into two parts: the robot’s navigator and the robot’s server/client. 2.2.1. Navigator of a robot The navigator of a robot was used to perform all functions of robot control including path planning, steer control, parse command code, transmitting control parameters to the tractor’s ECU, receiving feedback information from the tractor’s ECU, reading commands from the client/server and sending feedback information to the server/client. Fig. 3 illustrates a block diagram of control flow of the robot’s navigator. Firstly, the navigator read the navigation map and obtained position and posture data from the RTK GPS and IMU. Secondly, the steering angle (Dd) was calculated on the basis of lateral error and heading error bearing from the predetermined path by Eq. (1). Also, control parameters for the implement carried by the robot tractor, such as PTO (on/off) and hitch (up/down), were decoded from the codes embedded in the navigation map. In addition, the navigator read the command from the client/server of the leader–follower system.
2.1. Equipment
Dd ¼ ðku Du þ kd dÞ
ROBOT-1 (abbreviated as RT1, EG453, YANMAR Co., Ltd., Japan) and ROBOT-2 (abbreviated as RT2) were used as unmanned trac-
where d is the lateral error; Du is the heading error; ku and kd are control gains.
ð1Þ
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
271
Fig. 1. Platform of robot tractors.
Fig. 2. Communication structure of leader–follower system.
Fig. 3. Block diagram of control flow of navigator.
The lateral error, d, is the shortest distance from the current position of the robot to the predetermined path. The heading error, Du, is calculated from the relative angle between the desired angle and actual heading angle in the ground coordinates, where the desired angle is defined by the vector that is spanned by the orthogonal projection of the current robot position on the map trail and the other point that distance to the projection with a look ahead distance (Yang and Noguchi, 2014a). Finally, the steering angle and control commands were sent to the tractor’s ECU by the CAN-BUS. At the same time, the navigator
read the feedback information from the tractor’s ECU and sent it back to the server/client. 2.2.2. Server/client of the leader–follower system The server/client was used for cooperation and coordination of the two robots. For an easy understanding, suppose that the client was on RT1 and the server was on RT2. The client read RT1’s information through memory and sent it to the server through Bluetooth, while the server read RT2’s information through memory and received RT1’s status through Bluetooth. After calculation,
272
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 4. Flow chart of server/client.
Table 1 Format of communication message in leader–follower system. Command message format 1. 2. 3. 4. 5. 6. 7.
GPS time GPS status UTM_Easting UTM_Northing Yaw Work flag Turn status
8. 9. 10. 11. 12. 13. 14.
Danger status Engine speed Velocity PTO status Hitch status Start/stop Checksum
Fig. 6. Safety zone of robot.
Fig. 5. Method of processing IMU data.
the server sent messages to RT2’s navigator through memory and to RT1 through Bluetooth. The client received the message and transmitted it to RT1’s navigator, as shown in Fig. 4. The communication message of the leader–follower system is shown in Table 1. The calculation includes formation control, turning cooperation and safety evaluation. The frequencies of both the server and client were 5 Hz. 2.3. Fault tolerant Unlike a laboratory experiment, an experiment using a robot in the field is challenging because there are various kinds of disturbances. Disturbances can be divided into two categories: internal disturbances and external disturbances. Internal disturbances are from the system, such as GPS error, IMU error and transmission delay, and external disturbances are disturbances from the environment, such as uneven ground and animals or people in the field. Before conducting an operation in the field, the system should gain the ability to tolerate most of the disturbances.
Fig. 7. Simulation of routine of the two robots when they turn to adjacent path.
2.3.1. Internal disturbances Suppose the error of the GPS was eGi , the error of the IMU was eIi , and the error of transmission was eTi . For the leader–follower system, Gi and Ii were the data from the sensors. Define Gui and Iui as the real position and posture data that were used in the system, respectively.
Gui ¼ Gi þ f ðeGi ; eTi Þ
ð2Þ
Iui ¼ Ii þ f ðeIi ; eTi Þ
ð3Þ
where f ðeGi ; eTi Þ is the error caused by the GPS and transmission delay, and f ðeIi ; eTi Þ is the error caused by the IMU and transmission delay.
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
273
Fig. 8. Simulation of the two robots when they turn to adjacent path.
eGi can be caused by low accuracy status of the GPS. If Gi f is not ‘‘fix”, the accuracy of the RTK-GPS is more than 0.3 m. In this case, the robot stops and waits to ensure the accuracy. The server/client needs position and posture data for both robots to coordinate the two robots, and the current position of RT-i is defined as
Another factor that leads to GPS error is transmission delay. As mentioned in Section 2.2, both the GPS and IMU data were sent to the navigator, and then the server/client obtained these data from the navigator. There is a transmission delay in the radio transmission, and the predicted current position of RT-i is
8 R 0 < Gui x ¼ Gi x þ T 1 v it cosðIit hÞdt T0 T0 T1 : Gu y ¼ G y þ R T 01 v sinðI hÞdt iT0 it it iT1 T0
8 R < Gui x ¼ Gi x þ T 1 v it cosðIit hÞdt T0 T0 T1 : Gu y ¼ Gi y þ R T 1 v i sinðIi hÞdt t t iT1 T0 T0
ð4Þ
where GuiT1 is the current position data used in the system, GiT0 is the
last position data that Gi f is ‘‘fix” solution, v it is the velocity from the tractor’s ECU at the time of t, T 0 is the last time that Gi f is fix, T 01 is the estimated delay of the system, and Iit h is the yaw angle at the time of t.
ð5Þ
where T 1 is current time. Unlike the GPS, which receives signal from satellites, the two IMU from the two robots are independent from each other. In addition, the output of IMU (Ii ) do not include time. It is difficult to compare them at a given time because the clocks of the two PC
274
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 9. Simulation of velocity data for the two robots when they turn to adjacent path.
1
a ¼ ðI Ti:h I i:h Þ I Ti:h I i:G
ð8Þ
Delays of the system include sensor transmission delay (include GPS/IMU sensor delay, Bluetooth transmission delay), delay of the navigator, delay of the server/client, and delay of the tractor’s ECU. Eq. (5) was used to deal with sensor transmission delay. The navigator read the data from the sensors, and exchange data with server/client, as shown in Fig. 3. The frequency of both the navigator and server/client are 5 Hz, which is enough in this study. The delay of the tractor’s ECU (at the speed of 4.0 km/h) was 1.8 s.
Fig. 10. Trajectory of the two robots when they turn to adjacent path.
are not synchronized. Thus, the transmission delay of the IMU was ignored and the GPS position was used for fusing with the IMU data to improve the accuracy of yaw angle. Fig. 5 shows the method to modify IMU data. Given a period of time, T p , m sets of data of the IMU and GPS were stored as training examples, simplified as ðIi:hj ; Ii:Gj Þ; j 2 ½1; m, where Ii:Gj is the direction angle calculated by the GPS position.
Ii:Gj ¼ tan1
Gi yT j Gi yT j1 Gi xT j Gi xT j1
ð6Þ
where ðGi xT j ; Gi yT j Þ is the position data at the time of T j , (Gi xT j1 ; Gi yT j1 ) is the position data at the time of T j1 . Least squares regression (LSR) was used as the learning algorithm, as shown in Eq. (7), where a is a weights matrix calculated by Eq. (8) and ðI i:h ; I i:G Þ are the related inputs matrix.
I ui:h ¼ aI i:h
ð7Þ
2.3.2. External disturbances The external disturbances include uneven ground, humans or animals in the field, other vehicles or objects in the field. In general, the field is simplified as a two-dimensional plane, and the robot’s routine covers the whole plane to complete the work. However, the real field is three-dimensional considering the inclination of the robot. Thus, the position data from the RTK-GPS should be transformed by using posture data from the IMU. Position correction of the robot’s inclinations is done using the following equation:
0 1 3 3 2 m Gi x Gi X c B C 7 6 7 6 E ¼ G G Y y @n A 4 i c5 4 i 5 h Gi Z c Gi z 2
0 1 cosIi:b sinIi:h cosIi:a sinIi:b sinIi:h þ sinIi:a cosIi:h cosIi:a cosIi:h þ sinIi:a sinIi:b sinIi:h B C E ¼ @ cosIi:a sinIi:h þ sinIi:a sinIi:b cosIi:h cosIi:b cosIi:h cosIi:a sinIi:b cosIi:h sinIi:a sinIi:h A sinIi:a cosIi:b sinIi:b cosIi:a cosIi:b
ð9Þ
where Gi X c ; Gi Y c and Gi Z c are corrected positions of the GPS antenna; m and n are distance from the center of form of tractor to the GPS antenna, h is the height of the antenna from the surface of the ground in a vertical direction; and E is the transform matrix of the vehicle coordinate system to the X–Y coordinate system. A laser scanner was previously developed in our laboratory (Yang and Noguchi, 2014b). In that study, two laser scanners were used to cover all of the area surrounding the robot. The area
275
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 11. Experiment of the two robots when they turn to adjacent path.
surrounding the robot was divided into five areas: left area, right area, front area, rear area and center area. The wavelet transform (WT) method was used to detect obstacles in the front and rear areas, and the range-based obstacle detection (ROD) method was used to detect obstacles in the right, left and center areas. The distance error in that study was 0.1 m. In this article, the results obtained by using the laser scanner are not discussed.
A navigation map with six paths for each robot was made for the simulation, and each robot turned to the adjacent path. The total number of paths of the leader–follower system is 12. RT1’s navigation map differs from RT2’s navigation map. According to practical experience, when the robots conducted rotary tillage operation, the average velocity of the robots was 3.0 km/h and the maximum velocity was 3.5 km/h. The longitudinal distance between two robots, which means the distance between two robots in the path direction, was 12 m, which is calculated by Eq. (10).
3. Simulation and experiment results
dRT1betRT2 ¼ RT1 lrear þ RT2 lfront þ Before conducting the experiment, simulation was carried out to check the status of the leader–follower system in order to determine whether the two robots will collide in some situations.
1 v set t Em 2
stop
ð10Þ
where dRT1betRT2 is the set longitudinal distance between two robots, RT1 lrear is the distance from the center to rear of RT1, RT2 lfront is
276
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 12. Trajectory during headland turn when they turn to adjacent path.
the distance from center to front of RT2. v set is the set velocity of robots, and t Em stop is the stopping time of the robots. In this study, we took RT1 lrear and RT2 lfront as 3 m (large than real values, as shown in Fig. 6). v set was replaced by v max of tractor (not v max of robot), which is 15 km/h, t Em stop was 2.5 s. That is how 12 m was calculated. In addition, the minimum longitudinal distance between two robots was 8 m (3 m + 3 m + 0.5 ⁄ 1 m/s ⁄ 2.5 s). Thus, the longitudinal distance between two robots should be more than 8 m. The safety zone of each robot was simplified as a rectangle. The length and width of the robot were 4.4 m and 2.1 m, respectively, which is based on the size of the tractor and the implement carried by the robot tractor. Normally, the size of safety zone is 0.2 m larger than that of the robot. The length and width of safety zones were 4.6 m and 2.3 m, respectively, as shown in Fig. 6. The status (whether they are intersect or not) of two robots can be known by judging two safety zones. 3.1. Simulation results Fig. 7 shows the routine of the two robots. Hereafter, if the dashed line and dotted line appears in the same figure, the dashed line indicates RT1 and the dotted line indicates RT2. The robots turned to the adjacent path. The width of each path was 2 m and the width of each path set was thus 4 m. The path set, is the adjacent path of two robots which share the same path order, as shown in Fig. 7. The length of the path was 138 m. Each robot started from the related start point, and changed velocity according to the longitudinal distance between two robots. During headland turn
operation, the simulation software simulated the trajectory of the robot based on the distance between current path and next path (in this case, the distance is 4.0 m) and the robot followed the trajectory. The distance between the two robots and the distance between the two safety zones are shown in Fig. 8(a). When the work flag is 0, the robot is conducting a headland turn operation, and when the work flag is 1, the robot is conducting a work operation. We can judge whether the two robots were safe by judging these two distances. Fig. 8(b) shows the status of the two robots’ safety zones when the distance between the robots or between the safety zones is minimum. According to Fig. 8(b), the minimum distance between the two robots was 5.14 m, and the distance between the two safety zones was 1.51 m. The minimum distance between the two safety zones was 1.37 m, and the distance between the two robots was 5.53 m. The dashed line and dotted line indicates the trajectory of RT1 and RT2, respectively. The dashed rectangle and dotted rectangle indicates the safety zones of RT1 and RT2, respectively. Fig. 9 shows velocity data for the two robots. According to the velocity data, RT1 stopped and waited for 23.8 s, and RT2 stopped and waited for 25.4 s. It took 23.1 min for the two robots to complete the work. There is a difference between RT1’s waiting time and RT2’s waiting time, and the difference can be explained by the velocity data for RT2. At the beginning of each path, RT2 speeded up to catch up with RT1 since the initial longitudinal distance between the two robots was 1.2 m more than the setting value, which was 12 m. After a while, the two robots reached a balanced condition and maintained that condition. Thus, the time difference does not influence the results for efficiency. The distance between the two safety zones, which means the space distance between the two robots, was more than 1.3 m, and it can thus be concluded that it is safe for the two robots to work in this simulation. In conclusion, the leader–follower system can work without collision according to the results of simulation. And the experiment can be taken to testify the availability of the system. 3.2. Field experiment results 3.2.1. Turn to adjacent path experiment For an easy comparison with the results of simulation, the setting parameters (include velocity, path width, path length, path sequence, set longitudinal distance) in the field experiment were same as those in the simulation. Fig. 10 shows the trajectories of the two robots in the field experiment. The latitude and longitude of start point of RT1 is (43.07412111, 141.3359952). Each robot turned to the adjacent path. Considering the overlap of the implement (rotary cultivator), the width of path should be less than the width of implement. The width of each path was 2.0 m. Thus, the width between each path set was 4.0 m, the robot need to go backward during headland turn to ensure turn radius to enter the next path. Fig. 11(a) shows the distance between the two robots and the two safety zones. In this experiment, the minimum distance between the two robots was 4.09 m, and the distance between the two safety zones was 1.24 m. The minimum distance between the two safety zones was 0.8 m, and the distance between the two robots was 4.14 m. The status of the two robots’ safety zones when the distance between the robots or between the safety zones are shown in Fig. 11(b). Fig. 12 shows the status of safety zones embedded with time during turning. The x-axis is UTM-easting, the y-axis is UTMnorthing, and the z-axis is time. The rectangle was drawn every 2 s. It shows the two robots’ status during the headland turn coordination. The results for the 3-D figure show that there are
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
277
Fig. 13. Experiment of the two robots when they conduct skipping path turn method.
no rectangular zones that intersect during headland turn, which means it is a safe experiment. The minimum distance between two safety zones (0.8 m) is smaller than that shows by the simulation (1.37 m). There are two possible reasons for the difference. Firstly, there is a difference between the turning trajectory of simulation and experiment (the turning trajectory is different according to the field condition). Secondly, as mentioned in Section 2.3.1, the IMU of two robots are not synchronized, and the LSR method was used to modify the IMU data. However, the LSR method is more effective for a work operation than for a headland turn operation. From Fig. 11(b), it can be seen that only a small error of the IMU can lead to a large difference in distance between the two safety zones (a little rotates
of the rectangles will have a big effect on the distance). We can conclude that it is safe for the two robots to work in this experiment. The average error of distance between the two robots when they were working on the path was 0.13 m, and the RMS of this distance was 0.15 m. The average error of distance between the two safety zones when the two robots were working on the path was 0.14 m, and the RMS of this distance was 0.16 m. The accuracy of distance between the two robots is influenced by the GPS; however, the accuracy of distance between the two safety zones is influenced not only by the GPS but also by the IMU. That is the reason the distance error between two safety zones is larger than the distance error between two robots.
278
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 14. Trajectory during headland turn when they conduct skipping path turn method.
3.2.2. Skipping path turn experiment The waiting time of the two robots influences the work efficiency of the leader–follower system. One way to reduce the waiting time is to increase the longitudinal distance between the two robots, and another way is to use the skipping path turn method (The path order is A-1 ? D-4 ? B-2 ? E-5 ? C-3 ? F-6). It is easy to understand that increasing the longitudinal distance between the two robots decrease the waiting time. Another field experiment was taken to demonstrate that whether skipping path turn method will decrease the waiting time. The setting parameters in this experiment were the same as those in the experiment for turning to the adjacent path. The distance between the two robots and two safety zones are shown in Fig. 13(a). When the two robots skipped one path to enter the next path, the minimum distance between the two robots was 4.36 m, and the distance between the two safety zones was 1.72 m. The minimum distance between the two safety zones was 1.70 m, and the distance between the two robots was 4.38 m. When the two robots skipped two paths to enter the next path, the minimum distance between the two robots was 4.63 m, and the distance between the two safety zones was 1.38 m. The minimum distance between the two safety zones was 0.59 m, and the distance between the two robots was 5.26 m, as shown in Fig. 13(b). The distance between the two safety zones was 1.1 m less than that of a skip one path turn, but it was also a safe experiment. The trajectory of the two safety zones of the robots during a headland turn is shown in Fig. 14. The results for the 3-D figure show that there was no intersection of safety zones during the headland turn, which indicating that it is a safe experiment. The average errors of distances between the two robots and two safety zones were 0.13 m and 0.18 m, respectively. The RMS errors of distances between the two robots and two safety zones were 0.16 m and 0.22 m, respectively. These are acceptable values. According to Fig. 13(a), the distance between the two robots was unusual at 450 s and 700 s after the start of the experiment. To evaluate distance performance, data for distance must be combined with velocity data for the two robots, which are shown in Fig. 16. At 450 s after the start of the experiment, RT1 stopped since
Fig. 15. Experiment 1 of velocity data for the two robots.
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
279
Fig. 16. Experiment 2 of velocity data for the two robots.
Table 2 Waiting time of two robots during each turn.
Skip 0 path Skip one path Skip two paths
Waiting time of RT1 (s)
Waiting time of RT2 (s)
Total turning time (s)
Total time (min)
22.2
23.8
87.0
24.6
18.2
19.0
68.0
20.4
5.2
5.2
64.6
the accuracy of the GPS was not good. RT2 continued to work and the distance between two robots decreased. When the longitudinal distance between the two robots was less than 8.0 m, RT2 stopped and waited for RT1. When the accuracy of the GPS of RT1 was good enough for navigation, RT1 started working again. On the other hand, at 700 s, RT2’s GPS data were not good enough for navigation and it stopped working. RT1 continued to work and the distance between the two robots increased. When the longitudinal distance between the two robots was more than 16 m, RT1 stopped working and waited for RT2. After RT2 had started working again, RT1 also restarted. Thus, the leader–follower system can work well even if the accuracy of GPS data is not so high. 3.3. Efficiency and accuracy of the leader–follower system Fig. 15 shows velocity data for the two robots of experiment 1 (Turn to adjacent path experiment). According to the results of the experiment, RT1 stopped and waited for 22.2 s during each headland turn, 1.6 s shorter than simulation, RT2 stopped and waited for 23.8 s during each headland turn, 1.6 s shorter than simulation. The total work time (from point A to point D, as shown in Fig. 15) in the experiment was 24.6 min, almost the same as that shown by the simulation. The efficiency was 82.1 percent higher than that using a single robot, for which the total work time was 44.8 min. The total turning time, which is the time that both two robots need to complete a headland turn sequence (from point B to point C, as shown in Fig. 15), in the field experiment was 87.0 s, and the
total turning time in the simulation was 93.2 s. To evaluate the turning time, we need to combine it with velocity data for the two robots. In the simulation, the velocity of the robot tractor was 2.2 km/h; however, the real velocity of the robot tractor changed depending on the turning conditions. That is the reason why the actual turning time and actual waiting time of the robots were not the same as the simulation results. Also, the difference in velocity influenced the distances between the two robots and two safety zones. The difference between distances in the simulation and experiment might be caused by the trajectory difference between simulation and experiment, the IMU and the difference in velocity. As for the experiment 2 (skipping path turn experiment), shown in Fig. 16, when the two robots skipped one path to turn to the next path, the waiting times of RT1 and RT2 were 18.2 s and 19.0 s, respectively. When the two robots skipped two paths to turn to the next path, the waiting times of both RT1 and RT2 were 5.2 s. Table 2 shows the waiting times of the two robots using different turning methods. The total turning time by skipping one path was 68.0 s, while the total turning time by skipping two paths was 64.6 s. The total working time in the skipping path turn experiment was 20.4 min, 4.2 min less than that in the turn to the adjacent path experiment. Work efficiency was 95.1 percent higher than that using a single robot, which was 39.8 min (the single robot also used the skipping path turn method). The efficiency is limited by the length of field and the waiting time of the two robots. Fig. 17 shows the accuracy of each robot of the two experiments. Lateral error was used to evaluate the robot’s performance. Take the experiment 1 as example. The average lateral error of RT1 was 0.04 m, and the RMS of the lateral error of RT1 was 0.07 m. The average lateral error of RT2 was 0.03 m, 0.01 m less than that of RT1. The RMS of the lateral error of RT2 was 0.05 m. The average lateral error of both experiments was 0.03 m. The lateral errors of the two robots are shown in Table 3. The average lateral errors of both RT1 and RT2 were less than 0.05 m, and the RMS values were less than 0.07 m, which is high enough for agricultural field. In addition, the path width was 0.1 m less than the width of the implement carried by the robot taking into account overlapping. In the experiment, the path width was 2.0 m and the implement width was 2.1 m. From the lateral error results, we can see the
280
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
Fig. 17. Lateral error of the two robots.
4. Summary
Table 3 Comparison of lateral error. Experiment 1 (turn to adjacent path experiment)
Average RMS Max Min
Experiment 2 (skipping path turn experiment)
RT1 (m)
RT2 (m)
RT1 (m)
RT2 (m)
0.04 0.07 0.13 0.35
0.03 0.05 0.22 0.15
0.02 0.07 0.15 0.22
0.03 0.06 0.11 0.25
range of RT1’s lateral error is 0.2 m, and most of the lateral errors are narrowed in 0.15 m; the range of RT2’s lateral error is 0.15 m, and most of the lateral errors are narrowed in 0.10 m. Thus, the overlap of the path (0.1 m) can cover most of the conditions, which means the two robots can cover the whole work field.
This article showed the usefulness of the leader–follower system for field work. Two robots that individually track a desired path were used for cooperation and coordination in one field. To ensure safety of the leader–follower system, each robot tractor was simplified as a rectangular zone. By judging the two safety zones, it can be ensured that the two robots do not collide with each other. A laser scanner and a bumper switch were used to avoid collisions with other objects or humans. Methods to solve the problem of interior/exterior disturbances were discussed in Section 2. The results of a field experiment showed that it is possible for the two robots to work safely in one field. According to the results of the experiment, the minimum distance between the two robots was 4.09 m, and the minimum distance between the two safety
C. Zhang et al. / Computers and Electronics in Agriculture 121 (2016) 269–281
zones was 0.59 m. It was a small value but is acceptable. When the robot turned to the adjacent path, the waiting time on each path of the system was 22.2 s. However, when the robot skipped one path to enter the next path, the waiting time was 18.2 s in the headland turn process. When the robot skipped two paths to enter the next path, the waiting time was 5.2 s. The total time to finish an experimental field by one robot tractor was 39.8 min; by using the skipping path turn method, the total time to finish the same field was 20.4 min, the efficiency of the system was 95.1 percent higher than that of a conventional single robot. The efficiency of the leader–follower system compared with that of a conventional single robot was limited by field length, set longitudinal distance between the two robots and set velocity. Assuming that set velocities were the same, the set longitudinal distance between the two robots clearly affected the waiting time of the system. Thus, the leader– follower system would be more effective in a large field. The method of leader–follower system can be also used in multi-robot system. In the future, we are going to develop multirobot system that several robot tractors can cooperate and coordinate to work in the same field to reduce total work time and improve work efficiency. References Cheng, T.M., Savkin, A.V., Javed, F., 2011. Decentralized control of a group of mobile robots for deployment in sweep coverage. Robot. Autonomous Syst. 59, 497–507.
281
Kise, M., Zhang, Q., Noguchi, N., 2005. An obstacle identification algorithm for a laser range finder-based obstacle detector. Trans. ASAE 48, 1269–1278. Kondo, N., Monta, M., Noguchi, N., 2004. Agricultural Robotics (1). Corona Publishing Co., Ltd. Kosuge, K., Ishikawa, J., 1994. Task-oriented control of single-master multi-slave manipulator system. Rob. Autonomous Syst. 12 (1994), 95–105. Monta, M., Kondo, N., Nakatsuka, K., 1999. Man-machine cooperative system for agricultural robot (Part 2): human sensing system in working environment of manipulator. J. Jpn. Soc. Agric. Mach. 61, 91–100. Noguchi, N., Will, J., Reid, J., Zhang, Q., 2004. Development of a master–slave robot system for farm operations. Comput. Electron. Agric. 44, 1–19. Portugal, D., Rocha, R.P., 2013. Distributed multi-robot patrol: a scalable and faulttolerant framework. Robot. Autonomous Syst. 61, 1572–1587. Wojtara, T., Nonami, K., Shao, H., Yuasa, R., Amano, S., Waterman, D., Nobumoto, Y., 2005. Hydraulic master–slave land mine clearance robot hand controlled by pulse modulation. Mechatronics 15, 589–609. Yang, L., Noguchi, N., 2012. Human detection for a robot tractor using omnidirectional stereo vision. Comput. Electron. Agric. 89, 116–125. Yang, L., Noguchi, N., 2014a. Development of a wheel-type robot tractor and its utilization. In: Proceedings of the 19th World Congress of the IFAC. Cape Town, South Africa. pp. 11571–11576. http://dx.doi.org/10.3182/20140824-6-ZA1003.00952. Yang, L., Noguchi, N., 2014b. An active safety system using two laser scanners for a robot tractor. In: Proceedings of the 19th World Congress of the IFAC. Cape Town, South Africa. pp. 11577–11582. http://dx.doi.org/10.3182/20140824-6ZA-1003.00953. Yang, Y., Souissi, S., Defago, X., Takizawa, M., 2011. Fault-tolerant flocking for a group of autonomous mobile robots. J. Syst. Software 84, 29–36. Zajac, M., 2014. Online fault detection of a mobile robot with a parallelized particle filter. Neurocomputing 126, 151–165. Zhang, X., Geimer, M., Noack, P.O., Grandl, L., 2010. Development of an intelligent mastereslave system between agricultural vehicles. Proceedings of the 2010 IEEE Intelligent Vehicles Symposium. Institute of Electrical and Electronics Engineers, San Diego, CA, USA, pp. 250–255.