- Email: [email protected]
Scientific methods in mobile robotics
Robotics and Autonomous Systems ELSEVIER
Robotics and Autonomous Systems 24 (1998) 1-3
I
Editorial
Scientific methods in mobile robotics U l r i c h N e h m z o w a,., M i c h a e l R e c c e b, 1 a Department of Computer Science, University of Manchester, Manchester M13 9PL, UK b Department of Computer and Information Science, New Jersey Institute of Technology, University Heights, Newark, NJ 07102-1982, USA
The first UK conference "Towards Intelligent Mobile Robots" (TIMR '97) was held in 1997 in Manchester. It brought together an international group of mobile robot researchers with the goal to identify ways in which the field can develop beyond results that are merely "existence proofs", and move towards results that are quantitative and reproducible on different platforms and in different environments. This process includes the development of metrics and analysis tools that can put mobile robotics research on the road towards becoming a more scientific discipline, where key experiments are replicated and important findings are verified. This special issue of Robotics and Autonomous Systems collects some of the contributions to TIMR '97. It demonstrates a trend towards using quantitative methods in mobile robotics research, and shows that the community shares some common research practices. We do not yet have the research practice of natural sciences such as biochemistry or physics, where resuits are corroborated by independent research groups before they are accepted by the community, but the tools for such a practice are emerging. Mobile robotics is not a natural science, in that we are constructing new artificial autonomous systems, not analysing natural phenomena. However, it is closely related to the natural sciences, since many processes underlying the behaviour of mobile robots are rooted in natural phenomena. * Corresponding author. E-mail: [email protected]. 1E-mail: [email protected].
In mobile robotics, sensor noise and ambiguous, unreliable or inconsistent data have to be processed in real time. For reactive tasks such as obstacle avoidance and wall following, this variability can be overcome. However, for more complex tasks, like those that involve learning, the unreliability of robot sensors is a real problem. Wyatt et al. [9], for instance, present this clearly, and discuss the problems of obtaining reliable teaching signals from an unreliable Hall effect sensor. The design of a suitable reward function for self-supervised learning in these circumstances is a difficult task. "Naive" sensor models can quickly lead to overall failure of the mobile robot in its task, as the robotlearning example given above illustrates. There is a clear trend now in the mobile robotics community, to replace "naive" sensor models (such as the "single ray" model of sonar range finders) by more realistic and complete models [4,8,5]. "Wrong" use of sensors, as Walker et al. [8] argue, can lead to cumbersome and brittle solutions, while analogue computing through faithful sensor models can reduce computational load considerably. Hence the attempts by various contributors to this issue to develop faithful models of robot sensors [4,8,9]. Such models can serve as tools in the synthesis process of mobile robot behaviour. As in the synthesis, so in the analysis o f mobile robot behaviour some common practices can be observed. For instance, both qualitative and quantitative measures of robot performance are presented, and used for the evaluation of performance.
0921-8890/98/$ - see frontmatter © 1998 Published by Elsevier Science B.V. All fights reserved PII: S0921-8890(98)00017-7
u. Nehmzow, M. Recce/Robotics and Autonomous Systems 24 (1998) 1-3
Qualitative analysis methods used include the "blind" evaluation of video films [9], and qualitative correlation measures [1]. Wyatt et al. [9] use statistical correlation methods to compare the performance of different sensory filters, and Duckett and Nehmzow [3] use correlation methods to determine the performance of a mobile robot self-localisation system. A number of different experimental procedures are used for the experiments presented in this special issue. Blind evaluation of video tapes was mentioned before. Other groups use trace-replay mechanisms [3,6], in which sensor data obtained from a mobile robot moving in its target environment is logged and used for subsequent off-line analysis. This method allows comparison of different sensor signal processing techniques using identical data, while still using "real" sensor data, as opposed to simulated data. Billard and Dautenhahn [1] conduct experiments that are similar in nature both in simulation and on physical robots, although not explicitly to compare physical robots and their simulations, but to evaluate a language acquisition mechanism on different agents, and in different environments. Finally, there is the controversial issue of "benchmarks". The fundamental argument against benchmark tasks is that the laws governing robot-environment interaction are unknown, and that it is therefore impossible to characterise a robot, its task and its environment sufficiently to allow a precise definition of the benchmark [9]. However, we contend that benchmark tasks and comparative studies are essentially the same, because a benchmark can be viewed as the evaluation of robot performance by replicating someone else's task-robot-environment setup. If "benchmark" is meant to signify the precise replication of a particular robot-environment-task setup, then this is indeed not achievable and does not warrent significant effort. However, the replication of an experimental setup that is "essentially the same" as the original will serve a purpose in mobile robotics research; perhaps "comparative studies" is a good term for this type of work. All papers in this special issue demonstrate that a functioning mobile robot, performing a specific task in a target environment, is the result of a complex and iterative development process that is often guided by intuition, rather than knowledge. Braitenberg's illustration of downhill synthesis and uphill analysis [2],
which purports that it is easier to construct an agent exhibiting complex behaviour than it is to identify the underlying mechanisms producing that behaviour, has a surprising opposite here. While it is easier to construct an agent that exhibits some complex behaviour than it is to analyse its fundamental building blocks, we believe that for the design of a specific behaviour the opposite is true: the design of an agent exhibiting a clearly defined behaviour is more difficult than the agent's analysis, once the behaviour is established. This extended version of Braitenberg's illustration would explain why we have not yet managed to build intelligently behaving robots that mimic natural systems convincingly. In conclusion, then, the present issue demonstrates consensus in the need to analyse embedded systems, that is to consider robot, task and environment together. It presents attempts by several groups to quantify robot-environment interaction through statistical methods, and a survey of experimental methods currently in use in the mobile robotics research community. The debate about "benchmarks" is still ongoing, probably due to the loose definition of the term, and we look forward to TIMR '98 to resolve some of the questions raised by its predecessor and this current special issue on scientific methods in mobile robotics.
Acknowledgements We thank David Bisset for his support in organising "Towards Intelligent Mobile Robots 97" in Manchester, and for his valuable contributions to this special issue of Robotics and Autonomous Systems. We also thank the Department of Computer Science at Manchester, which supported TIMR '97 financially. The editors express their thanks to the reviewers of this special issue, who were conscientious, dependable, and very helpful. Finally, we thank Frans Groen and his team at Elsevier for advice and help in bringing together this special issue on "Scientific Methods in Mobile Robotics".
References [1] A. Billard, K. Dautenhahn, Grounding communication in autonomous robots: An experimental study, Robotics and Autonomous Systems 24 (1998) 71-79 (this issue).
U. Nehmzow, M. Recce /Robotics and Autonomous Systems 24 (1998) 1-3 [2] V. Braitenberg, Vehicles: Experiments in Synthetic Psychology, MIT Press, Cambridge, MA, 1984. [3] T. Duckett, U. Nehmzow, Mobile robot self-localisation and measurement of performance in middle-scale environments, Robotics and Autonomous Systems 24 (1998) 33-42 (this issue). [4] K. Hams, M. Recce, Experimental modelling of time-offlight sonar, Robotics and Autonomous Systems 24 (1998) 57-69 (this issue). [5] T. Lee, U. Nehmzow, R. Hubbold, Mobile robot simulation by means of acquired neural network models, in: Proceedings of the European Multiconference on Simulation, Manchester, 1998. [6] O. Lemon, U. Nehmzow, The scientific status of mobile robotics: Multi-resolution mapbuilding as a case study,
3
Robotics and Autonomous Systems 24 (1998) 5-15 (this issue). [7] U. Nehmzow, M. Recce, D. Bisset, in: Proceedings of the Towards Intelligent Mobile Robots 97, Technical Report Series, Department of Computer Science, Manchester University, Manchester 1997, http://www.cs.man.ac.uk/ csonly/cstechrep/titles97.html. [8] V.A. Walker, H. Peremans, J.C.T. Hallam, Good vibrations: Exploiting reflector motion to partition an acoustic environment, Robotics and Autonomous Systems 24 (1998) 43-55 (this issue). [9] J. Wyatt, J. Hoar, G. Hayes, Design, analysis and comparison of robot learners, Robotics and Autonomous Systems 24 (1998) 17-32 (this issue).
Robotics and Autonomous Systems 24 (1998) 1-3
I
Editorial
Scientific methods in mobile robotics U l r i c h N e h m z o w a,., M i c h a e l R e c c e b, 1 a Department of Computer Science, University of Manchester, Manchester M13 9PL, UK b Department of Computer and Information Science, New Jersey Institute of Technology, University Heights, Newark, NJ 07102-1982, USA
The first UK conference "Towards Intelligent Mobile Robots" (TIMR '97) was held in 1997 in Manchester. It brought together an international group of mobile robot researchers with the goal to identify ways in which the field can develop beyond results that are merely "existence proofs", and move towards results that are quantitative and reproducible on different platforms and in different environments. This process includes the development of metrics and analysis tools that can put mobile robotics research on the road towards becoming a more scientific discipline, where key experiments are replicated and important findings are verified. This special issue of Robotics and Autonomous Systems collects some of the contributions to TIMR '97. It demonstrates a trend towards using quantitative methods in mobile robotics research, and shows that the community shares some common research practices. We do not yet have the research practice of natural sciences such as biochemistry or physics, where resuits are corroborated by independent research groups before they are accepted by the community, but the tools for such a practice are emerging. Mobile robotics is not a natural science, in that we are constructing new artificial autonomous systems, not analysing natural phenomena. However, it is closely related to the natural sciences, since many processes underlying the behaviour of mobile robots are rooted in natural phenomena. * Corresponding author. E-mail: [email protected]. 1E-mail: [email protected].
In mobile robotics, sensor noise and ambiguous, unreliable or inconsistent data have to be processed in real time. For reactive tasks such as obstacle avoidance and wall following, this variability can be overcome. However, for more complex tasks, like those that involve learning, the unreliability of robot sensors is a real problem. Wyatt et al. [9], for instance, present this clearly, and discuss the problems of obtaining reliable teaching signals from an unreliable Hall effect sensor. The design of a suitable reward function for self-supervised learning in these circumstances is a difficult task. "Naive" sensor models can quickly lead to overall failure of the mobile robot in its task, as the robotlearning example given above illustrates. There is a clear trend now in the mobile robotics community, to replace "naive" sensor models (such as the "single ray" model of sonar range finders) by more realistic and complete models [4,8,5]. "Wrong" use of sensors, as Walker et al. [8] argue, can lead to cumbersome and brittle solutions, while analogue computing through faithful sensor models can reduce computational load considerably. Hence the attempts by various contributors to this issue to develop faithful models of robot sensors [4,8,9]. Such models can serve as tools in the synthesis process of mobile robot behaviour. As in the synthesis, so in the analysis o f mobile robot behaviour some common practices can be observed. For instance, both qualitative and quantitative measures of robot performance are presented, and used for the evaluation of performance.
0921-8890/98/$ - see frontmatter © 1998 Published by Elsevier Science B.V. All fights reserved PII: S0921-8890(98)00017-7
u. Nehmzow, M. Recce/Robotics and Autonomous Systems 24 (1998) 1-3
Qualitative analysis methods used include the "blind" evaluation of video films [9], and qualitative correlation measures [1]. Wyatt et al. [9] use statistical correlation methods to compare the performance of different sensory filters, and Duckett and Nehmzow [3] use correlation methods to determine the performance of a mobile robot self-localisation system. A number of different experimental procedures are used for the experiments presented in this special issue. Blind evaluation of video tapes was mentioned before. Other groups use trace-replay mechanisms [3,6], in which sensor data obtained from a mobile robot moving in its target environment is logged and used for subsequent off-line analysis. This method allows comparison of different sensor signal processing techniques using identical data, while still using "real" sensor data, as opposed to simulated data. Billard and Dautenhahn [1] conduct experiments that are similar in nature both in simulation and on physical robots, although not explicitly to compare physical robots and their simulations, but to evaluate a language acquisition mechanism on different agents, and in different environments. Finally, there is the controversial issue of "benchmarks". The fundamental argument against benchmark tasks is that the laws governing robot-environment interaction are unknown, and that it is therefore impossible to characterise a robot, its task and its environment sufficiently to allow a precise definition of the benchmark [9]. However, we contend that benchmark tasks and comparative studies are essentially the same, because a benchmark can be viewed as the evaluation of robot performance by replicating someone else's task-robot-environment setup. If "benchmark" is meant to signify the precise replication of a particular robot-environment-task setup, then this is indeed not achievable and does not warrent significant effort. However, the replication of an experimental setup that is "essentially the same" as the original will serve a purpose in mobile robotics research; perhaps "comparative studies" is a good term for this type of work. All papers in this special issue demonstrate that a functioning mobile robot, performing a specific task in a target environment, is the result of a complex and iterative development process that is often guided by intuition, rather than knowledge. Braitenberg's illustration of downhill synthesis and uphill analysis [2],
which purports that it is easier to construct an agent exhibiting complex behaviour than it is to identify the underlying mechanisms producing that behaviour, has a surprising opposite here. While it is easier to construct an agent that exhibits some complex behaviour than it is to analyse its fundamental building blocks, we believe that for the design of a specific behaviour the opposite is true: the design of an agent exhibiting a clearly defined behaviour is more difficult than the agent's analysis, once the behaviour is established. This extended version of Braitenberg's illustration would explain why we have not yet managed to build intelligently behaving robots that mimic natural systems convincingly. In conclusion, then, the present issue demonstrates consensus in the need to analyse embedded systems, that is to consider robot, task and environment together. It presents attempts by several groups to quantify robot-environment interaction through statistical methods, and a survey of experimental methods currently in use in the mobile robotics research community. The debate about "benchmarks" is still ongoing, probably due to the loose definition of the term, and we look forward to TIMR '98 to resolve some of the questions raised by its predecessor and this current special issue on scientific methods in mobile robotics.
Acknowledgements We thank David Bisset for his support in organising "Towards Intelligent Mobile Robots 97" in Manchester, and for his valuable contributions to this special issue of Robotics and Autonomous Systems. We also thank the Department of Computer Science at Manchester, which supported TIMR '97 financially. The editors express their thanks to the reviewers of this special issue, who were conscientious, dependable, and very helpful. Finally, we thank Frans Groen and his team at Elsevier for advice and help in bringing together this special issue on "Scientific Methods in Mobile Robotics".
References [1] A. Billard, K. Dautenhahn, Grounding communication in autonomous robots: An experimental study, Robotics and Autonomous Systems 24 (1998) 71-79 (this issue).
U. Nehmzow, M. Recce /Robotics and Autonomous Systems 24 (1998) 1-3 [2] V. Braitenberg, Vehicles: Experiments in Synthetic Psychology, MIT Press, Cambridge, MA, 1984. [3] T. Duckett, U. Nehmzow, Mobile robot self-localisation and measurement of performance in middle-scale environments, Robotics and Autonomous Systems 24 (1998) 33-42 (this issue). [4] K. Hams, M. Recce, Experimental modelling of time-offlight sonar, Robotics and Autonomous Systems 24 (1998) 57-69 (this issue). [5] T. Lee, U. Nehmzow, R. Hubbold, Mobile robot simulation by means of acquired neural network models, in: Proceedings of the European Multiconference on Simulation, Manchester, 1998. [6] O. Lemon, U. Nehmzow, The scientific status of mobile robotics: Multi-resolution mapbuilding as a case study,
3
Robotics and Autonomous Systems 24 (1998) 5-15 (this issue). [7] U. Nehmzow, M. Recce, D. Bisset, in: Proceedings of the Towards Intelligent Mobile Robots 97, Technical Report Series, Department of Computer Science, Manchester University, Manchester 1997, http://www.cs.man.ac.uk/ csonly/cstechrep/titles97.html. [8] V.A. Walker, H. Peremans, J.C.T. Hallam, Good vibrations: Exploiting reflector motion to partition an acoustic environment, Robotics and Autonomous Systems 24 (1998) 43-55 (this issue). [9] J. Wyatt, J. Hoar, G. Hayes, Design, analysis and comparison of robot learners, Robotics and Autonomous Systems 24 (1998) 17-32 (this issue).