- Email: [email protected]
Human Interfaces for Rescue Teams
ELSEVIER
Copyright © IFAC Telematics Applications in Automation and Robotics, Espoo. Finland. 2004
IFAC PUBLICATIONS www.clscvicr.comllocateJifac
ROBOTIHUMAN INTERFACES FOR RESCUE TEAMS
Frauke Driewer, Herbert Baier, Klaus Schilling Julius-Maximilians-University Wunburg. Informatics VII: Robotics and Telematics Am Hubland, 97074 Wunburg. Germany {driewer. baier. [email protected]
Abstract: Mobile robots can be of significant support for human teams in search and rescue situations. A user requirement analysis on basis of questionnaires distributed to fire fighters and emergency support people expresses notable interest in sending teleoperated robots into dangerous areas instead of risking human life. This paper summarises the requirements and the consequences for a telematic system with an intuitive user interface. which supports quick reaction capabilities. Design and implementation of an infrastructure addressing data flow in joint teams of humans and robots for data sharing, teleoperations and remote coordination are presented. Furthermore, robot vehicles used in that context and test scenarios are described. Copyright © 2004 IFAC
Keywords: teleoperation. telepresence, mobile robots. user interfaces. man-machine interaction, cooperation
l. INTRODUCTION
environment and the situation. The team members are equipped with sensors for localization and environment characterization. On basis of these measurements the coordinators know the position and status of all team members, as well as environmental conditions. Thus, an overview of the situation can be obtained that allows improved planning of tasks. The coordinators guide the human team members through the environment and command the mobile robots. This is supported by algorithms for efficient cooperative coverage in searching the complete area and by a user interface, which provides all the necessary information in an intuitive way, enabling quick reactions in time critical situations.
Mobile robots can assist or even replace humans in dangerous and hazardous situations. Typical application areas emerge in space, under water or in other harsh environments, where access is impossible or very dangerous for humans. The use of mobile robots for search and rescue services during catastrophes and disasters represents such a scenario. Robots with appropriate equipment can take sensor measurements to characterize the environment and localize themselves also in situations adverse for humans, such as at low-visibility conditions. Nevertheless, robots can only complement the humans with their far superior adaptive reaction capabilities. In order to take advantage of these complementary capabilities, telematics techniques can be used to support joint tearns of humans and robots by means of data sharing, teleoperation and remote coordination. Tasks for mobile vehicles are the exploration of unknown regions and the search for victims. During their mission the robotlhuman rescue team is supported by mission coordinators. They are located at a safe remote site (outside the region of accident) taking advantage of a well functioning infrastructure. The support provided by the remote coordinator varies from guidance of the team to provision of additional background knowledge about the
Fig. I. Human/robot team with remote coordinator.
59
stairs doors, windows, rooms, access routes, valves, pipelines, shut-off socks, energy systems, hazardous material and all fire fighting relevant objects, such as possible position of fire source, fire reporting centre, exits and entry ways, extinguish systems and sources of water. During the mission other data will be added dynamically to this map, such as the position of rescue personnel and victims or polluted areas. The requested accuracy for position of robots and humans, including victims, is about one meter and the required update time of information should be in the range between one and ten seconds. For dangerous materials the specification of the region is sufficient. Highly demanded was also the visualization and supervision of the breathing protection system. Most preferable way to display map information to the rescue personnel in place is a head-mounted display, followed by an arm-mounted display.
During discussions with professionals in search and rescue, such as fire fighters, it was obvious that there is a wide interest in the application of robots for emergency cases. Nevertheless, there exist also concerns against the use of new, possibly immature technology. Therefore, a careful investigation of the user's needs and requirements is absolutely essential for the development of rescue systems. This paper describes the requirements analyses based on questionnaires distributed to emergency support groups as well as on detailed interviews with fire departments and other institutions, dealing with emergencies. According to the user requirements a typical scenario was derived, robots were selected for tests and the basic system design for a human/robot rescue tearn supported by a remote coordinator is presented in this contribution. The presented work is part of the PeLoTe project (Building Presence through Localization for Hybrid Telematic Teams) funded by the European Community within the "Future and Emerging Technologies" program targeted on "presence" methods and applications.
The robots are considered to be especially useful for exploration of an unknown emergency area and search for victims. Currently, telerobots are preferred instead of fully autonomous robots. However, autonomous features supporting operations like path planning with obstacle avoidance and person following capabilities are desired. For future developments, fire fighters like to have autonomous robots that are able to accomplish tasks, such as searching wide areas for victims and exploration of dangerous areas on their own. Although, they are in some cases cautious about the reliability and adaptability of autonomous vehicles. Nevertheless, the employed telerobots are provided with autonomous low-level reactions and emergency handling situations, satisfying high criteria for robustness. Good mobility is an important feature for the robot, in order to take measurements in areas, which are difficult to access. Data acquisition from these areas is a crucial point. Camera pictures from the scene are required from almost all rescue personnel. While audio communication, which permits the rescue personnel to speak with found persons through the robot, was required from half of the participants. The robot design should exhibit a high robustness to extreme environment conditions, such as little oxygen, explosive atmosphere, high or low temperature regions.
2. USER REQUIREMENTS The user requirements are derived from answers to questionnaires sent to fire departments (Germany, Finland and Czech Republic) and govemmental disaster relief organizations (Germany). Different types of fire brigades, related to cities, airports and industrial plants, replied to the questionnaire. The proposed cooperating human/robot system seems to be especially interesting for professional fire fighters, e.g. in chemical factories or airports, or for rescue cases in big office buildings, as well as after emergencies with massive destruction, like for earthquakes. For modem buildings there are usually maps of the building available as well as additional information, e.g. storage areas of dangerous materials or positions of activated fire alarm sensors. Due to the progressing destruction caused by the emergency, this map might be continuously changing. In order to represent all the knowledge about the environment, the building map will be combined with additional information in a layer-based electronic format. This is available for the remote coordinators, as well as for the humans in the emergency area and will be continuously updated. In that way sharing of up-todate data between team members is possible and allows quick and meaningful warnings, when unexpected situations arise. The layer-based format (cr. Fig. 2), which allows display of currently required information, received good acceptance, since maps used nowadays contain often too much information at once. Information that was considered as useful, are the ground plan with
Fig. 2. Example for layer-based rescue map: Building map overlaid with additional information on fire sensors and storage sites for dangerous material.
60
........ ......
_-
- - 00.. _
...........
Fig. 4. Client/server system architecture. Fig. 3 summarizes the flow of infonnation and control between the remote coordinator and the team members in place. Video cameras and audio transmission devices provide a realistic impression about the situation in the emergency site without being there. The human team members and the remote coordinator are using audio transmission for their standard conversations. e.g. for the exchange of observations. The robots carry also audio transmission devices in order to work as communication platfonn to the found persons. Other kinds of infonnation are sensor and navigation data, as well as control commands. In order to achieve this system design two software architectures have been considered: client/server and event based archi tectures.
Fig. 3. Schematic of infonnation flow and processing in the human/robot team in place and the remote coordinators. The techniques developed in the PeLoTe project receive good acceptance from the rescue community. There is great interest in localization for the team members during the mission. The general concept of telematic issues (telerobots and remote coordinator) for rescue scenarios has been confinned by the users. Robust communication in an emergency case is crucial for the end-user system. In order to meet this criteria, strategies are needed that deal also with interrupted communication, as autonomous features for the robots and backup communication. In order to deploy the layer-based map. an informative, but intuitive user interface is necessary. This also includes methods to display the camera pictures and properties of the team members. as well as to broadcast information. message and warnings. Efficiency in personnel placement and task sharing was demanded, which will be provided by cooperation and coordination strategies.
Fig. 4 displays the principle of the client/server architecture. a centralized architecture. All data communication passes through the server, which also stores these data for documentation uses. The server only responds to requests from the clients and it will never send an update to a client without a previous request. Thus, the clients need to ask for updates. In such architecture, the server does not take care about the communication. If it breaks, the concerned client has to reestablish it. When a client need some periodically update. it should pool the server. The clients can be divided into following categories: user interfaces for the remote coordinators (teleoperating centre) and agents for the humans and robots in the team (environment). Exceptionally from the centralized architecture audio and video can be requested directly by the clients. These data do not necessarily pass through the server. Furthennore, a human member in place can control a robot directly.
3. SYSTEM DESIGN This section addresses a generic cooperative system composed of humans, robots and remote coordinators for handling a given rescue task. In the special case of a search and rescue scenario, the environment is quite hostile. The infonnation sharing between the team members needs to be very quick and efficient, since actions are time-<:ritical and the humans in place (rescue personnel and victims) are in danger. A well-arranged user interface should provide a complete overview of the situation. The visualized infonnation has to be presented in an efficient and easily interpretable way, such that the coordinator can react quickly and in a right manner to the emergency situation. The humans in place need a simplified user interface and a personal navigation system (Saarinen et al., 2004). Furthennore, the developed system should not give additional load to the fire fighters. Thus, tasks regarding the infrastructure of the system, as e.g. establishing communication. are mostly executed automatically by the software.
The human team members have their personal navigation system and a user interface similar to the one in the teleoperating center. The software of the robot team members includes sensor data processing and navigation control. Each client should register itself in the PeLoTe server before it can participate in the system. Hence, the server is able to trace and to spread infonnation from clients. An alternative to the client/server architecture offers the event based inter-process communication, which is decentralized. The event traffic is managed by a
61
postmaster background ~rocess (pnuf). It provi?es event handling, forwarding. and message passmg between processes. When pmd receives an event, it forwards it to all the processes that are registered to receive this event The global structure is shown in Fig. 5.
The complete configuration of the system is automatically retained in the server. The event based architecture requires the integration of a special process that carries out this task (confmang). All events need to pass this process to keep track of the actual configuration, which means increase of network traffic. The extension to wide-spread distributed applications is in a simpler way realized by the event based architecture. However. its general implementation is very complex, since all event flows must be exactly defined. An event can generate subsequent chains of events.
Postslave processes (psd) can be used to extend the system. Each post process (pmd and psds) can monitor a subset of the remaining processes. like human and robot agents. The post processes are organized hierarchically, where the pmd is the master. There are two types of event flow between the post processes, hierarchical and cooperative forwarding (cf. Fig. 6). Hierarchical forwarding describes the direct communication between pmd and psd. The cooperative forwarding uses another psd as bridge and can be used in the case of communication deficit. This feature provides not only extension, but also robustness against communication failures and allows local communication between processes monitored by a post process.
For the PeLoTe project the client/server architecture was chosen. A server was implemented and the integration of different modules is going on. The server is the main component for data sharing in the PeLoTe system and supports multitasking and multiuser capabilities as well as portability (platform independent). The server has a modular structure, with a hierarchical communication between the modules, which are organized in levels (cf. Fig. 7). The Kernel is the only module of the first level. Each of the modules, except the Kernel, communicates with exactly one other module from a level one step below. Communication between modules of the same level is not allowed. A module can communicate with any higher level modules. Extension of the server is straightforward with this modular hierarchical concept: If a new module should be included, only an interface needs to be provided by a lower level module.
Both architectures show strengths and weaknesses for the search and rescue application. In the event based architecture the post process handles the communication and therefore also its trOUbles. e.g. reestablishing of connections. This is not the case in the client/server architecture, since the server only responses to requests of the clients, which are themselves responsible for their connection. On the other hand. the continuous requests from the client side lead to unnecessary network traffic, if no new information is available.
The core of the server is the Kernel module. which is the central point for data sharing and control. It is responsible for the configuration management (manages the current status and configuration of the team members and environment), persistence (creates and maintains log files and save the configuration of the system) as well as authentication and authorization of clients. The application programming interface (API) module allows third party software to communicate locally with the server. An extension of the API is the RMI (Remote Method Invocation) module based on Java RMI, which enables remote communication. Thus, a client can call a remote object in the server in an easy and standard way.
Fig. 5. Event based architecture.
Fig. 7. a) Server modular architecture. b) Example: RMI modules request information from SRM module. The communication goes through API and Kernel modules.
Fig. 6. Extension of event based architecture.
62
So far, the following modules are included: • Planning: Responsible for the path planning of the multiple entities. • CoLo: Performs the cooperative localization tasks. • SRM: Deals with the tasks concerning the standard rescue map. The AMU module is an extension of SRM for the automated map update.
• •
•
In order to increase the usability of the PeLoTe system a graphical user interface (Gill) is designed. The GUI for the remote coordinator and for the human team members includes a two-dimensional map as well as the team member properties and data (sensor data, video). The remote coordinator can observe the environment and the team, update the map, command the robots and guide the humans from a remote place outside the emergency area. The GUI and especially the two dimensional map provides also a communication platfonn. Together with the audio communication and the video images, it generates a telepresence scenario increasing the impression of the concerned persons to be in the same place.
•
4. TEST SCENARIOS Reflecting the user requirements, test scenarios have been developed, which allows the evaluation of the design system. The experiments will address an indoor environment, which represents a typical large office building. The characteristics of these test fields are: many rooms, irregular shape and dead ends. Parameters as visibility, number of victims or configuration of the environment (e.g. source of fire, dangerous areas) will be changed. Moreover, different stages of difficulty can be arranged (Murphy et al.• 2000), e.g. the victims are easy to find or can be hidden behind objects. The tests will be performed by traditional human-only and robotlhuman teams. From the direct comparison, the benefits of the use of robots, as well as drawbacks and points for further improvements will be identified. Assessment factors are e.g. the time needed to complete the task by different teams and the safety for the rescue team. A more detailed evaluation will include the test observation by emergency professionals. documented in interviews directly after the test.
From the standards of user interface design (Wessel, 2002) and the user requirement analyses, guidelines for the man-machine interfaces have been compiled: •
•
User's language: The GUI uses the terminology used by firefighters. User's logic: Testing and evaluation by qualified test users will be perfonned to provide feedback to the programmers. User group: For designing the GUI, a user group is considered that is familiar with technical equipment, but most probably not an everyday user of computers. Before using the equipment in emergencies significant training practice has to be passed, so novice users are not considered. Standardization: Common code of practice. e.g. color or shapes is used. Icons and pictograms that are standard in the fire fighting community are utilized.
Feedback to user: The user is informed about the status of the system and the team members, e.g. the user receives alarms related to communication interrupts. Delay times are kept as small as possible. The system has multitasking abilities. User control: Background tasks are automated and invisible for the end-user, e.g. regular update of the position of the team members. However, each user has full control over hislher own system.
-HI
5. RESCUE ROBOTS
_~--_ -.._ -...... .....
As test vehicle the MERLIN (Mobile Experiment Robot for Locomotion and Intelligent Navigation) robot (Schilling and Meng, 2002) is used for the search and rescue scenario. The length is about 40 cm. It is equipped with a CI67 rnicrocontroller and radio link (Radio Packet Controller). The motors and the board are supplied by battery. The rover is equipped with ultrasonic and odometry sensors and performs autonomous low-level actions, such as collision avoidance.
I
---
..... _.- --.... - .. - --............. .. __ ~-
..-,~
--.a._ -
--
In order to provide further processing capacity and handle WLAN access, a PC 104 board is connected to the rnicrocontroller via serial port. Due to its CANbus interface the rover provides good flexibility and an excellent test bed for new sensors, control methods and algorithms.
Fig. 8. User interface design: The zoom window shows positions of human I, robot 2 and found victims, as well as dangerous areas and map updates.
63
There exist tracked and wheeled versions to offer suitable vehicles for a broad range of application scenarios. The tracked ven;ion of MERLIN provides the same features and is equipped with similar senson;, but different locomotion properties. The tracked wheels are more suitable for moving in destructed areas, but lead to increased power consumption and therefore to a reduced operations period. In order to relieve the operator, further autonomy features are implemented, as execution of movement with obstacle avoidance. This prevents the operator from the task of continuously teleoperating the robot. Another feature of the robot is the autonomous pursuit of a person. This is useful in order to send a small human-robot team as a group to the same target In this context work related to convoy driving and cooperative navigation has been analysed with the MERLIN robots (Gilioli and Schilling, 2(03). This work concerns the development of navigation algorithms for multiple mobile robots on basis of ultrasonic range measurements.
Fig. 9. Wheeled test vehicle MERLIN protection cover.
6. CONCLUSIONS
without
Fig. 10. Tracked MERLIN test vehicle sharing the same electronics and sensor data processing system with the wheeled ven;ion, only differing in the locomotion system.
From the evaluation of user requirements by questionnaires, an architecture for a heterogeneous human/robot search and rescue systems was derived, including in particular telepresence methods for remote coordinators. Generic techniques for cooperation of humans and robots in joint groups have been developed and related software systems are implemented. The core of the server is running and integration of modules is going on. Similar strategies in planning, cooperation, coordination, localization and man-machine interaction offer also interesting application potential in industrial, service robotic or educational context. Due to the user interface and data sharing the feeling of being "present" in place for the remote coordinator is increased. This allows a better communication between coordinator and humans in place, as well as an improved coordination of the team.
8. REFERENCES
Gilioli M. and K. Schilling. (2003). Autonomous Cooperative Localization of Mobile Robots Based on Ranging Systems. SPIE Aerosense conference proceedings "Unmanned Ground Vehicle Technology V", Orlando, USA, paper 5083-15. Murphy R., l. Casper, M. Micire and l. Hyams (2000). Assessment of the NIST Standard Test Bed for Urban Search and Rescue. NIST Workshop on Performance Metrics for Intelligent Systems 2000. Saarinen l., R. MazI, P. Emest, l. Suomela and L. Preucil. (2004). Sensors and Methods for Human Dead Reckoning. Proceedings of the 8th Conference on Intelligent Autonomous Systems IAS8, Amsterdam. Schilling K. and Q. Meng (2002). The MERLIN vehicles for outdoor applications, SPIE Aerosense conference proceedings "Unmanned Ground Vehicle Technology N", Orlando, USA. Wessel I. (2002). GUI-Design. Carl Hanser Verlag Munich, Germany. ISBN 3-446-21961-7.
7. ACKNOWLEDGEMENTS This work has been supported within the EU-project "PeLoTe - Building Presence through Localization for Hybrid Telematic Systems". The authors acknowledge the contributions and discussions with our consortium partners at the Czech Technical University in Prague, Helsinki Univen;ity of Technology, Certicon a.s. and Steinbeis Transferzentrum ARS.
64
Copyright © IFAC Telematics Applications in Automation and Robotics, Espoo. Finland. 2004
IFAC PUBLICATIONS www.clscvicr.comllocateJifac
ROBOTIHUMAN INTERFACES FOR RESCUE TEAMS
Frauke Driewer, Herbert Baier, Klaus Schilling Julius-Maximilians-University Wunburg. Informatics VII: Robotics and Telematics Am Hubland, 97074 Wunburg. Germany {driewer. baier. [email protected]
Abstract: Mobile robots can be of significant support for human teams in search and rescue situations. A user requirement analysis on basis of questionnaires distributed to fire fighters and emergency support people expresses notable interest in sending teleoperated robots into dangerous areas instead of risking human life. This paper summarises the requirements and the consequences for a telematic system with an intuitive user interface. which supports quick reaction capabilities. Design and implementation of an infrastructure addressing data flow in joint teams of humans and robots for data sharing, teleoperations and remote coordination are presented. Furthermore, robot vehicles used in that context and test scenarios are described. Copyright © 2004 IFAC
Keywords: teleoperation. telepresence, mobile robots. user interfaces. man-machine interaction, cooperation
l. INTRODUCTION
environment and the situation. The team members are equipped with sensors for localization and environment characterization. On basis of these measurements the coordinators know the position and status of all team members, as well as environmental conditions. Thus, an overview of the situation can be obtained that allows improved planning of tasks. The coordinators guide the human team members through the environment and command the mobile robots. This is supported by algorithms for efficient cooperative coverage in searching the complete area and by a user interface, which provides all the necessary information in an intuitive way, enabling quick reactions in time critical situations.
Mobile robots can assist or even replace humans in dangerous and hazardous situations. Typical application areas emerge in space, under water or in other harsh environments, where access is impossible or very dangerous for humans. The use of mobile robots for search and rescue services during catastrophes and disasters represents such a scenario. Robots with appropriate equipment can take sensor measurements to characterize the environment and localize themselves also in situations adverse for humans, such as at low-visibility conditions. Nevertheless, robots can only complement the humans with their far superior adaptive reaction capabilities. In order to take advantage of these complementary capabilities, telematics techniques can be used to support joint tearns of humans and robots by means of data sharing, teleoperation and remote coordination. Tasks for mobile vehicles are the exploration of unknown regions and the search for victims. During their mission the robotlhuman rescue team is supported by mission coordinators. They are located at a safe remote site (outside the region of accident) taking advantage of a well functioning infrastructure. The support provided by the remote coordinator varies from guidance of the team to provision of additional background knowledge about the
Fig. I. Human/robot team with remote coordinator.
59
stairs doors, windows, rooms, access routes, valves, pipelines, shut-off socks, energy systems, hazardous material and all fire fighting relevant objects, such as possible position of fire source, fire reporting centre, exits and entry ways, extinguish systems and sources of water. During the mission other data will be added dynamically to this map, such as the position of rescue personnel and victims or polluted areas. The requested accuracy for position of robots and humans, including victims, is about one meter and the required update time of information should be in the range between one and ten seconds. For dangerous materials the specification of the region is sufficient. Highly demanded was also the visualization and supervision of the breathing protection system. Most preferable way to display map information to the rescue personnel in place is a head-mounted display, followed by an arm-mounted display.
During discussions with professionals in search and rescue, such as fire fighters, it was obvious that there is a wide interest in the application of robots for emergency cases. Nevertheless, there exist also concerns against the use of new, possibly immature technology. Therefore, a careful investigation of the user's needs and requirements is absolutely essential for the development of rescue systems. This paper describes the requirements analyses based on questionnaires distributed to emergency support groups as well as on detailed interviews with fire departments and other institutions, dealing with emergencies. According to the user requirements a typical scenario was derived, robots were selected for tests and the basic system design for a human/robot rescue tearn supported by a remote coordinator is presented in this contribution. The presented work is part of the PeLoTe project (Building Presence through Localization for Hybrid Telematic Teams) funded by the European Community within the "Future and Emerging Technologies" program targeted on "presence" methods and applications.
The robots are considered to be especially useful for exploration of an unknown emergency area and search for victims. Currently, telerobots are preferred instead of fully autonomous robots. However, autonomous features supporting operations like path planning with obstacle avoidance and person following capabilities are desired. For future developments, fire fighters like to have autonomous robots that are able to accomplish tasks, such as searching wide areas for victims and exploration of dangerous areas on their own. Although, they are in some cases cautious about the reliability and adaptability of autonomous vehicles. Nevertheless, the employed telerobots are provided with autonomous low-level reactions and emergency handling situations, satisfying high criteria for robustness. Good mobility is an important feature for the robot, in order to take measurements in areas, which are difficult to access. Data acquisition from these areas is a crucial point. Camera pictures from the scene are required from almost all rescue personnel. While audio communication, which permits the rescue personnel to speak with found persons through the robot, was required from half of the participants. The robot design should exhibit a high robustness to extreme environment conditions, such as little oxygen, explosive atmosphere, high or low temperature regions.
2. USER REQUIREMENTS The user requirements are derived from answers to questionnaires sent to fire departments (Germany, Finland and Czech Republic) and govemmental disaster relief organizations (Germany). Different types of fire brigades, related to cities, airports and industrial plants, replied to the questionnaire. The proposed cooperating human/robot system seems to be especially interesting for professional fire fighters, e.g. in chemical factories or airports, or for rescue cases in big office buildings, as well as after emergencies with massive destruction, like for earthquakes. For modem buildings there are usually maps of the building available as well as additional information, e.g. storage areas of dangerous materials or positions of activated fire alarm sensors. Due to the progressing destruction caused by the emergency, this map might be continuously changing. In order to represent all the knowledge about the environment, the building map will be combined with additional information in a layer-based electronic format. This is available for the remote coordinators, as well as for the humans in the emergency area and will be continuously updated. In that way sharing of up-todate data between team members is possible and allows quick and meaningful warnings, when unexpected situations arise. The layer-based format (cr. Fig. 2), which allows display of currently required information, received good acceptance, since maps used nowadays contain often too much information at once. Information that was considered as useful, are the ground plan with
Fig. 2. Example for layer-based rescue map: Building map overlaid with additional information on fire sensors and storage sites for dangerous material.
60
........ ......
_-
- - 00.. _
...........
Fig. 4. Client/server system architecture. Fig. 3 summarizes the flow of infonnation and control between the remote coordinator and the team members in place. Video cameras and audio transmission devices provide a realistic impression about the situation in the emergency site without being there. The human team members and the remote coordinator are using audio transmission for their standard conversations. e.g. for the exchange of observations. The robots carry also audio transmission devices in order to work as communication platfonn to the found persons. Other kinds of infonnation are sensor and navigation data, as well as control commands. In order to achieve this system design two software architectures have been considered: client/server and event based archi tectures.
Fig. 3. Schematic of infonnation flow and processing in the human/robot team in place and the remote coordinators. The techniques developed in the PeLoTe project receive good acceptance from the rescue community. There is great interest in localization for the team members during the mission. The general concept of telematic issues (telerobots and remote coordinator) for rescue scenarios has been confinned by the users. Robust communication in an emergency case is crucial for the end-user system. In order to meet this criteria, strategies are needed that deal also with interrupted communication, as autonomous features for the robots and backup communication. In order to deploy the layer-based map. an informative, but intuitive user interface is necessary. This also includes methods to display the camera pictures and properties of the team members. as well as to broadcast information. message and warnings. Efficiency in personnel placement and task sharing was demanded, which will be provided by cooperation and coordination strategies.
Fig. 4 displays the principle of the client/server architecture. a centralized architecture. All data communication passes through the server, which also stores these data for documentation uses. The server only responds to requests from the clients and it will never send an update to a client without a previous request. Thus, the clients need to ask for updates. In such architecture, the server does not take care about the communication. If it breaks, the concerned client has to reestablish it. When a client need some periodically update. it should pool the server. The clients can be divided into following categories: user interfaces for the remote coordinators (teleoperating centre) and agents for the humans and robots in the team (environment). Exceptionally from the centralized architecture audio and video can be requested directly by the clients. These data do not necessarily pass through the server. Furthennore, a human member in place can control a robot directly.
3. SYSTEM DESIGN This section addresses a generic cooperative system composed of humans, robots and remote coordinators for handling a given rescue task. In the special case of a search and rescue scenario, the environment is quite hostile. The infonnation sharing between the team members needs to be very quick and efficient, since actions are time-<:ritical and the humans in place (rescue personnel and victims) are in danger. A well-arranged user interface should provide a complete overview of the situation. The visualized infonnation has to be presented in an efficient and easily interpretable way, such that the coordinator can react quickly and in a right manner to the emergency situation. The humans in place need a simplified user interface and a personal navigation system (Saarinen et al., 2004). Furthennore, the developed system should not give additional load to the fire fighters. Thus, tasks regarding the infrastructure of the system, as e.g. establishing communication. are mostly executed automatically by the software.
The human team members have their personal navigation system and a user interface similar to the one in the teleoperating center. The software of the robot team members includes sensor data processing and navigation control. Each client should register itself in the PeLoTe server before it can participate in the system. Hence, the server is able to trace and to spread infonnation from clients. An alternative to the client/server architecture offers the event based inter-process communication, which is decentralized. The event traffic is managed by a
61
postmaster background ~rocess (pnuf). It provi?es event handling, forwarding. and message passmg between processes. When pmd receives an event, it forwards it to all the processes that are registered to receive this event The global structure is shown in Fig. 5.
The complete configuration of the system is automatically retained in the server. The event based architecture requires the integration of a special process that carries out this task (confmang). All events need to pass this process to keep track of the actual configuration, which means increase of network traffic. The extension to wide-spread distributed applications is in a simpler way realized by the event based architecture. However. its general implementation is very complex, since all event flows must be exactly defined. An event can generate subsequent chains of events.
Postslave processes (psd) can be used to extend the system. Each post process (pmd and psds) can monitor a subset of the remaining processes. like human and robot agents. The post processes are organized hierarchically, where the pmd is the master. There are two types of event flow between the post processes, hierarchical and cooperative forwarding (cf. Fig. 6). Hierarchical forwarding describes the direct communication between pmd and psd. The cooperative forwarding uses another psd as bridge and can be used in the case of communication deficit. This feature provides not only extension, but also robustness against communication failures and allows local communication between processes monitored by a post process.
For the PeLoTe project the client/server architecture was chosen. A server was implemented and the integration of different modules is going on. The server is the main component for data sharing in the PeLoTe system and supports multitasking and multiuser capabilities as well as portability (platform independent). The server has a modular structure, with a hierarchical communication between the modules, which are organized in levels (cf. Fig. 7). The Kernel is the only module of the first level. Each of the modules, except the Kernel, communicates with exactly one other module from a level one step below. Communication between modules of the same level is not allowed. A module can communicate with any higher level modules. Extension of the server is straightforward with this modular hierarchical concept: If a new module should be included, only an interface needs to be provided by a lower level module.
Both architectures show strengths and weaknesses for the search and rescue application. In the event based architecture the post process handles the communication and therefore also its trOUbles. e.g. reestablishing of connections. This is not the case in the client/server architecture, since the server only responses to requests of the clients, which are themselves responsible for their connection. On the other hand. the continuous requests from the client side lead to unnecessary network traffic, if no new information is available.
The core of the server is the Kernel module. which is the central point for data sharing and control. It is responsible for the configuration management (manages the current status and configuration of the team members and environment), persistence (creates and maintains log files and save the configuration of the system) as well as authentication and authorization of clients. The application programming interface (API) module allows third party software to communicate locally with the server. An extension of the API is the RMI (Remote Method Invocation) module based on Java RMI, which enables remote communication. Thus, a client can call a remote object in the server in an easy and standard way.
Fig. 5. Event based architecture.
Fig. 7. a) Server modular architecture. b) Example: RMI modules request information from SRM module. The communication goes through API and Kernel modules.
Fig. 6. Extension of event based architecture.
62
So far, the following modules are included: • Planning: Responsible for the path planning of the multiple entities. • CoLo: Performs the cooperative localization tasks. • SRM: Deals with the tasks concerning the standard rescue map. The AMU module is an extension of SRM for the automated map update.
• •
•
In order to increase the usability of the PeLoTe system a graphical user interface (Gill) is designed. The GUI for the remote coordinator and for the human team members includes a two-dimensional map as well as the team member properties and data (sensor data, video). The remote coordinator can observe the environment and the team, update the map, command the robots and guide the humans from a remote place outside the emergency area. The GUI and especially the two dimensional map provides also a communication platfonn. Together with the audio communication and the video images, it generates a telepresence scenario increasing the impression of the concerned persons to be in the same place.
•
4. TEST SCENARIOS Reflecting the user requirements, test scenarios have been developed, which allows the evaluation of the design system. The experiments will address an indoor environment, which represents a typical large office building. The characteristics of these test fields are: many rooms, irregular shape and dead ends. Parameters as visibility, number of victims or configuration of the environment (e.g. source of fire, dangerous areas) will be changed. Moreover, different stages of difficulty can be arranged (Murphy et al.• 2000), e.g. the victims are easy to find or can be hidden behind objects. The tests will be performed by traditional human-only and robotlhuman teams. From the direct comparison, the benefits of the use of robots, as well as drawbacks and points for further improvements will be identified. Assessment factors are e.g. the time needed to complete the task by different teams and the safety for the rescue team. A more detailed evaluation will include the test observation by emergency professionals. documented in interviews directly after the test.
From the standards of user interface design (Wessel, 2002) and the user requirement analyses, guidelines for the man-machine interfaces have been compiled: •
•
User's language: The GUI uses the terminology used by firefighters. User's logic: Testing and evaluation by qualified test users will be perfonned to provide feedback to the programmers. User group: For designing the GUI, a user group is considered that is familiar with technical equipment, but most probably not an everyday user of computers. Before using the equipment in emergencies significant training practice has to be passed, so novice users are not considered. Standardization: Common code of practice. e.g. color or shapes is used. Icons and pictograms that are standard in the fire fighting community are utilized.
Feedback to user: The user is informed about the status of the system and the team members, e.g. the user receives alarms related to communication interrupts. Delay times are kept as small as possible. The system has multitasking abilities. User control: Background tasks are automated and invisible for the end-user, e.g. regular update of the position of the team members. However, each user has full control over hislher own system.
-HI
5. RESCUE ROBOTS
_~--_ -.._ -...... .....
As test vehicle the MERLIN (Mobile Experiment Robot for Locomotion and Intelligent Navigation) robot (Schilling and Meng, 2002) is used for the search and rescue scenario. The length is about 40 cm. It is equipped with a CI67 rnicrocontroller and radio link (Radio Packet Controller). The motors and the board are supplied by battery. The rover is equipped with ultrasonic and odometry sensors and performs autonomous low-level actions, such as collision avoidance.
I
---
..... _.- --.... - .. - --............. .. __ ~-
..-,~
--.a._ -
--
In order to provide further processing capacity and handle WLAN access, a PC 104 board is connected to the rnicrocontroller via serial port. Due to its CANbus interface the rover provides good flexibility and an excellent test bed for new sensors, control methods and algorithms.
Fig. 8. User interface design: The zoom window shows positions of human I, robot 2 and found victims, as well as dangerous areas and map updates.
63
There exist tracked and wheeled versions to offer suitable vehicles for a broad range of application scenarios. The tracked ven;ion of MERLIN provides the same features and is equipped with similar senson;, but different locomotion properties. The tracked wheels are more suitable for moving in destructed areas, but lead to increased power consumption and therefore to a reduced operations period. In order to relieve the operator, further autonomy features are implemented, as execution of movement with obstacle avoidance. This prevents the operator from the task of continuously teleoperating the robot. Another feature of the robot is the autonomous pursuit of a person. This is useful in order to send a small human-robot team as a group to the same target In this context work related to convoy driving and cooperative navigation has been analysed with the MERLIN robots (Gilioli and Schilling, 2(03). This work concerns the development of navigation algorithms for multiple mobile robots on basis of ultrasonic range measurements.
Fig. 9. Wheeled test vehicle MERLIN protection cover.
6. CONCLUSIONS
without
Fig. 10. Tracked MERLIN test vehicle sharing the same electronics and sensor data processing system with the wheeled ven;ion, only differing in the locomotion system.
From the evaluation of user requirements by questionnaires, an architecture for a heterogeneous human/robot search and rescue systems was derived, including in particular telepresence methods for remote coordinators. Generic techniques for cooperation of humans and robots in joint groups have been developed and related software systems are implemented. The core of the server is running and integration of modules is going on. Similar strategies in planning, cooperation, coordination, localization and man-machine interaction offer also interesting application potential in industrial, service robotic or educational context. Due to the user interface and data sharing the feeling of being "present" in place for the remote coordinator is increased. This allows a better communication between coordinator and humans in place, as well as an improved coordination of the team.
8. REFERENCES
Gilioli M. and K. Schilling. (2003). Autonomous Cooperative Localization of Mobile Robots Based on Ranging Systems. SPIE Aerosense conference proceedings "Unmanned Ground Vehicle Technology V", Orlando, USA, paper 5083-15. Murphy R., l. Casper, M. Micire and l. Hyams (2000). Assessment of the NIST Standard Test Bed for Urban Search and Rescue. NIST Workshop on Performance Metrics for Intelligent Systems 2000. Saarinen l., R. MazI, P. Emest, l. Suomela and L. Preucil. (2004). Sensors and Methods for Human Dead Reckoning. Proceedings of the 8th Conference on Intelligent Autonomous Systems IAS8, Amsterdam. Schilling K. and Q. Meng (2002). The MERLIN vehicles for outdoor applications, SPIE Aerosense conference proceedings "Unmanned Ground Vehicle Technology N", Orlando, USA. Wessel I. (2002). GUI-Design. Carl Hanser Verlag Munich, Germany. ISBN 3-446-21961-7.
7. ACKNOWLEDGEMENTS This work has been supported within the EU-project "PeLoTe - Building Presence through Localization for Hybrid Telematic Systems". The authors acknowledge the contributions and discussions with our consortium partners at the Czech Technical University in Prague, Helsinki Univen;ity of Technology, Certicon a.s. and Steinbeis Transferzentrum ARS.
64