- Email: [email protected]
GIM, TOWARDS THE FUTURE WORKSITE
SAMPLE PAGES TO BE FOLLOWED EXACTLY IN PREPARING SCRIPTS
GIM, TOWARDS THE FUTURE WORKSITE Suomela J 1., Saarinen J 1., Halme A 1, Vilenius M 2., Huhtala K. 2 1
2
GIM/Helsinki University of Technology (TKK), Automation Technology Laboratory GIM/Tampere University of Technology (TUT), Institute of Hydraulics and Automation www.gim.hut.fi,
[email protected]
Abstract: Future Worksite is a concept for robotic work machines working together in a common worksite. Newly founded GIM-Institute studies the subsystems and technologies in the area of field and service robotics. Research results are demonstrated and tested with real machines in the GIM-worksite, which is equipped with several robotic work machines, supporting infrastructure and remote operation station 180km away from the worksite. This paper describes how the infrastructure and communication/control architecture is constructed and the first test results of the long distance teleoperation tests. Copyright © 2007 IFAC Keywords: robot control, communication systems, teleoperation, distributed control, vehicle simulator.
1.INTRODUCTION Robots are slowly moving from factories to mines, construction sites, public places and homes. Vacuum cleaner and lawn mowing robots are already in the home and multipurpose robots like Wakamaru (Mitsubishi Heavy Industries, 2005) are coming. In the industrial site the development has started. The first real robotic systems replacing human driven work machines are already on the market. AutomineTM (Woof, M., 2005) by Sandvik and Autostrad (The Warren Centre, 2004) by Kalmar are good examples of commercial robotic work machines. Automine is a semiautonomous loading and hauling system for underground mines. It includes LHDs and dumpers, which both are capable to navigate autonomously without any additional infrastructure. Only loading and exceptional situations need the help of operator, who supervises - and occasionally teleoperates – several machines from a control room (Fig.1). Autostrad (Fig. 1) - autonomous straddle carrier system is based on a conventional, manned straddle carrier. This vehicle can autonomously pick up, carry and place shipping containers, allowing
movement of containers from land-side vehicles, to holding yards, to quay cranes and back. It was not a surprise that the first robotic work machine applications were underground mines and harbour terminals. Both environments are relatively well structured and slowly changing, which makes the autonomous operation easier comparing to construction sites or forest for example. Additionally in the both environments there is no need to have humans working in the same area with the machines. However, the next step will be in less structured environments in applications like forestry, earth moving, construction sites, janitorial services, etc. Commercial examples prove that the technology starts to be ready. However, there is still a long step before the robotic work machines are spread to all possible worksites. Especially perception, task learning/teaching and human robot interaction are crucial aspects in order to improve the performance of robotic vehicles. Additionally to these “hot topics“ in the robotic research also the other – already functional subsystems
– need improvements. The increasing energy prices generate a good motivation to develop the power generation and transmission systems and even totally new methods to carry out a work task.
Fig. 1. Automine control room and Autostrad Straddle carrier. The newly founded Finnish Center of Excellence (CoE) in Generic Intelligent Machines Research (GIM, www.gim.hut.fi) is focusing to the development of robotic work machines. The CoE period for GIM is 2008-2013 but GIM has started its operation as a research project to build up the infrastructure in the start of 2006. During the CoE period GIM will carry applied basic research projects covering the area of robotic work machines from locomotion systems up to human robotic interfacing. All research results are tested and demonstrated with real hardware in the GIMwork site reflecting the idea of Future Worksite explained in the following chapter. The background of GIM is in strong research tradition of the participating institutes: Automation Technology Laboratory (ATL) in Helsinki University of Technology and Institute of Hydraulics and Automation (IHA) in Tampere University of Technology. Institute is supported by the TEKES Finnish Academy (www.aka.fi), (www.tekes.fi) and the remarkably wide work machine industry in Finland (www.fima.fi).
where interesting research topics are raised and studied. Each RP is a studied by a team consisting of 3-5 senior scientists and several doctoral students. Senior scientists are typically involved in several RPs when doctoral students concentrate on a topic. In the case of field and service robotics the paper research and simulations are good tools but to be really credible the research results have to be proved with real hardware. RP 9, the integrator project, is a tool for this purpose. It includes a “Future worksite” type of infrastructure with several robotic machines and a control architecture, which allows easy (tele)control of the robots. All research results are integrated into the worksite and tested in practise. The basic funding for GIM comes from the Finnish Academy. Cooperation with the industry and international research institutes is promoted with targeted research projects financed by TEKES – Finnish Funding Agency for Technology and Innovation, industry and the EU-Framework Programmes. 3. FUTURE WORKSITE With “Future Worksite” the authors mean any type of worksite where mobile work machines carry their tasks in high level of autonomy (Fig. 2). It doesn’t need to be fully automated. Human workers as well as manually driven machines may also exist there. Local controls
maintenance exceptions
2. GIM RESEARCH GIM collects totally 40 scientists from its host units to carry out research during the years 2007-2013. In a way it’s a big six years long research project. Research is divided in 9 research packages: • • • • • • • • •
RP 1: Internal Re-evaluation and Education RP 2: Modularity of Generic Machines RP 3: Power and Energy Systems RP 4: Transmission of Power and Data RP 5: Control Architectures (task manager and planning system) RP 6: Perception and Navigation Systems RP 7: Human – Robotic Machine Interaction RP 8: Motion systems RP 9: INTEGRATOR project – integrate, validate and demonstrate
As the name indicates the first package is for internal evaluation and education in form of internal seminars, conferences and development meetings. RPs from 2 to 8 include the real research work. Each RP represents roughly a subsystem in a mobile robot. Naturally the research can’t cover each RP area fully but it’s a frame
Worksite
Machines
i n t e r n e t
Remote control room(s)
GIMArchitecture
Operator Model
communication common presence databases
Virtual environment
Production management Task design tools Human workers/ local operators
Fig. 2. Concept of Future Worksite. The idea is that most of the work can be performed with robotic vehicles and vehicle operators are not physically bound to the physical worksite. In fact the system includes three kinds of worksites: the physical work sites, operators’ worksites and virtual worksites where operators and machines meet. Physical worksites can be located globally – or even universally – anywhere. It could be somehow compared to the existing situation with telephone exchanges. A call to a company can end up to different continent than the company is located. After the answer it can be forwarded to a person sitting physically in the company or working in a third continent. This has become possible due to the development in telecommunication. If the development in robotic machines is added – as Automine and Autostrad show this is possible already today – the
operation of a worksite can really be outsourced or locate freely. This opens totally new business possibilities such as huge “call center” type “control centers” in countries with low salary level or machine manufacturers who – instead of machines – would sell only transported containers or ore.
changed to other type, which are more suitable to be used with the proportional valves. Two electrically controlled valves were added to lock the boom and the bucket if needed. This lock option saves power during the operation. In this study proportional valves with load sensing pressure measurement sensors were used (Vilenius et al., 2004).
However, even the future worksite already exists, there is a lot to do before the idea can be applied in all type of worksites such as construction sites, forestry, janitorial services, etc. At the moment the basic level navigation and perception of robotic work machines is sufficient for certain tasks but especially the machine learning, rapid task teaching, human robot interaction and the common situational awareness of the worksite entities have to be improved. 4. GIM WORKSITE The basic infrastructure of the GIM worksite is under development in “Integrator”-project during the years 2006 and 2007. The target is to construct basic infrastructure to serve the GIM research. Worksite includes a 20 m x 20 m test hall with 2-3 robotic work machines and a control/communication architecture allowing teleoperation and supervisory control over the Internet (see Ch. 5). Worksite and the machines are modelled in 3D in order to help the off-line testing of control algorithms.
Fig. 3. Mobile machine. (1) Electronic components, (2) Control and display module, (3) Proportional valve block.
4.1 Existing situation. Two small-scaled, skid steered loaders – Avant 320 (www.avanttecno.com) - are located in the covered worksite in Tampere. The worksite is equipped with supporting sub-systems such as cameras, WLAN communication, etc. A local remote control room is located in the vicinity of the worksite. The “real remote control room”, which basically can be anywhere or even distributed (where internet is available), is located in Espoo. The physical distance between Espoo and Tampere is 180 km. To ease the testing there are also couple of GIM-architecture compatible indoor robots in Espoo. The interface between a machine/robot and the GIM-architecture is independent of the robot structure thus it’s easy to connect different type of machines to the system. The remote machines are modified to be computer controlled. The leading idea in the instrumentation of the machines was to maintain them as a real mobile work machines (robust, environment tolerance, etc). The diesel hydraulic machine (Fig. 3) has three instrumentation parts needed for computer control. Number one includes all electronic components, number two is a control and display module, which runs the main program and number three is the place of the proportional valve block, which is installed in the front of the machine. An electrically controlled valve was added to control the number of hydraulic pumps in action (one or two). Because of the proportional valves, pressure filter was needed. Also, all original hoses were
Fig. 4. Two real machines and one simulated in the worksite. A snapshot from operation screen. At the moment the two Avants can be directly teleoperated over the internet. Additionally to the real machines there is a fully dynamic model of the Avant loader having the same control electronics (HIL) as the real machines. The model can be teleoperated as the real ones and it can also be augmented into the worksite by rendering it to the video image (Fig 4). The worksite is also modelled in 3D with 3D-laser scanner and the model can be used as virtual worksite for the vehicle model and also for the real ones. In the latter case the real machines are located with the aid of the worksite cameras and added into the model. 4.2 Near future actions. In the end of 2007 a third machine based on Avant 600 is added to the system. Comparing to existing machines the new one is bigger and has articulated steering, which makes the control easier comparing to the skid
steered ones. It will also be equipped with revolutionary digital hydraulics (Linjama, M. and Vilenius, M., 2007), which provides better control, efficiency and robustness of the hydraulic power transmission. At the moment the machines have only low level actuator control available and they can only be directly teleoperated without any autonomous functions. All three machines will be equipped with a “piloting unit” providing more computing power and taking care of navigation, perception, task control and communication inside the GIM-architecture. Needed sensors like cameras, laser scanners, IMUs, etc. will also be added. While writing this the piloting unit has just been tested with the simulated machine and the first navigation algorithms (monte-carlo based laser navigation with simple IMU and location based on ceiling camera) have proven to be functional in off-line tests. Piloting and navigation hardware and software will be integrated in the machines and tested during the autumn 2007. 5. GIMNET COMMUNICATION INFRASTRUCTURE The future worksite concept is not only development of robotic hardware. In this case the hardware development has been quite straight forward and there fore the most of the attention is put to the software development, which has a crucial role before the concept will be realized. The robots themselves have complex subsystems, which are interacting together. Additionally the robots are interacting with each others, with humans and with the worksite. This requires a communication infrastructure that enables all different parts of the system to - at least – communicate with each other.
5.1 GIMnet Overview GIMnet (Saarinen et.al. 2007) is a communication infrastructure designed to meet the needs of GIM. Basically, the system is a remote process communication implementation, which additionally functions as a base architecture for the software system. Figure 5 illustrates the basic idea of GIMnet. The hubs are the backbone of the network. A program called tcpHub performs the tasks of a hub. tcpHub is the “Network layer” of the GIMnet architecture. It is responsible for transferring packets from node to node over the GIMnet. The design of GIMnet allows scaling from a single local hub up to a large network of interconnected hubs forming a Virtual Private Network. The only requirement is that one of the hubs has one TCP port open for connections. This type of “ad-hoc” topology allows scalability and extendibility of the system. As an example, the bottom right box in figure 5 represents a machine in the network. There is a separate tcpHub running in the machine, allowing the internal processes to communicate through it, without being affected by the network. When the machine is connected to a larger network, accessing remote modules are not different – from programming point of view – from accessing local ones. Thus some of the machine’s functions can be moved outside the machine without any change in the programming. Due to this more computers can be easily added to the system to share the computing load. For research and development the system offers the possibility to develop and test software on one’s own desktop while keeping it connected to the rest of the system.
Some of the very basic requirements for the infrastructure are: • easy communication between all entities connected into the system • unlimited amount of robots and other working entities as well as control points • flexible connection of different type of machines • fast response times to allow closed loop teleoperation (if the used network allows) Additionally the infrastructure must tolerate and support complex systems. The multi-robot (tele-)control system is highly complex from the software point of view. The infrastructure must allow modular development to ease the design of the system. Also, the modularity encourages to code reuse in many cases. Finally the infrastructure must be usable. The usability means that people can use it (can make systems with it), the infrastructure works and the system using the infrastructure works. For example the firewalls nowadays are a pain in telecontrol. An infrastructure that does not take this into acount is unusable as a general solution.
Fig. 5. Principle of the GIMnet The software modules in figure 5 can be considered clients for the hub. The modules are separate processes running anywhere in the network. When connecting to the hub, the modules register their name and receive an ID, both of which can be later used to address the module.
The Network Interface (NIF) encapsulates the low level communication protocols. It wraps the OS-specific socket code (Unix/Linux and Windows) and the bitlevel protocol used by tcpHub into an OS-independent API, presenting all the basic operations like connecting to the hub, sending and receiving data packets, and querying peer IDs by name. The Generic Intelligent Machine Interface (GIMI) can be considered as an application layer, which provides an easily accessible API for the module developers. GIMI encapsulates the network so that the developers do not need to know the underlying structure; they only use the simple function interface. It also implements some important functions, which are required by almost all modules. Below is listed the main features of the GIMnet: • • • • • •
hardware abstraction layer. It is specifically designed to work with GIMnet. ASRobo schematics are illustrated in the figure 6. ASRobo allows multiple clients to connect to the ASRobo server, which acts as a communication bridge between the GIMnet and the robot devices. The device drivers have the abstract interface towards the clients. The bus drivers are used to share the robot resources. The modular structure and service based communication supports modular user interface developement. Each service is a pre-specified (“standard”) data interface. The user interface can offer components for each service providing virtually a possibility to control and observe any machine or system. This makes also the flexible login/out of different types of machines easy.
Distributed name and ID service Unicast, multicast, broadcast Synchronized and unsynchronized data transmission Automatic hub-to-hub and client-to-hub reconnect Service registration, subscription and listing Application level ping
6. BEYOND THE INFRASTRUCTURE The GIMnet provides a good base for the system software development but doesn’t alone fulfil all the requirements discussed.
Fig. 7. The modular user interface prototype
The machines in the worksite require control software. Due to the fact that different types of machines have different hardware, the robot control software has always a platform dependent part. However, on top of the machine software it’s possible to build a platform independent part acting as generic robot control software. In GIM this generic part is called ASRobo.
Figure 7 shows the prototype of the user interface. The user interface lists all the services available and the user can open the components he/she wants to use. The service discovery also has the location of the service specified (either user defined or IP based). This allows the interface to sort the available services in various ways. 7. THE FIRST RESULTS Until now the tests have been concentrated on the practical functionality of the basic infrastructure and the performance of the communication architecture. Two mobile machines (not yet robots) can be easily teleoperated over the GIMnet (and internet). Worksite and dynamic vehicle models work and can be augmented with real machines, which can be located with ceiling camera.
Fig. 6. ASRobo principle schematics ASRobo allows controlling robots with diverse hardware through a coherent interface; in essence, it’s a
GIMnet communication over internet was measured in order to find out the average system performance between Espoo and Tampere. In Internet the communication delays and efficient bandwidth can’t be predicted. However, the average quality of the network can be measured between two constant locations due to the fact that the basic performance of the network is
depending on connections between the communicating parties. Test results of long period continuous pinging between Espoo and Tampere are shown in figure 8. The left figure shows the histogram of the round-trip time of the TCP/IP packets sent between Espoo and Tampere. 99,5% of the packets have round trip time less than 10ms, the average being 4,91ms. The right figure shows the time the network has not been responsive within 200ms. During a week’s test there were two longer periods that the packets were not delivered. The longest time the network was down was 18 seconds and second longest 14s.
Fig. 8. The results of the network tests. On the left is illustrated a histogram of the ping times within one week (y-axis is percentages, x-axis is milliseconds). On the right is the network failures during the week (y-axis is seconds and x-axis is samples).
The practical direct teleoperation trials proved that both the communication architecture, communication and the machines were working well without problems and interrupts. The basic infrastructure is ready for the next development steps. 8. CONCLUSIONS GIM-Institute is a Finnish research centre focusing to the field and service robotics. Research covers all subsystems of a mobile robot from locomotion to task learning and human robot interaction. All research results will be demonstrated and integrated into a multi machine test system, which will gradually develop towards Future Worksite concept. Future Worksite includes a physical worksite with autonomous and semiautonomous machines, which are controlled from a remote control station. Operator and worksite are connected with a virtual worksite, where the work tasks are planned and the progress of the work can be followed. At the moment the basic infrastructure including the robotic machines, worksite, user interface and communication architecture is under work. Results from the first communication tests show that the communication architecture and low level control of the work machines are performing well forming a good basis for future development. Research work will be continued in cooperation with work machine industry and international research partners. Institute has basic funding and preliminary research plan to the end of 2013. REFERENCES
Fig. 9. Application level tests. On the left is round-trip time between client and local server. On the right is the histogram of the time spent in route clientremote server-local server-client. Y-axis is percentages (of the packets), X-axis is time in milliseconds. Similar tests were made also in the application level. The purpose was to show that the software does not cause lag or unpredictability in the system. The results can be seen in figure 9. The left side image shows that the local connection is very fast. Almost all the packets have round-trip time less than 0.8ms. This includes the application layer. The second (right-side) image illustrates the round-trip times of the route clientremote server-local server-client. This is practically the route the data travels during teleoperation. The payload was chosen to be 512B, which is significantly more than the average control data (however smaller than image stream). The average round-trip time was 7,46ms and almost all the packets were received within 10ms (99,0% within 10.28ms). Comparision between the bare network tests and the application level tests show that the base infrastructure caused approximately 1,88ms more delay.
Mitsubishi Heavy Industries, Wakamaru, http://www.mhi.co.jp/kobe/wakamaru/english/, updated 2005. The Warren Centre, http://www.warren.usyd.edu.au/bulletin/NO39/ed3 9art1.htm , e-bulletin, ISSUE 39, September 2004. Woof, M., Technology for Underground Loading and Hauling Systems Offers Exciting Prospects, Engineering and Mining Journal, April 2005. Vilenius J., Raneda A., Hyvönen M., Uusisalo J., Huhtala K., Valve Controlled Teleoperated Skid Steered Mobile Machine, Proc. Of IMECE 2004, Anaheim, USA, 2004. Linjama, M. and Vilenius, M. 2007. Digital Hydraulics – Towards perfect valve technology. Proceedings of the Tenth Scandinavian International Conference on Fluid Power, SICFP’07, May 21-23 2007, Vol 1, pp. 181-196. Jari Saarinen, Antti Maula, Renne Nissinen, Harri Kukkonen, Jussi Suomela and Aarne Halme, Gimnet - Infrastructure For Distributed Control Of Generic Intelligent Machines, The 13th IASTED International Conference on Telematics, August 29-31, 2007, Würzburg, Germany
GIM, TOWARDS THE FUTURE WORKSITE Suomela J 1., Saarinen J 1., Halme A 1, Vilenius M 2., Huhtala K. 2 1
2
GIM/Helsinki University of Technology (TKK), Automation Technology Laboratory GIM/Tampere University of Technology (TUT), Institute of Hydraulics and Automation www.gim.hut.fi,
[email protected]
Abstract: Future Worksite is a concept for robotic work machines working together in a common worksite. Newly founded GIM-Institute studies the subsystems and technologies in the area of field and service robotics. Research results are demonstrated and tested with real machines in the GIM-worksite, which is equipped with several robotic work machines, supporting infrastructure and remote operation station 180km away from the worksite. This paper describes how the infrastructure and communication/control architecture is constructed and the first test results of the long distance teleoperation tests. Copyright © 2007 IFAC Keywords: robot control, communication systems, teleoperation, distributed control, vehicle simulator.
1.INTRODUCTION Robots are slowly moving from factories to mines, construction sites, public places and homes. Vacuum cleaner and lawn mowing robots are already in the home and multipurpose robots like Wakamaru (Mitsubishi Heavy Industries, 2005) are coming. In the industrial site the development has started. The first real robotic systems replacing human driven work machines are already on the market. AutomineTM (Woof, M., 2005) by Sandvik and Autostrad (The Warren Centre, 2004) by Kalmar are good examples of commercial robotic work machines. Automine is a semiautonomous loading and hauling system for underground mines. It includes LHDs and dumpers, which both are capable to navigate autonomously without any additional infrastructure. Only loading and exceptional situations need the help of operator, who supervises - and occasionally teleoperates – several machines from a control room (Fig.1). Autostrad (Fig. 1) - autonomous straddle carrier system is based on a conventional, manned straddle carrier. This vehicle can autonomously pick up, carry and place shipping containers, allowing
movement of containers from land-side vehicles, to holding yards, to quay cranes and back. It was not a surprise that the first robotic work machine applications were underground mines and harbour terminals. Both environments are relatively well structured and slowly changing, which makes the autonomous operation easier comparing to construction sites or forest for example. Additionally in the both environments there is no need to have humans working in the same area with the machines. However, the next step will be in less structured environments in applications like forestry, earth moving, construction sites, janitorial services, etc. Commercial examples prove that the technology starts to be ready. However, there is still a long step before the robotic work machines are spread to all possible worksites. Especially perception, task learning/teaching and human robot interaction are crucial aspects in order to improve the performance of robotic vehicles. Additionally to these “hot topics“ in the robotic research also the other – already functional subsystems
– need improvements. The increasing energy prices generate a good motivation to develop the power generation and transmission systems and even totally new methods to carry out a work task.
Fig. 1. Automine control room and Autostrad Straddle carrier. The newly founded Finnish Center of Excellence (CoE) in Generic Intelligent Machines Research (GIM, www.gim.hut.fi) is focusing to the development of robotic work machines. The CoE period for GIM is 2008-2013 but GIM has started its operation as a research project to build up the infrastructure in the start of 2006. During the CoE period GIM will carry applied basic research projects covering the area of robotic work machines from locomotion systems up to human robotic interfacing. All research results are tested and demonstrated with real hardware in the GIMwork site reflecting the idea of Future Worksite explained in the following chapter. The background of GIM is in strong research tradition of the participating institutes: Automation Technology Laboratory (ATL) in Helsinki University of Technology and Institute of Hydraulics and Automation (IHA) in Tampere University of Technology. Institute is supported by the TEKES Finnish Academy (www.aka.fi), (www.tekes.fi) and the remarkably wide work machine industry in Finland (www.fima.fi).
where interesting research topics are raised and studied. Each RP is a studied by a team consisting of 3-5 senior scientists and several doctoral students. Senior scientists are typically involved in several RPs when doctoral students concentrate on a topic. In the case of field and service robotics the paper research and simulations are good tools but to be really credible the research results have to be proved with real hardware. RP 9, the integrator project, is a tool for this purpose. It includes a “Future worksite” type of infrastructure with several robotic machines and a control architecture, which allows easy (tele)control of the robots. All research results are integrated into the worksite and tested in practise. The basic funding for GIM comes from the Finnish Academy. Cooperation with the industry and international research institutes is promoted with targeted research projects financed by TEKES – Finnish Funding Agency for Technology and Innovation, industry and the EU-Framework Programmes. 3. FUTURE WORKSITE With “Future Worksite” the authors mean any type of worksite where mobile work machines carry their tasks in high level of autonomy (Fig. 2). It doesn’t need to be fully automated. Human workers as well as manually driven machines may also exist there. Local controls
maintenance exceptions
2. GIM RESEARCH GIM collects totally 40 scientists from its host units to carry out research during the years 2007-2013. In a way it’s a big six years long research project. Research is divided in 9 research packages: • • • • • • • • •
RP 1: Internal Re-evaluation and Education RP 2: Modularity of Generic Machines RP 3: Power and Energy Systems RP 4: Transmission of Power and Data RP 5: Control Architectures (task manager and planning system) RP 6: Perception and Navigation Systems RP 7: Human – Robotic Machine Interaction RP 8: Motion systems RP 9: INTEGRATOR project – integrate, validate and demonstrate
As the name indicates the first package is for internal evaluation and education in form of internal seminars, conferences and development meetings. RPs from 2 to 8 include the real research work. Each RP represents roughly a subsystem in a mobile robot. Naturally the research can’t cover each RP area fully but it’s a frame
Worksite
Machines
i n t e r n e t
Remote control room(s)
GIMArchitecture
Operator Model
communication common presence databases
Virtual environment
Production management Task design tools Human workers/ local operators
Fig. 2. Concept of Future Worksite. The idea is that most of the work can be performed with robotic vehicles and vehicle operators are not physically bound to the physical worksite. In fact the system includes three kinds of worksites: the physical work sites, operators’ worksites and virtual worksites where operators and machines meet. Physical worksites can be located globally – or even universally – anywhere. It could be somehow compared to the existing situation with telephone exchanges. A call to a company can end up to different continent than the company is located. After the answer it can be forwarded to a person sitting physically in the company or working in a third continent. This has become possible due to the development in telecommunication. If the development in robotic machines is added – as Automine and Autostrad show this is possible already today – the
operation of a worksite can really be outsourced or locate freely. This opens totally new business possibilities such as huge “call center” type “control centers” in countries with low salary level or machine manufacturers who – instead of machines – would sell only transported containers or ore.
changed to other type, which are more suitable to be used with the proportional valves. Two electrically controlled valves were added to lock the boom and the bucket if needed. This lock option saves power during the operation. In this study proportional valves with load sensing pressure measurement sensors were used (Vilenius et al., 2004).
However, even the future worksite already exists, there is a lot to do before the idea can be applied in all type of worksites such as construction sites, forestry, janitorial services, etc. At the moment the basic level navigation and perception of robotic work machines is sufficient for certain tasks but especially the machine learning, rapid task teaching, human robot interaction and the common situational awareness of the worksite entities have to be improved. 4. GIM WORKSITE The basic infrastructure of the GIM worksite is under development in “Integrator”-project during the years 2006 and 2007. The target is to construct basic infrastructure to serve the GIM research. Worksite includes a 20 m x 20 m test hall with 2-3 robotic work machines and a control/communication architecture allowing teleoperation and supervisory control over the Internet (see Ch. 5). Worksite and the machines are modelled in 3D in order to help the off-line testing of control algorithms.
Fig. 3. Mobile machine. (1) Electronic components, (2) Control and display module, (3) Proportional valve block.
4.1 Existing situation. Two small-scaled, skid steered loaders – Avant 320 (www.avanttecno.com) - are located in the covered worksite in Tampere. The worksite is equipped with supporting sub-systems such as cameras, WLAN communication, etc. A local remote control room is located in the vicinity of the worksite. The “real remote control room”, which basically can be anywhere or even distributed (where internet is available), is located in Espoo. The physical distance between Espoo and Tampere is 180 km. To ease the testing there are also couple of GIM-architecture compatible indoor robots in Espoo. The interface between a machine/robot and the GIM-architecture is independent of the robot structure thus it’s easy to connect different type of machines to the system. The remote machines are modified to be computer controlled. The leading idea in the instrumentation of the machines was to maintain them as a real mobile work machines (robust, environment tolerance, etc). The diesel hydraulic machine (Fig. 3) has three instrumentation parts needed for computer control. Number one includes all electronic components, number two is a control and display module, which runs the main program and number three is the place of the proportional valve block, which is installed in the front of the machine. An electrically controlled valve was added to control the number of hydraulic pumps in action (one or two). Because of the proportional valves, pressure filter was needed. Also, all original hoses were
Fig. 4. Two real machines and one simulated in the worksite. A snapshot from operation screen. At the moment the two Avants can be directly teleoperated over the internet. Additionally to the real machines there is a fully dynamic model of the Avant loader having the same control electronics (HIL) as the real machines. The model can be teleoperated as the real ones and it can also be augmented into the worksite by rendering it to the video image (Fig 4). The worksite is also modelled in 3D with 3D-laser scanner and the model can be used as virtual worksite for the vehicle model and also for the real ones. In the latter case the real machines are located with the aid of the worksite cameras and added into the model. 4.2 Near future actions. In the end of 2007 a third machine based on Avant 600 is added to the system. Comparing to existing machines the new one is bigger and has articulated steering, which makes the control easier comparing to the skid
steered ones. It will also be equipped with revolutionary digital hydraulics (Linjama, M. and Vilenius, M., 2007), which provides better control, efficiency and robustness of the hydraulic power transmission. At the moment the machines have only low level actuator control available and they can only be directly teleoperated without any autonomous functions. All three machines will be equipped with a “piloting unit” providing more computing power and taking care of navigation, perception, task control and communication inside the GIM-architecture. Needed sensors like cameras, laser scanners, IMUs, etc. will also be added. While writing this the piloting unit has just been tested with the simulated machine and the first navigation algorithms (monte-carlo based laser navigation with simple IMU and location based on ceiling camera) have proven to be functional in off-line tests. Piloting and navigation hardware and software will be integrated in the machines and tested during the autumn 2007. 5. GIMNET COMMUNICATION INFRASTRUCTURE The future worksite concept is not only development of robotic hardware. In this case the hardware development has been quite straight forward and there fore the most of the attention is put to the software development, which has a crucial role before the concept will be realized. The robots themselves have complex subsystems, which are interacting together. Additionally the robots are interacting with each others, with humans and with the worksite. This requires a communication infrastructure that enables all different parts of the system to - at least – communicate with each other.
5.1 GIMnet Overview GIMnet (Saarinen et.al. 2007) is a communication infrastructure designed to meet the needs of GIM. Basically, the system is a remote process communication implementation, which additionally functions as a base architecture for the software system. Figure 5 illustrates the basic idea of GIMnet. The hubs are the backbone of the network. A program called tcpHub performs the tasks of a hub. tcpHub is the “Network layer” of the GIMnet architecture. It is responsible for transferring packets from node to node over the GIMnet. The design of GIMnet allows scaling from a single local hub up to a large network of interconnected hubs forming a Virtual Private Network. The only requirement is that one of the hubs has one TCP port open for connections. This type of “ad-hoc” topology allows scalability and extendibility of the system. As an example, the bottom right box in figure 5 represents a machine in the network. There is a separate tcpHub running in the machine, allowing the internal processes to communicate through it, without being affected by the network. When the machine is connected to a larger network, accessing remote modules are not different – from programming point of view – from accessing local ones. Thus some of the machine’s functions can be moved outside the machine without any change in the programming. Due to this more computers can be easily added to the system to share the computing load. For research and development the system offers the possibility to develop and test software on one’s own desktop while keeping it connected to the rest of the system.
Some of the very basic requirements for the infrastructure are: • easy communication between all entities connected into the system • unlimited amount of robots and other working entities as well as control points • flexible connection of different type of machines • fast response times to allow closed loop teleoperation (if the used network allows) Additionally the infrastructure must tolerate and support complex systems. The multi-robot (tele-)control system is highly complex from the software point of view. The infrastructure must allow modular development to ease the design of the system. Also, the modularity encourages to code reuse in many cases. Finally the infrastructure must be usable. The usability means that people can use it (can make systems with it), the infrastructure works and the system using the infrastructure works. For example the firewalls nowadays are a pain in telecontrol. An infrastructure that does not take this into acount is unusable as a general solution.
Fig. 5. Principle of the GIMnet The software modules in figure 5 can be considered clients for the hub. The modules are separate processes running anywhere in the network. When connecting to the hub, the modules register their name and receive an ID, both of which can be later used to address the module.
The Network Interface (NIF) encapsulates the low level communication protocols. It wraps the OS-specific socket code (Unix/Linux and Windows) and the bitlevel protocol used by tcpHub into an OS-independent API, presenting all the basic operations like connecting to the hub, sending and receiving data packets, and querying peer IDs by name. The Generic Intelligent Machine Interface (GIMI) can be considered as an application layer, which provides an easily accessible API for the module developers. GIMI encapsulates the network so that the developers do not need to know the underlying structure; they only use the simple function interface. It also implements some important functions, which are required by almost all modules. Below is listed the main features of the GIMnet: • • • • • •
hardware abstraction layer. It is specifically designed to work with GIMnet. ASRobo schematics are illustrated in the figure 6. ASRobo allows multiple clients to connect to the ASRobo server, which acts as a communication bridge between the GIMnet and the robot devices. The device drivers have the abstract interface towards the clients. The bus drivers are used to share the robot resources. The modular structure and service based communication supports modular user interface developement. Each service is a pre-specified (“standard”) data interface. The user interface can offer components for each service providing virtually a possibility to control and observe any machine or system. This makes also the flexible login/out of different types of machines easy.
Distributed name and ID service Unicast, multicast, broadcast Synchronized and unsynchronized data transmission Automatic hub-to-hub and client-to-hub reconnect Service registration, subscription and listing Application level ping
6. BEYOND THE INFRASTRUCTURE The GIMnet provides a good base for the system software development but doesn’t alone fulfil all the requirements discussed.
Fig. 7. The modular user interface prototype
The machines in the worksite require control software. Due to the fact that different types of machines have different hardware, the robot control software has always a platform dependent part. However, on top of the machine software it’s possible to build a platform independent part acting as generic robot control software. In GIM this generic part is called ASRobo.
Figure 7 shows the prototype of the user interface. The user interface lists all the services available and the user can open the components he/she wants to use. The service discovery also has the location of the service specified (either user defined or IP based). This allows the interface to sort the available services in various ways. 7. THE FIRST RESULTS Until now the tests have been concentrated on the practical functionality of the basic infrastructure and the performance of the communication architecture. Two mobile machines (not yet robots) can be easily teleoperated over the GIMnet (and internet). Worksite and dynamic vehicle models work and can be augmented with real machines, which can be located with ceiling camera.
Fig. 6. ASRobo principle schematics ASRobo allows controlling robots with diverse hardware through a coherent interface; in essence, it’s a
GIMnet communication over internet was measured in order to find out the average system performance between Espoo and Tampere. In Internet the communication delays and efficient bandwidth can’t be predicted. However, the average quality of the network can be measured between two constant locations due to the fact that the basic performance of the network is
depending on connections between the communicating parties. Test results of long period continuous pinging between Espoo and Tampere are shown in figure 8. The left figure shows the histogram of the round-trip time of the TCP/IP packets sent between Espoo and Tampere. 99,5% of the packets have round trip time less than 10ms, the average being 4,91ms. The right figure shows the time the network has not been responsive within 200ms. During a week’s test there were two longer periods that the packets were not delivered. The longest time the network was down was 18 seconds and second longest 14s.
Fig. 8. The results of the network tests. On the left is illustrated a histogram of the ping times within one week (y-axis is percentages, x-axis is milliseconds). On the right is the network failures during the week (y-axis is seconds and x-axis is samples).
The practical direct teleoperation trials proved that both the communication architecture, communication and the machines were working well without problems and interrupts. The basic infrastructure is ready for the next development steps. 8. CONCLUSIONS GIM-Institute is a Finnish research centre focusing to the field and service robotics. Research covers all subsystems of a mobile robot from locomotion to task learning and human robot interaction. All research results will be demonstrated and integrated into a multi machine test system, which will gradually develop towards Future Worksite concept. Future Worksite includes a physical worksite with autonomous and semiautonomous machines, which are controlled from a remote control station. Operator and worksite are connected with a virtual worksite, where the work tasks are planned and the progress of the work can be followed. At the moment the basic infrastructure including the robotic machines, worksite, user interface and communication architecture is under work. Results from the first communication tests show that the communication architecture and low level control of the work machines are performing well forming a good basis for future development. Research work will be continued in cooperation with work machine industry and international research partners. Institute has basic funding and preliminary research plan to the end of 2013. REFERENCES
Fig. 9. Application level tests. On the left is round-trip time between client and local server. On the right is the histogram of the time spent in route clientremote server-local server-client. Y-axis is percentages (of the packets), X-axis is time in milliseconds. Similar tests were made also in the application level. The purpose was to show that the software does not cause lag or unpredictability in the system. The results can be seen in figure 9. The left side image shows that the local connection is very fast. Almost all the packets have round-trip time less than 0.8ms. This includes the application layer. The second (right-side) image illustrates the round-trip times of the route clientremote server-local server-client. This is practically the route the data travels during teleoperation. The payload was chosen to be 512B, which is significantly more than the average control data (however smaller than image stream). The average round-trip time was 7,46ms and almost all the packets were received within 10ms (99,0% within 10.28ms). Comparision between the bare network tests and the application level tests show that the base infrastructure caused approximately 1,88ms more delay.
Mitsubishi Heavy Industries, Wakamaru, http://www.mhi.co.jp/kobe/wakamaru/english/, updated 2005. The Warren Centre, http://www.warren.usyd.edu.au/bulletin/NO39/ed3 9art1.htm , e-bulletin, ISSUE 39, September 2004. Woof, M., Technology for Underground Loading and Hauling Systems Offers Exciting Prospects, Engineering and Mining Journal, April 2005. Vilenius J., Raneda A., Hyvönen M., Uusisalo J., Huhtala K., Valve Controlled Teleoperated Skid Steered Mobile Machine, Proc. Of IMECE 2004, Anaheim, USA, 2004. Linjama, M. and Vilenius, M. 2007. Digital Hydraulics – Towards perfect valve technology. Proceedings of the Tenth Scandinavian International Conference on Fluid Power, SICFP’07, May 21-23 2007, Vol 1, pp. 181-196. Jari Saarinen, Antti Maula, Renne Nissinen, Harri Kukkonen, Jussi Suomela and Aarne Halme, Gimnet - Infrastructure For Distributed Control Of Generic Intelligent Machines, The 13th IASTED International Conference on Telematics, August 29-31, 2007, Würzburg, Germany