Maintenance and Telematics for Robots (MainTelRob)

3rd IFAC Symposium on Telematics Applications The International Federation of Automatic Control November 11-13, 2013. Seoul, Korea

Maintenance and Telematics for Robots (MainTelRob) Felix Sittner ∗ Doris Aschenbrenner ∗ Michael Fritscher ∗ Ali Kheirkhah ∗ Markus Krauß ∗ Klaus Schilling ∗ ∗

Zentrum f¨ ur Telematik, Gerbrunn, 97218 Germany (Tel: 0049-931-3292954-{15;25;21;26;13;10}; e-mail: [email protected])

Abstract: This paper informs about a current challenge in industrial research and our first approaches to address this topic. Industrial robots are shipped to facilities all over the world and the manufacturers’ engineers often need to travel far to fix errors. To reduce downtimes and costs, plant manufacturers need easy to use and comprehensive telemaintenance services to support repair and maintenance tasks at remote facilities. We explain the requirements for such a novel remote maintenance system and the exemplary workflows we and our industrial partners aim to improve. We give a short survey of the state of the art in the fields of networked-control and augmented reality, followed by a description of our own ideas for adaptive control over networks and improved situation awareness. Subsequently, we present our first system designs and the testbed we created to verify our approaches during further development. Keywords: Control through networks, remote sensor data acquisition, telematic methods 1. INTRODUCTION The huge progress in the field of telecommunication and information technology allows to deploy increasingly ambitious services, which will help to reduce the need for costly long-distance travels. In combination with the control and automation technology in telematics, this offers novel possibilities to gather sensor and status data from remote industrial plants and even, if appropriate, to respond to real-time critical situations. Although simple telematics applications have been on the market for several years, many interesting research problems need yet to be solved to improve the reliability and efficiency of facilities through interactive services. The Zentrum f¨ ur Telematik e.V. (ZfT) and its industrial partners, Reis Robotics from Obernburg and Braun/Procter & Gamble Marktheidenfeld, cooperate on MainTelRob to solve these problems and create a comprehensive new telemaintenance system. 1.1 Project Goals In MainTelRob, we aim to employ user-friendly augmented reality techniques to intuitively represent a remote situation to an operator, e.g., by visualizing deviations. In addition, autonomous reactions and remote control are combined in order to absorb peak loads and enable injury prevention, especially during critical phases of operation. Both above mentioned essential elements of a user friendly telemaintenance system will be closely integrated to facilitate the remote maintenance of complex automation facilities. The project set-up and concrete tasks are divided into two main areas, which we explain based on Figure 1: ? This work is funded by the Bavarian Ministry of Economic Affairs, Infrastructure, Transport and Technology in its R&D program ”Information and Communication Technology”

978-3-902823-56-4/2013 © IFAC

113

Fig. 1. Overview of MainTelRob The robot and the local service technicians are situated at the production site of the customer, as depicted in the lower half. The robot is currently supervised by a condition monitoring system, which will be further developed within this project. The service technicians, possibly with differing qualifications, will be supported by a mobile environment. The assistance functions this mobile environment will provide will be researched and developed within MainTelRob. Our industrial partner Reis Robotics provides the telemaintenance center, depicted in the upper half of Figure 1, as a part of this project. The experts located there can use their telemaintenance environment to access the robot at the customer facility and communicate with the local service technicians. We will create various tools for them, so that they can help to solve problems without physically traveling to the factory. 10.3182/20131111-3-KR-2043.00010

IFAC TA 2013 November 11-13, 2013. Seoul, Korea

energy consumption. Here an expert appraisal comes in hand but it is too expensive in most cases. We will develop a remote interface through which an external expert can be consulted. Repair with external help For difficult repairs, advice by specialists from the robot manufacturing company is needed. The project will provide a maximum amount of situation awareness for these external experts, with the help of remote-control, video communication and augmented reality. We regard the use-case of motor exchange on a robot, a process currently not possible to support via telemaintenance. 2. STATE OF THE ART AND RESEARCH PROGRESS

Fig. 2. Injection molding machine 52 with Reis robotic arm 1.2 Industrial Setting and Scenarios During the requirements engineering process, the team focused on real scenarios, in which the established maintenance processes can be enhanced by telematic methods. We regarded the real machine depicted in Figure 2, which is installed at the Braun/P&G facility in Marktheidenfeld. It contains a robotic arm that manufactures the plastic base of electric toothbrushes by injection molding. One main task in this industrial research project was to identify modes of operation that measurably improve predictive maintenance. In this section, we give a brief overview of the six identified scenarios and their associated use-cases. Planned cyclic maintenance with facility downtime Frequent maintenance measurements are planned in regular intervals following manufacturers’ instructions. While different checklists are currently generated in a database, printed and processed by a service technician, this will be possible in the future with the help of the mobile device. Manual cyclic control on the enabled facility A machine operator collects the produced parts from the facility following a defined walking route. On this route the operator has to survey the status of the running facility. We will create a visual control tool for the mobile device, to guide the operator through these recurring tasks. Repair without external help In case the facility is stopped due to a mechanical breakdown, a service technician needs to find the cause of the failure and repair it. We develop a support device for the debugging routine and the error correction, i.e. by giving pictorial instructions for detailed working steps. Foresightful maintenance The facilities run for long periods. Many failures do not occur suddenly, breakdowns mostly happen as results of creeping processes like abrasion. We aim to improve the condition monitoring, so that these processes can be detected and resolved. In the project we focus on monitoring the mechanical motor load. Optimization with external help Service technicians try to optimize the facility regarding cycle times, material or 114

In this section, we give a short state-of-the-art and outline, which actual research findings and ideas we employ to facilitate the maintenance use-cases described above. First, we describe our ideas on network based robot control, which is needed in the latter three scenarios. Second, we survey the current state of Augmented Reality (AR) and which features we intend to use. And third, we present our first draft for system integration and security measures. 2.1 Control Feedback control systems are already widely applied in industry. Today, it is common for some applications to use network systems in the control loop, which means that a spatially separated controller and a plant are interconnected through networks. Such control loops are more flexible and modular. In addition, they allow novel applications, which need a separate controller, e.g. due to energy or room constraints. Examples of such systems are ¨ mobile sensor networks [Ogren et al., 2004], robot-assisted operations [Meng et al., 2004] and autonomous mobile robots [Seiler and Sengupta, 2001]. In MainTelRob, the following tasks are considered for the controllers: Bandwidth assignment The network path between the the remote expert and the local technician cannot be regarded as a stable line. Instead, a controller is needed to assign the available bandwidth and delay constraints to the individual services like video/audio-streaming, controlling/syncing the components and the link. Here, the network is a part of the plant and the services which are running over this network. Robot Control To ensure the stability and safe performance of a robot which is controlled through a network connection, the robot control system has to cope with delays, jitter and packet loss: Examples are reducing the velocity automatically, if the network conditions get worse and implementing transformations between the different ways of controlling the robot. Another concept is to prevent running into emergency-stop situations, e.g., by driving the arm into barriers defined by the security controller or by limiting the velocity and warning the operator in advance. We also aim to package or transform the data packets sent by the robot-steering into more network-affine variants and filter out unneeded data to provide more consistent and smaller data streams.

IFAC TA 2013 November 11-13, 2013. Seoul, Korea

Plant Actuators

System

Sensors

Network

Controller

Fig. 3. Overview of control networks Estimation of the movements and status changes of the robot in case of (short) network breakdowns In this situation, the system must decide whether prediction of the movement is possible or not (e.g. in case of not enough recent data). If it is possible, the forecast can be done using the model and known states. These approaches can be combined to get a whole local simulation of the machine running, without losing the ability to control it. In addition, complex calculations could be provided by the external center to avoid the need of extra hardware on the machine. Delay compensation One of the problems induced by networks in a control loop like in Fig. 3 is the time delay between the plant and the controller which can be dealt with in different ways: One approach is to fix the controller and try to choose and configure the network such that it fulfills the needs of the controller [Kweon et al., 2000]. This works only in case of stable local networks. Another solution, best for connections with short and constant delay, is to introduce a filter before the controller, which compensates the disruptions of the network [Liu and Goldsmith, 2004]. Approach For MainTelRob the controller has to be adjusted or even newly designed due to the random delay of the Internet, which connects the plant and the controller. In this case, a model for the network and its attributes must be found. In most works, the network is modeled as a binomial process which does not take delays into account, but only whether the packet is transferred successfully or not. Additionally, every packet is independent from others [Tatikonda, 2000]. Because of its simplicity, this model is useful for getting some fundamental results, e.g., for stability analysis [Seiler and Sengupta, 2003]. But the network can also be regarded as a delay element that issues predictable delays in the closed loop. There are proven approaches to treat this kind of control loops, like the Smith-Predictor, which treats the delay element as non-minimal-phase part of the plant and applies kind of an internal model control (IMC) for non-minimal-phase systems. To build a more realistic model of the network, the latency of the delay element will not be assumed as constant, but adopted to the current measurements and estimations in real-time [Astrom et al., 1994]. A more accurate model can be built using Markov chains, often in the first order, in which the next state depends on the current state. In some cases, it is beneficial to use higher order Markov chains. This approach is more reliable, because the Internet delays can also depend on the network load. A shared connection can, e.g., appear slower during a huge download. The work of Peter 115

Fig. 4. Reality-virtuality continuum for MainTelRob Seiler [Seiler, 2001], De Farias et. al [De Farias et al., 2000] and Costa [Costa et al., 2005] are examples of such an approach. Often a combination of an optimal state feedback, e.g., linear quadratic gaussan (LQG) and an optimal estimator like the Kalman-Filter is used to achieve an optimal controller in a closed loop with the Internet. A pioneer in this area is Nilsson [Nilsson et al., 1996, Nilsson and Bernhardsson, 1997]. The stability of such closed loops has been analyzed by [Zhang et al., 2001] and [Branicky et al., 2000]. 2.2 Augmented Reality While the term ”Virtual Reality” (VR) is already widely known, the demarcation of ”Augmented Reality” (AR) remains still unclear for many people, especially in the area of industrial applications [Mehler-Bicher et al., 2011]. AR enables the enhancement of normal perception with artificial information, while VR is a complete artificial construction. Consequently there is a ”reality-virtuality continuum” in which a consistent transition between real and virtual environment takes place [Milgram et al., 1995]. In Figure 4 this ”reality-virtuality-continuum” is depicted for the MainTelRob project context. On the left side is the real environment of the facility. For construction and programming, a pure virtual CAD model is already used which is shown on the right side. The mixture of both worlds is augmented reality. One task for the MainTelRob project will be to determine where exactly a solution has to be located in that continuum. AR-related problems in MainTelRob Considering the project scenarios described above, the main challenge will be to find the right point on the continuum for each scenario independently. That means answering the question how the scenario can be supported with augmented reality in order to provide a measurable benefit for every day work. We consider that ”situation awareness” as a crucial factor, which estimates whether the user realizes the actual situation, can interpret it correctly and project it to the near future. In theory, the whole research field of human machine interaction and software ergonomics is concerned. The ZfT has already done a lot of research on this topic [Driewer, 2009, Sauer, 2010] focusing explicitly on the human robot interaction. State of the Art The Gartner ”hype cycle” of 2009 [Fenn et al., 2009] depicts the adaption horizon for AR between five and ten years. Today half of the time has passed - and

IFAC TA 2013 November 11-13, 2013. Seoul, Korea

AR appears to be ready for a consumer electronic market. Thanks to the evolution of mobile devices like smartphones, former expensive products like head-mounted displays (HMD) are now possible for the mass market, e.g. Google Glass [Goo]. It is time to adopt these advances also for the fourth level of industrial revolution - Industry 4.0 where many different applications are considered not only possible but also as an opportunity to reduce costs [Ludwig and Reimann, 2005]. A complete survey of the research issues in AR is beyond this paper, but we outline the state of the art of the combination of industrial robots and AR. Those methods are used for offline programming [Duguleana et al., 2012] or more userfriendly programming [Ameri et al., 2010]. Our research group in cooperation with the University of W¨ urzburg has published on remote operation and monitoring [Leutert and Schilling, 2012] as well as on dynamic path planning and obstacle avoidance [Leutert et al., 2012]. The main challenge of the project is the adaptation of those results, mainly in the field of ”Spatial Augmented Reality” (with a stationary camera or display system [Bimber and Raskar, 2005]) on mobile applications.

2.3 Security and System Integration We intend to provide a secure framework for remote maintenance with integrated communication, video and VR/AR functionality. The timing constraints of the different applications sharing the available bandwidth must also be considered when designing a security solution. We start with a short description of our system design, followed by our security considerations. System overview Our telemaintenance system consists of multiple applications, which generate data to be exchanged between the remote maintenance center and facility: Manual and automatic robotic control, video streaming, file transfer, AR/VR and human communication. All generated data streams are collected and preprocessed by the adaptive management system (AMS) prior to encryption. The AMS will be designed as an adaptive control system that constantly measures and evaluates the connection parameters. If the measured Quality of Service (QoS) is not sufficient for a certain service, the system either triggers graceful degradation methods or informs the user that the requested service is currently not available. Security measures As a logged-in technician is enabled to remote control expensive machinery, proper authentication methods are mandatory. Digital signatures are needed to prevent falsification of commands during transmission. Encryption must be applied, to protect data interesting in terms of economic espionage: Detailed images of machinery, videos covering full production cycles and machine blueprints are exchanged for repair purposes. To summarize, our solution must provide sufficient security while distributing available bandwidth between different sub-applications.

Fig. 5. Schematic overview of the tracking problem

Research goals We have three main components in a mobile AR setting: tracking, display technology and real time rendering [Bimber and Raskar, 2005]. As described above, display technologies are currently undergoing a rapid progress. One goal of MainTelRob is the analysis of the application possibility of consumer electronic AR products for an industrial environment. Our research group in cooperation with the university of W¨ urzburg has already designed a spatial augmented reality set, working with robots built by Reis Robotics. This solution for realtime rendering will also be used for MainTelRob. Going from a spatial augmented reality solution to a mobile AR solution, the tracking of the users perspective is essential. Considering a portable device like a tablet PC, the position of this device and its onboard camera has to be determined very precisely to enable the fading of the virtual model. This problem is visualized in Figure 5: The tracking has to determine the projection of the relative coordinate system of the camera of the (stable) base coordinate system of the robot in real time. Hence the rendering techniques developed for spatial augmented reality settings can be used. 116

All above mentioned security measures (authentication, signatures and encryption) are already available within ready-to-use tunneling and virtual private network (VPN) software. We considered tunneling through the secure shell (SSH) protocol as well as common VPN solutions: Provider-provisioned VPNs resemble virtual site-to-site connections and offer a guaranteed QoS. These solutions often involve custom hardware, are expensive and not always encrypted [ISi09]. Hence, they are not suitable for our purpose. Freely available client-server VPN-software, for example OpenVPN [Ope], can be configured to work with different ciphers and authentication schemes. SSH-tunnels, which are run over TCP, offer authentication [Ylonen and Lonvick, 2006a] as well as encryption and integrity verification [Ylonen and Lonvick, 2006b, Bider and M., 2012]. SSH is also not an option, as TCP throughput can seriously decrease (”starvation”) in case of packet loss [Kurose and Ross, 2003] and retransmission of lost packets is of no use for our realtime services. Hence, we chose to utilize an encrypted client-server VPN over UDP. 3. TEST ENVIRONMENT We created models of expected network scenarios to evaluate our different proposed control techniques. This also helps us to identify the QoS above which certain controls and services are feasible. As we cannot simply set up multiple test sites around the world in this early project stage,

IFAC TA 2013 November 11-13, 2013. Seoul, Korea

we created a virtual test environment. Later, the fully developed system will also be tested on the PlanetLab [Pla] worldwide testbed. In the remainder of this section, we survey the used sources of information and shortly explain the implementation and the hardware set-up. robotics expert

Network parameters Prior to programming the emulator, we needed to determine sets of realistic network parameters for the emulation of multiple scenarios, which model connections between a service center in Germany and production sites in different countries. Important parameters are, e.g., throughput (commonly: ”bandwidth”), latency, jitter (delay variations), as well as the likeliness and distribution of packet loss and errors. We can assume that the telemaintenance center itself has a broadband internet connection with sufficient QoS. Hence, we focus our research on the QoS of the long-distance connections to, and the last-mile network conditions to be expected in the main export destinations of our industrial partners: Western Europe, North America, and the BRICS states (Brazil, Russia, India, China and South Africa). Primary data sources We first surveyed the freely available information on overall network availability and broadband access, which can, e.g., be found in the Reports from ITU [Broadband Commission, 2012] and Akamai [Akamai Technologies, Inc.]. The latter also lists peak and average bandwidths. In addition, we regarded the information provided on the websites of organizations running the national Internet exchanges in the BRICS states, government reports, Wikipedia and the submarine cable map [Telegeography]. The average long-distance delay from Germany to the national and regional internet exchanges could be estimated with the help of multiple pathping and traceroute tests towards them. The publicly available measurements of the PingER project [Matthews and Cottrell, 2000] provide data about the average delayes and jitter to expect on long distance connections between destinations in different countries. Kaune et al. researched how this measured jitter correlates with spacial adjacency and provide a mapping technique to extend the measured jitter ”landscape” based on geographical distance [Kaune et al., 2009]. SamKnows actively gathers data about broadband QoS for governments with the help of ”white-boxes”, special routers with measurement software. While the measurement techniques and setup are well documented and published [SamKnows, 2012], only a small part of the data itself is published. The statistics and data provided by the Measurement Lab (MLab) contain information about the connection quality on the last mile. Mlab hosts browser-based tools, which measure the QoS between end-users running test-software and the nearest Mlab server [Dovrolis et al., 2010]. Emulator implementation The emulator is a standardhardware personal computer, configured as router. It is equipped with multiple network interfaces and running the Ubuntu 12.0.4 LTS operating system. We implement the emulation for each scenario as a set of queuing disciplines (qdiscs) for the traffic control (tc) feature of the Linux kernel. Tc is part of the iproute2 kernel module and allows to configure filters and forwarding rules for ingress and egress traffic [Keller, 2006]. Each emulation is featuring multiple consecutive qdiscs, starting with a token bucket 117

network emulation

remote facility

Linux kernel tc / netem packet processing rules capacity & latency

error, jitter & loss models

Fig. 6. Test environment that restricts the maximum available bandwidth of the emulated ”link” to a given value. Delays, jitter and packet loss are implemented with the network emulation (netem) qdisc. Hardware setup Figure 6 depicts our local testbed setup: One notebook contains the software used by the robotics expert. The emulation is carried out on the Linux PC in the middle, which is configured as a router and equipped with two network interfaces. One interface is connected to the notebook, the other to the PC at the right, which represents the computer at the remote facility. Thus, all network traffic between the ”expert” and the ”facility” runs through the network emulation PC. 4. CONCLUSION In this paper we discussed the requirements and design foundations for a novel comprehensive system which aims to improve the state-of-the-art of telemaintenance services in terms of situation awareness and network-based adaptive control. Although we are still in an early stage of development, we envision that our solution will in the future be used to remotely monitor machinery. In addition, facility personnel will be able to use external expertise over the Internet. This will help to reduce downtimes in cases of failure and in addition enable robotics experts to optimize cycle times, material or energy consumption remotely for a great number of facilities. REFERENCES Google Glass Homepage. URL http://www.google.com/ glass/start/. OpenVPN Homepage. URL http://openvpn.net/. PlanetlLab Europe. URL http://www.planet-lab.eu. Akamai Technologies, Inc. The State of the Internet, 3rd Quarter, 2012 Report. 5(3). AE Ameri, B Akan, and B C¨ ur¨ ukl¨ u. Augmented reality meets industry: Interactive robot programming. SIGRAD, V¨ aster˚ as, 2010. Karl J Astrom, Chang C Hang, and BC Lim. A new smith predictor for controlling a process with an integrator and long dead-time. Automatic Control, IEEE Transactions on, 39(2):343–345, 1994. D. Bider and Baushke M. Sha-2 data integrity verification for the secure shell (ssh) transport layer protocol. IETF RFC 6668, July 2012.

IFAC TA 2013 November 11-13, 2013. Seoul, Korea

Oliver Bimber and Ramesh Raskar. Spatial Augmented Reality - Merging Real and Virtual Worlds. A K Peters, Ltd., 2005. M.S. Branicky, S.M. Phillips, and Wei Zhang. Stability of networked control systems: explicit analysis of delay. American Control Conference, 2000. Proceedings of the 2000, 4:2352 – 2357, June 2000. ITU Broadband Commission. The State of Broadband 2012: Achieving Digital Inclusion for All. September 2012. O. L. V. Costa, M. D. Fragoso, and R. P. Marques. Discrete-Time Markov Jump Linear Systems. Springer, 2005. D.P. De Farias, J.C. Geromel, J.B.R. Do Val, and O.L.V. Costa. Output feedback control of markov jump linear systems in continuous-time. IEEE Transactions on Automatic Control, 45(5):944–949, May 2000. doi: 10. 1109/9.855557. C. Dovrolis, K. Gummadi, A. Kuzmanovic, and S. Meinrath. Measurement Lab: Overview and an Invitation to the Research Community. ACM SIGCOMM Computer Communication Review, 40(3):53–56, July 2010. Frauke Driewer. Teleoperation Interfaces in Human-Robot Teams. PhD thesis, Universit¨ at W¨ urzburg - Lehrstuhl f¨ ur Informatik VII, 2009. Mihai Duguleana, Florin Girbacia, and Adrian Stavar. Offline programming of industrial robots using co-located environments. Advanced Materials Research, 463:1654– 1657, 2012. Jackie Fenn, Mark Raskino, and Brian Gammage. Gartner’s hype cycle special report for 2009. Gartner, Stamford, CT, 2009. ¨ Petter Ogren, Edward Fiorelli, and Naomi Ehrich Leonard. Cooperative control of mobile sensor networks: Adaptive gradient climbing in a distributed environment. IEEE Transactions on Automatic Control, 49:1292– 1302, 2004. ISi Work Group of the German Federal Office for Information Security (BSI). Virtuelles Privates Netz (ISi-VPN), BSI-Leitlinie zur Internet-Sicherheit (ISi-L), Version 1.0. 2009. S. Kaune, K. Pussep, C. Leng, A. Kovacevic, G. Tyson, and R. Steinmetz. Modelling the Internet Delay Space Based on Geographical Locations. In Parallel, Distributed and Network-based Processing, 2009 17th Euromicro International Conference on, pages 301–310, 2009. doi: 10.1109/PDP.2009.44. Ariane Keller. Manual, tc Packet Filtering and netem. July 2006. James Kurose and Keith Ross. Computer Networking: A Top-Down Approach Featuring the Internet. AddisonWesley / Pearson Education, Inc., second edition, 2003. Seok-kyu Kweon, Kang G. Shin, and Gary Workman. Achieving real-time communication over ethernet with adaptive traffic smoothing. In Proc. IEEE Real-Time Tech.. Appl. Symp., pages 90–100, 2000. Florian Leutert and Klaus Schilling. Support of power plant telemaintenance with robots by augmented reality methods. In Applied Robotics for the Power Industry (CARPI), 2012 2nd International Conference on, pages 45–49. IEEE, 2012. Florian Leutert, Christoph Freier, and Klaus Schilling. 3d-sensor based dynamic path planning and obstacle 118

avoidance for industrial manipulators. In Robotics; Proceedings of ROBOTIK 2012; 7th German Conference on, pages 1–6. VDE, 2012. Xiangheng Liu and Andrea Goldsmith. Kalman filtering with partial observation losses. In IEEE Trans. on Autom. Control, pages 4180–4186, 2004. Christine Ludwig and Christian Reimann. Augmented reality: Information im fokus. C-LAB Report. Paderborn, C-Lab, 2005. W. Matthews and L. Cottrell. The PingER project: active Internet performance monitoring for the HENP community. Communications Magazine, IEEE, 38(5): 130–136, 2000. ISSN 0163-6804. doi: 10.1109/35.841837. Anett Mehler-Bicher, Micheal Reiß, and Lothar Steiger. Augmented Reality - Theorie und Praxis. Oldenbourg Verlag M¨ unchen, 2011. Cai Meng, Tianmiao Wang, Wusheng Chou, Sheng Luan, Yuru Zhang, and Zengmin Tian. Remote surgery case: Robot-assisted teleneurosurgery. In ICRA, pages 819– 823, 2004. Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino. Augmented reality: A class of displays on the reality-virtuality continuum. In Photonics for Industrial Applications, pages 282–292. International Society for Optics and Photonics, 1995. Johan Nilsson and Bo Bernhardsson. Lqg control over a markov communication network. In In Proceedings of the 36th IEEE Conference on Decision and Control, pages 4586–4591, 1997. Johan Nilsson, Bo Bernhardsson, and Bj¨orn Wittenmark. Stochastic analysis and control of real-time systems with random time delays. Automatica, 34:57–64, 1996. SamKnows. SamKnows Test Methodology, Methodology and technical information relating to the SamKnows testing platform. April 2012. Markus Sauer. Mixed-Reality for Enhanced Robot Teleoperation. PhD thesis, Universit¨at W¨ urzburg - Lehrstuhl f¨ ur Informatik VII, 2010. P. Seiler and R. Sengupta. Analysis of communication losses in vehicle control problems. Proceedings of the 2001 American Control Conference., 2:1491–1496, 2001. doi: 10.1109/ACC.2001.945935. Peter Seiler and Raja Sengupta. An h∞ approach to networked control, January 2003. Peter Joseph Seiler. Coordinated Control of Unmanned Aerial Vehicles. Doctor of philosophy in engineeringmechanical engineering, University of California, Berkeley, 2001. Telegeography Global Bandwidth Research Service. The submarine cable map. URL http://www. submarinecablemap.com/. Sekhar Chandra Tatikonda. Control under communication constraints. PhD thesis, Massachusetts Institute of Technology. Dept. of Electrical Engineering and Computer Science., 2000. T. Ylonen and C. Lonvick. The secure shell (ssh) authentication protocol. IETF RFC 4252, January 2006a. T. Ylonen and C. Lonvick. The secure shell (ssh) transport layer protocol. IETF RFC 4253, January 2006b. Wei Zhang, Michael S Branicky, and Stephen M Phillips. Stability of networked control systems. Control Systems, IEEE, 21(1):84–99, 2001.