- Email: [email protected]
The Adaptive Management and Security System for Maintenance and Teleoperation of Industrial Robots*
4th IFAC Symposium on Telematics Applications 4th IFAC Symposium on Telematics Applications November 6-9, 2016. UFRGS, Porto Alegre, RS, Brazil 4th IFAC IFAC Symposium Symposium on Telematics Telematics Applications 4th on Applications November 6-9, 2016. UFRGS, Porto Alegre, RS, Brazil Available online at www.sciencedirect.com November RS, November 6-9, 6-9, 2016. 2016. UFRGS, UFRGS, Porto Porto Alegre, Alegre, RS, Brazil Brazil
ScienceDirect IFAC-PapersOnLine 49-30 (2016) 006–011
The Adaptive Management and Security The The Adaptive Adaptive Management Management and and Security Security System for Maintenance and Teleoperation System for Maintenance and Teleoperation System for Maintenance and Teleoperation of Industrial Robots of Industrial Robots of Industrial Robots
∗ ∗ ∗ Fritscher ∗ Felix Sittner ∗ Doris Aschenbrenner ∗ Fritscher Felix Sittner Doris Aschenbrenner ∗ ∗ ∗ ∗ ∗∗ ∗ Felix ∗ Doris ∗ Fritscher Sittner Aschenbrenner Markus Krauß Schilling ∗ Klaus ∗∗ Fritscher Felix Sittner Doris Aschenbrenner Markus Krauß ∗ Klaus Schilling ∗∗ ∗∗ ∗ Markus Krauß Krauß Klaus Klaus Schilling Schilling Markus ∗ u rr Telematik e.V., 97074 ∗ The authors are with the Zentrum f¨ The authors are with the Zentrum f¨ u e.V., 97074 ∗ ∗ W¨ The authors are with the Zentrum f¨ u rr Telematik Telematik e.V., 97074 [email protected] u rzburg, Germany The authors are with the Zentrum f¨ u Telematik e.V., 97074 [email protected] W¨ u rzburg, Germany ∗∗ [email protected] W¨ u rzburg, Germany Klaus Schilling is head of Chair VII at the Department of Computer ∗∗ [email protected] W¨ u rzburg, Germany Klaus Schilling is Chair VII at Department of ∗∗ ∗∗ Klaus Schilling is head headofof ofW¨ Chair VII 97074 at the the W¨ Department of Computer Computer Science, University u rzburg, u rzburg, Germany Klaus Schilling is head of Chair VII at the Department of Computer Science, University of W¨ u rzburg, 97074 W¨ u rzburg, Germany Science, of W¨ u rzburg, 97074 W¨ u rzburg, Germany [email protected] Science, University University of W¨ u rzburg, 97074 W¨ u rzburg, Germany [email protected]
Michael Michael Michael Michael
[email protected] [email protected]
Abstract: Within the project ”MainTelRob”, we research the opportunity support Abstract: Within the project ”MainTelRob”, we research the opportunity to to support maintemainteAbstract: Within the project ”MainTelRob”, we research the to maintenance tasks over the Internet. identified, that requires specific combination of services, Abstract: Within the projectWe ”MainTelRob”, wethis research theaa opportunity opportunity to support support maintenance tasks over the Internet. We identified, that this requires specific combination of services, nancevideo tasksstreaming, over the the Internet. Internet. We identified, identified, that thisdata requires a specific specific combination of services, services, e.g., communication services and transfer. By using the public Internet nance tasks over We that this requires a combination of e.g., video streaming, communication services and data transfer. By using the public Internet e.g., video streaming, communication services and data transfer. using the Internet emerges the need to transfer all data in aa secure However, there is no e.g., video streaming, communication services and and dataconfidential transfer. By Byway. using the public public Internet emerges the need to transfer all data in secure and confidential way. However, there is no emerges the need to transfer all data in a secure and confidential way. However, there is no framework to provide these functionalities out of the box. This is why we developed the Adaptive emerges the need to transfer all data in a secure and confidential way. However, there is no framework to provide these functionalities out of the box. This is why we developed the Adaptive framework to provide these functionalities out of the box. This is why we developed the Adaptive Management and Security System (AMS), a multi-layer architecture and framework providing framework to and provide these System functionalities out of the box. This is why we developed the providing Adaptive Management Security (AMS), a multi-layer architecture and framework Management and Security Security System (AMS), aa multi-layer multi-layer architecture andthe framework providing the building blocks to create tele-maintenance applications. In addition, AMS measures the Management and System (AMS), architecture and framework providing the building blocks to create tele-maintenance applications. In addition, the AMS measures the the building blocks to create tele-maintenance applications. In addition, the AMS measures the quality of an end-to-end connection over the Internet and adjusts the amount of data sent by the building blocks to create tele-maintenance applications. In addition, the AMS measures the quality of an end-to-end connection over the Internet and adjusts the amount of data sent by quality of an end-to-end connection over the Internet and adjusts the amount of data sent by the services, in order to use the given connection efficiently. In this publication, we provide quality of an end-to-end connection over the Internet and adjusts the amount of data sent byaa the services, in order to use the given connection efficiently. In this publication, we provide the services, in order to use the given connection efficiently. In this publication, we provide aa short overview of the state of the art and subsequently explain the architecture, as well as the the services, in of order state to use the given connection efficiently. In the thisarchitecture, publication, as we provide short the art subsequently explain as the short overview overview of the the state of of by theeach art and and subsequently explain the into architecture, as well well as the structure and tasks addressed layer. We also provide insight the first tests, in which short overview of the state of the art and subsequently explain the architecture, as well as the structure and tasks addressed by each layer. We also provide insight into the first tests, in which structure and tasks by each layer. We also provide insight into the tests, which aa prototype AMS was teleoperate industrial over the structure andimplementation tasks addressed addressedof bythe each layer. Weused alsoto provide insightan into the first first robot tests, in in which prototype implementation of the AMS was used to teleoperate an industrial robot over the aa prototype implementation of the AMS was used to teleoperate an industrial robot over the public Internet. prototype implementation of the AMS was used to teleoperate an industrial robot over the public Internet. public Internet. public Internet. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: Data streams, Data flow analysis, Network analyzers, Network identification, Data Keywords: Data streams, Data flow analysis, Network analyzers, Network identification, Data Keywords: Data streams, Data flow analysis, Network analyzers, Network identification, Data compression, Data Transmission, System security, Human-machine interface Keywords: Data streams, Data flow analysis, Network analyzers, Network identification, Data compression, Data Transmission, System security, Human-machine interface compression, Data Transmission, System security, Human-machine interface compression, Data Transmission, System security, Human-machine interface 1. INTRODUCTION the facility. The facility contains machines, aa robot and 1. INTRODUCTION INTRODUCTION the facility. The facility contains machines, and 1. the facility. The facility contains machines, aa robot robot and telemaintenance equipment: a computer, multiple cameras 1. INTRODUCTION the facility. The facility contains machines, robot and telemaintenance equipment: a computer, multiple cameras telemaintenance equipment: a computer, multiple cameras for video streaming and a mobile device. Center and In the current stage of globalization, new production telemaintenance equipment: a computer, multiple cameras for video streaming and a mobile device. Center and In the the current current stage stage of of globalization, globalization, new new production production facility for video video streaming and mobile device. Center Center and and In are connected over the Internet. plants set up in emerging nations. The specialized streaming and aa mobile device. In the are current stage of globalization, new production facility are connected over the Internet. plants are set up in emerging nations. The specialized for facility are connected over the Internet. plants are set up in emerging nations. The specialized machinery needed for these factories is often produced facility are connected over the Internet. plants are set up in emerging nations. The specialized main prerequisite is to provide the expert with a machinery needed needed for for these these factories factories is is often often produced produced The The prerequisite is provide the expert with machinery by special purpose machinery manufacturers in machinery needed for these factories is oftenresiding produced The main main prerequisite is to toon-site. provideThis the insight expert can withbeaaa good view of the situation by special purpose machinery manufacturers residing in The main prerequisite is to provide the expert with view of the situation on-site. This insight can be by special purpose machinery manufacturers residing in Europe or Northern America. The maintenance and reby special residing in good good view the on-site. insight be offered by aaof orchestrated combination of serEurope or purpose Northernmachinery America. manufacturers The maintenance maintenance and reregood view ofspecifically the situation situation on-site. This This insight can can be offered by specifically orchestrated combination of serEurope or Northern America. The and pair of the industrial robots inside the factories require Europe or Northern America. The maintenance and re- vices: offered by a specifically orchestrated combination of serRemote access to machinery data in combination pair of the industrial robots inside the factories require offered by a specifically orchestrated combination of serRemote access machinery data combination pair of the robots inside the factories require highly personnel, who are not pair of trained the industrial industrial robots the always factoriesavailable require vices: vices: video Remote access to toand machinery data in in services, combination with streaming communication e.g. highly trained personnel, whoinside are not not always available vices: Remote access to machinery data in combination with video streaming and communication services, e.g. highly trained personnel, who are always available on location. Here, Industrial internet solutions like telehighly trained personnel, who are not always available with chat videoand streaming and communication communication services,visual e.g. text Voice-over-IP (VoIP). In addition, on location. Here, Industrial internet solutions like telewith video streaming and services, e.g. text chat and Voice-over-IP (VoIP). In addition, visual on location. Here, Industrial internet solutions like telemaintenance can help bridge the gap. On the one hand, on location. Here, Industrial internet solutions likehand, tele- text chat and Voice-over-IP (VoIP). In addition, visual Augmented Reality (AR) overlays inserted into the camera maintenance can help bridge the gap. On the one text chat and Voice-over-IP (VoIP). In addition, visual Augmented Reality (AR) overlays inserted into the camera maintenance can the gap. On hand, they enable manufacturers to provide their maintenance can help help bridge bridge gap. assistance On the the one oneto Augmented Reality (AR) overlays inserted into the or video are used to provide guidance. As they enable enable manufacturers manufacturers to the provide assistance tohand, their pictures Augmented Realityview (AR) overlays inserted into the camera camera pictures or video view are used to provide guidance. As they to provide assistance to their customers over the Internet. Removing the need to travel they enable manufacturers to provide assistance to their pictures or video video view are areincludes used to to the provide guidance. As the targeted environment service technician customers over the Internet. Removing the need to travel pictures or view used provide guidance. As the targeted environment includes the service technician customers over the Internet. Removing the need to travel shall help to reduce downtimes in cases breakdowns, customers over the Internet. Removing theof need to travel the targeted environment includes the service technician repairing the machinery on-site, the industrial telemainteshall help to reduce downtimes in cases of breakdowns, the targeted environment includes the service technician repairing the machinery on-site, the industrial telemainteshall help to downtimes in of and new business models and markets for the shallalso helpcreate to reduce reduce downtimes in cases cases of breakdowns, breakdowns, repairing the on-site, industrial telemaintesystem should additionally provide modern means of and also create new business business models and markets markets for the the nance repairing the machinery machinery on-site, the the industrial telemaintenance system should additionally provide modern means of and also create new models and for manufacturers. On the other hand, these solutions can, and also create new business models and markets for the nance system should additionally provide modern means of communication. To enable the smooth operation of these manufacturers. On the other hand, these solutions can, nance system should additionally provideoperation modern means of communication. To enable the smooth of these manufacturers. On the other hand, these solutions can, in combination with multimedia technologies, facilitate manufacturers. On the other hand, these solutions can, communication. To enable thechallenges smooth operation operation ofbethese these desired services, four major need to adin combination with multimedia technologies, facilitate communication. To enable the smooth of desired services, four major challenges need to be adin combination with multimedia technologies, facilitate communication and transfer of knowledge between experts in combination and withtransfer multimedia technologies, services, four major challenges to be dressed in the system: needs communication of knowledge knowledge betweenfacilitate experts desired desired services, four First, majorteleoperation challenges need need tomodeling be adaddressed in the system: First, teleoperation needs modeling communication and of between experts in the highly industrialized and the local repair communication and transfer transfer countries of knowledge between experts dressed in the system: First, teleoperation needs modeling and remote control of industrial manipulators. Second, the in the highly industrialized countries and the local repair dressed in the system: First, teleoperation needs modeling remote control of industrial manipulators. Second, the in the industrialized countries personnel in emerging nations. in the highly highly industrialized countries and and the the local local repair repair and and remote control of industrial manipulators. Second, the teleoperation integrates humans in the control loop. There personnel in emerging nations. and remote control of industrial manipulators. Second, the teleoperation integrates humans in the control loop. There personnel in emerging nations. personnel in emerging nations. teleoperation integrates humans in the control control loop. There There are no existing models for the human teleoperator, yet. Within this publication, we regard the following use-case: teleoperation integrates humans in the loop. are no existing models for the human teleoperator, yet. Within this publication, we regard the following use-case: are no models for the teleoperator, yet. Within this publication, regard use-case: To address this issue, we aa strict user-centered There is aa manufacturer’s center from no existing existing models forpropose the human human teleoperator, yet. Within publication, we we telemaintenance regard the the following following use-case: To address this issue, we propose strict user-centered There is isthis manufacturer’s telemaintenance center from are To address this issue, we propose a strict user-centered There a manufacturer’s telemaintenance center from approach. Third, there are different prevalent end user which an engineer, the expert, provides technical expertise To address this issue, we propose a strict user-centered There is a manufacturer’s telemaintenance center from Third, there are different prevalent end user which an an engineer, engineer, the the expert, expert, provides provides technical technical expertise expertise approach. approach. Third, are different prevalent end user which access technologies to the differ in their to the local repair personnel, the service technicians, at approach. Third, there there areInternet, differentwhich prevalent end user which an engineer, the expert, provides technical expertise access technologies to the Internet, which differ in their to the local repair personnel, the service technicians, at access technologies to the Internet, which differ in their to the local repair personnel, the service technicians, at expectable Quality of Service (QoS). Hence, our framework access technologies to the Internet, which differ in their toThis the work local was repair personnel, the service technicians, at expectable Quality of Service (QoS). Hence, our framework funded by the Bavarian Ministry of Economic This work was funded by the Bavarian Ministry of Economic expectable Quality of (QoS). our framework must be able to detect the characteristics the provided expectable Quality of Service Service (QoS). Hence, Hence,of our framework must be able to detect the characteristics of the provided Affairs, Transport Technology as partof ofEconomic the R&D This work by Bavarian Ministry This Infrastructure, work was was funded funded by the theand Bavarian Ministry must be able to detect the characteristics of the provided Affairs, Infrastructure, Transport and Technology as partofofEconomic the R&D end-to-end connection, and decide which services are apmust be able to detect the characteristics of the provided program “Information and Communication Technology”. Affairs, Infrastructure, Transport and Technology as part of the R&D end-to-end connection, and decide which services are apAffairs, Infrastructure, Transport and Technology as part of the R&D program “Information and Communication Technology”. end-to-end connection, and decide which services are end-to-end connection, and decide which services are apapprogram program “Information “Information and and Communication Communication Technology”. Technology”. Copyright © 2016, 2016 IFAC 6 Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2016 IFAC 6 Copyright 2016 IFAC 6 Peer review© of International Federation of Automatic Copyright ©under 2016 responsibility IFAC 6 Control. 10.1016/j.ifacol.2016.11.114
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Michael Fritscher et al. / IFAC-PapersOnLine 49-30 (2016) 006–011
plicable. Fourth, it needs to orchestrate the transmission of data streams from multiple sources through a secured connection. This includes adapting the services to connection quality changes during ongoing telemaintenance sessions. In the next paragraph, we give a short survey of the state of the art in telemaintenance, while in the remainder of this publication, we present our approaches to the aforementioned challenges and offer insight into the first prototype tests.
7
Interaction
Interaction
Application
Application
Service
Service
Telematics Connection
Telematics Connection
Transport
Transport
Network
Network
Data Link
Data Link Physical
2. STATE OF THE ART
Fig. 1. Layers of the AMS architecture on top of the OSI transport layer
The term telemaintenance refers to the integration of computer science and communication technologies into the maintenance strategy (Chowdhury and Akram (2011); Mouzoune and Taibi (2014)). We understand the challenges in telemaintenance as a combination of teleoperation and remote condition monitoring. In this publication, we focus on the teleoperation aspect.
be supported are specified. Subsequently, the Application Layer and its user interface concepts are explained. This is followed by an overview of the Service Layer providing the underlying functionality for the Application Layer. In the last subsection, we survey the Telematics Connections Layer, which resides, as depicted, above the OSI transport layer. It provides a single entry point into the center or facility for the secured data transmission.
In general, ”a robotic teleoperation system allows to reproduce the actions of a human operator and to interact physically with objects and environments placed at a distance” (Melchiorri (2014)). Hence, it can be seen as a subcategory of human supervisory control (Sheridan (1992)). For example, visual feedback can provide information about the state of an industrial robot to the human operator in the control loop. For an overview of applications see Lichiardopol (2007); in this paper, we focus on teleoperation of industrial robots, which has been of early (Cederberg et al. (2002)) and late (Moradi Dalvand and Nahavandi (2014)) research interest. The application for teleoperation of industrial robots lies mainly in the context of hazardous environments (Pegman et al. (2006)), but in our scenario we cover maintenance scenarios similar to those for tunneling machines covered in David et al. (2014). As we aim to facilitate teleoperation via the Internet, the control perspective with time delay over the Internet (Slama et al. (2007)) has to be integrated as much as human perception of time delays (Vozar and Tilbury (2014)). There have been several proposals for teleoperation architectures, e.g. in Gray et al. (2007); Ortega et al. (2014). Our project partner Kuka Industries, former Reis Robotics, provides an architecture that includes basic teleoperation functionalities. However, there is a need for an enhanced architecture that also includes further services needed for human supervisory control, such as synchronized video feedback. The closest related work has been provided by Jia (2014), who covers multimedia streams, too (Elhajj et al. (2011)), but does not include other kinds of data needed for maintenance. To our best knowledge, there is no integrated teleoperation and maintenance architecture yet.
3.1 Interaction Layer Top-down, the first layer of our architecture is the Interaction Layer, which follows an user-centric design approach. This layer serves as an adaption layer between our architecture and the business processes at the facility and center. We performed a broad contextual analysis involving interviews with managers, engineers, repair personnel and production workers. Based thereon, we modelled the workflows of experts and service technicians and identified use-cases and teleoperation scenarios (Sittner et al. (2013)). For the AMS architecture teleoperation (as defined in Sheridan (1992)) is the key scenario. Here, the control loop is closed through the human and the control computer. We extend the model (as in Fig. 2) by the local service technician, which has a very important role due to safety issues. In Europe teleoperation of an industrial robot in an automation facility is only permitted if there is a local supervisor present. During our experiments, this service technician always has to press the dead man’s switch to enable the external control. Therefore, he also performs human supervisory control and mirrors the conceptual structure of the main teleoperator (the expert). As it can be derived from Fig. 2, communication mostly takes place between the expert and the service technician. Typical cases for maintenance calls are unknown problems at the production line, which need to be solved as quickly as possible. The expert knows the technical details of the robot quite well, but needs to get an impression of the periphery and the exact configuration of the robot in order to be able to analyze the failure. The AMS technology is designed especially for those questions by providing several cameras at the production line and a mobile camera in the tablet computer which the service technician carries as a Local Human Interactive System which can also serve as robot teach pendant (Aschenbrenner et al. (2014)).
3. AMS ARCHITECTURE The Adaptive Management and Security System (AMS) is designed as a modular multi-layer architecture, in which each layer provides functions for the layer above. Each layer comprises a manager component exchanging control information with the managers of adjacent layers. In the following sections, we explain the four layers of the AMS architecture top-down, as depicted in Fig. 1. We begin with the Interaction Layer, on which the work flows to
3.2 Application Layer The Application Layer provides the graphical user interface (GUI) to the user. This is also where the vendor 7
2016 IFAC TA 8 November 6-9, 2016. Porto Alegre, Brazil Michael Fritscher et al. / IFAC-PapersOnLine 49-30 (2016) 006–011
App Layer App Layer Component 1
Service Layer
TC Layer
Service
TC Layer Multiplexer
Routine C B A
…
CENTER
App Layer Component n
FACILITY
Manager
Service
…
TC Layer Component
Routine C
outine
B
A
Manager
App Manager
Service Manager
TC Manager
Fig. 2. Extended Human Supervisory Control Model
Fig. 5. Service Interactions
Fig. 3. GUI for the Remote Human Interactive System
facility. The small traffic light on the right indicates the current end-to-end Quality of Service between facility and center. Both figures depict the usage of the Paint App. Using this application, both expert and service technician can draw simple outlines into the video view, e.g., to point out areas of interest. Altogether, the expert is able to teleoperate the robot, communicate with the service technician, get video and data feedback and use the Paint App. The GUI on the mobile device used by the service technician, as shown in Fig. 4, is a reduced version of the expert’s GUI, with one switchable video view. The service technician can communicate and receive feedback from the camera and through the Paint App. 3.3 Service Layer The Service Layer provides the underlying functionalities for the components on the Application Layer. The main objective of the Service Layer is to facilitate re-usability by offering stackable generic and specialized components. The main components of the service layer are the ServiceManager and the services themselves. The ServiceManager is responsible for configuring, starting and stopping the local services. It exchanges management information with the managers of the neighbouring layers, as depicted in Fig. 5. In addition, the ServiceManager is designed to establish a single point of configuration: The master ServiceManager instance, typically located at the telemaintenance center provides the configuration for all slave AMS instances.
Fig. 4. Service technician using the Local Human Interactive System specific hard- and software components, often including their own GUI, are located. The GUIs are displayed on the Remote and the Local Human Interactive System depicted in Fig. 2. We extended our GUI framework, originally developed for the teleoperation of heterogeneous multirobot systems (Fritscher et al. (2012)), to support the teleoperation applications derived from the Interaction Layer.
Fig. 5 visualizes the structure of services and the way services interact with components of their own and adjacent layers. The broad arrows represent data flows while the thin arrows represent control connections. The Application Layer component depicted on the left relies on functionality or data provided by the service shown in the upper middle of the figure. The service itself may, as depicted, rely on other local services, e.g., for common tasks like time synchronization. For example, the storage service combines data from the video service and the robot service, synchronizes this data and stores it for condition monitoring. Services can communicate with remote services belonging to another AMS instance by using the TC Layer (right). In addition to these data streams, all
The prototype GUIs for the expert and for the service technician provide the basic applications needed for telemaintenance: The expert GUI, as depicted in Fig. 3, contains a text chat window (on the left). The robot can be controlled with the sliders, where also the current robot coordinates are shown. Three video views provide insight into the 8
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Michael Fritscher et al. / IFAC-PapersOnLine 49-30 (2016) 006–011
Center App Layer
Facility
Service Layer
TC Layer
Service Layer
Service data
Robot
Robot
Robot
Com
Com
Com
Com
AR VR
AR VR
Video
Video
Manager
Data
Storage
Manager
Control
...
Measurement 1
...
Cam
Measurement m
Data
Storage
Manager
Service data
ServiceConnection 1
Paint
...
TC Layer
Tunnel
ServiceConnection n
Paint Video
Config
TC Layer
App Layer
Provis
Data Show
...
9
Fig. 7. Composition of data streams in the encrypted tunnel
Config
Manager
Manager
Fig. 7 provides an overview of the data flows: The gray arrows depict the data originated by and intended for services. This data is sent through the secured tunnel to the AMS instance on the other side. The lower part of the figure visualizes the different kinds of connections in the tunnel: First, there is a control connection, used for information exchange between the managers of the TC Layer instances. Subsequently, there can be one or more ServiceConnections, each of which represents one connection from a local service to a remote one. The third kind of data transmitted is the test traffic generated by end-toend measurements originated by test routines belonging to the TC Layer. The manager uses data derived from these measurements to dynamically assign each service a fraction of the available network resources. In the following paragraph, we explain the main components of the TC Layer and how they interact in order to provide the main functionalities.
Fig. 6. Example AMS applications and services services keep a control connection to the ServiceManager depicted at the bottom of the figure. Services may contain so called routines, in which algorithms are implemented, or wrappers for external components, e.g., video streaming software or virtual machines. Each service contains a small internal Manager sub module taking care of communication with the ServiceManager. In addition, these sub modules are responsible for adapting the amount of data sent by the services. If the available bandwidth changes significantly, the LayerTC calculates the new budgets for the connections and send them to the ServiceManager. The services get the new budgets and adapt their transfers by e.g. changing video streaming parameters, using still pictures instead of video streams, disabling uncritical services and so on. So the available bandwidth is used in the optimal way (regarding used bandwidth and the priorities of the connections which are needed by the service bundle) by using adaptive connections instead of fixed ones. As of now, all services are custom implementations. But in general, a service may be any kind of software runnable as process of the underlying operating system.
Fig. 8 depicts the internal structure of the TC Layer, whereat the thin arrows represent control data or method calls and the broad arrows symbolize service data sent or received through the secured connection. The QoS measurement system consists of a set of components, as is pictured on the right (surrounded by the dashed outline). The main component is the Monitor, a system capable of starting different network test programs, the
As depicted in Fig. 6, each application is supported by one or more services: There are e.g. services for the transfer of video streams, chat functions and synchronized storage of machine data. Some generic services, like time synchronization or video streaming, are needed by other services, while specialized services provide access to vendor-specific software. Another important task fulfilled on the Service Layer is to provide access to the hardware, like robots, video cameras and external sensors. The ProvisConnector, for example, is a special Robot Service that integrates Reis Robotics’ “Provis” robot configuration and programming software into the AMS system.
Multiplexer InputLinkData InputLinkControl
Classful Classifier Policer Queuing
Output Link
VPN Network adapter interface
Shaper
3.4 Telematic Connections (TC) Layer
ServicesDB
The Telematics Connections Layer, or TC Layer, is located beneath the Service Layer of the AMS Architecture. It provides a secure connection between two or more endpoints, e.g. between a telemaintenance service center and the environment of a machine in a distant facility. This layer works as a stateful firewall and is responsible for admittance and blocking of service connections. In addition, it provides information about the end-to-end QoS between the AMS instances to the higher layers.
Measurement Monitor ParameterRequest
ConfigDB
Measurement DB
ProfileDB
TC Manager
Fig. 8. Schematic overview of the Telematics Connections Layer 9
2016 IFAC TA 10 November 6-9, 2016. Porto Alegre, Brazil Michael Fritscher et al. / IFAC-PapersOnLine 49-30 (2016) 006–011
Measurements. The Monitor pre-evaluates the outcome of these tests and places the results in the MeasurementDB. The TC Manager is depicted on the bottom of the figure. This component is responsible for bandwidth assignment and communicates with the ServiceManager for configuration purposes. The bandwidth assignment is calculated in the co-located Decider subsystem: The base configuration for the Decider is loaded from the ConfigurationDB, while the decisions are based on data from the MeasurementDB and a set of ServiceProfiles containing the constraints and priority of each admissible service. The TC Manager passes the Decider’s results to the TC Manager on the remote AMS system, to the Service Manger and to the Multiplexer. Information about admitted service connections is stored into the ServicesDB, along with a history of decisions regarding the service.
Fig. 9. Test person with mobile device following the expert’s instructions in front of the control cabinet
The Multiplexer is a QoS-aware throughput-limiting stateful firewall implementation, which guards the end of the secured connection towards the other AMS instance. The filtering rules, bandwidth assignment decisions and basic instructions regarding priorities and traffic classes are provided by the TC Manager. These instructions are translated into rules for Linux’ Netfilter / Iptables modules and queuing instructions.
The current prototype implementation of the AMS software was evaluated in a milestone test during October 2014 including our project partners Reis Robotics and Braun / Procter & Gamble. For this test, the service center was set up at Reis Robotics in Obernburg, Germany. It consisted of a Dell Latitude E6410 laptop, an additional monitor and the AMS software, a mobile phone and a Reis Robotics robot programming device. The facility to be serviced was the environment of an injection molding machine producing parts for electric toothbrushes, in the factory of Braun / P&G in Marktheidenfeld, Germany. It is comprised of an injection molding machine, an assembly adding membranes to the molded parts, and a Reis Cartesian robot handling the parts. In addition, the facility was equipped with a computer, two high definition IP cameras, and a mobile device (Microsoft Surface Pro tablet) with built-in camera to be used by the service technician. As the facility’s network is located inside the worldwide P&G virtual private network (VPN), the service center was connected to the P&G network by a host-to-site VPN via the P&G VPN gateway in Brussels. The goal of the test was to determine, whether the implemented software is adequate for use in a teleoperation scenario, in which the expert performs several tasks remotely and guides the service technician to identify and solve a problem concerning the robot. In addition, we wanted to evaluate to what extent the system behaviour while adapting to changes in the connection quality would impair the users’ workflows in a real maintenance scenario. This scenario was a motor exchange on one of the axis of the robot, which can normally only be done by a certified service technician.
The modules, which, comprise a QoS-supporting router and firewall are depicted within the Multiplexer-box in Fig. 8. The packets sent by the local services are first filtered by the stateful firewall, based on Netfilter / Iptables: Here, packets are classified and marked according to their service class, dropping packets not matching any rule. The policing is also implemented in the firewall: Traffic flows not matching the policy of their service class, e.g. by exceeding their bandwidth quota, are marked with an “excessive” flag. Depending on the settings for the service, excessive traffic can either be rejected by the firewall or delayed. The shaping and queuing of the outgoing traffic are implemented by creating a set of queues and disciplines for the traffic control module from Linux. The priorities of the different traffic classes are currently realized by priority queuing, based on the marks set by the firewall. In addition, traffic marked as excessive can be artificially delayed (shaping) by sending it through an additional queuing discipline. After traversing the queues, the packets are encapsulated and sent through the secured connection, which is set up using OpenVPN.
4. FIRST TESTS IN AN INDUSTRIAL ENVIRONMENT
In the initial setup, three video streams were transferred to the expert: The first provided an overview of the facility, the second stream was capturing the details of the assembly machine, while the third stream was captured by the camera of the service technician’s mobile device. After connecting, the expert was able to control and program the robot, like normally done over a local area network. The expert was able to steer the robot remotely using video and Provis’ AR feedback. Subsequently, the local service technician, as well as one participating scientist performed multiple tasks under expert supervision. For example: “go to the control cabinet and take a picture of the voltage information in the display of a certain component”, as shown in Fig. 9.
The evaluation of the telemaintenance and teleoperation scenarios poses a challenge, as only a limited number of specially trained experts are available and each teleoperation test causes a major interruption in the production line. Hence, we mainly focused on qualitative research: We evaluated different features in experimental testbeds and tested the applications’ functionality and usability in weekly experiments with service technicians at the production site. Short tests of the teleoperation components were carried out biweekly during the last year as explained in Aschenbrenner et al. (2015). 10
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Michael Fritscher et al. / IFAC-PapersOnLine 49-30 (2016) 006–011
To facilitate communication between the participants, both a chat functionality and a screenshot & paint functionality were available in the GUI. Then we triggered the bandwidth adaption routines with false inputs to emulate a deterioration of the end-to-end connection quality. This consisted on a drop from 3 Mbit/s to 300 kbit/s per video stream. Subsequently, the system adapted the video quality to prevent impairment of the higher priority services. The system needed 5 seconds on average to adapt (restarting the video streams), which was noticeable but still ok for the participants. Further advantages in this area made after the test using spare encoding processes dropped this delay to under one second.
11
ACKNOWLEDGEMENTS The authors would like to thank their project partners at Reis Robotics and Braun / Procter&Gamble for their participation and commitment to the MainTelRob project. REFERENCES Aschenbrenner, D., Leutert, F., Sittner, F., and Schilling, K. (2014). Einsatzm¨ oglichkeiten f¨ ur Mobile Ger¨ ate bei der Wartung von Industrierobotern. In SPS Drives Kongress. Aschenbrenner, D., Sittner, F., Fritscher, M., Krauss, M., and Schilling, K. (2015). Teleoperation of an industrial robot in an active production line. In Submitted Paper for: 2nd IFAC Conference on Embedded Systems, Computational Intelligence and Telematics in Control (CESCIT). IFAC. Cederberg, P., Olsson, M., and Bolmsj¨ o, G. (2002). Remote control of a standard abb robot system in real time using the robot application protocol (rap). In Proceedings of the 33rd ISR (International Symposium on Robotics) October, volume 7, 11. Chowdhury, S. and Akram, A. (2011). E-maintenance: Opportunities and challenges. In The 34th Information Systems Research Seminar in Scandinavia (IRIS), Turku, Finland, August 16-19, 2011, 68–81. Turku Centre for Computer Science. David, O., Russotto, F.X., Simoes, M.D.S., and Measson, Y. (2014). Collision avoidance, virtual guides and advanced supervisory control teleoperation techniques for high-tech construction: framework design. Automation in Construction, 44, 63–72. Elhajj, I.H., Dargham, N.B., Xi, N., and Jia, Y. (2011). Real-time adaptive content-based synchronization of multimedia streams. Advances in Multimedia, 2011, 4. Fritscher, M., Hess, R., and Schilling, K. (2012). Generic network and a browser-based hmi. In IARP RISE 2012 6th Workshop (RISE2012). Gray, J., Lippiello, V., Villani, L., and Siciliano, B. (2007). An open architecture for sensory feedback control of a dual-arm industrial robotic cell. Industrial Robot: An International Journal, 34(1), 46–53. Jia, Y. (2014). Teleoperation of mobile manipulators. Ph.D. thesis, Michigan State University. Lichiardopol, S. (2007). A survey on teleoperation. University of Eindhoven, Department Mechanical Engineering Dynamics and Control Group Eindhoven. Melchiorri, C. (2014). Robot teleoperation. In J. Baillieul and T. Samad (eds.), Encyclopedia of Systems and Control, 1–14. Springer London. Moradi Dalvand, M. and Nahavandi, S. (2014). Teleoperation of abb industrial robots. Industrial Robot: An International Journal, 41(3), 286–295. Mouzoune, A. and Taibi, S. (2014). Introducing e-maintenance 2.0. arXiv preprint arXiv:1401.8252. Ortega, J.G., Garc´ıa, J.G., Nieto, L.N., and Garc´ıa, A.S. (2014). Open software architecture for advanced control of robotic manipulators. Pegman, G., Desbats, P., Geffard, F., Piolain, G., and Coudray, A. (2006). Force-feedback teleoperation of an industrial robot in a nuclear spent fuel reprocessing plant. Industrial Robot: An International Journal, 33(3), 178–186. Sheridan, T.B. (1992). Telerobotics, automation, and human supervisory control. MIT press. Sittner, F., Aschenbrenner, D., Fritscher, M., Kheirkhah, A., Krauss, M., and Schilling, K. (2013). Maintenance and telematics for robots (maintelrob). In 3rd IFAC Symposium on Telematics Applications Conference (TA2013), Seoul. IFAC. Slama, T., Aubry, D., Oboe, R., and Kratz, F. (2007). Robust bilateral generalized predictive control for teleoperation systems. In Control & Automation, 2007. MED’07. Mediterranean Conference on, 1–6. IEEE. Vozar, S. and Tilbury, D.M. (2014). Driver modeling for teleoperation with time delay. In IFAC World Congress, volume 19, 3551–3556.
The experts and technicians were content with the features offered by the prototype. Both chat and paint features were considered very helpful, also because of the service technician being in a very noisy environment. Also, guiding the technician based on the video from the mobile device (”not that display!”), proved helpful. While the motor exchange was done with a certified service technician because of safety reasons, both he and the expert on the other side confirmed that using this tool enable also not certified service technicians to fulfill this job. So they expect the completed system to be a very helpful tool in future telemaintenance usecases. One important thing we learned was that especially the expert needs additional training for remote assisting in e.g. the field of responsibility (What happens if there is an accident because I forgot something or the service technician counts on me too much?) Because of these results, the next tests will be with a group of uncertified service technicians and a check list for experts. Additionally, the rules ”The service technician is responsible for his safety” and ”If there is something unclear, the service technician must always ask” will be introduced.
5. CONCLUSIONS In our research project MainTelRob - Maintenance and Telematics for Robots, we and our industrial partners’ engineers, technicians, and production workers identified usecases for the telemaintenance of industrial robots. In this paper we focused on a remote maintenance scenario involving teleoperation and motivated, why it was necessary to develop a novel architecture that provides all of the requested functionalities. Our Adaptive Management and Security System facilitates teleoperation of an industrial robot over the Internet. The framework combines a modular architecture and generic services with the ability to measure the end-to-end QoS of the used network connection and adjust the service combination adequately. The GUI part of the framework allows for fast integration of new features, next to the existing ones. To test our architecture in a real industrial telemaintenance setting, we implemented a prototype providing a specific combination of features. We tested our concepts and the prototype through continuous experiments in our testbed as well as in the production environment. In our future work, we plan to extend the system for further use cases, e.g. by providing enhanced Augmented Reality overlays. 11
ScienceDirect IFAC-PapersOnLine 49-30 (2016) 006–011
The Adaptive Management and Security The The Adaptive Adaptive Management Management and and Security Security System for Maintenance and Teleoperation System for Maintenance and Teleoperation System for Maintenance and Teleoperation of Industrial Robots of Industrial Robots of Industrial Robots
∗ ∗ ∗ Fritscher ∗ Felix Sittner ∗ Doris Aschenbrenner ∗ Fritscher Felix Sittner Doris Aschenbrenner ∗ ∗ ∗ ∗ ∗∗ ∗ Felix ∗ Doris ∗ Fritscher Sittner Aschenbrenner Markus Krauß Schilling ∗ Klaus ∗∗ Fritscher Felix Sittner Doris Aschenbrenner Markus Krauß ∗ Klaus Schilling ∗∗ ∗∗ ∗ Markus Krauß Krauß Klaus Klaus Schilling Schilling Markus ∗ u rr Telematik e.V., 97074 ∗ The authors are with the Zentrum f¨ The authors are with the Zentrum f¨ u e.V., 97074 ∗ ∗ W¨ The authors are with the Zentrum f¨ u rr Telematik Telematik e.V., 97074 [email protected] u rzburg, Germany The authors are with the Zentrum f¨ u Telematik e.V., 97074 [email protected] W¨ u rzburg, Germany ∗∗ [email protected] W¨ u rzburg, Germany Klaus Schilling is head of Chair VII at the Department of Computer ∗∗ [email protected] W¨ u rzburg, Germany Klaus Schilling is Chair VII at Department of ∗∗ ∗∗ Klaus Schilling is head headofof ofW¨ Chair VII 97074 at the the W¨ Department of Computer Computer Science, University u rzburg, u rzburg, Germany Klaus Schilling is head of Chair VII at the Department of Computer Science, University of W¨ u rzburg, 97074 W¨ u rzburg, Germany Science, of W¨ u rzburg, 97074 W¨ u rzburg, Germany [email protected] Science, University University of W¨ u rzburg, 97074 W¨ u rzburg, Germany [email protected]
Michael Michael Michael Michael
[email protected] [email protected]
Abstract: Within the project ”MainTelRob”, we research the opportunity support Abstract: Within the project ”MainTelRob”, we research the opportunity to to support maintemainteAbstract: Within the project ”MainTelRob”, we research the to maintenance tasks over the Internet. identified, that requires specific combination of services, Abstract: Within the projectWe ”MainTelRob”, wethis research theaa opportunity opportunity to support support maintenance tasks over the Internet. We identified, that this requires specific combination of services, nancevideo tasksstreaming, over the the Internet. Internet. We identified, identified, that thisdata requires a specific specific combination of services, services, e.g., communication services and transfer. By using the public Internet nance tasks over We that this requires a combination of e.g., video streaming, communication services and data transfer. By using the public Internet e.g., video streaming, communication services and data transfer. using the Internet emerges the need to transfer all data in aa secure However, there is no e.g., video streaming, communication services and and dataconfidential transfer. By Byway. using the public public Internet emerges the need to transfer all data in secure and confidential way. However, there is no emerges the need to transfer all data in a secure and confidential way. However, there is no framework to provide these functionalities out of the box. This is why we developed the Adaptive emerges the need to transfer all data in a secure and confidential way. However, there is no framework to provide these functionalities out of the box. This is why we developed the Adaptive framework to provide these functionalities out of the box. This is why we developed the Adaptive Management and Security System (AMS), a multi-layer architecture and framework providing framework to and provide these System functionalities out of the box. This is why we developed the providing Adaptive Management Security (AMS), a multi-layer architecture and framework Management and Security Security System (AMS), aa multi-layer multi-layer architecture andthe framework providing the building blocks to create tele-maintenance applications. In addition, AMS measures the Management and System (AMS), architecture and framework providing the building blocks to create tele-maintenance applications. In addition, the AMS measures the the building blocks to create tele-maintenance applications. In addition, the AMS measures the quality of an end-to-end connection over the Internet and adjusts the amount of data sent by the building blocks to create tele-maintenance applications. In addition, the AMS measures the quality of an end-to-end connection over the Internet and adjusts the amount of data sent by quality of an end-to-end connection over the Internet and adjusts the amount of data sent by the services, in order to use the given connection efficiently. In this publication, we provide quality of an end-to-end connection over the Internet and adjusts the amount of data sent byaa the services, in order to use the given connection efficiently. In this publication, we provide the services, in order to use the given connection efficiently. In this publication, we provide aa short overview of the state of the art and subsequently explain the architecture, as well as the the services, in of order state to use the given connection efficiently. In the thisarchitecture, publication, as we provide short the art subsequently explain as the short overview overview of the the state of of by theeach art and and subsequently explain the into architecture, as well well as the structure and tasks addressed layer. We also provide insight the first tests, in which short overview of the state of the art and subsequently explain the architecture, as well as the structure and tasks addressed by each layer. We also provide insight into the first tests, in which structure and tasks by each layer. We also provide insight into the tests, which aa prototype AMS was teleoperate industrial over the structure andimplementation tasks addressed addressedof bythe each layer. Weused alsoto provide insightan into the first first robot tests, in in which prototype implementation of the AMS was used to teleoperate an industrial robot over the aa prototype implementation of the AMS was used to teleoperate an industrial robot over the public Internet. prototype implementation of the AMS was used to teleoperate an industrial robot over the public Internet. public Internet. public Internet. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: Data streams, Data flow analysis, Network analyzers, Network identification, Data Keywords: Data streams, Data flow analysis, Network analyzers, Network identification, Data Keywords: Data streams, Data flow analysis, Network analyzers, Network identification, Data compression, Data Transmission, System security, Human-machine interface Keywords: Data streams, Data flow analysis, Network analyzers, Network identification, Data compression, Data Transmission, System security, Human-machine interface compression, Data Transmission, System security, Human-machine interface compression, Data Transmission, System security, Human-machine interface 1. INTRODUCTION the facility. The facility contains machines, aa robot and 1. INTRODUCTION INTRODUCTION the facility. The facility contains machines, and 1. the facility. The facility contains machines, aa robot robot and telemaintenance equipment: a computer, multiple cameras 1. INTRODUCTION the facility. The facility contains machines, robot and telemaintenance equipment: a computer, multiple cameras telemaintenance equipment: a computer, multiple cameras for video streaming and a mobile device. Center and In the current stage of globalization, new production telemaintenance equipment: a computer, multiple cameras for video streaming and a mobile device. Center and In the the current current stage stage of of globalization, globalization, new new production production facility for video video streaming and mobile device. Center Center and and In are connected over the Internet. plants set up in emerging nations. The specialized streaming and aa mobile device. In the are current stage of globalization, new production facility are connected over the Internet. plants are set up in emerging nations. The specialized for facility are connected over the Internet. plants are set up in emerging nations. The specialized machinery needed for these factories is often produced facility are connected over the Internet. plants are set up in emerging nations. The specialized main prerequisite is to provide the expert with a machinery needed needed for for these these factories factories is is often often produced produced The The prerequisite is provide the expert with machinery by special purpose machinery manufacturers in machinery needed for these factories is oftenresiding produced The main main prerequisite is to toon-site. provideThis the insight expert can withbeaaa good view of the situation by special purpose machinery manufacturers residing in The main prerequisite is to provide the expert with view of the situation on-site. This insight can be by special purpose machinery manufacturers residing in Europe or Northern America. The maintenance and reby special residing in good good view the on-site. insight be offered by aaof orchestrated combination of serEurope or purpose Northernmachinery America. manufacturers The maintenance maintenance and reregood view ofspecifically the situation situation on-site. This This insight can can be offered by specifically orchestrated combination of serEurope or Northern America. The and pair of the industrial robots inside the factories require Europe or Northern America. The maintenance and re- vices: offered by a specifically orchestrated combination of serRemote access to machinery data in combination pair of the industrial robots inside the factories require offered by a specifically orchestrated combination of serRemote access machinery data combination pair of the robots inside the factories require highly personnel, who are not pair of trained the industrial industrial robots the always factoriesavailable require vices: vices: video Remote access to toand machinery data in in services, combination with streaming communication e.g. highly trained personnel, whoinside are not not always available vices: Remote access to machinery data in combination with video streaming and communication services, e.g. highly trained personnel, who are always available on location. Here, Industrial internet solutions like telehighly trained personnel, who are not always available with chat videoand streaming and communication communication services,visual e.g. text Voice-over-IP (VoIP). In addition, on location. Here, Industrial internet solutions like telewith video streaming and services, e.g. text chat and Voice-over-IP (VoIP). In addition, visual on location. Here, Industrial internet solutions like telemaintenance can help bridge the gap. On the one hand, on location. Here, Industrial internet solutions likehand, tele- text chat and Voice-over-IP (VoIP). In addition, visual Augmented Reality (AR) overlays inserted into the camera maintenance can help bridge the gap. On the one text chat and Voice-over-IP (VoIP). In addition, visual Augmented Reality (AR) overlays inserted into the camera maintenance can the gap. On hand, they enable manufacturers to provide their maintenance can help help bridge bridge gap. assistance On the the one oneto Augmented Reality (AR) overlays inserted into the or video are used to provide guidance. As they enable enable manufacturers manufacturers to the provide assistance tohand, their pictures Augmented Realityview (AR) overlays inserted into the camera camera pictures or video view are used to provide guidance. As they to provide assistance to their customers over the Internet. Removing the need to travel they enable manufacturers to provide assistance to their pictures or video video view are areincludes used to to the provide guidance. As the targeted environment service technician customers over the Internet. Removing the need to travel pictures or view used provide guidance. As the targeted environment includes the service technician customers over the Internet. Removing the need to travel shall help to reduce downtimes in cases breakdowns, customers over the Internet. Removing theof need to travel the targeted environment includes the service technician repairing the machinery on-site, the industrial telemainteshall help to reduce downtimes in cases of breakdowns, the targeted environment includes the service technician repairing the machinery on-site, the industrial telemainteshall help to downtimes in of and new business models and markets for the shallalso helpcreate to reduce reduce downtimes in cases cases of breakdowns, breakdowns, repairing the on-site, industrial telemaintesystem should additionally provide modern means of and also create new business business models and markets markets for the the nance repairing the machinery machinery on-site, the the industrial telemaintenance system should additionally provide modern means of and also create new models and for manufacturers. On the other hand, these solutions can, and also create new business models and markets for the nance system should additionally provide modern means of communication. To enable the smooth operation of these manufacturers. On the other hand, these solutions can, nance system should additionally provideoperation modern means of communication. To enable the smooth of these manufacturers. On the other hand, these solutions can, in combination with multimedia technologies, facilitate manufacturers. On the other hand, these solutions can, communication. To enable thechallenges smooth operation operation ofbethese these desired services, four major need to adin combination with multimedia technologies, facilitate communication. To enable the smooth of desired services, four major challenges need to be adin combination with multimedia technologies, facilitate communication and transfer of knowledge between experts in combination and withtransfer multimedia technologies, services, four major challenges to be dressed in the system: needs communication of knowledge knowledge betweenfacilitate experts desired desired services, four First, majorteleoperation challenges need need tomodeling be adaddressed in the system: First, teleoperation needs modeling communication and of between experts in the highly industrialized and the local repair communication and transfer transfer countries of knowledge between experts dressed in the system: First, teleoperation needs modeling and remote control of industrial manipulators. Second, the in the highly industrialized countries and the local repair dressed in the system: First, teleoperation needs modeling remote control of industrial manipulators. Second, the in the industrialized countries personnel in emerging nations. in the highly highly industrialized countries and and the the local local repair repair and and remote control of industrial manipulators. Second, the teleoperation integrates humans in the control loop. There personnel in emerging nations. and remote control of industrial manipulators. Second, the teleoperation integrates humans in the control loop. There personnel in emerging nations. personnel in emerging nations. teleoperation integrates humans in the control control loop. There There are no existing models for the human teleoperator, yet. Within this publication, we regard the following use-case: teleoperation integrates humans in the loop. are no existing models for the human teleoperator, yet. Within this publication, we regard the following use-case: are no models for the teleoperator, yet. Within this publication, regard use-case: To address this issue, we aa strict user-centered There is aa manufacturer’s center from no existing existing models forpropose the human human teleoperator, yet. Within publication, we we telemaintenance regard the the following following use-case: To address this issue, we propose strict user-centered There is isthis manufacturer’s telemaintenance center from are To address this issue, we propose a strict user-centered There a manufacturer’s telemaintenance center from approach. Third, there are different prevalent end user which an engineer, the expert, provides technical expertise To address this issue, we propose a strict user-centered There is a manufacturer’s telemaintenance center from Third, there are different prevalent end user which an an engineer, engineer, the the expert, expert, provides provides technical technical expertise expertise approach. approach. Third, are different prevalent end user which access technologies to the differ in their to the local repair personnel, the service technicians, at approach. Third, there there areInternet, differentwhich prevalent end user which an engineer, the expert, provides technical expertise access technologies to the Internet, which differ in their to the local repair personnel, the service technicians, at access technologies to the Internet, which differ in their to the local repair personnel, the service technicians, at expectable Quality of Service (QoS). Hence, our framework access technologies to the Internet, which differ in their toThis the work local was repair personnel, the service technicians, at expectable Quality of Service (QoS). Hence, our framework funded by the Bavarian Ministry of Economic This work was funded by the Bavarian Ministry of Economic expectable Quality of (QoS). our framework must be able to detect the characteristics the provided expectable Quality of Service Service (QoS). Hence, Hence,of our framework must be able to detect the characteristics of the provided Affairs, Transport Technology as partof ofEconomic the R&D This work by Bavarian Ministry This Infrastructure, work was was funded funded by the theand Bavarian Ministry must be able to detect the characteristics of the provided Affairs, Infrastructure, Transport and Technology as partofofEconomic the R&D end-to-end connection, and decide which services are apmust be able to detect the characteristics of the provided program “Information and Communication Technology”. Affairs, Infrastructure, Transport and Technology as part of the R&D end-to-end connection, and decide which services are apAffairs, Infrastructure, Transport and Technology as part of the R&D program “Information and Communication Technology”. end-to-end connection, and decide which services are end-to-end connection, and decide which services are apapprogram program “Information “Information and and Communication Communication Technology”. Technology”. Copyright © 2016, 2016 IFAC 6 Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2016 IFAC 6 Copyright 2016 IFAC 6 Peer review© of International Federation of Automatic Copyright ©under 2016 responsibility IFAC 6 Control. 10.1016/j.ifacol.2016.11.114
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Michael Fritscher et al. / IFAC-PapersOnLine 49-30 (2016) 006–011
plicable. Fourth, it needs to orchestrate the transmission of data streams from multiple sources through a secured connection. This includes adapting the services to connection quality changes during ongoing telemaintenance sessions. In the next paragraph, we give a short survey of the state of the art in telemaintenance, while in the remainder of this publication, we present our approaches to the aforementioned challenges and offer insight into the first prototype tests.
7
Interaction
Interaction
Application
Application
Service
Service
Telematics Connection
Telematics Connection
Transport
Transport
Network
Network
Data Link
Data Link Physical
2. STATE OF THE ART
Fig. 1. Layers of the AMS architecture on top of the OSI transport layer
The term telemaintenance refers to the integration of computer science and communication technologies into the maintenance strategy (Chowdhury and Akram (2011); Mouzoune and Taibi (2014)). We understand the challenges in telemaintenance as a combination of teleoperation and remote condition monitoring. In this publication, we focus on the teleoperation aspect.
be supported are specified. Subsequently, the Application Layer and its user interface concepts are explained. This is followed by an overview of the Service Layer providing the underlying functionality for the Application Layer. In the last subsection, we survey the Telematics Connections Layer, which resides, as depicted, above the OSI transport layer. It provides a single entry point into the center or facility for the secured data transmission.
In general, ”a robotic teleoperation system allows to reproduce the actions of a human operator and to interact physically with objects and environments placed at a distance” (Melchiorri (2014)). Hence, it can be seen as a subcategory of human supervisory control (Sheridan (1992)). For example, visual feedback can provide information about the state of an industrial robot to the human operator in the control loop. For an overview of applications see Lichiardopol (2007); in this paper, we focus on teleoperation of industrial robots, which has been of early (Cederberg et al. (2002)) and late (Moradi Dalvand and Nahavandi (2014)) research interest. The application for teleoperation of industrial robots lies mainly in the context of hazardous environments (Pegman et al. (2006)), but in our scenario we cover maintenance scenarios similar to those for tunneling machines covered in David et al. (2014). As we aim to facilitate teleoperation via the Internet, the control perspective with time delay over the Internet (Slama et al. (2007)) has to be integrated as much as human perception of time delays (Vozar and Tilbury (2014)). There have been several proposals for teleoperation architectures, e.g. in Gray et al. (2007); Ortega et al. (2014). Our project partner Kuka Industries, former Reis Robotics, provides an architecture that includes basic teleoperation functionalities. However, there is a need for an enhanced architecture that also includes further services needed for human supervisory control, such as synchronized video feedback. The closest related work has been provided by Jia (2014), who covers multimedia streams, too (Elhajj et al. (2011)), but does not include other kinds of data needed for maintenance. To our best knowledge, there is no integrated teleoperation and maintenance architecture yet.
3.1 Interaction Layer Top-down, the first layer of our architecture is the Interaction Layer, which follows an user-centric design approach. This layer serves as an adaption layer between our architecture and the business processes at the facility and center. We performed a broad contextual analysis involving interviews with managers, engineers, repair personnel and production workers. Based thereon, we modelled the workflows of experts and service technicians and identified use-cases and teleoperation scenarios (Sittner et al. (2013)). For the AMS architecture teleoperation (as defined in Sheridan (1992)) is the key scenario. Here, the control loop is closed through the human and the control computer. We extend the model (as in Fig. 2) by the local service technician, which has a very important role due to safety issues. In Europe teleoperation of an industrial robot in an automation facility is only permitted if there is a local supervisor present. During our experiments, this service technician always has to press the dead man’s switch to enable the external control. Therefore, he also performs human supervisory control and mirrors the conceptual structure of the main teleoperator (the expert). As it can be derived from Fig. 2, communication mostly takes place between the expert and the service technician. Typical cases for maintenance calls are unknown problems at the production line, which need to be solved as quickly as possible. The expert knows the technical details of the robot quite well, but needs to get an impression of the periphery and the exact configuration of the robot in order to be able to analyze the failure. The AMS technology is designed especially for those questions by providing several cameras at the production line and a mobile camera in the tablet computer which the service technician carries as a Local Human Interactive System which can also serve as robot teach pendant (Aschenbrenner et al. (2014)).
3. AMS ARCHITECTURE The Adaptive Management and Security System (AMS) is designed as a modular multi-layer architecture, in which each layer provides functions for the layer above. Each layer comprises a manager component exchanging control information with the managers of adjacent layers. In the following sections, we explain the four layers of the AMS architecture top-down, as depicted in Fig. 1. We begin with the Interaction Layer, on which the work flows to
3.2 Application Layer The Application Layer provides the graphical user interface (GUI) to the user. This is also where the vendor 7
2016 IFAC TA 8 November 6-9, 2016. Porto Alegre, Brazil Michael Fritscher et al. / IFAC-PapersOnLine 49-30 (2016) 006–011
App Layer App Layer Component 1
Service Layer
TC Layer
Service
TC Layer Multiplexer
Routine C B A
…
CENTER
App Layer Component n
FACILITY
Manager
Service
…
TC Layer Component
Routine C
outine
B
A
Manager
App Manager
Service Manager
TC Manager
Fig. 2. Extended Human Supervisory Control Model
Fig. 5. Service Interactions
Fig. 3. GUI for the Remote Human Interactive System
facility. The small traffic light on the right indicates the current end-to-end Quality of Service between facility and center. Both figures depict the usage of the Paint App. Using this application, both expert and service technician can draw simple outlines into the video view, e.g., to point out areas of interest. Altogether, the expert is able to teleoperate the robot, communicate with the service technician, get video and data feedback and use the Paint App. The GUI on the mobile device used by the service technician, as shown in Fig. 4, is a reduced version of the expert’s GUI, with one switchable video view. The service technician can communicate and receive feedback from the camera and through the Paint App. 3.3 Service Layer The Service Layer provides the underlying functionalities for the components on the Application Layer. The main objective of the Service Layer is to facilitate re-usability by offering stackable generic and specialized components. The main components of the service layer are the ServiceManager and the services themselves. The ServiceManager is responsible for configuring, starting and stopping the local services. It exchanges management information with the managers of the neighbouring layers, as depicted in Fig. 5. In addition, the ServiceManager is designed to establish a single point of configuration: The master ServiceManager instance, typically located at the telemaintenance center provides the configuration for all slave AMS instances.
Fig. 4. Service technician using the Local Human Interactive System specific hard- and software components, often including their own GUI, are located. The GUIs are displayed on the Remote and the Local Human Interactive System depicted in Fig. 2. We extended our GUI framework, originally developed for the teleoperation of heterogeneous multirobot systems (Fritscher et al. (2012)), to support the teleoperation applications derived from the Interaction Layer.
Fig. 5 visualizes the structure of services and the way services interact with components of their own and adjacent layers. The broad arrows represent data flows while the thin arrows represent control connections. The Application Layer component depicted on the left relies on functionality or data provided by the service shown in the upper middle of the figure. The service itself may, as depicted, rely on other local services, e.g., for common tasks like time synchronization. For example, the storage service combines data from the video service and the robot service, synchronizes this data and stores it for condition monitoring. Services can communicate with remote services belonging to another AMS instance by using the TC Layer (right). In addition to these data streams, all
The prototype GUIs for the expert and for the service technician provide the basic applications needed for telemaintenance: The expert GUI, as depicted in Fig. 3, contains a text chat window (on the left). The robot can be controlled with the sliders, where also the current robot coordinates are shown. Three video views provide insight into the 8
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Michael Fritscher et al. / IFAC-PapersOnLine 49-30 (2016) 006–011
Center App Layer
Facility
Service Layer
TC Layer
Service Layer
Service data
Robot
Robot
Robot
Com
Com
Com
Com
AR VR
AR VR
Video
Video
Manager
Data
Storage
Manager
Control
...
Measurement 1
...
Cam
Measurement m
Data
Storage
Manager
Service data
ServiceConnection 1
Paint
...
TC Layer
Tunnel
ServiceConnection n
Paint Video
Config
TC Layer
App Layer
Provis
Data Show
...
9
Fig. 7. Composition of data streams in the encrypted tunnel
Config
Manager
Manager
Fig. 7 provides an overview of the data flows: The gray arrows depict the data originated by and intended for services. This data is sent through the secured tunnel to the AMS instance on the other side. The lower part of the figure visualizes the different kinds of connections in the tunnel: First, there is a control connection, used for information exchange between the managers of the TC Layer instances. Subsequently, there can be one or more ServiceConnections, each of which represents one connection from a local service to a remote one. The third kind of data transmitted is the test traffic generated by end-toend measurements originated by test routines belonging to the TC Layer. The manager uses data derived from these measurements to dynamically assign each service a fraction of the available network resources. In the following paragraph, we explain the main components of the TC Layer and how they interact in order to provide the main functionalities.
Fig. 6. Example AMS applications and services services keep a control connection to the ServiceManager depicted at the bottom of the figure. Services may contain so called routines, in which algorithms are implemented, or wrappers for external components, e.g., video streaming software or virtual machines. Each service contains a small internal Manager sub module taking care of communication with the ServiceManager. In addition, these sub modules are responsible for adapting the amount of data sent by the services. If the available bandwidth changes significantly, the LayerTC calculates the new budgets for the connections and send them to the ServiceManager. The services get the new budgets and adapt their transfers by e.g. changing video streaming parameters, using still pictures instead of video streams, disabling uncritical services and so on. So the available bandwidth is used in the optimal way (regarding used bandwidth and the priorities of the connections which are needed by the service bundle) by using adaptive connections instead of fixed ones. As of now, all services are custom implementations. But in general, a service may be any kind of software runnable as process of the underlying operating system.
Fig. 8 depicts the internal structure of the TC Layer, whereat the thin arrows represent control data or method calls and the broad arrows symbolize service data sent or received through the secured connection. The QoS measurement system consists of a set of components, as is pictured on the right (surrounded by the dashed outline). The main component is the Monitor, a system capable of starting different network test programs, the
As depicted in Fig. 6, each application is supported by one or more services: There are e.g. services for the transfer of video streams, chat functions and synchronized storage of machine data. Some generic services, like time synchronization or video streaming, are needed by other services, while specialized services provide access to vendor-specific software. Another important task fulfilled on the Service Layer is to provide access to the hardware, like robots, video cameras and external sensors. The ProvisConnector, for example, is a special Robot Service that integrates Reis Robotics’ “Provis” robot configuration and programming software into the AMS system.
Multiplexer InputLinkData InputLinkControl
Classful Classifier Policer Queuing
Output Link
VPN Network adapter interface
Shaper
3.4 Telematic Connections (TC) Layer
ServicesDB
The Telematics Connections Layer, or TC Layer, is located beneath the Service Layer of the AMS Architecture. It provides a secure connection between two or more endpoints, e.g. between a telemaintenance service center and the environment of a machine in a distant facility. This layer works as a stateful firewall and is responsible for admittance and blocking of service connections. In addition, it provides information about the end-to-end QoS between the AMS instances to the higher layers.
Measurement Monitor ParameterRequest
ConfigDB
Measurement DB
ProfileDB
TC Manager
Fig. 8. Schematic overview of the Telematics Connections Layer 9
2016 IFAC TA 10 November 6-9, 2016. Porto Alegre, Brazil Michael Fritscher et al. / IFAC-PapersOnLine 49-30 (2016) 006–011
Measurements. The Monitor pre-evaluates the outcome of these tests and places the results in the MeasurementDB. The TC Manager is depicted on the bottom of the figure. This component is responsible for bandwidth assignment and communicates with the ServiceManager for configuration purposes. The bandwidth assignment is calculated in the co-located Decider subsystem: The base configuration for the Decider is loaded from the ConfigurationDB, while the decisions are based on data from the MeasurementDB and a set of ServiceProfiles containing the constraints and priority of each admissible service. The TC Manager passes the Decider’s results to the TC Manager on the remote AMS system, to the Service Manger and to the Multiplexer. Information about admitted service connections is stored into the ServicesDB, along with a history of decisions regarding the service.
Fig. 9. Test person with mobile device following the expert’s instructions in front of the control cabinet
The Multiplexer is a QoS-aware throughput-limiting stateful firewall implementation, which guards the end of the secured connection towards the other AMS instance. The filtering rules, bandwidth assignment decisions and basic instructions regarding priorities and traffic classes are provided by the TC Manager. These instructions are translated into rules for Linux’ Netfilter / Iptables modules and queuing instructions.
The current prototype implementation of the AMS software was evaluated in a milestone test during October 2014 including our project partners Reis Robotics and Braun / Procter & Gamble. For this test, the service center was set up at Reis Robotics in Obernburg, Germany. It consisted of a Dell Latitude E6410 laptop, an additional monitor and the AMS software, a mobile phone and a Reis Robotics robot programming device. The facility to be serviced was the environment of an injection molding machine producing parts for electric toothbrushes, in the factory of Braun / P&G in Marktheidenfeld, Germany. It is comprised of an injection molding machine, an assembly adding membranes to the molded parts, and a Reis Cartesian robot handling the parts. In addition, the facility was equipped with a computer, two high definition IP cameras, and a mobile device (Microsoft Surface Pro tablet) with built-in camera to be used by the service technician. As the facility’s network is located inside the worldwide P&G virtual private network (VPN), the service center was connected to the P&G network by a host-to-site VPN via the P&G VPN gateway in Brussels. The goal of the test was to determine, whether the implemented software is adequate for use in a teleoperation scenario, in which the expert performs several tasks remotely and guides the service technician to identify and solve a problem concerning the robot. In addition, we wanted to evaluate to what extent the system behaviour while adapting to changes in the connection quality would impair the users’ workflows in a real maintenance scenario. This scenario was a motor exchange on one of the axis of the robot, which can normally only be done by a certified service technician.
The modules, which, comprise a QoS-supporting router and firewall are depicted within the Multiplexer-box in Fig. 8. The packets sent by the local services are first filtered by the stateful firewall, based on Netfilter / Iptables: Here, packets are classified and marked according to their service class, dropping packets not matching any rule. The policing is also implemented in the firewall: Traffic flows not matching the policy of their service class, e.g. by exceeding their bandwidth quota, are marked with an “excessive” flag. Depending on the settings for the service, excessive traffic can either be rejected by the firewall or delayed. The shaping and queuing of the outgoing traffic are implemented by creating a set of queues and disciplines for the traffic control module from Linux. The priorities of the different traffic classes are currently realized by priority queuing, based on the marks set by the firewall. In addition, traffic marked as excessive can be artificially delayed (shaping) by sending it through an additional queuing discipline. After traversing the queues, the packets are encapsulated and sent through the secured connection, which is set up using OpenVPN.
4. FIRST TESTS IN AN INDUSTRIAL ENVIRONMENT
In the initial setup, three video streams were transferred to the expert: The first provided an overview of the facility, the second stream was capturing the details of the assembly machine, while the third stream was captured by the camera of the service technician’s mobile device. After connecting, the expert was able to control and program the robot, like normally done over a local area network. The expert was able to steer the robot remotely using video and Provis’ AR feedback. Subsequently, the local service technician, as well as one participating scientist performed multiple tasks under expert supervision. For example: “go to the control cabinet and take a picture of the voltage information in the display of a certain component”, as shown in Fig. 9.
The evaluation of the telemaintenance and teleoperation scenarios poses a challenge, as only a limited number of specially trained experts are available and each teleoperation test causes a major interruption in the production line. Hence, we mainly focused on qualitative research: We evaluated different features in experimental testbeds and tested the applications’ functionality and usability in weekly experiments with service technicians at the production site. Short tests of the teleoperation components were carried out biweekly during the last year as explained in Aschenbrenner et al. (2015). 10
2016 IFAC TA November 6-9, 2016. Porto Alegre, Brazil Michael Fritscher et al. / IFAC-PapersOnLine 49-30 (2016) 006–011
To facilitate communication between the participants, both a chat functionality and a screenshot & paint functionality were available in the GUI. Then we triggered the bandwidth adaption routines with false inputs to emulate a deterioration of the end-to-end connection quality. This consisted on a drop from 3 Mbit/s to 300 kbit/s per video stream. Subsequently, the system adapted the video quality to prevent impairment of the higher priority services. The system needed 5 seconds on average to adapt (restarting the video streams), which was noticeable but still ok for the participants. Further advantages in this area made after the test using spare encoding processes dropped this delay to under one second.
11
ACKNOWLEDGEMENTS The authors would like to thank their project partners at Reis Robotics and Braun / Procter&Gamble for their participation and commitment to the MainTelRob project. REFERENCES Aschenbrenner, D., Leutert, F., Sittner, F., and Schilling, K. (2014). Einsatzm¨ oglichkeiten f¨ ur Mobile Ger¨ ate bei der Wartung von Industrierobotern. In SPS Drives Kongress. Aschenbrenner, D., Sittner, F., Fritscher, M., Krauss, M., and Schilling, K. (2015). Teleoperation of an industrial robot in an active production line. In Submitted Paper for: 2nd IFAC Conference on Embedded Systems, Computational Intelligence and Telematics in Control (CESCIT). IFAC. Cederberg, P., Olsson, M., and Bolmsj¨ o, G. (2002). Remote control of a standard abb robot system in real time using the robot application protocol (rap). In Proceedings of the 33rd ISR (International Symposium on Robotics) October, volume 7, 11. Chowdhury, S. and Akram, A. (2011). E-maintenance: Opportunities and challenges. In The 34th Information Systems Research Seminar in Scandinavia (IRIS), Turku, Finland, August 16-19, 2011, 68–81. Turku Centre for Computer Science. David, O., Russotto, F.X., Simoes, M.D.S., and Measson, Y. (2014). Collision avoidance, virtual guides and advanced supervisory control teleoperation techniques for high-tech construction: framework design. Automation in Construction, 44, 63–72. Elhajj, I.H., Dargham, N.B., Xi, N., and Jia, Y. (2011). Real-time adaptive content-based synchronization of multimedia streams. Advances in Multimedia, 2011, 4. Fritscher, M., Hess, R., and Schilling, K. (2012). Generic network and a browser-based hmi. In IARP RISE 2012 6th Workshop (RISE2012). Gray, J., Lippiello, V., Villani, L., and Siciliano, B. (2007). An open architecture for sensory feedback control of a dual-arm industrial robotic cell. Industrial Robot: An International Journal, 34(1), 46–53. Jia, Y. (2014). Teleoperation of mobile manipulators. Ph.D. thesis, Michigan State University. Lichiardopol, S. (2007). A survey on teleoperation. University of Eindhoven, Department Mechanical Engineering Dynamics and Control Group Eindhoven. Melchiorri, C. (2014). Robot teleoperation. In J. Baillieul and T. Samad (eds.), Encyclopedia of Systems and Control, 1–14. Springer London. Moradi Dalvand, M. and Nahavandi, S. (2014). Teleoperation of abb industrial robots. Industrial Robot: An International Journal, 41(3), 286–295. Mouzoune, A. and Taibi, S. (2014). Introducing e-maintenance 2.0. arXiv preprint arXiv:1401.8252. Ortega, J.G., Garc´ıa, J.G., Nieto, L.N., and Garc´ıa, A.S. (2014). Open software architecture for advanced control of robotic manipulators. Pegman, G., Desbats, P., Geffard, F., Piolain, G., and Coudray, A. (2006). Force-feedback teleoperation of an industrial robot in a nuclear spent fuel reprocessing plant. Industrial Robot: An International Journal, 33(3), 178–186. Sheridan, T.B. (1992). Telerobotics, automation, and human supervisory control. MIT press. Sittner, F., Aschenbrenner, D., Fritscher, M., Kheirkhah, A., Krauss, M., and Schilling, K. (2013). Maintenance and telematics for robots (maintelrob). In 3rd IFAC Symposium on Telematics Applications Conference (TA2013), Seoul. IFAC. Slama, T., Aubry, D., Oboe, R., and Kratz, F. (2007). Robust bilateral generalized predictive control for teleoperation systems. In Control & Automation, 2007. MED’07. Mediterranean Conference on, 1–6. IEEE. Vozar, S. and Tilbury, D.M. (2014). Driver modeling for teleoperation with time delay. In IFAC World Congress, volume 19, 3551–3556.
The experts and technicians were content with the features offered by the prototype. Both chat and paint features were considered very helpful, also because of the service technician being in a very noisy environment. Also, guiding the technician based on the video from the mobile device (”not that display!”), proved helpful. While the motor exchange was done with a certified service technician because of safety reasons, both he and the expert on the other side confirmed that using this tool enable also not certified service technicians to fulfill this job. So they expect the completed system to be a very helpful tool in future telemaintenance usecases. One important thing we learned was that especially the expert needs additional training for remote assisting in e.g. the field of responsibility (What happens if there is an accident because I forgot something or the service technician counts on me too much?) Because of these results, the next tests will be with a group of uncertified service technicians and a check list for experts. Additionally, the rules ”The service technician is responsible for his safety” and ”If there is something unclear, the service technician must always ask” will be introduced.
5. CONCLUSIONS In our research project MainTelRob - Maintenance and Telematics for Robots, we and our industrial partners’ engineers, technicians, and production workers identified usecases for the telemaintenance of industrial robots. In this paper we focused on a remote maintenance scenario involving teleoperation and motivated, why it was necessary to develop a novel architecture that provides all of the requested functionalities. Our Adaptive Management and Security System facilitates teleoperation of an industrial robot over the Internet. The framework combines a modular architecture and generic services with the ability to measure the end-to-end QoS of the used network connection and adjust the service combination adequately. The GUI part of the framework allows for fast integration of new features, next to the existing ones. To test our architecture in a real industrial telemaintenance setting, we implemented a prototype providing a specific combination of features. We tested our concepts and the prototype through continuous experiments in our testbed as well as in the production environment. In our future work, we plan to extend the system for further use cases, e.g. by providing enhanced Augmented Reality overlays. 11