Context-aware Maintenance Support for Augmented Reality Assistance and Synchronous Multi-user Collaboration

Available online at www.sciencedirect.com

ScienceDirect Procedia CIRP 59 (2017) 18 – 22

The 5th International Conference on Through-life Engineering Services (TESConf 2016)

Context-aware Maintenance Support for Augmented Reality Assistance and Synchronous Multi-user Collaboration Michael Abramovici, Mario Wolf*, Stefan Adwernat, Matthias Neges Chair for IT in Mechanical Engineering (ITM), Ruhr-Universität Bochum, Universitätstrasse 150, 44780 Bochum, Germany * Corresponding author. Tel.: +49-234-32-22115; fax: +49-234-32-14443. E-mail address: [email protected]

Abstract Maintenance processes are generally subdivided into tasks with specified goals for the concerned practitioners. Collaboration is achieved through coordination, cooperation and communication. Smart devices and the Internet of Things (IoT) improve communication between men and machine alike, so the potential gain on cooperation and coordination in an IoT-enabled environment is examined. In this approach a concept for a framework is proposed to create Augmented Reality based collaboration assistant systems. AR is not only a tool for the visualization of maintenance data, but also for team-wide communication and the display of warnings or other coordination-related indications. The framework is validated with the presented use case in a laboratory environment.

©2016 2016The The Authors. Published by Elsevier B.V. © Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license Peer-review under responsibility of the Programme Committee of the 5th International Conference on Through-life Engineering Services (http://creativecommons.org/licenses/by-nc-nd/4.0/). (TESConfunder 2016). Peer-review responsibility of the scientific committee of the The 5th International Conference on Through-life Engineering Services (TESConf 2016) Keywords: Collaboration; Augmented Reality; mobile devices; maintenance

1. Introduction “Industry 4.0” aims to achieve high flexibility and short lead times to face the growing competitive pressure between companies, therefore increasing automated production processes and resulting in a rising complexity of machinery. In this context maintaining one’s machinery becomes ever more important to guarantee reliability and thereby ensure the company’s supply capability [1]. Unsurprisingly, a major part of life cycle costs originates from the maintenance, repair and overhaul (MRO) processes. Therefore, it is essential to consider them in the life cycle engineering [2]. As terms like “Industry 4.0”, “Internet of Things”, “Brilliant Factory” or “Smart Planet” emerge more often it becomes clear that the increasing amount and complexity of data will continue to challenge today’s engineers. The term “Smart Maintenance” [3] is used to address new structures and platforms for maintenance services. It emphasizes the connection between asset management, sensor data and intelligent on-site maintenance support to create a reliable condition based maintenance strategy.

In the course of IoT-related innovation and emerging trends for product service systems (PSS) the improvements in prediction and simulation capabilities help to calculate the MRO costs [4]. Furthermore, the usage of standardized architectures for IoT enabled devices and the corresponding multi-platform support systems enable a vast advantage for the detection of errors, especially in big fleets of machines [5]. Interactive augmented reality (AR) technologies support the operators on shop-floor level as much as they provide the user of an AR application with a big amount of easy to understand context-relevant information. According to Syberfeldt et al. [6], the AR technology will be an integral part of future factories. Dini [7] states that even today the vast majority of AR application in a through life engineering (TES) context is used in maintenance (54%) or inspection (24%). Furthermore, a problem with AR applications is their static nature due to the high amount of preparation work needed. A dynamic, intelligent AR application would improve the state of the art in AR development for maintenance. Regarding the fact that Smart Devices are state of the art nowadays, the usage of these

2212-8271 © 2016 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the scientific committee of the The 5th International Conference on Through-life Engineering Services (TESConf 2016) doi:10.1016/j.procir.2016.09.042

Michael Abramovici et al. / Procedia CIRP 59 (2017) 18 – 22

devices is no hindrance for today’s personnel [8] and as a result, they could be utilized for AR supported maintenance. In order to reduce costly downtimes, maintenance activities could be performed by multiple technicians in a collaborative process. Leimeister [9] defines collaboration as a systematically organized activity performed by two or more individuals working jointly on the same material with the purpose of realizing a shared goal. To achieve this common goal, coordination, cooperation and communication among the related actors is necessary. The involved roles can be differentiated between the collaboration engineer, practitioner and facilitator [9]. In recurring high-value collaborative processes the collaboration engineer designs and documents the collaborative process, whereas the practitioner acts as a facilitator by both executing and organizing the defined process. Collaboration among its participants exists in different dimensions regarding the categories organization, location and time. The characteristics of each category can be divided into same and different organization, location and time [10]. Based on the authors‘ previous work [11] where contextsensitivity in machines was discussed a simplistic AR approach that can be used in the field by multiple technicians was designed. In the following sections we determine requirements for a collaborative AR-driven maintenance assistant, define the planned functionalities, describe the theory behind the graphbased evaluation and coordination of technicians and validate our approach in a laboratory environment. 2. Aims and requirements Based on our previous work [11] in which we used the mobile AR technology for interactive input methods, the main goal of the paper at hand is the improvement of on-site collaboration by means of interactive coordination and individual work instructions. The main tool to enable technicians to receive instructions and send and receive coordination relevant data (for simplicity called messages further on) is a smart device. The smart device hosts an

Figure 1: Overview of the proposed architecture

enhanced version of the assistant system described in [11]. As described before collaboration can be characterized through the organizational form, location and time that it is conducted in. The approach at hand will focus on synchronous collaboration on the same facility with working environments apart for the individual technicians. Porcelli et al. [12] mention a computer system for running the application as a necessary core component of typical AR systems. A digital camera captures the real scene, whereas a displaying device, such as a Head Mounted Display (HMD), a handheld computer or a projector is needed to see the augmented content. A tracking system tracks the position and movement of users as well as objects and links the augmented content to real elements. For the interaction with the above mentioned computer system, input devices such as gloves, tablets or PDAs deliver required data. Due to the advantages of user-friendly consumer hardware [11] and their vast acceptance [8] we focus on smart devices, which also combine the components stated before into a single device. HMDs might offer advantages over in the field of visualization [7], but lack the maturity level of industrial tablet PCs. According to Wang et al. [13] the multidisciplinary collaboration will be future standard, however there are current deficiencies of collaborative Augmented Reality systems. Approaches utilizing AR in a collaborative manner such as [14], [15] and [16] only to some extend address the elements of collaboration defined in [9] and previously described in the introductory section. Especially recurring high-value collaboration processes, on which we focus in this approach, require well-structured and easily executable activities performed by the practitioners [9]. In addition to the requirements regarding the AR system and the collaboration among the related actors, the collaborative maintenance process itself and the information retrieval need to be considered in our approach. It needs to support the core elements of collaboration, i.e. coordination, cooperation and communication. Furthermore, the users should be able to receive context-aware information about the machine’s status

19

20

Michael Abramovici et al. / Procedia CIRP 59 (2017) 18 – 22

and work instructions or information about co-working technicians on-site. To that extend the concepts published in [11] and [17] will be partly included into the approach for maintenance support for multiple technicians on-site. The building of such complex models is time-consuming and the processes concerning the planning/contracting of maintenance are manifold. The approach at hand therefore focuses on the support system onsite rather than on the early phases of maintenance planning. 3. Concept Haberfellner and Daenzer [18] cite that models as a depiction of reality must be convincing enough in regards to the situation and problem position. So the question of functionality and problem relevance is to be asked when developing a model. Harrison [19] states that the strength of graph databases lies especially on the use cases in which the correlation of elements is more important than the elements themselves. After the consideration of the preceding section and based on the graph-based model the authors described in [11], the changes in the given situation made it necessary to implement adaptations to the architecture. The considered processes in this concept include the creation of tasks to fulfill goals, the task decomposition into smaller activities to fulfill the whole task, the assignment of technicians to said activities and the actual on-site collaboration and individual maintenance assistance. It should be clarified that the functionalities of this approach are based on the considerations of Harrison, Haberfellner and Daenzer to find out which things are relevant to the given activities. Furthermore, it is necessary to focus on connections that exist between those things and the rules that bind things

Figure 2: Graph representation of the described problem

and connections together. In regard to Harrison [19] things in the real world can be displayed as knots in a graph. Connections between the individual things can be depicted with edges. Constraints ensure the necessary rules for validity of connections so that only certain types of knots may have certain types of in- and outgoing edges. According to Locke [20] a goal is a desired result or a desired status, so in the context of maintenance the general goal is the (functionally) desired status of the concerned machine. Goals can be characterized by the SMART criteria [20]: x Specific. Goals must be specific with precise instructions and desired status x Measurable. Goals must be measurable to judge the completion of a goal x Accepted. Goals must be accepted by the participants in regard of goal attainment x Realistic. Goals must be realistic so they can actually be completed x Terminated. Goals must be terminated for anticipatory actions These criteria offer a guideline which will be followed to enhance the author’s proven approach [11]. To enable the management of tasks, facilitators and practitioners for individual assignments, Figure 3 shows the created entities and their relations that can be managed in the graph-based model. In the following the concept will be made clear based on selected parts of the six-step concept for collaborative engineering [9] and the functionalities of our proven approach. As mentioned in the aims and requirements section the approach at hand will show potential for newly designed techniques bottom-up, starting in the operational phase. The focus lies strictly on recurring high-value collaboration, which

21

Michael Abramovici et al. / Procedia CIRP 59 (2017) 18 – 22

means that practitioners act as facilitators in the organization and execution of the collaboration. The infrastructure of the approach is depicted in Figure 1. Every machine that is to be serviced has to be part of the Internet of Things for the planning, tracking and execution of the collaborative work processes, whereas technicians are to be equipped with smart devices with an internet connection to make use of real-time synchronization of data between all concerned IoT entities.

“Pump 1”, no additional action is needed. The same goes for the status of valves in the follow-up task. Only for non-closed valves would an instruction be generated. The collaborative aspect adds another level of complexity to the maintenance support assistant. Generally, there are different views on the data in the graph-based model. The first is the administrative or master view on the model. Without restrictions one is able to see the progress of each technician in every activity he is involved in. Furthermore, the complete status of the machine on-site is visible (given a feasible type of visualization). The view of each technician is not as holistic as the so-called master view, because is neither necessary nor helpful. The technicians view is focused on the current activity to establish the desired state, which can be delayed or even interrupted by actions of other technicians or system failures and their resulting warnings. It is implicit that every status Pump Pump 2

Relay P2

Value Off

Manual Input

detached

Sensor

Activity Switch off Pump 2

Technician

John Doe

Figure 3: Relations between task planning entities

A Job in this context represents a terminated assignment to meet the customer’s needs for maintenance service. A Task is a distinct action to be fulfilled and coordinated, while an Activity is a specific, measurable, realistic action that is specifically assigned to a Technician in order to be executed. To be measurable the graph-based model connects the individual desired status to the concerning activity, giving the algorithms a specific value to calculate with. These functionalities are shown in Figure 2. Starting off with a task “Remove P1”, which is dependent on the completion of certain activities. These activities are connected to both a specific technician as well as the desired result of the activity. Results are not stored in an isolated matter, but in relation to the concerning sensor device, which is mounted to a specific part of the machine at hand. No work instruction would be created if the desired status of an activity is met before the technician starts working. Meaning that when “Pump 2” is already shut down after shutting down

Figure 5: Validation use case in graph representation

MI 1

Value

Activity Unscrew bracket

Value attached

Figure 4: Elements for the handling of manual inputs

update, whether it is automatic through the IoT-enabled machine or manual input from the technician, must be processed in the cloud collaboration platform, which in turn sends notifications to all concerned listening clients. For this matter sensory data is captured and processed automatically and manual inputs can be set where sensors are not applicable, i.e. for purely mechanical tasks like unscrewing a bracket. For the automatic generation of the graphical user interface (GUI) dialogs to capture non-sensor inputs, the possible states must be defined beforehand. For that matter a non-sensor kind of knot for the graph-based model is needed that can take

22

Michael Abramovici et al. / Procedia CIRP 59 (2017) 18 – 22

connections to desired and other valid states. Figure 4 shows the implementation of this pattern. The manual input class knots offer the same functionality as sensor class knots, with the exception of the missing automatic retrieval of data. As far as communication is concerned, the concept offers a general functionality to send a warning or call a colleague to take a look at the current working. While waiting for the completion of a relevant activity, the current progress of said activities is shown to the waiting technician.

4. Prototype implementation and validation To validate the capabilities of the presented approach a prototypical implementation was conducted with the test system being the hydraulic system used in [17]. We modelled a task as depicted in Figure 5 to coordinate two technicians and attached the desired values and dependencies accordingly like shown in Figure 2. The graph-based model is implemented in OrientDB with PHP web services. The Frontend for the technicians was developed using an Android Tablet and the PTC Vuforia SDK. In our validation scenario John Doe is in front of a hydraulic system, Laura Ipsum is (should be) on the backside of the machine. As can be seen in Figure 5 in both Tasks John Doe needs to take action before Laura Ipsum can go to work. A screenshot from the maintenance assistance application can be seen in Figure 6, in the situation when Laura Ipsum is waiting for the completion of the two tasks of John Doe.

Figure 6: Screenshot of the prototypical application

5. Conclusion The authors have shown that the approach at hand supports collaboration in a laboratory environment. Using the cloud-side dynamic data model created for single-user support and enhancing its capabilities to distribute collaboration-relevant messages to multiple users, the smart device based assistant system is able to support groups of technicians coordinating their work. Future work will focus on testing the approach with both a more complex system and more complex tasks. The approach will be enhanced in regard of the planning process of collaborative engineering in order to establish a holistic framework for collaborative engineering processes by means

of Augmented Reality. After the second laboratory experiment our goal is a real-world validation on an industrial site. References 1. Pawellek G (2016) Integrierte Instandhaltung und Ersatzteillogistik: Vorgehensweisen, Methoden, Tools, 2. Aufl. 2016. VDI-Buch. Springer Berlin Heidelberg, Berlin, Heidelberg 2. Dhillon B (2013) Life Cycle Costing: Techniques, Models and Applications. Taylor and Francis, Hoboken 3. Yokoyama A (2015) Innovative Changes for Maintenance of Railway by Using ICT–To Achieve “Smart Maintenance”. Procedia CIRP 38: 24–29. doi: 10.1016/j.procir.2015.07.074 4. Stark R, Grosser H, Beckmann-Dobrev B et al. (2014) Advanced Technologies in Life Cycle Engineering. Procedia CIRP 22: 3–14. doi: 10.1016/j.procir.2014.07.118 5. Lee J, Ardakani HD, Yang S et al. (2015) Industrial Big Data Analytics and Cyber-physical Systems for Future Maintenance & Service Innovation. Procedia CIRP 38: 3–7. doi: 10.1016/j.procir.2015.08.026 6. Syberfeldt A, Holm M, Danielsson O et al. (2016) Support Systems on the Industrial Shop-floors of the Future – Operators’ Perspective on Augmented Reality. Procedia CIRP 44: 108–113. doi: 10.1016/j.procir.2016.02.017 7. Dini G, Mura MD (2015) Application of Augmented Reality Techniques in Through-life Engineering Services. Procedia CIRP 38: 14–23. doi: 10.1016/j.procir.2015.07.044 8. Abramovici M, Göbel JC, Neges M (2015) Smart Engineering as Enabler for the 4th Industrial Revolution. In: Fathi M (ed) Integrated Systems: Innovations and Applications. Springer International Publishing, Cham, pp 163–170 9. Leimeister JM (2014) Collaboration Engineering. Springer Berlin Heidelberg, Berlin, Heidelberg 10. Lipnack J, Stamps J (1998) Virtuelle Teams: Projekte ohne Grenzen; Teambildung, virtuelle Orte, intelligentes Arbeiten, Vertrauen in Teams. Manager-Magazin-Edition. Ueberreuter, Wien, Frankfurt [Main] 11. Neges M, Wolf M, Abramovici M (2015) Secure Access Augmented Reality Solution for Mobile Maintenance Support Utilizing ConditionOriented Work Instructions. Procedia CIRP 38: 58–62. doi: 10.1016/j.procir.2015.08.036 12. Porcelli I, Rapaccini M, Espíndola DB et al. (2013) Technical and Organizational Issues about the Introduction of Augmented Reality in Maintenance and Technical Assistance Services. IFAC Proceedings Volumes 46(7): 257–262. doi: 10.3182/20130522-3-BR-4036.00024 13. Wang X, Kim MJ, Love PE et al. (2013) Augmented Reality in built environment: Classification and implications for future research. Automation in Construction 32: 1–13. doi: 10.1016/j.autcon.2012.11.021 14. Kaufmann H, Schmalstieg D (2003) Mathematics and geometry education with collaborative augmented reality. Computers & Graphics 27(3): 339– 345. doi: 10.1016/S0097-8493(03)00028-1 15. Shen Y, Ong SK, Nee A (2010) Augmented reality for collaborative product design and development. Design Studies 31(2): 118–145. doi: 10.1016/j.destud.2009.11.001 16. Ong SK, Shen Y (2009) A mixed reality environment for collaborative product design and development. CIRP Annals - Manufacturing Technology 58(1): 139–142. doi: 10.1016/j.cirp.2009.03.020 17. Abramovici M, Krebs A, Wolf M (2014) Approach to ubiquitous support for maintenance and repair jobs utilizing smart devices. In: Horváth I (ed) Tools and methods of competitive engineering: Digital proceedings of the Tenth International Symposium on Tools and Methods of Competitive Engineering - TMCE 2014, May 19 - 23, Budapest, Hungary. Faculty of Industrial Design Engineering Delft University of Technology, Delft, pp 83–93 18. Haberfellner R, Daenzer WF (eds) (2002) Systems Engineering: Methodik und Praxis, 11., durchges. Aufl. Verl. Industrielle Organisation, Zürich 19. Harrison G (2015) Next generation databases: NoSQL, NewSQL, and Big Data : what every professional needs to know about the future of databases in a world of NoSQL and Big Data. The expert's voice in Oracle. Apress (IOUG), New York 20. Locke EA, Latham GP, Smith KJ (1990) A theory of goal setting & task performance. Prentice Hall, Englewood Cliffs, N.J.