- Email: [email protected]
First steps to control a digitalized factory via Augmented Reality
14th IFAC IFACDesign Symposium on Analysis and Evaluation of Human Machine Systems 14th Symposium on 14th IFAC Symposium on Analysis Design and Evaluation of Human Machine Systems Tallinn, Estonia, Sept. 16-19, 2019 Available online at www.sciencedirect.com Analysis Design and Evaluation of Machine Systems 14th IFAC Symposium on Analysis Design and Evaluation of Human Human Machine Systems 14th IFAC Symposium on Tallinn, Estonia, Sept. 16-19, 2019 Tallinn, Estonia, Sept. 16-19, 2019 Analysis Design and Evaluation of Human Machine Systems Tallinn, Estonia, Sept. 16-19, 2019 Analysis Design Sept. and Evaluation of Human Machine Systems Tallinn, Estonia, 16-19, 2019 Tallinn, Estonia, Sept. 16-19, 2019
ScienceDirect
IFAC PapersOnLine 52-19 (2019) 1–6 First steps to control a digitalized factory via Augmented Reality First steps to control a digitalized factory via Augmented Reality First steps to control a digitalized factory via Augmented Reality First steps to control a digitalized factory via Augmented Reality First steps to control a digitalized factory via Augmented First steps to control Waldemar a digitalized factory Augmented Reality Reality Rempel, Benedictvia Bauer,
Waldemar Rempel, Benedict Bauer, Nicolaj C. Stache, Carsten Wittenberg Waldemar Rempel, Benedict Bauer, Waldemar Rempel, Benedict Bauer, Nicolaj C. Stache, Carsten Wittenberg Waldemar Rempel, Benedict Bauer, Nicolaj C. Stache, Carsten Wittenberg Nicolaj C. Stache, Carsten Wittenberg Waldemar Rempel, Benedict Bauer, Nicolaj C. Stache, Carsten Wittenberg Nicolaj C. Stache, Carsten Wittenberg *Faculty Mechanics & Electronics *Faculty Mechanics Electronics Heilbronn of& Science *Faculty Mechanics & Electronics *FacultyUniversity Mechanics &Applied Electronics Heilbronn University of Applied Science *Faculty Mechanics & Electronics Max-Planck-Str. 39, D-74081 Heilbronn, Germany Heilbronn University of Applied Science Heilbronn of&Applied Science *FacultyUniversity Mechanics Electronics Max-Planck-Str. 39, D-74081 Heilbronn, Germany Heilbronn University of Applied Science Email: {waldemar.rempel; benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Max-Planck-Str. 39, D-74081 Heilbronn, Germany Max-Planck-Str. 39, D-74081 Heilbronn, Germany Heilbronn University of Applied Science Email: {waldemar.rempel; benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Max-Planck-Str. 39, D-74081 Heilbronn, Germany Email: {waldemar.rempel; {waldemar.rempel; benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Email: benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Max-Planck-Str. 39, D-74081 Heilbronn, Germany Email: {waldemar.rempel; benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Email: {waldemar.rempel; benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Abstract: The technological capability of augmented reality (AR) is still increasing. Modern devices have Abstract: technological capability augmented (AR) Modern have available huge computing power and of several sensorsreality like cameras etc. increasing. The communication possibilities Abstract:aThe The technological capability of augmented reality (AR) is is still still increasing. Modern devices devices have Abstract: The technological capability of augmented reality (AR) is still increasing. Modern devices have available a huge computing power and several sensors like cameras etc. The communication possibilities Abstract: The technological capability of augmented reality (AR) is still increasing. Modern devices have cover all actual communication standards – also the industrial standards (except safety). The programming available a huge computing power and several sensors like cameras etc. The communication possibilities available aThe huge computing power and of several sensorsreality like cameras etc. increasing. The communication possibilities Abstract:actual technological capability (AR) is still Modern devices have cover standards ––augmented also the industrial standards safety). The available a hugecommunication computing power like cameras etc.(except The communication possibilities of ARall applications is simplified byand theseveral engines known from the gaming fields. This paper describes the cover all actual communication standards alsosensors the industrial standards (except safety). The programming programming cover all actual communication standards – also the industrial standards (except safety). The programming available a huge computing power and several sensors like cameras etc. The communication possibilities of AR applications is simplified by the engines known from the gaming fields. This paper describes the cover all actual communication standards – also the industrial standards (except safety). The programming first steps enabling the use of AR in the field of a production system. The production system is represented of AR ARall applications is simplified simplifiedstandards by the the engines engines known from standards the gaming gaming fields.safety). This paper paper describes the the of applications is by known from the fields. This describes cover actual communication – also industrial (except The programming first enabling of AR in the of aaathe production system. The production represented of applications is simplified by thefield engines known fromcomplexity. the gaming fields. Thissystem paper is describes the by AR a steps model factory the in ause laboratory format with sufficient The communication between first steps enabling the use of AR in the field of production system. The production system is represented first steps enabling the use of AR in the field of a production system. The production system is represented of ARmodel applications is simplified byformat the engines known fromcomplexity. the gamingThe fields. This paper describes the by factory in laboratory with sufficient communication the first steps the of AR in the field of aaaWLAN, production system. Thebus production represented ARaaadevice and the model factory is format realized via a typical field and OPCsystem UA. isbetween by modelenabling factory in aaause laboratory format with sufficient complexity. The communication between the by model factory in laboratory with sufficient complexity. The communication the first steps enabling the use of AR in the field of aaWLAN, production system. Thebus production system isbetween represented AR device and the model factory is realized via a typical field and OPC UA. by adevice modeland factory in a laboratory format with aWLAN, sufficient complexity. The communication between the AR the model factory is realized via a typical field bus and OPC UA. AR the The model factoryPublished is format realized via a All typical bus and OPC UA. factory by adevice modeland factory in aAuthors. laboratory with aWLAN, sufficient complexity. The communication between the Copyright 2019. by Elsevier Ltd. rightsfield reserved. Keywords: Supervisory Control, Reality, User-centered approach, AR device © and the model factoryAugmented is realized via WLAN, a typical field bus anddigitalized OPC UA. AR device and the model factory is realized via WLAN, a typical field bus and OPC UA. Keywords: Supervisory Control, Augmented Reality, User-centered approach, digitalized factory Keywords: Supervisory Supervisory Control, Control, Augmented Augmented Reality, Keywords: Reality, User-centered User-centered approach, approach, digitalized digitalized factory factory Keywords: Supervisory Control, Augmented Reality, User-centered approach, digitalized factory Keywords: Supervisory Control, Augmented Reality, User-centered approach, digitalized factory White et al (2007), Zhang and Sun (2005) investigate the 1. INTRODUCTION White et al (2007), Zhang and Sun (2005) investigate potential use of virtual hints in the form of text, diagramsthe or White et al (2007), Zhang and Sun investigate the 1. INTRODUCTION White et al (2007), Zhang and Sun (2005) (2005) investigate the 1. 1. INTRODUCTION INTRODUCTION potential use of virtual hints in the form of text, diagrams or White et al (2007), Zhang and Sun (2005) investigate the animations, as well as the spatial orientation and assignment potential use of virtual hints in the form of text, diagrams or 1. INTRODUCTION potential of virtual hints and in the form of text, diagramsthe or White et use al as (2007), Zhang Sun (2005) investigate 1.1 Mobile devices in industrial environment animations, well as the spatial orientation and assignment 1. INTRODUCTION potential use of virtual hints in the form of text, diagrams or of virtual information to real objects. Henderson, Feiner animations, as well as the spatial orientation and assignment animations, as well as the spatial orientation and assignment potential use of virtual hints in the form of text, diagrams or 1.1 Mobile devices in industrial environment 1.1 Mobile devices in industrial environment of virtual information to real objects. Henderson, Feiner animations, as well as the spatial orientation and assignment (2011) are showing the overlay of three-dimensional tools as of virtual information to real objects. Henderson, Feiner 1.1 Mobile devices in industrial environment of virtual information to real objects. Henderson, Feiner animations, as well as the spatial orientation and assignment 1.1 Mobile devices in industrial environment The idea of the ubiquitous computing (Weiser, 1991) is (2011) are showing the overlay of three-dimensional tools as of virtual information to real objects. Henderson, Feiner additional support for some production steps. The problem (2011) are showing the overlay of three-dimensional tools as 1.1 Mobile devices in industrialcomputing environment (2011) are showing the overlay ofobjects. three-dimensional tools as of virtual information to real Henderson, Feiner The idea of the ubiquitous (Weiser, 1991) is continued by the cyber-physical systems and also by the usage additional support for some production steps. The problem The idea of the ubiquitous computing (Weiser, 1991) is (2011) are showing the overlay of three-dimensional tools as here is the occlusions of the real world and the visualization of additional support for some production steps. The problem The idea of the ubiquitous computing (Weiser, 1991) is additional support for some production steps. The problem are occlusions showing the overlay of three-dimensional tools of as continued by systems also by the usage The idea devices. of the thecyber-physical ubiquitous computing 1991) is (2011) of mobile The meaningful usage and of(Weiser, mobile devices for here is is the the ofsome the real real world andsteps. the visualization visualization continued by the cyber-physical systems and also by the usage additional support for production The problem correct spatial depth. here occlusions of the world and the of continued by the cyber-physical systems and also by the usage The idea devices. of the The ubiquitous computing (Weiser, 1991) for is additional here is the occlusions the real world andsteps. the visualization of support forofsome production The problem of mobile mobile meaningful usage of mobile devices continued bymaintenance the cyber-physical systems and also bydevices the usage service and technicians is still in the research correct spatial depth. of devices. The meaningful usage of mobile for is the occlusions of the real world and the visualization of correct spatial depth. of mobile devices. The meaningful usage and of mobile for here continued bymaintenance the cyber-physical systems also bydevices the usage correct spatial depth. of the real world and the visualization of is the occlusions service and technicians is of still in the the research of mobile devices. The (e.g. meaningful usage mobile devices for here focus for decades Wittenberg, 2003). Technical service and maintenance technicians is still in research correct spatial depth. service and maintenance technicians is still in the research of mobile devices. The (e.g. meaningful usage of 2003). mobile devices for 1.3 Previous spatialwork depth. focus for decades Wittenberg, Technical service andlike maintenance technicians is still incapability theTechnical research limitations network(e.g. connections, graphical and correct focus decades Wittenberg, 2003). focus for for decades (e.g. Wittenberg, 2003). 1.3 Previous work service andlike maintenance technicians is still incapability theTechnical research 1.3 Previous work limitations network connections, graphical and focus for like decades (e.g. Wittenberg, 2003). Technical Previous work battery runtime slow downed the development ofcapability applications. limitations network connections, graphical and limitations network connections, graphical capability and 1.3 focus for like decades (e.g. Wittenberg, 2003). Technical ThePrevious focus ofwork past research on AR was on service and battery runtime slow downed the development applications. limitations like network connections, graphicalof capability and 1.3 battery runtime slow downed the development of applications. 1.3 Previous work battery runtime slow downed the development of applications. The focus of past research AR was 2015). on service and limitations like network connections, graphical capability and maintenance technicians (Buyer,on A service Since the development of tablet and of smartphones in The focus of on AR and battery runtime slow downed thecomputer development applications. The focus of past past research research onWittenberg, AR was was on on service service and maintenance technicians (Buyer, Wittenberg, 2015). A service battery runtime slow downed the development of applications. Since the development of tablet computer and smartphones in The focus of past research on AR was on service and technician in industrial maintenance area is a self-sufficient the last ten years the operating efficiency of mobile devices maintenance technicians (Buyer, Wittenberg, 2015). A service Since the development of tablet computer and smartphones in maintenance technicians (Buyer,onWittenberg, 2015). A service Since the development of tablet computer and smartphones in The focus of past research AR was on service and technician in industrial maintenance area is a self-sufficient the last ten years the operating efficiency of mobile devices maintenance technicians (Buyer, Wittenberg, 2015). A service Since the development of tablet computer and smartphones in and responsible team member of a workgroup. The basic increased tremendously. Simultaneous theof of technician in industrial maintenance area is aa self-sufficient the years efficiency mobile devices technician in industrial maintenance area is self-sufficient the last lasttheten ten years the the operating operating efficiency ofdevelopment mobile devices maintenance technicians (Buyer, Wittenberg, 2015). A service Since development of tablet computer and smartphones in and responsible responsible team member ofidentification, a workgroup. workgroup. The basic increased Simultaneous the of technician in system industrial maintenance is a self-sufficient the last tentremendously. years operating efficiency ofdevelopment mobile activities are checks, faultof fault analysis applications and the us-cases for these mobile devicesdevices in an team member The basic increased Simultaneous the of and responsible team member of aa area workgroup. The basic increased tremendously. Simultaneous theofdevelopment development of and technician in system industrial maintenance is a self-sufficient the last tentremendously. years the operating efficiency mobile activities are checks, fault identification, fault analysis applications and was us-cases for these these mobile devicesdevices in an an responsible team member ofidentification, a area workgroup. The basic increased tremendously. Simultaneous the development of and problem solving. This generally requires the use of industrial context intensified. activities are system checks, fault fault analysis applications and us-cases for mobile devices in activities are system checks, fault identification, fault analysis applications and us-cases for these mobile devices in an team member of a workgroup. Theuse basic increased tremendously. Simultaneous the development of and and responsible problem solving. Thisdocumentation, generally requires the of industrial context context was intensified. activities are system checks, fault identification, fault analysis applications and was us-cases for these mobile devices in an maintenanceand system process data and and problem solving. This generally requires the use industrial intensified. and problem solving. This fault generally requiresfault the analysis use of of industrial context intensified. are system checks, identification, applications and was us-cases for these mobile devices in an activities maintenancesystem process and problem and solving. Thisdocumentation, generally requires thedata useand of industrial context was intensified. communication with remote specialists. maintenanceand system documentation, process data and maintenanceand system documentation, process data and 1.2 Augmented Reality (AR) and problem solving. This generally requires the use of industrial context was intensified. communication with remote specialists. maintenanceand system documentation, process data and communication with remote specialists. 1.2 Augmented Augmented Reality Reality (AR) (AR) communication with remote specialists. maintenanceand system documentation, process data and 1.2 The first step inwith the user-centred development process is the 1.2 Augmented Reality (AR) communication remote specialists. 1.2 Augmented Reality (AR) with remote specialists. Augmented Reality (AR) is a visual extension of the reality. It communication The first step in the user-centred development the user requirement analysis. 25 persons (9 female,process 16 mal)is The first step in the user-centred process is the 1.2 Augmented Reality (AR) The first step in the user-centred development development process iswith the Augmented Reality (AR) is aa visual extension of the reality. It combines digital information such as videos, pictures, texts user requirement analysis. 25 persons (9 female, 16 mal) with Augmented Reality (AR) is visual extension of the reality. It The first step in the user-centred development process is the auser service and maintenance background took part in this requirement analysis. 25 persons (9 female, 16 mal) with Augmented Reality (AR) is a visual extension of the reality. It user requirement analysis. 25 persons (9 female, 16 mal) with The first step in the user-centred development process is the combines digital information such as videos, pictures, texts Augmented Reality (AR)in is reality, a visual extension ofpictures, thepicture reality. It user etc. with data that exists for example the of a service and maintenance background took part in this combines digital information such as videos, texts requirement analysis. 25 persons (9 female, 16 mal) with analysis (Buyer, Wittenberg, 2015, Wittenberg, 2016). service and maintenance background took part in this combines digital information such extension as videos,ofpictures, texts Augmented Reality (AR)in is reality, a visual thepicture reality. It aauser service and maintenance background took part in this requirement analysis. 25 persons (9 female, 16 mal) with etc. with data that exists for example the of combines digital information such as videos, pictures, texts aetc. digital supplementary data can help user of to aanalysis analysis (Buyer, Wittenberg, 2015, 2015, Wittenberg, 2016). etc. with data that exists in the picture service(Buyer, and maintenance background took2016). part in this Wittenberg, Wittenberg, with camera. data thatThe exists in reality, reality, for example thethe picture of combines digital information such for as example videos, pictures, texts analysis (Buyer, Wittenberg, 2015, Wittenberg, 2016). aAsservice and background took part in as thisa aaaccomplish digital supplementary can help the user to a result, the maintenance fast information retrieval is mentioned etc. with camera. data thatThe exists inThe reality, fordata example the picture of sever-al tasks. virtual extension occurs in real digital camera. The supplementary data can help the user to analysis (Buyer, Wittenberg, 2015, Wittenberg, 2016). aaccomplish digital supplementary can help the user to analysis etc. with camera. data thatThe exists inThe reality, fordata example the picture of (Buyer, Wittenberg, 2015, Wittenberg, 2016). As a result, the fast information retrieval is mentioned as a sever-al tasks. virtual extension occurs in real main aspect. This is confirmed by responses of the subjects atime digital camera. The supplementary data can help the user to As a result, the fast information retrieval is mentioned as so that relevant digital data can be retrieved by the user. accomplish sever-al tasks. The virtual virtualdata extension occurs in real real a result, the fast information retrieval is mentioned asonaa accomplish sever-al The extension occurs in aaccomplish digital camera. Thetasks. supplementary can help to As main aspect. This isprincipal confirmed by responses responses ofmentioned the subjectsasfor ona time so that that relevant digital data can be retrieved bythe theuser user. As a aspect. result, of the fast information retrieval isof sever-al tasks. The virtual extension occurs inuser. real the question the proceed in obtaining a contract main This is confirmed by the subjects on Especially big companies such as Google and Apple are time so relevant digital data can be retrieved by the maina aspect. Thisfast is confirmed by responses the subjectsasona time so that sever-al relevanttasks. digitalThe data can be retrievedoccurs by theinuser. result, the information retrieval isofmentioned accomplish virtual extension real the question in obtaining aa contract for Especially big companies such as be Google and by Apple are As main aspect. of This isprincipal confirmed by responses of the subjects on time so that relevant digital dataeveryday can retrieved the user. maintenance ofthe a system withproceed following points: „Use technical the question of the principal proceed in obtaining for working on big integrating AR into life and developing Especially companies such as Google Apple are the question of the principal proceed in obtaining a contract contract for Especially big companies such as Google and Apple are main aspect. This is confirmed by responses of the subjects on time so that relevant digital data can be retrieved by the user. maintenance of a system with following points: „Use technical working on integrating ARAR-Systems into everyday life andetdeveloping developing question“,of the principal proceed in obtaining acomparison“ contract for Especially big companies sucheveryday as Google Apple are the documents „carry out a target-performance maintenance of a system with following points: „Use technical new and more improved (Dörner al., 2013) working on integrating AR into life maintenance ofthe a system withproceed following points: „Use technical working on big integrating AR into everyday life and developing the question of principal in obtaining a contract for Especially companies such as Google Apple are documents “, out a target-performance comparison“ new and more improved (Dörner al., 2013) maintenance of„carry a system with following points: „Use technical working on integrating ARAR-Systems into everyday life andet developing such as „ Acquaintance with the system “.A problem often turn “, out a comparison“ (Bläss, Wittenberg, 2017). new more improved (Dörner al., documents “,of„carry „carry out a target-performance target-performance comparison“ new and and more improved AR-Systems (Dörner etdeveloping al., 2013) 2013) documents maintenance a system with following points: „Use technical working on integrating ARAR-Systems into everyday life andet such as „ Acquaintance with the system “.A problem often turn (Bläss, Wittenberg, 2017). documents “, „carry out a target-performance comparison“ new and more improved AR-Systems (Dörner et al., 2013) out is a decentralized storage of required documents (see fig. such as „ Acquaintance with the system “.A problem often (Bläss, Wittenberg, 2017). such as „ Acquaintance with the system “.A problem often turn turn (Bläss, Wittenberg, 2017). AR-Systems “, „carry out a target-performance comparison“ new and more (Dörner et al., 2013) documents The most knownimproved example out is a decentralized storage of required documents (see fig. such as „ Acquaintance with the system “.A problem often turn (Bläss, Wittenberg, 2017). for such a product is a data glass. 1). This leads to time-intensive searching and obtaining the out is a decentralized storage of required documents (see fig. out isasa „decentralized storage required (see turn fig. suchThis Acquaintance with theofsystem “.Adocuments problem often (Bläss, Wittenberg, 2017). The most known example for such aa product is aa data glass. These smart glasses are wearable AR-Systems that combine 1). leads to time-intensive searching and obtaining the The most known example for such product is data glass. out is a decentralized storage of required documents (see fig. leads searching and the The most known example for such a product is a data glass. 1). This leads to to time-intensive time-intensive searchingdocuments and obtaining obtaining the out This is a decentralized storage of required (see fig. These smart glasses are wearable combine The most known example for suchAR-Systems a product is that a data glass. 1). virtual reality with physical reality. These smart glasses are wearable AR-Systems that combine 1). This leads to time-intensive searching and obtaining the Thesemost smart glasses are wearable AR-Systems that combine The known example for such a product is a data glass. virtual reality with physical reality. These smart glasses are wearable AR-Systems that combine 1). This leads to time-intensive searching and obtaining the virtual reality with physical reality. virtual reality with physical reality. These smart glasses are wearable AR-Systems that combine virtual reality with physical reality. Copyright © 2019 IFAC virtual reality with 2405-8963 Copyright ©physical 2019. Thereality. Authors. Published by Elsevier Ltd. All1 rights reserved. Copyright 2019 IFAC 1 Copyright © 2019 IFAC 1 Peer review© of International Federation of Automatic Copyright ©under 2019 responsibility IFAC 1 Control. 10.1016/j.ifacol.2019.12.072 Copyright © 2019 IFAC 1 Copyright © 2019 IFAC 1
2019 IFAC HMS 2 Tallinn, Estonia, Sept. 16-19, 2019
Waldemar Rempel et al. / IFAC PapersOnLine 52-19 (2019) 1–6
documents before it can be effectively started on the appropriate system.
graphical representation of measured values. As played here from a bird's perspective, usually many objects are present on the playing surface and can be applied in accordance with each other An active object (multimeter) is applied to the passive object "S7-module". Other functions such as the graphical representation of a measured value can be triggered by selecting appropriate menu items of the active object For further information the user can to drag & drop the virtual multimeter on the real PLC module (fig. 3).
Fig. 1. Storage of maintenance documents (Buyer, Wittenberg, 2015) As a platform for the implementation of a first prototype a tablet computer was chosen. Via WLAN this tablet was connected to a Siemens PLC for data exchange.
Fig. 3. Selecting module-item to diagnose the Siemens S7IO-Modul to open module-items for measuring (Buyer, Wittenberg, 2015) The disadvantage of tablet computer is the necessity to hold the tablet computer in the hand. But a service and maintenance technician need the hands for the service tasks like working with tools. The logical next step lead to a data glass (Fig. 4).
Data glass PLC Fig. 2. Dragging the virtual multimeter as an active object (Buyer, Wittenberg, 2015) The developed approach for an intuitive information management system in augmented reality applications is shown in Fig. 2. The real-time camera image was overlaid with additional information from the automation system. The user's attention in the augmented world is controlled with the help of an avatar. The reaction change of the avatar presents the user with an invalid system state and draws attention to the appropriate object. Detailed information can be found on an object associated visualization surface. For Example, a digital multimeter symbolizes a bad value. Thus, eliminates decoding errors manually or with help of external tools and provides relevant information at first glance.
Touchpad Fig. 4: Data glass and PLC (Wittenberg, 2016)
The functionality is based on the object-object scheme known from simulation- and strategy games. Selecting inherent menu items of the active object can trigger other functions such as 2
2019 IFAC HMS Tallinn, Estonia, Sept. 16-19, 2019
Waldemar Rempel et al. / IFAC PapersOnLine 52-19 (2019) 1–6
3
2. LABORATORY EQUIPMENT 2.1 Smart factory in the Laboratory The Otto-Rettenmaier Research Laboratory for Digitalization contains a model factory for education and research activities. Fig. 5 shows the structure of this factory. Transports & logistics
Filling & Packing
Conveyor Station
Packaging Station Filling Station
Unpacking & Recycling
Process automation
Mixing Station
Unpacking Station
Fig. 7: Distribution Station Production automation
Recycling Station
Distribution Station
Buffer Station
•
Handling Station
Fig. 5: Structure of the model factory (Rempel et al., 2019)
Buffer Station (Figure 8): The function of the buffer station is to provide a storage for the caps. The caps wait on a conveyor belt until these caps are needed for the following productions steps. If bottles are for sealing the caps are transported via the conveyor belt to an area for pickup.
Fig. 8: Buffer Station Fig. 6: Model factory This model factory consists of the following subsystems (Wittenberg et al., 2019): •
Distribution Station (Figure 7): This subsystem is arranged before the buffer station. The function of the buffer station is to provide the cabs for bottles which are distributed in this system. The distribution station contains of a magazine for the cabs and a swivel arm for moving the cabs to the next subsystem.
3
•
Handling Station: A handling device grabs the cap from the pickup area and places this cap on a bottle in the filling station.
•
Filling Station (Figure 9): This subsystem contains a rotary table. The empty bottles are moved with this rotary table, are filled with liquids or solids, and sealed with the caps. At the end, the filled bottles are transported to further conveying belt.
2019 IFAC HMS 4 Tallinn, Estonia, Sept. 16-19, 2019
Waldemar Rempel et al. / IFAC PapersOnLine 52-19 (2019) 1–6
•
Recycling Station (Figure 12): In this subsystem each bottle is brought to its initial state. The caps are opened and moved to the distribution station. The content of the bottles is emptied by a vacuum and the empty bottles move via the last conveyor belt to the filling station.
Fig. 9: Filling Station •
Packing Station (Figure 10): The filled and sealed bottles are packed into a six-pack container.
Fig. 12: Recycling Station All bottles are equipped with RFID tags. The automation system fulfils the Industry 4.0/IIoT standards. 2.2 AR Device Actual, one of the most popular AR devices is the Microsoft HoloLens. The HoloLens contains the following subsystems (Figure 13): •
inertial measurement unit, IMU
Fig. 10: Packing Station
•
4 environmental understanding cameras (1)
•
•
Depth camera (2) with IR emitter (5)
•
HD video camera (3)
•
4 microphones
•
ambient light sensor (4)
•
Conveyor Station: This subsystem has the function to transport the six-packs with the bottles to the following subsystem – the unpacking station. Unpacking Station: At a time three bottles from the six-pack container are moved via another conveyor belt to the recycling station.
Fig. 11: Unpacking Station Fig. 13: Microsoft HoloLens (Rempel et al., 2019) 4
2019 IFAC HMS Tallinn, Estonia, Sept. 16-19, 2019
Waldemar Rempel et al. / IFAC PapersOnLine 52-19 (2019) 1–6
The HoloLens offers various interaction possibilities. Examples are the gazing – looking in a direction over a specific time period – or the interaction via gestures. Typical gestures are the air tap (with thumb and forefinger) as a symbol for a mouse click or the Bloom (spreading all fingers) as a symbol for home.
5
Based on this model the main menu and first dialogue windows are developed. Dialable areas (via gazing plus gesture) got a cover (red volume body). Normally, these volume bodies are transparent. These areas are recognizable through the change of the appearance of the cursor – similar to well-known web applications.
2.3 Communication between HoloLens and model factory The HoloLens is via WLAN connected to an access point in the model factory. This access point is included in the ProfiNET network of this model factory. The access point is used as a bridge between the HoloLens and the PLCs of the model factory. The communication standard is OPC UA (Fig. 14). HoloLens
WLAN
Access Point
OPC UA
ProfiNET
SPS
Modell Factory
Fig. 16: 3D model with overlays
Fig. 14: Communication structure (Rempel et al., 2019)
The PLCs in the lower part of the model factory are also shaped by a (green) volume body. These areas provide the possibilities for the display of factory information like measured values – sored in the PLCs. Figure 16 shows the model with the overlays.
3. HUMAN-MACHINE INTERFACE AND 3D MODEL OF THE MODEL FACTORY The application for the HoloLens was developed with the Unity programming interface. The first step is design of the 3D model of the factory (Figure 15). This is necessary to place the AR dialogues at the right position in the field of view. The 3D model itself is not displayed with the HoloLens.
4 OUTLOOK The described research project represents the first step in the development of an AR application with a focus on production systems. The logical next step is the user evaluation. A first usability test was performed as a student project. The analysis of this evaluation is not completed yet and will be included in the final paper. Based on this evaluation a suitable interface will be developed. REFERENCES Bläss, M, C. Wittenberg (2017): “An online user analysis regarding the usage of mobile Augmented and Virtual Reality devices in the field of robotics”. In: S. Lackey, J. Chen (Ed.): Virtual, Augmented and Mixed Reality, Cham: Springer, pp. 383-393. Buyer, S., Wittenberg, C. (2015): AR and maintenance visualisation of process data and engineering information. In: Constantine Stephanidis (Ed.): HCI International 2015 – Posters’ Extended Abstracts, Springer Cham, 2015, pp 159-162. Dörner, R., W. Broll, P. Grimm, B. Jung (2013): „Virtual und Augmented Reality (VR/AR)“. Berlin Heidelberg: Springer Vieweg. Henderson, S.,Feiner, S.( 2011). Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair. IEEE Transactions on Visualization and Computer Graphics, pp 1355 –1368. Rempel, W., B. Bauer, C. Wittenberg (2019): Der Einsatz von Augmented Reality in der Industrie 4.0 am Beispiel einer
Fig. 15: Modelling the 3D model of the factory 5
2019 IFAC HMS 6 Tallinn, Estonia, Sept. 16-19, 2019
Waldemar Rempel et al. / IFAC PapersOnLine 52-19 (2019) 1–6
Modellfabrik. In: B.Bauer, C. Wittenberg (Eds.): Tagungsband AALE 2019, VDE-Verlag, Berlin, pp 395401. Weiser, M (1991): “The computer of the 21st century,” Scientific American, vol. 265, no. 3, pp. 66–75. White, S., Lister, L. and Feiner, S. (2007). Visual Hints for Tangible Gestures in Augmented Reality. Proceedings of the 6th International Symposium on Mixed and Augmented Reality ISMAR, Nara, Japan, pp 47-50. Wittenberg, C. (2003): A Requirement Analysis for the use of Mobile Devices in Service and Maintenance. In: Proceedings 2003 IEEE International Conference on Systems, Man & Cybernetics, Vol. 4, pp 4033-4039, IEEE Conference Publications, New York. Wittenberg, C. (2016): Human-CPS Interaction – requirements and mobile human-machine interaction methods for the Industry 4.0. Proceedings of the 13th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems, IFACPapersOnLine, Volume 49, Issue 19, 2016, pp. 420-425. Wittenberg, Bauer, Stache (2019): A Smart Factory in a laboratory size for developing and testing innovative human-machine interaction concepts. Proceedings of the 10th International Conference on Applied Human Factors and Ergonomics, Springer Cham, in Press. Zhang, F. and Sun, H. (2005). Dynamic labeling management in virtual and augmented environments. Proceedings of the Ninth International Conference on Computer Aided Design and Computer Graphics, Hong Kong, China
6
ScienceDirect
IFAC PapersOnLine 52-19 (2019) 1–6 First steps to control a digitalized factory via Augmented Reality First steps to control a digitalized factory via Augmented Reality First steps to control a digitalized factory via Augmented Reality First steps to control a digitalized factory via Augmented Reality First steps to control a digitalized factory via Augmented First steps to control Waldemar a digitalized factory Augmented Reality Reality Rempel, Benedictvia Bauer,
Waldemar Rempel, Benedict Bauer, Nicolaj C. Stache, Carsten Wittenberg Waldemar Rempel, Benedict Bauer, Waldemar Rempel, Benedict Bauer, Nicolaj C. Stache, Carsten Wittenberg Waldemar Rempel, Benedict Bauer, Nicolaj C. Stache, Carsten Wittenberg Nicolaj C. Stache, Carsten Wittenberg Waldemar Rempel, Benedict Bauer, Nicolaj C. Stache, Carsten Wittenberg Nicolaj C. Stache, Carsten Wittenberg *Faculty Mechanics & Electronics *Faculty Mechanics Electronics Heilbronn of& Science *Faculty Mechanics & Electronics *FacultyUniversity Mechanics &Applied Electronics Heilbronn University of Applied Science *Faculty Mechanics & Electronics Max-Planck-Str. 39, D-74081 Heilbronn, Germany Heilbronn University of Applied Science Heilbronn of&Applied Science *FacultyUniversity Mechanics Electronics Max-Planck-Str. 39, D-74081 Heilbronn, Germany Heilbronn University of Applied Science Email: {waldemar.rempel; benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Max-Planck-Str. 39, D-74081 Heilbronn, Germany Max-Planck-Str. 39, D-74081 Heilbronn, Germany Heilbronn University of Applied Science Email: {waldemar.rempel; benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Max-Planck-Str. 39, D-74081 Heilbronn, Germany Email: {waldemar.rempel; {waldemar.rempel; benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Email: benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Max-Planck-Str. 39, D-74081 Heilbronn, Germany Email: {waldemar.rempel; benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Email: {waldemar.rempel; benedict.bauer; nicolaj.stache; carsten.wittenberg}@hs-heilbronn.de Abstract: The technological capability of augmented reality (AR) is still increasing. Modern devices have Abstract: technological capability augmented (AR) Modern have available huge computing power and of several sensorsreality like cameras etc. increasing. The communication possibilities Abstract:aThe The technological capability of augmented reality (AR) is is still still increasing. Modern devices devices have Abstract: The technological capability of augmented reality (AR) is still increasing. Modern devices have available a huge computing power and several sensors like cameras etc. The communication possibilities Abstract: The technological capability of augmented reality (AR) is still increasing. Modern devices have cover all actual communication standards – also the industrial standards (except safety). The programming available a huge computing power and several sensors like cameras etc. The communication possibilities available aThe huge computing power and of several sensorsreality like cameras etc. increasing. The communication possibilities Abstract:actual technological capability (AR) is still Modern devices have cover standards ––augmented also the industrial standards safety). The available a hugecommunication computing power like cameras etc.(except The communication possibilities of ARall applications is simplified byand theseveral engines known from the gaming fields. This paper describes the cover all actual communication standards alsosensors the industrial standards (except safety). The programming programming cover all actual communication standards – also the industrial standards (except safety). The programming available a huge computing power and several sensors like cameras etc. The communication possibilities of AR applications is simplified by the engines known from the gaming fields. This paper describes the cover all actual communication standards – also the industrial standards (except safety). The programming first steps enabling the use of AR in the field of a production system. The production system is represented of AR ARall applications is simplified simplifiedstandards by the the engines engines known from standards the gaming gaming fields.safety). This paper paper describes the the of applications is by known from the fields. This describes cover actual communication – also industrial (except The programming first enabling of AR in the of aaathe production system. The production represented of applications is simplified by thefield engines known fromcomplexity. the gaming fields. Thissystem paper is describes the by AR a steps model factory the in ause laboratory format with sufficient The communication between first steps enabling the use of AR in the field of production system. The production system is represented first steps enabling the use of AR in the field of a production system. The production system is represented of ARmodel applications is simplified byformat the engines known fromcomplexity. the gamingThe fields. This paper describes the by factory in laboratory with sufficient communication the first steps the of AR in the field of aaaWLAN, production system. Thebus production represented ARaaadevice and the model factory is format realized via a typical field and OPCsystem UA. isbetween by modelenabling factory in aaause laboratory format with sufficient complexity. The communication between the by model factory in laboratory with sufficient complexity. The communication the first steps enabling the use of AR in the field of aaWLAN, production system. Thebus production system isbetween represented AR device and the model factory is realized via a typical field and OPC UA. by adevice modeland factory in a laboratory format with aWLAN, sufficient complexity. The communication between the AR the model factory is realized via a typical field bus and OPC UA. AR the The model factoryPublished is format realized via a All typical bus and OPC UA. factory by adevice modeland factory in aAuthors. laboratory with aWLAN, sufficient complexity. The communication between the Copyright 2019. by Elsevier Ltd. rightsfield reserved. Keywords: Supervisory Control, Reality, User-centered approach, AR device © and the model factoryAugmented is realized via WLAN, a typical field bus anddigitalized OPC UA. AR device and the model factory is realized via WLAN, a typical field bus and OPC UA. Keywords: Supervisory Control, Augmented Reality, User-centered approach, digitalized factory Keywords: Supervisory Supervisory Control, Control, Augmented Augmented Reality, Keywords: Reality, User-centered User-centered approach, approach, digitalized digitalized factory factory Keywords: Supervisory Control, Augmented Reality, User-centered approach, digitalized factory Keywords: Supervisory Control, Augmented Reality, User-centered approach, digitalized factory White et al (2007), Zhang and Sun (2005) investigate the 1. INTRODUCTION White et al (2007), Zhang and Sun (2005) investigate potential use of virtual hints in the form of text, diagramsthe or White et al (2007), Zhang and Sun investigate the 1. INTRODUCTION White et al (2007), Zhang and Sun (2005) (2005) investigate the 1. 1. INTRODUCTION INTRODUCTION potential use of virtual hints in the form of text, diagrams or White et al (2007), Zhang and Sun (2005) investigate the animations, as well as the spatial orientation and assignment potential use of virtual hints in the form of text, diagrams or 1. INTRODUCTION potential of virtual hints and in the form of text, diagramsthe or White et use al as (2007), Zhang Sun (2005) investigate 1.1 Mobile devices in industrial environment animations, well as the spatial orientation and assignment 1. INTRODUCTION potential use of virtual hints in the form of text, diagrams or of virtual information to real objects. Henderson, Feiner animations, as well as the spatial orientation and assignment animations, as well as the spatial orientation and assignment potential use of virtual hints in the form of text, diagrams or 1.1 Mobile devices in industrial environment 1.1 Mobile devices in industrial environment of virtual information to real objects. Henderson, Feiner animations, as well as the spatial orientation and assignment (2011) are showing the overlay of three-dimensional tools as of virtual information to real objects. Henderson, Feiner 1.1 Mobile devices in industrial environment of virtual information to real objects. Henderson, Feiner animations, as well as the spatial orientation and assignment 1.1 Mobile devices in industrial environment The idea of the ubiquitous computing (Weiser, 1991) is (2011) are showing the overlay of three-dimensional tools as of virtual information to real objects. Henderson, Feiner additional support for some production steps. The problem (2011) are showing the overlay of three-dimensional tools as 1.1 Mobile devices in industrialcomputing environment (2011) are showing the overlay ofobjects. three-dimensional tools as of virtual information to real Henderson, Feiner The idea of the ubiquitous (Weiser, 1991) is continued by the cyber-physical systems and also by the usage additional support for some production steps. The problem The idea of the ubiquitous computing (Weiser, 1991) is (2011) are showing the overlay of three-dimensional tools as here is the occlusions of the real world and the visualization of additional support for some production steps. The problem The idea of the ubiquitous computing (Weiser, 1991) is additional support for some production steps. The problem are occlusions showing the overlay of three-dimensional tools of as continued by systems also by the usage The idea devices. of the thecyber-physical ubiquitous computing 1991) is (2011) of mobile The meaningful usage and of(Weiser, mobile devices for here is is the the ofsome the real real world andsteps. the visualization visualization continued by the cyber-physical systems and also by the usage additional support for production The problem correct spatial depth. here occlusions of the world and the of continued by the cyber-physical systems and also by the usage The idea devices. of the The ubiquitous computing (Weiser, 1991) for is additional here is the occlusions the real world andsteps. the visualization of support forofsome production The problem of mobile mobile meaningful usage of mobile devices continued bymaintenance the cyber-physical systems and also bydevices the usage service and technicians is still in the research correct spatial depth. of devices. The meaningful usage of mobile for is the occlusions of the real world and the visualization of correct spatial depth. of mobile devices. The meaningful usage and of mobile for here continued bymaintenance the cyber-physical systems also bydevices the usage correct spatial depth. of the real world and the visualization of is the occlusions service and technicians is of still in the the research of mobile devices. The (e.g. meaningful usage mobile devices for here focus for decades Wittenberg, 2003). Technical service and maintenance technicians is still in research correct spatial depth. service and maintenance technicians is still in the research of mobile devices. The (e.g. meaningful usage of 2003). mobile devices for 1.3 Previous spatialwork depth. focus for decades Wittenberg, Technical service andlike maintenance technicians is still incapability theTechnical research limitations network(e.g. connections, graphical and correct focus decades Wittenberg, 2003). focus for for decades (e.g. Wittenberg, 2003). 1.3 Previous work service andlike maintenance technicians is still incapability theTechnical research 1.3 Previous work limitations network connections, graphical and focus for like decades (e.g. Wittenberg, 2003). Technical Previous work battery runtime slow downed the development ofcapability applications. limitations network connections, graphical and limitations network connections, graphical capability and 1.3 focus for like decades (e.g. Wittenberg, 2003). Technical ThePrevious focus ofwork past research on AR was on service and battery runtime slow downed the development applications. limitations like network connections, graphicalof capability and 1.3 battery runtime slow downed the development of applications. 1.3 Previous work battery runtime slow downed the development of applications. The focus of past research AR was 2015). on service and limitations like network connections, graphical capability and maintenance technicians (Buyer,on A service Since the development of tablet and of smartphones in The focus of on AR and battery runtime slow downed thecomputer development applications. The focus of past past research research onWittenberg, AR was was on on service service and maintenance technicians (Buyer, Wittenberg, 2015). A service battery runtime slow downed the development of applications. Since the development of tablet computer and smartphones in The focus of past research on AR was on service and technician in industrial maintenance area is a self-sufficient the last ten years the operating efficiency of mobile devices maintenance technicians (Buyer, Wittenberg, 2015). A service Since the development of tablet computer and smartphones in maintenance technicians (Buyer,onWittenberg, 2015). A service Since the development of tablet computer and smartphones in The focus of past research AR was on service and technician in industrial maintenance area is a self-sufficient the last ten years the operating efficiency of mobile devices maintenance technicians (Buyer, Wittenberg, 2015). A service Since the development of tablet computer and smartphones in and responsible team member of a workgroup. The basic increased tremendously. Simultaneous theof of technician in industrial maintenance area is aa self-sufficient the years efficiency mobile devices technician in industrial maintenance area is self-sufficient the last lasttheten ten years the the operating operating efficiency ofdevelopment mobile devices maintenance technicians (Buyer, Wittenberg, 2015). A service Since development of tablet computer and smartphones in and responsible responsible team member ofidentification, a workgroup. workgroup. The basic increased Simultaneous the of technician in system industrial maintenance is a self-sufficient the last tentremendously. years operating efficiency ofdevelopment mobile activities are checks, faultof fault analysis applications and the us-cases for these mobile devicesdevices in an team member The basic increased Simultaneous the of and responsible team member of aa area workgroup. The basic increased tremendously. Simultaneous theofdevelopment development of and technician in system industrial maintenance is a self-sufficient the last tentremendously. years the operating efficiency mobile activities are checks, fault identification, fault analysis applications and was us-cases for these these mobile devicesdevices in an an responsible team member ofidentification, a area workgroup. The basic increased tremendously. Simultaneous the development of and problem solving. This generally requires the use of industrial context intensified. activities are system checks, fault fault analysis applications and us-cases for mobile devices in activities are system checks, fault identification, fault analysis applications and us-cases for these mobile devices in an team member of a workgroup. Theuse basic increased tremendously. Simultaneous the development of and and responsible problem solving. Thisdocumentation, generally requires the of industrial context context was intensified. activities are system checks, fault identification, fault analysis applications and was us-cases for these mobile devices in an maintenanceand system process data and and problem solving. This generally requires the use industrial intensified. and problem solving. This fault generally requiresfault the analysis use of of industrial context intensified. are system checks, identification, applications and was us-cases for these mobile devices in an activities maintenancesystem process and problem and solving. Thisdocumentation, generally requires thedata useand of industrial context was intensified. communication with remote specialists. maintenanceand system documentation, process data and maintenanceand system documentation, process data and 1.2 Augmented Reality (AR) and problem solving. This generally requires the use of industrial context was intensified. communication with remote specialists. maintenanceand system documentation, process data and communication with remote specialists. 1.2 Augmented Augmented Reality Reality (AR) (AR) communication with remote specialists. maintenanceand system documentation, process data and 1.2 The first step inwith the user-centred development process is the 1.2 Augmented Reality (AR) communication remote specialists. 1.2 Augmented Reality (AR) with remote specialists. Augmented Reality (AR) is a visual extension of the reality. It communication The first step in the user-centred development the user requirement analysis. 25 persons (9 female,process 16 mal)is The first step in the user-centred process is the 1.2 Augmented Reality (AR) The first step in the user-centred development development process iswith the Augmented Reality (AR) is aa visual extension of the reality. It combines digital information such as videos, pictures, texts user requirement analysis. 25 persons (9 female, 16 mal) with Augmented Reality (AR) is visual extension of the reality. It The first step in the user-centred development process is the auser service and maintenance background took part in this requirement analysis. 25 persons (9 female, 16 mal) with Augmented Reality (AR) is a visual extension of the reality. It user requirement analysis. 25 persons (9 female, 16 mal) with The first step in the user-centred development process is the combines digital information such as videos, pictures, texts Augmented Reality (AR)in is reality, a visual extension ofpictures, thepicture reality. It user etc. with data that exists for example the of a service and maintenance background took part in this combines digital information such as videos, texts requirement analysis. 25 persons (9 female, 16 mal) with analysis (Buyer, Wittenberg, 2015, Wittenberg, 2016). service and maintenance background took part in this combines digital information such extension as videos,ofpictures, texts Augmented Reality (AR)in is reality, a visual thepicture reality. It aauser service and maintenance background took part in this requirement analysis. 25 persons (9 female, 16 mal) with etc. with data that exists for example the of combines digital information such as videos, pictures, texts aetc. digital supplementary data can help user of to aanalysis analysis (Buyer, Wittenberg, 2015, 2015, Wittenberg, 2016). etc. with data that exists in the picture service(Buyer, and maintenance background took2016). part in this Wittenberg, Wittenberg, with camera. data thatThe exists in reality, reality, for example thethe picture of combines digital information such for as example videos, pictures, texts analysis (Buyer, Wittenberg, 2015, Wittenberg, 2016). aAsservice and background took part in as thisa aaaccomplish digital supplementary can help the user to a result, the maintenance fast information retrieval is mentioned etc. with camera. data thatThe exists inThe reality, fordata example the picture of sever-al tasks. virtual extension occurs in real digital camera. The supplementary data can help the user to analysis (Buyer, Wittenberg, 2015, Wittenberg, 2016). aaccomplish digital supplementary can help the user to analysis etc. with camera. data thatThe exists inThe reality, fordata example the picture of (Buyer, Wittenberg, 2015, Wittenberg, 2016). As a result, the fast information retrieval is mentioned as a sever-al tasks. virtual extension occurs in real main aspect. This is confirmed by responses of the subjects atime digital camera. The supplementary data can help the user to As a result, the fast information retrieval is mentioned as so that relevant digital data can be retrieved by the user. accomplish sever-al tasks. The virtual virtualdata extension occurs in real real a result, the fast information retrieval is mentioned asonaa accomplish sever-al The extension occurs in aaccomplish digital camera. Thetasks. supplementary can help to As main aspect. This isprincipal confirmed by responses responses ofmentioned the subjectsasfor ona time so that that relevant digital data can be retrieved bythe theuser user. As a aspect. result, of the fast information retrieval isof sever-al tasks. The virtual extension occurs inuser. real the question the proceed in obtaining a contract main This is confirmed by the subjects on Especially big companies such as Google and Apple are time so relevant digital data can be retrieved by the maina aspect. Thisfast is confirmed by responses the subjectsasona time so that sever-al relevanttasks. digitalThe data can be retrievedoccurs by theinuser. result, the information retrieval isofmentioned accomplish virtual extension real the question in obtaining aa contract for Especially big companies such as be Google and by Apple are As main aspect. of This isprincipal confirmed by responses of the subjects on time so that relevant digital dataeveryday can retrieved the user. maintenance ofthe a system withproceed following points: „Use technical the question of the principal proceed in obtaining for working on big integrating AR into life and developing Especially companies such as Google Apple are the question of the principal proceed in obtaining a contract contract for Especially big companies such as Google and Apple are main aspect. This is confirmed by responses of the subjects on time so that relevant digital data can be retrieved by the user. maintenance of a system with following points: „Use technical working on integrating ARAR-Systems into everyday life andetdeveloping developing question“,of the principal proceed in obtaining acomparison“ contract for Especially big companies sucheveryday as Google Apple are the documents „carry out a target-performance maintenance of a system with following points: „Use technical new and more improved (Dörner al., 2013) working on integrating AR into life maintenance ofthe a system withproceed following points: „Use technical working on big integrating AR into everyday life and developing the question of principal in obtaining a contract for Especially companies such as Google Apple are documents “, out a target-performance comparison“ new and more improved (Dörner al., 2013) maintenance of„carry a system with following points: „Use technical working on integrating ARAR-Systems into everyday life andet developing such as „ Acquaintance with the system “.A problem often turn “, out a comparison“ (Bläss, Wittenberg, 2017). new more improved (Dörner al., documents “,of„carry „carry out a target-performance target-performance comparison“ new and and more improved AR-Systems (Dörner etdeveloping al., 2013) 2013) documents maintenance a system with following points: „Use technical working on integrating ARAR-Systems into everyday life andet such as „ Acquaintance with the system “.A problem often turn (Bläss, Wittenberg, 2017). documents “, „carry out a target-performance comparison“ new and more improved AR-Systems (Dörner et al., 2013) out is a decentralized storage of required documents (see fig. such as „ Acquaintance with the system “.A problem often (Bläss, Wittenberg, 2017). such as „ Acquaintance with the system “.A problem often turn turn (Bläss, Wittenberg, 2017). AR-Systems “, „carry out a target-performance comparison“ new and more (Dörner et al., 2013) documents The most knownimproved example out is a decentralized storage of required documents (see fig. such as „ Acquaintance with the system “.A problem often turn (Bläss, Wittenberg, 2017). for such a product is a data glass. 1). This leads to time-intensive searching and obtaining the out is a decentralized storage of required documents (see fig. out isasa „decentralized storage required (see turn fig. suchThis Acquaintance with theofsystem “.Adocuments problem often (Bläss, Wittenberg, 2017). The most known example for such aa product is aa data glass. These smart glasses are wearable AR-Systems that combine 1). leads to time-intensive searching and obtaining the The most known example for such product is data glass. out is a decentralized storage of required documents (see fig. leads searching and the The most known example for such a product is a data glass. 1). This leads to to time-intensive time-intensive searchingdocuments and obtaining obtaining the out This is a decentralized storage of required (see fig. These smart glasses are wearable combine The most known example for suchAR-Systems a product is that a data glass. 1). virtual reality with physical reality. These smart glasses are wearable AR-Systems that combine 1). This leads to time-intensive searching and obtaining the Thesemost smart glasses are wearable AR-Systems that combine The known example for such a product is a data glass. virtual reality with physical reality. These smart glasses are wearable AR-Systems that combine 1). This leads to time-intensive searching and obtaining the virtual reality with physical reality. virtual reality with physical reality. These smart glasses are wearable AR-Systems that combine virtual reality with physical reality. Copyright © 2019 IFAC virtual reality with 2405-8963 Copyright ©physical 2019. Thereality. Authors. Published by Elsevier Ltd. All1 rights reserved. Copyright 2019 IFAC 1 Copyright © 2019 IFAC 1 Peer review© of International Federation of Automatic Copyright ©under 2019 responsibility IFAC 1 Control. 10.1016/j.ifacol.2019.12.072 Copyright © 2019 IFAC 1 Copyright © 2019 IFAC 1
2019 IFAC HMS 2 Tallinn, Estonia, Sept. 16-19, 2019
Waldemar Rempel et al. / IFAC PapersOnLine 52-19 (2019) 1–6
documents before it can be effectively started on the appropriate system.
graphical representation of measured values. As played here from a bird's perspective, usually many objects are present on the playing surface and can be applied in accordance with each other An active object (multimeter) is applied to the passive object "S7-module". Other functions such as the graphical representation of a measured value can be triggered by selecting appropriate menu items of the active object For further information the user can to drag & drop the virtual multimeter on the real PLC module (fig. 3).
Fig. 1. Storage of maintenance documents (Buyer, Wittenberg, 2015) As a platform for the implementation of a first prototype a tablet computer was chosen. Via WLAN this tablet was connected to a Siemens PLC for data exchange.
Fig. 3. Selecting module-item to diagnose the Siemens S7IO-Modul to open module-items for measuring (Buyer, Wittenberg, 2015) The disadvantage of tablet computer is the necessity to hold the tablet computer in the hand. But a service and maintenance technician need the hands for the service tasks like working with tools. The logical next step lead to a data glass (Fig. 4).
Data glass PLC Fig. 2. Dragging the virtual multimeter as an active object (Buyer, Wittenberg, 2015) The developed approach for an intuitive information management system in augmented reality applications is shown in Fig. 2. The real-time camera image was overlaid with additional information from the automation system. The user's attention in the augmented world is controlled with the help of an avatar. The reaction change of the avatar presents the user with an invalid system state and draws attention to the appropriate object. Detailed information can be found on an object associated visualization surface. For Example, a digital multimeter symbolizes a bad value. Thus, eliminates decoding errors manually or with help of external tools and provides relevant information at first glance.
Touchpad Fig. 4: Data glass and PLC (Wittenberg, 2016)
The functionality is based on the object-object scheme known from simulation- and strategy games. Selecting inherent menu items of the active object can trigger other functions such as 2
2019 IFAC HMS Tallinn, Estonia, Sept. 16-19, 2019
Waldemar Rempel et al. / IFAC PapersOnLine 52-19 (2019) 1–6
3
2. LABORATORY EQUIPMENT 2.1 Smart factory in the Laboratory The Otto-Rettenmaier Research Laboratory for Digitalization contains a model factory for education and research activities. Fig. 5 shows the structure of this factory. Transports & logistics
Filling & Packing
Conveyor Station
Packaging Station Filling Station
Unpacking & Recycling
Process automation
Mixing Station
Unpacking Station
Fig. 7: Distribution Station Production automation
Recycling Station
Distribution Station
Buffer Station
•
Handling Station
Fig. 5: Structure of the model factory (Rempel et al., 2019)
Buffer Station (Figure 8): The function of the buffer station is to provide a storage for the caps. The caps wait on a conveyor belt until these caps are needed for the following productions steps. If bottles are for sealing the caps are transported via the conveyor belt to an area for pickup.
Fig. 8: Buffer Station Fig. 6: Model factory This model factory consists of the following subsystems (Wittenberg et al., 2019): •
Distribution Station (Figure 7): This subsystem is arranged before the buffer station. The function of the buffer station is to provide the cabs for bottles which are distributed in this system. The distribution station contains of a magazine for the cabs and a swivel arm for moving the cabs to the next subsystem.
3
•
Handling Station: A handling device grabs the cap from the pickup area and places this cap on a bottle in the filling station.
•
Filling Station (Figure 9): This subsystem contains a rotary table. The empty bottles are moved with this rotary table, are filled with liquids or solids, and sealed with the caps. At the end, the filled bottles are transported to further conveying belt.
2019 IFAC HMS 4 Tallinn, Estonia, Sept. 16-19, 2019
Waldemar Rempel et al. / IFAC PapersOnLine 52-19 (2019) 1–6
•
Recycling Station (Figure 12): In this subsystem each bottle is brought to its initial state. The caps are opened and moved to the distribution station. The content of the bottles is emptied by a vacuum and the empty bottles move via the last conveyor belt to the filling station.
Fig. 9: Filling Station •
Packing Station (Figure 10): The filled and sealed bottles are packed into a six-pack container.
Fig. 12: Recycling Station All bottles are equipped with RFID tags. The automation system fulfils the Industry 4.0/IIoT standards. 2.2 AR Device Actual, one of the most popular AR devices is the Microsoft HoloLens. The HoloLens contains the following subsystems (Figure 13): •
inertial measurement unit, IMU
Fig. 10: Packing Station
•
4 environmental understanding cameras (1)
•
•
Depth camera (2) with IR emitter (5)
•
HD video camera (3)
•
4 microphones
•
ambient light sensor (4)
•
Conveyor Station: This subsystem has the function to transport the six-packs with the bottles to the following subsystem – the unpacking station. Unpacking Station: At a time three bottles from the six-pack container are moved via another conveyor belt to the recycling station.
Fig. 11: Unpacking Station Fig. 13: Microsoft HoloLens (Rempel et al., 2019) 4
2019 IFAC HMS Tallinn, Estonia, Sept. 16-19, 2019
Waldemar Rempel et al. / IFAC PapersOnLine 52-19 (2019) 1–6
The HoloLens offers various interaction possibilities. Examples are the gazing – looking in a direction over a specific time period – or the interaction via gestures. Typical gestures are the air tap (with thumb and forefinger) as a symbol for a mouse click or the Bloom (spreading all fingers) as a symbol for home.
5
Based on this model the main menu and first dialogue windows are developed. Dialable areas (via gazing plus gesture) got a cover (red volume body). Normally, these volume bodies are transparent. These areas are recognizable through the change of the appearance of the cursor – similar to well-known web applications.
2.3 Communication between HoloLens and model factory The HoloLens is via WLAN connected to an access point in the model factory. This access point is included in the ProfiNET network of this model factory. The access point is used as a bridge between the HoloLens and the PLCs of the model factory. The communication standard is OPC UA (Fig. 14). HoloLens
WLAN
Access Point
OPC UA
ProfiNET
SPS
Modell Factory
Fig. 16: 3D model with overlays
Fig. 14: Communication structure (Rempel et al., 2019)
The PLCs in the lower part of the model factory are also shaped by a (green) volume body. These areas provide the possibilities for the display of factory information like measured values – sored in the PLCs. Figure 16 shows the model with the overlays.
3. HUMAN-MACHINE INTERFACE AND 3D MODEL OF THE MODEL FACTORY The application for the HoloLens was developed with the Unity programming interface. The first step is design of the 3D model of the factory (Figure 15). This is necessary to place the AR dialogues at the right position in the field of view. The 3D model itself is not displayed with the HoloLens.
4 OUTLOOK The described research project represents the first step in the development of an AR application with a focus on production systems. The logical next step is the user evaluation. A first usability test was performed as a student project. The analysis of this evaluation is not completed yet and will be included in the final paper. Based on this evaluation a suitable interface will be developed. REFERENCES Bläss, M, C. Wittenberg (2017): “An online user analysis regarding the usage of mobile Augmented and Virtual Reality devices in the field of robotics”. In: S. Lackey, J. Chen (Ed.): Virtual, Augmented and Mixed Reality, Cham: Springer, pp. 383-393. Buyer, S., Wittenberg, C. (2015): AR and maintenance visualisation of process data and engineering information. In: Constantine Stephanidis (Ed.): HCI International 2015 – Posters’ Extended Abstracts, Springer Cham, 2015, pp 159-162. Dörner, R., W. Broll, P. Grimm, B. Jung (2013): „Virtual und Augmented Reality (VR/AR)“. Berlin Heidelberg: Springer Vieweg. Henderson, S.,Feiner, S.( 2011). Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair. IEEE Transactions on Visualization and Computer Graphics, pp 1355 –1368. Rempel, W., B. Bauer, C. Wittenberg (2019): Der Einsatz von Augmented Reality in der Industrie 4.0 am Beispiel einer
Fig. 15: Modelling the 3D model of the factory 5
2019 IFAC HMS 6 Tallinn, Estonia, Sept. 16-19, 2019
Waldemar Rempel et al. / IFAC PapersOnLine 52-19 (2019) 1–6
Modellfabrik. In: B.Bauer, C. Wittenberg (Eds.): Tagungsband AALE 2019, VDE-Verlag, Berlin, pp 395401. Weiser, M (1991): “The computer of the 21st century,” Scientific American, vol. 265, no. 3, pp. 66–75. White, S., Lister, L. and Feiner, S. (2007). Visual Hints for Tangible Gestures in Augmented Reality. Proceedings of the 6th International Symposium on Mixed and Augmented Reality ISMAR, Nara, Japan, pp 47-50. Wittenberg, C. (2003): A Requirement Analysis for the use of Mobile Devices in Service and Maintenance. In: Proceedings 2003 IEEE International Conference on Systems, Man & Cybernetics, Vol. 4, pp 4033-4039, IEEE Conference Publications, New York. Wittenberg, C. (2016): Human-CPS Interaction – requirements and mobile human-machine interaction methods for the Industry 4.0. Proceedings of the 13th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems, IFACPapersOnLine, Volume 49, Issue 19, 2016, pp. 420-425. Wittenberg, Bauer, Stache (2019): A Smart Factory in a laboratory size for developing and testing innovative human-machine interaction concepts. Proceedings of the 10th International Conference on Applied Human Factors and Ergonomics, Springer Cham, in Press. Zhang, F. and Sun, H. (2005). Dynamic labeling management in virtual and augmented environments. Proceedings of the Ninth International Conference on Computer Aided Design and Computer Graphics, Hong Kong, China
6