- Email: [email protected]
A new AR Interaction for Collaborative E-maintenance System
7th IFAC Conference on Manufacturing Modelling, Management, and Control International Federation of Automatic Control June 19-21, 2013. Saint Petersburg, Russia
A new AR Interaction for Collaborative E-maintenance System N. Zenati*. S. Benbelkacem*. M. Belhocine*. A. Bellarbi* *Development Centre of Advanced Technologies . Algeria (e-mail: [email protected]) Abstract: This paper describes a prototype of a collaborative e-maintenance system based on distributed Augmented Reality (AR). This technology is implemented to support specific assistance for users, which consists of providing all the information known as augmentation such as 3D virtual models and instructions required to perform tasks. The distributed aspect of this system will allows the remote collaboration between a distant expert and a local user equipped with a wearable computer. Two main aspects are studied and developed. The first aspect addresses a new collaboration strategy based on Service Oriented Architecture (SOA). The second aspect ensures the virtual objects transfer from the remote expert in real time. Experimental results are presented Keywords: augmented reality, e-maintenance, collaboration, tracking, web services.
Collaborative work between a remote expert and an operator is a main solution in many ways, such as with regard to quality control and feedback, although a system enabling remote interactions to be supported is needed (Bottechia et al., 2010). Our area of interest is the establishment of a distributed platform allowing collaboration between technicians and remote experts based on AR benefits. Two main aspects are studied and developed in our case. The first aspect addresses a new collaboration strategy based on Service Oriented Architecture (SOA) which offers efficient solutions in terms of information transfer and exchange, data heterogeneous management, etc. The second aspect ensures the virtual objects (maintenance procedures) transfer from the remote expert in real time. This paper is organized as follows: in section 2 related work of AR in maintenance assistance is given. Section 3 describes the principle of our system. Implementation and results are presented in the section 4. Finally, a conclusion and perspectives are given in the last section.
1. INTRODUCTION Augmented Reality (AR) is a new way of human-computerinteraction, where virtual objects are added to real scenes provided by a video camera in real time (Azuma, 1997) (Mackay, 1998). They are inserted in the right positions and complement the real picture. The digital information merges with the user’s environment so that the user can perceive currently important information directly where it is needed. AR is derived from Virtual Reality (VR) in which the user is completely immersed in an artificial world. In VR systems, there is no way for the user to interact with objects in the real world. Using AR technology, operator can thus interact with a mixed virtual and real world in a natural way (Fuchs et al;, 2003). An AR system that can enhances a user’s view of the surrounding scene with annotations on the scene content, has many potential applications. Azuma (Azuma, 1997) gives a description of various applications of AR systems including medical visualization, manufacturing and repair, robot path planning, architecture and design, robotics and tele-operation, games, entertainment and military aircraft. AR is considered as an ideal technology for industrial maintenance and service (Changzhi, 2006). It allows the user to see virtual objects increased on the real world through display devices such as PC, Laptops, Pocket PC, Videoprojector or Head Mounted Display. Using this system, the technician can interact with the virtual world and have additional information which can be a set of maintenance instructions given in the form of texts, images, video or audio augmentations. A number of maintenance platform based on AR have been developed such as STARMATE (Schwald et al., 2001), AMRA (Didier et al., 2005), ARVIKA (Weidenhausen et al., 2003) and ARMAR (Henderson et al., 2010). The goal of these platforms is to provide assistance to a user who has to perform maintenance tasks. Thanks to the explosion of bandwidth and the World Wide Web, real time assistance is becoming accessible. 978-3-902823-35-9/2013 © IFAC
2. RELATED WORK 2.1. Major AR research in maintenance Maintenance is one of the most promising fields where AR can be used to improve the current techniques and provide new solutions in the future. The application of AR in the maintenance started in the early 1990s. As one of the revolutionary AR maintenance system was proposed by Steve Feiner and his team (Feiner et al., 1993). This system called KARMA introduced the idea of Knowledge-based Augmented Reality for Maintenance Assistance, demonstrating a system for aiding a user in servicing a standard office laser printer. Afterward, other systems were proposed. Caudel and Mizell (Caudell et al., 1992) described the design and prototyping steps constructed wire harness for airplanes such as Boeing with an AR system instead of real boards, manuals and paper templates.
619
10.3182/20130619-3-RU-3018.00476
2013 IFAC MIM June 19-21, 2013. Saint Petersburg, Russia
Schwald (Schwald et al., 2003) proposed an AR prototype whose aim is first to provide assistance to a user who has to perform demanding working processes on complex mechanical elements and second to increase the skills of a user to perform such processes by passing him through training scenarios. Didier (Didier et al., 2005) implemented an AR system for mobile use in industrial applications such as train maintenance and repairs in industrial sites. The work of Feiner and Henderson (Henderson, et al., 2010) explored and evaluated the feasibility of developing prototype adaptive AR systems that can be used to investigate how real time computer graphics, overlaid on and registered with the actual equipment being maintained, can significantly increase the productivity of maintenance personnel, both during training and in the field. The aim of all these systems is to guide operators in the accomplishment of their tasks by giving additional information in form of augmentations in real time. Both of these conditions should reduce the risks of running errors according to Neumann's work (Neumann et al., 1996). However, in order to perform maintenance task, an operator can be assisted by a remote expert. Cooperation is considered as an effective and concrete articulation between them, involved action. The next section reviews the work in this space.
augmentation into video stream to increase the user’s view in real time and make him/her understand the tasks to be carried out. 3.1. Global architecture of the system Our work is based on a distant collaboration between technicians and experts. In order to build infrastructure for an exchange between consumers and providers via services through disparate areas, we propose an architecture called “SOA” (Service Oriented Architecture) to design our emaintenance platform. (Nickul, 2007). To address this need, a distributed platform is then required to facilitate the collaboration between a technician and the remote expert (Benbelkacem et al., 2011). Fig 1 illustrates a distributed and mobile platform allowing a dialogue between technicians and remote experts.
2.2. Augmented reality remote guiding system This section presents a recent development of an augmented reality remote guiding systems in maintenance assistance. Zhong in his work (Zhong et al., 2002) presented a prototype which enables operators equipped with an AR display device to be able to "share" their view with a remote expert. The operator can handle virtual objects in order to be trained in a task which is supervised by an expert. Sakata (Sakata et al., 2003) and Kurata (Kurata et al., 2004) , developed the wearable Active camera/Laser (WACL) system that involves the worker wearing a steerable camera/laser head. WACL allows the remote instructor not only to independently look into the worker’s task space, but also to point to real objects in the task space with the laser spot. Bottecchia (Bottecchia et al., 2010) presented the "T.A.C" system whose aim is to combine remote collaboration and industrial maintenance. This system enables the co presence of parties within the framework of a supervised maintenance task to be remotely simulated thanks to augmented and audio communication. Recently, J. Azpiazu (Azpiazu et al., 2011) proposed a mobile and remote augmented reality system to support effective maintenance in the railway sector. The use of augmented reality eases the integration and useful visualization of data meant to help the on-site worker to perform the operations; in a different business case, augmented reality enhances the communication between a remote expert and an on-site worker, allowing better information exchange between both actors.
Fig. 1. Overview of the proposed architecture When an equipment fails, the technician receives an alarm from a supervisor and proceeds to a first analysis to locate the error and its cause and formulate a diagnostic. So, through lack of competences he contacts a remote expert using internet network (chat, email exchanges, images and videos) in order to perform a detailed diagnosis. The technician equipped with a Head Mounted Display captures a video stream of the failure’s location and send it to the expert. The expert is equipped with a simple computer executing an application that receives video streaming from the operator (Fig. 2).
3. SYSTEM DESCRIPTION In this section, we will describe the distributed and the collaborative prototype built to support an AR interaction. With this system, the expert should be able to directly insert
Fig. 2. Interface for the expert 620
2013 IFAC MIM June 19-21, 2013. Saint Petersburg, Russia
The expert inserts augmentation (maintenance scenarios) into video stream to increase the operator’s view in real time. When the intervention is achieved, the technician fills out a maintenance report which contains all the information related to the failure and the intervention procedure. This report is transmitted to the supervisor (for validation) and stored in the interventions history database.
Web Server
WSDL
Service Broker
WSDL
Service Provider
SOAP
Send SOAP files and run web methods in WSDL documents
3.2. Web service for mobile distributed system The interaction between actors (experts and technicians) requires communication platforms. Despite the advantages of these technologies, it cannot manage effectively the heterogeneous running environment especially when using several communication tools as PC-Pocket, HMD, PC-Tablet and others. The Web service is a main solution to resolve this problem. The main role is to facilitate the data transmission between actors using Internet network (Zerarga et al., 2011). A Web Service is a software module which performs a set of discrete tasks. It can be soliciting and calling through a network, especially the World Wide Web, accessible via standard Internet protocols (Leymann, 2003). It is a set of services which can be invoked via internet network. Web Services are both characterized by the reuse facilitation. They are independent from used platforms (Windows, UNIX ...) and programming languages (C #, JAVA, VB ...). This interoperability form makes Web services one of the most technologies used to design distributed applications (Leymann, 2003). In our case, web service principle is adopted to establish a distributed mobile maintenance platform. A distributed system allows the communication between autonomous interconnected devices (computers, PDAs, processors, processes, lightweight process etc...). In the case of maintenance operation, this aspect allows a dialogue between technicians and exerts to perform remote maintenance operations. This principle has been approved for various reasons, namely: For simplicity implementation and use. For mobility of connected stations. For cost reduction. As a most important concept of Web Services, we are interested to use WCF (Windows Communication Foundation). It is a new feature of Dot NET Framework version. It supports the sending of HTTP data. HTTP protocol facilitates the communication in the Internet network (Scott, 2007). The actors consume Web Services by sending SOAP file. The services are described in a WSDL file and distributed through a UDDI registry. This allows easy collaboration between actors to resolve several technicians task. The actors communicate by exchanging SOAP files through a service provider in order to run web methods. These web methods are described in WSDL file which is hosted in the service broker. Fig.3 illustrates an example of a collaboration between three experts and one technician.
Expert
Expert
Expert
Technician
Fig. 3. Example of Collaboration between actors 3.3. Vision Based Tracking Tracking has been at the heart of research and development in AR since its inception in the 1960s (Fiala, 2004), (Comport et al., 2005). Tracking methods use cameras that capture real scene as sensing devices. So, to apply AR to maintenance support, the user’s positions and orientations in real time must be measured with high accuracy. Among the tracking technologies proposed in previous studies (Comport et al., 2005), marker-based tracking, which uses image processing technique to measure the relative position and orientation between a camera and markers (transformation matrix marker/camera), seems to be appropriate to industrial maintenance context. In fact, the most popular tracking method, which uses square markers, is ARToolkit (Kato et al., 1999). In fact, the results of augmentation based on visionbased methods are more precise as they are directly computed from features extracted from the images to be augmented. ARToolkit (Augmented Reality Toolkit) is an open source library software used to develop augmented reality applications. It uses vision techniques to calculate position and orientation of the camera relative to markers. The programmer can use this information to draw the 3D object and to insert it correctly in the real scene. In addition, ARToolkit guarantees the virtual object tracking when the camera (or user) changes the position. The library "ARToolkit" has several types of markers (Kato et al., 1999) (see Fig.4).
Hiro
Fig. 4. Example of ARToolkit markers The markers are in a black square form with a code inside. This marker is a simple form that can be easily identified for the increased of virtual object. The position and orientation of the camera can be calculated by identifying the markers in a video stream. However, despite the advances noted in the 621
2013 IFAC MIM June 19-21, 2013. Saint Petersburg, Russia
implementation of AR applications, ARToolKit still has certain deficiencies. To address these issues, a new version of ARToolKit named I-ARToolkit is proposed in our team (Bellarbi et al., 2010).
specific devices like optical or video see-through head mounted displays (HMD), PDA or tablet PC’s . Within the framework of maintenance, the operator must have his hand free to perform his task. That why we have decided to use an HMD. Our tests are performed on a car engine. Fig 6 shows an operator equipped with a Vuzix Wrap™ 920AR eyewear.
The proposed I-ARToolkit algorithm is given as follows: Open a video stream For each captured image o Calculate the optimal threshold value of the captured image using Otsu method ( Otsu, 1979) o Transform the captured image to a binary image using the calculated threshold value. o Detect black squares markers in the binary image. o Calculate the position and orientation of the camera (compute the transformation matrix) o Apply the stabilization algorithm (Bellarbi et al., 2010). o Superimpose the virtual objects on the captured image (using the calculated transformation matrix). 3.4. Collaboration between a technician and an expert
Fig. 6. A technician works with a Head Mounted Display
Thanks to telecommunications, an operator can be assisted by a remote expert when performing difficult tasks. The main difficulty lies in how to send the augmented scene from the expert to the technician in order to repair the broken equipment. The adopted approach consists in computing the technician's position (transformation matrix marker/camera) from the captured images. The captured images and the transformation matrix are both transmitted to the remote expert (see Fig. 5). This method gives good results since the markers detection is performed at the technician’s device. The transformation matrix and the images are transferred to the remote expert’s computer.
Being given the complexity of the equipments to be repaired, remote assistance using augmented reality tools seems to be an appropriate solution. The scene captured by the camera integrated in the on-site worker's glasses, is streamed in realtime to the expert. The expert views the video and proposes a maintenance procedure (Fig.7). When the corresponding marker is identified by AR system, the local operator can simultaneously see the virtual annotation information that overlay on the marker through HMD (see Fig.8). These annotations overlay images are real scenes images from the user’s viewpoint with three kinds of information shown in table.1.
Fig. 7. Remote support interaction
Fig. 5. Collaboration between technician and expert 4.
IMPLEMENTATION AND TRIALS
In this section a maintenance application using our platform is implemented. Unlike virtual reality which requires complete immersion, the operator using augmented reality can interact with mixed virtual and real world in a natural way using
Fig. 8. Augmented view of the technician
622
2013 IFAC MIM June 19-21, 2013. Saint Petersburg, Russia
Caudell, T., Mizell, D. (1992). Augmented reality: an application of heads-up display technology to manual manufacturing processes. Proceedings of the Twenty-fifth Hawaii international conference on system sciences. pp. 659–669. Comport, A. I., Kragic, D., Marchand, E., Chaumette, F. (2005). Robust real-time visual tracking: comparison, theoretical analysis and performance evaluation. In IEEE International Conference on Robotics and Automation (ICRA’05), pp.2841-2846, Barcelona, Spain. Didier, J.-Y., Roussel, D., Mallem, M., Otmane,S., Naudet, S., Pham, C., Bourgeois, S., Mégard, C. (2005). Amra : Augmented reality assistance in train maintenance tasks. Dans Workshop on Industrial Augmented Reality (ISMAR’05), Vienne, Autriche. Feiner, S., MacIntyre,B., Seligmann,D.(1993). KnowledgeBased Augmented Reality, Communication de l’ACM n°7, pp. 53-61. Fiala, M. (2004). Artag, An improved marker system based on ARToolkit. National Research Council, Canada. Fuchs, P., Moreau,G., Arnaldi,B., Burkhardt, J.M., Chauffaut, A. , Coquillart, S., Donikian,S., Duval, T., Grosjean, J., Harrouet, F. ( 2003). Le Traité de la réalité virtuelle. Les Presses de l’Ecole des Mines de Paris, deuxième édition, 2 volumes, ISBN 2-911762-47-9 et 2-911762-48-7, 915 pages. Goose, S., Guven,S., Zhang,X., Sudarsky,S., Navab,N.(2003). Mobile 3D Visualization an Interaction in an Industrial Environment. In Stephanidis C. (Eds), Universal Access in HCI, Inclusive Design in the Information Society. 2227 June 2003, Crete, Greece, London : Lawrence Erlbaum Associates,03, vol. 4, pp. 379-383. Henderson,S., and Feiner, S. (2010). Opportunistic tangible user interfaces for augmented reality. IEEE Transactions on Visualization and Computer Graphics, 16(1), pp 4-16. Kurata, T., Sakata, N., Kourogi, M., Kuzuoka, H. and Billinghurst, M. 2004. Remote collaboration using a shoulder-worn active camera/laser Wearable Computers.ISWC 2004. Eighth International Symposium, pp 62-69. Koller, D., Klinker, G., Rose, E., Breen,D. (1997). Real-time vision-based camera tracking for augmented reality applications. Proceedings of the ACM symposium on Virtual reality software and technology, pp. 87-94. Leymann, F. (2003). Web Services: Distributed Applications without Limits. Proceedings of 10th Conference on Database Systems for Business. Mackay, W.E. ( 1998). Augmented Reality: Linking real and virtual worlds - A new paradigm for interacting with computers, In Proceedings of AVI’98, ACM Conference on Advanced Visual Interfaces, l'Aquila, Italy, New York: ACM Press. Neumann,U., and Cho,Y.(1996). A Self-Tracking Augmented Reality System. Proc. of ACM Virtual Reality Software and Technology. pp. 109-115. Nickul, D. (2007). Service Oriented Architecture (SOA) and Specialized Messaging Patterns. Adobe Systems Incorporated, San-Jose, USA. Sakata, N., Kurata, T., Kato, T., Kourogi, M and Kuzuoka, H. (2003). WACL: supporting telecommunications using wearable active camera with laser pointer Wearable
Table 1. Kinds of annotations Virtual annotation information Textual annotation Pointing arrows Positioning frame
Function Showing the name of components which will install to the user Showing the orientation of component which will install to the user Showing the position of components Which will install to the user
5. CONCLUSIONS AND FUTURE WORK In this paper, we described a prototype developed for emaintenance which features augmented reality and collaboration. As a result, we are focused on synchronous and remote collaboration between technicians and experts to complete maintenance and repair tasks. The adopted strategy allows the technician to collaborate easily with a remote expert. In this paper, the car maintenance problem has been used as a test case application to show the potential of augmented reality by taking into the following objectives: Minimize the costs of maintenance tasks improve time taken to complete maintenance tasks Improve the communication between technicians and experts In the near future, we intend to improve the prototype in different ways. We plan to enhance the technician and interface and interaction methods in order to make manipulation of graphical objects easier. REFERENCES Ababsa,F., Mallem, M (2008). A Robust Circular Fiducial Detection Technique and Real-Time 3D Camera Tracking. Journal of Multimedia, Vol 3, No 4, pp. 34-41. Azpiazu, J., Siltanen, S., Multanen,M., Mäkiranta, A., N Barrena, N., Díez, A., Agirre, J.(2011). Remote support for maintenance tasks by the use of Augmented Reality: the ManuVAR project. 9th Congress on Virtual Reality Applications (CARVI). Vitoria-Gasteiz, Spain. Azuma, R.T.(1997). A Survey of Augmented Reality, Dans la revue Presence: Teleoperators and Virtual Environments, Vol.4, pp. 355-385. Bellarbi, A., Benbelkacem, S., Malek, M., Zenati-Henda, N., Belhocine, M. (2010). Amélioration des performances d'ARToolKit pour la réalisation d'applications de réalité augmentée. International Conference on Image and Signal Processing and their Applications, ISPA’2010, Biskra, Algeria. Benbelkacem, S., Zenati-Henda, N., Zerarga,F., Bellarbi,A., Belhocine,M., Tadjine M. (2011). Augmented Reality Platform for Collaborative E-Maintenance Systems. Andrew Yeh Ching Nee (ed), Augmented Reality - Some Emerging Application Areas. 266 pages. InTech. ISBN 978-953-307-422-1. Bottecchia, S., Cieutat, J.-M., Jessel J.-P. (2010) T.A.C: Augmented Reality System for Collaborative TeleAssistance in the Field of Maintenance through Internet. Augmented Human (AH’2010), Megève, France.
623
2013 IFAC MIM June 19-21, 2013. Saint Petersburg, Russia
Computers. Proceedings. Seventh IEEE International Symposium on, 2003, pp 53-56. Schwald, B.& de Laval, B.(2003). An Augmented Reality System for Training and Assistance to Maintenance in the Industrial Context, Journal of WSCG’2003, pp. 425-432. Otsu, N. (1979). A Threshold Selection Method from GrayLevel Histograms. IEEE Transactions on Systems, Man, and Cybernetics, Vol.9, pp. 62–66. Scott, K. (2007). Professional WCF Programming: .NET development with the Windows communication foundation. Amazon Edition, 430 pages. Wang, Wei., Qi, Yue., Wang, QingXing. (2011) An augmented reality Application Framework for Complex Equipment Collaborative Maintenance. In CDVE(2011) LNCS 6874, pp. 154-161. Weidenhausen, Jens., Stricker, C K. (2003). Arvika augmented reality for development, production and service. Computers & Graphics. Volume 27, Issue 6. Zerarga, F., Benbelkacem, S., Zenati N., Belhocine, M. (2011). Designing distributed mobile platform for emaintenance assistance. 11th International Conference on Automation and Mechatronics, 24-27 November 2011, Oran, Algeria, pp.485-488. Zhong,X., Boulange,P., and Georganas, N.D. (2002). Collaborative augmented reality: A prototype for industrial training. 21th Biennial Symposium on Communication, Canada.
624
A new AR Interaction for Collaborative E-maintenance System N. Zenati*. S. Benbelkacem*. M. Belhocine*. A. Bellarbi* *Development Centre of Advanced Technologies . Algeria (e-mail: [email protected]) Abstract: This paper describes a prototype of a collaborative e-maintenance system based on distributed Augmented Reality (AR). This technology is implemented to support specific assistance for users, which consists of providing all the information known as augmentation such as 3D virtual models and instructions required to perform tasks. The distributed aspect of this system will allows the remote collaboration between a distant expert and a local user equipped with a wearable computer. Two main aspects are studied and developed. The first aspect addresses a new collaboration strategy based on Service Oriented Architecture (SOA). The second aspect ensures the virtual objects transfer from the remote expert in real time. Experimental results are presented Keywords: augmented reality, e-maintenance, collaboration, tracking, web services.
Collaborative work between a remote expert and an operator is a main solution in many ways, such as with regard to quality control and feedback, although a system enabling remote interactions to be supported is needed (Bottechia et al., 2010). Our area of interest is the establishment of a distributed platform allowing collaboration between technicians and remote experts based on AR benefits. Two main aspects are studied and developed in our case. The first aspect addresses a new collaboration strategy based on Service Oriented Architecture (SOA) which offers efficient solutions in terms of information transfer and exchange, data heterogeneous management, etc. The second aspect ensures the virtual objects (maintenance procedures) transfer from the remote expert in real time. This paper is organized as follows: in section 2 related work of AR in maintenance assistance is given. Section 3 describes the principle of our system. Implementation and results are presented in the section 4. Finally, a conclusion and perspectives are given in the last section.
1. INTRODUCTION Augmented Reality (AR) is a new way of human-computerinteraction, where virtual objects are added to real scenes provided by a video camera in real time (Azuma, 1997) (Mackay, 1998). They are inserted in the right positions and complement the real picture. The digital information merges with the user’s environment so that the user can perceive currently important information directly where it is needed. AR is derived from Virtual Reality (VR) in which the user is completely immersed in an artificial world. In VR systems, there is no way for the user to interact with objects in the real world. Using AR technology, operator can thus interact with a mixed virtual and real world in a natural way (Fuchs et al;, 2003). An AR system that can enhances a user’s view of the surrounding scene with annotations on the scene content, has many potential applications. Azuma (Azuma, 1997) gives a description of various applications of AR systems including medical visualization, manufacturing and repair, robot path planning, architecture and design, robotics and tele-operation, games, entertainment and military aircraft. AR is considered as an ideal technology for industrial maintenance and service (Changzhi, 2006). It allows the user to see virtual objects increased on the real world through display devices such as PC, Laptops, Pocket PC, Videoprojector or Head Mounted Display. Using this system, the technician can interact with the virtual world and have additional information which can be a set of maintenance instructions given in the form of texts, images, video or audio augmentations. A number of maintenance platform based on AR have been developed such as STARMATE (Schwald et al., 2001), AMRA (Didier et al., 2005), ARVIKA (Weidenhausen et al., 2003) and ARMAR (Henderson et al., 2010). The goal of these platforms is to provide assistance to a user who has to perform maintenance tasks. Thanks to the explosion of bandwidth and the World Wide Web, real time assistance is becoming accessible. 978-3-902823-35-9/2013 © IFAC
2. RELATED WORK 2.1. Major AR research in maintenance Maintenance is one of the most promising fields where AR can be used to improve the current techniques and provide new solutions in the future. The application of AR in the maintenance started in the early 1990s. As one of the revolutionary AR maintenance system was proposed by Steve Feiner and his team (Feiner et al., 1993). This system called KARMA introduced the idea of Knowledge-based Augmented Reality for Maintenance Assistance, demonstrating a system for aiding a user in servicing a standard office laser printer. Afterward, other systems were proposed. Caudel and Mizell (Caudell et al., 1992) described the design and prototyping steps constructed wire harness for airplanes such as Boeing with an AR system instead of real boards, manuals and paper templates.
619
10.3182/20130619-3-RU-3018.00476
2013 IFAC MIM June 19-21, 2013. Saint Petersburg, Russia
Schwald (Schwald et al., 2003) proposed an AR prototype whose aim is first to provide assistance to a user who has to perform demanding working processes on complex mechanical elements and second to increase the skills of a user to perform such processes by passing him through training scenarios. Didier (Didier et al., 2005) implemented an AR system for mobile use in industrial applications such as train maintenance and repairs in industrial sites. The work of Feiner and Henderson (Henderson, et al., 2010) explored and evaluated the feasibility of developing prototype adaptive AR systems that can be used to investigate how real time computer graphics, overlaid on and registered with the actual equipment being maintained, can significantly increase the productivity of maintenance personnel, both during training and in the field. The aim of all these systems is to guide operators in the accomplishment of their tasks by giving additional information in form of augmentations in real time. Both of these conditions should reduce the risks of running errors according to Neumann's work (Neumann et al., 1996). However, in order to perform maintenance task, an operator can be assisted by a remote expert. Cooperation is considered as an effective and concrete articulation between them, involved action. The next section reviews the work in this space.
augmentation into video stream to increase the user’s view in real time and make him/her understand the tasks to be carried out. 3.1. Global architecture of the system Our work is based on a distant collaboration between technicians and experts. In order to build infrastructure for an exchange between consumers and providers via services through disparate areas, we propose an architecture called “SOA” (Service Oriented Architecture) to design our emaintenance platform. (Nickul, 2007). To address this need, a distributed platform is then required to facilitate the collaboration between a technician and the remote expert (Benbelkacem et al., 2011). Fig 1 illustrates a distributed and mobile platform allowing a dialogue between technicians and remote experts.
2.2. Augmented reality remote guiding system This section presents a recent development of an augmented reality remote guiding systems in maintenance assistance. Zhong in his work (Zhong et al., 2002) presented a prototype which enables operators equipped with an AR display device to be able to "share" their view with a remote expert. The operator can handle virtual objects in order to be trained in a task which is supervised by an expert. Sakata (Sakata et al., 2003) and Kurata (Kurata et al., 2004) , developed the wearable Active camera/Laser (WACL) system that involves the worker wearing a steerable camera/laser head. WACL allows the remote instructor not only to independently look into the worker’s task space, but also to point to real objects in the task space with the laser spot. Bottecchia (Bottecchia et al., 2010) presented the "T.A.C" system whose aim is to combine remote collaboration and industrial maintenance. This system enables the co presence of parties within the framework of a supervised maintenance task to be remotely simulated thanks to augmented and audio communication. Recently, J. Azpiazu (Azpiazu et al., 2011) proposed a mobile and remote augmented reality system to support effective maintenance in the railway sector. The use of augmented reality eases the integration and useful visualization of data meant to help the on-site worker to perform the operations; in a different business case, augmented reality enhances the communication between a remote expert and an on-site worker, allowing better information exchange between both actors.
Fig. 1. Overview of the proposed architecture When an equipment fails, the technician receives an alarm from a supervisor and proceeds to a first analysis to locate the error and its cause and formulate a diagnostic. So, through lack of competences he contacts a remote expert using internet network (chat, email exchanges, images and videos) in order to perform a detailed diagnosis. The technician equipped with a Head Mounted Display captures a video stream of the failure’s location and send it to the expert. The expert is equipped with a simple computer executing an application that receives video streaming from the operator (Fig. 2).
3. SYSTEM DESCRIPTION In this section, we will describe the distributed and the collaborative prototype built to support an AR interaction. With this system, the expert should be able to directly insert
Fig. 2. Interface for the expert 620
2013 IFAC MIM June 19-21, 2013. Saint Petersburg, Russia
The expert inserts augmentation (maintenance scenarios) into video stream to increase the operator’s view in real time. When the intervention is achieved, the technician fills out a maintenance report which contains all the information related to the failure and the intervention procedure. This report is transmitted to the supervisor (for validation) and stored in the interventions history database.
Web Server
WSDL
Service Broker
WSDL
Service Provider
SOAP
Send SOAP files and run web methods in WSDL documents
3.2. Web service for mobile distributed system The interaction between actors (experts and technicians) requires communication platforms. Despite the advantages of these technologies, it cannot manage effectively the heterogeneous running environment especially when using several communication tools as PC-Pocket, HMD, PC-Tablet and others. The Web service is a main solution to resolve this problem. The main role is to facilitate the data transmission between actors using Internet network (Zerarga et al., 2011). A Web Service is a software module which performs a set of discrete tasks. It can be soliciting and calling through a network, especially the World Wide Web, accessible via standard Internet protocols (Leymann, 2003). It is a set of services which can be invoked via internet network. Web Services are both characterized by the reuse facilitation. They are independent from used platforms (Windows, UNIX ...) and programming languages (C #, JAVA, VB ...). This interoperability form makes Web services one of the most technologies used to design distributed applications (Leymann, 2003). In our case, web service principle is adopted to establish a distributed mobile maintenance platform. A distributed system allows the communication between autonomous interconnected devices (computers, PDAs, processors, processes, lightweight process etc...). In the case of maintenance operation, this aspect allows a dialogue between technicians and exerts to perform remote maintenance operations. This principle has been approved for various reasons, namely: For simplicity implementation and use. For mobility of connected stations. For cost reduction. As a most important concept of Web Services, we are interested to use WCF (Windows Communication Foundation). It is a new feature of Dot NET Framework version. It supports the sending of HTTP data. HTTP protocol facilitates the communication in the Internet network (Scott, 2007). The actors consume Web Services by sending SOAP file. The services are described in a WSDL file and distributed through a UDDI registry. This allows easy collaboration between actors to resolve several technicians task. The actors communicate by exchanging SOAP files through a service provider in order to run web methods. These web methods are described in WSDL file which is hosted in the service broker. Fig.3 illustrates an example of a collaboration between three experts and one technician.
Expert
Expert
Expert
Technician
Fig. 3. Example of Collaboration between actors 3.3. Vision Based Tracking Tracking has been at the heart of research and development in AR since its inception in the 1960s (Fiala, 2004), (Comport et al., 2005). Tracking methods use cameras that capture real scene as sensing devices. So, to apply AR to maintenance support, the user’s positions and orientations in real time must be measured with high accuracy. Among the tracking technologies proposed in previous studies (Comport et al., 2005), marker-based tracking, which uses image processing technique to measure the relative position and orientation between a camera and markers (transformation matrix marker/camera), seems to be appropriate to industrial maintenance context. In fact, the most popular tracking method, which uses square markers, is ARToolkit (Kato et al., 1999). In fact, the results of augmentation based on visionbased methods are more precise as they are directly computed from features extracted from the images to be augmented. ARToolkit (Augmented Reality Toolkit) is an open source library software used to develop augmented reality applications. It uses vision techniques to calculate position and orientation of the camera relative to markers. The programmer can use this information to draw the 3D object and to insert it correctly in the real scene. In addition, ARToolkit guarantees the virtual object tracking when the camera (or user) changes the position. The library "ARToolkit" has several types of markers (Kato et al., 1999) (see Fig.4).
Hiro
Fig. 4. Example of ARToolkit markers The markers are in a black square form with a code inside. This marker is a simple form that can be easily identified for the increased of virtual object. The position and orientation of the camera can be calculated by identifying the markers in a video stream. However, despite the advances noted in the 621
2013 IFAC MIM June 19-21, 2013. Saint Petersburg, Russia
implementation of AR applications, ARToolKit still has certain deficiencies. To address these issues, a new version of ARToolKit named I-ARToolkit is proposed in our team (Bellarbi et al., 2010).
specific devices like optical or video see-through head mounted displays (HMD), PDA or tablet PC’s . Within the framework of maintenance, the operator must have his hand free to perform his task. That why we have decided to use an HMD. Our tests are performed on a car engine. Fig 6 shows an operator equipped with a Vuzix Wrap™ 920AR eyewear.
The proposed I-ARToolkit algorithm is given as follows: Open a video stream For each captured image o Calculate the optimal threshold value of the captured image using Otsu method ( Otsu, 1979) o Transform the captured image to a binary image using the calculated threshold value. o Detect black squares markers in the binary image. o Calculate the position and orientation of the camera (compute the transformation matrix) o Apply the stabilization algorithm (Bellarbi et al., 2010). o Superimpose the virtual objects on the captured image (using the calculated transformation matrix). 3.4. Collaboration between a technician and an expert
Fig. 6. A technician works with a Head Mounted Display
Thanks to telecommunications, an operator can be assisted by a remote expert when performing difficult tasks. The main difficulty lies in how to send the augmented scene from the expert to the technician in order to repair the broken equipment. The adopted approach consists in computing the technician's position (transformation matrix marker/camera) from the captured images. The captured images and the transformation matrix are both transmitted to the remote expert (see Fig. 5). This method gives good results since the markers detection is performed at the technician’s device. The transformation matrix and the images are transferred to the remote expert’s computer.
Being given the complexity of the equipments to be repaired, remote assistance using augmented reality tools seems to be an appropriate solution. The scene captured by the camera integrated in the on-site worker's glasses, is streamed in realtime to the expert. The expert views the video and proposes a maintenance procedure (Fig.7). When the corresponding marker is identified by AR system, the local operator can simultaneously see the virtual annotation information that overlay on the marker through HMD (see Fig.8). These annotations overlay images are real scenes images from the user’s viewpoint with three kinds of information shown in table.1.
Fig. 7. Remote support interaction
Fig. 5. Collaboration between technician and expert 4.
IMPLEMENTATION AND TRIALS
In this section a maintenance application using our platform is implemented. Unlike virtual reality which requires complete immersion, the operator using augmented reality can interact with mixed virtual and real world in a natural way using
Fig. 8. Augmented view of the technician
622
2013 IFAC MIM June 19-21, 2013. Saint Petersburg, Russia
Caudell, T., Mizell, D. (1992). Augmented reality: an application of heads-up display technology to manual manufacturing processes. Proceedings of the Twenty-fifth Hawaii international conference on system sciences. pp. 659–669. Comport, A. I., Kragic, D., Marchand, E., Chaumette, F. (2005). Robust real-time visual tracking: comparison, theoretical analysis and performance evaluation. In IEEE International Conference on Robotics and Automation (ICRA’05), pp.2841-2846, Barcelona, Spain. Didier, J.-Y., Roussel, D., Mallem, M., Otmane,S., Naudet, S., Pham, C., Bourgeois, S., Mégard, C. (2005). Amra : Augmented reality assistance in train maintenance tasks. Dans Workshop on Industrial Augmented Reality (ISMAR’05), Vienne, Autriche. Feiner, S., MacIntyre,B., Seligmann,D.(1993). KnowledgeBased Augmented Reality, Communication de l’ACM n°7, pp. 53-61. Fiala, M. (2004). Artag, An improved marker system based on ARToolkit. National Research Council, Canada. Fuchs, P., Moreau,G., Arnaldi,B., Burkhardt, J.M., Chauffaut, A. , Coquillart, S., Donikian,S., Duval, T., Grosjean, J., Harrouet, F. ( 2003). Le Traité de la réalité virtuelle. Les Presses de l’Ecole des Mines de Paris, deuxième édition, 2 volumes, ISBN 2-911762-47-9 et 2-911762-48-7, 915 pages. Goose, S., Guven,S., Zhang,X., Sudarsky,S., Navab,N.(2003). Mobile 3D Visualization an Interaction in an Industrial Environment. In Stephanidis C. (Eds), Universal Access in HCI, Inclusive Design in the Information Society. 2227 June 2003, Crete, Greece, London : Lawrence Erlbaum Associates,03, vol. 4, pp. 379-383. Henderson,S., and Feiner, S. (2010). Opportunistic tangible user interfaces for augmented reality. IEEE Transactions on Visualization and Computer Graphics, 16(1), pp 4-16. Kurata, T., Sakata, N., Kourogi, M., Kuzuoka, H. and Billinghurst, M. 2004. Remote collaboration using a shoulder-worn active camera/laser Wearable Computers.ISWC 2004. Eighth International Symposium, pp 62-69. Koller, D., Klinker, G., Rose, E., Breen,D. (1997). Real-time vision-based camera tracking for augmented reality applications. Proceedings of the ACM symposium on Virtual reality software and technology, pp. 87-94. Leymann, F. (2003). Web Services: Distributed Applications without Limits. Proceedings of 10th Conference on Database Systems for Business. Mackay, W.E. ( 1998). Augmented Reality: Linking real and virtual worlds - A new paradigm for interacting with computers, In Proceedings of AVI’98, ACM Conference on Advanced Visual Interfaces, l'Aquila, Italy, New York: ACM Press. Neumann,U., and Cho,Y.(1996). A Self-Tracking Augmented Reality System. Proc. of ACM Virtual Reality Software and Technology. pp. 109-115. Nickul, D. (2007). Service Oriented Architecture (SOA) and Specialized Messaging Patterns. Adobe Systems Incorporated, San-Jose, USA. Sakata, N., Kurata, T., Kato, T., Kourogi, M and Kuzuoka, H. (2003). WACL: supporting telecommunications using wearable active camera with laser pointer Wearable
Table 1. Kinds of annotations Virtual annotation information Textual annotation Pointing arrows Positioning frame
Function Showing the name of components which will install to the user Showing the orientation of component which will install to the user Showing the position of components Which will install to the user
5. CONCLUSIONS AND FUTURE WORK In this paper, we described a prototype developed for emaintenance which features augmented reality and collaboration. As a result, we are focused on synchronous and remote collaboration between technicians and experts to complete maintenance and repair tasks. The adopted strategy allows the technician to collaborate easily with a remote expert. In this paper, the car maintenance problem has been used as a test case application to show the potential of augmented reality by taking into the following objectives: Minimize the costs of maintenance tasks improve time taken to complete maintenance tasks Improve the communication between technicians and experts In the near future, we intend to improve the prototype in different ways. We plan to enhance the technician and interface and interaction methods in order to make manipulation of graphical objects easier. REFERENCES Ababsa,F., Mallem, M (2008). A Robust Circular Fiducial Detection Technique and Real-Time 3D Camera Tracking. Journal of Multimedia, Vol 3, No 4, pp. 34-41. Azpiazu, J., Siltanen, S., Multanen,M., Mäkiranta, A., N Barrena, N., Díez, A., Agirre, J.(2011). Remote support for maintenance tasks by the use of Augmented Reality: the ManuVAR project. 9th Congress on Virtual Reality Applications (CARVI). Vitoria-Gasteiz, Spain. Azuma, R.T.(1997). A Survey of Augmented Reality, Dans la revue Presence: Teleoperators and Virtual Environments, Vol.4, pp. 355-385. Bellarbi, A., Benbelkacem, S., Malek, M., Zenati-Henda, N., Belhocine, M. (2010). Amélioration des performances d'ARToolKit pour la réalisation d'applications de réalité augmentée. International Conference on Image and Signal Processing and their Applications, ISPA’2010, Biskra, Algeria. Benbelkacem, S., Zenati-Henda, N., Zerarga,F., Bellarbi,A., Belhocine,M., Tadjine M. (2011). Augmented Reality Platform for Collaborative E-Maintenance Systems. Andrew Yeh Ching Nee (ed), Augmented Reality - Some Emerging Application Areas. 266 pages. InTech. ISBN 978-953-307-422-1. Bottecchia, S., Cieutat, J.-M., Jessel J.-P. (2010) T.A.C: Augmented Reality System for Collaborative TeleAssistance in the Field of Maintenance through Internet. Augmented Human (AH’2010), Megève, France.
623
2013 IFAC MIM June 19-21, 2013. Saint Petersburg, Russia
Computers. Proceedings. Seventh IEEE International Symposium on, 2003, pp 53-56. Schwald, B.& de Laval, B.(2003). An Augmented Reality System for Training and Assistance to Maintenance in the Industrial Context, Journal of WSCG’2003, pp. 425-432. Otsu, N. (1979). A Threshold Selection Method from GrayLevel Histograms. IEEE Transactions on Systems, Man, and Cybernetics, Vol.9, pp. 62–66. Scott, K. (2007). Professional WCF Programming: .NET development with the Windows communication foundation. Amazon Edition, 430 pages. Wang, Wei., Qi, Yue., Wang, QingXing. (2011) An augmented reality Application Framework for Complex Equipment Collaborative Maintenance. In CDVE(2011) LNCS 6874, pp. 154-161. Weidenhausen, Jens., Stricker, C K. (2003). Arvika augmented reality for development, production and service. Computers & Graphics. Volume 27, Issue 6. Zerarga, F., Benbelkacem, S., Zenati N., Belhocine, M. (2011). Designing distributed mobile platform for emaintenance assistance. 11th International Conference on Automation and Mechatronics, 24-27 November 2011, Oran, Algeria, pp.485-488. Zhong,X., Boulange,P., and Georganas, N.D. (2002). Collaborative augmented reality: A prototype for industrial training. 21th Biennial Symposium on Communication, Canada.
624