- Email: [email protected]
A MONITOR-BASED AR SYSTEM AS A SUPPORT TOOL FOR INDUSTRIAL MAINTENANCE
A MONITOR-BASED AR SYSTEM AS A SUPPORT TOOL FOR INDUSTRIAL MAINTENANCE Vesna Nikolic, Peter F. Elzer, Christian Vetter Institute for Process and Production Control Technology Clausthal University of Technology Julius-Albert-Str. 6 D-38678 Clausthal-Zellerfeld Federal Republic of Germany
Abstract: A prototype of a monitor-based augmented reality (AR) system is presented in this paper. The system is completely mobile, consisting of a motorized controllable camera and a laptop with a head set. The human-system interaction is realized by useroriented interfaces and is speech-based. Advantages of the system are its simplicity and low cost combined with proven usability. Compared to other present tools for presentation of instructions for maintenance and operation, the use of this system results in time savings and lower error rates, as shown by experiments undertaken at the IPP. Copyright © 2006 IFAC Keywords: Augmented Reality, Monitor-based, Maintenance, Maintenance and Operating Instructions, Usability, Cognitive Time.
1. BACKGROUND In order to make work more efficient, the source of essential information for technicians in industrial maintenance must fulfil a large number of specific requirements. It has to be mobile and must neither influence safety and health of the personnel, nor the work flow and the work environment. It has to have an ergonomic design and be reliable. It should help to quickly find the information needed. Economical engineering as well as easy update of the tool are necessary. Furthermore, it should support fast input and/or reading of important contextual multimedia notes of the users. Such a system has to record automatically all maintenance activities. Its user-centered design is also very important: the information presented has to be adjusted to the user’s knowledge and/or experience. And, additionally, industry demands low cost and widely applicable solutions.
years (Feiner, et al., 1993; Mizell, 2001), but commercial applications are still rare. Four major problems contribute to this situation: poor ergonomics of head mounted displays, unsatisfactory technical performance of tracking systems, high hardware and engineering costs. Therefore, designing an AR system as a maintenance support tool is a challenging task. The AR technology itself is very promising (Elzer, et al., 2001), especially taking into account the results of several studies about its efficiency (Tang, et al., 2003; Alt, 2003; Wiedenmaier, et al., 2003). Although those studies have quite varying experimental set-ups and address serious problems in the implementation of head mounted displays, they show that using AR in maintenance or assembly leads either to time savings or reduction of error rates (thus improving accuracy), or both, along with the reduction of the cognitive load of its user.
The idea of using AR technology for supporting maintenance or assembly has existed for at least 15
902
with the following parameters: pan -170o to +170o, tilt -90o to +10o, and 16x optical zoom. The system was named CARIHBA, as an abbreviation for the German title: Computer Augmentierte Realität für InstandHaltungs- und BedienungsAnleitungen (= “Computer Augmented Reality for Maintenance and Operating Instructions”). The user-system interaction is speech-based, and is accomplished with commercial software. CARIHBA is completely mobile; the display is an integral part of the laptop while the camera is small and light enough and therefore easy to pack in the technician’s bag together with other maintenance tools. Mounting and starting of the system takes just a few minutes, without any need for calibration, independently of the user. The camera itself has to be calibrated only once in its operational life, at the very beginning of its use. Even with this system the user is partly immersed into the computer generated environment presented on the screen (Fig. 2). Fig. 1. Maintenance technician using the monitorbased AR system as a supporting tool for repair work at a pump. In this paper a simple and low cost monitor-based AR system is presented. It has been developed in the framework of a doctoral thesis at the IPP (Nikolic, 2006). Its functions and main features are described. The experiments conducted in order to assess its effectiveness are presented. In the conclusion some future work and other possible implementations of this system, e.g. in education, are addressed.
The user can observe his hands and his actions on the screen, together with the augmentation. As tests have shown, this contributes to user’s self-assurance, which is very important for trainees and novices, or even for experienced technicians in the case of completely new maintenance environments.
2. THE MONITOR-BASED AR SYSTEM 2.1 General Because of the abovementioned ergonomic and technical problems with such components the development efforts at the IPP were focused on an AR system that did not need a head mounted display or a tracking system. The work resulted in a different AR system, consisting of a laptop with a head set, one or more motorized controllable cameras (depending on the task and the environment) and the AR software (Fig. 1). The camera is to be mounted on a prearranged quick release fastener in the maintenance environment. The camera records the environment and the video picture is presented in real time on the screen of the laptop. Corresponding to pan, tilt and zoom chosen (and of course dependent on user and task), the AR software augments this picture with proper textual or graphical objects. The result is that the user can see his augmented environment on the laptop. He can change the camera view or zoom according to his needs. For the system realization we used a camera
Fig. 2. The immersion effect caused by using monitor-based AR system. The most important advantage of the system is the low cost of its hardware. It comprises only commercially available components so that user acceptance is given and maintenance costs of the system are very low.
903
Finally, during the use of CARIHBA the well known problems of using head mounted displays in common AR systems, like interposition (Drascic and Milgram, 1996), eyes accommodation (Rottenkolber, et al., 2004) and attention tunnelling (Yeh and Wickens, 2000) do not occur. This also holds for the implementation problems of tracking systems, as line of sight, accuracy or environmental depending performance (Behnke, 2005).
The basic system function of environment augmentation contributes to easy understanding of the elements of the facility or the equipment with their spatial relations.
One problem, however, remains: a certain amount of attention switching between the source of the information and the equipment to be maintained - similar to work with paper documentation. 2.2 System functionality Fixation of the camera on the fastener allows a defined mathematical description of the camera environment. The focal point of the camera (and its cardanic suspension) is actually the centre of a spherical coordinate system. Every visible point (i.e. object) in the space around the camera, which can be seen on its video picture, has a well-defined position described by the two angles: pan and tilt. To each object of the reality, various virtual objects can be allocated. The third dimension of the spherical coordinate system is cameras focal length, which corresponds to the chosen zoom. It is defined as the distance between the video picture and the centre of the sphere. Both the centre of the video picture and the centre of the coordinate system describe a half-line that intersects the sphere just at the actual coordinates pan and tilt. Together with the centre of the coordinate system all pixels of the video picture also describe halflines. The transformation between spherical coordinates and screen coordinates (and vice versa) of virtual objects can be accomplished by applying two rotations. 2.3 Modes of operation During the system development process, special attention was paid to the software features concerning its use in industrial maintenance. Three modes of operation with proper interfaces were created, one for each of the three main situations in maintenance: the overview, the scenario and the engineering mode.
Fig. 3. The overview mode interface of CARIHBA. All available information about one particular facility is ordered in a hierarchical structure, which is visualized as a pull down menu with seven levels. The user can select a specific piece of equipment and call for documents and actions about it, which are stored in the system. He can also retrieve the information needed from the plant legacy system - if available. Below the real-time video picture, the interface has a special pan-tilt field, where pan and tilt camera parameters can be changed simultaneously. This field shows the complete camera view and indicates all existing augmentations as white dots, so that the camera can be quickly moved to the points of interest. The chosen camera position can also be changed by using functions of “step back” or “default position”. The overview mode is completed with a system settings panel with a few functions for customizing the visualization of the information. The scenario mode leads the user through a series of steps, helping him to accomplish the chosen maintenance task (Fig. 4).
The overview mode helps the user to explore the environment (Fig. 3). After starting the system and logging in, the user selects his environment and defines the camera position. As a result, he can see his augmented environment on the screen. He can move the camera by means of voice control (i.e. change pan, tilt and zoom parameters) according to his needs and intentions, thus exploring his surroundings. He also gets additional textual information about the actual view, which is placed in a separate interface field.
Fig. 4. The scenario mode interface of CARIHBA.
904
Each step of one specific scenario includes up to four pieces of information: the augmented video picture, a textual task description, and eventually, a warning and/or a hint. The warnings are visualized in textual form in the middle of the video picture, framed by a red line. Hints are placed in a separate field of the interface, to the right of the task text field. Overall orientation in the scenario is supported by means of a status line, which shows both the total number of steps within the task as well as the number of already executed steps. The navigation is realized by means of the words “next” and “back”, which cause the loading of the next or the previous step in the task. A short flashing of the status panel, and a short sound, serve as system feedback. On a separate panel to the right of the video picture, three other useful functions are placed. The first one allows fast browsing through a task. The second one helps the user to save a certain multimedia note for his colleagues or to read their notes, respectively. The third function creates a log file with date and time stamps about the maintenance activity. The software allows for different presentations of augmentations and task texts for different user groups (novices, advanced users and experts). In the scenario mode the camera moves automatically, according to predefined position data for the steps in the scenario. This turned out to be a great help for the user in finding the specific location within the equipment or the facility, because he can simply follow the movement of the camera and resulting changes in the video picture, which give him additional information for orientation. The scenario mode has not been completed yet. Its design is currently under work, basically following the principle of "WYSIWYG". E.g. it shall be possible to edit augmentation texts or action diagrams via drag and drop. The engineering costs depend directly on the quality of such engineering tools. Nevertheless, the most important further development task appears to be technical support for engineering of the system without physical presence on the maintenance site, as usual by preparation of common technical documentation. 3. EXPERIMENTS 3.1 Aim A series of experiments was conducted in order to answer some questions. The first one concerned the smoothness of the maintenance work flow with CARIHBA as a support tool. It was important to observe how body and head movements change during the work with CARIHBA, compared with situations where somebody holds
paper documentation in his hand or looks at electronic documents on a laptop. Secondly, the effectiveness of CARIHBA was a part of the investigation. It was measured in two ways: firstly, as the time needed to execute several maintenance tasks, and secondly as the error rate during this execution. Finally, the personal opinion of the participants about the work with this particular instruction medium was an interesting topic. Therefore, every participant was interviewed after completion of each task and was asked to complete three questionnaires after the entire set of experiments. 3.2 Procedure As a medium for the provision of information during maintenance, CARIHBA differs from standard paper documentation in a number of ways: information is presented on a display; human-system communication is based on voice control; the system has a usercentered interface design with feedback possibilities and at last, the user gets additional self-assuredness by seeing his own hands on the screen, when touching augmented components in the real environment. In order to compare CARIHBA and paper documentation, it was necessary to express and distinguish influences of those characteristics on the efficiency of the maintenance work. Therefore, for the purpose of the experiments, four variants of instructions were designed, all providing the same quantity of presented information, but differing in the manner of its presentation and in interaction possibilities. The first one was called “paper”. Each step within a maintenance task was shown on a separate piece of paper (all numbered and bound) while the instruction itself was provided by means of both augmented (annotated) photo and related text. The second was called “monitor”, differing from the first one in the medium (a TFT monitor placed at the side of the maintenance workplace) and in the voicecontrolled navigation through the task. The third one additionally provided an interactive user-oriented interface, the same one as by already described monitor-based AR system. This third instruction was called “interface”. The fourth was CARIHBA (“carihba”), where instead of augmented photos (as by other three instructions), augmented real-time video pictures were shown. Tasks. Two pilot plants in the laboratory for process automation at IPP were chosen as the experimental environment. The first pilot plant is a model of a desalination water treatment plant. Amongst other equipment, it has five metering pumps, four tanks, vaporization and heating unit and complete process control equipment. The other one is a model of a production line for stamping of work pieces, also full automated.
905
A total of four maintenance tasks on those two pilot plants were formulated. They were designed to represent the majority of common maintenance activities. Therefore the tasks of demounting, inspection, mounting and start-up were defined. Each task included between 17 and 22 procedural steps (a total of 77 steps in all four tasks) and took not more then 8 minutes (in average) to be accomplished.
completion between the support tools (“paper” – P, “monitor” – M, “interface” – I, “carihba” – C). An alpha level of 0.05 (1-tailed) was used for all statistical tests (Table 1). Table 1. Statistical significance of cognitive time differences for various combinations of support tools and all maintenance tasks
Participants. Sixteen male undergraduate students of technical curricula at the Clausthal University of Technology participated in the experiments for a money award. No one had previous experience with AR systems or knowledge or experience concerning the two pilot plants, which had to be maintained. Measurement of effectiveness. All experiments were recorded on video. For each step within the each task, the completion time was measured, but for the evaluation only the cognitive time portion (Towne, 1985) was taken into account. These portions were quantified after the experiments on the basis of the video recordings. The term “cognitive time” was defined as the portion of time for the execution of one step that one person needs to direct her attention to the medium, read and interpret the instruction, to form a hypothesis about what has to be done, to direct her attention to the equipment, to inspect it, to discriminate und to select its parts, to transpose information from the medium to the equipment (Neumann and Majoros, 1998), to recheck the hypothesis, to make the decision to manipulate and for all attention switching underway. Cognitive time does not include time for the manipulation of an object or time to verify the correctness of the manipulation. It was postulated that a medium for provision of information in maintenance can affect only the cognitive time portion. The error rate was observed during the experiments and checked afterwards by analysing the video records. Performance. All participants had a short training of 10 minutes before the experiments, in order to become familiar with all four instruction forms as well as to practice voice control. During the experiments voice control was simulated through a human agent. All subjects completed all four tasks with one of the four support tools. One subject never worked twice on the same task or with the same medium. Thus the learning effect was excluded. 3.3 Results General. The common maintenance work can be executed with CARIHBA as support tool. All participants succeed in doing all assigned tasks. However, the position of the monitor and the position of the camera have to be carefully chosen for each task. Effectiveness. The Mann-U-Whitney test was used to estimate differences in cognitive time needed for task
Compared support tools M-I I-C p-values n.s.1 n.s. n.s. n.s. n.s. n.s. 0.022 n.s. n.s. n.s. n.s. n.s. P-M
Task Demounting Inspection Mounting Start-Up
P-C 0.010 0.010 0.022 0.042
Statistically significant differences for all tasks could only be found for “paper” and “carihba”. Cognitive times were lower with AR support up to 40%, depending on the task. For other combinations of support tools, significance in cognitive time differences for all four tasks could not be clearly identified, although the measurements showed a certain tendency. It can be expected that with a larger number of subjects and more tightly controlled psychological characteristics of the participants (such as the tendency to follow own rules and not the instructions or a certain superficiality), with longer training or work with experts as participants, such trends could be proved. Error rates were lowest with AR support i.e. with “carihba”. As the total error rate of all steps in all tasks was very low (less then 5%) no statistical evaluation was undertaken for error rates. From a total of 61 errors, only 40 were taken for further analysis, because the rest was not related to the design of the support tool (Table 2). Table 2. Number of errors in all maintenance tasks as a function of the support tool
Total errors
P 15
Support tool M I 11 8
C 6
These results correspond to the personal opinion of subjects about the usability of “carihba” as an instruction form. All participants could grade each tool after the experiments. The tools “interface” and “carihba” got considerably better grades than “paper” and “monitor”. On one hand, these results confirmed that CARIHBA is a well designed maintenance support tool. On the other hand, it has to be investigated for which tasks, environments and personal expertise the use of augmented (annotated) photos or the use of augmented real-time video is better suited.
1
n.s. = not significant
906
4. FURTHER WORK Future development of the described monitor-based AR system will be focused on system engineering questions. Solutions that imply high automation of the engineering processes will be preferred. Some investigations will also be undertaken in order to determine a minimum quantity and quality of augmentation information needed for different user groups in maintenance. At last, the IPP has a special interest to apply this AR solution to teaching courses. A very detailed visualized real-time exploration of the available pilot plants should be of great interest for students.
and P. Korhonen (Eds)), pp. 73-80. ACM Press, New York. Towne, D.M. (1985). Cognitive Workload in Fault Diagnosis. Report No. ONR-107; Contract with Engineering Psychology Group, Office of Naval Reasearch, Los Angeles. Behavioral Technology Laboratories, University of Southern California. Wiedenmaier S., O. Oehme, L. Schmidt, and H. Luczak, (2003). Augmented Reality (AR) for Assembly Processes Design and Experimental Evaluation. In: International Journal of Human Computer Interaction, Vol. 16 (3), pp. 497-514. Yeh, M. and C.D. Wickens (2000). Attention and Trust Biases in the Design of Augmented Reality Displays. Technical Report: Aviation Research Lab. University of Illinois, Urbana-Champaign, Savoy, IL.
REFERENCES Alt, T. (2003). Augmented Reality in der Produktion. Herbert Utz Verlag GmbH, München. Behnke, R. (2005). Verbesserung der Positionserkennung bei CAR Systemen durch Kopplung von bild- und trägheitsbasierten Verfahren. Papierflieger Verlag, Clausthal-Zellerfeld. Drascic, D. and P. Milgram (1996). Perceptual Issues in Augmented Reality. SPIE: Stereoscopic Displays and Virtual Reality Systems III, Vol. 2653, pp. 123-134. Elzer, P.F., R. Behnke, and A. Simon (2001). New Techniques for the Support of Maintenance and Training in Technical Systems. In: Proceedings of 8th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of HumanMachine-Systems, Kassel, Germany (G. Johannsen (Ed)), pp. 517-521. VDI Verlag GmbH, Düsseldorf. Feiner, S., B. Macintyre, and D. Seligmann (1993). Knowledge-based Augmented Reality. Communications of the ACM, Vol. 36 (7), pp. 53-62. Mizell, D. (2001). Boeing’s Wire Bundle Assembly Project. In: Fundamentals of Wearable Computers and Augmented Reality (W. Barfield and T. Caudell (Eds)), pp. 447-467. Lawrence Erlbaum Associates Publishers, London. Neumann, U. and A. Majoros (1998). Cognitive, Performance, and System Issues for Augmented Reality Applications in Manufacturing and Maintenance. In: Proceedings of IEEE VRAIS ’98, Atlanta, USA. IEEE Computer Society. Nikolic, V. (2006). Einsatz der Computer Augmented Reality in der Instandhaltung: eine alternative gebrauchstaugliche und kostengünstige Systemlösung. (in press). Rottenkolber, B., M. Edelmann and F. Höller (2004). Gestaltungsempfehlungen für Head Mounted Displays. In: Benutzerzentrierte Gestaltung von Augmented-Reality-Systemen (H. Luczak, L. Schmidt and F. Koller (Eds)). VDI Verlag GmbH, Düsseldorf. Tang, A., C. Owen, F. Biocca and W. Mou (2003). Comparative Effectiveness of Augmented Reality in Object Assembly. In: Proceedings of ACM CHI 2003 (V. Belloti, T. Erickson, G. Cockton,
907
Abstract: A prototype of a monitor-based augmented reality (AR) system is presented in this paper. The system is completely mobile, consisting of a motorized controllable camera and a laptop with a head set. The human-system interaction is realized by useroriented interfaces and is speech-based. Advantages of the system are its simplicity and low cost combined with proven usability. Compared to other present tools for presentation of instructions for maintenance and operation, the use of this system results in time savings and lower error rates, as shown by experiments undertaken at the IPP. Copyright © 2006 IFAC Keywords: Augmented Reality, Monitor-based, Maintenance, Maintenance and Operating Instructions, Usability, Cognitive Time.
1. BACKGROUND In order to make work more efficient, the source of essential information for technicians in industrial maintenance must fulfil a large number of specific requirements. It has to be mobile and must neither influence safety and health of the personnel, nor the work flow and the work environment. It has to have an ergonomic design and be reliable. It should help to quickly find the information needed. Economical engineering as well as easy update of the tool are necessary. Furthermore, it should support fast input and/or reading of important contextual multimedia notes of the users. Such a system has to record automatically all maintenance activities. Its user-centered design is also very important: the information presented has to be adjusted to the user’s knowledge and/or experience. And, additionally, industry demands low cost and widely applicable solutions.
years (Feiner, et al., 1993; Mizell, 2001), but commercial applications are still rare. Four major problems contribute to this situation: poor ergonomics of head mounted displays, unsatisfactory technical performance of tracking systems, high hardware and engineering costs. Therefore, designing an AR system as a maintenance support tool is a challenging task. The AR technology itself is very promising (Elzer, et al., 2001), especially taking into account the results of several studies about its efficiency (Tang, et al., 2003; Alt, 2003; Wiedenmaier, et al., 2003). Although those studies have quite varying experimental set-ups and address serious problems in the implementation of head mounted displays, they show that using AR in maintenance or assembly leads either to time savings or reduction of error rates (thus improving accuracy), or both, along with the reduction of the cognitive load of its user.
The idea of using AR technology for supporting maintenance or assembly has existed for at least 15
902
with the following parameters: pan -170o to +170o, tilt -90o to +10o, and 16x optical zoom. The system was named CARIHBA, as an abbreviation for the German title: Computer Augmentierte Realität für InstandHaltungs- und BedienungsAnleitungen (= “Computer Augmented Reality for Maintenance and Operating Instructions”). The user-system interaction is speech-based, and is accomplished with commercial software. CARIHBA is completely mobile; the display is an integral part of the laptop while the camera is small and light enough and therefore easy to pack in the technician’s bag together with other maintenance tools. Mounting and starting of the system takes just a few minutes, without any need for calibration, independently of the user. The camera itself has to be calibrated only once in its operational life, at the very beginning of its use. Even with this system the user is partly immersed into the computer generated environment presented on the screen (Fig. 2). Fig. 1. Maintenance technician using the monitorbased AR system as a supporting tool for repair work at a pump. In this paper a simple and low cost monitor-based AR system is presented. It has been developed in the framework of a doctoral thesis at the IPP (Nikolic, 2006). Its functions and main features are described. The experiments conducted in order to assess its effectiveness are presented. In the conclusion some future work and other possible implementations of this system, e.g. in education, are addressed.
The user can observe his hands and his actions on the screen, together with the augmentation. As tests have shown, this contributes to user’s self-assurance, which is very important for trainees and novices, or even for experienced technicians in the case of completely new maintenance environments.
2. THE MONITOR-BASED AR SYSTEM 2.1 General Because of the abovementioned ergonomic and technical problems with such components the development efforts at the IPP were focused on an AR system that did not need a head mounted display or a tracking system. The work resulted in a different AR system, consisting of a laptop with a head set, one or more motorized controllable cameras (depending on the task and the environment) and the AR software (Fig. 1). The camera is to be mounted on a prearranged quick release fastener in the maintenance environment. The camera records the environment and the video picture is presented in real time on the screen of the laptop. Corresponding to pan, tilt and zoom chosen (and of course dependent on user and task), the AR software augments this picture with proper textual or graphical objects. The result is that the user can see his augmented environment on the laptop. He can change the camera view or zoom according to his needs. For the system realization we used a camera
Fig. 2. The immersion effect caused by using monitor-based AR system. The most important advantage of the system is the low cost of its hardware. It comprises only commercially available components so that user acceptance is given and maintenance costs of the system are very low.
903
Finally, during the use of CARIHBA the well known problems of using head mounted displays in common AR systems, like interposition (Drascic and Milgram, 1996), eyes accommodation (Rottenkolber, et al., 2004) and attention tunnelling (Yeh and Wickens, 2000) do not occur. This also holds for the implementation problems of tracking systems, as line of sight, accuracy or environmental depending performance (Behnke, 2005).
The basic system function of environment augmentation contributes to easy understanding of the elements of the facility or the equipment with their spatial relations.
One problem, however, remains: a certain amount of attention switching between the source of the information and the equipment to be maintained - similar to work with paper documentation. 2.2 System functionality Fixation of the camera on the fastener allows a defined mathematical description of the camera environment. The focal point of the camera (and its cardanic suspension) is actually the centre of a spherical coordinate system. Every visible point (i.e. object) in the space around the camera, which can be seen on its video picture, has a well-defined position described by the two angles: pan and tilt. To each object of the reality, various virtual objects can be allocated. The third dimension of the spherical coordinate system is cameras focal length, which corresponds to the chosen zoom. It is defined as the distance between the video picture and the centre of the sphere. Both the centre of the video picture and the centre of the coordinate system describe a half-line that intersects the sphere just at the actual coordinates pan and tilt. Together with the centre of the coordinate system all pixels of the video picture also describe halflines. The transformation between spherical coordinates and screen coordinates (and vice versa) of virtual objects can be accomplished by applying two rotations. 2.3 Modes of operation During the system development process, special attention was paid to the software features concerning its use in industrial maintenance. Three modes of operation with proper interfaces were created, one for each of the three main situations in maintenance: the overview, the scenario and the engineering mode.
Fig. 3. The overview mode interface of CARIHBA. All available information about one particular facility is ordered in a hierarchical structure, which is visualized as a pull down menu with seven levels. The user can select a specific piece of equipment and call for documents and actions about it, which are stored in the system. He can also retrieve the information needed from the plant legacy system - if available. Below the real-time video picture, the interface has a special pan-tilt field, where pan and tilt camera parameters can be changed simultaneously. This field shows the complete camera view and indicates all existing augmentations as white dots, so that the camera can be quickly moved to the points of interest. The chosen camera position can also be changed by using functions of “step back” or “default position”. The overview mode is completed with a system settings panel with a few functions for customizing the visualization of the information. The scenario mode leads the user through a series of steps, helping him to accomplish the chosen maintenance task (Fig. 4).
The overview mode helps the user to explore the environment (Fig. 3). After starting the system and logging in, the user selects his environment and defines the camera position. As a result, he can see his augmented environment on the screen. He can move the camera by means of voice control (i.e. change pan, tilt and zoom parameters) according to his needs and intentions, thus exploring his surroundings. He also gets additional textual information about the actual view, which is placed in a separate interface field.
Fig. 4. The scenario mode interface of CARIHBA.
904
Each step of one specific scenario includes up to four pieces of information: the augmented video picture, a textual task description, and eventually, a warning and/or a hint. The warnings are visualized in textual form in the middle of the video picture, framed by a red line. Hints are placed in a separate field of the interface, to the right of the task text field. Overall orientation in the scenario is supported by means of a status line, which shows both the total number of steps within the task as well as the number of already executed steps. The navigation is realized by means of the words “next” and “back”, which cause the loading of the next or the previous step in the task. A short flashing of the status panel, and a short sound, serve as system feedback. On a separate panel to the right of the video picture, three other useful functions are placed. The first one allows fast browsing through a task. The second one helps the user to save a certain multimedia note for his colleagues or to read their notes, respectively. The third function creates a log file with date and time stamps about the maintenance activity. The software allows for different presentations of augmentations and task texts for different user groups (novices, advanced users and experts). In the scenario mode the camera moves automatically, according to predefined position data for the steps in the scenario. This turned out to be a great help for the user in finding the specific location within the equipment or the facility, because he can simply follow the movement of the camera and resulting changes in the video picture, which give him additional information for orientation. The scenario mode has not been completed yet. Its design is currently under work, basically following the principle of "WYSIWYG". E.g. it shall be possible to edit augmentation texts or action diagrams via drag and drop. The engineering costs depend directly on the quality of such engineering tools. Nevertheless, the most important further development task appears to be technical support for engineering of the system without physical presence on the maintenance site, as usual by preparation of common technical documentation. 3. EXPERIMENTS 3.1 Aim A series of experiments was conducted in order to answer some questions. The first one concerned the smoothness of the maintenance work flow with CARIHBA as a support tool. It was important to observe how body and head movements change during the work with CARIHBA, compared with situations where somebody holds
paper documentation in his hand or looks at electronic documents on a laptop. Secondly, the effectiveness of CARIHBA was a part of the investigation. It was measured in two ways: firstly, as the time needed to execute several maintenance tasks, and secondly as the error rate during this execution. Finally, the personal opinion of the participants about the work with this particular instruction medium was an interesting topic. Therefore, every participant was interviewed after completion of each task and was asked to complete three questionnaires after the entire set of experiments. 3.2 Procedure As a medium for the provision of information during maintenance, CARIHBA differs from standard paper documentation in a number of ways: information is presented on a display; human-system communication is based on voice control; the system has a usercentered interface design with feedback possibilities and at last, the user gets additional self-assuredness by seeing his own hands on the screen, when touching augmented components in the real environment. In order to compare CARIHBA and paper documentation, it was necessary to express and distinguish influences of those characteristics on the efficiency of the maintenance work. Therefore, for the purpose of the experiments, four variants of instructions were designed, all providing the same quantity of presented information, but differing in the manner of its presentation and in interaction possibilities. The first one was called “paper”. Each step within a maintenance task was shown on a separate piece of paper (all numbered and bound) while the instruction itself was provided by means of both augmented (annotated) photo and related text. The second was called “monitor”, differing from the first one in the medium (a TFT monitor placed at the side of the maintenance workplace) and in the voicecontrolled navigation through the task. The third one additionally provided an interactive user-oriented interface, the same one as by already described monitor-based AR system. This third instruction was called “interface”. The fourth was CARIHBA (“carihba”), where instead of augmented photos (as by other three instructions), augmented real-time video pictures were shown. Tasks. Two pilot plants in the laboratory for process automation at IPP were chosen as the experimental environment. The first pilot plant is a model of a desalination water treatment plant. Amongst other equipment, it has five metering pumps, four tanks, vaporization and heating unit and complete process control equipment. The other one is a model of a production line for stamping of work pieces, also full automated.
905
A total of four maintenance tasks on those two pilot plants were formulated. They were designed to represent the majority of common maintenance activities. Therefore the tasks of demounting, inspection, mounting and start-up were defined. Each task included between 17 and 22 procedural steps (a total of 77 steps in all four tasks) and took not more then 8 minutes (in average) to be accomplished.
completion between the support tools (“paper” – P, “monitor” – M, “interface” – I, “carihba” – C). An alpha level of 0.05 (1-tailed) was used for all statistical tests (Table 1). Table 1. Statistical significance of cognitive time differences for various combinations of support tools and all maintenance tasks
Participants. Sixteen male undergraduate students of technical curricula at the Clausthal University of Technology participated in the experiments for a money award. No one had previous experience with AR systems or knowledge or experience concerning the two pilot plants, which had to be maintained. Measurement of effectiveness. All experiments were recorded on video. For each step within the each task, the completion time was measured, but for the evaluation only the cognitive time portion (Towne, 1985) was taken into account. These portions were quantified after the experiments on the basis of the video recordings. The term “cognitive time” was defined as the portion of time for the execution of one step that one person needs to direct her attention to the medium, read and interpret the instruction, to form a hypothesis about what has to be done, to direct her attention to the equipment, to inspect it, to discriminate und to select its parts, to transpose information from the medium to the equipment (Neumann and Majoros, 1998), to recheck the hypothesis, to make the decision to manipulate and for all attention switching underway. Cognitive time does not include time for the manipulation of an object or time to verify the correctness of the manipulation. It was postulated that a medium for provision of information in maintenance can affect only the cognitive time portion. The error rate was observed during the experiments and checked afterwards by analysing the video records. Performance. All participants had a short training of 10 minutes before the experiments, in order to become familiar with all four instruction forms as well as to practice voice control. During the experiments voice control was simulated through a human agent. All subjects completed all four tasks with one of the four support tools. One subject never worked twice on the same task or with the same medium. Thus the learning effect was excluded. 3.3 Results General. The common maintenance work can be executed with CARIHBA as support tool. All participants succeed in doing all assigned tasks. However, the position of the monitor and the position of the camera have to be carefully chosen for each task. Effectiveness. The Mann-U-Whitney test was used to estimate differences in cognitive time needed for task
Compared support tools M-I I-C p-values n.s.1 n.s. n.s. n.s. n.s. n.s. 0.022 n.s. n.s. n.s. n.s. n.s. P-M
Task Demounting Inspection Mounting Start-Up
P-C 0.010 0.010 0.022 0.042
Statistically significant differences for all tasks could only be found for “paper” and “carihba”. Cognitive times were lower with AR support up to 40%, depending on the task. For other combinations of support tools, significance in cognitive time differences for all four tasks could not be clearly identified, although the measurements showed a certain tendency. It can be expected that with a larger number of subjects and more tightly controlled psychological characteristics of the participants (such as the tendency to follow own rules and not the instructions or a certain superficiality), with longer training or work with experts as participants, such trends could be proved. Error rates were lowest with AR support i.e. with “carihba”. As the total error rate of all steps in all tasks was very low (less then 5%) no statistical evaluation was undertaken for error rates. From a total of 61 errors, only 40 were taken for further analysis, because the rest was not related to the design of the support tool (Table 2). Table 2. Number of errors in all maintenance tasks as a function of the support tool
Total errors
P 15
Support tool M I 11 8
C 6
These results correspond to the personal opinion of subjects about the usability of “carihba” as an instruction form. All participants could grade each tool after the experiments. The tools “interface” and “carihba” got considerably better grades than “paper” and “monitor”. On one hand, these results confirmed that CARIHBA is a well designed maintenance support tool. On the other hand, it has to be investigated for which tasks, environments and personal expertise the use of augmented (annotated) photos or the use of augmented real-time video is better suited.
1
n.s. = not significant
906
4. FURTHER WORK Future development of the described monitor-based AR system will be focused on system engineering questions. Solutions that imply high automation of the engineering processes will be preferred. Some investigations will also be undertaken in order to determine a minimum quantity and quality of augmentation information needed for different user groups in maintenance. At last, the IPP has a special interest to apply this AR solution to teaching courses. A very detailed visualized real-time exploration of the available pilot plants should be of great interest for students.
and P. Korhonen (Eds)), pp. 73-80. ACM Press, New York. Towne, D.M. (1985). Cognitive Workload in Fault Diagnosis. Report No. ONR-107; Contract with Engineering Psychology Group, Office of Naval Reasearch, Los Angeles. Behavioral Technology Laboratories, University of Southern California. Wiedenmaier S., O. Oehme, L. Schmidt, and H. Luczak, (2003). Augmented Reality (AR) for Assembly Processes Design and Experimental Evaluation. In: International Journal of Human Computer Interaction, Vol. 16 (3), pp. 497-514. Yeh, M. and C.D. Wickens (2000). Attention and Trust Biases in the Design of Augmented Reality Displays. Technical Report: Aviation Research Lab. University of Illinois, Urbana-Champaign, Savoy, IL.
REFERENCES Alt, T. (2003). Augmented Reality in der Produktion. Herbert Utz Verlag GmbH, München. Behnke, R. (2005). Verbesserung der Positionserkennung bei CAR Systemen durch Kopplung von bild- und trägheitsbasierten Verfahren. Papierflieger Verlag, Clausthal-Zellerfeld. Drascic, D. and P. Milgram (1996). Perceptual Issues in Augmented Reality. SPIE: Stereoscopic Displays and Virtual Reality Systems III, Vol. 2653, pp. 123-134. Elzer, P.F., R. Behnke, and A. Simon (2001). New Techniques for the Support of Maintenance and Training in Technical Systems. In: Proceedings of 8th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of HumanMachine-Systems, Kassel, Germany (G. Johannsen (Ed)), pp. 517-521. VDI Verlag GmbH, Düsseldorf. Feiner, S., B. Macintyre, and D. Seligmann (1993). Knowledge-based Augmented Reality. Communications of the ACM, Vol. 36 (7), pp. 53-62. Mizell, D. (2001). Boeing’s Wire Bundle Assembly Project. In: Fundamentals of Wearable Computers and Augmented Reality (W. Barfield and T. Caudell (Eds)), pp. 447-467. Lawrence Erlbaum Associates Publishers, London. Neumann, U. and A. Majoros (1998). Cognitive, Performance, and System Issues for Augmented Reality Applications in Manufacturing and Maintenance. In: Proceedings of IEEE VRAIS ’98, Atlanta, USA. IEEE Computer Society. Nikolic, V. (2006). Einsatz der Computer Augmented Reality in der Instandhaltung: eine alternative gebrauchstaugliche und kostengünstige Systemlösung. (in press). Rottenkolber, B., M. Edelmann and F. Höller (2004). Gestaltungsempfehlungen für Head Mounted Displays. In: Benutzerzentrierte Gestaltung von Augmented-Reality-Systemen (H. Luczak, L. Schmidt and F. Koller (Eds)). VDI Verlag GmbH, Düsseldorf. Tang, A., C. Owen, F. Biocca and W. Mou (2003). Comparative Effectiveness of Augmented Reality in Object Assembly. In: Proceedings of ACM CHI 2003 (V. Belloti, T. Erickson, G. Cockton,
907