- Email: [email protected]
Interdisciplinary Communication and Comprehension in Factory Automation Engineering - A Concept for an Immersive Virtual Environment
Analysis, Design, and Evaluation of Human-Machine Systems 13th IFAC/IFIP/IFORS/IEA Symposium on Aug. IFAC/IFIP/IFORS/IEA 30 - Sept. 2, 2016. Kyoto, Japan on 13th Symposium Available onlineSystems at www.sciencedirect.com Analysis, Design, and Evaluation of Human-Machine Analysis, Design, and Evaluation of Human-Machine Systems Aug. 30 - Sept. 2, 2016. Kyoto, Japan Aug. 30 - Sept. 2, 2016. Kyoto, Japan
ScienceDirect
IFAC-PapersOnLineand 49-19 (2016) 227–232 Interdisciplinary Communication Comprehension in Factory Automation Interdisciplinary Communication and Comprehension in Automation Engineering - A Concept forand an Comprehension Immersive Virtual Environment Interdisciplinary Communication in Factory Factory Automation Engineering A Concept for an Immersive Virtual Environment EngineeringSebastian - A Concept an Immersive Virtual Environment Ulewicz, for Dorothea Pantförder, Birgit Vogel-Heuser
Sebastian Ulewicz, Dorothea Pantförder, Birgit Vogel-Heuser Sebastian Ulewicz, Dorothea Pantförder, Vogel-Heuser Technische Universität München – Institute for Automation andBirgit Information Systems, Munich, Germany (Tel: +49 89 289 16400; e-mail: {ulewicz; pantfoerder; vogel-heuser}@ais.mw.tum.de). Technische Universität München – Institute for Automation and Information Systems, Munich, Germany Technische Universität München – Institute for Automation and Information Systems, Munich, Germany (Tel: +49 89 289 16400; e-mail: {ulewicz; pantfoerder; vogel-heuser}@ais.mw.tum.de). (Tel: +49 89 289 16400; e-mail: {ulewicz; pantfoerder; vogel-heuser}@ais.mw.tum.de). Abstract: In engineering of factory automation systems, understanding and discussing complex systems is important a competitive development process. Understanding complex connections within these Abstract: In for engineering of factory automation systems, understanding and discussing complex systems Abstract: In engineering of factory automation systems, understanding and discussing complex systems systems while working with interdisciplinary teams scattered around the globe can be a very challenging is important for a competitive development process. Understanding complex connections within these is important for a competitive development process. Understanding complex connections within these task. Anwhile intuitive waywith for interdisciplinary comprehension and communication thiscan environment would be systems working teams scattered aroundwithin the globe be a very challenging systems while working with interdisciplinary teams scattered aroundenvironment the globe canfor be gaining a very challenging beneficial. For this reason, we propose a concept for a virtual a quicker task. An intuitive way for comprehension and communication within this environment would be task. An intuitive way for comprehension and communication environment would be understanding complex systems and theira connections, as awithin way forthis presentation and discussion beneficial. Forofthis reason, we propose concept for asa well virtual environment for gaining a quicker beneficial. For this reason, we propose a concept for a virtual environment for gaining a quicker using the concept of virtual reality.and This is connections, approached by usingasan immersive way of visualizing and understanding of complex systems their as well a way for presentation and discussion understanding of complex systems and their connections, as well as a way for presentation and discussion interacting with complex information. The environment itself as well as different ways of immersing in it using the concept of virtual reality. This is approached by using an immersive way of visualizing and using the concept of virtualthereality. Thisofis the approached by an immersive way of visualizing and are presented. Tocomplex highlight properties approach, an using application example ways is given. interacting with information. The environment itself as well as different of immersing in it interacting with complex information. The environment itself as well as different ways of immersing in it are presented. To highlight the properties of the approach, anHosting application example is given. © 2016, IFAC (International Federation of Automatic Control) by Elsevier Ltd. All rights reserved. Keywords: Factory automation, systems engineering, complex systems, communication environments, are presented. To highlight the properties of the approach, an application example is given. human factors, information analysis, man/machine interaction, virtual reality Keywords: Factory automation, systems engineering, complex systems, communication environments, Keywords: Factory automation, systems engineering, complex systems, communication environments, human factors, information analysis, man/machine interaction, virtual reality human factors, information analysis, man/machine interaction, virtual reality This is to instil a mutual understanding of engineering 1. INTRODUCTION artefacts andinstil their aconnections among development teams This is to mutual understanding of engineering This is to from instil a mutual understanding of working engineering 1. INTRODUCTION domainsdevelopment and in During the development process of complex factory stemming artefacts and theirheterogeneous connections among teams 1. INTRODUCTION artefacts and their connections among development teams different locations. automation systems, the involved engineers face severe stemming from heterogeneous domains and working in During the development process of complex factory stemming from heterogeneous domains and working in During the development and process of complex factory problems in comprehension communication. Due to fast different locations. automation systems, the involved engineers face severe different locations. automation systems,andthedynamic involved engineers face severe 2. STATE OF THE ART engineering team structures, engineers problems in cycles comprehension and communication. Due to fast problems in comprehension and communication. Due to fast are confronted withand a vast amount of complex 2. STATE OF THEautomation ART engineering cycles dynamic team structures,information engineers Complex information in factory engineering 2. STATE OF THE ART engineering cycles dynamicinteam structures, engineers which they have to and comprehend aofvery limitedinformation amount of are confronted with a vast amount complex mostly stems from complex connections between elements of are confronted with adevelopers vast amount complex information Complex information in factory automation engineering time. Inthey addition, areaof experts in their specific which have tothecomprehend in very limited amount of Complex information in factory automation engineering engineering documents. These connections often span across which they to comprehend in a very limitedengineering amount of mostly stems from complex connections between elements of domain, e.g.have mechanical construction, electrical time. In addition, the developers are experts in their specific mostly stems from complex connections between elements of different documents within or even between different time. In addition, the but developers are experts in their specific engineering documents. These connections often span across or computer science, lack an understanding of the tasks domain, e.g. mechanical construction, electrical engineering engineering documents. These connections often span across engineeringdocuments domains. Examples this kind of connection domain, e.g. mechanical construction, electricalbyengineering different within orforeven between different performed complex information created or computerand science, but lack an understanding of the the other tasks different documents within or evenofbetween different could be data flow connections offor parts a control program or computer science, but lack an understanding of the tasks engineering domains. Examples this kind of connection domains. While their specific expertise is necessary to create performed and complex information created by the other with engineering domains. Examples forrelating this kind of connection ports of a bus system and mechanical parts performed and complex information created by the other could be data flow connections of parts of a control program adomains. competitive, tech product, theis necessary integrationtoof the could be data flow connections of parts of a control program While high their specific expertise create (sensors and actuators) of the machine translating to a domains. While their specific expertise is necessary to create with ports of a bus system and relating mechanical parts different domains is imperative for reaching a working a competitive, high tech product, the integration of end the with portsmovement. of a bus system and relating mechanicalin parts physical Changes of single elements aresult. competitive, high technecessary product, tothegain integration mutual of the (sensors and actuators) of the machine translating tothisa is differentTherefore, domains isit imperative for reaching aa deep working end (sensors actuators) of the machine translating to a connectionand chain could have effects on other elements different domains is facing imperative for reaching a working end physical movement. Changes of single elements in and this understanding when difficulties conflicts, e.g.mutual when result. Therefore, it is necessary to orgain a deep physical movement. Changes of single elements in this should therefore be have discussed other involved result. Therefore, it making is necessary to gain a This deepsituation mutual connection chain could effects with on other elements and solving problems or design decisions. understanding when facing difficulties or conflicts, e.g. when connection chain could have effects on other elements and stakeholders. understanding when facing difficulties or conflicts, e.g. when therefore be discussed with other involved being difficult to solve, the development teams often should solvingalready problems or making design decisions. This situation should therefore be discussed with other involved solving problems or making design decisions. This situation stakeholders. are located in different locations around the globe. Explaining Systems Engineering approaches the problem of a mutual being already difficult to solve, the development teams often stakeholders. being already difficult to solve, the development design decisions, complex problems goals is teams even often more understanding of complex interconnections within and across are located in different locations aroundorthe globe. Explaining Systems Engineering approaches the problem of a mutual are located in different locations the globe. Explaining difficult being restricted to problems web around conferences Systems Engineering approaches thebyproblem of a mutual the different involved domains, e.g. creating that design decisions, complex or goalsorisphone even calls. more understanding complex interconnections withinmodels and across design decisions,cycles complex problems orshorten goals iswhile evengetting more understanding of As development of these systems of complex interconnections within and across define interfaces and connections between the domains. difficult being restricted to web conferences or phone calls. the different involved domains, e.g. by creating models that difficult being at restricted to time, web conferences or phone calls. more complex the same these problems intensify as the different involved domains, e.g. by creating models that Many ofinterfaces these works production automation As development cycles of these systems shorten while getting define and inconnections between theengineering domains. As development cycles of these systems shorten while getting there is less time to communicate and comprehend. This define and connections between thethedomains. rely onofinterfaces semi-formal modelling languages such as SysML more complex at the same time, these problems intensify as Many these works in production automation engineering more complex at the same time,improved these problems as situation would greatly by aintensify meanThis of Many of these works in production automation engineering (Vogel-Heuser et al., 2014). The main problems of these there is less time be to communicate and comprehend. on semi-formal modelling languages such as the SysML there is less time to communication communicate and This rely comprehension thatcomprehend. allows a more rely onissemi-formal modelling languages such as theand SysML works that theetmodels need to be created manually rely situation would and be greatly improved by a mean of (Vogel-Heuser 2014). The main problems of these situation be greatly improved byconveying a mean and of (Vogel-Heuser et al., natural andwould intuitive of understanding, al., 2014). Theunfamiliar main problems of these on modelling languages initially to all involved comprehension and way communication that allows a more is that the models need to be created manually and rely comprehension and communication that allows a more works discussing is that the shortcomings models need toarebesoftened created manually andthese rely domains. These natural andcomplex intuitiveinformation. way of understanding, conveying and works modelling languages initially unfamiliarbytore-using all involved natural and intuitive way of understanding, conveying and on on modelling languages initially unfamiliar to all involved documents for automatic generation of further artefacts or discussing complex information. These shortcomings are softened by re-using these The concept presented in this paper is to approach the domains. discussing complex information. domains. These shortcomings are softened by re-using these automatic analyses. An example for the former are code documents for automatic generation of further artefacts or problem of a mutual understanding of complex engineering The concept presented in this paper is to approach the documents for automatic generation of further or generation analyses. from design (Schütz et al.,artefacts 2014), the The concept presented in this paper is to approach the Andocuments example for the former are code information virtual environment which supports the automatic problem of athrough mutuala understanding of complex engineering automatic analyses. An example for the formeridentifying are code latter is regarded in works automatically problem of a of mutual understanding complex information. engineering generation from design documents (Schütz et al., 2014), the visualization anda interaction withofcomplex information through virtual environment which supports the generation from(Feldmann design documents (Schütz et al., problem 2014), the inconsistencies et al., 2015). Another is information through a virtual environment which supports the latter is regarded in works automatically identifying visualization of and interaction with complex information. latter is regarded in works automatically identifying visualization of and interaction with complex information. inconsistencies (Feldmann et al., 2015). Another problem is inconsistencies (Feldmann et al., 2015). Another problem is
Copyright © 2016 IFAC 233 2405-8963 © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Copyright 2016 responsibility IFAC 233Control. Peer review©under of International Federation of Automatic Copyright © 2016 IFAC 233 10.1016/j.ifacol.2016.10.529
2016 IFAC/IFIP/IFORS/IEA HMS 228 Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Sebastian Ulewicz et al. / IFAC-PapersOnLine 49-19 (2016) 227–232
that the information, namely complex interconnections within these models, is becoming increasingly hard to handle with a growing system size. The user quickly loses the ability to trace these interconnections across multiple hierarchies and at a macro scale.
projection techniques or even CAVEs (Creagh, 2003) are already being used to a great extent. This is due to the increased sense of presence which in turn facilitates imagining the real product. In addition, modern tools of interaction are used such as head tracking to allow changing the viewing angle in an intuitive way. The information displayed in these visualizations on the other hand quite often does not possess complex interconnections and does not include connections to other domains, which is the focus of our work.
There are ways to visualize such complex interconnections, e.g. directed graphs, specifying connections between elements. Extensive research has been done in the field of visualizing these types of graphs. A good overview over typical ways of visualizing 2D- and 3D-graphs is given by Herman et al. (2000). These approaches focus on displaying the connected information in a suitable way by avoiding occlusions and revealing hierarchies. In many cases, a perfect layout cannot be achieved as there are too many cross connections. In addition, static layouts also tend to get confusing once too many nodes and edges are present. In these cases, interaction with the graphs has proven to be useful. Different techniques, such as structure-based clustering (Roxborough and Sen, 1998) or content-based clustering (Wattenberg, 2006), or specific viewing tools such as the hyperbolic tree (Lamping, 1995) have proven to be useful. Even though these techniques of layouting and interaction are approaching the problem, in many cases the information can get so complex that new types of visualization and interaction should be explored.
3. CONCEPT The concept in this paper can be summarized as a virtual environment in which individual or multiple members of development teams can enter and meet to understand, present, explain and discuss complex information within a development process involving different domains. Different modes of immersing into this environment allow a wide variety of usage scenarios including remote access to this environment. In addition, different modes of visualization as well as interaction with this complex information are presented. 3.1. The Virtual Communication Environment The virtual environment (see Fig. 1) is to be designed in such a way that it is not distracting from the task to be performed, while remaining relatable to real environments to allow for a high degree of involvement and immersion. Thus, in its basic state, it is designed as a non-distracting open infinite plane in which the subject of discussion is located in the middle. The discussion partners can enter this environment and move around on this plane while being displayed as human avatars. The users can view the complex subject and interact with it to gain a deeper understanding of the interconnections.
In many historic works, it was investigated that displaying certain types of connected information in an integrated view can possess positive properties as the user can quickly relate this information in his mind (Wickens and Andre, 1990). In recent works, this approach was investigated for displaying complex information in process monitoring in 3D (Pantförder et al., 2009). It was found that in scenarios that increase the sense of involvement of the operator by allowing interaction with the 3D scene, increased the ability to spot problems more accurately compared to regular modes of displaying information in this field. Recently, virtual reality has been rediscovered by the gaming industry as it increases the sense of presence. Presence is largely influenced by the sense of immersion and involvement, where immersion is the feeling of physically being present in a non-physical environment and involvement is the ability to focus on a task or a subject (Witmer and Singer, 1998). With virtual reality, the subjects are secluded from the world when using a headset and are able to completely immerse into the digital world. Translating head and even body movements to the virtual world can appear to be more real compared to interacting with it using a mouse and keyboard. Thus, the interaction feels more natural. An overview over important properties of the sense of presence in virtual reality is given by Schuemie et al. (2001). Immersion has shown to be useful in a number of scenarios, such as operator training and data set analysis (Raja et al., 2004) as it can reduce information clutter for example, yet not every task profits from a high degree of immersion, e.g. tasks with a low degree of spatial information (Bowman and McMahan, 2007).
Fig. 1. The basic structure of the virtual environment To emphasize the focus on the subject, it is highlighted with intensified lighting in a substantially darker environment. The subject can be viewed from different angles by walking around it. To analyse details of the subject, the user can move closer or view the whole subject by moving further away. This type of interaction possesses many similarities to natural interaction with a subject in real live and therefore increases the intuitiveness of the visualization of the subject and subsequently increases the sense of presence. In case this
In virtual product development and design, modern display techniques, such as stereoscopic large screen displays, 234
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Sebastian Ulewicz et al. / IFAC-PapersOnLine 49-19 (2016) 227–232
229
mode of visualization is not sufficient to gain an appropriate understanding, a flying mode can be activated to have a bird’s eye view.
flow in contrast to colours, which would need an explanation. Through these improvements, increasing the understanding of information is thought to be improved.
The subject can be relating to something real, e.g. a product or a factory, or represent abstract information, e.g. a directed graph showing interconnections between programming entities. In many cases, hybrid views can be beneficial, visualizing abstract information over models relating to real (or to be real) subjects. When designed in such a way that related information (abstract or not) is put in close proximity, as proposed by Wickens (1990), relating the abstract information with the real subject can be facilitated for the user. This can lead to an increased speed during understanding this information or reaching conclusions regarding problems.
In 3D visualizations, interaction is imperative as entities can easily overlap and thus obstruct the view on other parts of the subject. Thus, navigation in this information is the first step of solving this problem. The navigation is designed in such a way that it is close to natural interaction with a subject. As mentioned before, head movements and body movements should be translated to the virtual scene, if possible. If this is not enough or not possible, interaction devices are needed to allow for further navigation, e.g. 3D mice. In many cases, changing the view is not enough, as certain parts of the scene can be completely obstructed. In other cases, relating data can be difficult due to an inadequate layout. In these cases, the second step to interacting with the scene has to be taken: Modification of the data. Here, moving entities in the scene is the most basic modification. Other modification techniques could be scaling or hiding objects. Opening objects to display more detailed information and sub-entities could be a way of representing a switch in hierarchies of structure- or contentbased clusters.
The concept proposed here is focusing on complex information in the form of extensive interconnections. Directed graphs were chosen as the form of display, as they are easily understandable. Fig. 2 shows an example for complex information as present in production automation engineering. The figure shows data flows between program entities within an automated control program. As can be seen, many interconnections overlap and cross, so a simple 2D layouting algorithm fails at making the connections understandable. Even though individual connections could be reconstructed by closer inspection, the macroscopic view of the information stays confusing. Subsequently, we investigated different modes of visualization and interaction for navigating and modifying this information.
For presenting or discussion information, these types of interaction have to be extended even further to allow for easy communication. As immersion into the virtual environment is achieved by a large angle of view, directing the attention of other users is very important. The most basic ways relating to intuitive actions could be pointing and laser pointing, i.e. showing a beam indicating the pointing direction. Highlighting entities of the scene is another way to direct the attention of the discussion partners. A very sophisticated way of quickly presenting relevant information could be a guided view by controlling the perspective of other users or playing back a pre-recorded flight through the data. As becoming apparent from this section, many different ideas of visualization and interaction were regarded for this concept, yet detailed investigation of the individual aspects is still to be done and is part of further research. 3.2. Ways of Immersion
Fig. 2. An example of a complex directed graph
We propose different ways of connecting to the virtual environment to allow versatile usage. Although there are many hybrid forms between the regarded modes of displaying and interacting with the environment in this section, three basic forms of immersion are regarded more closely to emphasize their different properties.
To decrease the time needed for understanding complex information in directed graphs, node and edge properties can be designed in a more intuitive way. For this, we present first ideas that will be investigated in more detail in future research. Node properties can be designed in a more intuitive way through the use of 3D visualization. The sense of immersion is thought to decrease the time for understanding this information even further. Materials or effects relating to real counterparts can emphasize certain properties such as “stone” for “protected” or “hard”, or “fire” for “exceeding a certain property”, e.g. a software entity exceeding a complexity measure index. Connections between nodes can be designed in a more intuitive way as well: instead of using arrows, particle streams can display different types and directions of information flows in integration. Using symbols as particles allows for a quick classification of this type of
Fig. 3. Basic ways of immersing into the virtual environment 235
2016 IFAC/IFIP/IFORS/IEA HMS 230 Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Sebastian Ulewicz et al. / IFAC-PapersOnLine 49-19 (2016) 227–232
The easiest way to connect to the virtual environment is a regular desktop PC. Understandably, the feeling of presence is very low due to the low degree of immersion caused by the small screen, only covering a low degree of the field of view. Even though human interaction is possible when regarding the information together, the small screen size hinders an adequate presentation and detail views while retaining an overview. In addition, the involvement with the scene can be disturbed easily by the work environment. These findings make this mode of connection a mediocre presentation and discussion tool. Yet, PCs are available in most working environments, so no additional investment is needed. Additionally, the PC is a very precise expert tool for editing, making it ideal for modifying or preparing information in the virtual environment.
is comparable to VR glasses, but the screen, including multiple or high end projectors and pixel processors require a high investment. As CAVEs extend this calculation and projection to multiple screens, they require an even higher investment. Table 1. Overview over modes of immersion desktop PC high low
VR wall / CAVE low (very) high
VR glasses
Precision low Immersion and very high comprehension Presentation suitable most not suitable and discussion suitable Price low (very) high low Use for Editing Discussion Understanding As summarized in Table I, the different modes of immersion into the virtual environment possess specific advantages and disadvantages, yet none of these modes are to be disregarded as they can be used for specific tasks. Due to a low investment and high precision regarding modifying data, PCs are a very suitable tool for editing and preparing information for the virtual environment. VR glasses are suitable for comprehending information due to the high degree of immersion and involvement realized by the seclusion from the outside environment and natural interaction. VR walls and CAVEs on the other hand are a promising tool regarding the presentation and discussion of information with multiple people, as human interaction is not hindered and the information can be displayed in high detail and interacted with in a natural manner.
A field of interacting with virtual environments gaining much attention lately is experiencing virtual reality using VR glasses (Oculus Rift, HTC Vive, etc.). In many ways, this type of entering into the virtual environment is the opposite of a desktop PC: Through stereoscopic vision, a wide viewing angle and head tracking, the sense of immersion is maximized. The complete seclusion to the outside environment increases this effect by supporting the involvement with the subject, as outside disturbances are minimized. New ways of hand tracking (e.g. Leap Motion) can be combined with VR glasses to further increase the sense of immersion through natural interaction. Even though stronger PC hardware is required to allow for low latencies and stereoscopic vision, the investment needed is for establishing this mode of interaction with the virtual environment is not a lot higher than buying a second screen and a new graphics card. Yet, there are downsides to this type of device. The displays used in these devices possess a relatively high resolution, yet this resolution is spread out onto the whole field of view resulting in a low perceived image quality. While this fact is sure to change in the future, this is a current limitation when considering this type of device for displaying high definition information. In addition, the complete seclusion also avoids natural interaction with colleagues. It is therefore inadequate for presentation and discussion of information with locally present parties.
3.3. The Technical Back End The requirements on the technical back-end are low reaction times, high refresh rates and the possibility to connect from remote locations. To preserve the sense of presence and minimize the risk of motion sickness, reaction times and refresh rates of visualization and movements or interaction are to be kept low. To allow for discussion of subjects from different locations, the approach needs to support logging into the virtual environment using different modes of immersion, as not every type might be available at every location.
A third mode of immersion are so called VR walls and CAVEs. These large projection screens are using either rear projection (projection from behind the screen) or front projection at a very acute angle. A CAVE goes one step further by positioning these screens in a cube in which the users can step into for an increased sense of immersion. These type of screens usually possess very high resolution per screen (~ 4K) and the possibility for stereoscopic vision and head tracking. This allows for displaying complex information while allowing for quickly regarding details of a subject as well as gaining an overview of the situation. Through the use of specialized head tracking cameras, the field of view can be adjusted in a natural way, allowing an intuitive visualization of the subject. This keeps the sense of presence at a high level, yet not as high as with VR glasses as the complete seclusion from the outside environment is not achieved. Regarding the investment, VR walls are quite expensive at this point. The hardware calculating the images
For this, a common server-client-structure is suitable. Closely comparable to common multiplayer games, a server is used to track and synchronize all user movements and the subject itself. This information is provided and transferred by the server to all clients which use this information to render the scene for their mode of visualization. This allows for a synchronized, low latency environment. On the other side, the client’s hardware has to be powerful enough to achieve low refresh rates and sensory translation to virtual movement. 4. APPLICATION SCENARIOS To exemplify the usage and positive properties of the communication environment, we present a use case spanning 236
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Sebastian Ulewicz et al. / IFAC-PapersOnLine 49-19 (2016) 227–232
231
the deep immersion and seclusion from the outside view, it is possible to quickly grasp the integrated subject, yet details and connections need to be investigated in more detail.
all modes of immersion. In this scenario, information is first created, modified and formatted on a PC, explored using VR glasses and then presented in a virtual environment spanning multiple remote locations. Creating and Editing Complex Information In this part of the example, a small automated production machine is designed in the domains of mechanical, electrical and software engineering. Each domain uses its appropriate tools for creation, such as CAD for the mechanical construction. To allow for an integrated view, a user compiles and prepares the engineering information of all involved domains. This is done by placing an integrated view of the mechanical and electrical construction on the ground, i.e. infinite plane. Relevant connections between the bus system (electrical engineering) and the actuators and sensors (mechanical engineering) are created. Through an analysis of the program, a directed graph of the control program is created, by dividing the individual software modules (nodes) and relating them via their calls and data flows (edges). The relation of the software modules to the bus system is then created, by connecting relevant variables with the bus channels. In addition, certain software modules can be directly related to hardware entities and are subsequently placed in close proximity. The rest of the graph is layouted hierarchically.
Fig. 5. Using VR glasses to naturally comprehend and interact with complex information For this, the user uses interaction techniques using hand tracking for highlighting and moving parts of the graph to his or her own liking (see Fig. 5). This helps the user to understand connections over multiple entities without losing the overview over the whole system. Through this interaction, the user is engaged with the subject in a natural and intense way. This allows for a deep understanding of the integrated information without needing to understand every detail of the individual domains or navigating through different files and programs. For instance, if an electrical engineer has to change a sensor type during engineering, he or she can explore the connections between this part and its connection and dependencies to the other domains to identify possible problems or inconsistencies. He or she can subsequently prepare this information for presentation and discussion with his or her colleagues.
The PC is a very suitable tool for this task as using a mouse and keyboard is very precise and allows for quick and complex interactions. Algorithms can help minimizing manual effort by automatically performing individual tasks, such as creating a dependency graph of the program. Fig. 4 shows a mock-up of the resulting subject of interest which is used in the following sections to explain information exploration and presentation.
Presentation and Discussion of Complex Information In this last part of the example, the user having a deeper understanding of the information is to present it to multiple colleagues working in different locations. This is done by displaying this information in the virtual environment. The user gathers some of his colleagues working in the same place in front of the VR wall to allow for natural human interaction. Through the server client structure, he or she can open the access to colleagues from different work locations. Some colleagues are subsequently joining the virtual environment via remote access. They are displayed as human 3D avatars to quickly grasp their point of view.
Fig. 4. An integrated view of a production system's hardware and abstracted software
The user can initiate the session using a guided presentation, giving a quick overview over the situation. This is done by directing views and movement of all colleagues in the scene by showing his or her perspective. Following the quick overview, the user then directs the attention of his colleagues by pointing out certain parts of the system using the laser point technique (see Fig. 6). He or she also highlights certain connections within the system to point out flaws or critical parts, which quickly starts a discussion about these points.
Comprehension of Complex Information In this part of the example, a user is to quickly understand the integrated automated production system and therefore enters the virtual environment using VR glasses. He or she can easily navigate through the virtual environment by moving around using the keyboard or a 3D mouse. Due to 237
2016 IFAC/IFIP/IFORS/IEA HMS 232 Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Sebastian Ulewicz et al. / IFAC-PapersOnLine 49-19 (2016) 227–232
Through the relatable representation of the scene, all users can quickly grasp the problems and decide accordingly.
Creagh, H. (2003). Cave automatic virtual environment. IEEE Electrical Insulation Conference and Electrical Manufacturing & Coil Winding Technology Conference. Proceedings, 499-504. Feldmann, S., Herzig, S.J.I., Kernschmidt, K., Wolfenstetter, T., Kammerl, D., Qamar, A., Lindemann, U., Krcmar, H., Paredis, C.J.J., Vogel-Heuser, B. (2015). A comparison of inconsistency management approaches using a mechatronic manufacturing system design case study. IEEE International Conference on Automation Science and Engineering (CASE), 158-165. Herman, I., Melançon, G., and Marshall, M. S. (2000). Graph visualization and navigation in information visualization: A survey. Visualization and Computer Graphics, IEEE Transactions on, 6(1), 24-43. Lamping, J., Rao, R., and Pirolli, P. (1995). A focus+ context technique based on hyperbolic geometry for visualizing large hierarchies. Proceedings of the SIGCHI conference on Human factors in computing systems, 401-408. Pantförder, D., Vogel-Heuser, B., and Schweizer, K. (2009). Benefit and evaluation of interactive 3D process data visualization for the presentation of complex problems. In Jacko, J.D. (ed.), Human-Computer Interaction. Novel Interaction Methods and Techniques, 869-878. Springer Berlin Heidelberg. Raja, D., Bowman, D. A., Lucas, J., and North, C. (2004). Exploring the Benefits of Immersion in Abstract Information Visualization. Proceedings of the 8th Immersive Projection Technology Workshop, 61–69. Roxborough, T. and Sen, A. (1997). Graph clustering using multiway ratio cut (Software demonstration). In DiBattista, G. (ed.), Graph Drawing, 291-296. Springer, Berlin Heidelberg. Schuemie, M.J., Van Der Straaten, P., Krijn, M., and Van Der Mast, C.A. (2001). Research on presence in virtual reality: A survey. CyberPsychology & Behavior, 4(2), 183-201. Schütz, D., Legat, C., and Vogel-Heuser, B. (2014). MDE of manufacturing automation software - Integrating SysML and standard development tools. 12th IEEE International Conference on Industrial Informatics (INDIN), 267-273. Vogel-Heuser, B., Schütz, D., Frank, T., and Legat, C. (2014). Model-driven engineering of Manufacturing Automation Software Projects – A SysML-based approach. Mechatronics, 24(7), 883-897. Wattenberg, M. (2006). Visual exploration of multivariate graphs. Proceedings of the SIGCHI conference on Human Factors in computing systems, 811-819. Wickens, C.D. and Andre, A.D. (1990). Proximity Compatibility and Information Display: Effects of Color, Space, and Objectness on Information Integration. Human Factors: The Journal of the Human Factors and Ergonomics Society, 32(1), 61-77. Witmer, B.G. and Singer, M.J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and virtual environments, 7(3), 225-240.
Fig. 6. Using a VR wall to discuss with remote parties To continue the example from the previous section, the electrical engineer identified needed changes regarding software drivers and differently placed mounts for the sensors. He or she can present the issues to his or her colleagues and reach a faster solution, as a better mutual understanding is achieved in the immersive environment. 5. CONCLUSION AND OUTLOOK In this paper we presented a concept for a virtual environment for understanding and discussing complex information in the factory automation domain. We focused on the investigation of complex interconnection within and between engineering documents. First concepts for viewing, presenting and interacting with this environment were presented. Presentation techniques were explained, allowing for intense discussion of problems or design decisions from stakeholders stemming from different engineering domains. Using an application example, the properties of the virtual environment were exemplified. Here, all steps of a creating, preparing, understanding and presenting complex information were highlighted using the example of a small automated production system. Even though this first concept seems very promising in our opinion, many aspects need to be investigated and developed in more detail. Devices and methods of interaction need to be developed in more detail as the cooperative aspect of a virtual environment using different modes of immersion and interaction raises many questions. For creating content for the virtual environment, automatically generating interconnection graphs from engineering documents is to be investigated in more detail. Subsequent improvement of 3D graph generation is another focus of research to allow emphasis of certain aspects, such as hierarchies or clusters. Parallel to these works, an ongoing evaluation of different aspects is to be conducted in order to analyse the properties of the concepts and allow for a subsequent improvement. REFERENCES Bowman, D. A., & McMahan, R. P. (2007). Virtual Reality: How Much Immersion Is Enough? Computer, 40(7), 36– 43. 238
ScienceDirect
IFAC-PapersOnLineand 49-19 (2016) 227–232 Interdisciplinary Communication Comprehension in Factory Automation Interdisciplinary Communication and Comprehension in Automation Engineering - A Concept forand an Comprehension Immersive Virtual Environment Interdisciplinary Communication in Factory Factory Automation Engineering A Concept for an Immersive Virtual Environment EngineeringSebastian - A Concept an Immersive Virtual Environment Ulewicz, for Dorothea Pantförder, Birgit Vogel-Heuser
Sebastian Ulewicz, Dorothea Pantförder, Birgit Vogel-Heuser Sebastian Ulewicz, Dorothea Pantförder, Vogel-Heuser Technische Universität München – Institute for Automation andBirgit Information Systems, Munich, Germany (Tel: +49 89 289 16400; e-mail: {ulewicz; pantfoerder; vogel-heuser}@ais.mw.tum.de). Technische Universität München – Institute for Automation and Information Systems, Munich, Germany Technische Universität München – Institute for Automation and Information Systems, Munich, Germany (Tel: +49 89 289 16400; e-mail: {ulewicz; pantfoerder; vogel-heuser}@ais.mw.tum.de). (Tel: +49 89 289 16400; e-mail: {ulewicz; pantfoerder; vogel-heuser}@ais.mw.tum.de). Abstract: In engineering of factory automation systems, understanding and discussing complex systems is important a competitive development process. Understanding complex connections within these Abstract: In for engineering of factory automation systems, understanding and discussing complex systems Abstract: In engineering of factory automation systems, understanding and discussing complex systems systems while working with interdisciplinary teams scattered around the globe can be a very challenging is important for a competitive development process. Understanding complex connections within these is important for a competitive development process. Understanding complex connections within these task. Anwhile intuitive waywith for interdisciplinary comprehension and communication thiscan environment would be systems working teams scattered aroundwithin the globe be a very challenging systems while working with interdisciplinary teams scattered aroundenvironment the globe canfor be gaining a very challenging beneficial. For this reason, we propose a concept for a virtual a quicker task. An intuitive way for comprehension and communication within this environment would be task. An intuitive way for comprehension and communication environment would be understanding complex systems and theira connections, as awithin way forthis presentation and discussion beneficial. Forofthis reason, we propose concept for asa well virtual environment for gaining a quicker beneficial. For this reason, we propose a concept for a virtual environment for gaining a quicker using the concept of virtual reality.and This is connections, approached by usingasan immersive way of visualizing and understanding of complex systems their as well a way for presentation and discussion understanding of complex systems and their connections, as well as a way for presentation and discussion interacting with complex information. The environment itself as well as different ways of immersing in it using the concept of virtual reality. This is approached by using an immersive way of visualizing and using the concept of virtualthereality. Thisofis the approached by an immersive way of visualizing and are presented. Tocomplex highlight properties approach, an using application example ways is given. interacting with information. The environment itself as well as different of immersing in it interacting with complex information. The environment itself as well as different ways of immersing in it are presented. To highlight the properties of the approach, anHosting application example is given. © 2016, IFAC (International Federation of Automatic Control) by Elsevier Ltd. All rights reserved. Keywords: Factory automation, systems engineering, complex systems, communication environments, are presented. To highlight the properties of the approach, an application example is given. human factors, information analysis, man/machine interaction, virtual reality Keywords: Factory automation, systems engineering, complex systems, communication environments, Keywords: Factory automation, systems engineering, complex systems, communication environments, human factors, information analysis, man/machine interaction, virtual reality human factors, information analysis, man/machine interaction, virtual reality This is to instil a mutual understanding of engineering 1. INTRODUCTION artefacts andinstil their aconnections among development teams This is to mutual understanding of engineering This is to from instil a mutual understanding of working engineering 1. INTRODUCTION domainsdevelopment and in During the development process of complex factory stemming artefacts and theirheterogeneous connections among teams 1. INTRODUCTION artefacts and their connections among development teams different locations. automation systems, the involved engineers face severe stemming from heterogeneous domains and working in During the development process of complex factory stemming from heterogeneous domains and working in During the development and process of complex factory problems in comprehension communication. Due to fast different locations. automation systems, the involved engineers face severe different locations. automation systems,andthedynamic involved engineers face severe 2. STATE OF THE ART engineering team structures, engineers problems in cycles comprehension and communication. Due to fast problems in comprehension and communication. Due to fast are confronted withand a vast amount of complex 2. STATE OF THEautomation ART engineering cycles dynamic team structures,information engineers Complex information in factory engineering 2. STATE OF THE ART engineering cycles dynamicinteam structures, engineers which they have to and comprehend aofvery limitedinformation amount of are confronted with a vast amount complex mostly stems from complex connections between elements of are confronted with adevelopers vast amount complex information Complex information in factory automation engineering time. Inthey addition, areaof experts in their specific which have tothecomprehend in very limited amount of Complex information in factory automation engineering engineering documents. These connections often span across which they to comprehend in a very limitedengineering amount of mostly stems from complex connections between elements of domain, e.g.have mechanical construction, electrical time. In addition, the developers are experts in their specific mostly stems from complex connections between elements of different documents within or even between different time. In addition, the but developers are experts in their specific engineering documents. These connections often span across or computer science, lack an understanding of the tasks domain, e.g. mechanical construction, electrical engineering engineering documents. These connections often span across engineeringdocuments domains. Examples this kind of connection domain, e.g. mechanical construction, electricalbyengineering different within orforeven between different performed complex information created or computerand science, but lack an understanding of the the other tasks different documents within or evenofbetween different could be data flow connections offor parts a control program or computer science, but lack an understanding of the tasks engineering domains. Examples this kind of connection domains. While their specific expertise is necessary to create performed and complex information created by the other with engineering domains. Examples forrelating this kind of connection ports of a bus system and mechanical parts performed and complex information created by the other could be data flow connections of parts of a control program adomains. competitive, tech product, theis necessary integrationtoof the could be data flow connections of parts of a control program While high their specific expertise create (sensors and actuators) of the machine translating to a domains. While their specific expertise is necessary to create with ports of a bus system and relating mechanical parts different domains is imperative for reaching a working a competitive, high tech product, the integration of end the with portsmovement. of a bus system and relating mechanicalin parts physical Changes of single elements aresult. competitive, high technecessary product, tothegain integration mutual of the (sensors and actuators) of the machine translating tothisa is differentTherefore, domains isit imperative for reaching aa deep working end (sensors actuators) of the machine translating to a connectionand chain could have effects on other elements different domains is facing imperative for reaching a working end physical movement. Changes of single elements in and this understanding when difficulties conflicts, e.g.mutual when result. Therefore, it is necessary to orgain a deep physical movement. Changes of single elements in this should therefore be have discussed other involved result. Therefore, it making is necessary to gain a This deepsituation mutual connection chain could effects with on other elements and solving problems or design decisions. understanding when facing difficulties or conflicts, e.g. when connection chain could have effects on other elements and stakeholders. understanding when facing difficulties or conflicts, e.g. when therefore be discussed with other involved being difficult to solve, the development teams often should solvingalready problems or making design decisions. This situation should therefore be discussed with other involved solving problems or making design decisions. This situation stakeholders. are located in different locations around the globe. Explaining Systems Engineering approaches the problem of a mutual being already difficult to solve, the development teams often stakeholders. being already difficult to solve, the development design decisions, complex problems goals is teams even often more understanding of complex interconnections within and across are located in different locations aroundorthe globe. Explaining Systems Engineering approaches the problem of a mutual are located in different locations the globe. Explaining difficult being restricted to problems web around conferences Systems Engineering approaches thebyproblem of a mutual the different involved domains, e.g. creating that design decisions, complex or goalsorisphone even calls. more understanding complex interconnections withinmodels and across design decisions,cycles complex problems orshorten goals iswhile evengetting more understanding of As development of these systems of complex interconnections within and across define interfaces and connections between the domains. difficult being restricted to web conferences or phone calls. the different involved domains, e.g. by creating models that difficult being at restricted to time, web conferences or phone calls. more complex the same these problems intensify as the different involved domains, e.g. by creating models that Many ofinterfaces these works production automation As development cycles of these systems shorten while getting define and inconnections between theengineering domains. As development cycles of these systems shorten while getting there is less time to communicate and comprehend. This define and connections between thethedomains. rely onofinterfaces semi-formal modelling languages such as SysML more complex at the same time, these problems intensify as Many these works in production automation engineering more complex at the same time,improved these problems as situation would greatly by aintensify meanThis of Many of these works in production automation engineering (Vogel-Heuser et al., 2014). The main problems of these there is less time be to communicate and comprehend. on semi-formal modelling languages such as the SysML there is less time to communication communicate and This rely comprehension thatcomprehend. allows a more rely onissemi-formal modelling languages such as theand SysML works that theetmodels need to be created manually rely situation would and be greatly improved by a mean of (Vogel-Heuser 2014). The main problems of these situation be greatly improved byconveying a mean and of (Vogel-Heuser et al., natural andwould intuitive of understanding, al., 2014). Theunfamiliar main problems of these on modelling languages initially to all involved comprehension and way communication that allows a more is that the models need to be created manually and rely comprehension and communication that allows a more works discussing is that the shortcomings models need toarebesoftened created manually andthese rely domains. These natural andcomplex intuitiveinformation. way of understanding, conveying and works modelling languages initially unfamiliarbytore-using all involved natural and intuitive way of understanding, conveying and on on modelling languages initially unfamiliar to all involved documents for automatic generation of further artefacts or discussing complex information. These shortcomings are softened by re-using these The concept presented in this paper is to approach the domains. discussing complex information. domains. These shortcomings are softened by re-using these automatic analyses. An example for the former are code documents for automatic generation of further artefacts or problem of a mutual understanding of complex engineering The concept presented in this paper is to approach the documents for automatic generation of further or generation analyses. from design (Schütz et al.,artefacts 2014), the The concept presented in this paper is to approach the Andocuments example for the former are code information virtual environment which supports the automatic problem of athrough mutuala understanding of complex engineering automatic analyses. An example for the formeridentifying are code latter is regarded in works automatically problem of a of mutual understanding complex information. engineering generation from design documents (Schütz et al., 2014), the visualization anda interaction withofcomplex information through virtual environment which supports the generation from(Feldmann design documents (Schütz et al., problem 2014), the inconsistencies et al., 2015). Another is information through a virtual environment which supports the latter is regarded in works automatically identifying visualization of and interaction with complex information. latter is regarded in works automatically identifying visualization of and interaction with complex information. inconsistencies (Feldmann et al., 2015). Another problem is inconsistencies (Feldmann et al., 2015). Another problem is
Copyright © 2016 IFAC 233 2405-8963 © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Copyright 2016 responsibility IFAC 233Control. Peer review©under of International Federation of Automatic Copyright © 2016 IFAC 233 10.1016/j.ifacol.2016.10.529
2016 IFAC/IFIP/IFORS/IEA HMS 228 Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Sebastian Ulewicz et al. / IFAC-PapersOnLine 49-19 (2016) 227–232
that the information, namely complex interconnections within these models, is becoming increasingly hard to handle with a growing system size. The user quickly loses the ability to trace these interconnections across multiple hierarchies and at a macro scale.
projection techniques or even CAVEs (Creagh, 2003) are already being used to a great extent. This is due to the increased sense of presence which in turn facilitates imagining the real product. In addition, modern tools of interaction are used such as head tracking to allow changing the viewing angle in an intuitive way. The information displayed in these visualizations on the other hand quite often does not possess complex interconnections and does not include connections to other domains, which is the focus of our work.
There are ways to visualize such complex interconnections, e.g. directed graphs, specifying connections between elements. Extensive research has been done in the field of visualizing these types of graphs. A good overview over typical ways of visualizing 2D- and 3D-graphs is given by Herman et al. (2000). These approaches focus on displaying the connected information in a suitable way by avoiding occlusions and revealing hierarchies. In many cases, a perfect layout cannot be achieved as there are too many cross connections. In addition, static layouts also tend to get confusing once too many nodes and edges are present. In these cases, interaction with the graphs has proven to be useful. Different techniques, such as structure-based clustering (Roxborough and Sen, 1998) or content-based clustering (Wattenberg, 2006), or specific viewing tools such as the hyperbolic tree (Lamping, 1995) have proven to be useful. Even though these techniques of layouting and interaction are approaching the problem, in many cases the information can get so complex that new types of visualization and interaction should be explored.
3. CONCEPT The concept in this paper can be summarized as a virtual environment in which individual or multiple members of development teams can enter and meet to understand, present, explain and discuss complex information within a development process involving different domains. Different modes of immersing into this environment allow a wide variety of usage scenarios including remote access to this environment. In addition, different modes of visualization as well as interaction with this complex information are presented. 3.1. The Virtual Communication Environment The virtual environment (see Fig. 1) is to be designed in such a way that it is not distracting from the task to be performed, while remaining relatable to real environments to allow for a high degree of involvement and immersion. Thus, in its basic state, it is designed as a non-distracting open infinite plane in which the subject of discussion is located in the middle. The discussion partners can enter this environment and move around on this plane while being displayed as human avatars. The users can view the complex subject and interact with it to gain a deeper understanding of the interconnections.
In many historic works, it was investigated that displaying certain types of connected information in an integrated view can possess positive properties as the user can quickly relate this information in his mind (Wickens and Andre, 1990). In recent works, this approach was investigated for displaying complex information in process monitoring in 3D (Pantförder et al., 2009). It was found that in scenarios that increase the sense of involvement of the operator by allowing interaction with the 3D scene, increased the ability to spot problems more accurately compared to regular modes of displaying information in this field. Recently, virtual reality has been rediscovered by the gaming industry as it increases the sense of presence. Presence is largely influenced by the sense of immersion and involvement, where immersion is the feeling of physically being present in a non-physical environment and involvement is the ability to focus on a task or a subject (Witmer and Singer, 1998). With virtual reality, the subjects are secluded from the world when using a headset and are able to completely immerse into the digital world. Translating head and even body movements to the virtual world can appear to be more real compared to interacting with it using a mouse and keyboard. Thus, the interaction feels more natural. An overview over important properties of the sense of presence in virtual reality is given by Schuemie et al. (2001). Immersion has shown to be useful in a number of scenarios, such as operator training and data set analysis (Raja et al., 2004) as it can reduce information clutter for example, yet not every task profits from a high degree of immersion, e.g. tasks with a low degree of spatial information (Bowman and McMahan, 2007).
Fig. 1. The basic structure of the virtual environment To emphasize the focus on the subject, it is highlighted with intensified lighting in a substantially darker environment. The subject can be viewed from different angles by walking around it. To analyse details of the subject, the user can move closer or view the whole subject by moving further away. This type of interaction possesses many similarities to natural interaction with a subject in real live and therefore increases the intuitiveness of the visualization of the subject and subsequently increases the sense of presence. In case this
In virtual product development and design, modern display techniques, such as stereoscopic large screen displays, 234
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Sebastian Ulewicz et al. / IFAC-PapersOnLine 49-19 (2016) 227–232
229
mode of visualization is not sufficient to gain an appropriate understanding, a flying mode can be activated to have a bird’s eye view.
flow in contrast to colours, which would need an explanation. Through these improvements, increasing the understanding of information is thought to be improved.
The subject can be relating to something real, e.g. a product or a factory, or represent abstract information, e.g. a directed graph showing interconnections between programming entities. In many cases, hybrid views can be beneficial, visualizing abstract information over models relating to real (or to be real) subjects. When designed in such a way that related information (abstract or not) is put in close proximity, as proposed by Wickens (1990), relating the abstract information with the real subject can be facilitated for the user. This can lead to an increased speed during understanding this information or reaching conclusions regarding problems.
In 3D visualizations, interaction is imperative as entities can easily overlap and thus obstruct the view on other parts of the subject. Thus, navigation in this information is the first step of solving this problem. The navigation is designed in such a way that it is close to natural interaction with a subject. As mentioned before, head movements and body movements should be translated to the virtual scene, if possible. If this is not enough or not possible, interaction devices are needed to allow for further navigation, e.g. 3D mice. In many cases, changing the view is not enough, as certain parts of the scene can be completely obstructed. In other cases, relating data can be difficult due to an inadequate layout. In these cases, the second step to interacting with the scene has to be taken: Modification of the data. Here, moving entities in the scene is the most basic modification. Other modification techniques could be scaling or hiding objects. Opening objects to display more detailed information and sub-entities could be a way of representing a switch in hierarchies of structure- or contentbased clusters.
The concept proposed here is focusing on complex information in the form of extensive interconnections. Directed graphs were chosen as the form of display, as they are easily understandable. Fig. 2 shows an example for complex information as present in production automation engineering. The figure shows data flows between program entities within an automated control program. As can be seen, many interconnections overlap and cross, so a simple 2D layouting algorithm fails at making the connections understandable. Even though individual connections could be reconstructed by closer inspection, the macroscopic view of the information stays confusing. Subsequently, we investigated different modes of visualization and interaction for navigating and modifying this information.
For presenting or discussion information, these types of interaction have to be extended even further to allow for easy communication. As immersion into the virtual environment is achieved by a large angle of view, directing the attention of other users is very important. The most basic ways relating to intuitive actions could be pointing and laser pointing, i.e. showing a beam indicating the pointing direction. Highlighting entities of the scene is another way to direct the attention of the discussion partners. A very sophisticated way of quickly presenting relevant information could be a guided view by controlling the perspective of other users or playing back a pre-recorded flight through the data. As becoming apparent from this section, many different ideas of visualization and interaction were regarded for this concept, yet detailed investigation of the individual aspects is still to be done and is part of further research. 3.2. Ways of Immersion
Fig. 2. An example of a complex directed graph
We propose different ways of connecting to the virtual environment to allow versatile usage. Although there are many hybrid forms between the regarded modes of displaying and interacting with the environment in this section, three basic forms of immersion are regarded more closely to emphasize their different properties.
To decrease the time needed for understanding complex information in directed graphs, node and edge properties can be designed in a more intuitive way. For this, we present first ideas that will be investigated in more detail in future research. Node properties can be designed in a more intuitive way through the use of 3D visualization. The sense of immersion is thought to decrease the time for understanding this information even further. Materials or effects relating to real counterparts can emphasize certain properties such as “stone” for “protected” or “hard”, or “fire” for “exceeding a certain property”, e.g. a software entity exceeding a complexity measure index. Connections between nodes can be designed in a more intuitive way as well: instead of using arrows, particle streams can display different types and directions of information flows in integration. Using symbols as particles allows for a quick classification of this type of
Fig. 3. Basic ways of immersing into the virtual environment 235
2016 IFAC/IFIP/IFORS/IEA HMS 230 Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Sebastian Ulewicz et al. / IFAC-PapersOnLine 49-19 (2016) 227–232
The easiest way to connect to the virtual environment is a regular desktop PC. Understandably, the feeling of presence is very low due to the low degree of immersion caused by the small screen, only covering a low degree of the field of view. Even though human interaction is possible when regarding the information together, the small screen size hinders an adequate presentation and detail views while retaining an overview. In addition, the involvement with the scene can be disturbed easily by the work environment. These findings make this mode of connection a mediocre presentation and discussion tool. Yet, PCs are available in most working environments, so no additional investment is needed. Additionally, the PC is a very precise expert tool for editing, making it ideal for modifying or preparing information in the virtual environment.
is comparable to VR glasses, but the screen, including multiple or high end projectors and pixel processors require a high investment. As CAVEs extend this calculation and projection to multiple screens, they require an even higher investment. Table 1. Overview over modes of immersion desktop PC high low
VR wall / CAVE low (very) high
VR glasses
Precision low Immersion and very high comprehension Presentation suitable most not suitable and discussion suitable Price low (very) high low Use for Editing Discussion Understanding As summarized in Table I, the different modes of immersion into the virtual environment possess specific advantages and disadvantages, yet none of these modes are to be disregarded as they can be used for specific tasks. Due to a low investment and high precision regarding modifying data, PCs are a very suitable tool for editing and preparing information for the virtual environment. VR glasses are suitable for comprehending information due to the high degree of immersion and involvement realized by the seclusion from the outside environment and natural interaction. VR walls and CAVEs on the other hand are a promising tool regarding the presentation and discussion of information with multiple people, as human interaction is not hindered and the information can be displayed in high detail and interacted with in a natural manner.
A field of interacting with virtual environments gaining much attention lately is experiencing virtual reality using VR glasses (Oculus Rift, HTC Vive, etc.). In many ways, this type of entering into the virtual environment is the opposite of a desktop PC: Through stereoscopic vision, a wide viewing angle and head tracking, the sense of immersion is maximized. The complete seclusion to the outside environment increases this effect by supporting the involvement with the subject, as outside disturbances are minimized. New ways of hand tracking (e.g. Leap Motion) can be combined with VR glasses to further increase the sense of immersion through natural interaction. Even though stronger PC hardware is required to allow for low latencies and stereoscopic vision, the investment needed is for establishing this mode of interaction with the virtual environment is not a lot higher than buying a second screen and a new graphics card. Yet, there are downsides to this type of device. The displays used in these devices possess a relatively high resolution, yet this resolution is spread out onto the whole field of view resulting in a low perceived image quality. While this fact is sure to change in the future, this is a current limitation when considering this type of device for displaying high definition information. In addition, the complete seclusion also avoids natural interaction with colleagues. It is therefore inadequate for presentation and discussion of information with locally present parties.
3.3. The Technical Back End The requirements on the technical back-end are low reaction times, high refresh rates and the possibility to connect from remote locations. To preserve the sense of presence and minimize the risk of motion sickness, reaction times and refresh rates of visualization and movements or interaction are to be kept low. To allow for discussion of subjects from different locations, the approach needs to support logging into the virtual environment using different modes of immersion, as not every type might be available at every location.
A third mode of immersion are so called VR walls and CAVEs. These large projection screens are using either rear projection (projection from behind the screen) or front projection at a very acute angle. A CAVE goes one step further by positioning these screens in a cube in which the users can step into for an increased sense of immersion. These type of screens usually possess very high resolution per screen (~ 4K) and the possibility for stereoscopic vision and head tracking. This allows for displaying complex information while allowing for quickly regarding details of a subject as well as gaining an overview of the situation. Through the use of specialized head tracking cameras, the field of view can be adjusted in a natural way, allowing an intuitive visualization of the subject. This keeps the sense of presence at a high level, yet not as high as with VR glasses as the complete seclusion from the outside environment is not achieved. Regarding the investment, VR walls are quite expensive at this point. The hardware calculating the images
For this, a common server-client-structure is suitable. Closely comparable to common multiplayer games, a server is used to track and synchronize all user movements and the subject itself. This information is provided and transferred by the server to all clients which use this information to render the scene for their mode of visualization. This allows for a synchronized, low latency environment. On the other side, the client’s hardware has to be powerful enough to achieve low refresh rates and sensory translation to virtual movement. 4. APPLICATION SCENARIOS To exemplify the usage and positive properties of the communication environment, we present a use case spanning 236
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Sebastian Ulewicz et al. / IFAC-PapersOnLine 49-19 (2016) 227–232
231
the deep immersion and seclusion from the outside view, it is possible to quickly grasp the integrated subject, yet details and connections need to be investigated in more detail.
all modes of immersion. In this scenario, information is first created, modified and formatted on a PC, explored using VR glasses and then presented in a virtual environment spanning multiple remote locations. Creating and Editing Complex Information In this part of the example, a small automated production machine is designed in the domains of mechanical, electrical and software engineering. Each domain uses its appropriate tools for creation, such as CAD for the mechanical construction. To allow for an integrated view, a user compiles and prepares the engineering information of all involved domains. This is done by placing an integrated view of the mechanical and electrical construction on the ground, i.e. infinite plane. Relevant connections between the bus system (electrical engineering) and the actuators and sensors (mechanical engineering) are created. Through an analysis of the program, a directed graph of the control program is created, by dividing the individual software modules (nodes) and relating them via their calls and data flows (edges). The relation of the software modules to the bus system is then created, by connecting relevant variables with the bus channels. In addition, certain software modules can be directly related to hardware entities and are subsequently placed in close proximity. The rest of the graph is layouted hierarchically.
Fig. 5. Using VR glasses to naturally comprehend and interact with complex information For this, the user uses interaction techniques using hand tracking for highlighting and moving parts of the graph to his or her own liking (see Fig. 5). This helps the user to understand connections over multiple entities without losing the overview over the whole system. Through this interaction, the user is engaged with the subject in a natural and intense way. This allows for a deep understanding of the integrated information without needing to understand every detail of the individual domains or navigating through different files and programs. For instance, if an electrical engineer has to change a sensor type during engineering, he or she can explore the connections between this part and its connection and dependencies to the other domains to identify possible problems or inconsistencies. He or she can subsequently prepare this information for presentation and discussion with his or her colleagues.
The PC is a very suitable tool for this task as using a mouse and keyboard is very precise and allows for quick and complex interactions. Algorithms can help minimizing manual effort by automatically performing individual tasks, such as creating a dependency graph of the program. Fig. 4 shows a mock-up of the resulting subject of interest which is used in the following sections to explain information exploration and presentation.
Presentation and Discussion of Complex Information In this last part of the example, the user having a deeper understanding of the information is to present it to multiple colleagues working in different locations. This is done by displaying this information in the virtual environment. The user gathers some of his colleagues working in the same place in front of the VR wall to allow for natural human interaction. Through the server client structure, he or she can open the access to colleagues from different work locations. Some colleagues are subsequently joining the virtual environment via remote access. They are displayed as human 3D avatars to quickly grasp their point of view.
Fig. 4. An integrated view of a production system's hardware and abstracted software
The user can initiate the session using a guided presentation, giving a quick overview over the situation. This is done by directing views and movement of all colleagues in the scene by showing his or her perspective. Following the quick overview, the user then directs the attention of his colleagues by pointing out certain parts of the system using the laser point technique (see Fig. 6). He or she also highlights certain connections within the system to point out flaws or critical parts, which quickly starts a discussion about these points.
Comprehension of Complex Information In this part of the example, a user is to quickly understand the integrated automated production system and therefore enters the virtual environment using VR glasses. He or she can easily navigate through the virtual environment by moving around using the keyboard or a 3D mouse. Due to 237
2016 IFAC/IFIP/IFORS/IEA HMS 232 Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Sebastian Ulewicz et al. / IFAC-PapersOnLine 49-19 (2016) 227–232
Through the relatable representation of the scene, all users can quickly grasp the problems and decide accordingly.
Creagh, H. (2003). Cave automatic virtual environment. IEEE Electrical Insulation Conference and Electrical Manufacturing & Coil Winding Technology Conference. Proceedings, 499-504. Feldmann, S., Herzig, S.J.I., Kernschmidt, K., Wolfenstetter, T., Kammerl, D., Qamar, A., Lindemann, U., Krcmar, H., Paredis, C.J.J., Vogel-Heuser, B. (2015). A comparison of inconsistency management approaches using a mechatronic manufacturing system design case study. IEEE International Conference on Automation Science and Engineering (CASE), 158-165. Herman, I., Melançon, G., and Marshall, M. S. (2000). Graph visualization and navigation in information visualization: A survey. Visualization and Computer Graphics, IEEE Transactions on, 6(1), 24-43. Lamping, J., Rao, R., and Pirolli, P. (1995). A focus+ context technique based on hyperbolic geometry for visualizing large hierarchies. Proceedings of the SIGCHI conference on Human factors in computing systems, 401-408. Pantförder, D., Vogel-Heuser, B., and Schweizer, K. (2009). Benefit and evaluation of interactive 3D process data visualization for the presentation of complex problems. In Jacko, J.D. (ed.), Human-Computer Interaction. Novel Interaction Methods and Techniques, 869-878. Springer Berlin Heidelberg. Raja, D., Bowman, D. A., Lucas, J., and North, C. (2004). Exploring the Benefits of Immersion in Abstract Information Visualization. Proceedings of the 8th Immersive Projection Technology Workshop, 61–69. Roxborough, T. and Sen, A. (1997). Graph clustering using multiway ratio cut (Software demonstration). In DiBattista, G. (ed.), Graph Drawing, 291-296. Springer, Berlin Heidelberg. Schuemie, M.J., Van Der Straaten, P., Krijn, M., and Van Der Mast, C.A. (2001). Research on presence in virtual reality: A survey. CyberPsychology & Behavior, 4(2), 183-201. Schütz, D., Legat, C., and Vogel-Heuser, B. (2014). MDE of manufacturing automation software - Integrating SysML and standard development tools. 12th IEEE International Conference on Industrial Informatics (INDIN), 267-273. Vogel-Heuser, B., Schütz, D., Frank, T., and Legat, C. (2014). Model-driven engineering of Manufacturing Automation Software Projects – A SysML-based approach. Mechatronics, 24(7), 883-897. Wattenberg, M. (2006). Visual exploration of multivariate graphs. Proceedings of the SIGCHI conference on Human Factors in computing systems, 811-819. Wickens, C.D. and Andre, A.D. (1990). Proximity Compatibility and Information Display: Effects of Color, Space, and Objectness on Information Integration. Human Factors: The Journal of the Human Factors and Ergonomics Society, 32(1), 61-77. Witmer, B.G. and Singer, M.J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and virtual environments, 7(3), 225-240.
Fig. 6. Using a VR wall to discuss with remote parties To continue the example from the previous section, the electrical engineer identified needed changes regarding software drivers and differently placed mounts for the sensors. He or she can present the issues to his or her colleagues and reach a faster solution, as a better mutual understanding is achieved in the immersive environment. 5. CONCLUSION AND OUTLOOK In this paper we presented a concept for a virtual environment for understanding and discussing complex information in the factory automation domain. We focused on the investigation of complex interconnection within and between engineering documents. First concepts for viewing, presenting and interacting with this environment were presented. Presentation techniques were explained, allowing for intense discussion of problems or design decisions from stakeholders stemming from different engineering domains. Using an application example, the properties of the virtual environment were exemplified. Here, all steps of a creating, preparing, understanding and presenting complex information were highlighted using the example of a small automated production system. Even though this first concept seems very promising in our opinion, many aspects need to be investigated and developed in more detail. Devices and methods of interaction need to be developed in more detail as the cooperative aspect of a virtual environment using different modes of immersion and interaction raises many questions. For creating content for the virtual environment, automatically generating interconnection graphs from engineering documents is to be investigated in more detail. Subsequent improvement of 3D graph generation is another focus of research to allow emphasis of certain aspects, such as hierarchies or clusters. Parallel to these works, an ongoing evaluation of different aspects is to be conducted in order to analyse the properties of the concepts and allow for a subsequent improvement. REFERENCES Bowman, D. A., & McMahan, R. P. (2007). Virtual Reality: How Much Immersion Is Enough? Computer, 40(7), 36– 43. 238