- Email: [email protected]
Measuring the potential of augmented reality in civil engineering
Advances in Engineering Software 90 (2015) 1–10
Contents lists available at ScienceDirect
Advances in Engineering Software journal homepage: www.elsevier.com/locate/advengsoft
Measuring the potential of augmented reality in civil engineering Sebastjan Meža, Žiga Turk, Matevž Dolenc∗ Chair of Construction Informatics, University of Ljubljana, Faculty of Civil and Geodetic Engineering, Jamova 2, 1000 Ljubljana, Slovenia
a r t i c l e
i n f o
Article history: Received 7 October 2014 Revised 4 February 2015 Accepted 11 June 2015 Available online 26 June 2015 Keywords: Augmented reality Mobile computing Computer integrated design Computer integrated engineering Civil engineering Project documentation Building information modelling BIM
a b s t r a c t Recently building information models have substantially improved the explicit semantic content of design information. Information models are used to integrate the initial phases of project development. On the construction site, however, the designs are still mostly represented as line-based paper drawings or projections on portable displays. A generic technology that can integrate information and situate it in time, place and context is augmented reality. The specific research issues addressed are (1) does augmented reality have a potential use in civil engineering, (2) how big – in comparison to other technologies - is this potential and (3) what are the main barriers to its adoption. The generic research issue was to develop a methodology for evaluation of potentials of technology. A prototype was built. It was tested on a real construction site to evaluate the potential of its use using the action-research method. A set of structured interviews with potential users was then conducted to compare the prototype to conventional presentation methods. Using this methodology it has been found out that augmented reality is expected to be as big a step as the transition from 2D line drawings to photorealistic 3D projections. The main barrier to the adoption is immature core virtual reality technology, conservative nature of construction businesses and size of building information models. © 2015 Elsevier Ltd. All rights reserved.
1. Introduction Tools for designing in construction have evolved through history. Pens, pencils and paper have been replaced with CAD (computer aided design) and BIM (building information modelling) software. Engineers, builders, planners and contractors also use various domain specific software to support their work. A priority for construction informatics research and practice has been to effectively integrate the construction processes using information technology [12]. Adequate standards, e.g. Industry Foundation Classes (IFC), have the potential of solving the problem of interoperability of software and representation of information in designing [15]. While the design phase is largely digitised and increasingly integrated around BIM, for a complete digitalisation of construction industry, structured information models would need to be available on construction site where the information is used to shape material world. However, on the construction site the IT infrastructure is not readily available. Things began to change with the introduction of mobile computing [10]. The field is still evolving. 1.1. Motivation The outputs of construction information processes (designs, plans and schedules) provide the control information for the material ∗
Corresponding author. Tel.: +386 1 4768 606. E-mail address: [email protected] (M. Dolenc).
http://dx.doi.org/10.1016/j.advengsoft.2015.06.005 0965-9978/© 2015 Elsevier Ltd. All rights reserved.
processes in construction [42]. The media to bring the information from the digital models to construction site where it is used to shape physical reality are still 2D documents such as floor plans, cross sections, sketches, etc. The construction site is integrated into the construction process using media and formats that pre-date computers. Situating information and establishing the relation between the real world of the construction site and design information remains the task of humans. In this task they are not assisted much by technology. Relevant information from the model has to be extracted, based on the user’s role in the project, location and time. The graphical representation of this information in 2D must be situated and contextualised with the physical 3D reality for which people rely on their spatial awareness. It is the technologically largely unassisted human mind that is bridging the gap between the real world of the construction site and the virtual world of the information model and is integrating the two. This is what engineers on site have been doing since the introduction of drawn design information centuries ago. The problem at hand is how to assist this process with technology. The hypothesis of our research of augmented reality (AR) was that by using a synthetic environment that enables the integration of 4D building information models into the live picture of real world it is possible to improve the understanding and ease the use of project information. It should be possible to measure this improvement. We claim that such synthetic environment is augmented reality. It is not just feasible, but it is also more effective than the more traditional, well-established presentations on blueprints or on-screen projections.
2
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
The specific questions the paper asks are (a) does AR have potential in structural engineering, (b) how big an improvement this technology is and (c) what are the barriers to its adoption. To answer these questions we had to develop a methodology, which is generic in nature and applicable to other technology related research in the interdisciplinary area of structural engineering and computer science. 1.2. On research methodology A lot of research in engineering in general and in construction informatics in particular does not analyse existing phenomena, as is the case in natural sciences. Rather, it synthesises (creates) new solutions and improves technologies. Establishing success or failure of this kind of research is methodologically difficult for at least two reasons. First, whatever the technology is doing to assist in the construction process has already been done without that technology for decades or centuries. So naturally it can be done with some additional help of new technology as well. It hardly can fail. In Karl Popper’s terms [17] the refutability of such research is questionable. Second, researchers can create prototypes that prove an idea but do not have the resources to create commercial grade software. Research prototypes lack robustness, friendliness and usability of commercial systems created by hundreds or perhaps thousands of programmers. Measuring the success of crude research prototypes does not do justice to the potential of the technology. In our research this problem was addressed by an innovative combination of several evaluation methods, both theoretical as well as empirical. Theoretical foundations are set on phenomenology. It provides philosophical basis for the hypothesis that augmented reality has a potential. A prototype was built. It was tested and studied in real life settings to assess the current technological capabilities and limitations. Finally the usefulness of the developed prototype was examined by conducting structured interviews with potential end-users. 1.3. Paper structure In the introduction the research context, goals, method and the hypothesis were defined. Section 2 presents the related work. Section 3 continues with the description of the design and implementation of the prototype. Section 4 presents the theoretical selfevaluation of the developed prototype and the preliminary field tests. Those results were used as input data for the empirical part of the research – the survey – that is presented in Section 5. In the conclusions the results are analysed and the hypothesis is revisited. 2. Background and related work In this section augmented reality is discussed from theoretical, technological and practical points of view. It explains our understanding of the building information modelling and its relation to construction project documentation. As construction can be understood as the materialisation – physical realisation of the project documentation [26,31,41], a philosophical discussion on relations among human mind, virtual and real environment is provided.
The other extreme of that continuum is a virtual environment, which allows engineers and designers to design objects in imagined, virtual, and designed, but not yet materialised world. Augmented reality is therefore the middle segment of continuum where virtual elements are added to real world [29]. Building information models are actually representations of designed reality, aimed at improving the perception of ideas and exchange information [41]. By advanced rendering of the models and surroundings, the user is offered a greater amount of information. This can reduce the possibility of misinterpretation of the designs [3]. With technological progress greater degree of realism can be achieved but according to this theory the perfect model cannot be established since reality presents a limit that can be approached but never reached [36]. Augmented reality on the other hand offers a different approach. It takes the live picture of real surroundings as a base to which virtual elements are added. With that in a given point in time and space the user’s interpretation of proposed digital solutions can be easier [14]. There are many terms used to define the segment of continuum between the extremes of the real and the virtual: mixed reality, amplified reality, augmented reality, mediated reality, diminished reality, augmented virtuality, virtualised reality [22]. In this paper the term, augmented reality (AR) is used since our application augments the insight to the real situation for the user. 2.2. Building information modelling/model By definition the building information modelling is a process of digital designing, which results in some form of a building information model (BIM). Ideally it should include all the data needed for the construction instead of the data being scattered throughout numerous drawings, folders, tables, reports, documents, etc. [9]. The basic premise of building information modelling is to enable frictionless collaboration of different actors (professions) at various stages of a building life cycle, integrated around a shared model. The actors may enter, retrieve, update or adopt the information in BIM and with that justify their roles as the participants in the construction process [6,24]. The 4D BIM is a model that includes the temporal properties. The 5th and 6th dimensions of BIM sometimes denote cost and facility management [32]. 2.3. Project documentation Formally speaking, the project documentation that is required by Slovenian legislation is defined by the Construction Act [40]. It should consist of the conceptual design, preliminary design, basic design, detailed design, and as-built design. In this paper the term “project documentation” is used broader than formally defined by Slovenia legislation. The term project documentation is used to denote a set of all documents needed for the construction. It includes all information contained in building information models. 2.4. Intersection between conscious real and virtual
2.1. Augmented reality Although virtual and real environments are two completely different entities it is practically impossible to make a clear boundary between them. They can be better presented with two poles of continuum [25], the real and the virtual. The virtual environment must be completely predefined since computers cannot make their own assumptions [16]. The real is a complex mixture of natural events and items that exist in one of the pole of the continuum. Reality, therefore, includes all that can be created, built, planned, observed, understood etc.
The role of augmented reality can theoretically be explained in the context of the meaning triangle in Fig. 1 [31]. The concept is an idea in the mind that refers to that specific referent (real world object). The symbol is a visual or audible signal symbolising the idea about that referent. The presented example shows that it is possible to establish a direct relation between referent-reference and reference-symbol (Fig. 1). The first is called referencing and the second modelling. The relation between the symbol and the object is more complicated as both exist outside the mind of the human. However, one could say
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
Fig. 1. The meaning triangle.
that construction is a process in which symbolic design representations (SYMBOL) are translated into real world buildings (OBJECT). Unless robots do this, human interpretation of symbols is essential. Augmented reality assists in this interpretation because it places the symbols over the picture of the real world. It is a superior technology to 2D plans and projections and virtual reality because these technologies keep the symbolic and the real apart with the human mind acting as the interface between the two. 2.5. Related work As early as 1996 Webster developed a working prototype called “Architectural anatomy” [2]. Application allowed a user to “see” structural rebar that obviously cannot be seen by a naked eye since it is surrounded by concrete. The result of the project MARS (Mobile Augmented Reality System) was a system that allowed a user to move freely in an open space and see virtual buildings added to the user perspective [33]. A significant amount of research effort was devoted to field of AR in the late 90s and in the beginning of last decade but due to the restrictions of available information technologies at that time (e.g. processing power, connectivity, size, etc.) the developed prototypes had limited practical value. Recent advancements in the field of mobile computing in conjunction with building information modelling have opened new opportunities for AR research and development. Today the most advanced mobile devices meet the minimum requirements of machine performance and portability needed for AR [39]. Therefore on the substantive side, the focus of research partly shifted to a critical assessment
3
of the usefulness of systems and not the practical feasibility of creating AR [1,23,27,34]. In recent years, the focus of the AR research is in the visual occlusion handling [4,13] as well as on improved tracking accuracy [19]. Tracking includes real-time calculation of both the location and the orientation of the viewpoint. It can be done with (1) reference images or with (2) sensors integrated in mobile devices, e.g. GPS, compass, gyroscope. Our solution [30] uses sensors, as presented use cases do not foresee extensive pre-preparations of the real construction site surroundings. On the other hand, the few existing commercial applications such as Bimar [7] and Augment [5] mainly rely on reference images, which makes the two approaches very different. The presented prototype was built on top of the existing components, which is the only feasible approach for academic environment. Therefore the research mainly deals with construction specific issues such as integration of BIM and AR [38] rather than innovation of AR display technology. 3. Prototype This Section first focuses on the information flows in the prototype. The IDEF0 method was chosen as a means of schematic abstraction since it allows an effortless presentation of the system activities, relations among them and external mechanisms by which they are influenced. Our system consists of four activities (Fig. 2). The first activity in the system is building information modelling (A0) which produces the building information model (BIM). Depending on the state of the project this may be a 3D model. Or in later stages the model may include the time component which results in the so-called 4D BIM. When modelling, variety of tools compatible with the IFC STEP standard can be used. The output of the first step is an IFC model that is used in all-subsequent steps. Normally the exchange of BIM between various software vendors takes place in the form of IFC files. Such an exchange can be carried out online using various servers; one of such is the open source BIM server [8] that has also been used in our prototype. BIM server in conjunction with file convertors is used in the second activity (A1) to prepare the L3D model, which can then be displayed in the generic AR application Layar [20]. This activity also depends on the stage of construction project. In case of construction project monitoring, the transformation is somewhat more complex since the model has to be modified so that it is compliant with the schedule. Activities A2 and A3 are similar. In both, the AR application Layar is used to display the L3D model. The difference is in the purpose of use. In case of 3D visualisation (A2), the purpose is to
Fig. 2. IDEF0 diagram of prototype functioning.
4
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
Fig. 3. The initial model.
determine whether the model in reality looks as it was imagined and designed in the virtual environment. The purpose of the activity A3 is to see whether the situation at the site corresponds to the one predicted in the schedule (Fig. 2). Our system can be used in two ways; (1) it can be used for the visualisation of 3D models in the phase of urban planning – A2 and (2) verifying whether the construction work in the field is being carried out in accordance with the schedule – A3. Practical demonstration of the two cases is shown later in this paper (Figs. 5d and 9d). 3.1. Practical implementation Structurally the system consists of a server and a mobile client component. A prototype system has been developed by the method of component-based software engineering (CBSE). CBSE is becoming increasingly popular method since it is time and cost efficient [37]. The CBSE is based on the idea that the most appropriate components are identified and assembled into a new complex system [11]. The server part of our system is dedicated to storing and managing data. It includes a BIM server, FTP server and web service. The BIM server provides effective information exchange between commercial programs compatible with the IFC standard [8]. The FTP server was used as an intermediate location where the models are temporary stored prior to being displayed with a generic AR display component. The web service is preparing a model suitable for display in a generic AR environment. The synchronisation between BIM and FTP servers and all the necessary transformations is automated. The mobile part is devoted to the presentation of BIM models on site. It consists of the custom made Android application and a generic AR display unit. The mobile application serves as a mean of communication between the user and servers. It is used as a schedule display unit and the user also has an insight into revisions of construction projects. There is a multitude of generic AR display apps e.g. Junaio, Layar, Metaio, MixAre, and many others, which can merge digital information and the live picture of real surroundings. Applications running on the mobile devices, mainly operating on the same principle, constantly monitor the user’s position and orientation in space. When a user reaches a zone from which the point of interest (POI) can be viewed, it is overlaid over the live picture of real surroundings [18]. In our case the POI is a 3D model, which was located on the FTP server. 4. Testing of the prototype The developed system can be used in two ways. First, for a presentation of the entire 3D model. This method is mainly intended to
visualise construction projects in the phase of urban planning. Second, in a 4D BIM scenario, which involves the verification of whether the construction is progressing according to the schedule. Testing was conducted in two steps. First, prototype self-evaluation was conducted. This was then followed by the field tests. 4.1. Reliable tests of the prototype In our system Layar is used as the AR display unit. Layar was chosen as it allows the creation of 3D model on the fly. The Layar L3D model converter – command line version – requires model in the format of OBJ/MTL to produce the L3D model suitable for display in the AR environment [20]. When creating 3D models, care should be taken of their size. L3D models should not contain more than 10,000 faces. In the prototype, the L3D model is generated automatically using IFC – OBJ/MTL and OBJ/MTL – L3D transformers. Mappings from IFC to L3D are performed sequentially. Autodesk Revit was used to create the architectural IFC model (Fig. 3). It consisted of 423 elements and its size was 6.7 MB. As the basis, a sample project that comes with the program was taken. Due to few IFC compatibility issues in Revit, some material properties had to be set manually to obtain the L3D model that accurately reflects the original one (Fig. 4). The experiment was repeated on the model of a real construction project of a multi-residential building “ECO silver house”. The IFC model of the load bearing structure used in the experiment consisted of 984 elements and its size was 2.34 MB (Fig. 5). In this set of tests our system performed as expected. As it turned out, the integration of 3D IFC model is more problematic than the actual display of the AR model. Especially time consuming is the process of determining the time dimension (schedule) of each element of the IFC model. Of course, the end result is restricted with the operation of generic AR display unit. It does not address the problem of visual occlusion and the imprecision of global positioning systems [30]. 4.2. Field tests The field tests were performed in order to test the reliability of our system on site and to obtain visual material for the final end user survey. The result of the AR visualisation of the 3D model is shown in (Fig. 6d). Most of inconveniences with which we were faced can be associated to the operation of the generic AR display component. Those can be further related to the inaccuracy of the mobile device’s sensors. In this test the progress of the construction of a real construction project in Ljubljana - “ECO silver house (Fig. 12) was monitored.
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
5
Fig. 4. The L3D model.
Fig. 5. The 4D model.
The prototype was used as an alternative method of time control of construction progress. The site was completely surrounded by the existing buildings and fences; therefore some views had the problem of the visual occlusion. 5. The survey In this section the results of a survey among potential users are presented. It was conducted, based on the experiences with the prototype. We wanted to gain an insight into the thinking of the AEC experts on the applicability of our AR solution and AR in general. The survey was conducted in the following steps. (1) Definition of starting points and formation of the research questions. (2) Definition of the research population and sampling. (3) Preparation of a short presentation of augmented reality. (4) Preparation of a questionnaire. (5) Preliminary substantive test of the questionnaire. (6) Execution of the structured interviews. (7) Analysis of the results. 5.1. The baseline The main goal of our research was to compare our AR system with the conventional presentation techniques and with that obtain an
empirical proof that augmented reality is not just feasible but also useful. When selecting the method, the following was considered. (1) The method had to allow analytical comparison of the established presentation techniques with AR. (2) Because AR was relatively unknown to the target audience; it was necessary to choose a method, which allowed a clear presentation of the topic. (3) The resulting data should allow subsequent quantitative analysis, therefore the answers to key questions had to be predefined [35]. In addition to the real world testing, questionnaires and interviews are most suitable methods of AR system evaluation. It is possible to efficiently collect subjective data regarding the users’ opinions and preferences without the need of specialised hardware and software equipment. We decided to preform structured interviews. The main advantage of the structured interview is the possibility of personal presentation of the topics covered and the opportunity to further clarify possible doubts with individual questions [21]. Prior to the start of each interview a short presentation of the topic was given. The introduction was aimed at excluding the risks associated with the terminology. Individual interviewees were acquainted with the meaning of the terms augmented reality, BIM and project documentation. The essential requirements for the existence of AR and explained the difference between the virtual and augmented
6
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
Fig. 6. Comparison of presentation methods - visualisation of preliminary studio.
reality were summarised. The potential advantages of AR were not exposed, as this could negatively affect the objectivity of the responses. The interviews were conducted in two steps. First the understanding of the interview questions and the multimedia presentation used for the interview as well as the expected length of the interview was tested on a small test group of four people. The findings of the pilot phase helped us in preparing the final questionnaire. The final questionnaire used in the interviews consisted of two parts. The first part included general demographic questions (gender, age, field of work, etc.). The second and main part of the questionnaire consisted of 11 questions: nine of them had pre-defined multiple choice answers and two in which the interviewees ware asked to comment their decisions. The interviews were carried out in January 2014 with representatives of AEC profession. Twenty persons from different Slovenian AEC companies were invited; fifteen of them accepted the invitation. Responses were obtained from all main target groups, which included architects and civil engineers. Each interview was carried out separately in duration between 20 and 55 min. In all the interviews questionnaires have been fully completed. 5.2. The results The first task in the interview was to compare four presentation techniques of project documentation: (1) 2D drawings, (2) 3D BIM on a computer, (3) 3D BIM on a tablet and (4) augmented reality. The participants were asked to compare those techniques and rate them with 1 (worst) to 10 (best). They had to evaluate understandability and usability of project documentation in two use cases discussed in Section 4. The visualisation of preliminary design and monitoring control of a construction process were evaluated. 5.2.1. Visualisation of preliminary design Interviewees were asked to rate the understandability and usability of presentational methods based on the Fig. 6. In addition to images, a demonstration video of system field tests was also shown to the interviewees.
The average ratings are presented in the following charts (Figs. 7 and 8). Separate values are presented for engineers and architects. In both cases augmented reality has achieved the highest score. The comparison of the understandability and the usability among engineers and architects shows that the engineers rated AR better. The result of the statistical analysis is consistent with the concluding comments of the survey. All architects included in the research emphasised the advantages of 2D-mode design, especially in the later stages of the construction projects. On the other hand they all saw a huge unexploited potential of AR, especially when communicating with clients, who usually lack the skills of understanding engineering drawings. 5.2.2. Monitoring of a construction progress Also in this use case interviewees were asked to compare the understandability and usability of four different presentation techniques (Fig. 9). They were offered a Gantt chart, 4D simulation of the construction, simulation on tablet PC and the idealised view of the project documentation. When inquiring about the understandability and usability in this use case we got the same ranking as in the previous use case. In both cases the best result was achieved by augmented reality, followed by the virtual simulations on the PC, 2D and finally simulations on tablets. On average ratings were slightly lower than in the first use case (Figs. 10 and 11). The comparison of the understandability and the usability among engineers and architects shows that the architects rated AR better than engineers. The comparison of 2D and 3D shows that engineers prefer Gantt charts over 4D simulations – just the opposite to the architects. 5.2.3. Assessment of the prototype The second task of the survey was to assess the performance and usability of the prototype. Interviewees were given the following task: “Imagine that you are on a construction site. Your task is to evaluate whether the work is carried out in accordance with the
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
Fig. 7. Understandability of project documentation in the visualisation preliminary studies.
Fig. 8. Usability of project documentation in the visualisation preliminary studies.
Fig. 9. Comparison of presentation methods for construction process monitoring.
7
8
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
Fig. 10. Understandability of project documentation in monitoring of construction.
Fig. 11. Usability of project documentation in monitoring of construction.
Fig. 12. Screenshot of 4D BIM AR.
planned schedule. You have a smart phone, which allows you to view real surroundings with the addition of virtual elements. The green colour indicates the item that is currently on schedule.” After the interviewees were provided with the instructions, they were shown the screenshot recorded on a real construction side (Fig. 12) and asked them the following question. Is it possible to determine whether the construction of the building is carried out in accordance with the schedule? More than 80% of the respondents agreed that it is possible to see that. Those who have answered yes were posed an additional question. “What is the state of the construction site? Is the construction on schedule; is it behind or ahead of schedule?” All answered correctly that the work is slightly behind schedule. Then they were asked to compare the realistic (Fig. 12) with an idealised (Fig. 13) view of augmented reality, to expose the most disturbing factors in real view and suggest what to add to the image that the application would be better. They were proposed three most
Fig. 13. Idealised view of AR.
obvious factors: (1) virtual model is not completely aligned with the surrounding area, (2) the construction site fence and the other elements located between the observation point and the building, (3) small size and resolution. The analysis of the responses indicated that the elements that obstruct the field view are the most disturbing. In addition to this visual occlusion problem the participants were most concerned with the amount of details contained in the augmented reality view. In the presented case only the element that is currently on schedule was marked. The detail about which task e.g. formworks, rebar, concreting, should be currently preformed was not provided. The respondents agreed that the current state of detail is sufficient for a rough assessment, mainly if the investor wants to quickly verify that the work is carried out in accordance with the schedule. 5.2.4. Concluding part of the survey The aim of this part is the assessment of the untapped potential of augmented reality in the AEC context. The questions of interest
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
9
Fig. 14. Potentially the most promising areas for the application of augmented reality.
were: (1) what the main advantages of augmented reality compared to virtual reality were, (2) in which phase of the construction project augmented reality could be most beneficial, (3) which participants of the architecture, engineering and construction sector could gain the most with AR and (4) what it would take to replace 2D drafts with AR. The interviewees were asked to compare the virtual reality and augmented reality and evaluate some features that are not available in virtual reality. Mainly all the augmented reality features were positively evaluated. The possibility to see the virtual models in real time on site from different perspectives gained the top rating. The possibility of free movement within the real space, without the use of the user interfaces, received the most concerns. Some interesting comments at this point were related to the inherent danger of construction site environment and risks posed by augmented reality equipment: - “Especially when one would use wearable mobile devices, e.g. glasses, special care should be taken, while in case of device failure this method of display could be dangerous.” - “I think that overlaying a part of the user’s field of view could be dangerous, especially with the use of wearable devices.” - “Most likely it would take quite some time to get used to the new user interface. I think that it could be dangerous if the user would have suddenly got too many information and thus forget about the real surrounding environment.” When asking about most promising areas for AR the topic classification from Topping [28] was used. Interviewees were asked to evaluate the AR potential of individual tasks with (1) very useful, (2) useful, (3) partly useful and (4) not useful. The analysis of the results showed that AR would be most useful in identifying and locating the existing building component locations and in the control of the compliance of the design and the actual building. Our first use case visualisation of 3D models has proven to be very promising, on the other hand time control – schedule compliance gained a slightly lower rating (Fig. 14). Some of the concluding remarks and comments of the survey: - “Construction is a specific, very traditional and conservative industry in which new technologies are very difficult to enforce. In any case, investing money in new technologies makes sense, especially on the long run. I believe that it will take a generation of engineers to fully adopt 3D principle of design and construction.” - “I believe that the technology is already available, especially for visualisation of 3D models in the field. Architects could presumably already take advantage of such systems. The problem lies in the conservatism of the construction industry. Constructors must first fully adopt 3D building information modelling, only then we can start to
discuss the monitoring of construction process progress with augmented reality.” - “The project documentation in paper form could probably be abandoned in earlier phases of construction projects. I believe that AR would be most beneficial in renovations and in the supervision of the compliance of the completed projects. But first it should be made obligatory to create PID at the handover of buildings in form of building information modelling.” - “In practice, drawing board and pencil remain the primary means of communication. As it allows us - architects to most easily document our thoughts. This is the case in the early design phase. CAD and building information modelling are most beneficial in later design stages. However, I believe that AR could be used as a communication bridge among the profession and general public.” 6. Conclusions and discussion Our research confirmed that augmented reality can significantly contribute to the understanding of project documentation in various stages of construction projects and is thus better integrating the construction phase with earlier stages of development. The claim of significance is based on the comparison of well-established conventional presentation techniques with augmented reality, both theoretically as well as empirically with the survey among potential users. The understandability and usability of project documentation was compared using the following techniques: (1) 2D plans, (2) BIM on a PC, (3) the use of tablet computers and (4) augmented reality. The techniques were compared with each other and quantitatively evaluated in the two use cases: (1) the visualisation of preliminary design and (2) the monitoring of construction progress. The comparison showed that augmented reality is at least one grade better than any other presentation technique. The cumulative of responses showed that the 3D mode is approximately 7% better than 2D, while AR could improve 3D up to 20%; however, only when taking into account certain assumptions. With that the paper’s hypothesis was essentially confirmed but it is necessary to discuss the conditions and assumptions. Augmented reality can facilitate the understanding of project documentation especially in the visualisation of 3D models in the field, but remains technologically constrained. Generally speaking the function of the prototype – visualisation of preliminary design – is more useful as the function of construction schedule supervision. The experts from the field of architecture, engineering and construction have assessed that the current functioning of the prototype presented in this paper can be most useful with the process of communication between the experts and the investors. It is also
10
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
possible to conclude that the presented prototype allows us to see and estimate whether the work on site is conducted in accordance with the expectations defined in the schedule. The quality of display is sufficient, although the most disturbing factor is the visual occlusion. It can be concluded that although augmented reality has a substantial potential it is unlikely that in the nearby future it could replace the conventional presentation techniques. The main barriers were found to be (a) GPS positioning in general and indoors positioning in particular, (b) visual occlusion, and (c) scalability in relation to the size of BIM models and end-user experience (frame-rates of virtual elements updates, general responsiveness, etc.). Some of the barriers could be removed by developing a specialised AR system with features like remote server side distributed near real-time video and image processing, advanced computer vision algorithms to help with unwanted visual occlusions, etc. The idea of using augmented reality needs to be developed in parallel with conventional methods, so that when the basic technology for augmented reality matures engineers and architects will be able to take advantage of it. Needless to say, well-formed digital models, such as BIM, are a prerequisite for AR as well. The three answers given above were reached using a methodology that combined theoretical argument why AR should work with empirical evidence, gathered by the survey, that AR works. This was possible even based on a limited academic prototype. As such prototypes are likely to precede many future technologies we believe this approach could generally be useful to avoid the “Popper Trap” when applying scientific method to technological problems.
Acknowledgement Research work presented in this paper was founded by the Slovenian Research Agency. Their support is greatly acknowledged. The authors also wish to thank all those who participated in this survey. In order to preserve anonymity they cannot be named. Without their help, this survey could not have been carried out.
References [1] Liverani A, Amati G, Caligiana G. Interactive control of manufacturing assemblies with mixed reality. Integr Comput Aided Eng 2006;13:163–72. [2] Webster A. Augmented reality in architectural construction, inspection, and renovation, computing in civil engineering. ASCE; 1996. p. 913–19. [3] A.B. Craig, Mobile augmented reality, Understanding augmented reality: concepts and applications. 2013. p. 209–220. http://dx.doi.org/10.1016/B978-0-240-824086.00007-2. [4] Behzadan AH, Kamat VR. Scalable algorithm for resolving incorrect occlusion in dynamic augmented reality engineering environments. Comput Aided Civil Infrastruct Eng 2010:3–19. doi:10.1111/j.1467-8667.2009.00601.x. [5] Augment, http://augmentedev.com [6] Björk B, Penttila H. A scenario for the development and an implementation of the building product model standard. Adv Eng Softw 1989:76–187. [7] Bimar, http://www.pointadvisory.com/bimar. [8] BIMServer, http://bimserver.org. [9] Eastman C, Teicholz P, Liston K. BIM handbook. Wiley; 2008. [10] Rebolj D, Menzel K. Mobile computing in construction. J Inf Technol Constr (ITcon) 2004;9:281–3. [11] Pour G. Component-based software development approach: new opportunities and challenges. In: Proceedings of the technology of object-oriented languages; 1998. p. 375–83. [12] Adeli H, Karim A. Construction scheduling, cost optimization, and management – a new model based on neurocomputing and object technologies. London: Spon Press; 2001.
[13] Bae H, Golparvar-Fard M, White J. High-precision vision-based mobile augmented reality system for context-aware architectural, engineering and construction facility management (AEC/FM) applications. Vis Eng 2013:1–13. doi:10.1186/22137459-1-3. [14] Davidson J, Campbel LA. Collaborative design in virtual space - Greenspace II: a shared environment for architectural design review, in, design computation, collaboration, reasoning. In: Proceedings of the conference on ACADIA 1996.; 1996. p. 165–79. [15] Zhang J, Yu F, Li D. Development and implementation of an industry foundation classes-based graphic information model for virtual construction. Comput Aided Civil Infrastruct Eng 2014;29:60–74. [16] Huggins JK. The assumptions of computing. In: Proceedings of the conference on ethics in the computer age, ECA ’94; 1994. p. 46–50. doi:10.1145/199544.199585. [17] Popper K. The logic of scientific discovery. London: Routledge; 1992. [18] Roche K. Pro iOS 5 augmented reality ISBN:1430239123 9781430239123. Apress Berkely; 2011. [19] Carozza L, Tingdahl D, Bosché F, Gool L, Markerless vision-. based augmented reality for urban planning. Comput Aided Civil Infrastruct Eng 2014;29:2–17. doi:10.1111/j.1467-8667.2012.00798.x. [20] Layar, http://www.layar.com/documentation/browser/3d-model-converter/. [21] M. Myers and M. Newman, The qualitative interview in IS research: examining the craft, Information and organization. 2007. p. 2–26. http://dx.doi.org/10.1016/j.infoandorg.2006.11.001. [22] Schnabel M. Framing mixed realities, mixed reality in architecture design and construction. Springer; 2007. p. 3–11. [23] Kostaras N, Xenos M. Usability evaluation of augmented reality systems. Intell Decis Technol 2012:139–49. [24] NBMS, The National BIM Standard-United States, http://www. nationalbimstandard.org/. [25] Milgram P, Takemura H. Augmented reality: A class of displays on the realityvirtuality continuum. Telemanip Telepresence Technol 1994;2351:282–92. [26] Duston PS, Shin H. Key areas and issues for augmented reality applications on construction sites. Mixed reality in architecture design and construction. Springer; 2009. p. 157–70. [27] Klinc R, Turk Ž, Dolenc M. Engineering collaboration 2.0: requirements and expectations, ITcon. Spec Issue Next Gener Constr it: Technol Foresight, Future Stud, Roadmapping, Scenar Plan 2009;14:473–88 http://www.itcon.org/2009/31. [28] Topping RE. Advanced asset knowledge through the use of augmented reality technologies. Austin: The University of Texas at Austin; 2011. [29] Azuma RT. A survey of augmented reality. Presence: Teleoper Virtual Environ 1997;6(4):355–85. [30] Meža S, Turk Ž, Dolenc M. Component based engineering of a mobile BIM-based augmented reality system. Autom Constr 2014;42:1–12. [31] Meža S, Turk Ž, Dolenc M. Integrating ideas, symbols, and physical objects in architecture engineering and construction. In: Proceedings of the conference on international post graduate research; 2013. p. 297–304. [32] Cerovšek T. A review and outlook for a ‘Building Information Model’ (BIM) a multi-standpoint framework for technological development. Adv Eng Inf 2011:224–44. doi:10.1016/j.aei.2010.06.003. [33] Hollerer T. Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system. Comput Graph 1999;23(6):779–85. [34] Olsson T, Savisalo A, Hakkarainen M, Woodward C. User evaluation of mobile augmented reality in architectural planning, eWork and eBusiness in Architecture. In: Gudnason G, Scherer R, editors. Engineering and Constr. Reykjavik, Island: ECPPM; 2012. p. 733–40. [35] Schultze U, Avital M. Designing interviews to generate rich data for information systems research. Inform Org 2011;21(1):1–16, ISSN 1471-7727. doi:10.1016/j.infoandorg.2010.11.001. [36] Barfield W, Caudel T. Wearable computers and augmented reality ISBN:0805829024 edt. Hillsdale: L. Erlbaum Associates Inc.; 2001. [37] Cai X, Lyn M, Wong K, Ko R. Component-based software engineering: technologies, development frameworks and quality assurance schemes. IEEE Computer Society; 2013. [38] Wang X, Love P, Kim MJ, Park C, Sing C, Hou L. A conceptual framework for integrating building information modeling with augmented reality. Autom Constr 2013;34:37–44. doi:10.1016/j.autcon.2012.10.012. [39] Aziz Z. Supporting site-based processes using context-aware virtual prototyping. J. Archit. Eng. 2012;17(2):79–83. doi:10.1061/(ASCE)AE.1943-5568.0000068. [40] ZGO - Pravilnik o projektni dokumentaciji. http://www.uradni-list.si/1/ content?id=86836, 2006. [41] Turk Ž. Phenomenological foundations of conceptual product modeling in architecture, engineering and construction. Artif Intel Eng 2001;15:83–92. doi:10.1016/S0954-1810(01)00008-5. [42] Turk Ž, Wasserfuhr R, Katranuschkov P, Amor R, Hannus M, Scherer RJ. Conceptual modelling of a concurrent engineering environment, concurrent engineering in construction. 1st International Conference on Concurrent Engineering in Construction, CEC’97, London, UK; 1997. p. 195–205.
Contents lists available at ScienceDirect
Advances in Engineering Software journal homepage: www.elsevier.com/locate/advengsoft
Measuring the potential of augmented reality in civil engineering Sebastjan Meža, Žiga Turk, Matevž Dolenc∗ Chair of Construction Informatics, University of Ljubljana, Faculty of Civil and Geodetic Engineering, Jamova 2, 1000 Ljubljana, Slovenia
a r t i c l e
i n f o
Article history: Received 7 October 2014 Revised 4 February 2015 Accepted 11 June 2015 Available online 26 June 2015 Keywords: Augmented reality Mobile computing Computer integrated design Computer integrated engineering Civil engineering Project documentation Building information modelling BIM
a b s t r a c t Recently building information models have substantially improved the explicit semantic content of design information. Information models are used to integrate the initial phases of project development. On the construction site, however, the designs are still mostly represented as line-based paper drawings or projections on portable displays. A generic technology that can integrate information and situate it in time, place and context is augmented reality. The specific research issues addressed are (1) does augmented reality have a potential use in civil engineering, (2) how big – in comparison to other technologies - is this potential and (3) what are the main barriers to its adoption. The generic research issue was to develop a methodology for evaluation of potentials of technology. A prototype was built. It was tested on a real construction site to evaluate the potential of its use using the action-research method. A set of structured interviews with potential users was then conducted to compare the prototype to conventional presentation methods. Using this methodology it has been found out that augmented reality is expected to be as big a step as the transition from 2D line drawings to photorealistic 3D projections. The main barrier to the adoption is immature core virtual reality technology, conservative nature of construction businesses and size of building information models. © 2015 Elsevier Ltd. All rights reserved.
1. Introduction Tools for designing in construction have evolved through history. Pens, pencils and paper have been replaced with CAD (computer aided design) and BIM (building information modelling) software. Engineers, builders, planners and contractors also use various domain specific software to support their work. A priority for construction informatics research and practice has been to effectively integrate the construction processes using information technology [12]. Adequate standards, e.g. Industry Foundation Classes (IFC), have the potential of solving the problem of interoperability of software and representation of information in designing [15]. While the design phase is largely digitised and increasingly integrated around BIM, for a complete digitalisation of construction industry, structured information models would need to be available on construction site where the information is used to shape material world. However, on the construction site the IT infrastructure is not readily available. Things began to change with the introduction of mobile computing [10]. The field is still evolving. 1.1. Motivation The outputs of construction information processes (designs, plans and schedules) provide the control information for the material ∗
Corresponding author. Tel.: +386 1 4768 606. E-mail address: [email protected] (M. Dolenc).
http://dx.doi.org/10.1016/j.advengsoft.2015.06.005 0965-9978/© 2015 Elsevier Ltd. All rights reserved.
processes in construction [42]. The media to bring the information from the digital models to construction site where it is used to shape physical reality are still 2D documents such as floor plans, cross sections, sketches, etc. The construction site is integrated into the construction process using media and formats that pre-date computers. Situating information and establishing the relation between the real world of the construction site and design information remains the task of humans. In this task they are not assisted much by technology. Relevant information from the model has to be extracted, based on the user’s role in the project, location and time. The graphical representation of this information in 2D must be situated and contextualised with the physical 3D reality for which people rely on their spatial awareness. It is the technologically largely unassisted human mind that is bridging the gap between the real world of the construction site and the virtual world of the information model and is integrating the two. This is what engineers on site have been doing since the introduction of drawn design information centuries ago. The problem at hand is how to assist this process with technology. The hypothesis of our research of augmented reality (AR) was that by using a synthetic environment that enables the integration of 4D building information models into the live picture of real world it is possible to improve the understanding and ease the use of project information. It should be possible to measure this improvement. We claim that such synthetic environment is augmented reality. It is not just feasible, but it is also more effective than the more traditional, well-established presentations on blueprints or on-screen projections.
2
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
The specific questions the paper asks are (a) does AR have potential in structural engineering, (b) how big an improvement this technology is and (c) what are the barriers to its adoption. To answer these questions we had to develop a methodology, which is generic in nature and applicable to other technology related research in the interdisciplinary area of structural engineering and computer science. 1.2. On research methodology A lot of research in engineering in general and in construction informatics in particular does not analyse existing phenomena, as is the case in natural sciences. Rather, it synthesises (creates) new solutions and improves technologies. Establishing success or failure of this kind of research is methodologically difficult for at least two reasons. First, whatever the technology is doing to assist in the construction process has already been done without that technology for decades or centuries. So naturally it can be done with some additional help of new technology as well. It hardly can fail. In Karl Popper’s terms [17] the refutability of such research is questionable. Second, researchers can create prototypes that prove an idea but do not have the resources to create commercial grade software. Research prototypes lack robustness, friendliness and usability of commercial systems created by hundreds or perhaps thousands of programmers. Measuring the success of crude research prototypes does not do justice to the potential of the technology. In our research this problem was addressed by an innovative combination of several evaluation methods, both theoretical as well as empirical. Theoretical foundations are set on phenomenology. It provides philosophical basis for the hypothesis that augmented reality has a potential. A prototype was built. It was tested and studied in real life settings to assess the current technological capabilities and limitations. Finally the usefulness of the developed prototype was examined by conducting structured interviews with potential end-users. 1.3. Paper structure In the introduction the research context, goals, method and the hypothesis were defined. Section 2 presents the related work. Section 3 continues with the description of the design and implementation of the prototype. Section 4 presents the theoretical selfevaluation of the developed prototype and the preliminary field tests. Those results were used as input data for the empirical part of the research – the survey – that is presented in Section 5. In the conclusions the results are analysed and the hypothesis is revisited. 2. Background and related work In this section augmented reality is discussed from theoretical, technological and practical points of view. It explains our understanding of the building information modelling and its relation to construction project documentation. As construction can be understood as the materialisation – physical realisation of the project documentation [26,31,41], a philosophical discussion on relations among human mind, virtual and real environment is provided.
The other extreme of that continuum is a virtual environment, which allows engineers and designers to design objects in imagined, virtual, and designed, but not yet materialised world. Augmented reality is therefore the middle segment of continuum where virtual elements are added to real world [29]. Building information models are actually representations of designed reality, aimed at improving the perception of ideas and exchange information [41]. By advanced rendering of the models and surroundings, the user is offered a greater amount of information. This can reduce the possibility of misinterpretation of the designs [3]. With technological progress greater degree of realism can be achieved but according to this theory the perfect model cannot be established since reality presents a limit that can be approached but never reached [36]. Augmented reality on the other hand offers a different approach. It takes the live picture of real surroundings as a base to which virtual elements are added. With that in a given point in time and space the user’s interpretation of proposed digital solutions can be easier [14]. There are many terms used to define the segment of continuum between the extremes of the real and the virtual: mixed reality, amplified reality, augmented reality, mediated reality, diminished reality, augmented virtuality, virtualised reality [22]. In this paper the term, augmented reality (AR) is used since our application augments the insight to the real situation for the user. 2.2. Building information modelling/model By definition the building information modelling is a process of digital designing, which results in some form of a building information model (BIM). Ideally it should include all the data needed for the construction instead of the data being scattered throughout numerous drawings, folders, tables, reports, documents, etc. [9]. The basic premise of building information modelling is to enable frictionless collaboration of different actors (professions) at various stages of a building life cycle, integrated around a shared model. The actors may enter, retrieve, update or adopt the information in BIM and with that justify their roles as the participants in the construction process [6,24]. The 4D BIM is a model that includes the temporal properties. The 5th and 6th dimensions of BIM sometimes denote cost and facility management [32]. 2.3. Project documentation Formally speaking, the project documentation that is required by Slovenian legislation is defined by the Construction Act [40]. It should consist of the conceptual design, preliminary design, basic design, detailed design, and as-built design. In this paper the term “project documentation” is used broader than formally defined by Slovenia legislation. The term project documentation is used to denote a set of all documents needed for the construction. It includes all information contained in building information models. 2.4. Intersection between conscious real and virtual
2.1. Augmented reality Although virtual and real environments are two completely different entities it is practically impossible to make a clear boundary between them. They can be better presented with two poles of continuum [25], the real and the virtual. The virtual environment must be completely predefined since computers cannot make their own assumptions [16]. The real is a complex mixture of natural events and items that exist in one of the pole of the continuum. Reality, therefore, includes all that can be created, built, planned, observed, understood etc.
The role of augmented reality can theoretically be explained in the context of the meaning triangle in Fig. 1 [31]. The concept is an idea in the mind that refers to that specific referent (real world object). The symbol is a visual or audible signal symbolising the idea about that referent. The presented example shows that it is possible to establish a direct relation between referent-reference and reference-symbol (Fig. 1). The first is called referencing and the second modelling. The relation between the symbol and the object is more complicated as both exist outside the mind of the human. However, one could say
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
Fig. 1. The meaning triangle.
that construction is a process in which symbolic design representations (SYMBOL) are translated into real world buildings (OBJECT). Unless robots do this, human interpretation of symbols is essential. Augmented reality assists in this interpretation because it places the symbols over the picture of the real world. It is a superior technology to 2D plans and projections and virtual reality because these technologies keep the symbolic and the real apart with the human mind acting as the interface between the two. 2.5. Related work As early as 1996 Webster developed a working prototype called “Architectural anatomy” [2]. Application allowed a user to “see” structural rebar that obviously cannot be seen by a naked eye since it is surrounded by concrete. The result of the project MARS (Mobile Augmented Reality System) was a system that allowed a user to move freely in an open space and see virtual buildings added to the user perspective [33]. A significant amount of research effort was devoted to field of AR in the late 90s and in the beginning of last decade but due to the restrictions of available information technologies at that time (e.g. processing power, connectivity, size, etc.) the developed prototypes had limited practical value. Recent advancements in the field of mobile computing in conjunction with building information modelling have opened new opportunities for AR research and development. Today the most advanced mobile devices meet the minimum requirements of machine performance and portability needed for AR [39]. Therefore on the substantive side, the focus of research partly shifted to a critical assessment
3
of the usefulness of systems and not the practical feasibility of creating AR [1,23,27,34]. In recent years, the focus of the AR research is in the visual occlusion handling [4,13] as well as on improved tracking accuracy [19]. Tracking includes real-time calculation of both the location and the orientation of the viewpoint. It can be done with (1) reference images or with (2) sensors integrated in mobile devices, e.g. GPS, compass, gyroscope. Our solution [30] uses sensors, as presented use cases do not foresee extensive pre-preparations of the real construction site surroundings. On the other hand, the few existing commercial applications such as Bimar [7] and Augment [5] mainly rely on reference images, which makes the two approaches very different. The presented prototype was built on top of the existing components, which is the only feasible approach for academic environment. Therefore the research mainly deals with construction specific issues such as integration of BIM and AR [38] rather than innovation of AR display technology. 3. Prototype This Section first focuses on the information flows in the prototype. The IDEF0 method was chosen as a means of schematic abstraction since it allows an effortless presentation of the system activities, relations among them and external mechanisms by which they are influenced. Our system consists of four activities (Fig. 2). The first activity in the system is building information modelling (A0) which produces the building information model (BIM). Depending on the state of the project this may be a 3D model. Or in later stages the model may include the time component which results in the so-called 4D BIM. When modelling, variety of tools compatible with the IFC STEP standard can be used. The output of the first step is an IFC model that is used in all-subsequent steps. Normally the exchange of BIM between various software vendors takes place in the form of IFC files. Such an exchange can be carried out online using various servers; one of such is the open source BIM server [8] that has also been used in our prototype. BIM server in conjunction with file convertors is used in the second activity (A1) to prepare the L3D model, which can then be displayed in the generic AR application Layar [20]. This activity also depends on the stage of construction project. In case of construction project monitoring, the transformation is somewhat more complex since the model has to be modified so that it is compliant with the schedule. Activities A2 and A3 are similar. In both, the AR application Layar is used to display the L3D model. The difference is in the purpose of use. In case of 3D visualisation (A2), the purpose is to
Fig. 2. IDEF0 diagram of prototype functioning.
4
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
Fig. 3. The initial model.
determine whether the model in reality looks as it was imagined and designed in the virtual environment. The purpose of the activity A3 is to see whether the situation at the site corresponds to the one predicted in the schedule (Fig. 2). Our system can be used in two ways; (1) it can be used for the visualisation of 3D models in the phase of urban planning – A2 and (2) verifying whether the construction work in the field is being carried out in accordance with the schedule – A3. Practical demonstration of the two cases is shown later in this paper (Figs. 5d and 9d). 3.1. Practical implementation Structurally the system consists of a server and a mobile client component. A prototype system has been developed by the method of component-based software engineering (CBSE). CBSE is becoming increasingly popular method since it is time and cost efficient [37]. The CBSE is based on the idea that the most appropriate components are identified and assembled into a new complex system [11]. The server part of our system is dedicated to storing and managing data. It includes a BIM server, FTP server and web service. The BIM server provides effective information exchange between commercial programs compatible with the IFC standard [8]. The FTP server was used as an intermediate location where the models are temporary stored prior to being displayed with a generic AR display component. The web service is preparing a model suitable for display in a generic AR environment. The synchronisation between BIM and FTP servers and all the necessary transformations is automated. The mobile part is devoted to the presentation of BIM models on site. It consists of the custom made Android application and a generic AR display unit. The mobile application serves as a mean of communication between the user and servers. It is used as a schedule display unit and the user also has an insight into revisions of construction projects. There is a multitude of generic AR display apps e.g. Junaio, Layar, Metaio, MixAre, and many others, which can merge digital information and the live picture of real surroundings. Applications running on the mobile devices, mainly operating on the same principle, constantly monitor the user’s position and orientation in space. When a user reaches a zone from which the point of interest (POI) can be viewed, it is overlaid over the live picture of real surroundings [18]. In our case the POI is a 3D model, which was located on the FTP server. 4. Testing of the prototype The developed system can be used in two ways. First, for a presentation of the entire 3D model. This method is mainly intended to
visualise construction projects in the phase of urban planning. Second, in a 4D BIM scenario, which involves the verification of whether the construction is progressing according to the schedule. Testing was conducted in two steps. First, prototype self-evaluation was conducted. This was then followed by the field tests. 4.1. Reliable tests of the prototype In our system Layar is used as the AR display unit. Layar was chosen as it allows the creation of 3D model on the fly. The Layar L3D model converter – command line version – requires model in the format of OBJ/MTL to produce the L3D model suitable for display in the AR environment [20]. When creating 3D models, care should be taken of their size. L3D models should not contain more than 10,000 faces. In the prototype, the L3D model is generated automatically using IFC – OBJ/MTL and OBJ/MTL – L3D transformers. Mappings from IFC to L3D are performed sequentially. Autodesk Revit was used to create the architectural IFC model (Fig. 3). It consisted of 423 elements and its size was 6.7 MB. As the basis, a sample project that comes with the program was taken. Due to few IFC compatibility issues in Revit, some material properties had to be set manually to obtain the L3D model that accurately reflects the original one (Fig. 4). The experiment was repeated on the model of a real construction project of a multi-residential building “ECO silver house”. The IFC model of the load bearing structure used in the experiment consisted of 984 elements and its size was 2.34 MB (Fig. 5). In this set of tests our system performed as expected. As it turned out, the integration of 3D IFC model is more problematic than the actual display of the AR model. Especially time consuming is the process of determining the time dimension (schedule) of each element of the IFC model. Of course, the end result is restricted with the operation of generic AR display unit. It does not address the problem of visual occlusion and the imprecision of global positioning systems [30]. 4.2. Field tests The field tests were performed in order to test the reliability of our system on site and to obtain visual material for the final end user survey. The result of the AR visualisation of the 3D model is shown in (Fig. 6d). Most of inconveniences with which we were faced can be associated to the operation of the generic AR display component. Those can be further related to the inaccuracy of the mobile device’s sensors. In this test the progress of the construction of a real construction project in Ljubljana - “ECO silver house (Fig. 12) was monitored.
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
5
Fig. 4. The L3D model.
Fig. 5. The 4D model.
The prototype was used as an alternative method of time control of construction progress. The site was completely surrounded by the existing buildings and fences; therefore some views had the problem of the visual occlusion. 5. The survey In this section the results of a survey among potential users are presented. It was conducted, based on the experiences with the prototype. We wanted to gain an insight into the thinking of the AEC experts on the applicability of our AR solution and AR in general. The survey was conducted in the following steps. (1) Definition of starting points and formation of the research questions. (2) Definition of the research population and sampling. (3) Preparation of a short presentation of augmented reality. (4) Preparation of a questionnaire. (5) Preliminary substantive test of the questionnaire. (6) Execution of the structured interviews. (7) Analysis of the results. 5.1. The baseline The main goal of our research was to compare our AR system with the conventional presentation techniques and with that obtain an
empirical proof that augmented reality is not just feasible but also useful. When selecting the method, the following was considered. (1) The method had to allow analytical comparison of the established presentation techniques with AR. (2) Because AR was relatively unknown to the target audience; it was necessary to choose a method, which allowed a clear presentation of the topic. (3) The resulting data should allow subsequent quantitative analysis, therefore the answers to key questions had to be predefined [35]. In addition to the real world testing, questionnaires and interviews are most suitable methods of AR system evaluation. It is possible to efficiently collect subjective data regarding the users’ opinions and preferences without the need of specialised hardware and software equipment. We decided to preform structured interviews. The main advantage of the structured interview is the possibility of personal presentation of the topics covered and the opportunity to further clarify possible doubts with individual questions [21]. Prior to the start of each interview a short presentation of the topic was given. The introduction was aimed at excluding the risks associated with the terminology. Individual interviewees were acquainted with the meaning of the terms augmented reality, BIM and project documentation. The essential requirements for the existence of AR and explained the difference between the virtual and augmented
6
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
Fig. 6. Comparison of presentation methods - visualisation of preliminary studio.
reality were summarised. The potential advantages of AR were not exposed, as this could negatively affect the objectivity of the responses. The interviews were conducted in two steps. First the understanding of the interview questions and the multimedia presentation used for the interview as well as the expected length of the interview was tested on a small test group of four people. The findings of the pilot phase helped us in preparing the final questionnaire. The final questionnaire used in the interviews consisted of two parts. The first part included general demographic questions (gender, age, field of work, etc.). The second and main part of the questionnaire consisted of 11 questions: nine of them had pre-defined multiple choice answers and two in which the interviewees ware asked to comment their decisions. The interviews were carried out in January 2014 with representatives of AEC profession. Twenty persons from different Slovenian AEC companies were invited; fifteen of them accepted the invitation. Responses were obtained from all main target groups, which included architects and civil engineers. Each interview was carried out separately in duration between 20 and 55 min. In all the interviews questionnaires have been fully completed. 5.2. The results The first task in the interview was to compare four presentation techniques of project documentation: (1) 2D drawings, (2) 3D BIM on a computer, (3) 3D BIM on a tablet and (4) augmented reality. The participants were asked to compare those techniques and rate them with 1 (worst) to 10 (best). They had to evaluate understandability and usability of project documentation in two use cases discussed in Section 4. The visualisation of preliminary design and monitoring control of a construction process were evaluated. 5.2.1. Visualisation of preliminary design Interviewees were asked to rate the understandability and usability of presentational methods based on the Fig. 6. In addition to images, a demonstration video of system field tests was also shown to the interviewees.
The average ratings are presented in the following charts (Figs. 7 and 8). Separate values are presented for engineers and architects. In both cases augmented reality has achieved the highest score. The comparison of the understandability and the usability among engineers and architects shows that the engineers rated AR better. The result of the statistical analysis is consistent with the concluding comments of the survey. All architects included in the research emphasised the advantages of 2D-mode design, especially in the later stages of the construction projects. On the other hand they all saw a huge unexploited potential of AR, especially when communicating with clients, who usually lack the skills of understanding engineering drawings. 5.2.2. Monitoring of a construction progress Also in this use case interviewees were asked to compare the understandability and usability of four different presentation techniques (Fig. 9). They were offered a Gantt chart, 4D simulation of the construction, simulation on tablet PC and the idealised view of the project documentation. When inquiring about the understandability and usability in this use case we got the same ranking as in the previous use case. In both cases the best result was achieved by augmented reality, followed by the virtual simulations on the PC, 2D and finally simulations on tablets. On average ratings were slightly lower than in the first use case (Figs. 10 and 11). The comparison of the understandability and the usability among engineers and architects shows that the architects rated AR better than engineers. The comparison of 2D and 3D shows that engineers prefer Gantt charts over 4D simulations – just the opposite to the architects. 5.2.3. Assessment of the prototype The second task of the survey was to assess the performance and usability of the prototype. Interviewees were given the following task: “Imagine that you are on a construction site. Your task is to evaluate whether the work is carried out in accordance with the
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
Fig. 7. Understandability of project documentation in the visualisation preliminary studies.
Fig. 8. Usability of project documentation in the visualisation preliminary studies.
Fig. 9. Comparison of presentation methods for construction process monitoring.
7
8
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
Fig. 10. Understandability of project documentation in monitoring of construction.
Fig. 11. Usability of project documentation in monitoring of construction.
Fig. 12. Screenshot of 4D BIM AR.
planned schedule. You have a smart phone, which allows you to view real surroundings with the addition of virtual elements. The green colour indicates the item that is currently on schedule.” After the interviewees were provided with the instructions, they were shown the screenshot recorded on a real construction side (Fig. 12) and asked them the following question. Is it possible to determine whether the construction of the building is carried out in accordance with the schedule? More than 80% of the respondents agreed that it is possible to see that. Those who have answered yes were posed an additional question. “What is the state of the construction site? Is the construction on schedule; is it behind or ahead of schedule?” All answered correctly that the work is slightly behind schedule. Then they were asked to compare the realistic (Fig. 12) with an idealised (Fig. 13) view of augmented reality, to expose the most disturbing factors in real view and suggest what to add to the image that the application would be better. They were proposed three most
Fig. 13. Idealised view of AR.
obvious factors: (1) virtual model is not completely aligned with the surrounding area, (2) the construction site fence and the other elements located between the observation point and the building, (3) small size and resolution. The analysis of the responses indicated that the elements that obstruct the field view are the most disturbing. In addition to this visual occlusion problem the participants were most concerned with the amount of details contained in the augmented reality view. In the presented case only the element that is currently on schedule was marked. The detail about which task e.g. formworks, rebar, concreting, should be currently preformed was not provided. The respondents agreed that the current state of detail is sufficient for a rough assessment, mainly if the investor wants to quickly verify that the work is carried out in accordance with the schedule. 5.2.4. Concluding part of the survey The aim of this part is the assessment of the untapped potential of augmented reality in the AEC context. The questions of interest
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
9
Fig. 14. Potentially the most promising areas for the application of augmented reality.
were: (1) what the main advantages of augmented reality compared to virtual reality were, (2) in which phase of the construction project augmented reality could be most beneficial, (3) which participants of the architecture, engineering and construction sector could gain the most with AR and (4) what it would take to replace 2D drafts with AR. The interviewees were asked to compare the virtual reality and augmented reality and evaluate some features that are not available in virtual reality. Mainly all the augmented reality features were positively evaluated. The possibility to see the virtual models in real time on site from different perspectives gained the top rating. The possibility of free movement within the real space, without the use of the user interfaces, received the most concerns. Some interesting comments at this point were related to the inherent danger of construction site environment and risks posed by augmented reality equipment: - “Especially when one would use wearable mobile devices, e.g. glasses, special care should be taken, while in case of device failure this method of display could be dangerous.” - “I think that overlaying a part of the user’s field of view could be dangerous, especially with the use of wearable devices.” - “Most likely it would take quite some time to get used to the new user interface. I think that it could be dangerous if the user would have suddenly got too many information and thus forget about the real surrounding environment.” When asking about most promising areas for AR the topic classification from Topping [28] was used. Interviewees were asked to evaluate the AR potential of individual tasks with (1) very useful, (2) useful, (3) partly useful and (4) not useful. The analysis of the results showed that AR would be most useful in identifying and locating the existing building component locations and in the control of the compliance of the design and the actual building. Our first use case visualisation of 3D models has proven to be very promising, on the other hand time control – schedule compliance gained a slightly lower rating (Fig. 14). Some of the concluding remarks and comments of the survey: - “Construction is a specific, very traditional and conservative industry in which new technologies are very difficult to enforce. In any case, investing money in new technologies makes sense, especially on the long run. I believe that it will take a generation of engineers to fully adopt 3D principle of design and construction.” - “I believe that the technology is already available, especially for visualisation of 3D models in the field. Architects could presumably already take advantage of such systems. The problem lies in the conservatism of the construction industry. Constructors must first fully adopt 3D building information modelling, only then we can start to
discuss the monitoring of construction process progress with augmented reality.” - “The project documentation in paper form could probably be abandoned in earlier phases of construction projects. I believe that AR would be most beneficial in renovations and in the supervision of the compliance of the completed projects. But first it should be made obligatory to create PID at the handover of buildings in form of building information modelling.” - “In practice, drawing board and pencil remain the primary means of communication. As it allows us - architects to most easily document our thoughts. This is the case in the early design phase. CAD and building information modelling are most beneficial in later design stages. However, I believe that AR could be used as a communication bridge among the profession and general public.” 6. Conclusions and discussion Our research confirmed that augmented reality can significantly contribute to the understanding of project documentation in various stages of construction projects and is thus better integrating the construction phase with earlier stages of development. The claim of significance is based on the comparison of well-established conventional presentation techniques with augmented reality, both theoretically as well as empirically with the survey among potential users. The understandability and usability of project documentation was compared using the following techniques: (1) 2D plans, (2) BIM on a PC, (3) the use of tablet computers and (4) augmented reality. The techniques were compared with each other and quantitatively evaluated in the two use cases: (1) the visualisation of preliminary design and (2) the monitoring of construction progress. The comparison showed that augmented reality is at least one grade better than any other presentation technique. The cumulative of responses showed that the 3D mode is approximately 7% better than 2D, while AR could improve 3D up to 20%; however, only when taking into account certain assumptions. With that the paper’s hypothesis was essentially confirmed but it is necessary to discuss the conditions and assumptions. Augmented reality can facilitate the understanding of project documentation especially in the visualisation of 3D models in the field, but remains technologically constrained. Generally speaking the function of the prototype – visualisation of preliminary design – is more useful as the function of construction schedule supervision. The experts from the field of architecture, engineering and construction have assessed that the current functioning of the prototype presented in this paper can be most useful with the process of communication between the experts and the investors. It is also
10
S. Meža et al. / Advances in Engineering Software 90 (2015) 1–10
possible to conclude that the presented prototype allows us to see and estimate whether the work on site is conducted in accordance with the expectations defined in the schedule. The quality of display is sufficient, although the most disturbing factor is the visual occlusion. It can be concluded that although augmented reality has a substantial potential it is unlikely that in the nearby future it could replace the conventional presentation techniques. The main barriers were found to be (a) GPS positioning in general and indoors positioning in particular, (b) visual occlusion, and (c) scalability in relation to the size of BIM models and end-user experience (frame-rates of virtual elements updates, general responsiveness, etc.). Some of the barriers could be removed by developing a specialised AR system with features like remote server side distributed near real-time video and image processing, advanced computer vision algorithms to help with unwanted visual occlusions, etc. The idea of using augmented reality needs to be developed in parallel with conventional methods, so that when the basic technology for augmented reality matures engineers and architects will be able to take advantage of it. Needless to say, well-formed digital models, such as BIM, are a prerequisite for AR as well. The three answers given above were reached using a methodology that combined theoretical argument why AR should work with empirical evidence, gathered by the survey, that AR works. This was possible even based on a limited academic prototype. As such prototypes are likely to precede many future technologies we believe this approach could generally be useful to avoid the “Popper Trap” when applying scientific method to technological problems.
Acknowledgement Research work presented in this paper was founded by the Slovenian Research Agency. Their support is greatly acknowledged. The authors also wish to thank all those who participated in this survey. In order to preserve anonymity they cannot be named. Without their help, this survey could not have been carried out.
References [1] Liverani A, Amati G, Caligiana G. Interactive control of manufacturing assemblies with mixed reality. Integr Comput Aided Eng 2006;13:163–72. [2] Webster A. Augmented reality in architectural construction, inspection, and renovation, computing in civil engineering. ASCE; 1996. p. 913–19. [3] A.B. Craig, Mobile augmented reality, Understanding augmented reality: concepts and applications. 2013. p. 209–220. http://dx.doi.org/10.1016/B978-0-240-824086.00007-2. [4] Behzadan AH, Kamat VR. Scalable algorithm for resolving incorrect occlusion in dynamic augmented reality engineering environments. Comput Aided Civil Infrastruct Eng 2010:3–19. doi:10.1111/j.1467-8667.2009.00601.x. [5] Augment, http://augmentedev.com [6] Björk B, Penttila H. A scenario for the development and an implementation of the building product model standard. Adv Eng Softw 1989:76–187. [7] Bimar, http://www.pointadvisory.com/bimar. [8] BIMServer, http://bimserver.org. [9] Eastman C, Teicholz P, Liston K. BIM handbook. Wiley; 2008. [10] Rebolj D, Menzel K. Mobile computing in construction. J Inf Technol Constr (ITcon) 2004;9:281–3. [11] Pour G. Component-based software development approach: new opportunities and challenges. In: Proceedings of the technology of object-oriented languages; 1998. p. 375–83. [12] Adeli H, Karim A. Construction scheduling, cost optimization, and management – a new model based on neurocomputing and object technologies. London: Spon Press; 2001.
[13] Bae H, Golparvar-Fard M, White J. High-precision vision-based mobile augmented reality system for context-aware architectural, engineering and construction facility management (AEC/FM) applications. Vis Eng 2013:1–13. doi:10.1186/22137459-1-3. [14] Davidson J, Campbel LA. Collaborative design in virtual space - Greenspace II: a shared environment for architectural design review, in, design computation, collaboration, reasoning. In: Proceedings of the conference on ACADIA 1996.; 1996. p. 165–79. [15] Zhang J, Yu F, Li D. Development and implementation of an industry foundation classes-based graphic information model for virtual construction. Comput Aided Civil Infrastruct Eng 2014;29:60–74. [16] Huggins JK. The assumptions of computing. In: Proceedings of the conference on ethics in the computer age, ECA ’94; 1994. p. 46–50. doi:10.1145/199544.199585. [17] Popper K. The logic of scientific discovery. London: Routledge; 1992. [18] Roche K. Pro iOS 5 augmented reality ISBN:1430239123 9781430239123. Apress Berkely; 2011. [19] Carozza L, Tingdahl D, Bosché F, Gool L, Markerless vision-. based augmented reality for urban planning. Comput Aided Civil Infrastruct Eng 2014;29:2–17. doi:10.1111/j.1467-8667.2012.00798.x. [20] Layar, http://www.layar.com/documentation/browser/3d-model-converter/. [21] M. Myers and M. Newman, The qualitative interview in IS research: examining the craft, Information and organization. 2007. p. 2–26. http://dx.doi.org/10.1016/j.infoandorg.2006.11.001. [22] Schnabel M. Framing mixed realities, mixed reality in architecture design and construction. Springer; 2007. p. 3–11. [23] Kostaras N, Xenos M. Usability evaluation of augmented reality systems. Intell Decis Technol 2012:139–49. [24] NBMS, The National BIM Standard-United States, http://www. nationalbimstandard.org/. [25] Milgram P, Takemura H. Augmented reality: A class of displays on the realityvirtuality continuum. Telemanip Telepresence Technol 1994;2351:282–92. [26] Duston PS, Shin H. Key areas and issues for augmented reality applications on construction sites. Mixed reality in architecture design and construction. Springer; 2009. p. 157–70. [27] Klinc R, Turk Ž, Dolenc M. Engineering collaboration 2.0: requirements and expectations, ITcon. Spec Issue Next Gener Constr it: Technol Foresight, Future Stud, Roadmapping, Scenar Plan 2009;14:473–88 http://www.itcon.org/2009/31. [28] Topping RE. Advanced asset knowledge through the use of augmented reality technologies. Austin: The University of Texas at Austin; 2011. [29] Azuma RT. A survey of augmented reality. Presence: Teleoper Virtual Environ 1997;6(4):355–85. [30] Meža S, Turk Ž, Dolenc M. Component based engineering of a mobile BIM-based augmented reality system. Autom Constr 2014;42:1–12. [31] Meža S, Turk Ž, Dolenc M. Integrating ideas, symbols, and physical objects in architecture engineering and construction. In: Proceedings of the conference on international post graduate research; 2013. p. 297–304. [32] Cerovšek T. A review and outlook for a ‘Building Information Model’ (BIM) a multi-standpoint framework for technological development. Adv Eng Inf 2011:224–44. doi:10.1016/j.aei.2010.06.003. [33] Hollerer T. Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system. Comput Graph 1999;23(6):779–85. [34] Olsson T, Savisalo A, Hakkarainen M, Woodward C. User evaluation of mobile augmented reality in architectural planning, eWork and eBusiness in Architecture. In: Gudnason G, Scherer R, editors. Engineering and Constr. Reykjavik, Island: ECPPM; 2012. p. 733–40. [35] Schultze U, Avital M. Designing interviews to generate rich data for information systems research. Inform Org 2011;21(1):1–16, ISSN 1471-7727. doi:10.1016/j.infoandorg.2010.11.001. [36] Barfield W, Caudel T. Wearable computers and augmented reality ISBN:0805829024 edt. Hillsdale: L. Erlbaum Associates Inc.; 2001. [37] Cai X, Lyn M, Wong K, Ko R. Component-based software engineering: technologies, development frameworks and quality assurance schemes. IEEE Computer Society; 2013. [38] Wang X, Love P, Kim MJ, Park C, Sing C, Hou L. A conceptual framework for integrating building information modeling with augmented reality. Autom Constr 2013;34:37–44. doi:10.1016/j.autcon.2012.10.012. [39] Aziz Z. Supporting site-based processes using context-aware virtual prototyping. J. Archit. Eng. 2012;17(2):79–83. doi:10.1061/(ASCE)AE.1943-5568.0000068. [40] ZGO - Pravilnik o projektni dokumentaciji. http://www.uradni-list.si/1/ content?id=86836, 2006. [41] Turk Ž. Phenomenological foundations of conceptual product modeling in architecture, engineering and construction. Artif Intel Eng 2001;15:83–92. doi:10.1016/S0954-1810(01)00008-5. [42] Turk Ž, Wasserfuhr R, Katranuschkov P, Amor R, Hannus M, Scherer RJ. Conceptual modelling of a concurrent engineering environment, concurrent engineering in construction. 1st International Conference on Concurrent Engineering in Construction, CEC’97, London, UK; 1997. p. 195–205.