- Email: [email protected]
Advanced interaction techniques in virtual environments
('omput, & Graphicv Vol. 17, No, 6, pp. 655-661. 1993
0097-8493/93 $6.00 + .00 © 1993 Pergamon Press Lid
Printed in Great Britain.
Virtual Reality
ADVANCED INTERACTION TECHNIQUES IN VIRTUAL ENVIRONMENTS MAURO FIGUEIREDO, t KLAUS BOHM ¢, and JoSl~ TEIXEIRA+ *Grupo de M~todos e Sistemas Grfificos, Departamento de Matem~itica-Universidade de Coimbra, Apartado 3008, 3000 Coimbra, Portugal *Zentrum ffir Graphische Datenverarbeitung e.V. (ZGDV), Wilheminenstral3e 7, 6100 Darmstadt, Germany Abstract--Fundamental to much of the Virtual Reality work is, in addition to high-level 3D graphical and multimedia scenes, research on advanced methods of interaction. The "visitor" of such virtual worlds must be able to act and behave intuitively, as he would in everyday situations, as well as receive expectable natural behaviour presented as feedback from the objects in the environment, in a way that he/she has the feeling of direct interaction with his/her application. In this paper we present several techniques to enrich the naturalness and enhance the user involvement in the virtual environment. We present how the user is enabled to grab objects without using any specific and elaborate hand gesture, which is more intuitive and close to the way humans are used to do. We also introduce a technique that makes it possible for the user to surround objects without any force-feedback interaction device. This technique allows the user to surround or "'walk" with the virtual hand on the object's surface and look for the best position to grab it. 1. INTRODUCTION Virtual Reality is a new type of human-computer interface that aims at creating the illusion of the user being immersed in a computer generated reality, providing a more direct communication link between users and the problem environment modeled by the computer system. The user interacts with a world containing three-dimensional objects in three-dimensional space that respond interactively to each other and the user actions. The attainment of illusion of involvement and participation place severe demands on all elements of a computer system. The primary constraint is the desire for natural and intuitive interactions. There are many examples of computer graphics applications in which the user's ability to interact with the pictures is as important as the quality of the pictures themselves. The key idea is that the virtual environment is trying to create a different world that one interacts with as if it were real. The user is reaching out and grabbing, pointing, pushing, and talking to things just as we do in the real world. In this way, the computer system must sense the user as unobtrusively and naturally as possible. This paper focus the realization of precise interactions with three-dimensional objects in virtual worlds for achieving natural interactions. Current state-of-the-art for 3D interaction is based on collision checks using bounding volumes. When collision checks are based only on bounding volumes very unnatural interactions happen. We implemented a precise collision check using also the object's polygonal information. With this information we enable the user to " t o u c h " and "feel" object's surfaces. Naturalness of interactions was improved with the help of a technique called surrounding that enables the user to grab 3D objects easily and intuitively using a 3D input device (e.g.. dataglove, spaceball ).
2. INTERACYION IN VIRTUAL ENVIRONMENTS Current state-of-the-art graphic workstations allow the creation of almost photorealistic images in less than 100 ms: some can even render two images in the same amount of time for stereoscopic viewing. The development of new input devices, interaction techniques, and interaction metaphors that can exploit the power of these workstations is, however, in its infancy. Virtual reality makes use of these capabilities in order to give the user the feeling of being part of an environment in which h e / s h e is able to manipulate virtual objects directly. But, in our opinion, the successful use of virtual reality in a wide range of applications depends critically upon the quality of interaction that can be achieved between this new technology and the users "within" it. If this match is not achieved then the technology cannot hope to realise its full potential. We need to analyse and evaluate the current stateof-the-art for 3D-interaction in order to see how far development has progressed and to determine the requirements for applying Virtual Reality to future applications. In order to systematically analyse three-dimensional interactions, a classification of two-dimensional (2D) interaction is briefly discussed. This classification and the lessons learned are then applied to three-dimensional (3D) interaction. See also [7]. This process of classification is illustrated using a progression of Interaction level diagrams. These diagrams illustrate a conceptual model of the different levels of interaction within the various dimensions. See also [ 5 ].
2.1. Two-dimensional interactions During the last decade, a set of well-defined 2D user interface paradigms has been established based upon the 2D output technology of faster devices (Fig. 1 ). At the level of input devices, the keyboard, mouse, 655
656
M. FIGUEIREDO, K. BOHMand J. TEIXEIRA
O,,,ce
Applications
IPrccesOontro,
I o,,MoOe,,no
UI-Toolkits
Metaphors
I
I oo,,,o0 MetaphorsI Direct Manipulation I
Interaction Tasks II Interaction Techniques Events
Devices
I Valuator
! MenusI 'con I Bu"ons IC'ick I Drag Ixt -Po,
! Input
Se,eoI,on I Pick
IButIonup,Oon IMo,io
IKeyboardlTablet IMousel Touch-Screen IJoystick[ Track-Ball ~/., Fig. 1. Reference model for 2D-interaction.
and other pointing devices provide elementary input events such as button up or down, motion, or key press. Based upon these elementary input events, fundamental interaction techniques were developed such as menus, icons, buttons, clicking, dragging, and so on. The next level are interaction tasks such as selection, picking, or valuator. Interaction tasks can be realized with a variety of different interaction techniques used to introduce interaction metaphors such as windowing, direct manipulation, or the desktop metaphor. Many 2D user interface toolkits such as O S F / M O T IF and Open Look (together with the metaphors they embody) are available, and provide a standard Look and Feel to a wide variety of applications including CAD, office systems, and process control. Their 2D user interface technology has become mature and fully developed. 2.2. Three-dimensional interactions The technology of 3 D interaction has just begun the same process of development as was applied to 2D interaction 10 years ago. We need to fill the gaps at each level of 3D interaction as was done in the past for 2D interaction. As input events in 3D virtual environments we see hand gestures as one category of events; there are others of course. Input events are the measurable behaviour of input devices such as position, mouse button presses, or positions of the fingers of a DataGlove. Interaction techniques can be defined as the meaningful combination of hand gestures, natural echo on the screen, and the manipulation of objects within the virtual environment. Interaction tasks include those to navigate within the scene, to identify an object, to move it from one place to another or to modify it. Examples of interaction techniques include grabbing an object, or releasing it (see also Fig. 2). As yet, we know of only a
few 3D metaphors such as virtual rooms or the principle of Direct Manipulation. Further metaphors and paradigms for 3D interaction and dialogue modeling are required. One of the most important technical issues in bringing Virtual Reality into a wide range of applications is the availability of toolkits. Only the wide availability of toolkits will enable the development of 3D-applications in a time and cost-effective manner. 3. ADVANCED INTERACTIONTECHNIQUES There are many examples of computer graphics applications in which the user's ability to interact with models is as important as the quality of the scene's images themselves. However, the interactive component of virtual reality is still behind the imagery and behavioral components. In this section we present our ideas to enhance interactions with three-dimensional objects in the virtual world for attaining natural and intuitive interactions. Most of the ideas described below are realized in the system GIVEN. GIVEN (Gesture based Interaction in Virtual ENvironments [1]) is being developed at the Computer Graphics Centre in Darmstadt-Germany, partly in cooperation with the Graphical Systems and Methods Group at the University of Coimbra-Portugal. The system implements a toolkit to assist in building virtual worlds and is used also as a testbed for 3D interaction concepts and methods. The visitor of a virtual environment created with this toolkit can navigate and manipulate objects in the environment by gesturing with his hand. 3,1. Precise collision detection as a f u n d a m e n t a l One of the main goals of virtual reality applications is to give the user the feeling of direct interaction with three-dimensional "real" objects as natural as possible.
Advanced interaction techniques in virtual environments
Applications
lOAD
657
IsimuIa,ionlcien,,.cMoe,in.lvirtua,
UI-Toolkits
Metaphors Interaction Tasks Interaction Techniques Events
3D-MenulvirtualSpherelPointinGrabbing g I ReleaseI GestureLanguage~ I 2D.MotionlButton click[TorquelForceI 6DMotionI Gestures
Joystick 13O-UouselspaceBal, lDataG,oveI 6D-TrackerlEye-Tracker ~
Input Devices
I
Fig. 2. Reference model tbr 3D-interaction.
The user of these systems wants to see simulated objects acting as if they were impenetrable and sense limits to motion and actions. Therefore, the fundamental of recognizing interactions between dynamic and static or dynamic and dynamic objects is collision checking. A very c o m m o n way to detect collision in virtual
Active Objcctg
Static Ob~eet.s
. . . . . . .
:
Collision
('.ollision Delectitm with Wor~t-Caxe Volumes
.
Handler Based on Bounding Volumes
~
Detection
(CDBV)
Pairs of Objects
Collision Derectio. with ,qpecific
Volum~ ~
L i
[.ist of' t)bjectS
e-
} : i
Polylledrical Intersection Module
(PtM)
',
ii ii
,= .,~
Ligt of Colliding Objects
Fig. 3. Precise collision detection manager's architecture.
environments is to use bounding boxes around objects, which are parallel to the coordinate axes. For example, in the first version of the G I V E N toolkit, the collision detection module (CDBV) was also based on bounding volumes implemented hierarchically with two levels of checking (Fig. 3), using worst-case and speci/ic bounding volumes. Using bounding volumes allows us to make collision checks very fast, but tbr a wide range of applications this is not sufficient to achieve supposed naturalness interactions in virtual worlds. Bounding volumes can be effectively used to state that two objects cannot intersect, however we cannot guarantee that two objects collide just because their bounding volumes intersect. For that reason, very unnatural and imprecise object interactions and situations can happen. Compared to interactions within applications like "'walk through in a virtual kitchen," interactions for applications like molecular modeling or solid modeling must be absolutely precise. When opening a cabinet, the interaction does not have to be 100% exact, but tbr controlling a robot that is repairing something dangerous, precision is fundamental. The dual issues of real-time and precision were tackled and a solution was found. In this way, the original hierarchical collision detection manager of the system G I V E N was extended in order to detect precisely which are the objects colliding in the virtual environment (see Fig. 3). The Precise Collision Detection Manager uses a boundary representation of solids for the implementation of a intersection algorithm. Collision detection is now made very precisely by intersecting two polyhedral objects and determining the vertices, edges, and faces where the intersection curve lies [4]. The precise collision detection algorithm works as follows:
658
M. FIGUEIREDO, K. BOHM and J. TEIXEIRA
for each moving object A that might collide with object B do calculate the Intersection Volume for object A and B for every face v o l u m e f v i o f A (i = 1 .. n, n - number of faces of object A) if face volume/i~i do not intersect the Intersection Volume then filter out this face for every face volume gv, o f B (i = 1 .. m, m - number of faces of object B) if face volume gv~ do not intersect the Intersection Volume then filter out this face for every face volume fv~ o f A (i = 1 .. n', n ' < n) for every face volume gvj o f B ( j = 1 . . m', m' -< m) if face volume fv, do not intersect face volume gvj then filter out this face pair intersect the remaining face pairs With this manager we are able to find out precisely if two objects are interacting or not enabling the user of applications that are based on the G I V E N toolkit to interact very precisely with the objects in the virtual environment. For applications such as telepresence this feature is very important, giving the opportunity to the operator of such systems to control very precisely their actions upon the environment and with improved safety. In an example illustrated in Fig. 4 the operator is manipulating a test tube and positioning precisely in an array of boxes. If we had bounding volumes around the object the user would be unable to go inside it. Naturalness of interaction with objects in virtual environments is also improved. As illustrated in Fig. 5, the user is able to touch the teapot and grab it in the same way he would normally do, because the user is able to get as close to objects as in real life. Furthermore, if a dataglove with tactile sensors is used, the user will have a real feeling of touching the objects being simulated. In this way, the user involvement in the virtual reality application will be greater because he will receive expectable behaviour from the objects in the virtual environment. He or she will directly manipulate virtual objects by pushing, grabbing, and releasing them only when touching its surface and not when (s) he is touching its bounding volumes.
3.2. Natural object interactions Hand gestures are a natural means for humans to manipulate the physical environment and using gestures offers a natural way to interface with virtual reality applications. For example in the G I V E N toolkit a small set of hand gestures is used by the visitor to perform operations such as navigating, grabbing, and releasing objects. This set of gestures constitutes a gesture language (Fig. 6) with which the visitor of a virtual environment can tell his intentions to the system. Thus, the visitor can easily "walk" around in the virtual world performing the appropriate gesture for navigating forward or backward and pointing in the desired direction, for example. He is also able to manipulate simulated threedimensional objects. An object can be grabbed using a "grab" gesture and later release it performing a "release" gesture. For navigating, reseting position, and to perform fast, but sometimes imprecise manipulations of 3D objects, gestures are well-suited. The user can communicate to the virtual environment his intentions using all the time the same input device, e.g., the dataglove, without any need to use complementary devices such as a keyboard or a mouse. In this way, the user is more concentrated in "visiting" and exploring the simulated environment rather than on which input
Fig. 4. Precise manipulation of a test tube in a hazardous environment,
Fig. 5. Natural interaction with a teapot in the virtual environment.
Advanced interaction techniques in virtual environments
Navigate
Navigate
forward
~ r d
Grab object
Rcle~ue
obic, t
659
"rum
Reset hand-curmt poaition
Fig. 6. Examples of a gesture language.
devices should be used at any moment. However, the manipulation of 3D objects with gestures does not reflect the interaction of objects in the real world, thus it is not natural. We could enhance the user involvement and fully create the illusion of immersion in a virtual world if we also allow the "visitor" to directly manipulate virtual objects without using gestures in the same way he is used to manipulate real objects. Our experiments revealed this desire to achieve natural interactions and we made the modest suggestion that interaction with virtual objects should also be realized in the same way as humans do in real life. The
:a-:,~.
user should be able to "touch" and surround virtual objects and grab them by touching them with the thumb or/and fingers, which is more natural then performing gestures. To find the best solution for this problem we need an input device (e.g., dataglove) with force feedback. Currently there is no dataglove with force feedback available on the market. Therefore, we developed a technique where we do not need force feedback. Our solution to accomplish natural interaction in virtual environments is based on the precise collision detection manager. The technique is called "surrounding object." 3.2.1. Surrounding technique. As virtual environ-
z.,~,:.--.--~ o'b
,
(a)
(b) Fig. 7. (a) Moving back the virtual hand in the direction of the last translation: (b) moving the virtual hand in the direction of the object's surface normal.
660
M. FIGUEIREDO,K. BOHM and J. TEIXEIRA
ments create a compelling illusion of a real. interactive world, the question of defining and characterizing the interactions between objects in the environment and the user is of great interest. In particular, there is a desire to let the user "'feel" and " t o u c h " objects. This would allow the user to surround or "walk," with the "'virtual" hand for example, on object's surfaces and perceive different material characteristics. The availability of a precise collision detection manager makes it possible for a "visitor" of a virtual environment to "'touch" an object's surface with the "virtual" hand. However, when the user is approaching a surface he will probably be inside the surface and not touching it because we can test the collision only by single time steps (Fig. 7). To give him the illusion that a solid surface is present the virtual hand should be moved back to the object's surface. Several techniques to move back can be used: It can be moved
~
l
~
Fig. 9. Pushing an object.
Fig. 8. Surrounding an object.
back as in Fig. 7a; or the object's surface information like the face normal can be used and the virtual hand is moved back as in Fig. 7b. We implemented the second approach, which lets the user walk around and surround objects easily. In this way, it is possible to surround objects without any force-feedback interaction devices. In Fig. 8 a picture is presented where it is possible to see the virtual hand surrounding a virtual object. Furthermore, this same information can be used when pushing an object. In Fig. 9 a sequence of pushing is presented. The object is moved away from the hand, and the hand can continue its m o v e m e n t without any object on front of it. In this way, with the surrounding technique the user can "walk" with the virtual hand on the object's surface and look for the best position to grab it. An example is presented in Fig. 10 where the user
Advanced interaction techniques in virtual environments
661
Table I. Execution speed for grabbing interactions. Objects
Complexity
Operation
Time
scene thumb and cube cube and cylinder
700 polygons 30 + 6 polygons 6 + 22 polygons
navigation precise collision detection precise collision detection
50 msec/frame 270 msec/frame 130 msec/frame
is grabbing a virtual object using a grab gesture from the gestural language vocabulary ( Fig. 10a ), and without performing any specific gesture that allows him to grab objects in the best suited manner (Figs. 5 and 10b). 4. CONCLUSIONS Our experiments with the G I V E N toolkit showed us that very unnatural object interactions happen in virtual environments if collision detection is only supported with bounding volumes. To increase naturalness of interactions we have found a way to detect precise collisions supported by a precise collision detection manager. The "visitor" of the G I V E N environment can now interact very precisely with virtual objects. This allows
the user to get "close" to objects and have the "feeling" of touching virtual objects enabling him to: identify faces with a virtual hand; grab an object when he is touching its surface; and position objects very precisely. To enhance the user involvement in the virtual environment he was also enabled to grab objects without using any specific and elaborate hand gesture defined by a gestural language, which is more intuitive and close to the way humans are used to do. In general there is a desire to let the user "'feel" and " t o u c h " objects. We also introduced a technique that makes it possible for the user to surround objects without any force-feedback interaction devices. This technique allows the user to surround or "walk," with the virtual hand for example, on object's surface and perceive its shape. In this way, the user can "walk" with the virtual hand on the object's surface and look for the best position to grab it. To test the overall performance several scenes were constructed and execution time for various tasks were measured on a Silicon Graphics 4 D / 3 2 0 VGX (Table 1 ). As shown on this table acceptable speed is achieved when the moving object has a small number of polygons.
Acknmdedgments--The authors would like to thank Professor Dr.-Ing. Jos6 Encarna¢~o and Dr-Ing. Wolfgang Hfibner for their support to this work. REFERENCES
Fig. 10(a). Grabbing a virtual tea cup via a grab gesture.
Fig. 10(b). Grabbing a virtual tea cup, naturally, without any specific gesture.
1. K. B6hm, W. Htibner, and K. V~i~in~inen,GIVEN: Gesture driven interactions in virtual environments, a toolkit approach to 3D interactions. Proceedings qfthe Interface to Real and Virtual Worlds Con[erenee, pp. 253-254, Montpelier, March 23-27 (1992). 2. L. M. Brill, Facing interface issues. Comp. Graph. World, 15, 148-57 (April 1992). 3. S. Bryson, Survey of virtual environment technologies and techniques, hnplementation ~[ lmmersive rTrtual Environments, Course Notes 9, SIGGRAPH '92, pp. 1.11.35 (1992). 4. M. Figueiredo, K. B6hm, and J. Teixeira, Precise object interactions using solid modeling techniques. In Proceedings O/IFIP TC 5.0 11 Working Conference on Modeling in Computer Graphics, B. Falcidieno and T. L. Kunii (Eds.), pp. 157-176, Italy (1993). 5. J. D. Foley, The human factors of computer graphics interaction techniques. IEEE Comp. Graph. Appl., 4, 1348 (November 1984). 6. J. D. Foley, Interfaces for advanced computing. Sci. Am., 126-135 (1987). 7. W. Htibner, K. V~i~inanen, and V. Burril, Advanced user interfaces for 3D and multimedia interaction, in Zentrum ./~ir Graphisehe Datenverarbeitung e. V. ZGDV, J. Encarnacao and J. Foley (Eds.), Report 71/93 (1993). 8. Krueger, M. Artflieial Reality 11, Addison-Wesley, Reading, MA (1991).
0097-8493/93 $6.00 + .00 © 1993 Pergamon Press Lid
Printed in Great Britain.
Virtual Reality
ADVANCED INTERACTION TECHNIQUES IN VIRTUAL ENVIRONMENTS MAURO FIGUEIREDO, t KLAUS BOHM ¢, and JoSl~ TEIXEIRA+ *Grupo de M~todos e Sistemas Grfificos, Departamento de Matem~itica-Universidade de Coimbra, Apartado 3008, 3000 Coimbra, Portugal *Zentrum ffir Graphische Datenverarbeitung e.V. (ZGDV), Wilheminenstral3e 7, 6100 Darmstadt, Germany Abstract--Fundamental to much of the Virtual Reality work is, in addition to high-level 3D graphical and multimedia scenes, research on advanced methods of interaction. The "visitor" of such virtual worlds must be able to act and behave intuitively, as he would in everyday situations, as well as receive expectable natural behaviour presented as feedback from the objects in the environment, in a way that he/she has the feeling of direct interaction with his/her application. In this paper we present several techniques to enrich the naturalness and enhance the user involvement in the virtual environment. We present how the user is enabled to grab objects without using any specific and elaborate hand gesture, which is more intuitive and close to the way humans are used to do. We also introduce a technique that makes it possible for the user to surround objects without any force-feedback interaction device. This technique allows the user to surround or "'walk" with the virtual hand on the object's surface and look for the best position to grab it. 1. INTRODUCTION Virtual Reality is a new type of human-computer interface that aims at creating the illusion of the user being immersed in a computer generated reality, providing a more direct communication link between users and the problem environment modeled by the computer system. The user interacts with a world containing three-dimensional objects in three-dimensional space that respond interactively to each other and the user actions. The attainment of illusion of involvement and participation place severe demands on all elements of a computer system. The primary constraint is the desire for natural and intuitive interactions. There are many examples of computer graphics applications in which the user's ability to interact with the pictures is as important as the quality of the pictures themselves. The key idea is that the virtual environment is trying to create a different world that one interacts with as if it were real. The user is reaching out and grabbing, pointing, pushing, and talking to things just as we do in the real world. In this way, the computer system must sense the user as unobtrusively and naturally as possible. This paper focus the realization of precise interactions with three-dimensional objects in virtual worlds for achieving natural interactions. Current state-of-the-art for 3D interaction is based on collision checks using bounding volumes. When collision checks are based only on bounding volumes very unnatural interactions happen. We implemented a precise collision check using also the object's polygonal information. With this information we enable the user to " t o u c h " and "feel" object's surfaces. Naturalness of interactions was improved with the help of a technique called surrounding that enables the user to grab 3D objects easily and intuitively using a 3D input device (e.g.. dataglove, spaceball ).
2. INTERACYION IN VIRTUAL ENVIRONMENTS Current state-of-the-art graphic workstations allow the creation of almost photorealistic images in less than 100 ms: some can even render two images in the same amount of time for stereoscopic viewing. The development of new input devices, interaction techniques, and interaction metaphors that can exploit the power of these workstations is, however, in its infancy. Virtual reality makes use of these capabilities in order to give the user the feeling of being part of an environment in which h e / s h e is able to manipulate virtual objects directly. But, in our opinion, the successful use of virtual reality in a wide range of applications depends critically upon the quality of interaction that can be achieved between this new technology and the users "within" it. If this match is not achieved then the technology cannot hope to realise its full potential. We need to analyse and evaluate the current stateof-the-art for 3D-interaction in order to see how far development has progressed and to determine the requirements for applying Virtual Reality to future applications. In order to systematically analyse three-dimensional interactions, a classification of two-dimensional (2D) interaction is briefly discussed. This classification and the lessons learned are then applied to three-dimensional (3D) interaction. See also [7]. This process of classification is illustrated using a progression of Interaction level diagrams. These diagrams illustrate a conceptual model of the different levels of interaction within the various dimensions. See also [ 5 ].
2.1. Two-dimensional interactions During the last decade, a set of well-defined 2D user interface paradigms has been established based upon the 2D output technology of faster devices (Fig. 1 ). At the level of input devices, the keyboard, mouse, 655
656
M. FIGUEIREDO, K. BOHMand J. TEIXEIRA
O,,,ce
Applications
IPrccesOontro,
I o,,MoOe,,no
UI-Toolkits
Metaphors
I
I oo,,,o0 MetaphorsI Direct Manipulation I
Interaction Tasks II Interaction Techniques Events
Devices
I Valuator
! MenusI 'con I Bu"ons IC'ick I Drag Ixt -Po,
! Input
Se,eoI,on I Pick
IButIonup,Oon IMo,io
IKeyboardlTablet IMousel Touch-Screen IJoystick[ Track-Ball ~/., Fig. 1. Reference model for 2D-interaction.
and other pointing devices provide elementary input events such as button up or down, motion, or key press. Based upon these elementary input events, fundamental interaction techniques were developed such as menus, icons, buttons, clicking, dragging, and so on. The next level are interaction tasks such as selection, picking, or valuator. Interaction tasks can be realized with a variety of different interaction techniques used to introduce interaction metaphors such as windowing, direct manipulation, or the desktop metaphor. Many 2D user interface toolkits such as O S F / M O T IF and Open Look (together with the metaphors they embody) are available, and provide a standard Look and Feel to a wide variety of applications including CAD, office systems, and process control. Their 2D user interface technology has become mature and fully developed. 2.2. Three-dimensional interactions The technology of 3 D interaction has just begun the same process of development as was applied to 2D interaction 10 years ago. We need to fill the gaps at each level of 3D interaction as was done in the past for 2D interaction. As input events in 3D virtual environments we see hand gestures as one category of events; there are others of course. Input events are the measurable behaviour of input devices such as position, mouse button presses, or positions of the fingers of a DataGlove. Interaction techniques can be defined as the meaningful combination of hand gestures, natural echo on the screen, and the manipulation of objects within the virtual environment. Interaction tasks include those to navigate within the scene, to identify an object, to move it from one place to another or to modify it. Examples of interaction techniques include grabbing an object, or releasing it (see also Fig. 2). As yet, we know of only a
few 3D metaphors such as virtual rooms or the principle of Direct Manipulation. Further metaphors and paradigms for 3D interaction and dialogue modeling are required. One of the most important technical issues in bringing Virtual Reality into a wide range of applications is the availability of toolkits. Only the wide availability of toolkits will enable the development of 3D-applications in a time and cost-effective manner. 3. ADVANCED INTERACTIONTECHNIQUES There are many examples of computer graphics applications in which the user's ability to interact with models is as important as the quality of the scene's images themselves. However, the interactive component of virtual reality is still behind the imagery and behavioral components. In this section we present our ideas to enhance interactions with three-dimensional objects in the virtual world for attaining natural and intuitive interactions. Most of the ideas described below are realized in the system GIVEN. GIVEN (Gesture based Interaction in Virtual ENvironments [1]) is being developed at the Computer Graphics Centre in Darmstadt-Germany, partly in cooperation with the Graphical Systems and Methods Group at the University of Coimbra-Portugal. The system implements a toolkit to assist in building virtual worlds and is used also as a testbed for 3D interaction concepts and methods. The visitor of a virtual environment created with this toolkit can navigate and manipulate objects in the environment by gesturing with his hand. 3,1. Precise collision detection as a f u n d a m e n t a l One of the main goals of virtual reality applications is to give the user the feeling of direct interaction with three-dimensional "real" objects as natural as possible.
Advanced interaction techniques in virtual environments
Applications
lOAD
657
IsimuIa,ionlcien,,.cMoe,in.lvirtua,
UI-Toolkits
Metaphors Interaction Tasks Interaction Techniques Events
3D-MenulvirtualSpherelPointinGrabbing g I ReleaseI GestureLanguage~ I 2D.MotionlButton click[TorquelForceI 6DMotionI Gestures
Joystick 13O-UouselspaceBal, lDataG,oveI 6D-TrackerlEye-Tracker ~
Input Devices
I
Fig. 2. Reference model tbr 3D-interaction.
The user of these systems wants to see simulated objects acting as if they were impenetrable and sense limits to motion and actions. Therefore, the fundamental of recognizing interactions between dynamic and static or dynamic and dynamic objects is collision checking. A very c o m m o n way to detect collision in virtual
Active Objcctg
Static Ob~eet.s
. . . . . . .
:
Collision
('.ollision Delectitm with Wor~t-Caxe Volumes
.
Handler Based on Bounding Volumes
~
Detection
(CDBV)
Pairs of Objects
Collision Derectio. with ,qpecific
Volum~ ~
L i
[.ist of' t)bjectS
e-
} : i
Polylledrical Intersection Module
(PtM)
',
ii ii
,= .,~
Ligt of Colliding Objects
Fig. 3. Precise collision detection manager's architecture.
environments is to use bounding boxes around objects, which are parallel to the coordinate axes. For example, in the first version of the G I V E N toolkit, the collision detection module (CDBV) was also based on bounding volumes implemented hierarchically with two levels of checking (Fig. 3), using worst-case and speci/ic bounding volumes. Using bounding volumes allows us to make collision checks very fast, but tbr a wide range of applications this is not sufficient to achieve supposed naturalness interactions in virtual worlds. Bounding volumes can be effectively used to state that two objects cannot intersect, however we cannot guarantee that two objects collide just because their bounding volumes intersect. For that reason, very unnatural and imprecise object interactions and situations can happen. Compared to interactions within applications like "'walk through in a virtual kitchen," interactions for applications like molecular modeling or solid modeling must be absolutely precise. When opening a cabinet, the interaction does not have to be 100% exact, but tbr controlling a robot that is repairing something dangerous, precision is fundamental. The dual issues of real-time and precision were tackled and a solution was found. In this way, the original hierarchical collision detection manager of the system G I V E N was extended in order to detect precisely which are the objects colliding in the virtual environment (see Fig. 3). The Precise Collision Detection Manager uses a boundary representation of solids for the implementation of a intersection algorithm. Collision detection is now made very precisely by intersecting two polyhedral objects and determining the vertices, edges, and faces where the intersection curve lies [4]. The precise collision detection algorithm works as follows:
658
M. FIGUEIREDO, K. BOHM and J. TEIXEIRA
for each moving object A that might collide with object B do calculate the Intersection Volume for object A and B for every face v o l u m e f v i o f A (i = 1 .. n, n - number of faces of object A) if face volume/i~i do not intersect the Intersection Volume then filter out this face for every face volume gv, o f B (i = 1 .. m, m - number of faces of object B) if face volume gv~ do not intersect the Intersection Volume then filter out this face for every face volume fv~ o f A (i = 1 .. n', n ' < n) for every face volume gvj o f B ( j = 1 . . m', m' -< m) if face volume fv, do not intersect face volume gvj then filter out this face pair intersect the remaining face pairs With this manager we are able to find out precisely if two objects are interacting or not enabling the user of applications that are based on the G I V E N toolkit to interact very precisely with the objects in the virtual environment. For applications such as telepresence this feature is very important, giving the opportunity to the operator of such systems to control very precisely their actions upon the environment and with improved safety. In an example illustrated in Fig. 4 the operator is manipulating a test tube and positioning precisely in an array of boxes. If we had bounding volumes around the object the user would be unable to go inside it. Naturalness of interaction with objects in virtual environments is also improved. As illustrated in Fig. 5, the user is able to touch the teapot and grab it in the same way he would normally do, because the user is able to get as close to objects as in real life. Furthermore, if a dataglove with tactile sensors is used, the user will have a real feeling of touching the objects being simulated. In this way, the user involvement in the virtual reality application will be greater because he will receive expectable behaviour from the objects in the virtual environment. He or she will directly manipulate virtual objects by pushing, grabbing, and releasing them only when touching its surface and not when (s) he is touching its bounding volumes.
3.2. Natural object interactions Hand gestures are a natural means for humans to manipulate the physical environment and using gestures offers a natural way to interface with virtual reality applications. For example in the G I V E N toolkit a small set of hand gestures is used by the visitor to perform operations such as navigating, grabbing, and releasing objects. This set of gestures constitutes a gesture language (Fig. 6) with which the visitor of a virtual environment can tell his intentions to the system. Thus, the visitor can easily "walk" around in the virtual world performing the appropriate gesture for navigating forward or backward and pointing in the desired direction, for example. He is also able to manipulate simulated threedimensional objects. An object can be grabbed using a "grab" gesture and later release it performing a "release" gesture. For navigating, reseting position, and to perform fast, but sometimes imprecise manipulations of 3D objects, gestures are well-suited. The user can communicate to the virtual environment his intentions using all the time the same input device, e.g., the dataglove, without any need to use complementary devices such as a keyboard or a mouse. In this way, the user is more concentrated in "visiting" and exploring the simulated environment rather than on which input
Fig. 4. Precise manipulation of a test tube in a hazardous environment,
Fig. 5. Natural interaction with a teapot in the virtual environment.
Advanced interaction techniques in virtual environments
Navigate
Navigate
forward
~ r d
Grab object
Rcle~ue
obic, t
659
"rum
Reset hand-curmt poaition
Fig. 6. Examples of a gesture language.
devices should be used at any moment. However, the manipulation of 3D objects with gestures does not reflect the interaction of objects in the real world, thus it is not natural. We could enhance the user involvement and fully create the illusion of immersion in a virtual world if we also allow the "visitor" to directly manipulate virtual objects without using gestures in the same way he is used to manipulate real objects. Our experiments revealed this desire to achieve natural interactions and we made the modest suggestion that interaction with virtual objects should also be realized in the same way as humans do in real life. The
:a-:,~.
user should be able to "touch" and surround virtual objects and grab them by touching them with the thumb or/and fingers, which is more natural then performing gestures. To find the best solution for this problem we need an input device (e.g., dataglove) with force feedback. Currently there is no dataglove with force feedback available on the market. Therefore, we developed a technique where we do not need force feedback. Our solution to accomplish natural interaction in virtual environments is based on the precise collision detection manager. The technique is called "surrounding object." 3.2.1. Surrounding technique. As virtual environ-
z.,~,:.--.--~ o'b
,
(a)
(b) Fig. 7. (a) Moving back the virtual hand in the direction of the last translation: (b) moving the virtual hand in the direction of the object's surface normal.
660
M. FIGUEIREDO,K. BOHM and J. TEIXEIRA
ments create a compelling illusion of a real. interactive world, the question of defining and characterizing the interactions between objects in the environment and the user is of great interest. In particular, there is a desire to let the user "'feel" and " t o u c h " objects. This would allow the user to surround or "walk," with the "'virtual" hand for example, on object's surfaces and perceive different material characteristics. The availability of a precise collision detection manager makes it possible for a "visitor" of a virtual environment to "'touch" an object's surface with the "virtual" hand. However, when the user is approaching a surface he will probably be inside the surface and not touching it because we can test the collision only by single time steps (Fig. 7). To give him the illusion that a solid surface is present the virtual hand should be moved back to the object's surface. Several techniques to move back can be used: It can be moved
~
l
~
Fig. 9. Pushing an object.
Fig. 8. Surrounding an object.
back as in Fig. 7a; or the object's surface information like the face normal can be used and the virtual hand is moved back as in Fig. 7b. We implemented the second approach, which lets the user walk around and surround objects easily. In this way, it is possible to surround objects without any force-feedback interaction devices. In Fig. 8 a picture is presented where it is possible to see the virtual hand surrounding a virtual object. Furthermore, this same information can be used when pushing an object. In Fig. 9 a sequence of pushing is presented. The object is moved away from the hand, and the hand can continue its m o v e m e n t without any object on front of it. In this way, with the surrounding technique the user can "walk" with the virtual hand on the object's surface and look for the best position to grab it. An example is presented in Fig. 10 where the user
Advanced interaction techniques in virtual environments
661
Table I. Execution speed for grabbing interactions. Objects
Complexity
Operation
Time
scene thumb and cube cube and cylinder
700 polygons 30 + 6 polygons 6 + 22 polygons
navigation precise collision detection precise collision detection
50 msec/frame 270 msec/frame 130 msec/frame
is grabbing a virtual object using a grab gesture from the gestural language vocabulary ( Fig. 10a ), and without performing any specific gesture that allows him to grab objects in the best suited manner (Figs. 5 and 10b). 4. CONCLUSIONS Our experiments with the G I V E N toolkit showed us that very unnatural object interactions happen in virtual environments if collision detection is only supported with bounding volumes. To increase naturalness of interactions we have found a way to detect precise collisions supported by a precise collision detection manager. The "visitor" of the G I V E N environment can now interact very precisely with virtual objects. This allows
the user to get "close" to objects and have the "feeling" of touching virtual objects enabling him to: identify faces with a virtual hand; grab an object when he is touching its surface; and position objects very precisely. To enhance the user involvement in the virtual environment he was also enabled to grab objects without using any specific and elaborate hand gesture defined by a gestural language, which is more intuitive and close to the way humans are used to do. In general there is a desire to let the user "'feel" and " t o u c h " objects. We also introduced a technique that makes it possible for the user to surround objects without any force-feedback interaction devices. This technique allows the user to surround or "walk," with the virtual hand for example, on object's surface and perceive its shape. In this way, the user can "walk" with the virtual hand on the object's surface and look for the best position to grab it. To test the overall performance several scenes were constructed and execution time for various tasks were measured on a Silicon Graphics 4 D / 3 2 0 VGX (Table 1 ). As shown on this table acceptable speed is achieved when the moving object has a small number of polygons.
Acknmdedgments--The authors would like to thank Professor Dr.-Ing. Jos6 Encarna¢~o and Dr-Ing. Wolfgang Hfibner for their support to this work. REFERENCES
Fig. 10(a). Grabbing a virtual tea cup via a grab gesture.
Fig. 10(b). Grabbing a virtual tea cup, naturally, without any specific gesture.
1. K. B6hm, W. Htibner, and K. V~i~in~inen,GIVEN: Gesture driven interactions in virtual environments, a toolkit approach to 3D interactions. Proceedings qfthe Interface to Real and Virtual Worlds Con[erenee, pp. 253-254, Montpelier, March 23-27 (1992). 2. L. M. Brill, Facing interface issues. Comp. Graph. World, 15, 148-57 (April 1992). 3. S. Bryson, Survey of virtual environment technologies and techniques, hnplementation ~[ lmmersive rTrtual Environments, Course Notes 9, SIGGRAPH '92, pp. 1.11.35 (1992). 4. M. Figueiredo, K. B6hm, and J. Teixeira, Precise object interactions using solid modeling techniques. In Proceedings O/IFIP TC 5.0 11 Working Conference on Modeling in Computer Graphics, B. Falcidieno and T. L. Kunii (Eds.), pp. 157-176, Italy (1993). 5. J. D. Foley, The human factors of computer graphics interaction techniques. IEEE Comp. Graph. Appl., 4, 1348 (November 1984). 6. J. D. Foley, Interfaces for advanced computing. Sci. Am., 126-135 (1987). 7. W. Htibner, K. V~i~inanen, and V. Burril, Advanced user interfaces for 3D and multimedia interaction, in Zentrum ./~ir Graphisehe Datenverarbeitung e. V. ZGDV, J. Encarnacao and J. Foley (Eds.), Report 71/93 (1993). 8. Krueger, M. Artflieial Reality 11, Addison-Wesley, Reading, MA (1991).