Shared virtual environments: The implications for tool builders

Cornput. & Graphics, Vol. 20, No. 2, pp. 185-189, 1996 Copyright Q 1996 EIse.vierScienceLtd Printed in Great Britain. All rights reserved 0097-8493/96 $lS.OO+0.00 00974493(9s)oo124-7

Techniques for Virtual Environments

SHARED

VIRTUAL

ENVIRONMENTS: THE IMPLICATIONS FOR TOOL BUILDERS MARK GREEN

Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada e-mail: [email protected] Abstract-The importance of software development tools will increase as virtual reality moves from the research stage to real world applications. A good set of software tools is required for the cost effective development of commercial VE applications, and without them it is unlikely that VR will develop into a commercially viable technology. A large number of the envisioned applications of VR involve some type. of collaboration. This collaboration may take the form of shared design spaces, medical consulting, games or shopping, but whatever the form, it places an extra demand on the development process. This paper presents some of the work that we have done in the area of software tools to support the development of shared virtual environments. These tools have been used by numerous groups to develop multi-user applications. Two of the key problems in the development of shared virtual environments, real-time and concurrency control, are discussed along with their implications fi,r software tools. Copyright 0 1996 Elsevier Science Ltd

1. INTRODUCTION

Of all the applications of virtual reality (VR), the ones generating the most interest are those involving multiple userssharingthe sameenvironment. These applications include conferencing, shared design spaces, various types of training simulators, games and simulated travel. With the advance of high speed networks, and the resulting demandfor interactive entertainment, shared virtual environments will

difficult task, with even fewer software tools to support it. Over the past 3 years we have been developingsoftwaretools, aspart of the MR Toolkit [ 11,for the developmentof sharedenvironments.This effort was largely motivated by severalof our users, and these tools have been used in the production

of

well known applications[2]. Thesetools are briefly describedin the following section, and the third section outlines what we considerto be the major becomea very active area of researchand develop- challengesin the developmentof software tools for ment. In theseapplications,userssharethe same3-D shared virtual environments. spaceand can interact with each other. This adds an

extra degree of complexity to the development of these applications,sinceeachusermust be informed of all the other users’ activities, and there must be

2. CURRENT RwEARcH

This sectionsummarizessomeof the work that we have done on sharedvirtual environments.It covers

support for joint actionsinvolving two or more users. some of things that we have learned from producing There is also the possibility that two or more users tools to support the developmentof theseenvironcould perform conflicting actions, resulting in contradictions in the environment. For example, if two users grab the samechair at the sametime and move it to two different locations, what is the final position of the chair? There is a need for concurrency control

ments, and provides the background discussed in the next section.

mechanisms that prevent this situation, but still allow two or more usersto cooperateon moving the chair to a new common position. If the two usersare at remotelocations,the applicationmust alsodealwith the problem of network lag, making it difficult to

in shared virtual environments and was motivated by a request from one of our main users. In 1989 we agreed to assist the Banff Centre for the Arts in a major project applying VR technology to the arts.

determine

involved

whether

the two

users simultaneously

for the issues

2.1. The MR Peers package

The MR P’eerspackage[3] wasour first experiment

Several of the projects proposed for this study multi-user

environments,

with

the best

grabbedthe chair, or one moved it after the other. These problems are discussedfurther in the third

known of thesebeing the Placeholderproject [2]. In response to this requestwe designedand implemen-

section of this paper.

ted the Peers package that became part of the MR

The development of single user virtual environ- Toolkit. ments, and the associatedsoftware tools, have The MR Toolkit proven to be a challenge. The development of multi-user environments has proven to be a more

is a low level toolkit supporting the development of VR applications. It provides drivers for standard VR devices, some standard

185

186

M. Green

interaction techniques,support for distributed applications, and other servicesby VR applications.One of the early designdecisionsin MR wasto provide support for distributed applications,in the sensethat a single user application could be distributed over multiple local workstations.The various components of the application communicate using a shared memory model. The original design of the MR Toolkit did not consider multi-user environments, sincewe felt that there wereenoughproblemsto deal with in singleuserenvironments.In order to handle multi-user environments, MR’s shared memory model was extended to allow multiple applications to sharethe samedata structure. This is basedon a replicated architecture, where each user runs his or her own copy of the application. The UDP layer of the IP protocol family is used as the basis for communicationsbetween the local versions of the application. The main reasonfor choosingthis layer was to reducethe lag in communicationsover wide areanetworks. When a userjoins a session,his or her copy of the application performs a rendezvousat a known port number on a known machine, and requeststo be addedto the session.By default, when the new user joins the session,all the other applications are informed, and a fully connectedgraph configuration is formed betweenall the applications.At this point, the new application requeststhe shareddata structuresthat it is interestedin, and the link betweenthe applications is formed. By default, all the current user’sinput devicesare echoedto all the other users. The main benefit of this approach is that it had a minimal impact on MR, making it quite easy to convert existing singleuserapplicationsto multi-user ones. The Peerspackagehas beenavailable for several years, and has beenusedin severalapplications.At the 1993 CyberConf we demonstrateda multi-user handball game that used the Peers package.This demonstration involved four players located at Austin Texas, Stanford California, Banff Alberta, and Edmonton Alberta. The Peerspackagewas also usedin the Placeholderproject. There are severalimportant things that we learned from this experience.The use of UDP significantly reduced network lag, an important consideration when there is a considerabledistancebetweenusers. Switching to a reliable protocol, such as TCP, can more than doublethe lag, sincean acknowledgement is required for each packet. When using UDP, lag can be noticeable,therefore, usinga more expensive protocol could destroy the feel of the interaction. In the case of local area networks the use of an unreliable protocol, such as UDP, is not a major problem, sinceerrors are rare. This is not the case when wide area networks are used,sincetheir error rates can be severalordersof magnitudegreater. The importance of reliability dependsupon the type of data being transmitted. This was nicely illustrated by the handball game.The position data

for eachuserdoesnot needto be reliably transmitted, sinceit is generatedon each update. If a position packet from one of the usersis lost or corrupted, there is no problem, sincea new packet will arrive in 0.1 s. In this gamea token is usedto indicate the current player. Only the current player can hit the ball and scorepoints. After the current player hits the ball, the token is passedto the next player. If the packet containing the token is lost, a deadlock situation arises,since none of the players can hit the ball. This problem rarely occurson a local area network, but occursquite frequently whenwide area networks are used. This problem was solved by construcing a special reliable sub-protocol for the token data. Our experiencesindicate that the type of protocol required dependsupon the type of network being usedand the characteristicsof the data. In general, we want to usethe cheapestprotocol for eachpieceof data in order to minimizenetwork delays.Providing automatic support for protocol selectionis an area that still needsmore research. 2.2. The environment manager

An environment manager is a program that is responsiblefor running and managingan environment. This is a key componentin the set of higher level softwaretools that we are currently developing. These tools allow programmersand non-programmers to develop virtual environments using interactive 3-D editors [4]. The application developercan use an interactive geometricalmodeler, JDCAD + , to define the geometry of the objects in the environment, and the sameprogram can be usedto add simple behaviors to these objects. The OML (Object Modeling Language)programminglanguage isusedto representall the objectsin the environment. The output from JDCAD+ is an OML program, allowing application developersto use either programming, interactive tools, or a combination of thesetechniquesto produceobjects.OML programs are converted to a byte code that is interpreted at run-time, and this interpretation processis managed by the environmentmanager. Our current environmentmanager,called EM [5], can be usedto construct both singleuserand shared environments.The OML object descriptionsare the samefor both singleuserandmulti-userapplications. The main input to EM is a script that specifiesthe objects in the environment, the behaviors to be executedin responseto useractionsand eventsin the environment, and in the caseof multi-user environments, the information that can be sharedbetween users.For each object, the designercan specify the subsetof its properties that are accessibleto other users.Other userscan be given read, write, or read/ write accessto individual propertiesof each object. Accessto all or part of the object can alsobe granted by token passing.This givesthe designera considerable amount of flexibility in accessspecification. Usingthis scriptinglanguage,both programmersand

Shared virtual environments non-programmers can quickly develop multi-user applications. One of the more interesting results of this research is the ability to move object code from one user to another. Since OML is an interpreted language, it is possible to send the OML code from one user to another, without concern for the type of workstation they are using. This can occur while the environment is running, so all the users do not need to have a complete set of objects when the environment starts running, and objects can be modified or added to the environment at run-time.

187

The experimentsthat have been run using this modelindicatethat even for a smallnumberof users this approachcan significantly reducethe amount of network traffic. This reduction in trafhc is particularly evident in caseswhereone or more of the users isinteractively manipulatingone of the objectsin the environment.In a flat environment, the new state of this object must be transmittedto eachuseron each update,and the local versionof its modelupdated.If the usercannot seethe objectbeingmanipulated,this is a wastedeffort, sinceat bestthe useronly needsto beinformed of the object’sfinal state. Without some type of structure, as provided by the virtual place 2.3. The Virtual Place Model model, environments that support direct object In her thesis,Yunqi Sun [6] proposedan architec- manipulationwill be unworkable. ture for multi-uservirtual environments,and one of 3. TWO KEY PROBLEMS the important componentsof this architecture is the Virtual Place Model. This modelpartitions a virtual This section discusses the two problemsthat we environment into a number of logical places,where feel are most important from the perspective of each place is a 3-D sub-volume of the entire software tools for the construction of multi-user environment. Places can be nested hierarchically, environments.Theseproblemsshould be considered and there can be links betweenplaces,allowing the by all researchersattempting to develop tools to userto view the activitiesin anotherplace.Theselinks support this type of application, and their solution need not correspond to geometrical relationships will be a major advance. betweenthe spaces,therefore, placesthat are geometrically separatedcan still be seenfrom the current 3.1. Real-time place. A placecan be either closedor open, with the Real-time is a problem in single user virtual distinction that the usercannot seethe contentsof a environments, but is more serious in multi-user neighbouringspacefrom insideof a closedspace.As environmentssinceit can prohibit the interactions part of her thesis research Sun implementedthis betweenseveralusers.There are two aspectsto realarchitecture on top of the MR Toolkit, and time in virtual environments.The first aspect,which performed a number of experimentalstudies. is commonto all real-timesystems,is the problem of One of the major contributions of this modelis the computations with hard deadlines.That is, a VR ability to control network traffic and response time as systemmust maintain a certain update rate, and in the size of the environment and number of users order to accomplishthis, it must setdeadlinesfor its grows. In a multi-userenvironment, eachusermust components.If it is to maintain a constant update be aware of any actions performed by other users rate, of say 10updates/s,all the computationsfor the that might possibly effect them. If the environment current update must be completedwithin 100 ms. has no structure, then by default, each user must The application must obtain input from the user, monitor the activities of all the other users,requiring perform any application specific operations, and a packet of information from each user on each display the current state of the environment within update. As the number of usersin the environment this time limit. The application must be able to grows, the network traffic will grow quadratically, allocatecomputationalresourcesto the various tasks and will quickly exhaustthe network bandwidth and to be performed so that all the taskscompletewith the local ability to handle this traffic. The virtual acceptableresultswithin their allocatedtime. This is placemodel imposesa structure on the environment a problemthat mustbe addressed by both singleuser that reduces the network traffic. From the user’s and multi-userVR applications.In the caseof single current place,it is possibleto efficiently computethe userapplications,the tasksto be performed in each setof all placesthat arepotentially visibleto the user, update are pre-determined,and do not changefrom and thus the placesthe usermust be aware of. The one update to another. This is not the casein multicurrent useronly needsto monitor the actionsof the user environments,since other userscan generate usersin theseplaces,and doesnot needto receiveany tasks that must be handled by the current user’s of the network traffic from the other usersin the application. This greatly complicatesthe scheduling environment. For a large environment with a problem, sincethe setof tasksto be scheduledin the significant number of usersthis can greatly reduce current update is not tied, and can vary from one the amount of network traffic. When the usermoves update to the next. Schedulingalgorithms for this from one placeto another, there could be a burst of type of problemarenot aswell developedasthosefor network traflic as the useris updated to the current the problemspresentedby singleuserenvironments. situation in the new place,but this is a relatively rare For example,in the caseof multi-userenvironments, event that will not have a significant effect on the amount cf computation performed in an update network traffic. often dependsupon the number of usersthat the

M. Green

188

current user is interacting with. In addition, if several users are trying to synchronize their actions, there may be a considerable demand on computation in each update. The other aspect of real-time is the relationship between simulation time and clock on the wall time. This is a problem that has rarely been addressed in VR research, but is crucial to the success of multiuser environments. Most VR applications involve some amount of simulation, which progresses at a certain time rate. The rate at which the simulation progresses must be matched to the real clock on the wall time, and this relationship must be the same for all users. If this is not the case, at the same real instant in time two users could see two different states of the simulation. This problem can occur in even simple applications, such as walk throughs. Consider the case of two users, at remote sites, walking through the design of a building together. It is likely that the two users will not have the same equipment configuration, and thus the update rates seen by the two users will be different. Some walk through applications move the user at a rate that is a function of its update rate, and not the real velocity at which the user wishes to move. Thus, even in this simple application, it could be quite difficult for the two users to move through the building together. This problem is far more complicated if the application is running some form of simulation. For the user with the faster machine configuration, this simulation will progress much further in a fixed length of time, and the environment that this user sees could be very different from that of other users. This will make any form of meaningful collaboration impossible.

3.2. Concurrency

control

Concurrency control is not a problem that is restricted to virtual reality, but is a commontheme through many areasof computerscience.In a shared environment there is always the possibility that two or more userswill attempt to interact with a single object at the sametime. If theseinteractionscausethe object to changestate in any way, they will produce multiple inconsistent versions of the environment. For most applicationsthis is a situation that mustbe avoided. A large number of techniquesfor solving this problem have been developedin other areasof computer science(for a good survey from the CSCW point of view see[7]), but it is not clear how useful thesealgorithms will be for VR. For example, one commontechniqueis the useof locks, wherethe user wanting to manipulatean object must first obtain a lock on that object before they can start the operation. This lock prevents other users from manipulating the object, as long as the first user retains the lock. The problem with this approachis that it may take sometime (on the order of seconds) to obtain the lock, and it doesnot allow severalusers to cooperateon manipulating the sameobject.

For this problem, there is a lot that we can learn from previouswork in CSCW. SinceCSCW is quite close to VR, there is a good chance that the techniquesdevelopedin this area can be transferred to VR. This transfermay becomplicatedby the large number of potential usersof a sharedenvironment, and the size of these environments. Most of the applications addressedby CSCW involve a small number of usersin a restricted application domain. Sevemlof the anticipatedapplicationsof VR involve large numbersof users(hundreds)sharingthe same environment. It is not clear that the techniques developedfor CSCW will scale to this number of users. In addition, a number of anticipated VR applications are quite large, for example virtual travel or shopping,which allow the user to travel over large areasand interact with a large number of objects. It might be possible to partition these environments,as suggestedby Sun [6], in such a way that efficientconcurrencycontrol algorithmscan be developed. One of the morepromisingtechniquesemployedin CSCW is to rely on normal socialprotocols to help solve the concurrencycontrol problem. For example, if one userseesthat another user is manipulating an objeci, they will normally not attempt to grab the object.away from the other user.This approachmay be more effective in VR, since it is quite easy to observethe actionsof other users.If a userseesthat other usersare observingan object, then he or she will be temptedto ask for their permissionto modify the object before attempting any modifications.The successof this type of social protocol will depend upon the user community, and needsto be investigated further. One of the key obstaclesto the development of concurrencycontrol techniquesin VR is network lag. If the network has a delay of dt seconds,then any concurrency control schemewill have period of indeterminacy of length dt. If two usersgrab the same(objectat the sameinstant of time, they will not be aware of the other user’sactions for at least dt seconds.To be safe,the applicationmay needto wait for dt secondsbefore it allows the user to progress with his or her modifications,or be preparedto back off an,ymodificationsthat are madewithin the first dr secondsof interaction. In the first case,the interface will be sluggish,sincethe start of someinteractions will be delayed,and in the secondcase,the interface will not behavein a predictable way, sinceit must undo someof the user’sactions.This may be a case wherethe socialprotocols are the best, sincewe can often anticipate what other peopleare going to do, for ar. least a short period before the action is performed.In the caseof two usersgrabbing for the sameobject, both userswill be able to seethe other user move towards the object, and can anticipate someneed for coordination betweentheir actions. Thus we may only need to use strict concurrency control algorithms in the caseof antagonisticusers that do not follow the normal socialprotocols.

Sharedvirtual environmenis 4.

CONCLUSIONS

This paper presentssomeof the key researchtopics in shared environments.The presentation of these researchtopics is basedon our experiencesover the past severalyearsin the production of sharedvirtual environments and the software tools used to construct them. REFERENCES

1. C. Shaw,M. Green,J. Liang,andY. Sun, Decoupled simulation

in virtual reality. ACM Transactions on Systems 11, 287-317 (1993). 2. B. Laurel, R. Strickland, and R. Tow, Placeholder: landscape and narrative in virtual environments. Information

189

Compukr Graphics 28, 118-126 (1994). 3. C. Shaw and M. Green, The MR Peers package. VRAIS’93 Conference Proceedings, 463-469 (1993). 4. S. Halhday and M. Green, A geometric modeling and animation system for virtual reality. VRST’94 Procedings, 71-84 (1994). 5. Q. Wang, M. Green, and C. Shaw, EM-An environment manager for building networked virtual environments. VRAZS’95 Proceedings (1995). 6. Y. Sun, A Hierarchical Modelfor Virtual Environments, Ph.D Thesis, Department of Computing Science, University of Alberta (1995). 7. S. Greenberg and D. Marwood, Real time groupware as a distributed system:concurrency control and its effects on the interface. CSCW’94 Proceedings, 207-217 (1994).