- Email: [email protected]
Virtual reality movies-real-time streaming of 3D objects
Computer Networks 31 Ž1999. 2215–2225 www.elsevier.comrlocatercomnet
Virtual reality movies-real-time streaming of 3D objects S. Olbrich ) , H. Pralle Institute for Computer Networks and Distributed Systems (RVS), Regional Scientific Computing Center for Lower Saxony (RRZN), UniÕersity of HannoÕer, Schlosswender Str. 5, D-30159 HannoÕer, Germany
Abstract Powerful servers for computation and storage, high-speed networking resources, and high-performance 3D graphics workstations, which are typically available in scientific research environments, potentially allow the development and productive application of advanced distributed high-quality multimedia concepts. Several bottlenecks, often caused by the inefficient design and software implementation of current systems, prevent utilization of the offered performance of existing hardware and networking resources. We present a system architecture, which supports streamed online presentation of series of 3D objects. In the case of expensive simulations on a supercomputer, results are increasingly represented as 3D geometry to support immersive exploration, presentation, and collaboration techniques. Accurate representation and high-quality display are fundamental requirements to avoid misinterpretation of the data. Our system consists of the following parts: a preprocessor to create a special 3D representation – optimized under transmission and streamed presentation issues in a high-performance working environment, an efficiently implemented streaming server, and a client. The client was implemented as a web browser plugin, integrating a viewer with high-quality virtual reality presentation Žstereoscopic displays., interaction Žtracking devices., and hyperlinking capabilities. q 1999 Elsevier Science B.V. All rights reserved. Keywords: Virtual reality; VRML; Scientific visualization; Streaming; Browser; Plugin
1. Introduction Scientific and industrial research environments increasingly provide powerful operating platforms, such as high-speed networks, high-performance server and client systems, and high-quality 3D graphics systems. These are potentially meeting the requirements of high quality applications in typical visualization scenarios, where complex 3D objects – for example results of a simulation – have to be handled interactively. Available client-server systems for distributed online presentation of virtual 3D scenes in the WWW )
Corresponding author. Tel.: q49-511-762-3078; Fax: q49511-762-3003; e-mail: [email protected]
are based on Internet standards, such as TCPrIP, URL w5x, HTTP w13x, MIME w14x, HTML w4x, and VRML Version 1.0 w3x or VRML97 w22x. They reveal several constraints regarding performance, quality, and functionality aspects which often prohibit useful application. This is in contrast to specific virtual reality systems, which demonstrate the performance potential, but are essentially designed as stand-alone systems. Using VRML viewers, high latency, low navigation frame rates, little support of high-quality virtual reality presentation and interaction techniques have been observed, which are caused by inefficient representation, transport, and presentation protocol design and implementation. While delays between user-requested changes of static scenes already prevent
1389-1286r99r$ - see front matter q 1999 Elsevier Science B.V. All rights reserved. PII: S 1 3 8 9 - 1 2 8 6 Ž 9 9 . 0 0 0 9 7 - 3
2216
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
interactive production, real-time streaming of sequences of scenes is obviously in general left out of consideration. But such a mechanism is required for exploration of three-dimensional, time-dependent phenomena, utilizing virtual-reality techniques in order to get Virtual Reality MoÕies, freely to navigate with on-the-fly presentation.
2. Related work Interactive viewing of 3D models was originally considered in the context of specialized ‘Virtual Reality Systems’, consisting of a local storage system, a high-performance 3D rendering engine, stereoscopic displays, and 3D interaction devices. After the – often time-consuming – startup phase, loading the often proprietary file formats for the model data and behaviour in such a stand-alone system, all manipulation is based on the memorybased representation. When Internet-based, hyper-linked multimedia documents became generally available and useful to science and industry, integration and standardization of 3D scene descriptions in the World Wide Web were required. In 1994, after presentation of a 3D user interface for the Internet w37x, public discussion – based on an e-mail list – was started. The Open Inventor ŽSilicon Graphics. file format w50x was then chosen as a foundation for the first specification of a Virtual Reality Modeling Language ŽVRML-1.0. w3x. A revised version, ŽVRML97. w22x, including dynamic elements, was standardized by collaboration of the Internet community and ISO. Further development concerns, among other aspects, compressed binary coding w23x. A framework and representation of multimedia documents including 3D scenes is also provided by MPEG-4, specifying a binary representation of interactive graphics and audio-visual scenes, known as BIFS ŽBInary Format for Scenes. w11,41x. Several methods for the preparation of complex 3D surface models to reduce their complexity and data volume are known. They can be classified as: Ø Mesh decimation, smoothing w25,26,38,39x and multiresolution modeling w9,24,27x, Ø Topology-oriented geometric compression w16,43, 45,46x and progressive coding w12,15,44x,
Ø Primitive-packing, such as ‘Triangle stripping’. Techniques of the first two classes, on principle, introduce deviations in the representation, which should generally be avoided for scientific data visualization applications in order to maintain accurate exploration of results of expensive simulations on supercomputers. Besides this, reconstruction from the compressed code can be a very time-consuming process – e.g., ca. 35 000 triangles per second in w45x; 800 000 triangles per second in w16x, and therefore introduce a significant bottleneck in powerful networking and rendering environments. Triangle stripping is a useful technique not only to reduce the data volume Ž3:1 for large connected meshes., but also to optimize the rendering process Žsee Figs. 1 and 4.. Besides VRML97 viewers, such as CosmoPlayer Žnow ‘PlatinumPlayer’, announced as open source w35x. and VRwave w1x, several other 3D viewers have been implemented: first integrating 3D presentation in the WWW ŽCLRMosaic, Web OOGL., then considering efficient display Že.g., i3D w2x. or prototyping new concepts. An approach for embedding complex 3D objects in digital documents is presented in w9x, resulting in a hierarchical mesh reduction technique with progressive transmission and display capabilities, available as an ActiveX control for Microsoft Windows and a web-browser helper application for Unix. Streaming concepts are discussed in the context of transmission and on-the-fly presentation and control of video and audio media, separately presented or integrated in 3D worlds ŽVRML97 only supports video or audio objects as complete files., as a method to incrementally update minor changes in virtual scenes w17x, or transmission and adaptive presentation of progressive multi-level meshes of static scenes w15x. Concepts for sharing distributed virtual environments are investigated in w6,7x. The connection of scientific data visualization and immersive virtual reality techniques is discussed in w8,10,18,19x. Representation, communication, and presentation aspects in conjunction with online presentation of 3D objects are discussed in w33x, resulting in an efficient concept for interactive handling preprocessed VRML-1.0-based complex static models, uti-
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
2217
Fig. 1. Complexity of typical rendering primitives: Independent triangles versus connected mesh.
lizing the capacity of high-performance networking, computing, and 3D graphics environment. 3. Requirements We mainly focus on the visualization of scientific results for exploration and presentation purposes. Large multi-dimensional datasets from measurements or high-performance computations are characteristic for this field of application. This raw data Že.g., scalar or vector fields. have to be filtered to symbolic Že.g., isosurfaces, streamlines. and mapped to three-dimensional geometric Že.g., triangles, lines. representations, which are then rendered on a two-dimensional raster display. Immersive virtual reality environments, as illustrated in Fig. 2, will increasingly provide powerful means to support virtual laboratories, which take advantage of multi-sensor, intuitive user interfaces. In the following sections, we consider the distributed presentation of sequences of complex 3D objects at the geometric level of abstraction. This enables virtual reality techniques, such as stereoscopic projection and 3D navigation. Partitioning at a higher level would support more interaction, such as the choice of filtering and mapping methods and parameters. But for the interesting grand-challenge problems – such as exploration of large-scale unsteady fluid flows – these steps could not be offered at interactive rates, since the data
volume or computation efforts are often prohibitive w28x. The online-presentation of series of these prepared 3D objects has to avoid bottlenecks, which could prevent efficient and accurate exploration. An overview of the requirements is given in Table 1, where we classify the aspects which influence per-
Fig. 2. Virtual reality environment for scientific visualization.
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
2218
Table 1 Classification of requirements standards Žexamples. Presentation
Communication
Representation
Ø Optimized implementation
Quality
Ø Short latency - Startup - Navigation Ø High framerates - Transform rate - Fill rate - Usage of efficient primitives Ø Resolution - Pixels - Intensity - Color Ø Antialiasing Ø Color Management - Configurable destination ICC profile
Functionality
Ø Progressive rendering
Ø Streaming, e. g.
Ø Stereoscopic viewing Ø Navigational comfort - Tracking - 3D input devices Ø Multiplatform support - MS Windows - UNIX
- control: RTSP - media data: HTTP Ø Synchronization - intrastream - interstream Ø Media-specific scaling - frame rate - resolution - level of detail Ø Networks - LAN, WAN - Ethernet, ATM Ø Protocols - IP, TCP, RTSP Ø Services - HTTP
Ø Client-side effort - Decode - Decompress - Parse - Building scene graph - Calculation of normal vectors Ø Data volume Ø Resolution - Time - Geometry - Normals - Color, Texture Ø Compression - Lossless! Ø Color profiles support - sRGB default Ø Needed attributes for Virtual Reality presentation - Focal distance Ø Sequences of 3D objects: ‘Virtual Reality Movie’
Performance
Standards
Ø Generic WWW-Browser - Netscape - Plugin Interface Ø APIs - 3D Graphics - GUI
Ø Quality of Service - High bitrates - Short latency - Low jitter - Low error rate
formance, quality, and functionality and attach them to the representation, communication and presentation instances of the processing pipeline. 4. Concept and implementation After identification of the bottlenecks and studying feasibility of the given application scenario requirements using state-of-the art equipment, we have designed and implemented an optimized viewer architecture. It consists of three components, which are discussed in the next sections: Ø Preprocessor: VRML to DVR, a new 3D file format,
Ø WWW Server Ø Dynamic content - CGI Ø Caching, Proxy techniques Ø Multimedia, 3DrVR data formats
Ø DVR streaming server, based on Real-Time Streaming Protocol ŽRTSP., Ø New Viewer: Netscape plugin, called DocShowVR. By using this software in a high-performance networking and graphics environment, sequences of high-quality 3D scenes can be played out and interactively navigated while viewing smooth 3D animations. The operating sequence is characterized by the following steps Žnumbers corresponding to Fig. 3.: 1. The web client reads a DVRS object ŽMIME type: applicationrx-docshow-vrs, extension: .dvrs. via HTTP, which was possibly requested by an EMBED tag in an HTML page.
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
2219
Fig. 3. 3D streaming system.
2. After the appropriate DVRS plugin ‘DocShowVR’ Žsee Section 4.3. is loaded by the web browser, it establishes a connection to the 3D streaming server ŽDVRS, see Section 4.2., based on the reference information ŽIP address, port number. and attributes Žframe rate, etc.. contained in the DVRS data. 3. The 3D streaming server reads 3D objects from files in DVR format Žsee Section 4.1. and delivers them to the client, interleaved by additional delimiting PDUs. 4. After reading the first 3D object, the client executes some further actions: Ø Reading DVR data: Transfer from the streaming connection into a 3D object memory buffer. Ø Rendering: Transformation of the 3D objects, based on the current virtual camera position, orientation, and view angle, to the display device. Ø VR Navigation: Modification of the virtual camera parameters, according to the input device control.
Ø RTSP, VCR Metaphor: Control handling, e.g., instruct the server to stop, to play-out from a new position with a modified frame rate, or to deliver an alternate data set Žin case of existing multiple levels of detail on the server.. The binaries, the web-based VRML-to-DVR converter service, and several application examples are publicly available: http://www.dfn-expo.de /Technologie/DocShow-VR/ 4.1. Preprocessor: VRML to DVR, a new 3D file format In order to avoid the overhead of calculations that are usually done by the viewer software in typical WWW-integrated applications, the 3D scene descriptions are preprocessed. Computer-intensive processes, such as: Ø decompressing, Ø converting from ASCII to binary representation, Ø building a scene graph, Ø calculation of normal vectors,
2220
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
Ø packing optimized display lists – such as triangle strips Žas described in Section 2, see also Fig. 4., are now executed as a preparation step, resulting in a new 3D file format called ‘DVR’. The DVR files are
then placed on a web server ŽMIME type: applicationrx-docshow-vr, extension: .dvr. or our DVR streaming server Žsee Section 4.2.. This partitioning approach could be thought as splitting a conventional
Fig. 4. Preprocessing results of three application scenarios at RRZNrRVS: DVR data volumes, and immediate mode rendering rates, times, bitrates, based on embedded DVR plugin ŽDocShow-VR 1.3.. Window sizes: 1. 512 = 512 pixel, 2. and 3. 640 = 480 pixel. Platforms: Ži. HP Kayak XW, fx6 graphics, 2 = Pentium II Xeon, 450 MHz, Windows NT 4.0, Žii. SGI Onyx2 Infinite Reality, 4 = MIPS R10000, 195 MHz, Irix 6.5.4.
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
scene graph based virtual reality API – such as Iris Performer, Open Inventor, or Optimizer – at a usually hidden level, where structures of optimized rendering primitives are built in memory, in order to take advantage of adequately optimized rendering routines, which are also provided within this API in a capsulated manner. Since the standardized 3D file format in the Internet is VRML w3,22x, we have developed tools to provide efficient and convenient conversion from VRML to DVR. This supports processing of existing 3D scenes, which could possibly be first converted to VRML by other available tools w49x and then to DVR by our tool. VRML output of 3D modeling tools, such as 3D Studio, and visualization systems, such as AVS Žwith the public domain module ‘write VRML’. w47x, AVSrExpress, Ensight and VTK w38x, have been successfully prepared in this way. Besides that, a class library was implemented to support direct DVR data generation, which is intended to be integrated into VTK. The preprocessor accepts static 3D scene descriptions in the ASCII formats VRML-1.0 or VRML97 and converts them into an own binary representation ŽIEEE format, network byte ordering. of the restructured, linearized scene graph to support efficient on-the-fly rendering of the data stream at the client side. The C q q implementation is based on the available VRML-1.0 parser library QvLib w42x, an e-mail message from Jan Hardenberg, consisting of fragments of an implementation of a rudimentary, OpenGL-based VRML-1.0 viewer w20x, and the VRML-2.0 reader and scene graph classes of OpenInventor ŽTGS, Version 2.5.. The current release is provided as a batch-oriented command-line tool for Microsoft Windows 95r98rNT and SGI Irix. It has also been integrated in an interactive online conversion service via HTTP form-based upload w29x. We have also experimented with a transparent mechanism, using a caching proxy w33x, based on a configurable http filter process webfilt w48x. 4.2. DVR streaming serÕer, based on RTSP In addition to a usual WWW server which delivers single DVR scenes or DVR streaming reference and control information Žnew format called DVRS., we need a dedicated 3D streaming server. We have
2221
implemented a prototype, applying the Real Time Streaming Protocol ŽRTSP, RFC 2326. w40x for this purpose. The DVR transport protocol ŽDVRP. is realized via a TCP-based transmission, simply using the native DVR files, interleaved with a new separator protocol data unit ŽPDU., which can be considered as an extension of the DVR file PDUs. An intra-stream synchronization is realized based on the server-side delivery times. Optionally the intended mean frame rate is maintained by occasionally omitting scene files. 4.3. New Õiewer: Netscape plugin, called DocShowVR To achieve short latency, streaming capabilities and support of embedded 3D scenes in HTML pages, we have implemented an optimized viewer as a plugin for Netscape browsers. As such, by using the Netscape Plugin API w31,32x, it is tightly coupled to the data delivery to allow streamed, progressive display of single DVR scenes via HTTP or – after reading DVRS information – to connect to our 3D streaming server. After RTSP startup, 3D animations are displayed in real-time, overlapped with the streamed TCP-based DVR data transmission, where display Žup to two graphics pipes. and transport are implemented in separate threads, which leads to a speed-up by parallelization of the communication and rendering processes. This RTSP-based streaming scenario is currently only implemented on SGI Irix. The renderer is efficiently implemented for UNIX ŽSGI Irix, Sun Solaris, HPrUX. and Win32 ŽMicrosoft Windows 95r98rNT. platforms, based on OpenGL w30,34x, the de-facto 3D graphics API standard, and utilizes platform-specific OpenGL extensions, such as vertex arrays Žhigh-speed polygon rendering. or multisampling antialiasing Žhigh-quality presentation.. For UNIX platforms not providing an OpenGL runtime environment, plugin versions linked with the OpenGL emulation library Mesa w36x were created. Stereoscoping viewing is supported on several platforms ŽSGI: any Irix workstation; Sun: Elite 3D; Windows NT: Diamond FireGL 1000, HP Kayak XW., using the quad-buffer stereo mode in conjunction with shutter glasses or large screen stereo projection. Head tracking systems ŽCrystal Eyes CE-VR,
2222
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
Table 2 ICC-based Color management functions on different platforms
Initialization Open Profile Close Profile Get Tag Value Create Color Transformation Transform Colors
Microsoft Windows 98 ICM 2.0
SGI Irix 6.5 Coloratura
Sun Solaris KCMS
OpenColorProfile CloseColorProfile GetColorProfileElement CreateMultiProfileTransform TranslateBitmapBits
cmsOpen cmsImportProfile cmsCloseProfile cmsGetTag cmsCreateTfm cmsApplyTfm
KcsAvailable KcsLoadProfile KcsFreeProfile KcsGetAttribute KcsConnectProfiles KcsEvaluate
Polhemus Fastrak, Intersense. have also been integrated, in order to get an immersive virtual reality system. Color management w21x support was integrated to increase reproduction quality of textures as scene components. The presentation of images in TIFF RGB and CMYK formats, originally based on Sam Leffler’s TIFF library, takes the eventually embedded ICC-conforming source profile as well as a monitor-specific ICC destination profile into account, and converts color space and gamut to match the intended colors. It was implemented on SGI Irix 6.5, Sun Solaris, and Microsoft Windows 98, by applying the available color management system ŽCMS. APIs Coloratura, KCMS and ICM 2.0, respectively Žused CMS functions listed in Table 2.. On these platforms the plugin is also capable of viewing TIFF images ŽMIME type: imagertiff or imagerx-tiff, extension: .tif.. Support of collaborative work in performing bidirectional synchronization of the virtual camera between two clients is already implemented in the Unix version. In the future we would like to synchronize not only navigation in 3D space, but also implement 3D tele-pointers and annotations, and a synchronization of the time axis for at least two users in the streaming scenario.
5. Evaluation We built a testbed for the evaluation of the described production process in a typical scientific data visualization application. The components are: 1. DVR dataset: Ø A simulation of an unsteady fluid flow phenomenon Žoceanic convection w38x. on a supercomputer ŽSiemensrFujitsu VPP 300. resulted
in raw data of 740 timesteps, each consisting of temperature and velocity information at 161 = 161 = 31 grid points, stored as 32-bit float values. This dataset was postprocessed by an AVSrExpress application into colored slicers, isosurfaces and arrows, stored as a sequence of 3D scenes in VRML-1.0 format by the module ‘OutputVRML’, and then converted to DVR by our ‘wrl1toDVR’ tool. Ø 740 DVR files, between 5 542 044 bytes and 9 309 468 bytes in size Žwhole DVR dataset: 5.3 Gbyte., each containing about 100 000 primitives Žlines, polygons.. An example of a rendered image is shown in Fig. 4 as case 2. 2. 3D streaming client-server configuration Ø As a client system, where 3D rendering performance is a main issue, we used an SGI Onyx2 Žrack system, 4 processors R10000r195 MHz, 2 Gbytes main memory, 9 Gbytes SCSI disk, Irix 6.5.4. with two Infinite Reality graphics subsystems Žeach with two 64 MB Raster Manager boards.. Ø A 3D streaming serÕer was also installed on this machine for development purposes and testing capabilities in a high-performance TCPrIP scenario Žloopback communication.. The DVR files were stored on a 140 Gbyte RAID system, connected via Fibre Channel. Ø A separate 3D streaming serÕer was implemented on an SGI Origin200 ŽR10000r225 MHz, 512 Mbytes main memory, 9 q 18 Gbyte SCSI disk, Irix 6.5.4.. In this case the DVR files were stored on a 18 Gbyte SCSI disk. Ø A high-speed network between the Onyx2 and the Origin200 was realized by a back-to-back Gigabit Ethernet connection. As a performance optimization, jumbo frame support was enabled on both machines.
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
2223
Table 3 Throughput measurement results Žtypical values in Mbitrs. of the scenario ‘Oceanic convection’ Žframes 720–739.
Read DVR files, one read block per file Žsecond try: cached. TCPrIP transmission from server to client Žttcp -l1048576 -n100 -b262144. 3D rendering ŽDocShow-VR.: OpenGL immediate mode DocShow-VR client-server maximal framerate streaming pipeline 4 framesrsecond 2 framesrsecond
Client SGI Onyx2
Server 1 SGI Onyx2
Server 2 SGI Origin200
-
250–350 Ž980–1020.
109–110 Ž1090–1170.
-
790–950 Žloopback.
560–670 ŽGigabit Ethernet.
380–420
-
-
-
280–320 251 126
324–326 251 126
We measured the throughput of the critical instances separately and the over-all performance of the processing pipeline. Then we justified read and write block sizes and socket buffer sizes in the implementation of our client and server software. In Table 3 we present some characteristic results. Interesting experiences in this scenario are: Ø The throughput in the distributed configuration is higher than in the case of executing client and server on the same machine. Reasons could be: - Lower CPU clock on the client machine Ž195 MHz versus 225 MHz., - Overhead introduced by running 3 processes simultaneously. Ø The sustained overall throughput is significantly slower than the rendering speed, which seemed to be the limiting factor. Possible reasons are: - Swapping latency caused by double-buffered display. Average time: 1rŽ2)frame_rate. Ž1r144 s for 72 Hz frame rate., - X11rMotif event processing at every frame, - Initialization of OpenGL rendering state at startup of every frame. In Fig. 4 we illustrate the rendering performance in typical application scenarios.
6. Conclusion Our concept of preprocessing, real-time streaming, and efficient, high-quality presentation of 3D scenes as Virtual Reality MoÕies, embedded in a WWW browser, has proven to be a powerful tool, in
particular for high-performance application environments, such as exploration or presentation of visualized supercomputer simulation results on a 3D graphics workstation. This is a potential Gigabit-network killer-application, since the geometric rendering data throughput of a state-of-the-art 3D graphics system is in the order of Gbitrs, and the requirements for real-time transmission are similar in a streaming scenario. For example: N triangles, organized as a triangle strip with 3D normals per every 3D vertex have a volume of Ž N q 2.)24 bytes in the case of the usual 32-bit representation for coordinate and normal vector components – this leads with N 4 2 for 6 Mio. trianglesrs to a data rate of above 1 Gbitrs. Already existing advanced distributed application scenarios will require the integration of collaborative functionalities – such as synchronization of viewing parameters, annotations, and video-conferencing in a multi-user environment – and the integration of further media types – such as video and audio streams as parts of an immersive virtual reality presentation.
Acknowledgements This work is partly funded by the DFN-Verein ŽGerman Research Network., with funds from the BMBF ŽGerman Federal Ministry for Education and Research. in the project ‘DFN-Expo’. The authors wish to thank A. von Berg ŽRVS. for the discussion about the high-performance network issues and configuration of Gigabit Ethernet.
2224
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
References w17x w1x K. Andrews, A. Pesendorfer, M. Pichler, K.H. Wagenbrunn, J. Wolte, Looking inside VRwave: the architecture and interface of the VRwave VRML97 browser, Proc. VRML ’98 Symposium, Monterey, CA, 1998. http:rrwww2.iicm.edur keithrpublications.html w2x J.-F., Balaguer, E. Gobbetti, i3D: a high-speed 3D web browser, Proc. VRML ’95 Symposium, San Diego, 1995. http:rrwww.crs4.itr ; gobettir w3x G. Bell, A. Parisi, M. Pesce, The Virtual Reality Modeling Language – Version 1.0 Specification, 9 November 1995. http:rrwww.vrml.orgrSpecificationsrVRML1.0r w4x T. Berners-Lee, D. Connolly, Hypertext Markup Language – HTML 2.0, RFC 1866, 3 November 1995. ftp:rrnis. nsf.netrdocumentsrrfcr w5x T. Berners-Lee, L. Masinter, M. McCahill, Uniform Resource Locators ŽURL., RFC 1737, 20 December 1994. ftp:rrnis.nsf.netrdocumentsrrfcr w6x W. Broll, DWTP – an Internet protocol for shared virtual environments, Proc. VRML ’98 Symposium, Monterey, CA, 1 9 9 8 . h ttp :r r ece.u w aterlo o .car v rm l9 8 r p ap ersr content.html w7x D. Brutzman, M. Zyda, K. Watsen, M. Macedonia, Virtual Reality Transfer Protocol ŽVRTP. design rationale, Workshop on Enabling Technology: Infrastructure for Collaborative Enterprises ŽWET ICE.: Sharing a Distributed Virtual Reality, MIT, 18–20 June 1997. http:rrwww.stl.nps. navy.milr ; brutzmanrvrtprvrtp_design.ps w8x S. Bryson, C. Levit, The virtual windtunnel, IEEE Computer Graphics and Applications 12 Ž4. Ž1992. . w9x S. Campagna, L. Kobbelt, H. Seidel, Enhancing digital documents by including 3D-models, Computers and Graphics 22 Ž6. Ž1998. . w10x B. Corrie, D. Sitsky, P. Mackerras, Integrating high performance computing and virtual environments, Proc. Seventh Parallel Computing Workshop, Canberra, Australia, September 1997. http:rrcap.anu.edu.aurcaprprojectsrparvisr bibliographyrBCDSPM97.ps.gz w11x P.K. Doenges, T. Capin, K. Capin, F. Lavagetto, J. Ostermann, I.S. Pandzic, E.E. Petajan, MPEG-4: audiorvideo and synthetic graphicsraudio for mixed media, Image Communication Journal 5 Ž4. Ž1997. . w12x K. Engel, R. Grosso, T. Ertl, Progressive Iso-surfaces on the Web, Proc. IEEE Visualization 1998. http:rrwww9. informatik.uni-erlangen.dergerrresearchrpub1998r w13x R. Fielding, J. Gettys, J. Mogul, H. Nielsen, T. Berners-Lee, Hypertext Transfer Protocol - HTTPr1.1, RFC 2068, 3 January 1997. ftp:rrnis.nsf.netrdocumentsrrfcr w14x N. Freed, N. Borenstein, Multipurpose Internet Mail Extensions ŽMIME., RFC 2049, 2 December 1996. ftp:rrnis.nsf. netrdocumentsrrfcr w15x A. Gueziec, G. Taubin, B. Horn, F. Lazarus, A framework ´ for streaming geometry in VRML, IEEE Computer Graphics and Applications 19 Ž2. Ž1999. . w16x S. Gumhold, W. Straber, Real time compression of triangle
w18x w19x
w20x
w21x
w22x
w23x
w24x
w25x
w26x
w27x
w28x
w29x
w30x
w31x
w32x
w33x
mesh connectivity, ACM SIGGRAPH ’98 Conference Proc., July 19–24, 1998. O. Grau, Representation of temporal changes of flexible 3-D objects, International Workshop on Synthetic-Natural Hybrid Coding and Three-Dimensional Imaging, 1997. H. Haase, Symbiosis of virtual reality and scientific visualization system, Computer Graphics Forum 15 Ž3. Ž1996. . H. Haase, F. Dai, J. Strassner, M. Gobel, Immersive investi¨ gation of scientific data, in: G. Nielson et al. ŽEds.., Scientific Visualization – Overviews, Methodologies and Techniques, IEEE Computer Society Press, Rockville, MD, 1997. J. Hardenberg, RE: QvLib questions, VRML Hypermail Archive, 27 March 1995. http:rrvag.vrml.orgrwww-vrmlr archr1107.html M. Has, T. Newman, Color management: current practice and the adoption of a new standard, 1996. http:rrwww. color.orgroverview.html ISOrIEC 14772-1: The Virtual Reality Modeling Language ŽVRML97. – Part 1: Functional specification and UTF-8 encoding, International Standard Organization, 1997. http:rrwww.vrml.orgrSpecificationsrVRML97r ISOrIEC 14772-3: The Virtual Reality Modeling Language ŽVRML97. – Part 3: Compressed Binary Format Specification, Editor’s Draft 5, 1997. R. Klein, Multiresolution representations for surface meshes, Proc. SCCG, 1997. http:rrwww.gris.uni-tuebingen.der peoplerstaffrreinhardrmai97.ps.gz R. Klein, W. Straber, Handling of very large 3D-surfacedatasets using mesh simplification and multiresolution modeling, Tutorial, Computer Graphics International 1998, Hannover, Germany, June 22–26, 1998. http:rrwww.gris. uni-tuebingen.derpeoplerstaffrreinhardrCGI-Tutorialr L. Kobbelt, S. Campagna, H. Seidel, A general framework for mesh decimation, Graphics Interface Proc., Vancouver, BC, 18–20 June 1998. L. Kobbelt, S. Campagna, J. Vorsatz, H. Seidel, Interactive multi-resolution modeling on arbitrary meshes, ACM SIGGRAPH ‘98 Conference Proc., July 19–24, 1998. D.E. Lane, Scientific visualization of large-scale unsteady fluid flows, in: G.M. Nielson et al. ŽEds.., Scientific Visualization – Overviews, Methodologies and Techniques, IEEE Computer Society Press, Rockville, MD, 1997. E. Nebel, L. Masinter, Form-based File Upload in HTML, RFC 1867, 7 November 1995. ftp:rrnis.nsf.netr documentsrrfcr J. Neider, T. Davis, M. Woo, OpenGL Programming Guide – The Official Guide to Learning OpenGL, Release 1. Addison-Wesley, Reading, MA, 1993. Netscape, Netscape Navigator LiveConnectrPlug-In Software Development Kit, 1998. http:rrhome.netscape.comr comprodrdevelopment_partnersrplugin_apir Netscape, Plug-In Guide – Communicator 4.0, January 1998. http:r r developer.netscape.comr docsr manualsrcommunicatorrpluginr S. Olbrich, H. Pralle, High-performance online presentation of complex 3D scenes, IFIP Conference on High Performance Networking, Wien, 1998.
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225 w34x OPC – The OpenGL Performance Characterization Projekt, Viewperf Information and Results. http:rrwww.specbench. orgrgpcropc.staticrviewin;1.html w35x T. Parisi, Corrections: keeping web 3D on course, Keynote address at the VRML ’99 Symposium, Paderborn, 1999. w36x B. Paul, The Mesa 3-D graphics library. http:rrwww.ssec. wisc.edur ; brianprMesa.html w37x M. Pesce, P. Kennard, A. Parisi, Cyberspace, First WWW Conference, May 1994. http:rrwww.hyperreal.orgr ; mpescerwww.html w38x S. Raasch, D. Etling, Modeling deep ocean convection: large eddy simulation in comparison with laboratory experiments, J. Phys. Oceanogr. 28 Ž1998. 1796–1802. w39x W. Schroeder, K. Martin, B. Lorensen, The Visualization Toolkit, 2nd ed., Prentice-Hall, Englewood Cliffs, NJ, 1997. w40x W. Schroeder, Polygon reduction techniques, IEEE Visualization 1995. w41x H. Schulzrinne, A. Rao, R. Lanphier, Real Time Streaming Protocol ŽRTSP., RFC 2326, 14 April 1998. ftp:rrnis.nsf. netrdocumentsrrfcr w42x P. Strauss, G. Bell, The VRML Programming Library – QvLib, Version 1.0 beta 1. 1995. http:rrvag.vrml.orgr www-vrmlrvrml.techrqv.html w43x G. Taubin, J. Rossignac, Geometric compression through topology surgery, IBM Research technical report RC-20340, 16 January 1996. http:rrwww.research.ibm.comrvrmlr binaryrpdfsribm20340.pdf w44x G. Taubin, A. Gueziec, W. Horn, F. Lazarus, Progressive forest split compression, ACM SIGGRAPH ’98 Conference Proc., July 19–24, 1998. w45x G. Taubin, J. Rossignac, Geometric compression through topologic surgery, ACM Transactions on Graphics 17 Ž2. Ž1998. . w46x C. Touma, C. Gotsman, Triangle mesh compression, Graphics Interface Proc., Vancouver, BC, 18–20 June 1998. w47x C. Upson, T. Faulhaber, D. Kamins, D. Laidlaw, D. Schlegel, J. Vroom, R. Gurwitz, A. van Dam, The application visualization system: a computational environment for scientific visualization, IEEE Computer Graphics and Applications, July 1989. w48x J.-S. Vockler, A quick glance at webfilt, RVS, University ¨ Hannover, 3 September 1997.
2225
w49x Web 3D Consortium, The VRML Repository. http:rrwww. web3d.orgrvrmlrvrml.htm w50x J. Wernecke, The Inventor Mentor: Programming ObjectOriented 3D Graphics with Open Inventor. Release 2, Addison-Wesley, Reading, MA, 1994. Stephan Olbrich received his diploma in electrical engineering from the University of Hannover in 1987. From 1987 to 1989 he developed hardware and software at Altec Electronic GmbH in Hannover. Since 1989 he has worked as a research staff member at the Regional Scientific Computing Centre and the Institute for Computer Networks and Distributed Systems. His major interests include multimedia communication and distributed virtual reality systems. Helmut Pralle, University Professor, Dr.-Ing., studied engineering sciences at the Technische Hochschule Karlsruhe and Hannover. Afterwards he started to work as an assistant at the Institute for Applied Mathematics in the Division for Electronic Calculation and at the Computer Centre of the Technical University of Hannover. He received his doctorate in Control Theory in 1967. From 1968 to 1971 he was Head of the Computer Centre of the University of Hannover, since 1971 he has been Director of the Regional Computing Centre of Lower Saxony. In 1986 he got the Chair of the Institute for Computer Networks and Distributed Systems at the Department of Electronical Engineering and Information Technology of Hannover. Since 1996 he has been a member of the administrative council and the executive committee of the DFN Association. He delivers expert reports and vocational guidance for the German Research Council ŽDFG., ministries and the industry. His major interests include distributed information systems and high-speed data communication.
Virtual reality movies-real-time streaming of 3D objects S. Olbrich ) , H. Pralle Institute for Computer Networks and Distributed Systems (RVS), Regional Scientific Computing Center for Lower Saxony (RRZN), UniÕersity of HannoÕer, Schlosswender Str. 5, D-30159 HannoÕer, Germany
Abstract Powerful servers for computation and storage, high-speed networking resources, and high-performance 3D graphics workstations, which are typically available in scientific research environments, potentially allow the development and productive application of advanced distributed high-quality multimedia concepts. Several bottlenecks, often caused by the inefficient design and software implementation of current systems, prevent utilization of the offered performance of existing hardware and networking resources. We present a system architecture, which supports streamed online presentation of series of 3D objects. In the case of expensive simulations on a supercomputer, results are increasingly represented as 3D geometry to support immersive exploration, presentation, and collaboration techniques. Accurate representation and high-quality display are fundamental requirements to avoid misinterpretation of the data. Our system consists of the following parts: a preprocessor to create a special 3D representation – optimized under transmission and streamed presentation issues in a high-performance working environment, an efficiently implemented streaming server, and a client. The client was implemented as a web browser plugin, integrating a viewer with high-quality virtual reality presentation Žstereoscopic displays., interaction Žtracking devices., and hyperlinking capabilities. q 1999 Elsevier Science B.V. All rights reserved. Keywords: Virtual reality; VRML; Scientific visualization; Streaming; Browser; Plugin
1. Introduction Scientific and industrial research environments increasingly provide powerful operating platforms, such as high-speed networks, high-performance server and client systems, and high-quality 3D graphics systems. These are potentially meeting the requirements of high quality applications in typical visualization scenarios, where complex 3D objects – for example results of a simulation – have to be handled interactively. Available client-server systems for distributed online presentation of virtual 3D scenes in the WWW )
Corresponding author. Tel.: q49-511-762-3078; Fax: q49511-762-3003; e-mail: [email protected]
are based on Internet standards, such as TCPrIP, URL w5x, HTTP w13x, MIME w14x, HTML w4x, and VRML Version 1.0 w3x or VRML97 w22x. They reveal several constraints regarding performance, quality, and functionality aspects which often prohibit useful application. This is in contrast to specific virtual reality systems, which demonstrate the performance potential, but are essentially designed as stand-alone systems. Using VRML viewers, high latency, low navigation frame rates, little support of high-quality virtual reality presentation and interaction techniques have been observed, which are caused by inefficient representation, transport, and presentation protocol design and implementation. While delays between user-requested changes of static scenes already prevent
1389-1286r99r$ - see front matter q 1999 Elsevier Science B.V. All rights reserved. PII: S 1 3 8 9 - 1 2 8 6 Ž 9 9 . 0 0 0 9 7 - 3
2216
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
interactive production, real-time streaming of sequences of scenes is obviously in general left out of consideration. But such a mechanism is required for exploration of three-dimensional, time-dependent phenomena, utilizing virtual-reality techniques in order to get Virtual Reality MoÕies, freely to navigate with on-the-fly presentation.
2. Related work Interactive viewing of 3D models was originally considered in the context of specialized ‘Virtual Reality Systems’, consisting of a local storage system, a high-performance 3D rendering engine, stereoscopic displays, and 3D interaction devices. After the – often time-consuming – startup phase, loading the often proprietary file formats for the model data and behaviour in such a stand-alone system, all manipulation is based on the memorybased representation. When Internet-based, hyper-linked multimedia documents became generally available and useful to science and industry, integration and standardization of 3D scene descriptions in the World Wide Web were required. In 1994, after presentation of a 3D user interface for the Internet w37x, public discussion – based on an e-mail list – was started. The Open Inventor ŽSilicon Graphics. file format w50x was then chosen as a foundation for the first specification of a Virtual Reality Modeling Language ŽVRML-1.0. w3x. A revised version, ŽVRML97. w22x, including dynamic elements, was standardized by collaboration of the Internet community and ISO. Further development concerns, among other aspects, compressed binary coding w23x. A framework and representation of multimedia documents including 3D scenes is also provided by MPEG-4, specifying a binary representation of interactive graphics and audio-visual scenes, known as BIFS ŽBInary Format for Scenes. w11,41x. Several methods for the preparation of complex 3D surface models to reduce their complexity and data volume are known. They can be classified as: Ø Mesh decimation, smoothing w25,26,38,39x and multiresolution modeling w9,24,27x, Ø Topology-oriented geometric compression w16,43, 45,46x and progressive coding w12,15,44x,
Ø Primitive-packing, such as ‘Triangle stripping’. Techniques of the first two classes, on principle, introduce deviations in the representation, which should generally be avoided for scientific data visualization applications in order to maintain accurate exploration of results of expensive simulations on supercomputers. Besides this, reconstruction from the compressed code can be a very time-consuming process – e.g., ca. 35 000 triangles per second in w45x; 800 000 triangles per second in w16x, and therefore introduce a significant bottleneck in powerful networking and rendering environments. Triangle stripping is a useful technique not only to reduce the data volume Ž3:1 for large connected meshes., but also to optimize the rendering process Žsee Figs. 1 and 4.. Besides VRML97 viewers, such as CosmoPlayer Žnow ‘PlatinumPlayer’, announced as open source w35x. and VRwave w1x, several other 3D viewers have been implemented: first integrating 3D presentation in the WWW ŽCLRMosaic, Web OOGL., then considering efficient display Že.g., i3D w2x. or prototyping new concepts. An approach for embedding complex 3D objects in digital documents is presented in w9x, resulting in a hierarchical mesh reduction technique with progressive transmission and display capabilities, available as an ActiveX control for Microsoft Windows and a web-browser helper application for Unix. Streaming concepts are discussed in the context of transmission and on-the-fly presentation and control of video and audio media, separately presented or integrated in 3D worlds ŽVRML97 only supports video or audio objects as complete files., as a method to incrementally update minor changes in virtual scenes w17x, or transmission and adaptive presentation of progressive multi-level meshes of static scenes w15x. Concepts for sharing distributed virtual environments are investigated in w6,7x. The connection of scientific data visualization and immersive virtual reality techniques is discussed in w8,10,18,19x. Representation, communication, and presentation aspects in conjunction with online presentation of 3D objects are discussed in w33x, resulting in an efficient concept for interactive handling preprocessed VRML-1.0-based complex static models, uti-
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
2217
Fig. 1. Complexity of typical rendering primitives: Independent triangles versus connected mesh.
lizing the capacity of high-performance networking, computing, and 3D graphics environment. 3. Requirements We mainly focus on the visualization of scientific results for exploration and presentation purposes. Large multi-dimensional datasets from measurements or high-performance computations are characteristic for this field of application. This raw data Že.g., scalar or vector fields. have to be filtered to symbolic Že.g., isosurfaces, streamlines. and mapped to three-dimensional geometric Že.g., triangles, lines. representations, which are then rendered on a two-dimensional raster display. Immersive virtual reality environments, as illustrated in Fig. 2, will increasingly provide powerful means to support virtual laboratories, which take advantage of multi-sensor, intuitive user interfaces. In the following sections, we consider the distributed presentation of sequences of complex 3D objects at the geometric level of abstraction. This enables virtual reality techniques, such as stereoscopic projection and 3D navigation. Partitioning at a higher level would support more interaction, such as the choice of filtering and mapping methods and parameters. But for the interesting grand-challenge problems – such as exploration of large-scale unsteady fluid flows – these steps could not be offered at interactive rates, since the data
volume or computation efforts are often prohibitive w28x. The online-presentation of series of these prepared 3D objects has to avoid bottlenecks, which could prevent efficient and accurate exploration. An overview of the requirements is given in Table 1, where we classify the aspects which influence per-
Fig. 2. Virtual reality environment for scientific visualization.
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
2218
Table 1 Classification of requirements standards Žexamples. Presentation
Communication
Representation
Ø Optimized implementation
Quality
Ø Short latency - Startup - Navigation Ø High framerates - Transform rate - Fill rate - Usage of efficient primitives Ø Resolution - Pixels - Intensity - Color Ø Antialiasing Ø Color Management - Configurable destination ICC profile
Functionality
Ø Progressive rendering
Ø Streaming, e. g.
Ø Stereoscopic viewing Ø Navigational comfort - Tracking - 3D input devices Ø Multiplatform support - MS Windows - UNIX
- control: RTSP - media data: HTTP Ø Synchronization - intrastream - interstream Ø Media-specific scaling - frame rate - resolution - level of detail Ø Networks - LAN, WAN - Ethernet, ATM Ø Protocols - IP, TCP, RTSP Ø Services - HTTP
Ø Client-side effort - Decode - Decompress - Parse - Building scene graph - Calculation of normal vectors Ø Data volume Ø Resolution - Time - Geometry - Normals - Color, Texture Ø Compression - Lossless! Ø Color profiles support - sRGB default Ø Needed attributes for Virtual Reality presentation - Focal distance Ø Sequences of 3D objects: ‘Virtual Reality Movie’
Performance
Standards
Ø Generic WWW-Browser - Netscape - Plugin Interface Ø APIs - 3D Graphics - GUI
Ø Quality of Service - High bitrates - Short latency - Low jitter - Low error rate
formance, quality, and functionality and attach them to the representation, communication and presentation instances of the processing pipeline. 4. Concept and implementation After identification of the bottlenecks and studying feasibility of the given application scenario requirements using state-of-the art equipment, we have designed and implemented an optimized viewer architecture. It consists of three components, which are discussed in the next sections: Ø Preprocessor: VRML to DVR, a new 3D file format,
Ø WWW Server Ø Dynamic content - CGI Ø Caching, Proxy techniques Ø Multimedia, 3DrVR data formats
Ø DVR streaming server, based on Real-Time Streaming Protocol ŽRTSP., Ø New Viewer: Netscape plugin, called DocShowVR. By using this software in a high-performance networking and graphics environment, sequences of high-quality 3D scenes can be played out and interactively navigated while viewing smooth 3D animations. The operating sequence is characterized by the following steps Žnumbers corresponding to Fig. 3.: 1. The web client reads a DVRS object ŽMIME type: applicationrx-docshow-vrs, extension: .dvrs. via HTTP, which was possibly requested by an EMBED tag in an HTML page.
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
2219
Fig. 3. 3D streaming system.
2. After the appropriate DVRS plugin ‘DocShowVR’ Žsee Section 4.3. is loaded by the web browser, it establishes a connection to the 3D streaming server ŽDVRS, see Section 4.2., based on the reference information ŽIP address, port number. and attributes Žframe rate, etc.. contained in the DVRS data. 3. The 3D streaming server reads 3D objects from files in DVR format Žsee Section 4.1. and delivers them to the client, interleaved by additional delimiting PDUs. 4. After reading the first 3D object, the client executes some further actions: Ø Reading DVR data: Transfer from the streaming connection into a 3D object memory buffer. Ø Rendering: Transformation of the 3D objects, based on the current virtual camera position, orientation, and view angle, to the display device. Ø VR Navigation: Modification of the virtual camera parameters, according to the input device control.
Ø RTSP, VCR Metaphor: Control handling, e.g., instruct the server to stop, to play-out from a new position with a modified frame rate, or to deliver an alternate data set Žin case of existing multiple levels of detail on the server.. The binaries, the web-based VRML-to-DVR converter service, and several application examples are publicly available: http://www.dfn-expo.de /Technologie/DocShow-VR/ 4.1. Preprocessor: VRML to DVR, a new 3D file format In order to avoid the overhead of calculations that are usually done by the viewer software in typical WWW-integrated applications, the 3D scene descriptions are preprocessed. Computer-intensive processes, such as: Ø decompressing, Ø converting from ASCII to binary representation, Ø building a scene graph, Ø calculation of normal vectors,
2220
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
Ø packing optimized display lists – such as triangle strips Žas described in Section 2, see also Fig. 4., are now executed as a preparation step, resulting in a new 3D file format called ‘DVR’. The DVR files are
then placed on a web server ŽMIME type: applicationrx-docshow-vr, extension: .dvr. or our DVR streaming server Žsee Section 4.2.. This partitioning approach could be thought as splitting a conventional
Fig. 4. Preprocessing results of three application scenarios at RRZNrRVS: DVR data volumes, and immediate mode rendering rates, times, bitrates, based on embedded DVR plugin ŽDocShow-VR 1.3.. Window sizes: 1. 512 = 512 pixel, 2. and 3. 640 = 480 pixel. Platforms: Ži. HP Kayak XW, fx6 graphics, 2 = Pentium II Xeon, 450 MHz, Windows NT 4.0, Žii. SGI Onyx2 Infinite Reality, 4 = MIPS R10000, 195 MHz, Irix 6.5.4.
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
scene graph based virtual reality API – such as Iris Performer, Open Inventor, or Optimizer – at a usually hidden level, where structures of optimized rendering primitives are built in memory, in order to take advantage of adequately optimized rendering routines, which are also provided within this API in a capsulated manner. Since the standardized 3D file format in the Internet is VRML w3,22x, we have developed tools to provide efficient and convenient conversion from VRML to DVR. This supports processing of existing 3D scenes, which could possibly be first converted to VRML by other available tools w49x and then to DVR by our tool. VRML output of 3D modeling tools, such as 3D Studio, and visualization systems, such as AVS Žwith the public domain module ‘write VRML’. w47x, AVSrExpress, Ensight and VTK w38x, have been successfully prepared in this way. Besides that, a class library was implemented to support direct DVR data generation, which is intended to be integrated into VTK. The preprocessor accepts static 3D scene descriptions in the ASCII formats VRML-1.0 or VRML97 and converts them into an own binary representation ŽIEEE format, network byte ordering. of the restructured, linearized scene graph to support efficient on-the-fly rendering of the data stream at the client side. The C q q implementation is based on the available VRML-1.0 parser library QvLib w42x, an e-mail message from Jan Hardenberg, consisting of fragments of an implementation of a rudimentary, OpenGL-based VRML-1.0 viewer w20x, and the VRML-2.0 reader and scene graph classes of OpenInventor ŽTGS, Version 2.5.. The current release is provided as a batch-oriented command-line tool for Microsoft Windows 95r98rNT and SGI Irix. It has also been integrated in an interactive online conversion service via HTTP form-based upload w29x. We have also experimented with a transparent mechanism, using a caching proxy w33x, based on a configurable http filter process webfilt w48x. 4.2. DVR streaming serÕer, based on RTSP In addition to a usual WWW server which delivers single DVR scenes or DVR streaming reference and control information Žnew format called DVRS., we need a dedicated 3D streaming server. We have
2221
implemented a prototype, applying the Real Time Streaming Protocol ŽRTSP, RFC 2326. w40x for this purpose. The DVR transport protocol ŽDVRP. is realized via a TCP-based transmission, simply using the native DVR files, interleaved with a new separator protocol data unit ŽPDU., which can be considered as an extension of the DVR file PDUs. An intra-stream synchronization is realized based on the server-side delivery times. Optionally the intended mean frame rate is maintained by occasionally omitting scene files. 4.3. New Õiewer: Netscape plugin, called DocShowVR To achieve short latency, streaming capabilities and support of embedded 3D scenes in HTML pages, we have implemented an optimized viewer as a plugin for Netscape browsers. As such, by using the Netscape Plugin API w31,32x, it is tightly coupled to the data delivery to allow streamed, progressive display of single DVR scenes via HTTP or – after reading DVRS information – to connect to our 3D streaming server. After RTSP startup, 3D animations are displayed in real-time, overlapped with the streamed TCP-based DVR data transmission, where display Žup to two graphics pipes. and transport are implemented in separate threads, which leads to a speed-up by parallelization of the communication and rendering processes. This RTSP-based streaming scenario is currently only implemented on SGI Irix. The renderer is efficiently implemented for UNIX ŽSGI Irix, Sun Solaris, HPrUX. and Win32 ŽMicrosoft Windows 95r98rNT. platforms, based on OpenGL w30,34x, the de-facto 3D graphics API standard, and utilizes platform-specific OpenGL extensions, such as vertex arrays Žhigh-speed polygon rendering. or multisampling antialiasing Žhigh-quality presentation.. For UNIX platforms not providing an OpenGL runtime environment, plugin versions linked with the OpenGL emulation library Mesa w36x were created. Stereoscoping viewing is supported on several platforms ŽSGI: any Irix workstation; Sun: Elite 3D; Windows NT: Diamond FireGL 1000, HP Kayak XW., using the quad-buffer stereo mode in conjunction with shutter glasses or large screen stereo projection. Head tracking systems ŽCrystal Eyes CE-VR,
2222
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
Table 2 ICC-based Color management functions on different platforms
Initialization Open Profile Close Profile Get Tag Value Create Color Transformation Transform Colors
Microsoft Windows 98 ICM 2.0
SGI Irix 6.5 Coloratura
Sun Solaris KCMS
OpenColorProfile CloseColorProfile GetColorProfileElement CreateMultiProfileTransform TranslateBitmapBits
cmsOpen cmsImportProfile cmsCloseProfile cmsGetTag cmsCreateTfm cmsApplyTfm
KcsAvailable KcsLoadProfile KcsFreeProfile KcsGetAttribute KcsConnectProfiles KcsEvaluate
Polhemus Fastrak, Intersense. have also been integrated, in order to get an immersive virtual reality system. Color management w21x support was integrated to increase reproduction quality of textures as scene components. The presentation of images in TIFF RGB and CMYK formats, originally based on Sam Leffler’s TIFF library, takes the eventually embedded ICC-conforming source profile as well as a monitor-specific ICC destination profile into account, and converts color space and gamut to match the intended colors. It was implemented on SGI Irix 6.5, Sun Solaris, and Microsoft Windows 98, by applying the available color management system ŽCMS. APIs Coloratura, KCMS and ICM 2.0, respectively Žused CMS functions listed in Table 2.. On these platforms the plugin is also capable of viewing TIFF images ŽMIME type: imagertiff or imagerx-tiff, extension: .tif.. Support of collaborative work in performing bidirectional synchronization of the virtual camera between two clients is already implemented in the Unix version. In the future we would like to synchronize not only navigation in 3D space, but also implement 3D tele-pointers and annotations, and a synchronization of the time axis for at least two users in the streaming scenario.
5. Evaluation We built a testbed for the evaluation of the described production process in a typical scientific data visualization application. The components are: 1. DVR dataset: Ø A simulation of an unsteady fluid flow phenomenon Žoceanic convection w38x. on a supercomputer ŽSiemensrFujitsu VPP 300. resulted
in raw data of 740 timesteps, each consisting of temperature and velocity information at 161 = 161 = 31 grid points, stored as 32-bit float values. This dataset was postprocessed by an AVSrExpress application into colored slicers, isosurfaces and arrows, stored as a sequence of 3D scenes in VRML-1.0 format by the module ‘OutputVRML’, and then converted to DVR by our ‘wrl1toDVR’ tool. Ø 740 DVR files, between 5 542 044 bytes and 9 309 468 bytes in size Žwhole DVR dataset: 5.3 Gbyte., each containing about 100 000 primitives Žlines, polygons.. An example of a rendered image is shown in Fig. 4 as case 2. 2. 3D streaming client-server configuration Ø As a client system, where 3D rendering performance is a main issue, we used an SGI Onyx2 Žrack system, 4 processors R10000r195 MHz, 2 Gbytes main memory, 9 Gbytes SCSI disk, Irix 6.5.4. with two Infinite Reality graphics subsystems Žeach with two 64 MB Raster Manager boards.. Ø A 3D streaming serÕer was also installed on this machine for development purposes and testing capabilities in a high-performance TCPrIP scenario Žloopback communication.. The DVR files were stored on a 140 Gbyte RAID system, connected via Fibre Channel. Ø A separate 3D streaming serÕer was implemented on an SGI Origin200 ŽR10000r225 MHz, 512 Mbytes main memory, 9 q 18 Gbyte SCSI disk, Irix 6.5.4.. In this case the DVR files were stored on a 18 Gbyte SCSI disk. Ø A high-speed network between the Onyx2 and the Origin200 was realized by a back-to-back Gigabit Ethernet connection. As a performance optimization, jumbo frame support was enabled on both machines.
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
2223
Table 3 Throughput measurement results Žtypical values in Mbitrs. of the scenario ‘Oceanic convection’ Žframes 720–739.
Read DVR files, one read block per file Žsecond try: cached. TCPrIP transmission from server to client Žttcp -l1048576 -n100 -b262144. 3D rendering ŽDocShow-VR.: OpenGL immediate mode DocShow-VR client-server maximal framerate streaming pipeline 4 framesrsecond 2 framesrsecond
Client SGI Onyx2
Server 1 SGI Onyx2
Server 2 SGI Origin200
-
250–350 Ž980–1020.
109–110 Ž1090–1170.
-
790–950 Žloopback.
560–670 ŽGigabit Ethernet.
380–420
-
-
-
280–320 251 126
324–326 251 126
We measured the throughput of the critical instances separately and the over-all performance of the processing pipeline. Then we justified read and write block sizes and socket buffer sizes in the implementation of our client and server software. In Table 3 we present some characteristic results. Interesting experiences in this scenario are: Ø The throughput in the distributed configuration is higher than in the case of executing client and server on the same machine. Reasons could be: - Lower CPU clock on the client machine Ž195 MHz versus 225 MHz., - Overhead introduced by running 3 processes simultaneously. Ø The sustained overall throughput is significantly slower than the rendering speed, which seemed to be the limiting factor. Possible reasons are: - Swapping latency caused by double-buffered display. Average time: 1rŽ2)frame_rate. Ž1r144 s for 72 Hz frame rate., - X11rMotif event processing at every frame, - Initialization of OpenGL rendering state at startup of every frame. In Fig. 4 we illustrate the rendering performance in typical application scenarios.
6. Conclusion Our concept of preprocessing, real-time streaming, and efficient, high-quality presentation of 3D scenes as Virtual Reality MoÕies, embedded in a WWW browser, has proven to be a powerful tool, in
particular for high-performance application environments, such as exploration or presentation of visualized supercomputer simulation results on a 3D graphics workstation. This is a potential Gigabit-network killer-application, since the geometric rendering data throughput of a state-of-the-art 3D graphics system is in the order of Gbitrs, and the requirements for real-time transmission are similar in a streaming scenario. For example: N triangles, organized as a triangle strip with 3D normals per every 3D vertex have a volume of Ž N q 2.)24 bytes in the case of the usual 32-bit representation for coordinate and normal vector components – this leads with N 4 2 for 6 Mio. trianglesrs to a data rate of above 1 Gbitrs. Already existing advanced distributed application scenarios will require the integration of collaborative functionalities – such as synchronization of viewing parameters, annotations, and video-conferencing in a multi-user environment – and the integration of further media types – such as video and audio streams as parts of an immersive virtual reality presentation.
Acknowledgements This work is partly funded by the DFN-Verein ŽGerman Research Network., with funds from the BMBF ŽGerman Federal Ministry for Education and Research. in the project ‘DFN-Expo’. The authors wish to thank A. von Berg ŽRVS. for the discussion about the high-performance network issues and configuration of Gigabit Ethernet.
2224
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225
References w17x w1x K. Andrews, A. Pesendorfer, M. Pichler, K.H. Wagenbrunn, J. Wolte, Looking inside VRwave: the architecture and interface of the VRwave VRML97 browser, Proc. VRML ’98 Symposium, Monterey, CA, 1998. http:rrwww2.iicm.edur keithrpublications.html w2x J.-F., Balaguer, E. Gobbetti, i3D: a high-speed 3D web browser, Proc. VRML ’95 Symposium, San Diego, 1995. http:rrwww.crs4.itr ; gobettir w3x G. Bell, A. Parisi, M. Pesce, The Virtual Reality Modeling Language – Version 1.0 Specification, 9 November 1995. http:rrwww.vrml.orgrSpecificationsrVRML1.0r w4x T. Berners-Lee, D. Connolly, Hypertext Markup Language – HTML 2.0, RFC 1866, 3 November 1995. ftp:rrnis. nsf.netrdocumentsrrfcr w5x T. Berners-Lee, L. Masinter, M. McCahill, Uniform Resource Locators ŽURL., RFC 1737, 20 December 1994. ftp:rrnis.nsf.netrdocumentsrrfcr w6x W. Broll, DWTP – an Internet protocol for shared virtual environments, Proc. VRML ’98 Symposium, Monterey, CA, 1 9 9 8 . h ttp :r r ece.u w aterlo o .car v rm l9 8 r p ap ersr content.html w7x D. Brutzman, M. Zyda, K. Watsen, M. Macedonia, Virtual Reality Transfer Protocol ŽVRTP. design rationale, Workshop on Enabling Technology: Infrastructure for Collaborative Enterprises ŽWET ICE.: Sharing a Distributed Virtual Reality, MIT, 18–20 June 1997. http:rrwww.stl.nps. navy.milr ; brutzmanrvrtprvrtp_design.ps w8x S. Bryson, C. Levit, The virtual windtunnel, IEEE Computer Graphics and Applications 12 Ž4. Ž1992. . w9x S. Campagna, L. Kobbelt, H. Seidel, Enhancing digital documents by including 3D-models, Computers and Graphics 22 Ž6. Ž1998. . w10x B. Corrie, D. Sitsky, P. Mackerras, Integrating high performance computing and virtual environments, Proc. Seventh Parallel Computing Workshop, Canberra, Australia, September 1997. http:rrcap.anu.edu.aurcaprprojectsrparvisr bibliographyrBCDSPM97.ps.gz w11x P.K. Doenges, T. Capin, K. Capin, F. Lavagetto, J. Ostermann, I.S. Pandzic, E.E. Petajan, MPEG-4: audiorvideo and synthetic graphicsraudio for mixed media, Image Communication Journal 5 Ž4. Ž1997. . w12x K. Engel, R. Grosso, T. Ertl, Progressive Iso-surfaces on the Web, Proc. IEEE Visualization 1998. http:rrwww9. informatik.uni-erlangen.dergerrresearchrpub1998r w13x R. Fielding, J. Gettys, J. Mogul, H. Nielsen, T. Berners-Lee, Hypertext Transfer Protocol - HTTPr1.1, RFC 2068, 3 January 1997. ftp:rrnis.nsf.netrdocumentsrrfcr w14x N. Freed, N. Borenstein, Multipurpose Internet Mail Extensions ŽMIME., RFC 2049, 2 December 1996. ftp:rrnis.nsf. netrdocumentsrrfcr w15x A. Gueziec, G. Taubin, B. Horn, F. Lazarus, A framework ´ for streaming geometry in VRML, IEEE Computer Graphics and Applications 19 Ž2. Ž1999. . w16x S. Gumhold, W. Straber, Real time compression of triangle
w18x w19x
w20x
w21x
w22x
w23x
w24x
w25x
w26x
w27x
w28x
w29x
w30x
w31x
w32x
w33x
mesh connectivity, ACM SIGGRAPH ’98 Conference Proc., July 19–24, 1998. O. Grau, Representation of temporal changes of flexible 3-D objects, International Workshop on Synthetic-Natural Hybrid Coding and Three-Dimensional Imaging, 1997. H. Haase, Symbiosis of virtual reality and scientific visualization system, Computer Graphics Forum 15 Ž3. Ž1996. . H. Haase, F. Dai, J. Strassner, M. Gobel, Immersive investi¨ gation of scientific data, in: G. Nielson et al. ŽEds.., Scientific Visualization – Overviews, Methodologies and Techniques, IEEE Computer Society Press, Rockville, MD, 1997. J. Hardenberg, RE: QvLib questions, VRML Hypermail Archive, 27 March 1995. http:rrvag.vrml.orgrwww-vrmlr archr1107.html M. Has, T. Newman, Color management: current practice and the adoption of a new standard, 1996. http:rrwww. color.orgroverview.html ISOrIEC 14772-1: The Virtual Reality Modeling Language ŽVRML97. – Part 1: Functional specification and UTF-8 encoding, International Standard Organization, 1997. http:rrwww.vrml.orgrSpecificationsrVRML97r ISOrIEC 14772-3: The Virtual Reality Modeling Language ŽVRML97. – Part 3: Compressed Binary Format Specification, Editor’s Draft 5, 1997. R. Klein, Multiresolution representations for surface meshes, Proc. SCCG, 1997. http:rrwww.gris.uni-tuebingen.der peoplerstaffrreinhardrmai97.ps.gz R. Klein, W. Straber, Handling of very large 3D-surfacedatasets using mesh simplification and multiresolution modeling, Tutorial, Computer Graphics International 1998, Hannover, Germany, June 22–26, 1998. http:rrwww.gris. uni-tuebingen.derpeoplerstaffrreinhardrCGI-Tutorialr L. Kobbelt, S. Campagna, H. Seidel, A general framework for mesh decimation, Graphics Interface Proc., Vancouver, BC, 18–20 June 1998. L. Kobbelt, S. Campagna, J. Vorsatz, H. Seidel, Interactive multi-resolution modeling on arbitrary meshes, ACM SIGGRAPH ‘98 Conference Proc., July 19–24, 1998. D.E. Lane, Scientific visualization of large-scale unsteady fluid flows, in: G.M. Nielson et al. ŽEds.., Scientific Visualization – Overviews, Methodologies and Techniques, IEEE Computer Society Press, Rockville, MD, 1997. E. Nebel, L. Masinter, Form-based File Upload in HTML, RFC 1867, 7 November 1995. ftp:rrnis.nsf.netr documentsrrfcr J. Neider, T. Davis, M. Woo, OpenGL Programming Guide – The Official Guide to Learning OpenGL, Release 1. Addison-Wesley, Reading, MA, 1993. Netscape, Netscape Navigator LiveConnectrPlug-In Software Development Kit, 1998. http:rrhome.netscape.comr comprodrdevelopment_partnersrplugin_apir Netscape, Plug-In Guide – Communicator 4.0, January 1998. http:r r developer.netscape.comr docsr manualsrcommunicatorrpluginr S. Olbrich, H. Pralle, High-performance online presentation of complex 3D scenes, IFIP Conference on High Performance Networking, Wien, 1998.
S. Olbrich, H. Praller Computer Networks 31 (1999) 2215–2225 w34x OPC – The OpenGL Performance Characterization Projekt, Viewperf Information and Results. http:rrwww.specbench. orgrgpcropc.staticrviewin;1.html w35x T. Parisi, Corrections: keeping web 3D on course, Keynote address at the VRML ’99 Symposium, Paderborn, 1999. w36x B. Paul, The Mesa 3-D graphics library. http:rrwww.ssec. wisc.edur ; brianprMesa.html w37x M. Pesce, P. Kennard, A. Parisi, Cyberspace, First WWW Conference, May 1994. http:rrwww.hyperreal.orgr ; mpescerwww.html w38x S. Raasch, D. Etling, Modeling deep ocean convection: large eddy simulation in comparison with laboratory experiments, J. Phys. Oceanogr. 28 Ž1998. 1796–1802. w39x W. Schroeder, K. Martin, B. Lorensen, The Visualization Toolkit, 2nd ed., Prentice-Hall, Englewood Cliffs, NJ, 1997. w40x W. Schroeder, Polygon reduction techniques, IEEE Visualization 1995. w41x H. Schulzrinne, A. Rao, R. Lanphier, Real Time Streaming Protocol ŽRTSP., RFC 2326, 14 April 1998. ftp:rrnis.nsf. netrdocumentsrrfcr w42x P. Strauss, G. Bell, The VRML Programming Library – QvLib, Version 1.0 beta 1. 1995. http:rrvag.vrml.orgr www-vrmlrvrml.techrqv.html w43x G. Taubin, J. Rossignac, Geometric compression through topology surgery, IBM Research technical report RC-20340, 16 January 1996. http:rrwww.research.ibm.comrvrmlr binaryrpdfsribm20340.pdf w44x G. Taubin, A. Gueziec, W. Horn, F. Lazarus, Progressive forest split compression, ACM SIGGRAPH ’98 Conference Proc., July 19–24, 1998. w45x G. Taubin, J. Rossignac, Geometric compression through topologic surgery, ACM Transactions on Graphics 17 Ž2. Ž1998. . w46x C. Touma, C. Gotsman, Triangle mesh compression, Graphics Interface Proc., Vancouver, BC, 18–20 June 1998. w47x C. Upson, T. Faulhaber, D. Kamins, D. Laidlaw, D. Schlegel, J. Vroom, R. Gurwitz, A. van Dam, The application visualization system: a computational environment for scientific visualization, IEEE Computer Graphics and Applications, July 1989. w48x J.-S. Vockler, A quick glance at webfilt, RVS, University ¨ Hannover, 3 September 1997.
2225
w49x Web 3D Consortium, The VRML Repository. http:rrwww. web3d.orgrvrmlrvrml.htm w50x J. Wernecke, The Inventor Mentor: Programming ObjectOriented 3D Graphics with Open Inventor. Release 2, Addison-Wesley, Reading, MA, 1994. Stephan Olbrich received his diploma in electrical engineering from the University of Hannover in 1987. From 1987 to 1989 he developed hardware and software at Altec Electronic GmbH in Hannover. Since 1989 he has worked as a research staff member at the Regional Scientific Computing Centre and the Institute for Computer Networks and Distributed Systems. His major interests include multimedia communication and distributed virtual reality systems. Helmut Pralle, University Professor, Dr.-Ing., studied engineering sciences at the Technische Hochschule Karlsruhe and Hannover. Afterwards he started to work as an assistant at the Institute for Applied Mathematics in the Division for Electronic Calculation and at the Computer Centre of the Technical University of Hannover. He received his doctorate in Control Theory in 1967. From 1968 to 1971 he was Head of the Computer Centre of the University of Hannover, since 1971 he has been Director of the Regional Computing Centre of Lower Saxony. In 1986 he got the Chair of the Institute for Computer Networks and Distributed Systems at the Department of Electronical Engineering and Information Technology of Hannover. Since 1996 he has been a member of the administrative council and the executive committee of the DFN Association. He delivers expert reports and vocational guidance for the German Research Council ŽDFG., ministries and the industry. His major interests include distributed information systems and high-speed data communication.