- Email: [email protected]
Virtual reality for orthognathic surgery: The augmented reality environment concept
J Oral Maxillofac Surg 55:456-462, 1997
Virtual Reality for Orthogna thic Surgery: The Augmented Reality Environment Concept WERNER
ARNE WAGNER, MD* MICHAEL RASSE, MD, DDS,t MILLESI, MD, DDS,$ AND ROLF EWERS, MD, DD, DDS, PhD,§
Purpose: The objective of this study was to apply virtual reality technology to osteotomies of the facial skeleton. Materials and Methods: Augmented reality can be considered a hybrid of virtual and real environment spaces, which are coregistered and simultaneously visualized. Using a see-through HMD (head-mounted display) and Interventional Video Tomography intraoperatively, partial visual immersion into a patient-related virtual data space augments the surgeon’s perception as shown in an experimental study and clinical applications. Results: Without limiting the surgical judgment, offering continuous observation of the operating field, the presented technology additionally provides visual access to invisible data of anatomy, physiology, and function and thus guarantees unencumbered and fluent surgery. Conclusion: Despite current shortcomings, augmented reality technology proved to be particularly well suited for use in osteotomies of the facial skeleton.
Virtual Reality (VR) has captured the imagination of the media and the public, yet its applications in medicine are only beginning to be explored. A search of the Medline Database of the National Library of Medicine performed in January 1995 showed that reports of applying virtual reality tools in medicinelm6 have increased by 500% since their introduction into the medical literature, a fact that gives evidence of the growing popularity and importance of VR in medicine.7 In 1987 Watanabe et a18,9introduced a new device for computed intraoperative navigational assistance. Guided by the ‘ ‘Neuronavigator,” they fixed the patient’s skull with a head clamp to assure intraoperative Received from the Clinic Vienna, Waehringer Guertel * Resident. t Senior Consultant. $ Consultant. Q Professor and Head of Address correspondence versitaetsklinik fur Kiefer-und ringer Guertel 18-20, 1090 0 1997 American
Association
0278-2391/97/5505-0005$3.00/O
of Maxillofacial Surgery, University 18-20, 1090 Vienna, Austria.
Clinic. and reprint requests Gesichtschirurgie, Wien, Austria.
of
to Dr Wagner: UniAKH Wien, Waeh-
of Oral and Maxillofacial
Surgeons
calibration. Watanabe et al used a pointing device in connection with a mechanical tracking arm that allowed real-time visualization of the tip of his instrument as an overlay graphic point on computed tomography (CT) images. Advantages of this and similar systems”-13 were appreciated, and their application in neurosurgical procedures were found to be useful. In an attempt to overcome the disadvantages inherent in current systems, the Clinic of Maxillofacial Surgery at the University of Vienna adapted the VPS system by ARTMA Biomedical Inc (Vienna, Austria), for use in the maxillofacial region. The ultimate objective was to provide support to the surgeon during the operation by visualizing invisible topography and instruments in a nonobtrusive way. Our goal was to explore this concept and to promote the development of a system in which the surgeon was partially immersed within the virtual data space. Partial immersion is a hybrid of virtual and real environment spaces and thus can be termed composite reality14 or augmented reality. To accomplish this, the surgeon wears a see-through head-mounted display (HMD)15-1s to see virtual medical structures orthotopically superimposed on the real patient. Thus, the technology should enhance rather than replace the real environment, and thereby aug-
WAGNER
457
ET AL
FIGURE 1. Computer videophotograph as seen through the HMD. Sterolithographic skull model with virtual overlay graphic indicating the preplanned Le Fort I osteotomy line in blue, both palatal arteries and the tip of the saw in red, handpiece of the saw in blue.
mentation of the surgeon’s perception should improve rather than limit surgical judgment. When applying VR technology to osteotomies of the facial skeleton, the objective was to transfer points, lines, and planes from cephalometric drawings, stereolithographic skull models, splints, and imaging data to the patient. Virtual visualization should be accomplished intraoperatively by superimposition of these data. Materials
and Methods
Superimposition of virtual points, lines, and planes on the patient was made possible by defining a spatial relationship between these structures and fiducial points. Fiducials are defined points on a patient that are also discernible in images used for three-dimensional (3D) data acquisition. Preoperatively, anatomic structures, their positional changes caused by surgical procedures, and the osteotomy are simulated in virtual space. During surgery, the position of the bone segment or soft tissue to be obtained is visualized as a real-time overlay. The osteotomy is visualized and superimposed on the patient. This superimposition is brought to the surgeon’s eye by use of a partially immersive HMD (Fig 1). The surgeon watches the operating field and, at the same time, recognizes computer-generated navigational data in a pseudo-natural and unencumbering way. The proprietary principle of IVT (ARTMA), is the dynamic correlation of video stereophotogrametry with the patient’s real-time position data. These data are provided by 6-degrees-offreedom sensors, tracking the subject’s anatomy, and additionally, medical instruments and imaging devices. Advancements in VR technology have led to the development of various digitizing technologies that match
virtual environments to the real world. In the presented configuration, the ARTMA Virtual Patient System is equipped with an electromagnetic tracking device. A rigidly attached sensor continuously registers spatially defined data during the real-time movement of any structure and provides permanent updating in real time. A video camera is used for acquisition of live real-time single-image frames. For every individual video frame, the position and orientation of the charged couplet device (CCD) imaging plane is unambiguously defined and the optical characteristics and distortion of the video image are simulated in a mathematical system of equations. It is then possible to project sensor data on the imaging plane for simultaneous orthotopic matching of virtual (imaged) data with real world (patient) data. The virtual data then become any structure identified and marked in the cephalograms, CT scans, or magnetic resonance (MR) images that can be backprojected in the correct spatial position on the IVT image set. The unique advantage of this interactive backprojection technology is manipulation of the virtual computer-generated graphic overlay production. Anatomic structures are reconstructed in three dimensions from planar cephalograms by stereophotometric analysis. 1g*20A stereophotometric equational algorithm involving at least eight intrinsic fiducial markers is used. They are not fixed but are reconstructed and matched by digitizing their position using a digitizing stylus. The position of the fiducial markers can be confirmed at any time during the clinical study or surgical procedure and updated so that the augmented reality field can be continuously monitored for accuracy by the surgeon. In this system, the extension of the IVT fusion of the virtual imaged data with digitally processed live video sequences can be considered augmented reality of live image fusion. The partial immersive visor display deliv-
FIGURE 2. Hardware setup. The operating microscope provides video input. Note the computer workstation and an HMD prototype.
458
AUGMENTED
REALITY
ENVIRONMENT
FOR
ORTHOGNATHIC
FIG1 JRE! light weigl with an video 3 can
ers spatial localizing and navigational information and virtual imaging of nonvisual structures to the operating surgeon in an unencumbered, less restricting way, without interrupting the flow of the surgery. Clinical
Experience
Before actual surgical intervention, the planned procedure was simulated in a preclinical experimental study using a stereolithographic skull model.21 Figure 2 shows the intraoperative visualization of a pre-
FIGURE 4. Sterile sheathed 3D sensor rigidly head tracks every movement of the head.
attached
to the fore-
SURGERY
Surgeon usi ng a see-through I -IMD dditional mini ature -a intraoperativ ely.
planned Le Fort I osteotomy line as a blue virtual overlay graphic. The handpiece of the saw is visualized in blue, whereas the blade of the saw and the palatal arteries on both sides are discernible as red lines. When planning the model operation, we assumed that this could reduce the risk of damage of the palatal vessels intraoperatively. During the actual Le Fort I osteotomy, we found it appropriate to visualize the positioning of the upper jaw in combination with the simultaneous soft tissue
FIGURE 5. Image coordinate 3D sensor stylus for digitizing coordinate systems of virtual patient.
transformation The surgeon uses a fiducial points, thereby matching the preoperative planning and the real
WAGNER
ET AL
FIGURE 6. Computer videophotograph as seen through the HMD. Virtual overlay graphic of the preplanned osteotomy line superimposed on the maxillary bone guides the tip of the oscillating saw (arrow).
changes. The predicted and intended soft tissue changes could be compared with the actual acquired profile in real time. If not determined by the occlusion, intraoperative decisions concerning the amount of advancement could be made with respect to the cephalometric analysis. The cephalometric analysis can be visualized by virtual overlay graphics intraoperatively. Conversely, the patient’s actual profile line can be real time overlayed on the cephalograms. This applies to any point, line, plane, or structure of the patient’s face. Figure 3 shows the hardware setup. In this particular case, an operating microscope was used for video input. When investigating the benefits of various HMDs
FIGURE 7. Surgeon’s perspective. When using a seethrough HMD the physician views the operating field enchanced by fusion of cyberspace-graphics with the real world. The yellow line indicates the preoperatively planned osteotomy.
(Fig 3), this see-through HMD in combination with a miniature camera provided reasonable visualization quality. Preoperatively, a sterile sheathed 3D tracker was rigidly attached to the patient’s forehead and draped completely (Fig 4), defining a patient-related coordinate system. In the next step, fiducial markers were digitized for image coordinate transformation. Repeated intraoperatively, this controls accuracy (Fig 5). Figures 6 and 7 show the operating field from the surgeon’s perspective. The preplanned osteotomy is superimposedon the maxilla, serving asintraoperative navigation information for the oscillating saw. The resulting Le Fort I osteotomy matches the virtual super-
460
AUGMENTED
REALITY
ENVIRONMENT
FOR
ORTHOGNATHIC
SURGERY
FIGURE! 8. Computer videophotograph of four split-screen views showing virtual control of the positioning procedure. A virtual interincisional point (arrows) provides numeric information about the upper jaw’s spatial position in real time.
imposition. Virtual control of positioning of the maxilla (Fig S), by use of a virtual interincisal points providing numeric information on the spatial position of the upper jaw, was used in addition to the prefabricated occlusal splint. When simulating soft tissue changes of the upper lip, as shown in Figures 9 and 10, the preoperative and preplanned postoperative overlay graphics on the drawing as well as on the postoperative live video
show accurate matching of the virtually planned procedures and the surgical result. The blue lines correlate with the upper lip contour; the two colored points indicate the preoperative and postoperative incisor position (Figs 9, 11). Discussion Since 1989, we have investigated various applications of augmented reality environment technology in
FIGURE 9. Computer videophotograph of four split-screen views of the subject. The two colored points (short arrows) correspond to the preoperative and postoperative incisor position. The blue lines (long arrows) indicate the upper lip contour and thus show planned soft tissue changes intraoperatively.
WAGNER
ET AL
461
FIGURE 10. Computer videophotograph as seen through the HMD. Virtual superimposition of these graphics on the patient show correct soft tissue advancement.
maxillofacial surgery.22-26 Subsequently, we used VR technology in orthognathic surgery. In 1995, we introduced the virtual visualization of preplanned osteotomy lines and the virtual visualization of positioning of osteotomized bone segments intraoperatively. Accurate matching and simultaneous visualization of virtual data and the real world can be termed “augmented reality.” To obtain this, the user wears a seethrough HMD to see virtual medical structures superimposed on the real patient. Partially immersive methods allow complete interaction with the real world and simultaneously make accessible the virtual data envi-
FIGURE 11. Computer videophotograph as seen through the HMD. Correct position of the maxilla postoperatively indicated by the two colored interincision points.
ronment, thereby providing the safety needed when performing surgical procedures. Virtual image-guided surgery either reduced or did not add operative time, but it did increase preoperative preparation time. Preoperative data collection, image processing, preparations for image coordinate transformation, and preoperative planning are time consuming. In general, surgery using virtual image guidance was believed to provide additional safety compared with surgery performed without it. Being able to view an unrestricted surgical field and at the same time recognize virtual anatomic and functional data orthotopi-
DISCUSSION
462
tally superimposed on the real patient allows surgery to become less invasive. In orthognathic surgery and in osteotomies of the facial skeleton, augmented reality concepts have, for the first time, allowed transparent transfer of preoperative planning to intraoperative visualization. Whenever bone segments are transferred, virtual structures serve as guidelines intraoperatively and, in addition show every motion of the osteotomized bone in relation to imaging data and cephalometric drawing (Figs 8, 9). For instance, in correction of posttraumatic enophthalmos, augmented reality can be used for intraoperative visualization of a symmetrical position of the globe. System accuracy and consistancy are over-riding concerns. Because the surgeon relies on other cues and skills during surgery, he or she can always check the accuracy of the system using fiducial markers and noting what is encountered compared with what the virtual images show. The system showed good, but still insufficient, accuracy for extremely accurate procedures (ie, less than 1 mm) in the operating room environment, because the system’s coregistration error cannot be expected to decrease the tracking system’s error. For instance, for determination of occlusal relations, a splint is still better. In this sense, it can be stated that it is not always possible to be highly accurate; therefore, it is of paramount importance that the surgeon can always quantify system inaccuracy by direct visualization. Despite little shortcomings, however, our system has already proved to be a valuable addition to our surgical armamentarium and is therefore used in selected cases. References 1. Dunkley P: Virtual reality in medical training. Lancet 343: 1218, 1994 2. Frisbie AG: Advances in educational technology and journeys into virtual reality. J-Allied-IVD, CD-I, Health 22:131, 1993 3. Kaltenbom KF, Rienhoff 0: Virtual reality in medicine. Methods Inf Med 32:407, 1993 4. Noar MD: Endoscopy simulation: A brave new world. Endoscopy 23:147, 1991
5. Noar MD: Endoscopy simulation training devices. Endoscopy 24: 159. 1992 6. Vannier i’&W: Computer applications in radiology. Curr Opin Radio1 3:258. 1991 7. Gupta SC, Klein SA, Barker JH, et al: Introduction of new technology to clinical practice: A guide for assessment of new VR applications. J Med Virtual Reality 1: 16, 1995 8. Watanabe E. Watanabe S. Manaka S. et al: Three-dimensional digitizer ;Neuronavigaior): New equipment for CT-guided stereotaxis surgerv. Surg Neurol 27:543. 1987 9. Watanabe E, Mayaiagi Y: Kosugi Y, et al: Open surgery assisted by the neuronavigator, a stereotactic, articulated, sensitive arm. Neurosurgery 28:792, 1991 10. Fialkov JA, Phillips JH, Gruss JS, et al: stereotactic system for guiding complex craniofacial reconstruction. Plast Reconstr Surg 89:340, 1992 11. Giorgi C, Luzzara M: A computer controlled stereotactic arm: Virtual reality in neurosurgical procedures. Acta Neurochir Sum11Wien 58:75, 1993 12. Has&id S, Miihling J. ZGller J: Intraoperative navigation in oral and maxillofacial surgery. Int J Oral Maxillofac Surg 24:111, 1995 13. Reinhardt H, Meyer H, Amrein E: A computer-assisted device for the intraooerative CT-correlated localization of brain tumors. Eur Su;g Res 20:51, 1988 14. Weinberg R: Neurosurgerv for third millenium: Neurosurgical topics: Vol 2. Ame&& Association of Neurological %urgeons, 1992, pp 47-63 15. Stix G: See-through view: Virtual reality may guide physicians hands. Sci Am 267:166, 1992 16. Stix G: Reach out: Touch is added to virtual reality simulations. Sci Am 264:134, 1991 17. Wickham JEA: Future developments: Minimally invasive sur-zerv. Br Med J 308:193. 1994 18. Wi&ms M, Wann JP: Binocular vision in a virtual world. Ouhthalmic Phvsiol Out 13:387. 1993 19. Lo&et-Higgins fiC: A computer ‘dgorithm for reconstructing a scene from two projections. Nature 293:133, 1981 20. Metz CE, Fencil LE: Determination of three-dimensional structure in biplane radiography without prior howledge of the relationship between the two views. Theor Med Phys 16:45, 1989 21. Fencil LE, Metz CE: Propagation and reduction of error in three-dimensional structure determined from biolane views of unknown orientation. Med Phys 17:951, 1996 22. Ploder 0, Wagner A, Enislidis G. et al: Comuuter assisted intraopera&e v&ualization of den&l implants: Augmented reality in medicine. Der Radiologe 35:569, 1995 23. Wagner A, Ploder 0, Enislidis G, et al: Virtual image guided navigation in tumor surgery: Technical innovation. J Craniomaxillofac Surg 23:271, 1995 24. Wagner A, Ploder 0, Enislidis G, et al: NAVIGOS: Navigation assistance by a virtual image guiding operation system.Initial description and model operation on a stereolithographic skull model. Stomatol 93:87-90, 1996 25. Wagner A, Ploder 0, Truppe M, et al: Image guided surgery. Int J Oral Maxillofac Surg 25:147, 1996 26. Wagner A, Ploder 0, Zuniga J, et al: Augmented reality environment for temporomandibular joint motion analysis. Int J Adult Orthod Orthognath Surg 11:127, 1996 I
.
J Oral Maxillofac Surg 55:462-463. 1997
Discussion Virtual Reality in Orthognathic Surgery: The Augmented Reality Environment Concept Herman F. Sailer, MD, DDS and Ulrich Longerich, MD, DDS University
Hospital
Zurich,
Zurich,
Switzerland
The Virtual Patient System (VPS) described in this article and the further development, the Interventional Videotomography (IVT), offer the possibilities of so-called composite or augmented reality. VPS and IVT are based on the same software, with the difference being that IVT transmits the image data added to a special conference system via Internet,
ISDN, or TCP/IP.
Virtual Reality for Orthogna thic Surgery: The Augmented Reality Environment Concept WERNER
ARNE WAGNER, MD* MICHAEL RASSE, MD, DDS,t MILLESI, MD, DDS,$ AND ROLF EWERS, MD, DD, DDS, PhD,§
Purpose: The objective of this study was to apply virtual reality technology to osteotomies of the facial skeleton. Materials and Methods: Augmented reality can be considered a hybrid of virtual and real environment spaces, which are coregistered and simultaneously visualized. Using a see-through HMD (head-mounted display) and Interventional Video Tomography intraoperatively, partial visual immersion into a patient-related virtual data space augments the surgeon’s perception as shown in an experimental study and clinical applications. Results: Without limiting the surgical judgment, offering continuous observation of the operating field, the presented technology additionally provides visual access to invisible data of anatomy, physiology, and function and thus guarantees unencumbered and fluent surgery. Conclusion: Despite current shortcomings, augmented reality technology proved to be particularly well suited for use in osteotomies of the facial skeleton.
Virtual Reality (VR) has captured the imagination of the media and the public, yet its applications in medicine are only beginning to be explored. A search of the Medline Database of the National Library of Medicine performed in January 1995 showed that reports of applying virtual reality tools in medicinelm6 have increased by 500% since their introduction into the medical literature, a fact that gives evidence of the growing popularity and importance of VR in medicine.7 In 1987 Watanabe et a18,9introduced a new device for computed intraoperative navigational assistance. Guided by the ‘ ‘Neuronavigator,” they fixed the patient’s skull with a head clamp to assure intraoperative Received from the Clinic Vienna, Waehringer Guertel * Resident. t Senior Consultant. $ Consultant. Q Professor and Head of Address correspondence versitaetsklinik fur Kiefer-und ringer Guertel 18-20, 1090 0 1997 American
Association
0278-2391/97/5505-0005$3.00/O
of Maxillofacial Surgery, University 18-20, 1090 Vienna, Austria.
Clinic. and reprint requests Gesichtschirurgie, Wien, Austria.
of
to Dr Wagner: UniAKH Wien, Waeh-
of Oral and Maxillofacial
Surgeons
calibration. Watanabe et al used a pointing device in connection with a mechanical tracking arm that allowed real-time visualization of the tip of his instrument as an overlay graphic point on computed tomography (CT) images. Advantages of this and similar systems”-13 were appreciated, and their application in neurosurgical procedures were found to be useful. In an attempt to overcome the disadvantages inherent in current systems, the Clinic of Maxillofacial Surgery at the University of Vienna adapted the VPS system by ARTMA Biomedical Inc (Vienna, Austria), for use in the maxillofacial region. The ultimate objective was to provide support to the surgeon during the operation by visualizing invisible topography and instruments in a nonobtrusive way. Our goal was to explore this concept and to promote the development of a system in which the surgeon was partially immersed within the virtual data space. Partial immersion is a hybrid of virtual and real environment spaces and thus can be termed composite reality14 or augmented reality. To accomplish this, the surgeon wears a see-through head-mounted display (HMD)15-1s to see virtual medical structures orthotopically superimposed on the real patient. Thus, the technology should enhance rather than replace the real environment, and thereby aug-
WAGNER
457
ET AL
FIGURE 1. Computer videophotograph as seen through the HMD. Sterolithographic skull model with virtual overlay graphic indicating the preplanned Le Fort I osteotomy line in blue, both palatal arteries and the tip of the saw in red, handpiece of the saw in blue.
mentation of the surgeon’s perception should improve rather than limit surgical judgment. When applying VR technology to osteotomies of the facial skeleton, the objective was to transfer points, lines, and planes from cephalometric drawings, stereolithographic skull models, splints, and imaging data to the patient. Virtual visualization should be accomplished intraoperatively by superimposition of these data. Materials
and Methods
Superimposition of virtual points, lines, and planes on the patient was made possible by defining a spatial relationship between these structures and fiducial points. Fiducials are defined points on a patient that are also discernible in images used for three-dimensional (3D) data acquisition. Preoperatively, anatomic structures, their positional changes caused by surgical procedures, and the osteotomy are simulated in virtual space. During surgery, the position of the bone segment or soft tissue to be obtained is visualized as a real-time overlay. The osteotomy is visualized and superimposed on the patient. This superimposition is brought to the surgeon’s eye by use of a partially immersive HMD (Fig 1). The surgeon watches the operating field and, at the same time, recognizes computer-generated navigational data in a pseudo-natural and unencumbering way. The proprietary principle of IVT (ARTMA), is the dynamic correlation of video stereophotogrametry with the patient’s real-time position data. These data are provided by 6-degrees-offreedom sensors, tracking the subject’s anatomy, and additionally, medical instruments and imaging devices. Advancements in VR technology have led to the development of various digitizing technologies that match
virtual environments to the real world. In the presented configuration, the ARTMA Virtual Patient System is equipped with an electromagnetic tracking device. A rigidly attached sensor continuously registers spatially defined data during the real-time movement of any structure and provides permanent updating in real time. A video camera is used for acquisition of live real-time single-image frames. For every individual video frame, the position and orientation of the charged couplet device (CCD) imaging plane is unambiguously defined and the optical characteristics and distortion of the video image are simulated in a mathematical system of equations. It is then possible to project sensor data on the imaging plane for simultaneous orthotopic matching of virtual (imaged) data with real world (patient) data. The virtual data then become any structure identified and marked in the cephalograms, CT scans, or magnetic resonance (MR) images that can be backprojected in the correct spatial position on the IVT image set. The unique advantage of this interactive backprojection technology is manipulation of the virtual computer-generated graphic overlay production. Anatomic structures are reconstructed in three dimensions from planar cephalograms by stereophotometric analysis. 1g*20A stereophotometric equational algorithm involving at least eight intrinsic fiducial markers is used. They are not fixed but are reconstructed and matched by digitizing their position using a digitizing stylus. The position of the fiducial markers can be confirmed at any time during the clinical study or surgical procedure and updated so that the augmented reality field can be continuously monitored for accuracy by the surgeon. In this system, the extension of the IVT fusion of the virtual imaged data with digitally processed live video sequences can be considered augmented reality of live image fusion. The partial immersive visor display deliv-
FIGURE 2. Hardware setup. The operating microscope provides video input. Note the computer workstation and an HMD prototype.
458
AUGMENTED
REALITY
ENVIRONMENT
FOR
ORTHOGNATHIC
FIG1 JRE! light weigl with an video 3 can
ers spatial localizing and navigational information and virtual imaging of nonvisual structures to the operating surgeon in an unencumbered, less restricting way, without interrupting the flow of the surgery. Clinical
Experience
Before actual surgical intervention, the planned procedure was simulated in a preclinical experimental study using a stereolithographic skull model.21 Figure 2 shows the intraoperative visualization of a pre-
FIGURE 4. Sterile sheathed 3D sensor rigidly head tracks every movement of the head.
attached
to the fore-
SURGERY
Surgeon usi ng a see-through I -IMD dditional mini ature -a intraoperativ ely.
planned Le Fort I osteotomy line as a blue virtual overlay graphic. The handpiece of the saw is visualized in blue, whereas the blade of the saw and the palatal arteries on both sides are discernible as red lines. When planning the model operation, we assumed that this could reduce the risk of damage of the palatal vessels intraoperatively. During the actual Le Fort I osteotomy, we found it appropriate to visualize the positioning of the upper jaw in combination with the simultaneous soft tissue
FIGURE 5. Image coordinate 3D sensor stylus for digitizing coordinate systems of virtual patient.
transformation The surgeon uses a fiducial points, thereby matching the preoperative planning and the real
WAGNER
ET AL
FIGURE 6. Computer videophotograph as seen through the HMD. Virtual overlay graphic of the preplanned osteotomy line superimposed on the maxillary bone guides the tip of the oscillating saw (arrow).
changes. The predicted and intended soft tissue changes could be compared with the actual acquired profile in real time. If not determined by the occlusion, intraoperative decisions concerning the amount of advancement could be made with respect to the cephalometric analysis. The cephalometric analysis can be visualized by virtual overlay graphics intraoperatively. Conversely, the patient’s actual profile line can be real time overlayed on the cephalograms. This applies to any point, line, plane, or structure of the patient’s face. Figure 3 shows the hardware setup. In this particular case, an operating microscope was used for video input. When investigating the benefits of various HMDs
FIGURE 7. Surgeon’s perspective. When using a seethrough HMD the physician views the operating field enchanced by fusion of cyberspace-graphics with the real world. The yellow line indicates the preoperatively planned osteotomy.
(Fig 3), this see-through HMD in combination with a miniature camera provided reasonable visualization quality. Preoperatively, a sterile sheathed 3D tracker was rigidly attached to the patient’s forehead and draped completely (Fig 4), defining a patient-related coordinate system. In the next step, fiducial markers were digitized for image coordinate transformation. Repeated intraoperatively, this controls accuracy (Fig 5). Figures 6 and 7 show the operating field from the surgeon’s perspective. The preplanned osteotomy is superimposedon the maxilla, serving asintraoperative navigation information for the oscillating saw. The resulting Le Fort I osteotomy matches the virtual super-
460
AUGMENTED
REALITY
ENVIRONMENT
FOR
ORTHOGNATHIC
SURGERY
FIGURE! 8. Computer videophotograph of four split-screen views showing virtual control of the positioning procedure. A virtual interincisional point (arrows) provides numeric information about the upper jaw’s spatial position in real time.
imposition. Virtual control of positioning of the maxilla (Fig S), by use of a virtual interincisal points providing numeric information on the spatial position of the upper jaw, was used in addition to the prefabricated occlusal splint. When simulating soft tissue changes of the upper lip, as shown in Figures 9 and 10, the preoperative and preplanned postoperative overlay graphics on the drawing as well as on the postoperative live video
show accurate matching of the virtually planned procedures and the surgical result. The blue lines correlate with the upper lip contour; the two colored points indicate the preoperative and postoperative incisor position (Figs 9, 11). Discussion Since 1989, we have investigated various applications of augmented reality environment technology in
FIGURE 9. Computer videophotograph of four split-screen views of the subject. The two colored points (short arrows) correspond to the preoperative and postoperative incisor position. The blue lines (long arrows) indicate the upper lip contour and thus show planned soft tissue changes intraoperatively.
WAGNER
ET AL
461
FIGURE 10. Computer videophotograph as seen through the HMD. Virtual superimposition of these graphics on the patient show correct soft tissue advancement.
maxillofacial surgery.22-26 Subsequently, we used VR technology in orthognathic surgery. In 1995, we introduced the virtual visualization of preplanned osteotomy lines and the virtual visualization of positioning of osteotomized bone segments intraoperatively. Accurate matching and simultaneous visualization of virtual data and the real world can be termed “augmented reality.” To obtain this, the user wears a seethrough HMD to see virtual medical structures superimposed on the real patient. Partially immersive methods allow complete interaction with the real world and simultaneously make accessible the virtual data envi-
FIGURE 11. Computer videophotograph as seen through the HMD. Correct position of the maxilla postoperatively indicated by the two colored interincision points.
ronment, thereby providing the safety needed when performing surgical procedures. Virtual image-guided surgery either reduced or did not add operative time, but it did increase preoperative preparation time. Preoperative data collection, image processing, preparations for image coordinate transformation, and preoperative planning are time consuming. In general, surgery using virtual image guidance was believed to provide additional safety compared with surgery performed without it. Being able to view an unrestricted surgical field and at the same time recognize virtual anatomic and functional data orthotopi-
DISCUSSION
462
tally superimposed on the real patient allows surgery to become less invasive. In orthognathic surgery and in osteotomies of the facial skeleton, augmented reality concepts have, for the first time, allowed transparent transfer of preoperative planning to intraoperative visualization. Whenever bone segments are transferred, virtual structures serve as guidelines intraoperatively and, in addition show every motion of the osteotomized bone in relation to imaging data and cephalometric drawing (Figs 8, 9). For instance, in correction of posttraumatic enophthalmos, augmented reality can be used for intraoperative visualization of a symmetrical position of the globe. System accuracy and consistancy are over-riding concerns. Because the surgeon relies on other cues and skills during surgery, he or she can always check the accuracy of the system using fiducial markers and noting what is encountered compared with what the virtual images show. The system showed good, but still insufficient, accuracy for extremely accurate procedures (ie, less than 1 mm) in the operating room environment, because the system’s coregistration error cannot be expected to decrease the tracking system’s error. For instance, for determination of occlusal relations, a splint is still better. In this sense, it can be stated that it is not always possible to be highly accurate; therefore, it is of paramount importance that the surgeon can always quantify system inaccuracy by direct visualization. Despite little shortcomings, however, our system has already proved to be a valuable addition to our surgical armamentarium and is therefore used in selected cases. References 1. Dunkley P: Virtual reality in medical training. Lancet 343: 1218, 1994 2. Frisbie AG: Advances in educational technology and journeys into virtual reality. J-Allied-IVD, CD-I, Health 22:131, 1993 3. Kaltenbom KF, Rienhoff 0: Virtual reality in medicine. Methods Inf Med 32:407, 1993 4. Noar MD: Endoscopy simulation: A brave new world. Endoscopy 23:147, 1991
5. Noar MD: Endoscopy simulation training devices. Endoscopy 24: 159. 1992 6. Vannier i’&W: Computer applications in radiology. Curr Opin Radio1 3:258. 1991 7. Gupta SC, Klein SA, Barker JH, et al: Introduction of new technology to clinical practice: A guide for assessment of new VR applications. J Med Virtual Reality 1: 16, 1995 8. Watanabe E. Watanabe S. Manaka S. et al: Three-dimensional digitizer ;Neuronavigaior): New equipment for CT-guided stereotaxis surgerv. Surg Neurol 27:543. 1987 9. Watanabe E, Mayaiagi Y: Kosugi Y, et al: Open surgery assisted by the neuronavigator, a stereotactic, articulated, sensitive arm. Neurosurgery 28:792, 1991 10. Fialkov JA, Phillips JH, Gruss JS, et al: stereotactic system for guiding complex craniofacial reconstruction. Plast Reconstr Surg 89:340, 1992 11. Giorgi C, Luzzara M: A computer controlled stereotactic arm: Virtual reality in neurosurgical procedures. Acta Neurochir Sum11Wien 58:75, 1993 12. Has&id S, Miihling J. ZGller J: Intraoperative navigation in oral and maxillofacial surgery. Int J Oral Maxillofac Surg 24:111, 1995 13. Reinhardt H, Meyer H, Amrein E: A computer-assisted device for the intraooerative CT-correlated localization of brain tumors. Eur Su;g Res 20:51, 1988 14. Weinberg R: Neurosurgerv for third millenium: Neurosurgical topics: Vol 2. Ame&& Association of Neurological %urgeons, 1992, pp 47-63 15. Stix G: See-through view: Virtual reality may guide physicians hands. Sci Am 267:166, 1992 16. Stix G: Reach out: Touch is added to virtual reality simulations. Sci Am 264:134, 1991 17. Wickham JEA: Future developments: Minimally invasive sur-zerv. Br Med J 308:193. 1994 18. Wi&ms M, Wann JP: Binocular vision in a virtual world. Ouhthalmic Phvsiol Out 13:387. 1993 19. Lo&et-Higgins fiC: A computer ‘dgorithm for reconstructing a scene from two projections. Nature 293:133, 1981 20. Metz CE, Fencil LE: Determination of three-dimensional structure in biplane radiography without prior howledge of the relationship between the two views. Theor Med Phys 16:45, 1989 21. Fencil LE, Metz CE: Propagation and reduction of error in three-dimensional structure determined from biolane views of unknown orientation. Med Phys 17:951, 1996 22. Ploder 0, Wagner A, Enislidis G. et al: Comuuter assisted intraopera&e v&ualization of den&l implants: Augmented reality in medicine. Der Radiologe 35:569, 1995 23. Wagner A, Ploder 0, Enislidis G, et al: Virtual image guided navigation in tumor surgery: Technical innovation. J Craniomaxillofac Surg 23:271, 1995 24. Wagner A, Ploder 0, Enislidis G, et al: NAVIGOS: Navigation assistance by a virtual image guiding operation system.Initial description and model operation on a stereolithographic skull model. Stomatol 93:87-90, 1996 25. Wagner A, Ploder 0, Truppe M, et al: Image guided surgery. Int J Oral Maxillofac Surg 25:147, 1996 26. Wagner A, Ploder 0, Zuniga J, et al: Augmented reality environment for temporomandibular joint motion analysis. Int J Adult Orthod Orthognath Surg 11:127, 1996 I
.
J Oral Maxillofac Surg 55:462-463. 1997
Discussion Virtual Reality in Orthognathic Surgery: The Augmented Reality Environment Concept Herman F. Sailer, MD, DDS and Ulrich Longerich, MD, DDS University
Hospital
Zurich,
Zurich,
Switzerland
The Virtual Patient System (VPS) described in this article and the further development, the Interventional Videotomography (IVT), offer the possibilities of so-called composite or augmented reality. VPS and IVT are based on the same software, with the difference being that IVT transmits the image data added to a special conference system via Internet,
ISDN, or TCP/IP.