VIRTUAL OTOSCOPY

COMPUTERS IN OTOLARYNGOLOGY

0030-6665 /98 $8.00

+ .OO

VIRTUAL OTOSCOPY Robert Frankenthaler, MD, Vik Moharir, MD, Ron Kikinis, MD, Peter van Kipshagen, Ferenc Jolesz, MD, Chris Umans, and Marvin P. Fried, MD, FACS

Three-dimensional reconstructions as an advanced tool have been researched for otolaryngology-head and neck surgery.*This development was paralleled by advances in imaging Over the past several years, with the improvement in computer technology, a new threedimensional display and visualization tool has emerged-virtual endosVirtual endoscopy consists of a computer-generated three-dimensional view of the patient’s anatomy based on their CT or MR images. The computer rendering provides a continuous luminal view within which one can navigate along the inner surfaces, similar to traditional endoscopy. In addition, virtual endoscopy also can display an external global view in three dimensions and a view of the related CT or MR slice for added orientation and e~amination.~ A number of virtual endoscopic techniques now exist (e.g., bronchoscopy, colonoscopy, and angiography);23,24 however, this article focuses on virtual otoscopy. Specifically, we (1)examine the limitations of traditional imaging techniques and display; (2) discuss potential benefits of virtual endoscopy/otoscopy; (3) describe the process of constructing virtual otoscopic views; (4) show virtual otoscopy images and illustrate their features; and (5) analyze how virtual otoscopy can assist in addressing surgical problems.

*References 1-3,5,10,12-16,20,22,25,27 and 28.

From the Department of Otolaryngology, Joint Center for Otolaryngology, Harvard Medical School (RF, VM, MF); Department of Radiology, Surgical Planning Laboratory (RK), and Division of MR Imaging and Image Guided Therapy (FJ),Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts; University of Leiden, Leiden, The Netherlands (PK); and Computer Science Division, University of California-Berkeley (CU),Berkeley, California OTOLARYNGOLOGICCLINICS OF NORTH AMERICA VOLUME 31 NUMBER 2 APRIL 1998

383

384

FRANKENTHALER et a1

LIMITATIONS OF IMAGING AND DISPLAY

A restricted operative visual field blocks the surgeon’s ability to sight important structures beyond and through the exposed layers of tissue. The advent of minimal access surgery compounds this problem. The ability to plan a trajectory to the target area through tissue volumes is not possible by direct visualization. The surgeon often is aware of the presence of important structures, such as blood vessels, viscera, and nerves only after the removal of the preceding layer. Current imaging techniques can help, but have their limitations in assisting the surgeon intraoperatively; to visualize beyond the skin or exposed tissue layers, the following methods are applicable. Endoscopy

Endoscopy uses optical and video technology to view the inner surface of hollow organs in an uninterrupted fashion. It provides access to many regions and the examiner has full control of the scope, view point, and view angle. In most cases, a natural orifice provides access. Endoscopy only gives information on the surface anatomy of the lumen and practically no information on the anatomy within or beyond the wall. This limitation prevents evaluation of the transmural extent of tumor and limits identifying concomitant localization of contiguous structures. During endoscopy the patient requires sedation and still may be un~omfortable.~ Traditional endoscopy always carries the risk of viscus perforation. The benefit of significant reduction in tissue damage, when compared to surgical exploration, has established endoscopy as a minimally invasive procedure in otolaryngology and other specialties. Currently no method exists for endoscopy through the full extent of the ear. The canal is simply too small and structures such as the tympanic membrane and ossicles prevent the advancement of an endoscope. Noninvasive Imaging Techniques

Noninvasive imaging techniques, such as ultrasound, CT, and MR imaging allow the surgeon to visualize volumes of tissue beyond an exposed surface, enabling identification of anatomic structures and their practical relationships. Ultrasound can be in real time, providing continuous interactive display. However, bone/air interfaces may distort the ultrasound image and make it difficult to interpret. CT can provide an excellent display of bone and air, but soft tissue structures have a low contrast resolution. MR imaging provides excellent soft tissue characterization; however, delineation of bony structures is suboptimal. The major problem with these noninvasive imaging techniques is that the anatomy is displayed in a two-dimensional cross-sectional view; as a result, the surgeon has to mentally integrate a series of images into a three-dimensional s t r u c t ~ r eThis . ~ becomes especially difficult when examining com-

VIRTUAL OTOSCOPY

385

plex anatomic entities, such as the tubular structures of the inner ear, that pass back and forth through multiple cross-sectional slices. Virtual endoscopy is a three-dimensional reconstruction of a patient’s anatomy based on the individual’s CT and MR imaging data. Not only could one navigate through the reconstruction, simulating traditional endoscopic technique, but a concomitant display of the global view and a view of the related CT or MR slice provide an assistance beyond traditional endoscopy in localizing and analyzing tissue. Furthermore, a virtual otoscopy allows visualization in a three-dimensional format where currently one cannot see without performing surgery-the middle and inner ear.

MATERIAL AND METHODS OF VIRTUAL ENDOSCOPY Image Acquisition All routine CT or MR imaging techniques that provide high resolution cross-sectional images can be post-processed to obtain three-dimensional reconstructions. CT scanning for virtual endoscopy is performed using a continuously rotating helical CT system. For MR imaging, T1weighted three-dimensional volumetric gradient-echo sequences are ne~essary.~ The authors used only CT data for the first virtual otoscopy. However, a later reconstruction incorporated both CT and MR images. A cadaveric head of a 55-year-old female constituted the initial virtual otoscopy specimen.

Image Processing The CT images were transferred, from their respected workstations, using internal network connections, to computers (SPARC-20,Sun Microsystems Inc., Mountain View, CAI at our institution’s surgical planning laboratory (SPL). The images underwent refinement using a filtering program. The next step, “segmentation,” consisted of an automatic and manual technique for isolating and outlining anatomic structures. Segmented entities included temporal bone, the three ossicula, saccule, utricle, internal auditory canal, internal carotid artery, facial nerve, stapedius muscle, eustachian tube, and geniculate ganglion. These segmented anatomic structures were stacked into individual three-dimensional objects. Finally, a special computer program combined all of the three-dimensional objects into a complete model, which was used for virtual otoscopy.

RESULTS The authors completed a three-dimensional model of the specimen’s temporal bone, with its associated structures, and displayed it on a Sun

386

FRANKENTHALER et a1

Workstation. Using in-house developed computer software, extensive analysis of the reconstruction took place.

Camera Guidance The two main ways in which the virtual camera moved through the ear consisted of 1. Manual camera movement: Using the mouse, an operator could change the camera position and field of view. A Sun Workstation equipped with the ZX graphics accelerator board (Sun Microsystems, Mountain View, CA) displayed the virtual otoscopy. The computer screen included the virtual endoscopiccamera view, the external view, and a display of the nearest CT slice: (Fig. 1, Plate 1).Virtual endoscopy provided the ability to add or remove anatomic components as desired. From any given location, the operator could quickly obtain a 360" view of the surrounding tissue. The zoom feature allowed one to enlarge the area being studied: (Fig. 2, Plate 1 shows the virtual otoscopy control panelL7 2. Automatic path planning: Most of the tubular organs that were examined presented a challenge for manual camera movement. This was particularly difficult within confined spaces. To guide the virtual endoscope, we adapted a robot path planning algorithm. We treated the camera as a point robot and the walls of the middle ear and internal and external auditory canals as obstacles. We programmed starting and end point coordinates into the computer, which then followed the shortest path between the coordinates while avoiding the walls. During virtual otoscopy, structures that lay in the flight path, such as the tympanic membrane and ossicula, were inspected and then passed through for a continuous study.7

Flight Path A 648 frame 'fly-through of the middle and inner ear was completed. The animation begins by emulating the view obtained during a conventional otoscopy, approachingthe external auditory canal. The virtual camera then passes through the eardrum and changes to 90" optics for an endoscopic view. Next, the virtual camera backs into a recess of the middle ear cavity for a clear view of the incus, malleus, stapes, and stapedius muscle. The camera then moves along the middle ear cavity for a short distance until the cavity walls dissolve to reveal the inner ear structures. The view now includes the facial nerve, the geniculate ganglion, the greater petrosal foramen, the carotid artery, the internal auditory, and the cochlea and semicircular canals, all identified by different colors: (Fig. 3, Plate 2). The camera moves outside and converts to 30"optics. It provides close-up views of a variety of structures. Finally, the camera backs away

VIRTUAL OTOSCOI'Y

387

Figure 1. This is how virtual otoscopy appears on the computer screen. A shows the camera view. B shows the external global view. C represents the related CT slice, with the arrow pointing to the camera location.

for a view from inside the skull where the entry points of the carotid artery and facial nerve are visible.8 DISCUSSION

Virtual otoscopy displays a full three-dimensional representation of the relevant anatomy and allows emulation of a traditional endoscopic viewing with several key advantages: in traditional endoscopy the lumen limits visualization of tissue; however, virtual endoscopy provides the ability to see structures beyond the lumen wall in a three-dimensional

388

FRANKENTHALER et a1

Figure 2. Virtual otoscopy control panel.

format. The simultaneous display of three different views, camera, global, and the related CT or MR slice, results in improved localization of tissue when compared to traditional endoscopy. This could enable a surgeon to better anticipate areas that may present a challenge during the operation. A full three-dimensional global view also could allow a surgeon to plan the best trajectory to target tissue. The three-dimensional format gives excellent representation of the full tubular structures of the temporal bone. The virtual camera overcomes traditional limitations of maneuverability and space by permitting an endoscopic procedure to take place in the middle and inner ear without the risk of perforation. The ability to easily perform a 360" examination at any given location provides a powerful examination tool. Finally, the zoom feature allows close examination of desired areas in a way traditional imaging techniques could not. Based on such promising results, virtual otoscopy can play a tremendous role in analysis of ear and temporal bone pathology. Tumors of the petrous apex and middle ear could be visualized better. Greater information could be obtained on congenital anomalies and the jugular bulb. Finally, the ear atlas that was constructed can play a valuable role in medical education: (Fig. 4A and B, Plate 2). Some problems, however, were encountered during the virtual otoscopy construction; the quality of the initial CT dictates the quality of the virtual otoscopy, and the automated segmentation process sometimes loses details. Initial start-up requires a computer workstation, a software engineer, and a research fellow. The entire reconstruction process, from image transfer to display of the final product, requires several hours (not including the learning curve that exists in performing reconstructions).

VIRTUAL OTOSCOPY

389

Figure 3. Series of frames from the virtual otoscopy (A-C).

Obviously, virtual otoscopy currently would not help in emergency situations. However, if the advancements in computers continue at their current rapid pace, many of these limitations may be overcome in the near future. Furthermore, virtual endoscopy would not help if the endoscopist wanted to acquire a biopsy, or if the patient needed interventional treatment (e.g., cauterization of bleeders). CONCLUSION

Virtual otoscopy provides an excellent way to combine the advantages of traditional endoscopy, CT and MR imaging. This may have po-

390

FRANKENTHALER et a1

Figure 4. A, Temporal bone. 6, Temporal bone removed and inner structures magnified.

tential benefits in surgical evaluation and procedures. To statisticallyconfirm the accuracy and precision of virtual otoscopy, more data should be collected and compared with intraoperative findings. Though time and cost factors presently limit widespread use of virtual otoscopy, future developments in computers and software may overcome any current

VIRTUAL OTOSCOPY

391

drawbacks. Overall, virtual otoscopy stands as a promising new technique for analysis and elucidation of the middle ear, inner ear, and temporal bone.

References 1. Altobelli DE, Kikinis R, et al: Computer assisted three-dimensional planning in craniofacial surgery. Plast Reconstr Surg 92:57&585,1993 2. Cline HE, Dumoulin CL, et a 1 3D reconstruction of the brain from magnetic resonance images using a connectivity algorithm. Magn Reson Imaging 5:345-352,1987 3. Eisele DW, Pichtsmeier WJ, et a 1 Three-dimensional models for head and neck tumor treatment planning. Laryngoscope 104:433-439,1994 4. Fried MP, Hsu L, et a1 Image-guided surgery in a new magnetic resonance suite: Preclinical consideration. Laryngoscope 106411417,1996 5. Girod S, Keeve E, Girod B Advanced interactive craniofacial surgery planning by 3D simulation and visualization. Int J Oral Maxillofac Surg 24120-125,1995 6. Jolesz F: Image-guided procedures. Radiology, in press 7. Jolesz FA, Lorensen WE, et al: Interactive virtual endoscopy, in press 8. Kikinis R, Umans C, et a 1 Virtual Otoscopy. Presentation at “Image Conference,” Scottsdale, AZ, June 23-28,1996 9. Kikinis R, Langham GP, et al: Computer assisted interactive three-dimensional planning for neurosurgical procedures. Neurosurgery 38:640-651,1996 10. Korves S, Klimek L: Image- and model-based surgical planning in otolaryngology. J

Otolaryngol24265-270,1995 11. Lorrensen WE, Jolesz FA, Kikinis R The exploration of cross-sectional data with a virtual

endoscope. In Satava RM, Morgan K, Sieberg, et a1 (eds): Interactive Technology and the New Paradigm for Health-Care: Medicine Meets Virtual Reality 111 Proceedings. Amsterdam, Holland, 10s Press, 1995, pp 221-230 12. Marentette LJ, Maisel RH: Three-dimensional CT reconstruction in midfacial surgery. Otolaryngol Head Neck Surg 98:38-52,1988 13. Marsh JL, Vannier Mw: The ”Third” dimension in Craniofacial surgery. Plast Reconstr Surg 71:759-767,1983 14. Marsh JL,Vannier M W , et a1 Computerized imaging for soft tissue and osseous reconstruction in the Head and Neck. Clin Plast Surg 12279-291,1985 15. McEwan CNN, Fukuta K Recent advances in medical imaging: Surgery planning and simulation. World J Surg 13:343-348,1989 16. Ossoff RH,Reinisch L Computer assisted surgical techniques: A vision for the future of otolaryngology-head and neck surgery. J Otolaryngol23:354-359,1994 17. Rosenberg SI, Siverstein F, et a 1 Endoscopy in otology and neurootology. Am J Otol 15:168-172,1994 18. Schenck JF, Jolesz FA, et al: Superconduction open-configuration MR imaging system for image-guided therapy. Radiology 195:805-814,1995 19. Shenton ME, Kikinis R, et al: Harvard Brain Atlas: A teaching and visualization tool. 20. Shubert 0, Sartor K, et al: Three-dimensional computed display of otosurgical operation sites by spiraI CT. Neuroradiology 386634568,1996 21. Valvassori GE: Update of computed tomography and magnetic resonance in otology. Am J Otol15:203-206,1994 22. Vannier MW, Marsh JL: Three-dimensional imaging, surgical planning, and imageguided therapy. Radio1Clin North Am 34:545-563,1996 23. Vining DJ, Padhani AR, Wood S, et al: Virtual bronchoscopyy: A new perspective for viewing the tracheobronchial tree [abstract]. Radiology 189:438, 1993 24. Vining DJ, Shifrin RY, Grishaw EK, et al: Virtual colonoscopy [abstract]. Radiology: 469, 1994 25. Wiengand DA, Page RB, et al: The surgical workstation: Surgical planning using generic software. Otolaryngol Head Neck Surg 109:434440,1993

392

FRANKENTHALER et a1

26. Yokoyama T, Uemura K. Surgical approach to the internal auditory meatus in acoustic neuroma surgery: Significance of preoperative high-resolution computed tomography. Neurosurgery 39:965-969,1996 27. Zinreich SJ, Mattox DE, et al: 3-D CT for cranial facial and laryngeal surgery. Laryngoscope 98:1212-1219,1988 28. Zonneveld FW, Lobregt S, et al: Three-dimensional imaging in craniofacial surgery. World J Surg 13:328-342,1989

Address reprint requests to Ron Kikinis, MD Department of Radiology, Surgical Planning Laboratory Brigham and Women’s Hospital Harvard Medical School 75 Francis Street Boston, MA 02115