- Email: [email protected]
Live video manipulator for endoscopy and natural orifice transluminal endoscopic surgery (with videos)
Tang et al
Live video manipulator for endoscopy and NOTES
21. Ichikawa D, Kurioka H, Yamaguchi T, et al. Postoperative complications following gastrectomy for gastric cancer during the last decade. Hepato-gastroenterology 2004;51:613-7. 22. Kodera Y, Sasako M, Yamamoto S, et al. Identification of risk factors for the development of complications following extended and superextended lymphadenectomies for gastric cancer. Br J Surg 2005;92:1103-9. 23. Felsher J, Farres H, Chand B, et al. Mucosal apposition in endoscopic suturing. Gastrointest Endosc 2003;58:867-70. 24. Dennert B, Ramirez FC, Sanowski RA. A prospective evaluation of the endoscopic spectrum of overtube-related esophageal mucosal injury. Gastrointest Endosc 1997;45:134-7.
Received October 18, 2007. Accepted March 25, 2008. Current affiliations: Institute of Digestive Disease (P.W.C., J.Y.L., E.K.N., C.C.L., K.F.T., J.J.S.), Department of Surgery (P.W.C., J.Y.L., E.K.N., C.C.L., S.S.C.), Department of Medicine and Therapeutics (J.J.S.), Department of Anatomical and Cellular Pathology (K.F.T.), Department of Microbiology (M.H.), The Chinese University of Hong Kong, Hong Kong. Reprint requests: Philip W. Y. Chiu, Institute of Digestive Disease, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong.
Live video manipulator for endoscopy and natural orifice transluminal endoscopic surgery (with videos) Shou-jiang Tang, MD, Richard Bergs, MS, Saad F. Jazrawi, MD, Christopher O. Olukoga, MD, Jeffrey Caddedu, MD, Raul Fernandez, PhD, Daniel J. Scott, MD Dallas, Arlington, Texas, USA
Background: During fluoroscopy, radiologists and gastroenterologists are able to manipulate live fluoroscopic video for better orientation and visualization. During endoscopy and natural orifice transluminal endoscopic surgery (NOTES), this function is not currently available. Particularly during NOTES, the endoscopic image is sometimes inverted, and off-axis operation is required. Objective: Our purpose was to develop and test a prototype live video manipulator (LVM) for endoscopy, laparoscopy, and NOTES. Design: Prospective ex vivo and in vivo feasibility study. Interventions: We developed a prototype LVM software for video image manipulation that can be easily installed on any computer. The video input is streamed into the computer and can be displayed on a standard monitor. LVM was tested ex vivo in the following functions: (1) instant live video rotation, (2) vertical or horizontal video inversion, (3) mirror imaging, and (4) digital zooming. These functions were also tested during upper and lower GI endoscopy, ERCP, diagnostic laparoscopy, and various transvaginal NOTES procedures (cholecystectomy, gastroenterostomy, and sleeve gastrectomy) in porcine models. Main Outcome Measurements: Image quality observation between unmanipulated and manipulated live videos. Results: LVM reliably and easily performed live video manipulation during these tests. Besides standard definition video signals, LVM is fully compatible with high-definition video endoscopy. Three observers reported that the subjective image quality was the same in specified areas between manipulated and unmanipulated live videos. Limitations: Observation and feasibility study. Conclusions: LVM reliably and conveniently performed live video manipulations. LVM requires minimal equipment, capital investment, and maintenance, and is easy to set up. LVM can be a useful tool in many medical imaging studies, including endoscopy, laparoscopy, and NOTES, either as a built-in technology or as an as-needed add-on feature.
Copyright ª 2008 by the American Society for Gastrointestinal Endoscopy 0016-5107/$32.00 doi:10.1016/j.gie.2008.04.018
Image processing techniques, broadly defined as the manipulation of signals such as photographs and video sequences, are ubiquitous in medicine, astronomy, video communication, and electronic games.1 Video-assisted endoluminal and intracavitary procedures, robotic-assisted
www.giejournal.org
Volume 68, No. 3 : 2008 GASTROINTESTINAL ENDOSCOPY 559
Abbreviations: LVM, live video manipulator; NOTES, natural orifice transluminal endoscopic surgery.
Live video manipulator for endoscopy and NOTES
surgery, and laparoscopy are well established in medicine2-5 and rely on live video presentation and manipulation.6,7 During these video-assisted techniques, the ideal vision system should provide surgeons with the same quality of visual information as with open surgery, namely, high resolution, excellent color quality, precise spatial information, and a constant clear view for optimal surgical intervention.6 Unfortunately, many important problems in surgical image processing remain unsolved. For example, during natural orifice transluminal endoscopic surgery (NOTES), the endoscopic image is occasionally inverted, and off-axis manipulation is required.2 When endotherapy is performed in patients with certain altered anatomic features, the anatomic orientation is unconventional (ie, ampulla appears inverted in Billroth II anatomy).3 To obviate these challenges, we developed and tested a prototype live video manipulator (LVM) for endoscopy, NOTES, and any video-based procedure and surgery. The concept of digitally manipulating video is protected under a filing with the University of Texas Southwestern Medical Center. To our knowledge, neither video manipulation nor a video manipulator for endoscopy has been reported in the medical literature.
METHODS LVM One of the investigators (R. B.) developed a prototype LVM software (Appendix) that can be easily installed on any computer. Any video source, including S video, composite, RBG, and so forth, can be streamed through a video capture device (Dazzle DVD Recorder, Pinnacle Systems, Mountain View, Calif) and into a computer installed with LVM (Fig. 1). The Dazzle DVD recorder is a plug-andplay device and its suggested retail price is US$49.99. The manipulated video signal is directly exported to a monitor for viewing. The manipulated video signal can be recorded as well. Special training is not required to be proficient in the use of this software. The on-screen control panel of the software is self-explanatory. In addition, LVM can capture still images in JPEG format for printout and presentation.
Ex vivo image and video quality tests An Olympus colonoscope (H180 AL/I, Olympus America, Center Valley, Pa) was used to test both image and video quality in terms of resolution, contrast, color reproduction, and time sequence differences between the source and manipulated videos. Two physicians and one engineer participated in the following comparison tests. A Kodak digital science imaging test chart TL-5003 (Eastman Kodak, Rochester, NY) was used for resolution test comparison. Mini-ColorChecker (Munsell Color, X-rite, Grand Rapids, Mich) and an investigator’s fingertip were used for image color reproduction tests. The video signal was split into 2 560 GASTROINTESTINAL ENDOSCOPY Volume 68, No. 3 : 2008
Tang et al
Capsule Summary What is already known on this topic d
During natural orifice transluminal endoscopic surgery (NOTES), the endoscopic image may be inverted, off-axis manipulation is required, and the anatomic orientation may be unfavorable in some patients with certain altered anatomy.
What this study adds to our knowledge d
In a prospective feasibility study of a prototype live video manipulator, 3 observers reported that the subjective image quality was the same in specific areas when comparing live video images with and without manipulation.
cables from an Olympus CV-180 video processor through the composite video output. One cable was connected to a high-definition flat-panel monitor (Olympus OEV-191H, 1080 lines resolution) and printer. The other was connected to a laptop computer installed with LVM by a video capture device (Dazzle DVD recorder). The manipulated video signal was then connected to a second high-definition monitor and printer. The 2 monitors were placed side by side for comparing video quality and time sequence difference. This 2-monitor comparison test was recorded with a digital camcorder (DCR-SR300, Sony Corporation, Tokyo, Japan).
In vivo feasibility testing during endoscopy and NOTES Besides the original video display, video manipulation was performed and displayed on a second monitor during upper and lower endoscopy (n Z 3 each), ERCP (n Z 3), diagnostic laparoscopy (n Z 3), and various transvaginal NOTES procedures (n Z 6, including cholecystectomy, gastroenterostomy, and sleeve gastrectomy) in porcine models. The instituional animal care and utilization committee approved these laparoscopy and NOTES procedures, and our institutional review board approved clinical endoscopy. Olympus CV-160 video processors and non-high definition endoscopes were used for clinical endoscopy and NOTES because of equipment limitations. We compared video quality in terms of resolution, contrast, color reproduction, and time sequence differences between the source and manipulated videos.
RESULTS Throughout the study period, there was no LVM software anomaly or failure. www.giejournal.org
Tang et al
Live video manipulator for endoscopy and NOTES
Figure 1. Schematic illustration of live video manipulator set up during endoscopy and NOTES.
laydwhich was less than 120 millisecondsdwas not noticeable during the 1-monitor test.
In vivo feasibility testing during endoscopy and NOTES
Even with high-definition video signals, all 3 observers reported that the subjective image quality (resolution and color reproduction) was the same between manipulated and unmanipulated live videos (Video 1; available online at www.giejournal.org). The manipulated video stream was smooth and without visible flicker or distortion. During the 2-monitor comparison test, a minimal time delay was present in the manipulated video attributable to computer processing overhead. However, this de-
During upper and lower endoscopy, ERCP, and transvaginal NOTES, the 3 observers felt that the subjective image quality was the same between manipulated and unmanipulated live videos (Figs. 2 through 4, Video 2; available online at www.giejournal.org). The millisecondorder time delay of the manipulated video visible during the 2-monitor test was not noticeable on the LVM monitor alone, and it did not affect or impair endoscopic or NOTES procedures in any regard. During routine diagnostic upper and lower endoscopy and uncomplicated laparoscopy, the manipulated video did not offer extra benefits. During laparoscopy, the laparoscope had to be held without rotation to keep the abdominal wall on the top in the view (horizon-corrected images). Otherwise, the laparoscopic view of the abdominal cavity is rotated. With LVM, the horizon view was easily ‘‘corrected’’ without rotation of the laparoscope. Applying 180-degree video rotation during ERCP in patients with normal gastric anatomy, the biliary ductal orientation and cannulation direction became that of patients with Roux-en-Y or Billroth II anatomy (Fig. 2, Video 2). During NOTES, with video rotation between 90 and 180 degrees, optimal spatial visualization of the abdominal cavity and organs could be easily maintained and endoscopic vision was comparable to that of open or laparoscopic surgery (Figs. 3 and 4, Video 2). The endoscopists and surgeons
www.giejournal.org
Volume 68, No. 3 : 2008 GASTROINTESTINAL ENDOSCOPY 561
Figure 2. Endoscopic image of the ampulla during ERCP. This patient has normal gastric anatomy, with video rotation of 180 degrees the biliary ductal orientation and cannulation direction is similar to that in patients of Roux-en-Y or Billroth II anatomy.
Ex vivo image and video quality tests
Live video manipulator for endoscopy and NOTES
Tang et al
Figure 3. Endoscopic images of abdominal cavity during NOTES. With video rotation between 90-180 degrees, optimal endoscopic visualization of the abdominal cavity and organs is easily maintained and endoscopic vision is comparable to that during open or laparoscopic surgery.
be easily identified. Without video manipulation, the surgeons felt that they would not have been able to complete the dissection.
DISCUSSION
preferred and relied on the manipulated video display in certain situations, such as with retroflexed views, off-axis visualization, multiple instruments manipulating on the tissue from different angles, and so forth. We found that digital zooming was useful under some circumstances. For example, 1 case of gallbladder dissection during transvaginal cholecystectomy was not possible without digital enhancement of our magnetically guided camera and cauterizer.8 Although there was some loss of mucosal or serosal details from image pixilation with digital zooming, organ and tissue structures could still
During fluoroscopy, the radiologists and endoscopists are able to manipulate live fluoroscopic video by means such as horizontal and vertical image inversion, image rotation, and magnificationdall of which enhance orientation and visualization. During endoscopy and NOTES, this function is not currently available. In the current study, we developed and demonstrated the feasibility of a prototype LVM in clinical endoscopy and NOTES. Besides standard-definition video signals, LVM is fully compatible with high-definition video endoscopy. With highdefinition video signals, observers felt that the subjective image quality was the same in the specified areas between manipulated and unmanipulated live videos. The video feed and display was smooth during all tests. Although a slight (50- to 120-millisecond) time delay was noticed at the manipulated video during a 2-monitor test, this difference was not noticeable during the 1-monitor ex vivo and in vivo tests. Most important, the delay did not affect or impair endoscopic and NOTES procedures in any regard. Although video inversion can potentially impair the inexperienced operator’s ability to navigate, we have not encountered any interference during our initial experience with multiple LVM procedures. For example, when ERCP is performed in a patient either in the supine or prone position, the fluoroscopic images are inverted. However, once the images are corrected horizontally, the visualization and the ERCP procedure themselves remain the same. Other possible drawbacks of LVM include extra
562 GASTROINTESTINAL ENDOSCOPY Volume 68, No. 3 : 2008
www.giejournal.org
Figure 4. Endoscopic images of abdominal cavity during NOTES. With video rotation between 90-180 degrees, optimal endoscopic visualization of the abdominal cavity and organs is easily maintained and endoscopic vision is comparable to that during open or laparoscopic surgery.
Tang et al
Live video manipulator for endoscopy and NOTES
Figure 5. Image of on-screen representation of the software control panel.
equipment, extra wiring, the need for an assistant for video manipulation (applicable to the current LVM prototype computer setup), potential software anomalies or failures (which we have not encountered), and minimal image quality degradation from splitting of video signals from the endoscopy processor card. Reassuringly, we have not noticed a significant difference in terms of image quality degradation. We have noticed that there is a change of the accessory channel ‘‘location’’ on the monitor after video inversion. For example, the accessory view will not come out of a diagnostic gastroscope at 9 o’clock on the monitor after video inversion. However, during NOTES, we have not found that this phenomenon interfered with our procedures. LVM use in other endoscopic procedures is questionable and may not be necessary. We did not find that the manipulated video offered extra benefits during routine diagnostic upper and lower endoscopy. Endoscopists are accustomed to off-axis visualization during most of the diagnostic procedures after years of training and practice. Because our current prototype LVM was installed on a standard laptop computer and there were concerns about biologic fluid contamination, an assistant operated LVM according to the endoscopists’ and surgeons’ verbal instructions. However, once it is packaged as a stand-alone device with ergonomic controls, the endoscopists and surgeons would be able to manipulate video independently.
We also envision a future LVM system with voice-recognition capability to further enhance its effectiveness with endoscopists and surgeons. In conclusion, LVM reliably and conveniently performed live video manipulations with both standard and high-definition video signals. LVM can be a useful tool in many medical imaging studies including endoscopy, laparoscopy and NOTESdeither as a built-in technology or as an as-needed add-on feature.
www.giejournal.org
Volume 68, No. 3 : 2008 GASTROINTESTINAL ENDOSCOPY 563
DISCLOSURE The authors report that there are no disclosures relevant to this publication.
REFERENCES 1. Huang TS, Aizawa K. Image processing: some challenging problems. Proc Natl Acad Sci USA 1993;90:9766-9. 2. Rattner DW, Kalloo A. The ASGE/SAGES working group on Natural Orifice Translumenal Endoscopic Surgery White Paper. Gastrointest Endosc 2006;63:199-203. 3. Feitoza AB, Baron TH. Endoscopy and ERCP in the setting of previous upper GI tract surgery, I: reconstruction without alteration of pancreaticobiliary anatomy. Gastrointest Endosc 2001;54:743-9.
Live video manipulator for endoscopy and NOTES 4. Downey RJ. Complications after video-assisted thoracic surgery [review]. Chest Surg Clin North Am 1998;8:907-17. 5. Patil PV, Hanna GB, Cuschieri A. Effect of the angle between the optical axis of the endoscope and the instruments’ plane on monitor image and surgical performance. Surg Endosc 2004;18:111-4. 6. Buess GF, Schurr MO, Fischer SC. Robotics and allied technologies in endoscopic surgery. Arch Surg 2000;135:229-35. 7. Micali S, Virgili G, Vannozzi E, et al. Feasibility of telementoring between Baltimore (USA) and Rome (Italy): the first five cases. J Endourol 2000;14:493-6. 8. Scott DJ, Tang SJ, Fernandez R, et al. Completely transvaginal NOTES cholecystectomy using magnetically anchored instruments. Surg Endosc 2007;21:2308-16.
Tang et al
Received December 11, 2007. Accepted April 12, 2008. Current affiliations: Division of Digestive and Liver Diseases, Department of Internal Medicine (S.-J.T., S.F.J.), Department of Surgery (C.O.O., D.J.S.), and Department of Urology (J.C.), University of Texas Southwestern Medical Center, Dallas, Texas, USA, Texas Manufacturing Assistance Center, Automation and Robotics Research Institute, University of Texas (R.B., R.F.), Arlington, Texas, USA. Reprint requests: Shou-jiang Tang, MD, Parkland Memorial Hospital, Division of Digestive and Liver Diseases, Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, TX 75390-9151.
APPENDIX LVM allows for the following manipulation (Fig. 5, 2-control panel, and Video 1): 1. Video rotation: arbitrarily able to rotate 360 degrees clockwise or counterclockwise; this can be achieved with the mouse (click on the dial) or the page-up/page-down buttons on the computer keyboard. 2. Horizontal and vertical video inversion. 3. Mirror imaging. 4. Color correction: brightness and contrast for each color are independently adjustable, and values may be reset to the initial setting with a single click. 5. Digital zoom: the magnification power ranges from 2 to 6 and it is shown on the video feed. 6. Pan: after digital zooming, image can be horizontally and vertically repositioned. 7. Drawing tools: drawings and annotations can be superimposed on the video feed live; usually used for edge detection, but can also be used to highlight features. There is a ‘‘return to default’’ button, which resets all controls and restores the image to the natural position (unmanipulated video). There is a separate button for clearing boxes, circles, and such that are drawn on the video. In addition, LVM can capture still images in JPEG format for printing out and presentation. During video manipulation, it takes the image about 50 to 120 milliseconds to go through the program. Computers with dual-core processors perform best with video manipulation: while 1 processor is handling 1 frame, another processor takes the next image frame. If LVM is installed on a stand-alone dedicated hardware, a very simple single processor could handle the entire video manipulation because it would not have the overhead a conventional computer does. The time delay currently present would be nearly eliminated by the use of dedicated hardware. High definition is defined as either 1240 720 or 1920 1080 resolution. The Dazzle has a maximum capture resolution of 720 480, which is referred to as NTSC or standard definition resolution. The video is left in this format (720 480) through the manipulations and then is rescaled to an output of 800 600, which is SVGA. The output can be scaled to 1024 768, referred to as XGA, but there is a slight degradation in speed (ie, slower frame rate) or degradation in the image because it is being stretched from 720 480 to 1024 768. Although technically a resolution of 1024 768 would have enough vertical lines to be classified as high definition, the horizontal lines would have to be stretched to create the 1240 from the 1024. Because of the limitations of the current prototype LVM and video capturing device, some degradation in video quality is expected if a high-definition endoscope and HD monitor are used for the endoscopic examination. In this study, we used a Toshiba Satellite laptop computer, model number P105-S9337 (Toshiba America, New York, NY). It has a dual-core processor, and a built-in television output was used to connect the system to conventional operating room monitors. The computer uses an Intel Core 2 Duo Processor T7200, which is a 64-bit capable, 2.00 gigahertz, 4 megabyte level 2 cache, 667 megahertz front side bus. The level 2 cache memory is processor only memory. The front side bus is the communication speed between components and the processor. The motherboard chipset is Mobile Intel 945PM Express Chipset. It has 2048 megabytes of PC5300 DDR2 RAM. The hard drive is 200 gigabyte. The video card is a PCI-Express 16 NVIDIA GeForce Go 7900 GS with 265 megabytes of GDDR3 memory of its own. This graphics card is capable of video output through S-Video, and is what is used to connect to the monitor in the operating room. The output resolution is set to 800 600 so that the video fills the screen. The operating system is Windows XP Professional 5.1, Service Pack 2.
564 GASTROINTESTINAL ENDOSCOPY Volume 68, No. 3 : 2008
www.giejournal.org
Live video manipulator for endoscopy and NOTES
21. Ichikawa D, Kurioka H, Yamaguchi T, et al. Postoperative complications following gastrectomy for gastric cancer during the last decade. Hepato-gastroenterology 2004;51:613-7. 22. Kodera Y, Sasako M, Yamamoto S, et al. Identification of risk factors for the development of complications following extended and superextended lymphadenectomies for gastric cancer. Br J Surg 2005;92:1103-9. 23. Felsher J, Farres H, Chand B, et al. Mucosal apposition in endoscopic suturing. Gastrointest Endosc 2003;58:867-70. 24. Dennert B, Ramirez FC, Sanowski RA. A prospective evaluation of the endoscopic spectrum of overtube-related esophageal mucosal injury. Gastrointest Endosc 1997;45:134-7.
Received October 18, 2007. Accepted March 25, 2008. Current affiliations: Institute of Digestive Disease (P.W.C., J.Y.L., E.K.N., C.C.L., K.F.T., J.J.S.), Department of Surgery (P.W.C., J.Y.L., E.K.N., C.C.L., S.S.C.), Department of Medicine and Therapeutics (J.J.S.), Department of Anatomical and Cellular Pathology (K.F.T.), Department of Microbiology (M.H.), The Chinese University of Hong Kong, Hong Kong. Reprint requests: Philip W. Y. Chiu, Institute of Digestive Disease, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong.
Live video manipulator for endoscopy and natural orifice transluminal endoscopic surgery (with videos) Shou-jiang Tang, MD, Richard Bergs, MS, Saad F. Jazrawi, MD, Christopher O. Olukoga, MD, Jeffrey Caddedu, MD, Raul Fernandez, PhD, Daniel J. Scott, MD Dallas, Arlington, Texas, USA
Background: During fluoroscopy, radiologists and gastroenterologists are able to manipulate live fluoroscopic video for better orientation and visualization. During endoscopy and natural orifice transluminal endoscopic surgery (NOTES), this function is not currently available. Particularly during NOTES, the endoscopic image is sometimes inverted, and off-axis operation is required. Objective: Our purpose was to develop and test a prototype live video manipulator (LVM) for endoscopy, laparoscopy, and NOTES. Design: Prospective ex vivo and in vivo feasibility study. Interventions: We developed a prototype LVM software for video image manipulation that can be easily installed on any computer. The video input is streamed into the computer and can be displayed on a standard monitor. LVM was tested ex vivo in the following functions: (1) instant live video rotation, (2) vertical or horizontal video inversion, (3) mirror imaging, and (4) digital zooming. These functions were also tested during upper and lower GI endoscopy, ERCP, diagnostic laparoscopy, and various transvaginal NOTES procedures (cholecystectomy, gastroenterostomy, and sleeve gastrectomy) in porcine models. Main Outcome Measurements: Image quality observation between unmanipulated and manipulated live videos. Results: LVM reliably and easily performed live video manipulation during these tests. Besides standard definition video signals, LVM is fully compatible with high-definition video endoscopy. Three observers reported that the subjective image quality was the same in specified areas between manipulated and unmanipulated live videos. Limitations: Observation and feasibility study. Conclusions: LVM reliably and conveniently performed live video manipulations. LVM requires minimal equipment, capital investment, and maintenance, and is easy to set up. LVM can be a useful tool in many medical imaging studies, including endoscopy, laparoscopy, and NOTES, either as a built-in technology or as an as-needed add-on feature.
Copyright ª 2008 by the American Society for Gastrointestinal Endoscopy 0016-5107/$32.00 doi:10.1016/j.gie.2008.04.018
Image processing techniques, broadly defined as the manipulation of signals such as photographs and video sequences, are ubiquitous in medicine, astronomy, video communication, and electronic games.1 Video-assisted endoluminal and intracavitary procedures, robotic-assisted
www.giejournal.org
Volume 68, No. 3 : 2008 GASTROINTESTINAL ENDOSCOPY 559
Abbreviations: LVM, live video manipulator; NOTES, natural orifice transluminal endoscopic surgery.
Live video manipulator for endoscopy and NOTES
surgery, and laparoscopy are well established in medicine2-5 and rely on live video presentation and manipulation.6,7 During these video-assisted techniques, the ideal vision system should provide surgeons with the same quality of visual information as with open surgery, namely, high resolution, excellent color quality, precise spatial information, and a constant clear view for optimal surgical intervention.6 Unfortunately, many important problems in surgical image processing remain unsolved. For example, during natural orifice transluminal endoscopic surgery (NOTES), the endoscopic image is occasionally inverted, and off-axis manipulation is required.2 When endotherapy is performed in patients with certain altered anatomic features, the anatomic orientation is unconventional (ie, ampulla appears inverted in Billroth II anatomy).3 To obviate these challenges, we developed and tested a prototype live video manipulator (LVM) for endoscopy, NOTES, and any video-based procedure and surgery. The concept of digitally manipulating video is protected under a filing with the University of Texas Southwestern Medical Center. To our knowledge, neither video manipulation nor a video manipulator for endoscopy has been reported in the medical literature.
METHODS LVM One of the investigators (R. B.) developed a prototype LVM software (Appendix) that can be easily installed on any computer. Any video source, including S video, composite, RBG, and so forth, can be streamed through a video capture device (Dazzle DVD Recorder, Pinnacle Systems, Mountain View, Calif) and into a computer installed with LVM (Fig. 1). The Dazzle DVD recorder is a plug-andplay device and its suggested retail price is US$49.99. The manipulated video signal is directly exported to a monitor for viewing. The manipulated video signal can be recorded as well. Special training is not required to be proficient in the use of this software. The on-screen control panel of the software is self-explanatory. In addition, LVM can capture still images in JPEG format for printout and presentation.
Ex vivo image and video quality tests An Olympus colonoscope (H180 AL/I, Olympus America, Center Valley, Pa) was used to test both image and video quality in terms of resolution, contrast, color reproduction, and time sequence differences between the source and manipulated videos. Two physicians and one engineer participated in the following comparison tests. A Kodak digital science imaging test chart TL-5003 (Eastman Kodak, Rochester, NY) was used for resolution test comparison. Mini-ColorChecker (Munsell Color, X-rite, Grand Rapids, Mich) and an investigator’s fingertip were used for image color reproduction tests. The video signal was split into 2 560 GASTROINTESTINAL ENDOSCOPY Volume 68, No. 3 : 2008
Tang et al
Capsule Summary What is already known on this topic d
During natural orifice transluminal endoscopic surgery (NOTES), the endoscopic image may be inverted, off-axis manipulation is required, and the anatomic orientation may be unfavorable in some patients with certain altered anatomy.
What this study adds to our knowledge d
In a prospective feasibility study of a prototype live video manipulator, 3 observers reported that the subjective image quality was the same in specific areas when comparing live video images with and without manipulation.
cables from an Olympus CV-180 video processor through the composite video output. One cable was connected to a high-definition flat-panel monitor (Olympus OEV-191H, 1080 lines resolution) and printer. The other was connected to a laptop computer installed with LVM by a video capture device (Dazzle DVD recorder). The manipulated video signal was then connected to a second high-definition monitor and printer. The 2 monitors were placed side by side for comparing video quality and time sequence difference. This 2-monitor comparison test was recorded with a digital camcorder (DCR-SR300, Sony Corporation, Tokyo, Japan).
In vivo feasibility testing during endoscopy and NOTES Besides the original video display, video manipulation was performed and displayed on a second monitor during upper and lower endoscopy (n Z 3 each), ERCP (n Z 3), diagnostic laparoscopy (n Z 3), and various transvaginal NOTES procedures (n Z 6, including cholecystectomy, gastroenterostomy, and sleeve gastrectomy) in porcine models. The instituional animal care and utilization committee approved these laparoscopy and NOTES procedures, and our institutional review board approved clinical endoscopy. Olympus CV-160 video processors and non-high definition endoscopes were used for clinical endoscopy and NOTES because of equipment limitations. We compared video quality in terms of resolution, contrast, color reproduction, and time sequence differences between the source and manipulated videos.
RESULTS Throughout the study period, there was no LVM software anomaly or failure. www.giejournal.org
Tang et al
Live video manipulator for endoscopy and NOTES
Figure 1. Schematic illustration of live video manipulator set up during endoscopy and NOTES.
laydwhich was less than 120 millisecondsdwas not noticeable during the 1-monitor test.
In vivo feasibility testing during endoscopy and NOTES
Even with high-definition video signals, all 3 observers reported that the subjective image quality (resolution and color reproduction) was the same between manipulated and unmanipulated live videos (Video 1; available online at www.giejournal.org). The manipulated video stream was smooth and without visible flicker or distortion. During the 2-monitor comparison test, a minimal time delay was present in the manipulated video attributable to computer processing overhead. However, this de-
During upper and lower endoscopy, ERCP, and transvaginal NOTES, the 3 observers felt that the subjective image quality was the same between manipulated and unmanipulated live videos (Figs. 2 through 4, Video 2; available online at www.giejournal.org). The millisecondorder time delay of the manipulated video visible during the 2-monitor test was not noticeable on the LVM monitor alone, and it did not affect or impair endoscopic or NOTES procedures in any regard. During routine diagnostic upper and lower endoscopy and uncomplicated laparoscopy, the manipulated video did not offer extra benefits. During laparoscopy, the laparoscope had to be held without rotation to keep the abdominal wall on the top in the view (horizon-corrected images). Otherwise, the laparoscopic view of the abdominal cavity is rotated. With LVM, the horizon view was easily ‘‘corrected’’ without rotation of the laparoscope. Applying 180-degree video rotation during ERCP in patients with normal gastric anatomy, the biliary ductal orientation and cannulation direction became that of patients with Roux-en-Y or Billroth II anatomy (Fig. 2, Video 2). During NOTES, with video rotation between 90 and 180 degrees, optimal spatial visualization of the abdominal cavity and organs could be easily maintained and endoscopic vision was comparable to that of open or laparoscopic surgery (Figs. 3 and 4, Video 2). The endoscopists and surgeons
www.giejournal.org
Volume 68, No. 3 : 2008 GASTROINTESTINAL ENDOSCOPY 561
Figure 2. Endoscopic image of the ampulla during ERCP. This patient has normal gastric anatomy, with video rotation of 180 degrees the biliary ductal orientation and cannulation direction is similar to that in patients of Roux-en-Y or Billroth II anatomy.
Ex vivo image and video quality tests
Live video manipulator for endoscopy and NOTES
Tang et al
Figure 3. Endoscopic images of abdominal cavity during NOTES. With video rotation between 90-180 degrees, optimal endoscopic visualization of the abdominal cavity and organs is easily maintained and endoscopic vision is comparable to that during open or laparoscopic surgery.
be easily identified. Without video manipulation, the surgeons felt that they would not have been able to complete the dissection.
DISCUSSION
preferred and relied on the manipulated video display in certain situations, such as with retroflexed views, off-axis visualization, multiple instruments manipulating on the tissue from different angles, and so forth. We found that digital zooming was useful under some circumstances. For example, 1 case of gallbladder dissection during transvaginal cholecystectomy was not possible without digital enhancement of our magnetically guided camera and cauterizer.8 Although there was some loss of mucosal or serosal details from image pixilation with digital zooming, organ and tissue structures could still
During fluoroscopy, the radiologists and endoscopists are able to manipulate live fluoroscopic video by means such as horizontal and vertical image inversion, image rotation, and magnificationdall of which enhance orientation and visualization. During endoscopy and NOTES, this function is not currently available. In the current study, we developed and demonstrated the feasibility of a prototype LVM in clinical endoscopy and NOTES. Besides standard-definition video signals, LVM is fully compatible with high-definition video endoscopy. With highdefinition video signals, observers felt that the subjective image quality was the same in the specified areas between manipulated and unmanipulated live videos. The video feed and display was smooth during all tests. Although a slight (50- to 120-millisecond) time delay was noticed at the manipulated video during a 2-monitor test, this difference was not noticeable during the 1-monitor ex vivo and in vivo tests. Most important, the delay did not affect or impair endoscopic and NOTES procedures in any regard. Although video inversion can potentially impair the inexperienced operator’s ability to navigate, we have not encountered any interference during our initial experience with multiple LVM procedures. For example, when ERCP is performed in a patient either in the supine or prone position, the fluoroscopic images are inverted. However, once the images are corrected horizontally, the visualization and the ERCP procedure themselves remain the same. Other possible drawbacks of LVM include extra
562 GASTROINTESTINAL ENDOSCOPY Volume 68, No. 3 : 2008
www.giejournal.org
Figure 4. Endoscopic images of abdominal cavity during NOTES. With video rotation between 90-180 degrees, optimal endoscopic visualization of the abdominal cavity and organs is easily maintained and endoscopic vision is comparable to that during open or laparoscopic surgery.
Tang et al
Live video manipulator for endoscopy and NOTES
Figure 5. Image of on-screen representation of the software control panel.
equipment, extra wiring, the need for an assistant for video manipulation (applicable to the current LVM prototype computer setup), potential software anomalies or failures (which we have not encountered), and minimal image quality degradation from splitting of video signals from the endoscopy processor card. Reassuringly, we have not noticed a significant difference in terms of image quality degradation. We have noticed that there is a change of the accessory channel ‘‘location’’ on the monitor after video inversion. For example, the accessory view will not come out of a diagnostic gastroscope at 9 o’clock on the monitor after video inversion. However, during NOTES, we have not found that this phenomenon interfered with our procedures. LVM use in other endoscopic procedures is questionable and may not be necessary. We did not find that the manipulated video offered extra benefits during routine diagnostic upper and lower endoscopy. Endoscopists are accustomed to off-axis visualization during most of the diagnostic procedures after years of training and practice. Because our current prototype LVM was installed on a standard laptop computer and there were concerns about biologic fluid contamination, an assistant operated LVM according to the endoscopists’ and surgeons’ verbal instructions. However, once it is packaged as a stand-alone device with ergonomic controls, the endoscopists and surgeons would be able to manipulate video independently.
We also envision a future LVM system with voice-recognition capability to further enhance its effectiveness with endoscopists and surgeons. In conclusion, LVM reliably and conveniently performed live video manipulations with both standard and high-definition video signals. LVM can be a useful tool in many medical imaging studies including endoscopy, laparoscopy and NOTESdeither as a built-in technology or as an as-needed add-on feature.
www.giejournal.org
Volume 68, No. 3 : 2008 GASTROINTESTINAL ENDOSCOPY 563
DISCLOSURE The authors report that there are no disclosures relevant to this publication.
REFERENCES 1. Huang TS, Aizawa K. Image processing: some challenging problems. Proc Natl Acad Sci USA 1993;90:9766-9. 2. Rattner DW, Kalloo A. The ASGE/SAGES working group on Natural Orifice Translumenal Endoscopic Surgery White Paper. Gastrointest Endosc 2006;63:199-203. 3. Feitoza AB, Baron TH. Endoscopy and ERCP in the setting of previous upper GI tract surgery, I: reconstruction without alteration of pancreaticobiliary anatomy. Gastrointest Endosc 2001;54:743-9.
Live video manipulator for endoscopy and NOTES 4. Downey RJ. Complications after video-assisted thoracic surgery [review]. Chest Surg Clin North Am 1998;8:907-17. 5. Patil PV, Hanna GB, Cuschieri A. Effect of the angle between the optical axis of the endoscope and the instruments’ plane on monitor image and surgical performance. Surg Endosc 2004;18:111-4. 6. Buess GF, Schurr MO, Fischer SC. Robotics and allied technologies in endoscopic surgery. Arch Surg 2000;135:229-35. 7. Micali S, Virgili G, Vannozzi E, et al. Feasibility of telementoring between Baltimore (USA) and Rome (Italy): the first five cases. J Endourol 2000;14:493-6. 8. Scott DJ, Tang SJ, Fernandez R, et al. Completely transvaginal NOTES cholecystectomy using magnetically anchored instruments. Surg Endosc 2007;21:2308-16.
Tang et al
Received December 11, 2007. Accepted April 12, 2008. Current affiliations: Division of Digestive and Liver Diseases, Department of Internal Medicine (S.-J.T., S.F.J.), Department of Surgery (C.O.O., D.J.S.), and Department of Urology (J.C.), University of Texas Southwestern Medical Center, Dallas, Texas, USA, Texas Manufacturing Assistance Center, Automation and Robotics Research Institute, University of Texas (R.B., R.F.), Arlington, Texas, USA. Reprint requests: Shou-jiang Tang, MD, Parkland Memorial Hospital, Division of Digestive and Liver Diseases, Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, TX 75390-9151.
APPENDIX LVM allows for the following manipulation (Fig. 5, 2-control panel, and Video 1): 1. Video rotation: arbitrarily able to rotate 360 degrees clockwise or counterclockwise; this can be achieved with the mouse (click on the dial) or the page-up/page-down buttons on the computer keyboard. 2. Horizontal and vertical video inversion. 3. Mirror imaging. 4. Color correction: brightness and contrast for each color are independently adjustable, and values may be reset to the initial setting with a single click. 5. Digital zoom: the magnification power ranges from 2 to 6 and it is shown on the video feed. 6. Pan: after digital zooming, image can be horizontally and vertically repositioned. 7. Drawing tools: drawings and annotations can be superimposed on the video feed live; usually used for edge detection, but can also be used to highlight features. There is a ‘‘return to default’’ button, which resets all controls and restores the image to the natural position (unmanipulated video). There is a separate button for clearing boxes, circles, and such that are drawn on the video. In addition, LVM can capture still images in JPEG format for printing out and presentation. During video manipulation, it takes the image about 50 to 120 milliseconds to go through the program. Computers with dual-core processors perform best with video manipulation: while 1 processor is handling 1 frame, another processor takes the next image frame. If LVM is installed on a stand-alone dedicated hardware, a very simple single processor could handle the entire video manipulation because it would not have the overhead a conventional computer does. The time delay currently present would be nearly eliminated by the use of dedicated hardware. High definition is defined as either 1240 720 or 1920 1080 resolution. The Dazzle has a maximum capture resolution of 720 480, which is referred to as NTSC or standard definition resolution. The video is left in this format (720 480) through the manipulations and then is rescaled to an output of 800 600, which is SVGA. The output can be scaled to 1024 768, referred to as XGA, but there is a slight degradation in speed (ie, slower frame rate) or degradation in the image because it is being stretched from 720 480 to 1024 768. Although technically a resolution of 1024 768 would have enough vertical lines to be classified as high definition, the horizontal lines would have to be stretched to create the 1240 from the 1024. Because of the limitations of the current prototype LVM and video capturing device, some degradation in video quality is expected if a high-definition endoscope and HD monitor are used for the endoscopic examination. In this study, we used a Toshiba Satellite laptop computer, model number P105-S9337 (Toshiba America, New York, NY). It has a dual-core processor, and a built-in television output was used to connect the system to conventional operating room monitors. The computer uses an Intel Core 2 Duo Processor T7200, which is a 64-bit capable, 2.00 gigahertz, 4 megabyte level 2 cache, 667 megahertz front side bus. The level 2 cache memory is processor only memory. The front side bus is the communication speed between components and the processor. The motherboard chipset is Mobile Intel 945PM Express Chipset. It has 2048 megabytes of PC5300 DDR2 RAM. The hard drive is 200 gigabyte. The video card is a PCI-Express 16 NVIDIA GeForce Go 7900 GS with 265 megabytes of GDDR3 memory of its own. This graphics card is capable of video output through S-Video, and is what is used to connect to the monitor in the operating room. The output resolution is set to 800 600 so that the video fills the screen. The operating system is Windows XP Professional 5.1, Service Pack 2.
564 GASTROINTESTINAL ENDOSCOPY Volume 68, No. 3 : 2008
www.giejournal.org