- Email: [email protected]
The Telemedicine benchmark—a general tool to measure and compare the performance of video conferencing equipment in the telemedicine area
Computer Methods and Programs in Biomedicine 60 (1999) 133 – 141 www.elsevier.com/locate/cmpb
The Telemedicine benchmark—a general tool to measure and compare the performance of video conferencing equipment in the telemedicine area Peter J. Klutke a,*, Paolo Mattioli b,1, Fabio Baruffaldi b,2, Aldo Toni b,3, Karl-H. Englmeier a,4 a
National Research Center for En6ironment and Health, Institute of Medical Informatics and Health Ser6ices Research, Ingolsta¨dter Landstraße 1, D-85764 Neuherberg, Germany b Instituti Ortopedici Rizzoli, Laboratorio di Tecnologia Medica, Via di barbiano 1 /10, I-40136 Bologna, Italy Received 11 November 1998; received in revised form 25 February 1999; accepted 25 February 1999
Abstract In this paper, we describe the ‘Telemedicine Benchmark’ (TMB), which is a set of standard procedures, protocols and measurements to test reliability and levels of performance of data exchange in a telemedicine session. We have put special emphasis on medical imaging, i.e. digital image transfer, joint viewing and editing and 3D manipulation. With the TMB, we can compare the aptitude of different video conferencing software systems for telemedicine issues and the effect of different network technologies (ISDN, xDSL, ATM, Ethernet). The evaluation criteria used are length of delays and functionality. For the application of the TMB, a data set containing radiological images and medical reports was set up. Considering the Benchmark protocol, this data set has to be exchanged between the partners of the session. The Benchmark covers file transfer, whiteboard usage, application sharing and volume data analysis and compression. The TMB has proven to be a useful tool in several evaluation issues. © 1999 Elsevier Science Ireland Ltd. All rights reserved. Keywords: Telemedicine; Performance; Evaluation; Video conferencing; ISDN; Medical imaging
1. Introduction, aims and scope * Corresponding author. Tel.: + 49-89-31874457; fax: +4989-31874243. E-mail addresses: [email protected] (P.J. Klutke), [email protected] (P. Mattioli), [email protected] (F. Baruffaldi), [email protected] (A. Toni), [email protected] (K.H. Englmeier) 1 Tel.: + 39-51-6366864; fax: + 39-51-6366863 2 Tel.: + 39-51-6366864; fax: + 39-51-6366863 3 Tel.: + 39-51-6366864; fax: + 39-51-6366863 4 Tel.: + 49-89-31874457; fax: + 49-89-31874243
In times of scarce resources, especially in the health care sector, it is crucial to evaluate the cost-effectiveness of solutions in telemedicine [1,2]. Video conferencing equipment costs have decreased to one third in the last 3 years. Thus, they have become affordable to the general practitioner, enabling him to invite the second opinion
0169-2607/99/$ - see front matter © 1999 Elsevier Science Ireland Ltd. All rights reserved. PII: S 0 1 6 9 - 2 6 0 7 ( 9 9 ) 0 0 0 2 7 - 9
134
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
of an expert. For linkages among hospitals, these new low-priced systems are used with buoyancy also. On the other hand, only very little in-depth research on well-defined environments has been carried out in this field. A thorough evaluation of these new techniques for remote communication in medicine is still in is infancy [3,4]. What is more, the software of the video conferencing systems has been improved significantly during the last 2 years. Application sharing, the possibility of joint working on the same application, is meanwhile available and may be used for computer supported collaborative work (CSCW) on medical files, statistics or medical images by sharing the related program. Our Telemedicine Benchmark (TMB) is a set of standard procedures, protocols and measurements to test reliability and levels of performance of data exchange in a telemedicine session. The presentation of the TMB is the scope of this paper. The evaluation criteria used are delay times and software functionality. In the protocol, both synchronous and asynchronous communication is tested. Medical use of asynchronous communication (store and forward procedures) is for example the transfer of medical images; the field of synchronous communication (real time joint working) includes the subsequent collaboration and discussion on the medical images. We have chosen examples of data exchange in radiology and orthopedics because of the high communication requirements of medical image transfer. Using image file transfer without loss of information, in contrary to the recording of images by H.320 compliant cameras avoids to deal with assessment of lossy compressed medical images, which has already been evaluated [5,6]. Furthermore, the protocol was designed with respect to the ACR Standard for Teleradiology [7]. The TMB was developed during the project PH-Net, which was sponsored by the European Commission, Contract No. 45491, within the framework of TEN-ISDN (Trans European Networks using ISDN). The main aims of PH-Net are: The improvement of knowledge regarding the health care area reality, to assist strategic decisions, planning and management.
The improvement of the quality of care by means of information sharing among professionals and specialists involved in the care process. The improvement of cost-effectiveness of careprocesses using standard teleconferencing equipment over ISDN lines to implement teleconsulting, teleradiology and telepathology. During the project PH-Net, the methods of communication and data exchange to be performed in the TMB have proven to be a realistic portrayal of clinical routine work. 2. General conditions to apply the telemedicine benchmark
2.1. Explanations and definitions Application sharing: A communication tool, that allows to share an application on one computer to the computer of a connected partner. This partner can make inputs to and see outputs of the shared application as if it ran on his computer. The application sharing tool is for example implemented in Microsoft NetMeeting™. LP: The partner who rings up the other partner (LP, local partner). Regarding application sharing, the LP (in this context called the ‘Host’) makes available one of his applications to RP. RP: The partner that is rung up by the other partner (RP, remote partner). Regarding application sharing, the RP (in this context called the ‘Guest’) is made available one of the applications of LP.
2.2. Requirements The TMB can be applied to every telemedicine application that disposes of the following features: Audio and video connection for remote sessions Data exchange by file transfer and/or whiteboard and/or application sharing It is applicable to different operating systems (Windows 3.1x, Windows 95, Windows NT, MacOS, UNIX). The following software, for the most part available through the Internet, is required:
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
IDL (Interactive Data Language®), a data visualization software from Research Systems, Boulder, Colorado, USA OSIRIS, a viewer for medical image data in DICOM format [8] Text editor (e.g. Microsoft® Wordpad) Zip compression software (e.g. WinZip®)
2.3. Preliminary set-up of the computer The computer has to be prepared carefully to guarantee equal circumstances for every TMB session. All settings that have proven to have an effect on the performance or the functionality of the whole system have to be brought into line. This includes the following aspects: The necessary software to be shared has to be installed. The screen settings as well as the number of colors have to be equalized. The screen savers have to be deactivated. The settings of the conferencing software have to be harmonized and adjusted to automatic acceptance of incoming calls and automatic storage of incoming files in an empty receipt directory.
2.4. Method of time delay measurement The time measurements are carried out by a chronometer. The sensitivity of the time measurements is 91 s. Two different arrangements are possible to adopt the TMB. If there are two telemedicine systems in one room for a local test, one has to refer to chapter 2.4.4: ‘Local sessions’; if there is a session with the two systems spatially separated, chapter 2.4.2: ‘Remote sessions’ is used. Chapter 2.4.3 ‘Guest only with application sharing (local and remote sessions)’ is relevant for both arrangements.
2.4.1. Local sessions To measure the time delay of an action ACT in a local session, LP and RP are the same person, that has to use the following steps: 1. Identify the end of ACT. It is marked in the TMB protocol by an asterisk (*). An example is the pushing of a button to load an image.
135
2. Identify the effect of the end of ACT. It is marked in the TMB protocol by a number sign (c ). The effect could be for example the appearance of the image. 3. Perform ACT on the LP side. Start your chronometer at the end of ACT. 4. Watch the RP screen. Stop your chronometer if the effect of the end of ACT appears on the RP screen AND the concerning window is completely restored (if it had been disturbed or had to be built up). 5. Write down the result of the time measurement in the TMB protocol immediately.
2.4.2. Remote sessions Measuring time delays in a remote session is a little more complicated than in a local session. The start of an action ACT by one partner has to be indicated to the other partner by audio. This is done by the countdown phrase ‘3-2-1-GO’. The other partner has to confirm that the effect of the end of ACT appeared on his screen. This is done by the conventional phrase ‘STOP’. For most actions (ACT) performed, two delays are measured, one by the local partner and one by the remote partner. They are obtained by applying the following steps. 1. LP and RP identify the end of ACT. It is marked in the TMB protocol by an asterisk (*). 2. LP and RP identify the effect of the end of ACT. It is marked in the TMB protocol by a number sign (c). 3. LP performs ACT on the his site. LP counts down ‘3-2-1-GO’ with ‘GO’ at the end of ACT and simultaneously starts his chronometer (at the end of ACT). 4. RP starts his chronometer when he receives the ‘GO’ of LP by audio. 5. RP watches his screen. He stops his chronometer if the effect of the end of ACT appears on his screen AND the concerning window is completely restored (if it had been disturbed or had to be built up). Simultaneously, he says ‘STOP’ to LP. 6. LP stops his chronometer when he receives ‘STOP’ of RP.
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
136
During the protocol always LP is the host and RP is the guest. RP starts the action ACT on his computer and measures the delay until the result of ACT appears on his screen. No audio delays have to be considered if this procedure is performed in remote sessions. The TMB protocol indicates when to use this measurement method.
2.5. Unpredictable e6ents, a6ailability of actions and general remarks
Fig. 1. Time delay measurement in remote sessions.
7. Both partners exchange their measured values and write them down in the TMB protocol immediately. Fig. 1 shows how the two time measurement results are obtained and why the time measurement by LP is longer twice the VC system audio delay than the time measurement by RP. Consequently, the best approximation of the action duration is obtained by the average of the two measured values.
2.4.3. Guest only with application sharing (local and remote sessions) The difference to the other two procedures is that only the guest of an application sharing session performs a measurement. As defined, the shared application runs on the host’s computer.
During the performance of the protocol, all unpredictable events are recorded together with the related action. Among them include interruption of the connection, the crash of the conferencing software, the crash of an application shared, sudden long time delays and a reboot of the system. This allows to identify weak points in the video conferencing software as well as a general statement regarding the stability and toughness of the whole system. The identification of the availability of every action of the TMB protocol helps to assess the overall capabilities of the conferencing software.
3. Description of the telemedicine benchmark
3.1. The telemedicine benchmark data set The TMB data set (Table 1) contains all the files needed to perform the TMB. These files are
Table 1 The telemedicine benchmark data set File name
File description
Size (bytes)
RX-post.tif TC-pre.dc Pictures.tif Report.txt RX-post.zip Pictures.zip Report.zip Bench.zip mr000xx.dcm
X-ray of a hip arthroplasty, scanned, 2000×1000 pixel, 8 Bit CT image for research purposes, DICOM 3.0 format, coronal cut, 512×512 pixel, 12 Bit Colored TIFF image with constant gray background showing post surgery exercises Standard text file containing a short medical report Zipped RX-post.tif (by WinZip 6.x) Zipped Pictures.tif (by WinZip 6.x) Zipped Report.txt (by WinZip 6.x) Zipped Benchmark files (by WinZip 6.x) 20 MR images, colon with contrast medium, DICOM 3.0 format, coronal cut, 132522 Bytes each, 256×256 Pixel, 12 Bit Zipped mr00040.dcm (by WinZip 6.x) Zipped mr00001.dcm to mr00020.dcm (by WinZip 6.x)
2 009 042 525 676 256 798 847 622 505 42 771 617 906 276 2 650 440
mr00040.zip mr01-20.zip
79 711 1 512 390
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
for example transferred by file transfer or are shared among the partners by application sharing of the related application.
3.2. The telemedicine benchmark protocol The TMB protocol describes in detail the procedure to apply the TMB. It consists of the main paragraphs file transfer, whiteboard usage, application sharing, volume data analysis and compression. Each paragraph may be applied independently from the others, as required by the user. Of course the TMB protocol requires the important hard- and software settings to be taken down before the start of the session.
3.2.1. File transfer File transfer will be of frequent use in clinical routine, transferring medical files, reports or images. It regards asynchronous as well as synchronous communication. With the automatic acceptance of incoming calls and the automatic storage of incoming files in a pre-selected directory the files may be transferred without any user interaction on the site of RP. Of course file transfer also makes sense during a conference. All files of the TMB data set are transferred and the necessary time is measured as indicated in the TMB protocol. One can derive the effectiveness of internal compression techniques as well as the utilization of the available bandwidth during file transfer. 3.2.2. Whiteboard usage Both sites enable the whiteboard (sometimes also called ‘notepad’) of their conferencing software. This is a work environment visible on the screens of all connected sites. Each partner can add or modify the contents of the whiteboard. The changes are immediately transferred to all connected sites. If data in the whiteboard is changed, this has usually no effect on the appropriate original file. This work environment is suitable for simple joint viewing and annotating of medical documents. Clinical routine usage may be the comparison of medical files or discussion on statistics.
137
During the TMB protocol, an X-ray image is visualized in the whiteboard. Then a region of interest and a line are drawn onto the image, telepointing is performed and the image is scrolled. Finally, a colored illustration and a snapshot of an application window are imported in the whiteboard. Time delays are measured as indicated in the TMB protocol. The results of the time delay measurements allow to assess the overall performance of the whiteboard as well as the identification of a suitable operational area of this tool given the tested hardware and software configuration.
3.2.3. Application sharing Application sharing provides more features than the common work environment ‘whiteboard’. It allows joint working with any windows application. This application is only running on the computer of one partner, called the host. The host has to share his application with the other partner, the guest. The guest has an identical user interface on his screen, showing the same as on the host’s screen and allowing interaction just as if the application was running on the guest’s computer [9]. Of course, the guest’s user interaction has to be sent to the host’s computer and the resulting changes on the screen have to be sent back to the guest’s computer. The time delay caused by this information transfer for selected actions is investigated in this section. For medical issues, the most time-critical issue is the collaboration on medical images. To test this aspect, the DICOM viewer OSIRIS for medical images [8] that is common available through the Internet is used for application sharing. A CT image is displayed with this viewer, annotations are made and the gray-value windowing is changed, which may be in particular important for remote education and training. Almost the same is done for an X-ray image. Finally, a word processing system is used to simulate dealing with patient data and medical reports. Text visualization as well as copy and paste operations are tested. The results of the time delay measurements do not only allow to assess the overall performance of application sharing and the intelligent use of
138
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
the bandwidth available. They also evaluate the use of internal compression techniques, supporting the decision whether the usage of the whiteboard or of application sharing is more efficient in a certain situation.
3.2.4. Volume data analysis By the volume data analysis section of the TMB, it was intended to get an idea of the possibilities for remote 3D image analysis. This task demands a lot as it deals with a huge amount of data. Twenty CT images with 512× 512 pixels each result in 10 Mb of data. Transferring these files in uncompressed form results in a great time delay. However, for a 3D analysis of medical image data, interactive working on the data set should be possible. This means maximum response times of 5 s. The 3D visualization can be achieved by application sharing. The shared program is IDL (Interactive Data Language) v5.0, which provides 3D visualization features. Using application sharing for 3D image analysis provides the following advantage. The volume data is stored only on one partner’s side and the CPU of his computer has to perform the 3D operations. All results of 3D operations are viewed and examined on the screen, that means they are 2D information. Only this information has to be transferred by the application sharing tool. The reduction of the amount of information to be transferred from three dimensions (volume data) to two dimensions (screen data) makes application sharing feasible for 3D remote image analysis. For the TMB, the IDL demo with its appropriate data set was used. The following actions were tested: Application sharing of the IDL slicer, an interactive 3D visualization tool. Visualization of a yellow rectangle to visualize a cutting plane through CT volume data. Visualization of the cut-out slice. Deletion of the 3D image. Display of a shaded surface representing a threshold in the CT volume data. Change of the z-rotation of a small wire netting cube representing the volume data.
Display of the rotated surface according to the chosen change of the z-rotation. Rotating the wire netting of a human heart once by a mouse movement. Releasing the mouse button to make the shaded 3D surface appear. It is possible to convert DICOM files of a medical slice series (CT, MR) into raw data that can easily be visualized with the IDL slicer tool.
3.2.5. Compression During the TMB, we have obtained transfer times for files in compressed and uncompressed format. Together with time measurements for compression and decompression of these files we can answer the question if it makes sense to compress files before transferring them as regards the time needed for the whole action. For the calculation we use the following variables: fu fc h g
th tu tc tu tc
uncompressed file size (Mb) compressed file size (Mb) compression factor (= fc/fu) specific time for compression and decompression referring to uncompressed. data (s/ Mb) constant user overhead for the handling of the compression software (s) specific transfer time for uncompressed data (s/Mb) specific transfer time for compressed data (s/Mb) whole time needed for file transfer without compression (s) whole time needed for file transfer with compression (s)
The time for transferring the file is calculated as follows: whole time needed= time for compression + transfer time tu = 0+ tu × fu (without compression)
(1)
tc = th + g× fu + tc × fc (with compression)
(2)
with
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
139
If Eq. (7) is not true, no positive break-even file size f\0 exists and it never makes sense to compress files of this type before transferring them.
4. Results, discussion and outlook
Fig. 2. The definition of the ‘break-even file size’.
fc = fu × h
(3)
one gets: tu = fu ×tu
(4)
tc = th + fu ×(g+tc ×h)
(5)
It only makes sense to compress a file before transferring it if tu( fu)\tc( fu)
(6)
For th \ 0 this can only happen for large fu if tu \ (g+ tc × h)
(7)
Eq. (7) means that the gradient of tu( fu) must be larger than the gradient of tc( fu). Fig. 2 shows the dependencies between the variables. If Eq. (7) is true and th \0, the two straight lines in Fig. 2 intersect. A ‘break-even file size’ f may be defined as follows. If a file size is larger than the break-even file size, it saves time to compress the file before transferring it. If the size of a file is exactly the break-even file size, we get tu(f)/tc(f)
(8)
Therefore, one gets the break-even file size f from Eq. (8) as f = th = (tu −tc × h− g)
(9)
The TMB covers the exchange of 2- and 3-D data and its joint analysis, because these fields make the greatest demands on channel capacity usage. Further developments concerning one dimensional data analysis (e.g. electrocardiography data) and dynamic digital imaging (e.g. images coming from an optical microscope) are going to be implemented. The exchange of patient records, included in the TMB as text file exchange, may also be intensified. This field, however, demands a lot more of the clinical information system’s interoperability than of the overall performance and is therefore not quite within the scope of the TMB. To test specific features like dynamic bandwidth allocation or audio and/or video protocol changes an advanced section is being developed. It has not been integrated in the basic TMB because the knowledge required to apply these measurements is considerably higher. No special technical or medical expertise, however, should be required to perform the basic TMB. Moreover, the TMB is an excellent tool to introduce inexperienced users into the current possibilities of telemedicine. Finally, the TMB is scaleable, i.e. you can only use parts of it or even just one action. The user of the TMB has to decide which parts are relevant for him. As the TMB uses commonly available software (Wordpad™ text editor, OSIRIS DICOM viewer, IDL™ data visualization software, Zip compression tools) to be shared among partners during the TMB or for compression topics, the TMB is not dependent on a certain operating system. Except for the Microsoft™ Wordpad™ all other necessary software is available through the Internet, first of all free of charge. The Microsoft™ Wordpad™ is included in Windows 95/NT™. Other operating systems offer similar tools. The TMB allows to test and compare different telemedicine systems with different settings and
140
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
configurations thus supporting the user to choose the most cost-efficient solution. These different systems may be characterized by different network technologies (e.g. ISDN Point to Point, TCP/IP Routing by ISDN with and without encryption, xDSL, fiberglass, ATM) [10], different video conferencing systems (e.g. Intel ProShare 200™, Intel™ Business Video Conferencing, PictureTel Live 200™, Aethra SDV8000™, Aethra Eycona V™), different software configurations (e.g. preferences of video conferencing system, screen settings), different operating systems (e.g. Windows 95™, Windows 98™, Windows NT 4.0™, MacOS™, UNIX™) and different hardware configurations (e.g. main processor, RAM, hard disk capacity, graphics card). With the TMB, future buyers of VC systems can evaluate different equipment before buying. For those who have already bought equipment, the results of the TMB measurements help to decide upon the best use of different data communication components like ‘Whiteboard’ or ‘Application Sharing’ to take care of a given task like a medical image transfer. The TMB and the related results are of use for the manufacturers of video conferencing or telemedicine equipment also. They get a clear catalogue of demands for their systems as well as comparable evaluation results, which specify the capacity of their systems, help to identify weak points and possible improvements. Scientists can develop new algorithms optimized for specific tasks contained in the TMB. A detailed description of measurement results applying the TMB to different systems is not within the scope of this paper. Another publication presents the results of the TMB’s application to the four desktop video conferencing systems Intel ProShare 200™, Intel™ Business Video Conferencing, PictureTel™ Live 200 and Aethra™ SDV8000 [11]. The results show that even low-cost desktop video conferencing solutions are suitable to be applied in the telemedicine area (e.g. Orthopedics teleconsulting). Here the
TMB has proven to be an excellent tool for evaluation and comparison. The evaluation of the TMB by measuring time delays or response times happens on the application layer of the ISO-OSI network reference model. The causes of the time delays are not only due to the application used, but also due to effects related to lower layers. The preliminary uniform set-up of the systems and the recording of hardand software settings intend to harmonize and/or identify these effects. It is of great importance to include them into the final discussion of the TMB measurement results. We developed a web page (http:// www.tecno.ior.it/telemed/projects/TMBench/index.html) where the TMB data set and TMB protocol will be available free of charge after filling out an agreement form. This page also offers explanation documents describing step by step the application of the TMB to different standard video conferencing systems. They facilitate a user-friendly application of the TMB to a specific system.
Acknowledgements The development of the TMB would not have been possible without the foundation of the European Union. Special thanks are due to the project leader, Chiara Mancini, for the good project organization. We would also like to thank Danny van Reeth, Pascal Vanaverbeke, Evangelos V. Kopsacheilis, Lambros Makris for their collaboration regarding the international TMB sessions.
References [1] C.P. Friedman, C.W. Wyatt, Evaluation Methods in Medical Informatics, Springer-Verlag, New York, 1997. [2] M. Gerneth, Evaluation of Cost and Effect of Telemedicine Applications, Proceedings of TELEMED’97-Telematik im Gesundheitswesen, 7 – 8 November 1997, Berlin, Germany, pp. 42 – 54 [3] M.J. Field (Ed.), Telemedicine — A Guide to Assessing Telecommunications in Health Care, National Academy Press, Washington, 1996.
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141 [4] R. Wright, C. Loughrey, Teleradiology, BMJ 310 (1995) 1392 – 1393. [5] M. Testoni, P. Mattioli, F. Baruffaldi, P. Trentani, G. Bianchi, F. Fanton, M. Mieti, A. Toni, X-ray film and paired digitised image evaluation in total hip arthroplasty: a statistical comparison regarding the diagnostic capability. J. Bone Joint Surg., Proceedings of the Eighth Conference of European Orthopaedic Research Society (EORS), 7 – 10 May 1998, Amsterdam, accepted for publication. [6] L.G. Yamamoto, Using JPEG image compression to facilitate telemedicine, Am. J. Emerg. Med. 13 (1995) 55 – 57. [7] ACR Standard for Teleradiology, Revised 1996 (Res. 26) [8] Y. Ligier, O. Ratib, M. Logean, C. Girard, OSIRIS: a
.
141
medical image manipulation system, Comput. J. 11 (1994) 212 – 218. [9] P. Mattioli, P.J. Klutke, F. Baruffaldi, M. Viceconti, A. Toni, K.H. Englmeier, A study of the application sharing capabilities in telemedicine, Comput. Methods Programs Biomed. 58 (1999) 89 – 97. [10] W. Lamotte, E. Flerackers, F. Van Reeth, R. Earnshaw, J.M. De Matos, Visinet: collaborative 3D visualization and VR over ATM networks, IEEE Comput. Graph. 17 (1997) 66 – 75. [11] P. Mattioli, P.J. Klutke, F. Baruffaldi, A. Villar-Guzman, A. Toni, K.H. Englmeier, Technical validation of lowcost videoconferencing systems applied in orthopaedic teleconsulting services, Comput. Methods Programs Biomed. 60 (1999) 143 – 152.
The Telemedicine benchmark—a general tool to measure and compare the performance of video conferencing equipment in the telemedicine area Peter J. Klutke a,*, Paolo Mattioli b,1, Fabio Baruffaldi b,2, Aldo Toni b,3, Karl-H. Englmeier a,4 a
National Research Center for En6ironment and Health, Institute of Medical Informatics and Health Ser6ices Research, Ingolsta¨dter Landstraße 1, D-85764 Neuherberg, Germany b Instituti Ortopedici Rizzoli, Laboratorio di Tecnologia Medica, Via di barbiano 1 /10, I-40136 Bologna, Italy Received 11 November 1998; received in revised form 25 February 1999; accepted 25 February 1999
Abstract In this paper, we describe the ‘Telemedicine Benchmark’ (TMB), which is a set of standard procedures, protocols and measurements to test reliability and levels of performance of data exchange in a telemedicine session. We have put special emphasis on medical imaging, i.e. digital image transfer, joint viewing and editing and 3D manipulation. With the TMB, we can compare the aptitude of different video conferencing software systems for telemedicine issues and the effect of different network technologies (ISDN, xDSL, ATM, Ethernet). The evaluation criteria used are length of delays and functionality. For the application of the TMB, a data set containing radiological images and medical reports was set up. Considering the Benchmark protocol, this data set has to be exchanged between the partners of the session. The Benchmark covers file transfer, whiteboard usage, application sharing and volume data analysis and compression. The TMB has proven to be a useful tool in several evaluation issues. © 1999 Elsevier Science Ireland Ltd. All rights reserved. Keywords: Telemedicine; Performance; Evaluation; Video conferencing; ISDN; Medical imaging
1. Introduction, aims and scope * Corresponding author. Tel.: + 49-89-31874457; fax: +4989-31874243. E-mail addresses: [email protected] (P.J. Klutke), [email protected] (P. Mattioli), [email protected] (F. Baruffaldi), [email protected] (A. Toni), [email protected] (K.H. Englmeier) 1 Tel.: + 39-51-6366864; fax: + 39-51-6366863 2 Tel.: + 39-51-6366864; fax: + 39-51-6366863 3 Tel.: + 39-51-6366864; fax: + 39-51-6366863 4 Tel.: + 49-89-31874457; fax: + 49-89-31874243
In times of scarce resources, especially in the health care sector, it is crucial to evaluate the cost-effectiveness of solutions in telemedicine [1,2]. Video conferencing equipment costs have decreased to one third in the last 3 years. Thus, they have become affordable to the general practitioner, enabling him to invite the second opinion
0169-2607/99/$ - see front matter © 1999 Elsevier Science Ireland Ltd. All rights reserved. PII: S 0 1 6 9 - 2 6 0 7 ( 9 9 ) 0 0 0 2 7 - 9
134
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
of an expert. For linkages among hospitals, these new low-priced systems are used with buoyancy also. On the other hand, only very little in-depth research on well-defined environments has been carried out in this field. A thorough evaluation of these new techniques for remote communication in medicine is still in is infancy [3,4]. What is more, the software of the video conferencing systems has been improved significantly during the last 2 years. Application sharing, the possibility of joint working on the same application, is meanwhile available and may be used for computer supported collaborative work (CSCW) on medical files, statistics or medical images by sharing the related program. Our Telemedicine Benchmark (TMB) is a set of standard procedures, protocols and measurements to test reliability and levels of performance of data exchange in a telemedicine session. The presentation of the TMB is the scope of this paper. The evaluation criteria used are delay times and software functionality. In the protocol, both synchronous and asynchronous communication is tested. Medical use of asynchronous communication (store and forward procedures) is for example the transfer of medical images; the field of synchronous communication (real time joint working) includes the subsequent collaboration and discussion on the medical images. We have chosen examples of data exchange in radiology and orthopedics because of the high communication requirements of medical image transfer. Using image file transfer without loss of information, in contrary to the recording of images by H.320 compliant cameras avoids to deal with assessment of lossy compressed medical images, which has already been evaluated [5,6]. Furthermore, the protocol was designed with respect to the ACR Standard for Teleradiology [7]. The TMB was developed during the project PH-Net, which was sponsored by the European Commission, Contract No. 45491, within the framework of TEN-ISDN (Trans European Networks using ISDN). The main aims of PH-Net are: The improvement of knowledge regarding the health care area reality, to assist strategic decisions, planning and management.
The improvement of the quality of care by means of information sharing among professionals and specialists involved in the care process. The improvement of cost-effectiveness of careprocesses using standard teleconferencing equipment over ISDN lines to implement teleconsulting, teleradiology and telepathology. During the project PH-Net, the methods of communication and data exchange to be performed in the TMB have proven to be a realistic portrayal of clinical routine work. 2. General conditions to apply the telemedicine benchmark
2.1. Explanations and definitions Application sharing: A communication tool, that allows to share an application on one computer to the computer of a connected partner. This partner can make inputs to and see outputs of the shared application as if it ran on his computer. The application sharing tool is for example implemented in Microsoft NetMeeting™. LP: The partner who rings up the other partner (LP, local partner). Regarding application sharing, the LP (in this context called the ‘Host’) makes available one of his applications to RP. RP: The partner that is rung up by the other partner (RP, remote partner). Regarding application sharing, the RP (in this context called the ‘Guest’) is made available one of the applications of LP.
2.2. Requirements The TMB can be applied to every telemedicine application that disposes of the following features: Audio and video connection for remote sessions Data exchange by file transfer and/or whiteboard and/or application sharing It is applicable to different operating systems (Windows 3.1x, Windows 95, Windows NT, MacOS, UNIX). The following software, for the most part available through the Internet, is required:
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
IDL (Interactive Data Language®), a data visualization software from Research Systems, Boulder, Colorado, USA OSIRIS, a viewer for medical image data in DICOM format [8] Text editor (e.g. Microsoft® Wordpad) Zip compression software (e.g. WinZip®)
2.3. Preliminary set-up of the computer The computer has to be prepared carefully to guarantee equal circumstances for every TMB session. All settings that have proven to have an effect on the performance or the functionality of the whole system have to be brought into line. This includes the following aspects: The necessary software to be shared has to be installed. The screen settings as well as the number of colors have to be equalized. The screen savers have to be deactivated. The settings of the conferencing software have to be harmonized and adjusted to automatic acceptance of incoming calls and automatic storage of incoming files in an empty receipt directory.
2.4. Method of time delay measurement The time measurements are carried out by a chronometer. The sensitivity of the time measurements is 91 s. Two different arrangements are possible to adopt the TMB. If there are two telemedicine systems in one room for a local test, one has to refer to chapter 2.4.4: ‘Local sessions’; if there is a session with the two systems spatially separated, chapter 2.4.2: ‘Remote sessions’ is used. Chapter 2.4.3 ‘Guest only with application sharing (local and remote sessions)’ is relevant for both arrangements.
2.4.1. Local sessions To measure the time delay of an action ACT in a local session, LP and RP are the same person, that has to use the following steps: 1. Identify the end of ACT. It is marked in the TMB protocol by an asterisk (*). An example is the pushing of a button to load an image.
135
2. Identify the effect of the end of ACT. It is marked in the TMB protocol by a number sign (c ). The effect could be for example the appearance of the image. 3. Perform ACT on the LP side. Start your chronometer at the end of ACT. 4. Watch the RP screen. Stop your chronometer if the effect of the end of ACT appears on the RP screen AND the concerning window is completely restored (if it had been disturbed or had to be built up). 5. Write down the result of the time measurement in the TMB protocol immediately.
2.4.2. Remote sessions Measuring time delays in a remote session is a little more complicated than in a local session. The start of an action ACT by one partner has to be indicated to the other partner by audio. This is done by the countdown phrase ‘3-2-1-GO’. The other partner has to confirm that the effect of the end of ACT appeared on his screen. This is done by the conventional phrase ‘STOP’. For most actions (ACT) performed, two delays are measured, one by the local partner and one by the remote partner. They are obtained by applying the following steps. 1. LP and RP identify the end of ACT. It is marked in the TMB protocol by an asterisk (*). 2. LP and RP identify the effect of the end of ACT. It is marked in the TMB protocol by a number sign (c). 3. LP performs ACT on the his site. LP counts down ‘3-2-1-GO’ with ‘GO’ at the end of ACT and simultaneously starts his chronometer (at the end of ACT). 4. RP starts his chronometer when he receives the ‘GO’ of LP by audio. 5. RP watches his screen. He stops his chronometer if the effect of the end of ACT appears on his screen AND the concerning window is completely restored (if it had been disturbed or had to be built up). Simultaneously, he says ‘STOP’ to LP. 6. LP stops his chronometer when he receives ‘STOP’ of RP.
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
136
During the protocol always LP is the host and RP is the guest. RP starts the action ACT on his computer and measures the delay until the result of ACT appears on his screen. No audio delays have to be considered if this procedure is performed in remote sessions. The TMB protocol indicates when to use this measurement method.
2.5. Unpredictable e6ents, a6ailability of actions and general remarks
Fig. 1. Time delay measurement in remote sessions.
7. Both partners exchange their measured values and write them down in the TMB protocol immediately. Fig. 1 shows how the two time measurement results are obtained and why the time measurement by LP is longer twice the VC system audio delay than the time measurement by RP. Consequently, the best approximation of the action duration is obtained by the average of the two measured values.
2.4.3. Guest only with application sharing (local and remote sessions) The difference to the other two procedures is that only the guest of an application sharing session performs a measurement. As defined, the shared application runs on the host’s computer.
During the performance of the protocol, all unpredictable events are recorded together with the related action. Among them include interruption of the connection, the crash of the conferencing software, the crash of an application shared, sudden long time delays and a reboot of the system. This allows to identify weak points in the video conferencing software as well as a general statement regarding the stability and toughness of the whole system. The identification of the availability of every action of the TMB protocol helps to assess the overall capabilities of the conferencing software.
3. Description of the telemedicine benchmark
3.1. The telemedicine benchmark data set The TMB data set (Table 1) contains all the files needed to perform the TMB. These files are
Table 1 The telemedicine benchmark data set File name
File description
Size (bytes)
RX-post.tif TC-pre.dc Pictures.tif Report.txt RX-post.zip Pictures.zip Report.zip Bench.zip mr000xx.dcm
X-ray of a hip arthroplasty, scanned, 2000×1000 pixel, 8 Bit CT image for research purposes, DICOM 3.0 format, coronal cut, 512×512 pixel, 12 Bit Colored TIFF image with constant gray background showing post surgery exercises Standard text file containing a short medical report Zipped RX-post.tif (by WinZip 6.x) Zipped Pictures.tif (by WinZip 6.x) Zipped Report.txt (by WinZip 6.x) Zipped Benchmark files (by WinZip 6.x) 20 MR images, colon with contrast medium, DICOM 3.0 format, coronal cut, 132522 Bytes each, 256×256 Pixel, 12 Bit Zipped mr00040.dcm (by WinZip 6.x) Zipped mr00001.dcm to mr00020.dcm (by WinZip 6.x)
2 009 042 525 676 256 798 847 622 505 42 771 617 906 276 2 650 440
mr00040.zip mr01-20.zip
79 711 1 512 390
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
for example transferred by file transfer or are shared among the partners by application sharing of the related application.
3.2. The telemedicine benchmark protocol The TMB protocol describes in detail the procedure to apply the TMB. It consists of the main paragraphs file transfer, whiteboard usage, application sharing, volume data analysis and compression. Each paragraph may be applied independently from the others, as required by the user. Of course the TMB protocol requires the important hard- and software settings to be taken down before the start of the session.
3.2.1. File transfer File transfer will be of frequent use in clinical routine, transferring medical files, reports or images. It regards asynchronous as well as synchronous communication. With the automatic acceptance of incoming calls and the automatic storage of incoming files in a pre-selected directory the files may be transferred without any user interaction on the site of RP. Of course file transfer also makes sense during a conference. All files of the TMB data set are transferred and the necessary time is measured as indicated in the TMB protocol. One can derive the effectiveness of internal compression techniques as well as the utilization of the available bandwidth during file transfer. 3.2.2. Whiteboard usage Both sites enable the whiteboard (sometimes also called ‘notepad’) of their conferencing software. This is a work environment visible on the screens of all connected sites. Each partner can add or modify the contents of the whiteboard. The changes are immediately transferred to all connected sites. If data in the whiteboard is changed, this has usually no effect on the appropriate original file. This work environment is suitable for simple joint viewing and annotating of medical documents. Clinical routine usage may be the comparison of medical files or discussion on statistics.
137
During the TMB protocol, an X-ray image is visualized in the whiteboard. Then a region of interest and a line are drawn onto the image, telepointing is performed and the image is scrolled. Finally, a colored illustration and a snapshot of an application window are imported in the whiteboard. Time delays are measured as indicated in the TMB protocol. The results of the time delay measurements allow to assess the overall performance of the whiteboard as well as the identification of a suitable operational area of this tool given the tested hardware and software configuration.
3.2.3. Application sharing Application sharing provides more features than the common work environment ‘whiteboard’. It allows joint working with any windows application. This application is only running on the computer of one partner, called the host. The host has to share his application with the other partner, the guest. The guest has an identical user interface on his screen, showing the same as on the host’s screen and allowing interaction just as if the application was running on the guest’s computer [9]. Of course, the guest’s user interaction has to be sent to the host’s computer and the resulting changes on the screen have to be sent back to the guest’s computer. The time delay caused by this information transfer for selected actions is investigated in this section. For medical issues, the most time-critical issue is the collaboration on medical images. To test this aspect, the DICOM viewer OSIRIS for medical images [8] that is common available through the Internet is used for application sharing. A CT image is displayed with this viewer, annotations are made and the gray-value windowing is changed, which may be in particular important for remote education and training. Almost the same is done for an X-ray image. Finally, a word processing system is used to simulate dealing with patient data and medical reports. Text visualization as well as copy and paste operations are tested. The results of the time delay measurements do not only allow to assess the overall performance of application sharing and the intelligent use of
138
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
the bandwidth available. They also evaluate the use of internal compression techniques, supporting the decision whether the usage of the whiteboard or of application sharing is more efficient in a certain situation.
3.2.4. Volume data analysis By the volume data analysis section of the TMB, it was intended to get an idea of the possibilities for remote 3D image analysis. This task demands a lot as it deals with a huge amount of data. Twenty CT images with 512× 512 pixels each result in 10 Mb of data. Transferring these files in uncompressed form results in a great time delay. However, for a 3D analysis of medical image data, interactive working on the data set should be possible. This means maximum response times of 5 s. The 3D visualization can be achieved by application sharing. The shared program is IDL (Interactive Data Language) v5.0, which provides 3D visualization features. Using application sharing for 3D image analysis provides the following advantage. The volume data is stored only on one partner’s side and the CPU of his computer has to perform the 3D operations. All results of 3D operations are viewed and examined on the screen, that means they are 2D information. Only this information has to be transferred by the application sharing tool. The reduction of the amount of information to be transferred from three dimensions (volume data) to two dimensions (screen data) makes application sharing feasible for 3D remote image analysis. For the TMB, the IDL demo with its appropriate data set was used. The following actions were tested: Application sharing of the IDL slicer, an interactive 3D visualization tool. Visualization of a yellow rectangle to visualize a cutting plane through CT volume data. Visualization of the cut-out slice. Deletion of the 3D image. Display of a shaded surface representing a threshold in the CT volume data. Change of the z-rotation of a small wire netting cube representing the volume data.
Display of the rotated surface according to the chosen change of the z-rotation. Rotating the wire netting of a human heart once by a mouse movement. Releasing the mouse button to make the shaded 3D surface appear. It is possible to convert DICOM files of a medical slice series (CT, MR) into raw data that can easily be visualized with the IDL slicer tool.
3.2.5. Compression During the TMB, we have obtained transfer times for files in compressed and uncompressed format. Together with time measurements for compression and decompression of these files we can answer the question if it makes sense to compress files before transferring them as regards the time needed for the whole action. For the calculation we use the following variables: fu fc h g
th tu tc tu tc
uncompressed file size (Mb) compressed file size (Mb) compression factor (= fc/fu) specific time for compression and decompression referring to uncompressed. data (s/ Mb) constant user overhead for the handling of the compression software (s) specific transfer time for uncompressed data (s/Mb) specific transfer time for compressed data (s/Mb) whole time needed for file transfer without compression (s) whole time needed for file transfer with compression (s)
The time for transferring the file is calculated as follows: whole time needed= time for compression + transfer time tu = 0+ tu × fu (without compression)
(1)
tc = th + g× fu + tc × fc (with compression)
(2)
with
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
139
If Eq. (7) is not true, no positive break-even file size f\0 exists and it never makes sense to compress files of this type before transferring them.
4. Results, discussion and outlook
Fig. 2. The definition of the ‘break-even file size’.
fc = fu × h
(3)
one gets: tu = fu ×tu
(4)
tc = th + fu ×(g+tc ×h)
(5)
It only makes sense to compress a file before transferring it if tu( fu)\tc( fu)
(6)
For th \ 0 this can only happen for large fu if tu \ (g+ tc × h)
(7)
Eq. (7) means that the gradient of tu( fu) must be larger than the gradient of tc( fu). Fig. 2 shows the dependencies between the variables. If Eq. (7) is true and th \0, the two straight lines in Fig. 2 intersect. A ‘break-even file size’ f may be defined as follows. If a file size is larger than the break-even file size, it saves time to compress the file before transferring it. If the size of a file is exactly the break-even file size, we get tu(f)/tc(f)
(8)
Therefore, one gets the break-even file size f from Eq. (8) as f = th = (tu −tc × h− g)
(9)
The TMB covers the exchange of 2- and 3-D data and its joint analysis, because these fields make the greatest demands on channel capacity usage. Further developments concerning one dimensional data analysis (e.g. electrocardiography data) and dynamic digital imaging (e.g. images coming from an optical microscope) are going to be implemented. The exchange of patient records, included in the TMB as text file exchange, may also be intensified. This field, however, demands a lot more of the clinical information system’s interoperability than of the overall performance and is therefore not quite within the scope of the TMB. To test specific features like dynamic bandwidth allocation or audio and/or video protocol changes an advanced section is being developed. It has not been integrated in the basic TMB because the knowledge required to apply these measurements is considerably higher. No special technical or medical expertise, however, should be required to perform the basic TMB. Moreover, the TMB is an excellent tool to introduce inexperienced users into the current possibilities of telemedicine. Finally, the TMB is scaleable, i.e. you can only use parts of it or even just one action. The user of the TMB has to decide which parts are relevant for him. As the TMB uses commonly available software (Wordpad™ text editor, OSIRIS DICOM viewer, IDL™ data visualization software, Zip compression tools) to be shared among partners during the TMB or for compression topics, the TMB is not dependent on a certain operating system. Except for the Microsoft™ Wordpad™ all other necessary software is available through the Internet, first of all free of charge. The Microsoft™ Wordpad™ is included in Windows 95/NT™. Other operating systems offer similar tools. The TMB allows to test and compare different telemedicine systems with different settings and
140
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141
configurations thus supporting the user to choose the most cost-efficient solution. These different systems may be characterized by different network technologies (e.g. ISDN Point to Point, TCP/IP Routing by ISDN with and without encryption, xDSL, fiberglass, ATM) [10], different video conferencing systems (e.g. Intel ProShare 200™, Intel™ Business Video Conferencing, PictureTel Live 200™, Aethra SDV8000™, Aethra Eycona V™), different software configurations (e.g. preferences of video conferencing system, screen settings), different operating systems (e.g. Windows 95™, Windows 98™, Windows NT 4.0™, MacOS™, UNIX™) and different hardware configurations (e.g. main processor, RAM, hard disk capacity, graphics card). With the TMB, future buyers of VC systems can evaluate different equipment before buying. For those who have already bought equipment, the results of the TMB measurements help to decide upon the best use of different data communication components like ‘Whiteboard’ or ‘Application Sharing’ to take care of a given task like a medical image transfer. The TMB and the related results are of use for the manufacturers of video conferencing or telemedicine equipment also. They get a clear catalogue of demands for their systems as well as comparable evaluation results, which specify the capacity of their systems, help to identify weak points and possible improvements. Scientists can develop new algorithms optimized for specific tasks contained in the TMB. A detailed description of measurement results applying the TMB to different systems is not within the scope of this paper. Another publication presents the results of the TMB’s application to the four desktop video conferencing systems Intel ProShare 200™, Intel™ Business Video Conferencing, PictureTel™ Live 200 and Aethra™ SDV8000 [11]. The results show that even low-cost desktop video conferencing solutions are suitable to be applied in the telemedicine area (e.g. Orthopedics teleconsulting). Here the
TMB has proven to be an excellent tool for evaluation and comparison. The evaluation of the TMB by measuring time delays or response times happens on the application layer of the ISO-OSI network reference model. The causes of the time delays are not only due to the application used, but also due to effects related to lower layers. The preliminary uniform set-up of the systems and the recording of hardand software settings intend to harmonize and/or identify these effects. It is of great importance to include them into the final discussion of the TMB measurement results. We developed a web page (http:// www.tecno.ior.it/telemed/projects/TMBench/index.html) where the TMB data set and TMB protocol will be available free of charge after filling out an agreement form. This page also offers explanation documents describing step by step the application of the TMB to different standard video conferencing systems. They facilitate a user-friendly application of the TMB to a specific system.
Acknowledgements The development of the TMB would not have been possible without the foundation of the European Union. Special thanks are due to the project leader, Chiara Mancini, for the good project organization. We would also like to thank Danny van Reeth, Pascal Vanaverbeke, Evangelos V. Kopsacheilis, Lambros Makris for their collaboration regarding the international TMB sessions.
References [1] C.P. Friedman, C.W. Wyatt, Evaluation Methods in Medical Informatics, Springer-Verlag, New York, 1997. [2] M. Gerneth, Evaluation of Cost and Effect of Telemedicine Applications, Proceedings of TELEMED’97-Telematik im Gesundheitswesen, 7 – 8 November 1997, Berlin, Germany, pp. 42 – 54 [3] M.J. Field (Ed.), Telemedicine — A Guide to Assessing Telecommunications in Health Care, National Academy Press, Washington, 1996.
P.J. Klutke et al. / Computer Methods and Programs in Biomedicine 60 (1999) 133–141 [4] R. Wright, C. Loughrey, Teleradiology, BMJ 310 (1995) 1392 – 1393. [5] M. Testoni, P. Mattioli, F. Baruffaldi, P. Trentani, G. Bianchi, F. Fanton, M. Mieti, A. Toni, X-ray film and paired digitised image evaluation in total hip arthroplasty: a statistical comparison regarding the diagnostic capability. J. Bone Joint Surg., Proceedings of the Eighth Conference of European Orthopaedic Research Society (EORS), 7 – 10 May 1998, Amsterdam, accepted for publication. [6] L.G. Yamamoto, Using JPEG image compression to facilitate telemedicine, Am. J. Emerg. Med. 13 (1995) 55 – 57. [7] ACR Standard for Teleradiology, Revised 1996 (Res. 26) [8] Y. Ligier, O. Ratib, M. Logean, C. Girard, OSIRIS: a
.
141
medical image manipulation system, Comput. J. 11 (1994) 212 – 218. [9] P. Mattioli, P.J. Klutke, F. Baruffaldi, M. Viceconti, A. Toni, K.H. Englmeier, A study of the application sharing capabilities in telemedicine, Comput. Methods Programs Biomed. 58 (1999) 89 – 97. [10] W. Lamotte, E. Flerackers, F. Van Reeth, R. Earnshaw, J.M. De Matos, Visinet: collaborative 3D visualization and VR over ATM networks, IEEE Comput. Graph. 17 (1997) 66 – 75. [11] P. Mattioli, P.J. Klutke, F. Baruffaldi, A. Villar-Guzman, A. Toni, K.H. Englmeier, Technical validation of lowcost videoconferencing systems applied in orthopaedic teleconsulting services, Comput. Methods Programs Biomed. 60 (1999) 143 – 152.