- Email: [email protected]
Experiences with a workstation prototype for softcopy reading within the bavarian mammography recertification program1
Computer Assisted Radiology and Surgery
Original Investigations
Experiences With a Workstation Prototype for Softcopy Reading Within the Bavarian Mammography Recertification Program1 Workstations and Education Jo¨rg Riesmeier, MSc, Marco Eichelberg, PhD, Hans-Peter Hellemann, DP, Joachim Kieschke, MSP, Thomas Wilkens, DI
Rationale and Objectives: In January 2002, the Bavarian Statutory Health Care Administration (“Kassena¨rztliche Vereinigung Bayerns”, KVB) started a recertification program for quality assurance and quality improvement in mammography reading. Materials and Methods: All accredited radiologists and gynecologists are asked to prove their qualification every 1–2 years. The recertification program requires the physicians to read 50 cases randomly selected from a larger collection of high-quality test cases. The portion of malignant and benign cases corresponds to the requirements of the German National Association of Statutory Health Insurance Physicians (“Kassena¨rztliche Bundesvereinigung”, KBV). In order to perform the recertification in a manageable manner a decentralized approach (test at different locations in parallel) was preferred over a centralized one. Therefore, the X-ray films were digitized and converted to DICOM Digital Mammography format to be read on a softcopy device. To verify the applicability of digitized mammograms for recertification purposes, a comparative study with 32 trained radiologists and gynecologists was performed. Results: A system of two high-resolution/high-contrast monitors (2048 ⫻ 2560 pixels, ⱖ350 cd/m2) in combination with a 5 mega-pixel dual-head graphics adapter with calibrated output was chosen for the mammography workstation. The software was implemented according to the particular requirements of this program. As a result, the comparative study showed that there was no significant difference in the error rate of the reported findings between conventional film and softcopy reading. Conclusion: The first intermediate results of this quality initiative are promising. As of 2003, the test is mandatory for all mammography-reading physicians in Bavaria. Key Words. Mammography; reporting; quality assurance; recertification program; DICOM. ©
AUR, 2004
In January 2002, the Bavarian Statutory Health Care Administration (“Kassena¨rztliche Vereinigung Bayerns,”
Acad Radiol 2004; 11:407– 418 1 From the OFFIS e.V., Escherweg 2, Oldenburg, Germany 26121, OFFIS CARE GmbH, Industriestr. 9, Oldenburg 26121 (J.R., M.E., T.W.), NCA Mikroelektronic GmbH (J.K.), Mu¨nchen, OFFIS e.V., (H.-P.H.), Oldenburg, Germany. Received November 25, 2003; revision requested December 17; revision received and accepted December 23. e-mail: [email protected]
©
AUR, 2004 doi:10.1016/j.acra.2003.12.004
KVB) started a recertification program for quality assurance and quality improvement in mammography reading. The basis for this program was established by an agreement between the German National Association of Statutory Health Insurance Physicians (“Kassena¨rztliche Bundesvereinigung,” KBV) and the German public health care insurance companies. All accredited radiologists and gynecologists are asked to prove their qualification every 1–2 years. The participating physicians are required to read 50 cases randomly selected from a larger collection
407
RIESMEIER ET AL
(initially 150) of high quality test cases. Each case consists of 4 mammographic views (cranio– caudal and medio–lateral oblique view, left and right breast) acquired on conventional X-ray film. The portion of malignant and benign cases corresponds to the requirements of the KBV, ie, among 50 cases there are 19 –29 carcinomas, 1–2 double carcinomas and several benign lesions. The cases were pre-selected on film for being either BI-RADS (Breast Imaging Reporting and Data System) (1) 1 (negative) and 2 (benign finding) or 5 (highly suspicious of malignancy, do biopsy). All malignant cases had proven histology, all benign cases had a present follow-up report and were at least two years old. Another requirement was that the lesions had to be visible in both cranio– caudal and medio–lateral oblique view. Uncertain cases with the need for an extra examination were removed from the collection. The gold standard for each case was defined by a radiologic expert commission: the BI-RADS classification was derived from the referring physician’s report whereas the location of a finding was specified based on a consensus decision. However, the location was not part of the test and, therefore, not assessed—it was included for the reason of scientific completeness only. During the test, a participating physician is asked to read all 200 images (50 cases with 4 images each) and to report his findings separately for the left and the right breast. To pass the test, both sensitivity (true positive rate) and specificity (true negative rate) must be at least 95%. Because specificity is not as easy to reproduce, the rule was set to 2 false negatives (overlooked carcinomas) and 7 errors in total (which is equivalent to 5–7 false positives). The first quarter of 2002 was planned as a trial period where physicians were invited to participate voluntarily. A certificate for successfully passing the test is being issued since May 2002. MATERIAL AND METHODS To perform the recertification in a manageable manner a decentralized approach (test at different locations in parallel) was preferred over a centralized one. Moreover, it was planned to distribute sample mammography images among interested physicians for exercise purposes. However, because the mammography images used for the recertification program were only available on conventional X-ray film this would require the creation of physical copies. For reasons of quality and economy it was decided to digitize the films instead and to read the images
408
Academic Radiology, Vol 11, No 4, April 2004
on softcopy. Because of the strict requirements for mammograms regarding spatial and contrast resolution the films were digitized using a high-quality CCD (Charge Coupled Device) scanner (570 dpi, 4096 shades of grey) to be viewed on an appropriate display workstation. Apart from the fact that digital images can be copied an infinite number of times without any loss of information—which ensures equal conditions for all participants—the use of digital equipment significantly facilitates the preparation and evaluation of a test (see “Workstation Prototype”). To read digitized mammograms on a softcopy device, an appropriate display workstation was required. It was decided not to use an available workstation offered by a commercial company but to develop a new one specifically for the purpose of this recertification program. The main reason was to avoid complaints from competitors with regard to the choice of a particular product. The usage of a commercially available mammography workstation for recertification purposes could tempt physicians to purchase the same station for their practice to feel better prepared for the test. This would definitely distort the market in this field. Another reason for developing a new workstation was that none of the existing systems could be tailored to the particular requirements of the program as well as a dedicated specialized system could. Other research groups had similar experiences and, therefore, developed dedicated systems for their purposes (eg, (2) and (3)). Another question that needed to be answered before the start of the tests was whether the use of digitized mammograms and softcopy equipment is appropriate for recertification purposes, given that users receive an initial training in handling the computer system. It is quite obvious that the characteristics of a mammography image displayed on a softcopy system deviate significantly from a conventional mammography, both in terms of spatial and contrast resolution as well as in the way images can be “processed”: While conventional screen/film reading allows the use of tools such as a magnifying glass or changing the brightness of the light box, a computer based system supports tools such as window level and width adjustment or zoom. The Comparative Study section summarizes the results of a comparative study that has been performed to answer this question. RESULTS Workstation Prototype The development of the mammography workstation started in September 2001, only a few weeks before a
Academic Radiology, Vol 11, No 4, April 2004
first prototype of the system was expected to be presented at a press conference (4). Because of the short time it was decided to focus on the selection of appropriate hardware and to limit the software development part to a “rapid prototype” that was later completely redesigned (“second generation”). In addition, an initial set of mammography films was digitized and converted to standard DICOM (Digital Imaging and Communications in Medicine) (5) format. The basic functionality requirement of the workstation was that it display mammography images and input the associated findings. During the design phase, the main objectives for the mammography workstation were identified as follows: • • • • •
high-quality presentation of mammography images, easy to use and easy to administer, limited set of essential functions only, standard conformance, and system portability.
To meet the first and most important requirement, a system of two high-resolution/high-contrast monitors (2048 ⫻ 2560 pixels, ⱖ 350 cd/m2) in combination with a 5 mega-pixel dual-head graphics adapter with calibrated output was chosen. A third, conventional monitor was used to display the input mask for the findings. Although the software to be developed was required to be portable, it was decided to use an adequately equipped Unix workstation (1 GByte RAM, 160 GByte hard disk) as the main development platform. The reason for this decision was that a Unix-based system could be protected much easier from unauthorized manipulation than most other systems, thus simplifying the work of system administrators once the systems were deployed. Figure 1 shows a picture of this first workstation prototype (a) and a later-developed standard PC version (b). Before any test, the mammography films must be digitized and converted to DICOM format (see “Scanning Software”). Furthermore, the gold standard must be specified and stored in the system. This is the first of three processes the workstation software can be used for. For this purpose, the software allows the display of four image views of each case and allows findings to be entered according to the BI-RADS classification as well as the location of the finding (see Fig. 2). It is possible to specify more than one BI-RADS code and location to be valid. Finally, this “gold standard mode” allows the user to adjust the default value of interest (VOI) window that
WORKSTATION PROTOTYPE FOR SOFTCOPY READING
is used when an image is displayed to the user for the first time. The second process, the performance of the test, is handled by the “test mode” of the software. The user interface is almost identical to that of the previously described “gold standard mode.” The input mask (Fig. 2) allows the user to navigate step-by-step through the test cases and to enter the detected findings in arbitrary order. Two large check marks for the left and the right sides indicate that a case has been processed completely (ie, BI-RADS code and location—in the case of a positive finding— have been filled). The viewer shows the mammography images with the initial hanging protocol (cranio-caudal view of both sides to ensure that the user saw all mammograms in full resolution) and the default VOI window. In the default configuration with three monitors, the images of the right breast are always displayed on the left monitor and vice versa. In addition, a number of pre-defined hanging protocols and single image views are provided to enable the user to determine the location of a finding. Images are always scaled to fit into the maximum available space on the screen. However, the resolution of a high-quality digitized mammography film (570 dpi) is much higher than what can be displayed on any monitor available today. Therefore, the electronic counterpart of a magnifying glass can be used to show particular image details with original resolution (one pixel on the screen corresponds to one pixel in the DICOM image). Moreover, it is possible to adjust the VOI window interactively with the mouse to improve the visibility of lowcontrast details, to reset the VOI window to the default settings as well as to other VOI windows stored in the DICOM image, to invert the image (white becomes black and vice versa), and to perform a couple of other essential functions available in most image-viewing stations. Before closing the application, the software checks whether all test cases have been processed completely and warns the user if this is not the case. Finally, the total processing time (difference between start and end time) is noted in the system for future evaluation. The third process concerns the evaluation of the test and is managed by the “review mode” of the software. The input mask simultaneously shows the user input (recorded during the “test mode”) and the gold standard. To facilitate the evaluation two check marks are used to indicate correct findings. In addition, the total number of errors (wrong findings) and warnings (minor deviations) is displayed in the title bar of the window. In “review
409
RIESMEIER ET AL
Academic Radiology, Vol 11, No 4, April 2004
Figure 1. Pictures of the workstation prototype (a) and a standard PC version with dual display (b).
mode,” the image viewer is hidden by default but can be activated on demand. Moreover, a scanned version of the clinical findings report (including histology) can be displayed for each case. This enables a supervisor performing the evaluation to explain any deviation of the test results to the participant in an unambiguous manner. Though still a prototype, the software is highly configurable. Because of short development time, a relatively simple text file was preferred over a more complex data-
410
base system for storing the configuration information. This information includes • general information: default screen layout of the image viewer, start and end time of the test, number of the last processed case, directory where the DICOM images and clinical findings reports are stored, etc., • user information: name and additional identifying information of the person performing the test,
Academic Radiology, Vol 11, No 4, April 2004
Figure 2.
WORKSTATION PROTOTYPE FOR SOFTCOPY READING
Input mask of the viewing software used to report findings.
• a list of test cases the user can navigate through, • for each test case: gold standard and user input (BIRADS code and location for the left and the right breast), filename of the clinical findings report (if any), information on the four DICOM images (one for each view), • for each DICOM image: filename, default and current user-defined VOI window. To grant equal treatment to all participants, the 50 test cases are randomly selected taking the above-mentioned KBV requirements into account. The basic principle of the algorithm is based on the “drawing of lots.” The four different classes of findings (carcinomas, double carcinomas, benign lesions, and inconspicuous cases) are associated with four boxes. First, all lots each representing a single case are put into the corresponding box. Then the number of lots to be drawn from each box is determined by randomly selecting a number within a given range (eg, 19 –29 for cases with carcinomas). The remainder from 50 is assigned to the box with the inconspicuous cases. Afterward, the determined number of lots (cases) is randomly selected from each of the four boxes (eg, 22 with carcinomas, 2 with double carcinomas, 15 with benign lesions, and 11 inconspicuous cases). Finally, the order of the 50 selected cases is arranged in a random order. This algorithm ensures that neither the number of cases for
each class nor the concrete cases within each class nor the order of the test cases are predictable. To summarize, a typical workflow of a test session within the scope of the recertification program would look like the following: 1.
A supervisor registers the new participant to the system. The recorded information includes name, address, birth date, and identification (identity card data and KVB certification number) of the person. 2. A new set of 50 cases is randomly selected from the collection of test cases using the above-described algorithm. The workstation software is started in “test mode” allowing the participant to read the first set of images. Participants who are not familiar with the handling of the system receive a short standardized introduction before the test. 3. Immediately after the test has been completed, the results are evaluated in the participant’s presence. The supervisor shows the deviations from the gold standard and is able to refer to the clinical findings report if required. Finally, the system automatically prints a detailed report for every incorrect case and a summary of the test results.
Based on the experiences gained so far, it is intended that a “third generation” of the workstation software be devel-
411
RIESMEIER ET AL
oped. One of the considerations is to omit the third monitor currently used to display the input mask. This would, of course, require the same functionality to be integrated into the two remaining high-resolution/high-brightness monitors without significantly reducing the available space for the mammography images, eg, by specifying the location of a finding directly in the mammography image. Another intention is to decrease the latency when switching to another test case, which is mainly caused by the huge amount of pixel data that must be loaded and processed. At the moment the waiting time of a few seconds per case does not really affect the user because a complete test session with 50 cases typically takes a few hours. Nevertheless, it interrupts the workflow of the user, which is particularly harmful in a “test situation.” Home Edition In addition to the workstation software, a “home edition” operating on a standard Windows PC has been developed allowing physicians to practice the test procedure at home and get used to working with the software. The main challenge was to minimize the hardware requirements for running the software without changing the visual appearance and handling of the system too much. If the “home edition” looked completely different it would be of little help. Because of the system-independent development of the workstation software only a few changes were required to port the software to the Windows operating system. The user interface and handling had to be slightly modified to reflect the fact that one cannot assume multiple screens to be connected to a standard PC. Therefore, the three main application windows (two image viewers and the input mask) are shown on a single monitor. The two image viewers are arranged side by side covering the whole screen (Fig. 3 (b)); the input mask is typically hidden in the background but can be brought to the front by a single keystroke. Moreover, support for computer mice with only two buttons has been added because they are still widely used—the workstation version of the software required three buttons to use the complete functionality. Based on this “home edition” of the software, a training CD-ROM has been composed and distributed. Because of the limited capacity of this storage medium (650 MByte) and the relatively small size of working memory (RAM) typically found in a PC, the size of the mammography images had to be reduced. A compromise was found in scaling down the images by factor 2 and additionally compressing them in a lossless manner (Run
412
Academic Radiology, Vol 11, No 4, April 2004
Length Encoding, RLE). The size of the resulting DICOM images was around 4 MByte, which is about an eighth of the original size. That way, 35 cases fit on a single CD-ROM. Another advantage of the reduced image size is an improved load time: CD-ROM drives are still considerably slower than hard disks and the overhead for the RLE decompression is insignificant compared with this saving of time. As mentioned before, the main purpose of this training system is to “learn” the handling of the software, not to perform primary diagnosis. Having this goal in mind, the reduced resolution of the mammography images can be tolerated just as the reduced quality of most consumer monitors. However, to get the best possible results from the available display system, a visual test pattern is displayed before the first test case. This allows the user to adjust the brightness and contrast settings of the monitor to get the best possible results. A more sophisticated calibration procedure as it is performed on the mammography workstation would be desirable but is not feasible at the moment due to the demand of additional hardware (eg, a photometer) and an experienced operator—this might change in the future. The 35 cases on the CD-ROM are, of course, not part of the collection of test cases used for the recertification. However, the corresponding gold standard definitions and clinical findings reports are included. This allows the user to check his results for correctness in the same way as it is done after the “real” test. A first edition of the CD-ROM called “Mammo Lite” (Fig. 3 (a)) was presented at a press conference of the KVB in September 2002. About 250 copies had been distributed among the journalists and interested physicians. At the same time it was announced that a copy of this training CD-ROM could be ordered at cost. Due to the numerous advance orders, a second revised edition with 1000 copies was produced in November 2002. The general feedback was predominantly positive. Only a few problems have been reported that were mainly related to the system configuration and not to the handling of the training software. However, most of these problems could be solved by calling the telephone hotline that was installed together with the distribution of the CD-ROMs. To avoid frequently observed problems with the initial operation during the test stage, a start-up application had been added to the CD-ROM, which automatically checks whether the present PC fulfils the minimum system requirements (graphics resolution and RAM size). This ap-
Academic Radiology, Vol 11, No 4, April 2004
WORKSTATION PROTOTYPE FOR SOFTCOPY READING
Figure 3. Training software “Mammo Lite”—CD with cover (a) and screenshot of the user interface (b).
plication is also used to start the training or review mode and to show the user manual. The software is fully operational from CD-ROM; ie, no installation procedure is required. This fact is of vital importance to smoothly introduce this training system with minimum effort in the medical domain.
Scanning Software The presentation of the first workstation prototype required only a few test cases to show the feasibility. At that time, the main focus for acquiring digitized mammograms was on image quality and not the way of scanning. For a small number of films it was, therefore, acceptable
413
RIESMEIER ET AL
Figure 4.
Scanning software “DicomMammoScan”—interactive selection of the region of interest.
to manually scan each film and store the resulting pixel data first in a standard image format—in our case, TIFF. The DICOM image file (Digital Mammography X-ray Image Storage For Presentation SOP Class) was then created by calling a script that would in turn call several small command line tools. Because the TIFF file did not contain any information on the image view and most other header information required for DICOM mammography images, this data had to be retrieved from the filename or to be “invented” if not crucial for image display. This was not very handy and in particular required the attention of the operator when assigning the filenames. Because the conversion from TIFF to DICOM format also included a non-linear transformation of the greyscale values (linear optical density to luminance), quality control could only take place on the basis of the converted DICOM image, eg, to check whether the important “skin line” of the breast is visible. Thus, an insufficiently scanned film could only be detected at the very end of the processing pipeline. The cropping of image data (ie, reduction of the image to a rectangular region of interest) was cumbersome and often required the same film to be scanned multiple times. Cropping was essential to avoid bright areas at the film borders or the patient data, which is typically exposed into the film to be visible without truncating the breast tissue, though. It is obvious that digitizing hundreds of films requires a more sophisticated solution that is easier to use and al-
414
Academic Radiology, Vol 11, No 4, April 2004
lows for a better quality control. Therefore, a dedicated scanning software, called “DicomMammoScan” (Fig. 4), has been developed which controls the film scanner by a software component provided by the manufacturer of the device. The new tool covers the entire scanning procedure previously handled by a couple of separate applications. It is possible to scan a single mammography film or a set of four (according to the four image views of a case). All scanner settings can be configured separately for both modes and are automatically stored as the default. However, it is also possible to store and reload particular settings (eg, profiles). Typically, the four images of a case are scanned in one pass but if only a single image needs to be scanned or re-scanned this is also supported by the software. An “application wizard,” as known from many software installations, guides the user through the individual steps of the scanning procedure. First, the user is asked to insert the films into the feed tray of the scanner in a predefined order (in case more than one film is to be scanned). After successfully digitizing the image data the user is able to specify both the region of interest (ROI) and the default VOI window either interactively with the mouse or by manual input. Optionally, the ROI that is used to crop the image can be fixed to ensure an identical image size for all four views. Finally, the DICOM images are created. Both filenames and header information are
WORKSTATION PROTOTYPE FOR SOFTCOPY READING
Academic Radiology, Vol 11, No 4, April 2004
Table 1 Number and Type of Incorrectly Reported Results, Sensitivity, and Specificity Depending on Reporting Medium
Reported mammae Incorrect results From original film From scanned image Correct results (%) Original film Scanned image
Total
Malignant findings (sensitivity)
Non-malignant findings (specificity)
3198 148 73 75 95.4 95.4 95.3
960 63 26 37 93.4 94.6 92.3
2238 85 47 38 96.2 95.8 96.6
generated automatically based on the current user configuration. The “DicomMammoScan” software facilitates the complex and tedious procedure of scanning mammography films. The total time to scan and convert a set of four films could be reduced by a factor of 2–3. Now, the limiting factor is the hardware and not the user. Although we expect a trained layperson to be able to deal with this important task we still count on the experience of a medical physicist who is able to directly assess the image quality of mammography films and digitized image data. Comparative Study It is quite obvious that the characteristics of a mammography image displayed on a softcopy system deviates significantly from a conventional mammography, both in terms of spatial and contrast resolution as well as in the way images can be “processed”: While conventional screen/film reading allows the use of tools such as a magnifying glass or changing the brightness of the light box, a computer based system supports tools such as window level and width adjustment or zoom. Therefore, one of the questions the project tried to answer was whether a use of digitized mammograms and softcopy equipment is appropriate for recertification purposes, given that users receive an initial training in handling the computer system. The goal of a comparative study funded by the Bavarian Ministry for Labour and Social Affairs (6) was to examine the influence of the reporting medium (original film on a light box vs. digitized image on a computer monitor) on the result of mammography reporting. A number of factors might be relevant in this context:
• • • •
quality of the digitization process characteristics of the cases individual professional qualification of the participant individual familiarity of the participant with the reporting medium
To take the first two influencing factors into account the selection of mammography cases and a quality check of the scanned images was performed by a radiologic expert commission. Each participant had to read the same number of cases on both reporting media. This allowed to compensate for the differences in the professional qualification of the participants. Because a physician would be able to remember a conspicuous finding seeing it a second time on the other reporting medium a “2 period change-over plan” with regard to the reporting medium was developed: In a first pass 16 of the 32 participants had to read 25 mammography cases on hardcopy (original film on a light box) and 25 on softcopy (digitized images on a computer monitor). The cases were randomly selected from the collection of test cases under consideration of the KBV requirements. In a second pass the 16 remaining participants had to read the same cases on the respective other medium. To evaluate the results the following parameters were determined: • error rate: proportion of wrong results with regard to the total number of reported mammae • sensitivity: proportion of false negatives with regard to the total number of reported mammae with malignant findings • specificity: proportion of false positives with regard to the total number of reported mammae with non-malignant findings As Table 1 shows, the sensitivity for the original film was higher (94.6% to 92.3%) whereas the specificity was slightly lower (95.8% to 96.6%). The proportion of cor-
Table 2 Agreement of Reported Results at Pairwise Comparison of Complementary Reportings Scanned Images Original Film
Malignant
Non-malignant
Total
Malignant Non-malignant Total
406 58 464
72 993 1,065
478 1,051 1,529
415
RIESMEIER ET AL
rect results was almost identical (95.4% to 95.3%). If less information is perceived on a mammography image, it can be expected that the proportion of false negatives will increase and the proportion of false positives will decrease. To check the degree of agreement of the reported results depending on the reporting medium the 100 mammographies (50 cases each with left and right breast) were complementary assigned to a pair of participants, ie, each case was read on both media by each pair of participants. This gave a total number of 1,529 pairwise collected reporting results (see Table 2). A total of 91.5% (406⫹993⫽1,399 out of 1,529) were identical which is a very high agreement. Additionally, the equivalence of the reporting results was evaluated by determining the differences of the error rate, the sensitivity and the specificity of a particular participant for both media. Equivalence was defined in case the upper 95% confidence interval of the mean of the differences was below 0.05. For the total error rate and the specificity the equivalence could be verified. For the sensitivity the proof of equivalence failed. However, because the deviation was relatively small and the mean of the differences was far below the critical value of 0.05 it can be expected that an extension of the study would also verify the equivalence of the sensitivity because the standard error would then be smaller. Even though equivalence could not yet be verified for all criteria the agreement of the reported results is very high. Furthermore, this initial study compared original films with scanned images. For a decentralized approach as described above physical copies with significant quality losses would be essential though. When comparing the results for film copies and scanned original images the conclusion would, therefore, probably be more in favor of the softcopy approach. In addition to the descriptive analysis described above, the technical report (7) of the comparative study also includes typical measures as Cohen’s kappa value and other common descriptions. Unfortunately, the study is currently only available in German language. DISCUSSION The mammography workstation for recertification purposes is currently installed at three KVB offices in Bavaria: Munich, Nuremberg and Wu¨ rzburg. Other German states have already expressed their interest in our solution. The system has been running without major prob-
416
Academic Radiology, Vol 11, No 4, April 2004
lems for almost two years now (as of December 2003). Unix as an operating system ensures stability with little maintenance effort. Due to comprehensive access restrictions and a couple of automation scripts, the workstation rarely requires the help of a technical expert. Even more important is the fact that the system is accepted by the medical professionals who have to perform the recertification test. Up to now only four participants out of more than 400 asked to use the conventional light box with films instead of reading the images at the mammography workstation. To not exclude physicians who have reservations about modern technology or concerns about the equivalence of softcopy and hardcopy reporting, taking the test using a conventional light box was an option to all participants. However, the system is sufficiently easy to use to allow also inexperienced users to cope with it after a short introduction. It is particularly appreciated that the printed result is available immediately after the test has been finished and that the user input can be compared directly with the gold standard and to the clinical findings report on the monitor. Furthermore, the automatic selection of test cases on an access-controlled system excludes possible manipulations. After scanning several hundreds of X-ray films we found the digitizing to be the crucial part of the whole project. As mentioned before, mammograms make great demands on scanners regarding spatial and contrast resolution. Different scan technologies claim to provide the best results but we experienced that there is currently no ideal solution. All scanners jam on films that slightly stick together or sometimes load a film skewed. Apart from these mechanical problems, greyscale consistency is also still an issue. Conventionally acquired X-ray films strongly differ in their range of optical density. This is particularly true if films from different sources are used as in our collection of test cases. Therefore, the “exposure time” of the CCD scanner needs to be adjusted manually. Scanners supporting the DICOM Grayscale Standard Display Function (GSDF) are not yet available as far as we know. This part of the DICOM standard has been published years ago (8) and since then quite successfully penetrated the domain of medical display systems and printers (9). As a matter of course, the mammography workstation developed in this project fully complies with the GSDF and the relevant DIN standard (10) requiring output calibration on a regular basis and controlled ambient light conditions during image display. The American Association of Physicists in Medicine (AAPM) developed both a document on “Assessment of Display Performance
Academic Radiology, Vol 11, No 4, April 2004
for Medical Imaging Systems” (11) which compiles many of these aspects and a set of test patterns which allow to assess the quality of a particular display system. To achieve standardized viewing conditions the rooms’ windows at the three locations where the system is currently installed are pasted up to keep the changing sunlight out. Instead, the room is slightly illuminated by an indirect constant light source. Furthermore, the display system is calibrated regularly according to the manufacturer’s guidelines. Finally, the overall display consistency is ensured by measuring the luminance emitted from the monitor (including ambient light) by means of a luminance meter. To achieve similar results for the digitizing process, an automatic quality control procedure using standardized test patterns exposed to the film would be desirable. One vision is that some time in the future the recertification process could be performed at the physician’s home or practice. This approach would be a completely decentralized solution. However, it would also require appropriate hardware, particularly a workstation with calibrated display system, to be present at the physician’s home or practice. Furthermore, it is much more complicated to protect a computer system at the physician’s home or practice from unauthorized manipulations than a few systems at the KVB offices. Even if the gold standard definitions and clinical findings reports, ie, the correct test results, would never be present on the physician’s system, malfunctions of the software or hardware could have a negative effect on the test. In the first instance, it is consequently planned to only refine the “home edition” of the software. Today’s high-end PC systems are already capable of processing and displaying mammograms in full resolution. A DVD (Digital Versatile Disc) could replace the CD-ROM as storage medium and store the same amount of images in original resolution. Other formerly specialized hardware like highbrightness monitors and high-resolution graphics adapters, partly equipped with dual heads, is entering the consumer market and becomes, therefore, more and more affordable. Greyscale consistency by calibrating the output of a display system is also already available to consumer systems, either completely implemented in hardware or partly in software. Another consideration in this context is to create a mobile test system moving around from practice to practice. In conclusion, the first intermediate results of this quality initiative are promising. A few months after the trial period was started, the hospital physicians in Bavaria
WORKSTATION PROTOTYPE FOR SOFTCOPY READING
joined the recertification program. As of 2003, the test is mandatory for all mammography-reading physicians in Bavaria. The introduction of a corresponding federal German recertification program is intended and actually required by legislation until the end of this year. Whether our mammography workstation is adopted or whether digital images are used at all—instead of conventional X-ray films—is not clear yet, though. However, the equivalence of using digitized images on a digital mammography workstation and conventional film on a light box as part of the federal recertification program has been defined. Publications from other research groups—mainly from mammography screening projects—also indicate “that a dedicated reading station allows soft-copy reading of digitized mammograms without loss of quality” (12). However, they restrict their conclusion by requiring “that CAD is available for microcalcification detection.” This requirement is certainly sensible in the screening or diagnostic reading context but not within a recertification program where the professional qualification of the participants is to be tested. Other groups propose to pre-process digital mammograms “for improving their visualization on softcopy display systems” (13). As for the workstation developed during this project, it is intended to further enhance the software according to the experiences gained so far. We already made a development plan for a “third generation” of the workstation software. Furthermore, we intend to release the software as Open Source and, therefore, allow everyone to validate the test system in detail. REFERENCES 1. American College of Radiology (ACR). Breast Imaging Reporting and Data System (BI-RADS), 3rd ed, http://www.acr.org/departments/ stand_accred/birads/, 1998, Accessed January 9, 2004 2. Hemminger BM.Softcopy Display Requirements for Digital Mammography. In: Mun SK, ed. Medical Imaging 2002: Visualization, Image-Guided Procedures, and Display. Proceedings of SPIE 2002; 4681:366 –379 3. Evertsz CJG, Bo¨ dicker A, Brady M, Hendriks J, Ju¨ rgens H, Karssemeijer N, et al. Soft-Copy Reading Environment for Screening Mammography—SCREEN. In: Yaffe MJ, ed. 5th International Workshop on Digital Mammography, Toronto, Ontario, June 2000. Madison, WI: Medical Physics Publishing, 2001, 566 –572 4. Kassena¨ rztliche Vereinigung Bayerns. KVB stellt digitale Befundungsstation vor. Press Release, October 2001, https://www.kvb.de/ servlet/PB/menu/1024330/ 5. NEMA Standards Publication PS 3.x. Digital Imaging and Communications in Medicine (DICOM). Rosslyn, VA: National Electrical Manufacturers Association, 2003. 6. Kassena¨ rztliche Vereinigung Bayerns, Mammographie—Neue Standards in der Qualita¨ t setzen, Digitale Befundungsstation der KVB. Press Release, March 2002, https://www.kvb.de/servlet/PB/menu/1036000/ 7. Kieschke J, Jensch P. Studie zur Beurteilung der Verwendbarkeit digitalisierter konventioneller Aufnahmen bei der Rezertifizierung der Befundung von Mammographien. Technical Report, OFFIS, 2002
417
RIESMEIER ET AL
8. Eichelberg M, Riesmeier J, Kleber K, Holstein J, Oosterwijk HJ, Jensch P. Consistency of Softcopy and Hardcopy: Preliminary Experiences with the new DICOM Extensions for Image Display. In: Blaine GJ, Siegel EL, eds. PACS Design and Evaluation: Engineering and Clinical Issues. Proceedings of SPIE 2000;3980:57– 67. 9. Riesmeier J, Eichelberg M, Kleber K, Gro¨ nemeyer DHW, Oosterwijk HJ, Jensch P. DICOM image display consistency: a test environment. In: Siegel EL, Huang HK, eds. Medical Imaging 2001: PACS and Integrated Medical Information Systems: Design and Evaluation. Proceedings of SPIE 2001;4323:47–56. 10. Deutsches Institut fu¨ r Normung (DIN). Sicherung der Bildqualita¨ t in ro¨ ntgendiagnostischen Betrieben—Teil 57: Abnahmepru¨ fung an Bildwiedergabegera¨ ten. DIN V 6868-57 (2001).
418
Academic Radiology, Vol 11, No 4, April 2004
11. Samei E, Badano A, Chakraborty D, Compton K, Cornelius C, Corrigan K, et al. Assessment of Display Performance for Medical Imaging Systems. Draft Report of the American Association of Physicists in Medicine (AAPM) Task Group 18, Version 9.0, October 2002. 12. Roelofs T, van Woudenberg S, Hendricks J, Bo¨ dicker A, Evertsz C, Karssemeijer M. Performance Evaluation of a Digital Reading Station for Screening Mammography. In: Peitgen HO, ed. 6th International Workshop on Digital Mammography, June 22–25, 2002, Bremen Germany. Berlin: Springer-Verlag, 2003; 455– 459. 13. Fan J, Dallas WJ, Roehrig H, Krupinski EA.Improving visualization of digital mammograms on the CRT display system. In: Galloway RL, ed. Medical Imaging 2003: Visualization, Image-Guided Procedures, and Display. Proceedings of SPIE 2003;5029:746 –753.
Original Investigations
Experiences With a Workstation Prototype for Softcopy Reading Within the Bavarian Mammography Recertification Program1 Workstations and Education Jo¨rg Riesmeier, MSc, Marco Eichelberg, PhD, Hans-Peter Hellemann, DP, Joachim Kieschke, MSP, Thomas Wilkens, DI
Rationale and Objectives: In January 2002, the Bavarian Statutory Health Care Administration (“Kassena¨rztliche Vereinigung Bayerns”, KVB) started a recertification program for quality assurance and quality improvement in mammography reading. Materials and Methods: All accredited radiologists and gynecologists are asked to prove their qualification every 1–2 years. The recertification program requires the physicians to read 50 cases randomly selected from a larger collection of high-quality test cases. The portion of malignant and benign cases corresponds to the requirements of the German National Association of Statutory Health Insurance Physicians (“Kassena¨rztliche Bundesvereinigung”, KBV). In order to perform the recertification in a manageable manner a decentralized approach (test at different locations in parallel) was preferred over a centralized one. Therefore, the X-ray films were digitized and converted to DICOM Digital Mammography format to be read on a softcopy device. To verify the applicability of digitized mammograms for recertification purposes, a comparative study with 32 trained radiologists and gynecologists was performed. Results: A system of two high-resolution/high-contrast monitors (2048 ⫻ 2560 pixels, ⱖ350 cd/m2) in combination with a 5 mega-pixel dual-head graphics adapter with calibrated output was chosen for the mammography workstation. The software was implemented according to the particular requirements of this program. As a result, the comparative study showed that there was no significant difference in the error rate of the reported findings between conventional film and softcopy reading. Conclusion: The first intermediate results of this quality initiative are promising. As of 2003, the test is mandatory for all mammography-reading physicians in Bavaria. Key Words. Mammography; reporting; quality assurance; recertification program; DICOM. ©
AUR, 2004
In January 2002, the Bavarian Statutory Health Care Administration (“Kassena¨rztliche Vereinigung Bayerns,”
Acad Radiol 2004; 11:407– 418 1 From the OFFIS e.V., Escherweg 2, Oldenburg, Germany 26121, OFFIS CARE GmbH, Industriestr. 9, Oldenburg 26121 (J.R., M.E., T.W.), NCA Mikroelektronic GmbH (J.K.), Mu¨nchen, OFFIS e.V., (H.-P.H.), Oldenburg, Germany. Received November 25, 2003; revision requested December 17; revision received and accepted December 23. e-mail: [email protected]
©
AUR, 2004 doi:10.1016/j.acra.2003.12.004
KVB) started a recertification program for quality assurance and quality improvement in mammography reading. The basis for this program was established by an agreement between the German National Association of Statutory Health Insurance Physicians (“Kassena¨rztliche Bundesvereinigung,” KBV) and the German public health care insurance companies. All accredited radiologists and gynecologists are asked to prove their qualification every 1–2 years. The participating physicians are required to read 50 cases randomly selected from a larger collection
407
RIESMEIER ET AL
(initially 150) of high quality test cases. Each case consists of 4 mammographic views (cranio– caudal and medio–lateral oblique view, left and right breast) acquired on conventional X-ray film. The portion of malignant and benign cases corresponds to the requirements of the KBV, ie, among 50 cases there are 19 –29 carcinomas, 1–2 double carcinomas and several benign lesions. The cases were pre-selected on film for being either BI-RADS (Breast Imaging Reporting and Data System) (1) 1 (negative) and 2 (benign finding) or 5 (highly suspicious of malignancy, do biopsy). All malignant cases had proven histology, all benign cases had a present follow-up report and were at least two years old. Another requirement was that the lesions had to be visible in both cranio– caudal and medio–lateral oblique view. Uncertain cases with the need for an extra examination were removed from the collection. The gold standard for each case was defined by a radiologic expert commission: the BI-RADS classification was derived from the referring physician’s report whereas the location of a finding was specified based on a consensus decision. However, the location was not part of the test and, therefore, not assessed—it was included for the reason of scientific completeness only. During the test, a participating physician is asked to read all 200 images (50 cases with 4 images each) and to report his findings separately for the left and the right breast. To pass the test, both sensitivity (true positive rate) and specificity (true negative rate) must be at least 95%. Because specificity is not as easy to reproduce, the rule was set to 2 false negatives (overlooked carcinomas) and 7 errors in total (which is equivalent to 5–7 false positives). The first quarter of 2002 was planned as a trial period where physicians were invited to participate voluntarily. A certificate for successfully passing the test is being issued since May 2002. MATERIAL AND METHODS To perform the recertification in a manageable manner a decentralized approach (test at different locations in parallel) was preferred over a centralized one. Moreover, it was planned to distribute sample mammography images among interested physicians for exercise purposes. However, because the mammography images used for the recertification program were only available on conventional X-ray film this would require the creation of physical copies. For reasons of quality and economy it was decided to digitize the films instead and to read the images
408
Academic Radiology, Vol 11, No 4, April 2004
on softcopy. Because of the strict requirements for mammograms regarding spatial and contrast resolution the films were digitized using a high-quality CCD (Charge Coupled Device) scanner (570 dpi, 4096 shades of grey) to be viewed on an appropriate display workstation. Apart from the fact that digital images can be copied an infinite number of times without any loss of information—which ensures equal conditions for all participants—the use of digital equipment significantly facilitates the preparation and evaluation of a test (see “Workstation Prototype”). To read digitized mammograms on a softcopy device, an appropriate display workstation was required. It was decided not to use an available workstation offered by a commercial company but to develop a new one specifically for the purpose of this recertification program. The main reason was to avoid complaints from competitors with regard to the choice of a particular product. The usage of a commercially available mammography workstation for recertification purposes could tempt physicians to purchase the same station for their practice to feel better prepared for the test. This would definitely distort the market in this field. Another reason for developing a new workstation was that none of the existing systems could be tailored to the particular requirements of the program as well as a dedicated specialized system could. Other research groups had similar experiences and, therefore, developed dedicated systems for their purposes (eg, (2) and (3)). Another question that needed to be answered before the start of the tests was whether the use of digitized mammograms and softcopy equipment is appropriate for recertification purposes, given that users receive an initial training in handling the computer system. It is quite obvious that the characteristics of a mammography image displayed on a softcopy system deviate significantly from a conventional mammography, both in terms of spatial and contrast resolution as well as in the way images can be “processed”: While conventional screen/film reading allows the use of tools such as a magnifying glass or changing the brightness of the light box, a computer based system supports tools such as window level and width adjustment or zoom. The Comparative Study section summarizes the results of a comparative study that has been performed to answer this question. RESULTS Workstation Prototype The development of the mammography workstation started in September 2001, only a few weeks before a
Academic Radiology, Vol 11, No 4, April 2004
first prototype of the system was expected to be presented at a press conference (4). Because of the short time it was decided to focus on the selection of appropriate hardware and to limit the software development part to a “rapid prototype” that was later completely redesigned (“second generation”). In addition, an initial set of mammography films was digitized and converted to standard DICOM (Digital Imaging and Communications in Medicine) (5) format. The basic functionality requirement of the workstation was that it display mammography images and input the associated findings. During the design phase, the main objectives for the mammography workstation were identified as follows: • • • • •
high-quality presentation of mammography images, easy to use and easy to administer, limited set of essential functions only, standard conformance, and system portability.
To meet the first and most important requirement, a system of two high-resolution/high-contrast monitors (2048 ⫻ 2560 pixels, ⱖ 350 cd/m2) in combination with a 5 mega-pixel dual-head graphics adapter with calibrated output was chosen. A third, conventional monitor was used to display the input mask for the findings. Although the software to be developed was required to be portable, it was decided to use an adequately equipped Unix workstation (1 GByte RAM, 160 GByte hard disk) as the main development platform. The reason for this decision was that a Unix-based system could be protected much easier from unauthorized manipulation than most other systems, thus simplifying the work of system administrators once the systems were deployed. Figure 1 shows a picture of this first workstation prototype (a) and a later-developed standard PC version (b). Before any test, the mammography films must be digitized and converted to DICOM format (see “Scanning Software”). Furthermore, the gold standard must be specified and stored in the system. This is the first of three processes the workstation software can be used for. For this purpose, the software allows the display of four image views of each case and allows findings to be entered according to the BI-RADS classification as well as the location of the finding (see Fig. 2). It is possible to specify more than one BI-RADS code and location to be valid. Finally, this “gold standard mode” allows the user to adjust the default value of interest (VOI) window that
WORKSTATION PROTOTYPE FOR SOFTCOPY READING
is used when an image is displayed to the user for the first time. The second process, the performance of the test, is handled by the “test mode” of the software. The user interface is almost identical to that of the previously described “gold standard mode.” The input mask (Fig. 2) allows the user to navigate step-by-step through the test cases and to enter the detected findings in arbitrary order. Two large check marks for the left and the right sides indicate that a case has been processed completely (ie, BI-RADS code and location—in the case of a positive finding— have been filled). The viewer shows the mammography images with the initial hanging protocol (cranio-caudal view of both sides to ensure that the user saw all mammograms in full resolution) and the default VOI window. In the default configuration with three monitors, the images of the right breast are always displayed on the left monitor and vice versa. In addition, a number of pre-defined hanging protocols and single image views are provided to enable the user to determine the location of a finding. Images are always scaled to fit into the maximum available space on the screen. However, the resolution of a high-quality digitized mammography film (570 dpi) is much higher than what can be displayed on any monitor available today. Therefore, the electronic counterpart of a magnifying glass can be used to show particular image details with original resolution (one pixel on the screen corresponds to one pixel in the DICOM image). Moreover, it is possible to adjust the VOI window interactively with the mouse to improve the visibility of lowcontrast details, to reset the VOI window to the default settings as well as to other VOI windows stored in the DICOM image, to invert the image (white becomes black and vice versa), and to perform a couple of other essential functions available in most image-viewing stations. Before closing the application, the software checks whether all test cases have been processed completely and warns the user if this is not the case. Finally, the total processing time (difference between start and end time) is noted in the system for future evaluation. The third process concerns the evaluation of the test and is managed by the “review mode” of the software. The input mask simultaneously shows the user input (recorded during the “test mode”) and the gold standard. To facilitate the evaluation two check marks are used to indicate correct findings. In addition, the total number of errors (wrong findings) and warnings (minor deviations) is displayed in the title bar of the window. In “review
409
RIESMEIER ET AL
Academic Radiology, Vol 11, No 4, April 2004
Figure 1. Pictures of the workstation prototype (a) and a standard PC version with dual display (b).
mode,” the image viewer is hidden by default but can be activated on demand. Moreover, a scanned version of the clinical findings report (including histology) can be displayed for each case. This enables a supervisor performing the evaluation to explain any deviation of the test results to the participant in an unambiguous manner. Though still a prototype, the software is highly configurable. Because of short development time, a relatively simple text file was preferred over a more complex data-
410
base system for storing the configuration information. This information includes • general information: default screen layout of the image viewer, start and end time of the test, number of the last processed case, directory where the DICOM images and clinical findings reports are stored, etc., • user information: name and additional identifying information of the person performing the test,
Academic Radiology, Vol 11, No 4, April 2004
Figure 2.
WORKSTATION PROTOTYPE FOR SOFTCOPY READING
Input mask of the viewing software used to report findings.
• a list of test cases the user can navigate through, • for each test case: gold standard and user input (BIRADS code and location for the left and the right breast), filename of the clinical findings report (if any), information on the four DICOM images (one for each view), • for each DICOM image: filename, default and current user-defined VOI window. To grant equal treatment to all participants, the 50 test cases are randomly selected taking the above-mentioned KBV requirements into account. The basic principle of the algorithm is based on the “drawing of lots.” The four different classes of findings (carcinomas, double carcinomas, benign lesions, and inconspicuous cases) are associated with four boxes. First, all lots each representing a single case are put into the corresponding box. Then the number of lots to be drawn from each box is determined by randomly selecting a number within a given range (eg, 19 –29 for cases with carcinomas). The remainder from 50 is assigned to the box with the inconspicuous cases. Afterward, the determined number of lots (cases) is randomly selected from each of the four boxes (eg, 22 with carcinomas, 2 with double carcinomas, 15 with benign lesions, and 11 inconspicuous cases). Finally, the order of the 50 selected cases is arranged in a random order. This algorithm ensures that neither the number of cases for
each class nor the concrete cases within each class nor the order of the test cases are predictable. To summarize, a typical workflow of a test session within the scope of the recertification program would look like the following: 1.
A supervisor registers the new participant to the system. The recorded information includes name, address, birth date, and identification (identity card data and KVB certification number) of the person. 2. A new set of 50 cases is randomly selected from the collection of test cases using the above-described algorithm. The workstation software is started in “test mode” allowing the participant to read the first set of images. Participants who are not familiar with the handling of the system receive a short standardized introduction before the test. 3. Immediately after the test has been completed, the results are evaluated in the participant’s presence. The supervisor shows the deviations from the gold standard and is able to refer to the clinical findings report if required. Finally, the system automatically prints a detailed report for every incorrect case and a summary of the test results.
Based on the experiences gained so far, it is intended that a “third generation” of the workstation software be devel-
411
RIESMEIER ET AL
oped. One of the considerations is to omit the third monitor currently used to display the input mask. This would, of course, require the same functionality to be integrated into the two remaining high-resolution/high-brightness monitors without significantly reducing the available space for the mammography images, eg, by specifying the location of a finding directly in the mammography image. Another intention is to decrease the latency when switching to another test case, which is mainly caused by the huge amount of pixel data that must be loaded and processed. At the moment the waiting time of a few seconds per case does not really affect the user because a complete test session with 50 cases typically takes a few hours. Nevertheless, it interrupts the workflow of the user, which is particularly harmful in a “test situation.” Home Edition In addition to the workstation software, a “home edition” operating on a standard Windows PC has been developed allowing physicians to practice the test procedure at home and get used to working with the software. The main challenge was to minimize the hardware requirements for running the software without changing the visual appearance and handling of the system too much. If the “home edition” looked completely different it would be of little help. Because of the system-independent development of the workstation software only a few changes were required to port the software to the Windows operating system. The user interface and handling had to be slightly modified to reflect the fact that one cannot assume multiple screens to be connected to a standard PC. Therefore, the three main application windows (two image viewers and the input mask) are shown on a single monitor. The two image viewers are arranged side by side covering the whole screen (Fig. 3 (b)); the input mask is typically hidden in the background but can be brought to the front by a single keystroke. Moreover, support for computer mice with only two buttons has been added because they are still widely used—the workstation version of the software required three buttons to use the complete functionality. Based on this “home edition” of the software, a training CD-ROM has been composed and distributed. Because of the limited capacity of this storage medium (650 MByte) and the relatively small size of working memory (RAM) typically found in a PC, the size of the mammography images had to be reduced. A compromise was found in scaling down the images by factor 2 and additionally compressing them in a lossless manner (Run
412
Academic Radiology, Vol 11, No 4, April 2004
Length Encoding, RLE). The size of the resulting DICOM images was around 4 MByte, which is about an eighth of the original size. That way, 35 cases fit on a single CD-ROM. Another advantage of the reduced image size is an improved load time: CD-ROM drives are still considerably slower than hard disks and the overhead for the RLE decompression is insignificant compared with this saving of time. As mentioned before, the main purpose of this training system is to “learn” the handling of the software, not to perform primary diagnosis. Having this goal in mind, the reduced resolution of the mammography images can be tolerated just as the reduced quality of most consumer monitors. However, to get the best possible results from the available display system, a visual test pattern is displayed before the first test case. This allows the user to adjust the brightness and contrast settings of the monitor to get the best possible results. A more sophisticated calibration procedure as it is performed on the mammography workstation would be desirable but is not feasible at the moment due to the demand of additional hardware (eg, a photometer) and an experienced operator—this might change in the future. The 35 cases on the CD-ROM are, of course, not part of the collection of test cases used for the recertification. However, the corresponding gold standard definitions and clinical findings reports are included. This allows the user to check his results for correctness in the same way as it is done after the “real” test. A first edition of the CD-ROM called “Mammo Lite” (Fig. 3 (a)) was presented at a press conference of the KVB in September 2002. About 250 copies had been distributed among the journalists and interested physicians. At the same time it was announced that a copy of this training CD-ROM could be ordered at cost. Due to the numerous advance orders, a second revised edition with 1000 copies was produced in November 2002. The general feedback was predominantly positive. Only a few problems have been reported that were mainly related to the system configuration and not to the handling of the training software. However, most of these problems could be solved by calling the telephone hotline that was installed together with the distribution of the CD-ROMs. To avoid frequently observed problems with the initial operation during the test stage, a start-up application had been added to the CD-ROM, which automatically checks whether the present PC fulfils the minimum system requirements (graphics resolution and RAM size). This ap-
Academic Radiology, Vol 11, No 4, April 2004
WORKSTATION PROTOTYPE FOR SOFTCOPY READING
Figure 3. Training software “Mammo Lite”—CD with cover (a) and screenshot of the user interface (b).
plication is also used to start the training or review mode and to show the user manual. The software is fully operational from CD-ROM; ie, no installation procedure is required. This fact is of vital importance to smoothly introduce this training system with minimum effort in the medical domain.
Scanning Software The presentation of the first workstation prototype required only a few test cases to show the feasibility. At that time, the main focus for acquiring digitized mammograms was on image quality and not the way of scanning. For a small number of films it was, therefore, acceptable
413
RIESMEIER ET AL
Figure 4.
Scanning software “DicomMammoScan”—interactive selection of the region of interest.
to manually scan each film and store the resulting pixel data first in a standard image format—in our case, TIFF. The DICOM image file (Digital Mammography X-ray Image Storage For Presentation SOP Class) was then created by calling a script that would in turn call several small command line tools. Because the TIFF file did not contain any information on the image view and most other header information required for DICOM mammography images, this data had to be retrieved from the filename or to be “invented” if not crucial for image display. This was not very handy and in particular required the attention of the operator when assigning the filenames. Because the conversion from TIFF to DICOM format also included a non-linear transformation of the greyscale values (linear optical density to luminance), quality control could only take place on the basis of the converted DICOM image, eg, to check whether the important “skin line” of the breast is visible. Thus, an insufficiently scanned film could only be detected at the very end of the processing pipeline. The cropping of image data (ie, reduction of the image to a rectangular region of interest) was cumbersome and often required the same film to be scanned multiple times. Cropping was essential to avoid bright areas at the film borders or the patient data, which is typically exposed into the film to be visible without truncating the breast tissue, though. It is obvious that digitizing hundreds of films requires a more sophisticated solution that is easier to use and al-
414
Academic Radiology, Vol 11, No 4, April 2004
lows for a better quality control. Therefore, a dedicated scanning software, called “DicomMammoScan” (Fig. 4), has been developed which controls the film scanner by a software component provided by the manufacturer of the device. The new tool covers the entire scanning procedure previously handled by a couple of separate applications. It is possible to scan a single mammography film or a set of four (according to the four image views of a case). All scanner settings can be configured separately for both modes and are automatically stored as the default. However, it is also possible to store and reload particular settings (eg, profiles). Typically, the four images of a case are scanned in one pass but if only a single image needs to be scanned or re-scanned this is also supported by the software. An “application wizard,” as known from many software installations, guides the user through the individual steps of the scanning procedure. First, the user is asked to insert the films into the feed tray of the scanner in a predefined order (in case more than one film is to be scanned). After successfully digitizing the image data the user is able to specify both the region of interest (ROI) and the default VOI window either interactively with the mouse or by manual input. Optionally, the ROI that is used to crop the image can be fixed to ensure an identical image size for all four views. Finally, the DICOM images are created. Both filenames and header information are
WORKSTATION PROTOTYPE FOR SOFTCOPY READING
Academic Radiology, Vol 11, No 4, April 2004
Table 1 Number and Type of Incorrectly Reported Results, Sensitivity, and Specificity Depending on Reporting Medium
Reported mammae Incorrect results From original film From scanned image Correct results (%) Original film Scanned image
Total
Malignant findings (sensitivity)
Non-malignant findings (specificity)
3198 148 73 75 95.4 95.4 95.3
960 63 26 37 93.4 94.6 92.3
2238 85 47 38 96.2 95.8 96.6
generated automatically based on the current user configuration. The “DicomMammoScan” software facilitates the complex and tedious procedure of scanning mammography films. The total time to scan and convert a set of four films could be reduced by a factor of 2–3. Now, the limiting factor is the hardware and not the user. Although we expect a trained layperson to be able to deal with this important task we still count on the experience of a medical physicist who is able to directly assess the image quality of mammography films and digitized image data. Comparative Study It is quite obvious that the characteristics of a mammography image displayed on a softcopy system deviates significantly from a conventional mammography, both in terms of spatial and contrast resolution as well as in the way images can be “processed”: While conventional screen/film reading allows the use of tools such as a magnifying glass or changing the brightness of the light box, a computer based system supports tools such as window level and width adjustment or zoom. Therefore, one of the questions the project tried to answer was whether a use of digitized mammograms and softcopy equipment is appropriate for recertification purposes, given that users receive an initial training in handling the computer system. The goal of a comparative study funded by the Bavarian Ministry for Labour and Social Affairs (6) was to examine the influence of the reporting medium (original film on a light box vs. digitized image on a computer monitor) on the result of mammography reporting. A number of factors might be relevant in this context:
• • • •
quality of the digitization process characteristics of the cases individual professional qualification of the participant individual familiarity of the participant with the reporting medium
To take the first two influencing factors into account the selection of mammography cases and a quality check of the scanned images was performed by a radiologic expert commission. Each participant had to read the same number of cases on both reporting media. This allowed to compensate for the differences in the professional qualification of the participants. Because a physician would be able to remember a conspicuous finding seeing it a second time on the other reporting medium a “2 period change-over plan” with regard to the reporting medium was developed: In a first pass 16 of the 32 participants had to read 25 mammography cases on hardcopy (original film on a light box) and 25 on softcopy (digitized images on a computer monitor). The cases were randomly selected from the collection of test cases under consideration of the KBV requirements. In a second pass the 16 remaining participants had to read the same cases on the respective other medium. To evaluate the results the following parameters were determined: • error rate: proportion of wrong results with regard to the total number of reported mammae • sensitivity: proportion of false negatives with regard to the total number of reported mammae with malignant findings • specificity: proportion of false positives with regard to the total number of reported mammae with non-malignant findings As Table 1 shows, the sensitivity for the original film was higher (94.6% to 92.3%) whereas the specificity was slightly lower (95.8% to 96.6%). The proportion of cor-
Table 2 Agreement of Reported Results at Pairwise Comparison of Complementary Reportings Scanned Images Original Film
Malignant
Non-malignant
Total
Malignant Non-malignant Total
406 58 464
72 993 1,065
478 1,051 1,529
415
RIESMEIER ET AL
rect results was almost identical (95.4% to 95.3%). If less information is perceived on a mammography image, it can be expected that the proportion of false negatives will increase and the proportion of false positives will decrease. To check the degree of agreement of the reported results depending on the reporting medium the 100 mammographies (50 cases each with left and right breast) were complementary assigned to a pair of participants, ie, each case was read on both media by each pair of participants. This gave a total number of 1,529 pairwise collected reporting results (see Table 2). A total of 91.5% (406⫹993⫽1,399 out of 1,529) were identical which is a very high agreement. Additionally, the equivalence of the reporting results was evaluated by determining the differences of the error rate, the sensitivity and the specificity of a particular participant for both media. Equivalence was defined in case the upper 95% confidence interval of the mean of the differences was below 0.05. For the total error rate and the specificity the equivalence could be verified. For the sensitivity the proof of equivalence failed. However, because the deviation was relatively small and the mean of the differences was far below the critical value of 0.05 it can be expected that an extension of the study would also verify the equivalence of the sensitivity because the standard error would then be smaller. Even though equivalence could not yet be verified for all criteria the agreement of the reported results is very high. Furthermore, this initial study compared original films with scanned images. For a decentralized approach as described above physical copies with significant quality losses would be essential though. When comparing the results for film copies and scanned original images the conclusion would, therefore, probably be more in favor of the softcopy approach. In addition to the descriptive analysis described above, the technical report (7) of the comparative study also includes typical measures as Cohen’s kappa value and other common descriptions. Unfortunately, the study is currently only available in German language. DISCUSSION The mammography workstation for recertification purposes is currently installed at three KVB offices in Bavaria: Munich, Nuremberg and Wu¨ rzburg. Other German states have already expressed their interest in our solution. The system has been running without major prob-
416
Academic Radiology, Vol 11, No 4, April 2004
lems for almost two years now (as of December 2003). Unix as an operating system ensures stability with little maintenance effort. Due to comprehensive access restrictions and a couple of automation scripts, the workstation rarely requires the help of a technical expert. Even more important is the fact that the system is accepted by the medical professionals who have to perform the recertification test. Up to now only four participants out of more than 400 asked to use the conventional light box with films instead of reading the images at the mammography workstation. To not exclude physicians who have reservations about modern technology or concerns about the equivalence of softcopy and hardcopy reporting, taking the test using a conventional light box was an option to all participants. However, the system is sufficiently easy to use to allow also inexperienced users to cope with it after a short introduction. It is particularly appreciated that the printed result is available immediately after the test has been finished and that the user input can be compared directly with the gold standard and to the clinical findings report on the monitor. Furthermore, the automatic selection of test cases on an access-controlled system excludes possible manipulations. After scanning several hundreds of X-ray films we found the digitizing to be the crucial part of the whole project. As mentioned before, mammograms make great demands on scanners regarding spatial and contrast resolution. Different scan technologies claim to provide the best results but we experienced that there is currently no ideal solution. All scanners jam on films that slightly stick together or sometimes load a film skewed. Apart from these mechanical problems, greyscale consistency is also still an issue. Conventionally acquired X-ray films strongly differ in their range of optical density. This is particularly true if films from different sources are used as in our collection of test cases. Therefore, the “exposure time” of the CCD scanner needs to be adjusted manually. Scanners supporting the DICOM Grayscale Standard Display Function (GSDF) are not yet available as far as we know. This part of the DICOM standard has been published years ago (8) and since then quite successfully penetrated the domain of medical display systems and printers (9). As a matter of course, the mammography workstation developed in this project fully complies with the GSDF and the relevant DIN standard (10) requiring output calibration on a regular basis and controlled ambient light conditions during image display. The American Association of Physicists in Medicine (AAPM) developed both a document on “Assessment of Display Performance
Academic Radiology, Vol 11, No 4, April 2004
for Medical Imaging Systems” (11) which compiles many of these aspects and a set of test patterns which allow to assess the quality of a particular display system. To achieve standardized viewing conditions the rooms’ windows at the three locations where the system is currently installed are pasted up to keep the changing sunlight out. Instead, the room is slightly illuminated by an indirect constant light source. Furthermore, the display system is calibrated regularly according to the manufacturer’s guidelines. Finally, the overall display consistency is ensured by measuring the luminance emitted from the monitor (including ambient light) by means of a luminance meter. To achieve similar results for the digitizing process, an automatic quality control procedure using standardized test patterns exposed to the film would be desirable. One vision is that some time in the future the recertification process could be performed at the physician’s home or practice. This approach would be a completely decentralized solution. However, it would also require appropriate hardware, particularly a workstation with calibrated display system, to be present at the physician’s home or practice. Furthermore, it is much more complicated to protect a computer system at the physician’s home or practice from unauthorized manipulations than a few systems at the KVB offices. Even if the gold standard definitions and clinical findings reports, ie, the correct test results, would never be present on the physician’s system, malfunctions of the software or hardware could have a negative effect on the test. In the first instance, it is consequently planned to only refine the “home edition” of the software. Today’s high-end PC systems are already capable of processing and displaying mammograms in full resolution. A DVD (Digital Versatile Disc) could replace the CD-ROM as storage medium and store the same amount of images in original resolution. Other formerly specialized hardware like highbrightness monitors and high-resolution graphics adapters, partly equipped with dual heads, is entering the consumer market and becomes, therefore, more and more affordable. Greyscale consistency by calibrating the output of a display system is also already available to consumer systems, either completely implemented in hardware or partly in software. Another consideration in this context is to create a mobile test system moving around from practice to practice. In conclusion, the first intermediate results of this quality initiative are promising. A few months after the trial period was started, the hospital physicians in Bavaria
WORKSTATION PROTOTYPE FOR SOFTCOPY READING
joined the recertification program. As of 2003, the test is mandatory for all mammography-reading physicians in Bavaria. The introduction of a corresponding federal German recertification program is intended and actually required by legislation until the end of this year. Whether our mammography workstation is adopted or whether digital images are used at all—instead of conventional X-ray films—is not clear yet, though. However, the equivalence of using digitized images on a digital mammography workstation and conventional film on a light box as part of the federal recertification program has been defined. Publications from other research groups—mainly from mammography screening projects—also indicate “that a dedicated reading station allows soft-copy reading of digitized mammograms without loss of quality” (12). However, they restrict their conclusion by requiring “that CAD is available for microcalcification detection.” This requirement is certainly sensible in the screening or diagnostic reading context but not within a recertification program where the professional qualification of the participants is to be tested. Other groups propose to pre-process digital mammograms “for improving their visualization on softcopy display systems” (13). As for the workstation developed during this project, it is intended to further enhance the software according to the experiences gained so far. We already made a development plan for a “third generation” of the workstation software. Furthermore, we intend to release the software as Open Source and, therefore, allow everyone to validate the test system in detail. REFERENCES 1. American College of Radiology (ACR). Breast Imaging Reporting and Data System (BI-RADS), 3rd ed, http://www.acr.org/departments/ stand_accred/birads/, 1998, Accessed January 9, 2004 2. Hemminger BM.Softcopy Display Requirements for Digital Mammography. In: Mun SK, ed. Medical Imaging 2002: Visualization, Image-Guided Procedures, and Display. Proceedings of SPIE 2002; 4681:366 –379 3. Evertsz CJG, Bo¨ dicker A, Brady M, Hendriks J, Ju¨ rgens H, Karssemeijer N, et al. Soft-Copy Reading Environment for Screening Mammography—SCREEN. In: Yaffe MJ, ed. 5th International Workshop on Digital Mammography, Toronto, Ontario, June 2000. Madison, WI: Medical Physics Publishing, 2001, 566 –572 4. Kassena¨ rztliche Vereinigung Bayerns. KVB stellt digitale Befundungsstation vor. Press Release, October 2001, https://www.kvb.de/ servlet/PB/menu/1024330/ 5. NEMA Standards Publication PS 3.x. Digital Imaging and Communications in Medicine (DICOM). Rosslyn, VA: National Electrical Manufacturers Association, 2003. 6. Kassena¨ rztliche Vereinigung Bayerns, Mammographie—Neue Standards in der Qualita¨ t setzen, Digitale Befundungsstation der KVB. Press Release, March 2002, https://www.kvb.de/servlet/PB/menu/1036000/ 7. Kieschke J, Jensch P. Studie zur Beurteilung der Verwendbarkeit digitalisierter konventioneller Aufnahmen bei der Rezertifizierung der Befundung von Mammographien. Technical Report, OFFIS, 2002
417
RIESMEIER ET AL
8. Eichelberg M, Riesmeier J, Kleber K, Holstein J, Oosterwijk HJ, Jensch P. Consistency of Softcopy and Hardcopy: Preliminary Experiences with the new DICOM Extensions for Image Display. In: Blaine GJ, Siegel EL, eds. PACS Design and Evaluation: Engineering and Clinical Issues. Proceedings of SPIE 2000;3980:57– 67. 9. Riesmeier J, Eichelberg M, Kleber K, Gro¨ nemeyer DHW, Oosterwijk HJ, Jensch P. DICOM image display consistency: a test environment. In: Siegel EL, Huang HK, eds. Medical Imaging 2001: PACS and Integrated Medical Information Systems: Design and Evaluation. Proceedings of SPIE 2001;4323:47–56. 10. Deutsches Institut fu¨ r Normung (DIN). Sicherung der Bildqualita¨ t in ro¨ ntgendiagnostischen Betrieben—Teil 57: Abnahmepru¨ fung an Bildwiedergabegera¨ ten. DIN V 6868-57 (2001).
418
Academic Radiology, Vol 11, No 4, April 2004
11. Samei E, Badano A, Chakraborty D, Compton K, Cornelius C, Corrigan K, et al. Assessment of Display Performance for Medical Imaging Systems. Draft Report of the American Association of Physicists in Medicine (AAPM) Task Group 18, Version 9.0, October 2002. 12. Roelofs T, van Woudenberg S, Hendricks J, Bo¨ dicker A, Evertsz C, Karssemeijer M. Performance Evaluation of a Digital Reading Station for Screening Mammography. In: Peitgen HO, ed. 6th International Workshop on Digital Mammography, June 22–25, 2002, Bremen Germany. Berlin: Springer-Verlag, 2003; 455– 459. 13. Fan J, Dallas WJ, Roehrig H, Krupinski EA.Improving visualization of digital mammograms on the CRT display system. In: Galloway RL, ed. Medical Imaging 2003: Visualization, Image-Guided Procedures, and Display. Proceedings of SPIE 2003;5029:746 –753.