- Email: [email protected]
Three-dimensional reconstruction from serial sections in PC-Windows platform by using 3D_Viewer
Computer Methods and Programs in Biomedicine (2004) 76, 143—154
Three-dimensional reconstruction from serial sections in PC-Windows platform by using 3D---Viewer Yi-Hua Xu, Garet Lahvis, Harlene Edwards, Henry C. Pitot* McArdle Laboratory for Cancer Research, University of Wisconsin-Madison, 1400 University Avenue, Madison, WI 53706-1599, USA Received 17 April 2004; accepted 17 April 2004
KEYWORDS Three-dimensional reconstruction; PC Windows; 3D Viewer; Image align; Image transform-shift-overlay technique; Movie-frame-image; MS Visual Basic
Summary Three-dimensional (3D) reconstruction from serial sections allows identification of objects of interest in 3D and clarifies the relationship among these objects. 3D Viewer, developed in our laboratory for this purpose, has four major functions: image alignment, movie frame production, movie viewing, and shift-overlay image generation. Color images captured from serial sections were aligned; then the contours of objects of interest were highlighted in a semi-automatic manner. These 2D images were then automatically stacked at different viewing angles, and their composite images on a projected plane were recorded by an image transformshift-overlay technique. These composition images are used in the object-rotation movie show. The design considerations of the program and the procedures used for 3D reconstruction from serial sections are described. This program, with a digital image-capture system, a semi-automatic contours highlight method, and an automatic image transform-shift-overlay technique, greatly speeds up the reconstruction process. Since images generated by 3D Viewer are in a general graphic format, data sharing with others is easy. 3D Viewer is written in MS Visual Basic 6, obtainable from our laboratory on request. © 2004 Elsevier Ireland Ltd. All rights reserved.
1. Introduction and background The three-dimensional (3D) reconstruction of tissue structures from serial sections has always been, at least in theory, a useful tool to expand the ‘‘lost’’ dimension of microscopy in order to appreciate the structures of tissues, cells, and subcellular components. Prior to the advent of more modern technologies, the methods of three-dimensional reconstruction were somewhat cumbersome and
*Corresponding author. Tel.: +1 608 262 3247. E-mail address: [email protected] (H.C. Pitot).
certainly time consuming. In recent years, with the use of computers and various methods for converting images into informational content, the technique of three-dimensional reconstruction has found application in histology, histopathology, and clinical radiology [1—6]. Some two decades ago, 3D reconstruction was based on the use of a digitizer upon which an individual section was projected and the outlines of the objects traced manually for data input [7]. More recently, the technology for 3D reconstructions has developed rapidly. A number of commercial 3D reconstruction applications are available in SGI, Sun, and HP platforms [8]. Most of these are relatively
0169-2607/$ — see front matter © 2004 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.cmpb.2004.04.002
144
Y.-H. Xu et al.
Fig. 1 Hardware configuration used to obtain a color digital image. The following equipment is used: (1) an Olympus SZH zoom stereo microscope; (2) an OLY-750 3-CCD color digital camera; both are from Olympus American Inc.; (3) a Navitar video Zoom 7000 lens from Navitar Inc.; (4) a FlashBus video frame grabber card from Integral technologies Inc.; (5) a desktop PC (Dell Precision 410 workstation). See text for details.
expensive. In contrast, software for 3D reconstruction with platforms readily applicable to PC’s are rare. 3D reconstruction that could be generated on a PC-Windows platform would be useful to research laboratories, since PC’s are widely used, are of low cost, and can run a variety of graphic software packages. In order to take advantage of this less expensive system, we have developed an application software, 3D Viewer. This paper describes the development and availability of this software as well as methods used to obtain 3D reconstruction from serial sections on the PC-Windows platform.
This application software was designed to work cooperatively with Adobe Photoshop, a commercial graphic software. 3D Viewer supplies four basic functions that are essential for 3D reconstruction, while the other graphic functions can be found in Adobe Photoshop. These four functions are: (a) image aligning; (b) movie-frame-image generation; (c) movie viewing; and (d) shift-overlay image generation. Among these functions, the movie-frame-image generation and movie viewing are the most important. In 3D Viewer, each of these functions forms a sub program.
2. Hardware configuration
3.1. Image transform-shift-overlay technique and movie-frame-imagegeneration sub program
A color image capture system is required to introduce color microscopic images into the PC computer in a digital graphic file format. Digital images of serial sections are obtained by an Olympus OLY-750 CCD camera and a PC with a frame grabber card through a microscope (high magnification) or a Navitar Zoom 7000 lens (low magnification). The hardware configuration is shown in Fig. 1. Management of color information and 3D reconstruction requires the computer to multi-task several programs, including 3D Viewer and Adobe Photoshop. To keep these software programs running smoothly, an IBM-compatible PC requires a minimum speed of 450 MHz and substantial memory (256 MB or more). In this laboratory, we currently use a Dell Dimension 4400 computer with Pentium 4 microprocessor (2.0 GHz) and 1000 MB RAM, which has a Windows XP professional operating system.
3. Software development An application program, 3D Viewer, was developed in this laboratory to perform 3D reconstructions.
The movie-frame-image-generation sub program is designed to make a set of movie-frame-images from different view angles. Each movie-frame-image is created by overlaying all 2D images from serial sections in cutting sequence in a view angle. A technique termed image transform-shift-overlay is used. Fig. 2 shows the method to make the movie-frame-images from a view angle. 2D images from the serial sections are stacked in sequence (with the distance D corresponding to the cutting distance) to be assembled as a tissue block in which the interested object is located. When this image stack (object) is rotated at the front of the projection plane (PP), the projected image formed on PP is changed from time to time. During the rotation, each image forms at a different time point in sequence. They are the movie-frame-images of the object rotation show. When this image stack is perpendicular to the viewer (Fig. 2A), the resultant projected image of this image stack on the projection plane (PP) is simply an overlay of all 2D images. This image can be easily obtained by drawing the 2D image over and over onto the destination
3D reconstruction from serial sections in PC-Windows platform by using 3D Viewer
Fig. 2 The image transform-shift-overlay technique utilized to make the movie-frame-image from a stack of 2D images. Upper: Bird’s eye view showing the relationship of a stack of 2D images, a project plane (PP), and the human eyes which look at the project plane. 2D images from the serial sections are stacked together in cutting sequence with a distance D corresponding to the distance between slides. Thus, this set of 2D stacked images was re-assembled back into a tissue block. This tissue block, where the object of interest is located, is rotated at the front of a project plane (PP). At each time point during the rotation, there is a projected image formed on the project plane. Each projected image actually is one frame image in a rotation movie series of this object. As shown in this figure, this projected image is the combination (overlay) of each individual 2D image. (A) 2D stack images are perpendicular to the viewer. (B) 2D stacked images rotated to an angle, ␣. Bottom: an example of making projected images on the PP when 2D stacked images are rotated at different positions. To simplify the situation, this set of 2D images has only five images. There is a circle in image 1 (small, in black) and image 5 (large, in gray) respectively, and the images 2—4 are blank. (C) Projected image of 2D stacked images formed on PP when the rotation angle is zero. As shown in the figure, the width of the 2D stacked images is not changed and the projected location of image 1 (X1 , Y1 , top left corner) is the same as the projected location of image 5 (X5 , Y5 , top left corner). The projected image is formed by overlaying all images in the stack at the same location with their original sizes. (D) Projected image of 2D stacked images formed on PP when the rotation angle is ␣. Each image in the stack transforms when it is projected onto PP. It’s width becomes short (new width = original width * cos ␣). Please note the following facts: (1) where the projected images of image 1 and 5 are transformed, two round circles change into oval shape due to the transformation; (2) the projected locations of these 2D images on the PP (X1 , Y1 ; X5 , Y5 ; top left corners of image 1 and 5) are different; (3) the projected image of image 5 is on
145
PictureBox in stack sequence at the same location as shown on the left side of the figure (Fig. 2C). Rotation of this stack of 2D images by an angle (α), as shown on the right side of the figure (Fig. 2B), results in a change of the composite image on the projection plane. Although the resulting image is still an overlay of all 2D images, there are two differences. The width of each projected 2D image is shortened, the overlay becomes incomplete, and each image is located in a different place (Fig. 2D). To make this composite image, each 2D image is first compressed (transformed) and then drawn into a PictureBox at different locations (shift-overlay). If the rotation angle is known, the new width of the images of the serial section can be calculated. The locations of the images 1—5 (X1 , Y1 , . . . , X5 , Y5 ) can be calculated as well. Thus, it is possible to make the movie-frame-image in any given view angle. One problem may occur: the image previously drawn into the PictureBox will be covered by the next image owing to the opaque nature of the image. In order to show the objects of interest on the images underneath, the drawing mode should be set to transparent. This can be done in Visual Basic by using the Draw method of ImageList control and setting its DrawMode to transparent and MaskColor to RGB(255,255,255) [9]. The user interface of the movie-frame-image generation program is shown on Fig. 3. On the left side of the window, there is a control panel to collect the information needed to make the movie-frame-images. Several TextBoxes are provided for the user to enter the rotation angle range, the number of frames, the distance of the slides, scale information, and the destination directory for the resulting movie-frame-image files. A PictureBox is used for the operator to pick up the background color of the movie-frame-image. Two Option controls are provided for the user to select the rotation directions. When the operator clicks the Load Images button, a set of 2D image files can be selected by using the multiple selection method; an array of Image controls is used in the program to store these images. From the information entered by the operator, the rotation
the top of image 1. If one wishes to make the projected image in different rotation angles, the original 2D image should be compressed in width (transformed) first and then be drawn onto a destination PictureBox, in sequence one by one, at specific locations (shift-overlay). The degree of transformation (new width of the image) and the location of each 2D image can be calculated when the rotation angle ␣ is known. See text for further details.
146
Y.-H. Xu et al.
Fig. 3 The user interface of the movie-frame-image generation program. On the left side is a control panel for the user to enter the information needed to build the movie-frame-images. The user should enter the scale information of the 2D images and the distance between the slides, select the background color, set the rotation angle range, rotation direction, and the number of the movie frames. All of these parameters are needed to generate the movie-frame-images. The user can then load all of the aligned and highlighted 2D images into the program by using the multiple selection method. By clicking the Make Rotation Image. . . button, the user can get a set of movie-frame-images and a scale information file in the directory as assigned. The picture above shows one movie-frame-image made and displayed inside the PictureBox. Here, the background of the movie-frame-image that the user selected is white. However, the background can be set to any color, as shown in Fig. 5.
angle for each frame is determined, and the new width/height and starting point of each image is calculated. At this time point, each image stored in the Image control array is transferred to a temporary PictureBox where it is transformed (compressed). The Windows API function, StretchBlt, is used for this purpose [10]. This compressed image is then stored into an ImageList control and is then drawn into a destination PictureBox at the location Xn , Yn (where n is the number identification (ID) of the stacked 2D images). Transparent mode is used in this drawing process. The default transparent color is set to white, RGB(255,255,255). This process is repeated until all 2D images from the serial section are drawn into the destination PictureBox. One movie-frame-image is made and shown on the PictureBox in the center of the window. It is then stored as the BMP graphic file in the
directory the user selected. The computer then starts to produce the next movie-frame-image automatically. When all the movie-frame-image files are made, the computer will collect all scale information and store it in a file named 3D Result.dot in the same directory as the one where the movie-frame-images are found. This information file will be used in the movie viewer program. When the images from the serial section are flipped and stacked in a reversed sequence, the resultant movie-frame-image is the rear view of this stack (Fig. 4). Using this technique, we can obtain two sets of movie-frame-images, one in a front view and one in a rear view, which can be used in the 3D reconstruction viewer sub-program. Flip image is possible in Visual Basic by using the PaintPicture method of the ImageList control
3D reconstruction from serial sections in PC-Windows platform by using 3D Viewer
147
Fig. 4 Relationship of stack sequence and point of view. Different points of view of the 2D stacked images can be obtained by changing the stack sequence and flipping the images. (A) Front view; (B) rear view. Please note that the stack sequence is reversed and the 2D image flipped in comparing the two sets.
with a negative value for the destination image width [9].
3.2. 3D reconstruction viewer sub-program The 3D reconstruction viewer sub-program is designed to run the movie as well as display a single frame image. Fig. 5 shows the user interface of the 3D reconstruction viewer program. Motion picture effects are possible in Visual Basic programming by displaying a set of movie-frame-images one by one in sequence in a PictureBox. On the left side are several control panels. One is used to display information, and the others are used to control the movie show and image display. When the user clicks the Load Images button, a set of movie-frame-image files is loaded into an array of Image controls, which is used in the program to store these images. When the user clicks the Make Movie button, the movie show begins. The images stored in the array of Image controls are then transferred to the display PictureBox in sequence at a certain time interval. The speed of the motion picture show is controlled by a Slider control which changes the length of interval of a Timer control. Two play mode Option controls (one way, two way) and two view point Option controls (front view or rear view) are provided for the operator to choose the movie play mode and the view point mode. When the user clicks the stop button, the movie is stopped and the main PictureBox displays a single movie-frame-image. A display image Slider control is used to control which single movie-frame-image is displayed. This software is written with MS Visual Basic 6, professional edition (Microsoft Corporation) [9] and
can be obtained from our laboratory upon request [11].
4. Principal procedures in 3D reconstruction 4.1. Production and staining of serial section 4.1.1. Rat liver sample Altered hepatic foci (AHF) were induced in rat liver in vivo by using a standard protocol with diethylnitrosamine (DEN) as an initiator at a dose of 10 mg/kg body weight (after mitotic stimulation induced by partial hepatectomy); phenobarbital (PB) was the promoting agent, 500 ppm in drinking water for 4 months [12]. After the rat was sacrificed, a liver tissue sample was fixed and dehydrated in various concentrations of ethanol followed by xylene, and then embedded in low-melting paraffin (58 ◦ C). This tissue block was serially sectioned at 5 m and treated with an antibody specific for the glutathione S-transferase (GSTP) protein by methods previously described [13]. 4.1.2. Mouse liver sample A C57 Bl/6 mouse was sacrificed at approximately 24 h of age. The liver was preserved in 10% buffered formalin and embedded in paraffin to form a tissue block. To produce two markers for alignment prior to section, we inserted two decalcified fish bones into the paraffin portion of the tissue block perpendicular to the cutting plane. These two markers allowed alignment of each of the sections used in the development of the image. Serial sections were cut at 5 m thickness and were stained
148
Y.-H. Xu et al.
Fig. 5 The user interface of the 3D reconstruction viewer program. The figure depicts the movie show of a computer simulation sphere reconstruction sample. The information panel and a 1 cm scale are designed to show the rotation and scale information. The movie panel is used to control the movie show. After a set of movie-frame-images and a scale information file are loaded into the program, the user can click the Make Movie button to start the movie show and use the slider to control the speed of the show. View point can be changed by selecting different options in the View From panel. The background of this movie-frame-image is black. Actually, the user can select any background color preferred in the stage of movie-frame-image generation. The single frame image show is made possible by clicking the Stop button. At this time, the movie show is stopped, and the display window shows a single movie-frame-image. The user may use the slider inside the Display Image panel to control which single movie-frame-image is displayed.
with hematoxylin and eosin. Every 20th section was used to develop the mouse liver surface 3D reconstruction.
4.2. Image capture Biological samples (microscopic) were observed with an Olympus SZH stereo zoom microscope or a Zoom 7000 Navitar lens (macroscopic), based on the sample size of interest. The largest slide from the serial sections was chosen to set up the magnification factor. Color images of the slides were captured by an OLY-750 digital color camera and a FlashBus frame grabber card. They were imported into Adobe Photoshop by Twain interface. Each of the 2D images of the slide was then adjusted by using the Auto contrast and Auto Level commands in the Image menu and was then stored as JPG files.
4.3. Image alignment Accurate image alignment can be achieved in 3D Viewer by using the image align utility if the alignment markers are established during the tissue preparation [14—16]. In Adobe Photoshop, select the image which has similar orientation as the control image and copy it into the clipboard. From the main menu of the 3D Viewer, select the Align image command in the Utility menu. The image align utility window is now shown on the screen. Fig. 6 shows the detailed procedures involved in the alignment process. The key point of image alignment is to know the angle (orientation) difference of two images and the rotation center. This is achieved by clicking on the alignment markers A and B on the image. The mouse cursor location can be registered with the event mouse down, thereby allowing the orientation angles (the angle between line AB and
3D reconstruction from serial sections in PC-Windows platform by using 3D Viewer
149
Fig. 6 Image alignment by using 3D Viewer image alignment utility. (A) Two image files overlaid before alignment. (B) The first image was set as the control image. It was loaded into the program, and the locations of alignment markers A and B were recorded. This information can be saved into files and retrieved for later use. (C) The second image, the image to be aligned, was loaded into the program, and the locations of alignment markers A and B were recorded. (D) The rotation of the second image was obtained by clicking the Rotate button since the computer can calculate the rotation angle based on the location of these alignment makers. A message box was shown on the screen to prompt the user to pick up the option (rotation-shift, rotation-only) preferred. Note that the rotated image size was enlarged to avoid truncation. The resultant image is stored inside the clipboard and can be pasted in Adobe Photoshop to form a new (aligned) file. (E) Two image files overlaid after rotate-shift option was picked. This option further shifted the rotated second image and cropped it to match with the control image (image 1); thus, the overlay was perfect. (F) Two image files overlaid after the rotate-only option was picked. This option kept the rotated image intact. Note that the orientations of these two images were the same now, but the locations were different. The user could get a perfect overlay by storing different images into different layers in an Adobe Photoshop document file and moving the upper layer of the image thereafter. In certain conditions, the rotation-only option is necessary to avoid data truncation (see discussion in text for details). The alignment process for each image can be finished within 1 min.
150 the horizon line) to be calculated. The orientation angle of the image to be rotated can be determined also. This second image then rotates into the same orientation as the control image with alignment at the anatomic maker A. This rotate-shifted image is then copy-pasted onto an Adobe Photoshop document (PSD) file to form a new layer. This process can be done within one minute. After image alignment, multiple image files are converted into multiple layers in a PSD file. The layer sequence must be the same as the cut section sequence. The alignment markers for the control image are entered only once. The user can continue pasting and aligning images. This information can be stored into a file for subsequent retrieval. Images without alignment markers may be roughly aligned by using Photoshop’s built-in functions, but the result is much less satisfactory.
4.4. Highlighting of interested objects For highlighting the area of interest and for clearing the remainder of the image, Adobe Photoshop offers a number of powerful tools. The outline of
Y.-H. Xu et al. the object can be highlighted by selecting the object and performing the Stroke command in the Edit menu. Objects with specific color can be selected by using the Color Range command in the Select menu and can then be highlighted by filling with any color. Fig. 7 depicts the results of some techniques used to highlight the rat liver tissue section and preneoplastic lesions therein. These procedures each take about one minute. After highlighting, one 2D image file becomes a layer inside a Photoshop document file. When all the objects of interest on every layer of the Adobe Photoshop document file are highlighted, each layer is converted to a GIF file by using Save from the commands in the File menu. These files are named according to the cutting sequence (layer number). Since these files are from the single Photoshop document file, they are all of the same size (graphic size, pixel X—Y dimension).
4.5. Making of a movie frame This process can be finished in 3D Viewer by using the movie-frame-image generation program.
Fig. 7 Techniques used to highlight objects of interest in Adobe Photoshop. (A) An image captured from a rat liver section stained by the GST marker. Focal lesions can be seen on the image that show a brownish color, while the tissue background is a shade of blue. The image background color is white. These color differences can be used to draw the tissue outline and focal lesion easily. (B) Liver tissue is selected by using the magic wand tool to select the white area and then using the Inverse command in the Selection menu. A blank layer is added onto this image. The selection is then outlined by the blue color by using the Stroke command in the Edit menu. (C) Focal lesions are selected by using the Color Range in Selection menu and are then filled with red color onto the new added layer. (D) After the edit process (add missing foci and erase background), the image underneath is removed, leaving only the added layer. Thus, the tissue outline and focal lesions are highlighted and other areas are all set to the white color.
3D reconstruction from serial sections in PC-Windows platform by using 3D Viewer As shown in Fig. 3, aligned and highlighted 2D images from the serial sections are loaded by clicking the Load Images button. Scale information and the cutting distance should be entered first. Options for movie-frame-image generation include: background color, range of the rotation angle, rotation direction, frame number. By depressing the Make Rotation Image. . . button, the program starts the process of drawing the movie-frame-image at a different angle one by one, using the image transform and image-overlay techniques as described. A set of movie-frame-images files (BMP files) is generated with scale information and stored in a selectable destination directory. Fig. 3 shows the rotation frame maker program window when one movie-frame-image is completed. These BMP formatted movie-frame-image files can be converted into JPG format graph files in Photoshop to save storage disk space.
151
4.6. Data presentation 4.6.1. Movie viewing and single-frame image display To run the movie show, the operator should click the Load Images button. Movie-frame-images generated previously are loaded into the computer with scale information. A 1 cm scale is presented on the screen to provide a size impression of the 3D structure of the object. By clicking the Make Movie button inside the Movie control panel, the movie begins (Fig. 5). The Slider controls the movie show’s speed. By selecting front or rear view mode, the user can see opposing views of the show. The playing sequence is controlled by the play mode (one way and two way). The Stop button inside the Movie control panel stops the movie. The Slider inside the Display Image control panel can be used to display a single frame image in the main PictureBox.
Fig. 8 A shift-overlay image plot of a mouse liver sample. This type of image can be generated by 3D Viewer by using the image shift-overlay plot utility. It is made by overlaying the 2D images from the serial sections. The user can select the start image and stop image and adjust the shift distance. Thus, a certain portion of the object can be viewed in detail.
152 4.6.2. Image printing and shift-overlay image generation To this point, a number of images are generated during the 3D reconstruction process: (1) all the original 2D images captured from the serial sections are stored in JPG format, although some of them may be in TIF, BMP format. (2) The aligned and highlighted images are stored in GIF format. (3) The generated movie-frame-images are in BMP or JPG format. These image formats are popular and therefore can be viewed or printed in many graphic applications. 3D Viewer offers another type of image, the shift-overlay image, which is the shift-overlay of the original 2D aligned and highlighted images from the serial sections. The background color of the resulting image, the shift distance, as well as the starting and ending images, can be controlled by the operator. As a result, a BMP file is generated, as shown in Fig. 8. This method offers a way to see detail in some portions of 3D reconstruction.
5. Results and discussion With a 3D Viewer and the help of Adobe Photoshop, 3D reconstructions can be performed in a PC-Windows platform. The system described here has four features: (1) images of the serial section are captured by a digital camera and then used directly; thus, many processes previously required, such as film development and printing, have been eliminated owing to this improvement in the hardware. (2) Accurate image alignment can be achieved in the 3D Viewer by using the image align utility, where the alignment markers are established during the tissue preparation. (3) Tedious tracing of the object’s outline with a digitizer to obtain the contour data is replaced by the highlighting of interested objects, a technique that can be performed in a semi-automatic manner. (4) The earlier time-consuming point and contour data registration in the Z-axis is replaced by the image transform-shift-overlay technique, which is fully automatic. 3D Viewer provides the impression of the three dimensions of the structure and size in four ways. (1) 2D image stacking show. During the movie-frame-image generation process, stacking of successive aligned 2D images one by one to form a movie-frame-image can be viewed in the PictureBox at different view angles. (2) 3D movie show. The user may load a set of movie-frame-images into the 3D reconstruction viewer program and observe the movie show in RGB color from two points of view (front and rear). (3) Single
Y.-H. Xu et al. movie-frame-image show. When the movie stops, each single movie-frame-image can be displayed by using a slider control. (4) Shift-overlay image show. An image of any portion of the 3D structure can be generated and shown. Recent advances in PC technology allowed successful development of the 3D Viewer. For instance, the image transform-shift-overlay technique is the core of the 3D Viewer program. Image transform is a complicated process in programming, involving manipulation of pixel color data one by one. This can now be simply performed in Visual Basic, a Windows application developing tool, by using ImageList control and some of Windows Application Programming Interface (API) functions [10]. Since these API functions actually are functions of the Windows operating system, they are executed very rapidly. The total 3D reconstruction process is greatly simplified and saves time. This system works on the PC-Windows platform and appears to have a number of advantages. Since PCs with considerable memory and hard disk space are reasonably priced and the software required is readily available, it is a relatively inexpensive system. PC hardware can be easily upgraded when needed. Most of the potential users already have the knowledge to use the PC and some of the graphic software such as Photoshop, and thus do not need special training. Since 3D Viewer can be obtained from our laboratory upon request, expensive software may not be necessary. Meanwhile, 3D reconstruction data obtained with such instrumentation can readily be shared with others via image files. Since the image files may be compressed, the cost of hard disk and other storage media is subsequently decreased as well. In order to accomplish 3D reconstruction efficiently, we have found that the following are important factors: 1. During the image capture process, the largest slide to be captured should be used to set the magnification factor in order to avoid truncation. 2. For accurate alignment of the 2D images, alignment markers should be inserted into the tissue block outside the tissue itself before sectioning. Anatomic markers, such as blood vessels, are not recommended because they are not perpendicular to the cutting surface and tend to be curved. This is especially important when the tissue block to be observed is thick. One choice is to drill holes with a mechanical drill; however, this technique must make holes in the tissue itself, thus destroying tissue in its path. The mechanical drill bit usually makes a larger hole than
3D reconstruction from serial sections in PC-Windows platform by using 3D Viewer the drill’s diameter owing to the vibration. Recently some reports showed that a laser beam can be used to make holes [14]; however, this may require expensive equipment. Insertion of other material into the paraffin portion of the tissue block is another choice [15,16]. A range of materials has been tested in our laboratory, such as surgical suture thread, cactus spines, fishing string, and fish bone. Among them, we found that fish bone is the best. Obviously, the alignment marker should be placed very close to the tissue in order to allow the largest magnification. 3. The image alignment is made possible by a rotation-shift process. Sometimes this process may cause data loss. For example, if the objects of interest occupy the full image before rotation, some portion of the objects of interest will be moved outside the image margin after rotation. Thus, the data in this portion are lost. In anticipation of such data loss, we designed, in the image align utility, two options for the operator. The operator may use the shift-rotation or the rotation-only function. Use of the rotation-only function results in a simple rotation of the image. The resultant image is somewhat larger than the original image without any truncation. The new image can be copied onto a new layer of a Photoshop document file, which serves to collect aligned image files. This layer can be shifted to the right location by using the Move tool in Adobe Photoshop because, when the blending mode is set to darken in the layer window, the underlying image (here the previous file’s aligned image) can be seen through. The image size can be increased later by using the Canvas size command in the Image menu. With this technique, the data loss can be avoided. 4. The object highlight process at first seems to be an artifact in the 3D reconstruction. However, the purpose of this process is to emphasize the object of interest and to clear the remainder of the image. The latter is more important than the former. The user may keep the original color of the object if desired; however, the rest of the image must be cleared. These cleared areas should be set to transparent in the image-overlay process. Otherwise this area will block the view path and cover the image underneath. The Object highlight process takes the most time in 3D reconstruction. Previously it was a fully manual process; now it can be done in a semi-automatic manner since Adobe Photoshop offers a number of powerful functions [17,18]. 5. The aligned, highlighted 2D image files can be stored as individual image files or as the different layers in a single Adobe Photoshop doc-
153
ument file. There is an advantage to storing these as PSD files: the image size can be enlarged as needed (such as when image objects are truncated after rotation). One change in image size causes every layer to change. These different layers in multi-layered PSD files can be de-assembled into a set of same-sized image files by using the Save for web command in the File Menu. 6. The format used for final alignment and for highlighting image files should be GIF. These files are used in a movie-frame-images generation program in which the transparent drawing technique is used. Only one RGB color [RGB(255,255,255)] can be used as the transparent color. However, white color on an image may consist of real white [RGB(255,255,255)] and some near-white colors [the color seen by naked eye as white, but not RGB(255,255,255), such as RGB(255,254,254), RGB(255,254,253)]. These near-white colors are difficult to identify and will remain opaque in the image-overlay process. However, the operator can easily delete all near-white colors from the color table in Adobe Photoshop using the Save for Web command in GIF format [19]. 7. In the movie-frame-image generation process, the number of movie frames should be at least 10 in order to run the movie smoothly. The more movie frames, the smoother the movie show, but also the longer the time required to finish the job. The actual number of the frame images to be made is doubled because a set of movie frames from the rear view is also made. The default format of the resultant frame image file is BMP. However, BMP files usually are very large and occupy a lot of disk space. These files can be converted into JPG files and decreased in size since no transparent function should be performed at this stage.
6. Future plans The objects of interest usually have a special color, which is different from the rest of the image. We plan to develop software to separate the objects with specific RGB color more efficiently and thus highlight them readily.
Acknowledgements The authors express their sincere appreciation to Mrs. Mary Jo Markham and Mrs. Kristen Adler for their expert technical typing of the manuscript and
154 to Dr. Ilse Riegel for her critical editorial comments. The development of this software was supported in part by grants from the National Cancer Institute of the United States (CA07175, CA22484, and CA45700).
References [1] 3D Reconstruction home page at http://biocomp.stanford. edu/3dreconstruction/index.html. [2] D.M. Hyde, D.J. Magliano, E. Reus, N.K. Tyler, S. Nichols, W.S. Tyler, Computer-assisted morphometry: point, intersection, and profile counting and three-dimensional reconstruction, Microsc. Res. Technol. 21 (1992) 262—270. [3] J. Aferzon, R.I. Chau, D.F. Cowan, A microcomputer based system for three-dimensional reconstructions from tomographic or histologic sections, Anal. Quant. Cytol. Histol. 13 (1991) 80—88. [4] J.W. Hoffmeister, G.C. Rinehart, M.W. Vannier, Threedimensional surface reconstructions using a general purpose image processing system, Comput. Med. Imaging Graph. 14 (1990) 35—42. [5] J. Spacek, T. Dolezal, J. Hanus, M. Hartmann, Computeraided three-dimensional reconstructions of biological objects using simple facilities, Folia Biol. (Praha) 35 (1989) 329—337. [6] N. Baba, K. Kanaya, Three-dimensional reconstruction from serial section images by computer graphics, Scanning Microsc. Suppl. 2 (1988) 303—314. [7] J.R. Salisbury, W.F. Whimster, Progress in computergenerated three-dimensional reconstruction, J. Pathol. 170 (1993) 223—227.
Y.-H. Xu et al. [8] 3D Reconstruction software homepage at http://biocomp. stanford.edu/3dreconstruction/software/index.html. [9] Microsoft Corporation, Visual Basic on Line for Visual Basic 6.0 Professional Version, CD Material. [10] The API guide homepage at http://www.allapi.net/agnet/ apiguide.php. [11] Pitot Laboratory web page at http://mcardle.oncology. wisc.edu/pitot/. [12] H.C. Pitot, L. Barsness, T. Goldsworthy, T. Kitagawa, Biochemical characterisation of stages of hepatocarcinogenesis after a single dose of diethylnitrosamine, Nature 271 (1978) 456—458. [13] S. Hendrich, H.A. Campbell, H.C. Pitot, Quantitative stereological evaluation of four histochemical markers of altered foci in multistage hepatocarcinogenesis in the rat, Carcinogenesis 8 (1987) 1245—1250. [14] G.E. Clarke, P.W. Hamilton, W.A. Montgomery, Aligning histological serial sections for three-dimensional reconstruction using an excimer laser beam, Pathol. Res. Pract. 189 (1993) 563—566. [15] K. Brandle, A new method for aligning histological serial sections for three-dimensional reconstruction, Comput. Biomed. Res. 22 (1989) 52—62. [16] M.H. Deverell, W.F. Whimster, A method of image registration for three-dimensional reconstruction of microscopic structures using an IBAS 2000 image analysis system, Pathol. Res. Pract. 185 (1989) 602—605. [17] K. Binder, T. Alsoach, Selections, Paths, and Masks, Photoshop 4 Complete, Hayden Books, Indianapolis, 1997, pp. 131—183. [18] G.D. Bouton, B.M. Bouton, Inside Adobe Photoshop 5, New Riders Publishing, Indianapolis, 1998, pp. 384— 389. [19] Optimizing Colors in GIF and PNG-8 Images, Adobe Photoshop 6 Help Content, CD Material.
Three-dimensional reconstruction from serial sections in PC-Windows platform by using 3D---Viewer Yi-Hua Xu, Garet Lahvis, Harlene Edwards, Henry C. Pitot* McArdle Laboratory for Cancer Research, University of Wisconsin-Madison, 1400 University Avenue, Madison, WI 53706-1599, USA Received 17 April 2004; accepted 17 April 2004
KEYWORDS Three-dimensional reconstruction; PC Windows; 3D Viewer; Image align; Image transform-shift-overlay technique; Movie-frame-image; MS Visual Basic
Summary Three-dimensional (3D) reconstruction from serial sections allows identification of objects of interest in 3D and clarifies the relationship among these objects. 3D Viewer, developed in our laboratory for this purpose, has four major functions: image alignment, movie frame production, movie viewing, and shift-overlay image generation. Color images captured from serial sections were aligned; then the contours of objects of interest were highlighted in a semi-automatic manner. These 2D images were then automatically stacked at different viewing angles, and their composite images on a projected plane were recorded by an image transformshift-overlay technique. These composition images are used in the object-rotation movie show. The design considerations of the program and the procedures used for 3D reconstruction from serial sections are described. This program, with a digital image-capture system, a semi-automatic contours highlight method, and an automatic image transform-shift-overlay technique, greatly speeds up the reconstruction process. Since images generated by 3D Viewer are in a general graphic format, data sharing with others is easy. 3D Viewer is written in MS Visual Basic 6, obtainable from our laboratory on request. © 2004 Elsevier Ireland Ltd. All rights reserved.
1. Introduction and background The three-dimensional (3D) reconstruction of tissue structures from serial sections has always been, at least in theory, a useful tool to expand the ‘‘lost’’ dimension of microscopy in order to appreciate the structures of tissues, cells, and subcellular components. Prior to the advent of more modern technologies, the methods of three-dimensional reconstruction were somewhat cumbersome and
*Corresponding author. Tel.: +1 608 262 3247. E-mail address: [email protected] (H.C. Pitot).
certainly time consuming. In recent years, with the use of computers and various methods for converting images into informational content, the technique of three-dimensional reconstruction has found application in histology, histopathology, and clinical radiology [1—6]. Some two decades ago, 3D reconstruction was based on the use of a digitizer upon which an individual section was projected and the outlines of the objects traced manually for data input [7]. More recently, the technology for 3D reconstructions has developed rapidly. A number of commercial 3D reconstruction applications are available in SGI, Sun, and HP platforms [8]. Most of these are relatively
0169-2607/$ — see front matter © 2004 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.cmpb.2004.04.002
144
Y.-H. Xu et al.
Fig. 1 Hardware configuration used to obtain a color digital image. The following equipment is used: (1) an Olympus SZH zoom stereo microscope; (2) an OLY-750 3-CCD color digital camera; both are from Olympus American Inc.; (3) a Navitar video Zoom 7000 lens from Navitar Inc.; (4) a FlashBus video frame grabber card from Integral technologies Inc.; (5) a desktop PC (Dell Precision 410 workstation). See text for details.
expensive. In contrast, software for 3D reconstruction with platforms readily applicable to PC’s are rare. 3D reconstruction that could be generated on a PC-Windows platform would be useful to research laboratories, since PC’s are widely used, are of low cost, and can run a variety of graphic software packages. In order to take advantage of this less expensive system, we have developed an application software, 3D Viewer. This paper describes the development and availability of this software as well as methods used to obtain 3D reconstruction from serial sections on the PC-Windows platform.
This application software was designed to work cooperatively with Adobe Photoshop, a commercial graphic software. 3D Viewer supplies four basic functions that are essential for 3D reconstruction, while the other graphic functions can be found in Adobe Photoshop. These four functions are: (a) image aligning; (b) movie-frame-image generation; (c) movie viewing; and (d) shift-overlay image generation. Among these functions, the movie-frame-image generation and movie viewing are the most important. In 3D Viewer, each of these functions forms a sub program.
2. Hardware configuration
3.1. Image transform-shift-overlay technique and movie-frame-imagegeneration sub program
A color image capture system is required to introduce color microscopic images into the PC computer in a digital graphic file format. Digital images of serial sections are obtained by an Olympus OLY-750 CCD camera and a PC with a frame grabber card through a microscope (high magnification) or a Navitar Zoom 7000 lens (low magnification). The hardware configuration is shown in Fig. 1. Management of color information and 3D reconstruction requires the computer to multi-task several programs, including 3D Viewer and Adobe Photoshop. To keep these software programs running smoothly, an IBM-compatible PC requires a minimum speed of 450 MHz and substantial memory (256 MB or more). In this laboratory, we currently use a Dell Dimension 4400 computer with Pentium 4 microprocessor (2.0 GHz) and 1000 MB RAM, which has a Windows XP professional operating system.
3. Software development An application program, 3D Viewer, was developed in this laboratory to perform 3D reconstructions.
The movie-frame-image-generation sub program is designed to make a set of movie-frame-images from different view angles. Each movie-frame-image is created by overlaying all 2D images from serial sections in cutting sequence in a view angle. A technique termed image transform-shift-overlay is used. Fig. 2 shows the method to make the movie-frame-images from a view angle. 2D images from the serial sections are stacked in sequence (with the distance D corresponding to the cutting distance) to be assembled as a tissue block in which the interested object is located. When this image stack (object) is rotated at the front of the projection plane (PP), the projected image formed on PP is changed from time to time. During the rotation, each image forms at a different time point in sequence. They are the movie-frame-images of the object rotation show. When this image stack is perpendicular to the viewer (Fig. 2A), the resultant projected image of this image stack on the projection plane (PP) is simply an overlay of all 2D images. This image can be easily obtained by drawing the 2D image over and over onto the destination
3D reconstruction from serial sections in PC-Windows platform by using 3D Viewer
Fig. 2 The image transform-shift-overlay technique utilized to make the movie-frame-image from a stack of 2D images. Upper: Bird’s eye view showing the relationship of a stack of 2D images, a project plane (PP), and the human eyes which look at the project plane. 2D images from the serial sections are stacked together in cutting sequence with a distance D corresponding to the distance between slides. Thus, this set of 2D stacked images was re-assembled back into a tissue block. This tissue block, where the object of interest is located, is rotated at the front of a project plane (PP). At each time point during the rotation, there is a projected image formed on the project plane. Each projected image actually is one frame image in a rotation movie series of this object. As shown in this figure, this projected image is the combination (overlay) of each individual 2D image. (A) 2D stack images are perpendicular to the viewer. (B) 2D stacked images rotated to an angle, ␣. Bottom: an example of making projected images on the PP when 2D stacked images are rotated at different positions. To simplify the situation, this set of 2D images has only five images. There is a circle in image 1 (small, in black) and image 5 (large, in gray) respectively, and the images 2—4 are blank. (C) Projected image of 2D stacked images formed on PP when the rotation angle is zero. As shown in the figure, the width of the 2D stacked images is not changed and the projected location of image 1 (X1 , Y1 , top left corner) is the same as the projected location of image 5 (X5 , Y5 , top left corner). The projected image is formed by overlaying all images in the stack at the same location with their original sizes. (D) Projected image of 2D stacked images formed on PP when the rotation angle is ␣. Each image in the stack transforms when it is projected onto PP. It’s width becomes short (new width = original width * cos ␣). Please note the following facts: (1) where the projected images of image 1 and 5 are transformed, two round circles change into oval shape due to the transformation; (2) the projected locations of these 2D images on the PP (X1 , Y1 ; X5 , Y5 ; top left corners of image 1 and 5) are different; (3) the projected image of image 5 is on
145
PictureBox in stack sequence at the same location as shown on the left side of the figure (Fig. 2C). Rotation of this stack of 2D images by an angle (α), as shown on the right side of the figure (Fig. 2B), results in a change of the composite image on the projection plane. Although the resulting image is still an overlay of all 2D images, there are two differences. The width of each projected 2D image is shortened, the overlay becomes incomplete, and each image is located in a different place (Fig. 2D). To make this composite image, each 2D image is first compressed (transformed) and then drawn into a PictureBox at different locations (shift-overlay). If the rotation angle is known, the new width of the images of the serial section can be calculated. The locations of the images 1—5 (X1 , Y1 , . . . , X5 , Y5 ) can be calculated as well. Thus, it is possible to make the movie-frame-image in any given view angle. One problem may occur: the image previously drawn into the PictureBox will be covered by the next image owing to the opaque nature of the image. In order to show the objects of interest on the images underneath, the drawing mode should be set to transparent. This can be done in Visual Basic by using the Draw method of ImageList control and setting its DrawMode to transparent and MaskColor to RGB(255,255,255) [9]. The user interface of the movie-frame-image generation program is shown on Fig. 3. On the left side of the window, there is a control panel to collect the information needed to make the movie-frame-images. Several TextBoxes are provided for the user to enter the rotation angle range, the number of frames, the distance of the slides, scale information, and the destination directory for the resulting movie-frame-image files. A PictureBox is used for the operator to pick up the background color of the movie-frame-image. Two Option controls are provided for the user to select the rotation directions. When the operator clicks the Load Images button, a set of 2D image files can be selected by using the multiple selection method; an array of Image controls is used in the program to store these images. From the information entered by the operator, the rotation
the top of image 1. If one wishes to make the projected image in different rotation angles, the original 2D image should be compressed in width (transformed) first and then be drawn onto a destination PictureBox, in sequence one by one, at specific locations (shift-overlay). The degree of transformation (new width of the image) and the location of each 2D image can be calculated when the rotation angle ␣ is known. See text for further details.
146
Y.-H. Xu et al.
Fig. 3 The user interface of the movie-frame-image generation program. On the left side is a control panel for the user to enter the information needed to build the movie-frame-images. The user should enter the scale information of the 2D images and the distance between the slides, select the background color, set the rotation angle range, rotation direction, and the number of the movie frames. All of these parameters are needed to generate the movie-frame-images. The user can then load all of the aligned and highlighted 2D images into the program by using the multiple selection method. By clicking the Make Rotation Image. . . button, the user can get a set of movie-frame-images and a scale information file in the directory as assigned. The picture above shows one movie-frame-image made and displayed inside the PictureBox. Here, the background of the movie-frame-image that the user selected is white. However, the background can be set to any color, as shown in Fig. 5.
angle for each frame is determined, and the new width/height and starting point of each image is calculated. At this time point, each image stored in the Image control array is transferred to a temporary PictureBox where it is transformed (compressed). The Windows API function, StretchBlt, is used for this purpose [10]. This compressed image is then stored into an ImageList control and is then drawn into a destination PictureBox at the location Xn , Yn (where n is the number identification (ID) of the stacked 2D images). Transparent mode is used in this drawing process. The default transparent color is set to white, RGB(255,255,255). This process is repeated until all 2D images from the serial section are drawn into the destination PictureBox. One movie-frame-image is made and shown on the PictureBox in the center of the window. It is then stored as the BMP graphic file in the
directory the user selected. The computer then starts to produce the next movie-frame-image automatically. When all the movie-frame-image files are made, the computer will collect all scale information and store it in a file named 3D Result.dot in the same directory as the one where the movie-frame-images are found. This information file will be used in the movie viewer program. When the images from the serial section are flipped and stacked in a reversed sequence, the resultant movie-frame-image is the rear view of this stack (Fig. 4). Using this technique, we can obtain two sets of movie-frame-images, one in a front view and one in a rear view, which can be used in the 3D reconstruction viewer sub-program. Flip image is possible in Visual Basic by using the PaintPicture method of the ImageList control
3D reconstruction from serial sections in PC-Windows platform by using 3D Viewer
147
Fig. 4 Relationship of stack sequence and point of view. Different points of view of the 2D stacked images can be obtained by changing the stack sequence and flipping the images. (A) Front view; (B) rear view. Please note that the stack sequence is reversed and the 2D image flipped in comparing the two sets.
with a negative value for the destination image width [9].
3.2. 3D reconstruction viewer sub-program The 3D reconstruction viewer sub-program is designed to run the movie as well as display a single frame image. Fig. 5 shows the user interface of the 3D reconstruction viewer program. Motion picture effects are possible in Visual Basic programming by displaying a set of movie-frame-images one by one in sequence in a PictureBox. On the left side are several control panels. One is used to display information, and the others are used to control the movie show and image display. When the user clicks the Load Images button, a set of movie-frame-image files is loaded into an array of Image controls, which is used in the program to store these images. When the user clicks the Make Movie button, the movie show begins. The images stored in the array of Image controls are then transferred to the display PictureBox in sequence at a certain time interval. The speed of the motion picture show is controlled by a Slider control which changes the length of interval of a Timer control. Two play mode Option controls (one way, two way) and two view point Option controls (front view or rear view) are provided for the operator to choose the movie play mode and the view point mode. When the user clicks the stop button, the movie is stopped and the main PictureBox displays a single movie-frame-image. A display image Slider control is used to control which single movie-frame-image is displayed. This software is written with MS Visual Basic 6, professional edition (Microsoft Corporation) [9] and
can be obtained from our laboratory upon request [11].
4. Principal procedures in 3D reconstruction 4.1. Production and staining of serial section 4.1.1. Rat liver sample Altered hepatic foci (AHF) were induced in rat liver in vivo by using a standard protocol with diethylnitrosamine (DEN) as an initiator at a dose of 10 mg/kg body weight (after mitotic stimulation induced by partial hepatectomy); phenobarbital (PB) was the promoting agent, 500 ppm in drinking water for 4 months [12]. After the rat was sacrificed, a liver tissue sample was fixed and dehydrated in various concentrations of ethanol followed by xylene, and then embedded in low-melting paraffin (58 ◦ C). This tissue block was serially sectioned at 5 m and treated with an antibody specific for the glutathione S-transferase (GSTP) protein by methods previously described [13]. 4.1.2. Mouse liver sample A C57 Bl/6 mouse was sacrificed at approximately 24 h of age. The liver was preserved in 10% buffered formalin and embedded in paraffin to form a tissue block. To produce two markers for alignment prior to section, we inserted two decalcified fish bones into the paraffin portion of the tissue block perpendicular to the cutting plane. These two markers allowed alignment of each of the sections used in the development of the image. Serial sections were cut at 5 m thickness and were stained
148
Y.-H. Xu et al.
Fig. 5 The user interface of the 3D reconstruction viewer program. The figure depicts the movie show of a computer simulation sphere reconstruction sample. The information panel and a 1 cm scale are designed to show the rotation and scale information. The movie panel is used to control the movie show. After a set of movie-frame-images and a scale information file are loaded into the program, the user can click the Make Movie button to start the movie show and use the slider to control the speed of the show. View point can be changed by selecting different options in the View From panel. The background of this movie-frame-image is black. Actually, the user can select any background color preferred in the stage of movie-frame-image generation. The single frame image show is made possible by clicking the Stop button. At this time, the movie show is stopped, and the display window shows a single movie-frame-image. The user may use the slider inside the Display Image panel to control which single movie-frame-image is displayed.
with hematoxylin and eosin. Every 20th section was used to develop the mouse liver surface 3D reconstruction.
4.2. Image capture Biological samples (microscopic) were observed with an Olympus SZH stereo zoom microscope or a Zoom 7000 Navitar lens (macroscopic), based on the sample size of interest. The largest slide from the serial sections was chosen to set up the magnification factor. Color images of the slides were captured by an OLY-750 digital color camera and a FlashBus frame grabber card. They were imported into Adobe Photoshop by Twain interface. Each of the 2D images of the slide was then adjusted by using the Auto contrast and Auto Level commands in the Image menu and was then stored as JPG files.
4.3. Image alignment Accurate image alignment can be achieved in 3D Viewer by using the image align utility if the alignment markers are established during the tissue preparation [14—16]. In Adobe Photoshop, select the image which has similar orientation as the control image and copy it into the clipboard. From the main menu of the 3D Viewer, select the Align image command in the Utility menu. The image align utility window is now shown on the screen. Fig. 6 shows the detailed procedures involved in the alignment process. The key point of image alignment is to know the angle (orientation) difference of two images and the rotation center. This is achieved by clicking on the alignment markers A and B on the image. The mouse cursor location can be registered with the event mouse down, thereby allowing the orientation angles (the angle between line AB and
3D reconstruction from serial sections in PC-Windows platform by using 3D Viewer
149
Fig. 6 Image alignment by using 3D Viewer image alignment utility. (A) Two image files overlaid before alignment. (B) The first image was set as the control image. It was loaded into the program, and the locations of alignment markers A and B were recorded. This information can be saved into files and retrieved for later use. (C) The second image, the image to be aligned, was loaded into the program, and the locations of alignment markers A and B were recorded. (D) The rotation of the second image was obtained by clicking the Rotate button since the computer can calculate the rotation angle based on the location of these alignment makers. A message box was shown on the screen to prompt the user to pick up the option (rotation-shift, rotation-only) preferred. Note that the rotated image size was enlarged to avoid truncation. The resultant image is stored inside the clipboard and can be pasted in Adobe Photoshop to form a new (aligned) file. (E) Two image files overlaid after rotate-shift option was picked. This option further shifted the rotated second image and cropped it to match with the control image (image 1); thus, the overlay was perfect. (F) Two image files overlaid after the rotate-only option was picked. This option kept the rotated image intact. Note that the orientations of these two images were the same now, but the locations were different. The user could get a perfect overlay by storing different images into different layers in an Adobe Photoshop document file and moving the upper layer of the image thereafter. In certain conditions, the rotation-only option is necessary to avoid data truncation (see discussion in text for details). The alignment process for each image can be finished within 1 min.
150 the horizon line) to be calculated. The orientation angle of the image to be rotated can be determined also. This second image then rotates into the same orientation as the control image with alignment at the anatomic maker A. This rotate-shifted image is then copy-pasted onto an Adobe Photoshop document (PSD) file to form a new layer. This process can be done within one minute. After image alignment, multiple image files are converted into multiple layers in a PSD file. The layer sequence must be the same as the cut section sequence. The alignment markers for the control image are entered only once. The user can continue pasting and aligning images. This information can be stored into a file for subsequent retrieval. Images without alignment markers may be roughly aligned by using Photoshop’s built-in functions, but the result is much less satisfactory.
4.4. Highlighting of interested objects For highlighting the area of interest and for clearing the remainder of the image, Adobe Photoshop offers a number of powerful tools. The outline of
Y.-H. Xu et al. the object can be highlighted by selecting the object and performing the Stroke command in the Edit menu. Objects with specific color can be selected by using the Color Range command in the Select menu and can then be highlighted by filling with any color. Fig. 7 depicts the results of some techniques used to highlight the rat liver tissue section and preneoplastic lesions therein. These procedures each take about one minute. After highlighting, one 2D image file becomes a layer inside a Photoshop document file. When all the objects of interest on every layer of the Adobe Photoshop document file are highlighted, each layer is converted to a GIF file by using Save from the commands in the File menu. These files are named according to the cutting sequence (layer number). Since these files are from the single Photoshop document file, they are all of the same size (graphic size, pixel X—Y dimension).
4.5. Making of a movie frame This process can be finished in 3D Viewer by using the movie-frame-image generation program.
Fig. 7 Techniques used to highlight objects of interest in Adobe Photoshop. (A) An image captured from a rat liver section stained by the GST marker. Focal lesions can be seen on the image that show a brownish color, while the tissue background is a shade of blue. The image background color is white. These color differences can be used to draw the tissue outline and focal lesion easily. (B) Liver tissue is selected by using the magic wand tool to select the white area and then using the Inverse command in the Selection menu. A blank layer is added onto this image. The selection is then outlined by the blue color by using the Stroke command in the Edit menu. (C) Focal lesions are selected by using the Color Range in Selection menu and are then filled with red color onto the new added layer. (D) After the edit process (add missing foci and erase background), the image underneath is removed, leaving only the added layer. Thus, the tissue outline and focal lesions are highlighted and other areas are all set to the white color.
3D reconstruction from serial sections in PC-Windows platform by using 3D Viewer As shown in Fig. 3, aligned and highlighted 2D images from the serial sections are loaded by clicking the Load Images button. Scale information and the cutting distance should be entered first. Options for movie-frame-image generation include: background color, range of the rotation angle, rotation direction, frame number. By depressing the Make Rotation Image. . . button, the program starts the process of drawing the movie-frame-image at a different angle one by one, using the image transform and image-overlay techniques as described. A set of movie-frame-images files (BMP files) is generated with scale information and stored in a selectable destination directory. Fig. 3 shows the rotation frame maker program window when one movie-frame-image is completed. These BMP formatted movie-frame-image files can be converted into JPG format graph files in Photoshop to save storage disk space.
151
4.6. Data presentation 4.6.1. Movie viewing and single-frame image display To run the movie show, the operator should click the Load Images button. Movie-frame-images generated previously are loaded into the computer with scale information. A 1 cm scale is presented on the screen to provide a size impression of the 3D structure of the object. By clicking the Make Movie button inside the Movie control panel, the movie begins (Fig. 5). The Slider controls the movie show’s speed. By selecting front or rear view mode, the user can see opposing views of the show. The playing sequence is controlled by the play mode (one way and two way). The Stop button inside the Movie control panel stops the movie. The Slider inside the Display Image control panel can be used to display a single frame image in the main PictureBox.
Fig. 8 A shift-overlay image plot of a mouse liver sample. This type of image can be generated by 3D Viewer by using the image shift-overlay plot utility. It is made by overlaying the 2D images from the serial sections. The user can select the start image and stop image and adjust the shift distance. Thus, a certain portion of the object can be viewed in detail.
152 4.6.2. Image printing and shift-overlay image generation To this point, a number of images are generated during the 3D reconstruction process: (1) all the original 2D images captured from the serial sections are stored in JPG format, although some of them may be in TIF, BMP format. (2) The aligned and highlighted images are stored in GIF format. (3) The generated movie-frame-images are in BMP or JPG format. These image formats are popular and therefore can be viewed or printed in many graphic applications. 3D Viewer offers another type of image, the shift-overlay image, which is the shift-overlay of the original 2D aligned and highlighted images from the serial sections. The background color of the resulting image, the shift distance, as well as the starting and ending images, can be controlled by the operator. As a result, a BMP file is generated, as shown in Fig. 8. This method offers a way to see detail in some portions of 3D reconstruction.
5. Results and discussion With a 3D Viewer and the help of Adobe Photoshop, 3D reconstructions can be performed in a PC-Windows platform. The system described here has four features: (1) images of the serial section are captured by a digital camera and then used directly; thus, many processes previously required, such as film development and printing, have been eliminated owing to this improvement in the hardware. (2) Accurate image alignment can be achieved in the 3D Viewer by using the image align utility, where the alignment markers are established during the tissue preparation. (3) Tedious tracing of the object’s outline with a digitizer to obtain the contour data is replaced by the highlighting of interested objects, a technique that can be performed in a semi-automatic manner. (4) The earlier time-consuming point and contour data registration in the Z-axis is replaced by the image transform-shift-overlay technique, which is fully automatic. 3D Viewer provides the impression of the three dimensions of the structure and size in four ways. (1) 2D image stacking show. During the movie-frame-image generation process, stacking of successive aligned 2D images one by one to form a movie-frame-image can be viewed in the PictureBox at different view angles. (2) 3D movie show. The user may load a set of movie-frame-images into the 3D reconstruction viewer program and observe the movie show in RGB color from two points of view (front and rear). (3) Single
Y.-H. Xu et al. movie-frame-image show. When the movie stops, each single movie-frame-image can be displayed by using a slider control. (4) Shift-overlay image show. An image of any portion of the 3D structure can be generated and shown. Recent advances in PC technology allowed successful development of the 3D Viewer. For instance, the image transform-shift-overlay technique is the core of the 3D Viewer program. Image transform is a complicated process in programming, involving manipulation of pixel color data one by one. This can now be simply performed in Visual Basic, a Windows application developing tool, by using ImageList control and some of Windows Application Programming Interface (API) functions [10]. Since these API functions actually are functions of the Windows operating system, they are executed very rapidly. The total 3D reconstruction process is greatly simplified and saves time. This system works on the PC-Windows platform and appears to have a number of advantages. Since PCs with considerable memory and hard disk space are reasonably priced and the software required is readily available, it is a relatively inexpensive system. PC hardware can be easily upgraded when needed. Most of the potential users already have the knowledge to use the PC and some of the graphic software such as Photoshop, and thus do not need special training. Since 3D Viewer can be obtained from our laboratory upon request, expensive software may not be necessary. Meanwhile, 3D reconstruction data obtained with such instrumentation can readily be shared with others via image files. Since the image files may be compressed, the cost of hard disk and other storage media is subsequently decreased as well. In order to accomplish 3D reconstruction efficiently, we have found that the following are important factors: 1. During the image capture process, the largest slide to be captured should be used to set the magnification factor in order to avoid truncation. 2. For accurate alignment of the 2D images, alignment markers should be inserted into the tissue block outside the tissue itself before sectioning. Anatomic markers, such as blood vessels, are not recommended because they are not perpendicular to the cutting surface and tend to be curved. This is especially important when the tissue block to be observed is thick. One choice is to drill holes with a mechanical drill; however, this technique must make holes in the tissue itself, thus destroying tissue in its path. The mechanical drill bit usually makes a larger hole than
3D reconstruction from serial sections in PC-Windows platform by using 3D Viewer the drill’s diameter owing to the vibration. Recently some reports showed that a laser beam can be used to make holes [14]; however, this may require expensive equipment. Insertion of other material into the paraffin portion of the tissue block is another choice [15,16]. A range of materials has been tested in our laboratory, such as surgical suture thread, cactus spines, fishing string, and fish bone. Among them, we found that fish bone is the best. Obviously, the alignment marker should be placed very close to the tissue in order to allow the largest magnification. 3. The image alignment is made possible by a rotation-shift process. Sometimes this process may cause data loss. For example, if the objects of interest occupy the full image before rotation, some portion of the objects of interest will be moved outside the image margin after rotation. Thus, the data in this portion are lost. In anticipation of such data loss, we designed, in the image align utility, two options for the operator. The operator may use the shift-rotation or the rotation-only function. Use of the rotation-only function results in a simple rotation of the image. The resultant image is somewhat larger than the original image without any truncation. The new image can be copied onto a new layer of a Photoshop document file, which serves to collect aligned image files. This layer can be shifted to the right location by using the Move tool in Adobe Photoshop because, when the blending mode is set to darken in the layer window, the underlying image (here the previous file’s aligned image) can be seen through. The image size can be increased later by using the Canvas size command in the Image menu. With this technique, the data loss can be avoided. 4. The object highlight process at first seems to be an artifact in the 3D reconstruction. However, the purpose of this process is to emphasize the object of interest and to clear the remainder of the image. The latter is more important than the former. The user may keep the original color of the object if desired; however, the rest of the image must be cleared. These cleared areas should be set to transparent in the image-overlay process. Otherwise this area will block the view path and cover the image underneath. The Object highlight process takes the most time in 3D reconstruction. Previously it was a fully manual process; now it can be done in a semi-automatic manner since Adobe Photoshop offers a number of powerful functions [17,18]. 5. The aligned, highlighted 2D image files can be stored as individual image files or as the different layers in a single Adobe Photoshop doc-
153
ument file. There is an advantage to storing these as PSD files: the image size can be enlarged as needed (such as when image objects are truncated after rotation). One change in image size causes every layer to change. These different layers in multi-layered PSD files can be de-assembled into a set of same-sized image files by using the Save for web command in the File Menu. 6. The format used for final alignment and for highlighting image files should be GIF. These files are used in a movie-frame-images generation program in which the transparent drawing technique is used. Only one RGB color [RGB(255,255,255)] can be used as the transparent color. However, white color on an image may consist of real white [RGB(255,255,255)] and some near-white colors [the color seen by naked eye as white, but not RGB(255,255,255), such as RGB(255,254,254), RGB(255,254,253)]. These near-white colors are difficult to identify and will remain opaque in the image-overlay process. However, the operator can easily delete all near-white colors from the color table in Adobe Photoshop using the Save for Web command in GIF format [19]. 7. In the movie-frame-image generation process, the number of movie frames should be at least 10 in order to run the movie smoothly. The more movie frames, the smoother the movie show, but also the longer the time required to finish the job. The actual number of the frame images to be made is doubled because a set of movie frames from the rear view is also made. The default format of the resultant frame image file is BMP. However, BMP files usually are very large and occupy a lot of disk space. These files can be converted into JPG files and decreased in size since no transparent function should be performed at this stage.
6. Future plans The objects of interest usually have a special color, which is different from the rest of the image. We plan to develop software to separate the objects with specific RGB color more efficiently and thus highlight them readily.
Acknowledgements The authors express their sincere appreciation to Mrs. Mary Jo Markham and Mrs. Kristen Adler for their expert technical typing of the manuscript and
154 to Dr. Ilse Riegel for her critical editorial comments. The development of this software was supported in part by grants from the National Cancer Institute of the United States (CA07175, CA22484, and CA45700).
References [1] 3D Reconstruction home page at http://biocomp.stanford. edu/3dreconstruction/index.html. [2] D.M. Hyde, D.J. Magliano, E. Reus, N.K. Tyler, S. Nichols, W.S. Tyler, Computer-assisted morphometry: point, intersection, and profile counting and three-dimensional reconstruction, Microsc. Res. Technol. 21 (1992) 262—270. [3] J. Aferzon, R.I. Chau, D.F. Cowan, A microcomputer based system for three-dimensional reconstructions from tomographic or histologic sections, Anal. Quant. Cytol. Histol. 13 (1991) 80—88. [4] J.W. Hoffmeister, G.C. Rinehart, M.W. Vannier, Threedimensional surface reconstructions using a general purpose image processing system, Comput. Med. Imaging Graph. 14 (1990) 35—42. [5] J. Spacek, T. Dolezal, J. Hanus, M. Hartmann, Computeraided three-dimensional reconstructions of biological objects using simple facilities, Folia Biol. (Praha) 35 (1989) 329—337. [6] N. Baba, K. Kanaya, Three-dimensional reconstruction from serial section images by computer graphics, Scanning Microsc. Suppl. 2 (1988) 303—314. [7] J.R. Salisbury, W.F. Whimster, Progress in computergenerated three-dimensional reconstruction, J. Pathol. 170 (1993) 223—227.
Y.-H. Xu et al. [8] 3D Reconstruction software homepage at http://biocomp. stanford.edu/3dreconstruction/software/index.html. [9] Microsoft Corporation, Visual Basic on Line for Visual Basic 6.0 Professional Version, CD Material. [10] The API guide homepage at http://www.allapi.net/agnet/ apiguide.php. [11] Pitot Laboratory web page at http://mcardle.oncology. wisc.edu/pitot/. [12] H.C. Pitot, L. Barsness, T. Goldsworthy, T. Kitagawa, Biochemical characterisation of stages of hepatocarcinogenesis after a single dose of diethylnitrosamine, Nature 271 (1978) 456—458. [13] S. Hendrich, H.A. Campbell, H.C. Pitot, Quantitative stereological evaluation of four histochemical markers of altered foci in multistage hepatocarcinogenesis in the rat, Carcinogenesis 8 (1987) 1245—1250. [14] G.E. Clarke, P.W. Hamilton, W.A. Montgomery, Aligning histological serial sections for three-dimensional reconstruction using an excimer laser beam, Pathol. Res. Pract. 189 (1993) 563—566. [15] K. Brandle, A new method for aligning histological serial sections for three-dimensional reconstruction, Comput. Biomed. Res. 22 (1989) 52—62. [16] M.H. Deverell, W.F. Whimster, A method of image registration for three-dimensional reconstruction of microscopic structures using an IBAS 2000 image analysis system, Pathol. Res. Pract. 185 (1989) 602—605. [17] K. Binder, T. Alsoach, Selections, Paths, and Masks, Photoshop 4 Complete, Hayden Books, Indianapolis, 1997, pp. 131—183. [18] G.D. Bouton, B.M. Bouton, Inside Adobe Photoshop 5, New Riders Publishing, Indianapolis, 1998, pp. 384— 389. [19] Optimizing Colors in GIF and PNG-8 Images, Adobe Photoshop 6 Help Content, CD Material.