Mtcron and Microscopica Acta, Vol. 15, No. 4, pp. 221 226, 1984. Pr,nted in Great Britain.
0739626084 S3.00÷0.~ 1985 Perganion Press Ltd.
COMPUTER-AIDED STEREOGRAPHIC REPRESENTATION OF AN OBJECT RECONSTRUCTED FROM MICROGRAPHS OF SERIAL THIN SECTIONS NoRlo BABA,* MIcHIAKI NAKA,* Y0sHIN0RI MURANAKA,t SHIN-IcHI NAKAMURA4 ISAMU KINo~ and KoIcHI KANAYA* * Department of Electrical Engineering, Kogakuin University, 1-24-2, Nishishinjuku, Shinjuku-ku Tokyo, Japan, t Electron Microscope Laboratory, Hamamatsu University School of Medicine 3600, Handa-cho, Hamamatsu, Japan and ~ Department of Pathology, Hamamatsu University School of Medicine 3600, Handa-cho, Hamamatsu, Japan
(Received 3 May 1984: in revised lorin 20 July 1984)
Abstract—A method is described which allows a three-dimensional objectto be reconstructed from micrographs ofserial thin sections using computer graphic techniques. The reconstructed object, which can be rotated threedimensionally, is displayed on a colour visual display unit and the surface ofthe object is shaded in order that it can be observed to provide an illusion of a three-dimensional structure. Moreover, the technique makes it possible to represent an inner structure when seen through an outer one, also to observe other sectioned face views. The method as described here allows rapid visual evaluation of the results of three-dimensional reconstruction from serial thin sections when recorded with the aid of a light or an electron microscope.
INTRODUCTION A transmission microscope provides only twodimensional information from an image. Recording photographs of serial thin sections of a specimen with a microscope, on the other hand, is one method to investigate the threedimensional structure of an object. The established technique employed to reconstruct an object is the stacking of a number of contoured sections with transparent plastic spacer plates or cut balsa-wood sheet materials (Henderson and Unwin, 1975; Blundell and Johnson, 1976). In these techniques, however, it is sometimes difficult to construct atransparent model which will allow the inner structure to be observed through the outer one. In addition, the models built using the above materials are not always suitable for the method of sectioning to observe other sequences of face views by trial and error. The present technique using a colour graphic system by microcomputer is proposed as a suitable method to solve the problem and allows rapid visual evaluation of three-dimensional reconstructed results without the need to construct models. Recently, some similar stereographic representation techniques have been described
(Herman, 1980; Heel, 1983). However, the method reported here is different, as it describes a simple procedure for stereographic representation of a reconstructed object from micrographs of serial thin sections. In this case the reconstructed model is presented with several threedimensional contours displayed with the aid of some current graphic techniques including shading. The present technique was developed from techniques in which contour lines or planar contours obtained from serial section images were drawn graphically by a computer (Stevens, 1977; Perkins and Green, 1982; Johnson and Capowski, 1983; Wong et al., 1983). Furthermore, the present technique has also been applied to practical stereo images of a biomedical specimen using a transparent model as mentioned above. The principle of the shading algorithm was previously given in earlier publications by Newman and Sproull (1975) and Harrington (1983). In this paper we have applied the principle of the shading technique to the stereographic representation. In the threedimensional reconstruction, the alignment of images of serial thin sections when stacking them in sequence presents one of the difficult problems. 221
222
N. Baba, M. Naka, Y. Muranaka. S-i. Nakamura, I. Kino and K. Kanava
This is also the case for a series of high-resolution electron micrographs. The main aim of this paper is to provide a simple method and procedure for applying current graphic techniques to some stereographic representations of the reconstructed results. In our initial studies we used a series of light microscope images, which were then adapted and utilized for a series of electron micrograph images. The computer processing and algorithm are described below, also practical examples of stereographic observations of a biomedical object reconstructed from micrographs of serial sections taken with a light microscope. SPECIMEN MATERIAL AND MICROGRAPHS RECORDED AS SERIAL SECTIONS The specimens used for microscopic observation and three-dimensional reconstruction were 1
minute colonic adenomas in human tissues from familial adenomatosis coli. Details of the material and specimen preparation were described by Nakamura and Kino (1984). Figure 1 shows micrographs of serial sections of the specimen used for the computer-aided reconstruction. These images were taken with a light microscope. The thickness of the section was estimated to be 3-5 ~.tm.The images show serial cross-sections of the specimen through the :-axis. The arrows in Fig. 1 indicate a bud of a single gland adenoma. PROCED URES FOR RECONSTRUCTION AND STEREOGRAPHIC REPRESENTATION The processing method for three-dimensional reconstruction and stereographic representation of the reconstructed object is described below under (A), (B). (C) and (D). The processing system used consisted of a digitizer, D-SCAN
_______~
__
I ____L
___
2
____
~
3
~bf ~~JI
6
-
~__
-~
~
__
10
___
I:
—
7
-
11
-
__j~j 4
_
-
i
~
.
_
8 -n
__
__
-.
___
.
Fig. I. Ts~elvcserial sections recorded in a light microscope from an adenoma I • three-dimensional reconstruction.
which was iis~das ti sample fsr
Computer-aided Reconstruction from Micrographs
DT-3 100 tablet and a 16-bit microcomputer, SORD M343, with a colour CR1 display and a magnetic disk (20MB). The processing is shown in the flow diagram of Fig. 2. (A) The serial micrographs are processed by ________________________________ Trace contour lines of objects observed~J in each micrograph in order of thin sect [series using a digitizer. °
I
_______________________________ [Divide
the contour lines into many points.J
F Store
p
three-dimensional coordinates of points into memoryt three—dimensional contours I of the objects are constructed with sets of [many small triangle planes (see Fig.2).
I
1sotate the reconstructed objects three—dinensio~ii] by calculating new coordinates of three points F I forming each triangle plane rotated for ~ s~ I around the x-, y- and n-axis, respectively, ‘Number each of the triangle element planes in of the shortness of the new z-coordinatest Ithe projection plane is the x-y plane.
@ lorder
(
p Perform the shading processing. Calculate the brightness ~c~~”i’j~ on each (small triangle element plane constructing F the object surface which is illuminated in direction 1 (see Fig.21
F
La
p (Draw each projected small triangle in a color with the brightness of 5~S order of the number which numbered [above block ~j.
T
n~_~spare~~ ____________
____________
[are overlapping not drawn.~J parts F
I[are Over drawn. lapping
par~
Fig. 2. Flow diagram showing the processing stages in block form for three-dimensional reconstruction and stereographic representation.
the digitizer. Contour lines of objects observed in each micrograph are traced manually and the lines are divided into a large number of points. An example of a region for tracing is shown in Fig. 3a, which is a traced result of the sixth micrograph shown in Fig. 1. There are several types of tablet, including the RAND tablet, voltage gradient tablet, acoustic tablet and electro—acoustic tablet, which are currently available as suitable graphic input devices (Newman and Sproull, 1975). Some of the more recent tablets are considered to be convenient for providing the input of graphic image data to computers (Green et a!., 1979). The contour of many obtained from lines each(consisting micrograph werepoints) then stacked in the order of the series. In the computer processing, the three-dimensional coordinate of these points (forming the contour lines) together with an estimate of the section thickness were fed into a storage device. It should be mentioned here that some points or parts (within the image) are necessary to form a base for aligning the successive images in the stacking procedure. In this stacking procedure, the part of the surface epithelium and corresponding part in each micrograph (indicated by double arrows) were used as the bases for alignment. This was achieved by visual adjustment and drawing the contour lines on the CRT. Thus a threedimensional contour of the object (surface of the object) is represented by a set of many image points. The contour is then linked by a set of a number of small triangular planes as shown in Fig. 3. A flat plane is formed only if three sets of
—
2
-~
-
223
,
~~~ça’ Fig. 3. Schematic diagram for the stereographic representation procedure, also for an explanation of the triangular element planes constructing the surface of a reconstructed object; (a) shows an example of tracing the contour lines in the sixth micrograph of Fig. 1.
224
N. Baba. M. Naka, Y. Muranaka, S-i. Nakamura, I. Kino and K. Kanava
i)~ .5°
I
~2.2 ~ .2i-us~us .ci a .,~ .~
.9
5)
-so — na.~~ a.. c~.a soIf) 5)5) — 5)5t nit) ~ a 2i2~..i~ ~~~~50 a..— nan .o~~ ,)~ .~ a 5) 0 .5 5° us ~ii —
II_
_______________________
2
i-
a_s
.~
.5°
~
5) ‘so5)5° 5° ~
tn
a 5°
us,) S.’ ~ 5°. a
_______________________
~ ,~.a
nit)
5)—
Ifs, M’ s”
U
5°
Li.J
Computer-aided Reconstruction from Micrographs
the x-, y- and z-coordinates of the three points are determined. (B) The reconstructed object is then rotated three-dimensionally, which is achieved by calculating the new coordinates of the three points forming each triangular plane rotated for 0,,., 0~ and 0~around the x-, y- and z-axes, respectively, Subsequently, each of the triangular plane elements used for constructing the threedimensional contour is numbered in the order of the shortness of each calculated z-coordinate, which represents the distance from the x—y projection plane. (C) The following processing to shade the surface of the reconstructed object was performed, as it can be viewed to give an illusion of a three-dimensional structure (Baba et a!., 1982). It is assumed that the three-dimensional object is illuminated in the direction of I as shown in Fig. 3. Moreover, it is also assumed that the brightness on each small triangle element plane at the surface is calculated by the scalar product of B~ = Hi. 1 (Lambert’s law), where n~ = (a~ x b~)/2. It can be seen in Fig. 3 that the vector product n represents the area of the triangular plane and the normal direction to the plane. Therefore, shading of the surface of the object is made by displaying each projected triangular element plane on a CRT with the brightness B. In the case shown here for simple processing, the shadowing effect due to neighbouring objects is neglected. (D) Each of the projected small triangles is drawn with the brightness of B, in the order of the number which was allocated in the second processing stage described above, where the overlapping parts are not drawn. When a transparent model (see Fig. 4g) is constructed the overlapping parts are also drawn. In the practical processing using the present CR1, the brightness was represented by a number of colour dots plotted inside the projected small triangle, in which the position of each dot was determined by a random number. PRACTICAL EXAMPLES OF STEREOGRAPHIC REPRESENTATIONS Figure 4 shows practical examples for stereo viewing the result of three-dimensional reconstruction from the micrograph series (Fig. 1).~ Figures 4a, b and c are results rotated for 30 0 and 30 around the x-axis, respectively; in which these are also further rotated for 30C around —
—
225
the y-axis. The x- and y-axes are indicated in Fig. 1 and the 2-axis is perpendicular to the plate of Fig. 1. Figure 4b shows an underside view of the object. In this model any part to be presented can be displayed in colour which is, therefore, different from any neighbouring parts. In the case of a bud of a single gland adenoma, for example, it is drawn in red and the neighbouring region is light blue. Figures 4d, e and f show differences of various directions of illumination 1 (.eZ~). Figure 4g shows a transparent model in which the inner three-dimensional contours, indicated by arrows (~)in Figure 3a (in green and pink), can be seen through outer contours (in light blue and red). In this model, part of the left side contour is omitted. This technique is advantageous for observing the relation between an inner and outer structure. Figure 4h is an upper side view of the object. Figure 4i shows an example of cutting off the model (compare with Fig. 4h). This was simply performed by omitting the drawing processing of the triangular element planes used for constructing the three-dimensional contours, except for a region bounded by a boundary plane (cutting plane). The boundary plane in the example of Fig. 4i is the y—z plane. This technique can also be utilized for observing other sectioned face views. CONCLUSION The techniques presented here provide a rapid and relatively simple way for observing and evaluating the results of three-dimensional reconstruction from serial thin section micrographs recorded with a light or an electron microscope. Acknowledgements——We would like to thank Yasuo Odagiri and Kazuteru Chinone for composing the graphic system with a microcomputer and programming the system software.
REFERENCES Baba, N., Hijikata, A. and Kanaya, K., 1982. Threedimensional representation method of electron microscopic image with shading techniques by computer processing. In: Proc. 10th mt. Conyr. Electron Microsc., Hamburg, Vol 1, 501 502. Blundell, T. L. and Johnson, L. N., 1976. Protein Crystallography. Academic Press, New York. Green, R. J., Perkins, W. J., Piper, E. A. and Stcnning, B. F.. 1979. The transfer of selective image data to a computer using a conductive tablet. J. hiomed Enqng, I: 240. Harrington, S., 1983. Computer Graphics. McGraw-Hill, New York, 355 373.
226
N. Baba. M. Naka, Y. Muranaka. S-i. Nakamura, I. Kino and K. Kanaya
Heel, M. V., 1983. Stereographic representation of threedimensional density distributions. Ultramicro,scopy, 11: 307-314. Henderson, R. and Unwin, P. N. T., 1975. Three-dimensional model of purple membrane obtained by electron microscopy. Nature, Lond., 257: 28-32. Herman, G. T., 1980. Image Reconstruction from Projections. Academic Press, New York, 260 276. Johnson, E. M. and Capowski, J. J., 1983. A system for the three-dimensional reconstruction of biological structures, Computers biomed. Res., 16: 79 87. Nakamura, S. and Kino, 1., 1984. Morphogenesis of minute adenomas in familial polyposis cob. J. natn Cancer Inst., 73: 41—49.
Newman, W. M. and Sproull, R. F., 1975. Principles oj Interactive Computer Graphics. McGraw-Hill. New York, 389—410. Perkins, W. J. and Green, R. J.. 1982. Three-dimensional reconstruction of biological sections. J. hiotned. Engng. 4: 3743. Stevens, B. J., 1977. Variation in number and volume of the mitochondria in yeast according to growth conditions. A study based on serial sectioning and computer graphics representation. Biol. cell., 28: 37-56. Wong, Yu-Man M., Thompson, R. P., Cobb. L. and Fitzharris, T. P., 1983. Computer reconstruction of serial sections. Computers hiomed. Ret., 16, 580-586.