- Email: [email protected]
A new method for three-dimensional reconstruction from serial sections by computer graphics using “meta-balls”: Reconstruction of “hepatoskeletal system” formed by Ito cells in the cod liver
COMPUTERS
AND
BIOMEDICAL
A New Method Serial Sections Reconstruction
HARUYUKI
*Department
RESEARCH
23,
37-45 (l!@o)
for Three-Dimensional Reconstruction from by Computer Graphics Using “Meta-Balls”: of “Hepatoskeletal System” Formed by Ito Cells in the Cod Liver TATSUMI,* EIJI TAKAOKI,~ AND HISAO FUJITA*
KOICHI
OMURA,$
ofAnatomy, Osaka University Medical School, I-3-57Nakanoshima, 530 Japan; tMeta Corporation, Japan; and *Osaka Gukuin University,
Kiia-ku, Japan
Osaku.
Received January 4, 1989
A new method is described for three-dimensional reconstruction from serial ultrathin sections, using “meta-balls” as a primitive of object modeling in computer graphics (CG). We take advantage of the meta-ball’s blobbiness characteristic and its rendering system for reconstructing images. The two-dimensional outline data from serial sections are converted into three-dimensional meta-ball data with a graphic editor, “Metack,” by which we can check the correctness of the data conversion. Then the converted data are visualized on a color display with a CC rendering software, “Tracy.” This reconstruction method is applied in studying the spatial distribution of Ito cells (fat storing cells) in the cod liver in relation to the blood capillary. By observing the reconstructed images, we can easily understand the three-dimensional relationship between the Ito cells and the blood capillary. The Ito cells surround the blood capillary and extend their cytoplasmic processes into the interparenchymal space to make a well-developed network system. These findings would support a concept of “hepatoskeletal system” formed by Ito cells in the cod liver. Therefore, we think that the meta-ball reconstruction method is useful in morphological study. 8 tsso ’ Academic
Press, Inc.
1. INTRODUCTION In morphology, observations are the first and most essential step to study. Light and then electron microscopes were devised, bringing about great epochs in anatomy and other disciplines. They have enabled us to understand the fine structures of tissues and cells, but the method for preparing the sections itself limits comprehension of the three-dimensional structure, since the data obtained from thin sections are two-dimensional. To circumvent this problem, reconstruction by computer graphics is a suitable approach, providing us with a good understanding of the three-dimensional structure (I, 2), but reliable methods are not available to reconstruct complicated, slender, and branching structures, such as cod Ito cells (fat storing cells) from serial sections. So a need exists 37 OolO-4809/90 $3.00 Copyright 0 1!?30 by Academic Press, Inc. All rights of reproduction in any form reserved.
TATSUMI
38
ET Al..
for a method of three-dimensional reconstruction to demonstrate complicated structures. Though beauty is not always scientific, morphologists generally prefer beautiful pictures. Neat pictures often lead us to new findings and support the researcher’s conclusions well as compared with confusing pictures. In fact, good pictures are very informative as well as beautiful. Keeping this in mind, we attempted to develop a new method for three-dimensional reconstruction. This study is a first trial application of a new object modeling primitive, the “meta-ball.” in three-dimensional reconstruction from serial ultrathin sections. In 1951, Ito cells (fat storing cells) were discovered in the Disse space of the human liver by Ito (3); since then the morphology and physiology of this cell have been well studied by many researchers (4-11). Ito cells have the ability to store lipid droplets in which vitamin A is highly concentrated. In addition, the Ito cells in the cod (Gadus morhua macrocephalus) liver have thick bundles of intermediate filaments in their cytoplasm and well-developed desmosomes. by which the Ito cells are interconnected with one another (12). Although scanning electron microscopy is a powerful tool for three-dimensional morphological study, it does not reveal the internal morphology and the architecture covered by the other structures. So precise identification for the structure is difficult. By transmission electron microscopy, we can more easily identify the structure. but the micrographs of serial sections bring only fragmentary information on the three-dimensional structure. For complete understanding, it is necessary to get information about the three-dimensional structure as well as the internal structure of the same object. For the hepatic Ito cells of the cod, the threedimensional architecture must be studied as a whole in a wide area in order to make it easier to understand a concept of “hepatoskeletal system” formed by the Ito cells. Using the new reconstruction method we have developed, we made an attempt to study the three-dimensional architecture of the cod Ito cells in relation to the blood capillary. 2. METHODS
AND MATERIALS
2.1. Meta-Balls
Meta-balls (a kind of modified Blinn’s surface drawing model (13)) were developed as a primitive of object modeling in computer graphics by Omura’s CG group in Osaka University to express complicated objects with ease (14). The meta-ball is represented by a potential function MJ~,which consists of the following piecewise functions: wi(X* Y, Z) = G(rj) = d,[l - 3(v,lb,)2]
(0 I r; 5 b;/3)
= (3&‘2)[ 1 - (riibi)]’
(b,/3 cr r; 5 b;)
= 0
(bi I Tj)
111
THREE-DIMENSIONAL
RECONSTRUCTION
ri = d/(X - Ci,)* + (y -
BY META-BALL
C;,)’
+
(Z, -
Ci,)*
39
121
ci: position of the center d;: value of the center = weight of the meta-ball bi: radius. The surface of the meta-ball (meta-surface) is defined as those points where the value of this function equals some threshold amount, C: w(x, y, z) = c.
[31
Since the value of the threshold is irrelevant, we set it to some canonical value such as 1, according to Blinn (13). The meta-ball has a blobbiness characteristic if we set a fusion switch on. For example, two meta-balls, located in close vicinity less than 2b(b = radius), influence their own meta-surface, that is, invisible points become visible where the sum of the value of the functions at those points is equal to the threshold. Moving closer, the meta-balls fuse smoothly, showing the blobbiness characteristic, when the fusion switch is on (Fig. 1). This blobbiness is affected by the value of the weight (d) as well as the radius (b) of the meta-ball. On reconstructing objects from serial sections, we take advantage of this blobbiness characteristic in smoothing the surfaces of the piled meta-balls. The meta-balls data, rendered in Fig. 1, are described as follows in a shape file. This data format is called an “image score”: { Mtl
me1 { len 20.6782 20.6782 4.000000 mov 0.0000000 0.000000 0.000000 rot 0.000000 0.000000 0.0000000 wgt 3.000000
Mt2
me1 { len 20.6782 20.6782 4.000000 mov 0.0000000 -20.000000 0.000000 rot 0.000000 0.000000 0.0000000 wgt 3 .oOOooo
Mtl, Mt2: labels for the meta-balls mel: meta-ellipsoid primitive len: radius of the ellipsoid (x, y, z) mov: position of the center of the ellipsoid rot: rotation of the ellipsoid (x, y, z) wgt: weight of the meta-ball.
(x, y, z)
40 2.2 Software
TATSUMI
ET AL
and Hardware
2.2.1. Metack. Metack was programmed as a meta-ball graphic editor in order to handle meta-balls visually and interactively on a color display connected to a computer and to reproduce outline data obtained through serial sections. With Metack, we entered and edited meta-balls on the display, representing a threedimensional view (Fig. 2), and saved the meta-ball data onto a floppy or hard disc as an image score as described above. The image score is a data format, readable by a rendering software, “Tracy,” which is described in the next paragraph. Metack has a mode to make a moire, which shows an outline of a slice of the fused meta-balls at a given level. Using this mode, we checked the correctness of data conversion from outline data into meta-ball data. Metack works on the NEC PC 9801 series personal computer (8086 or 80286 CPU) and Sony NEWS workstation (68020 CPU). 2.2.2. Tracy and the LINKS-l system. Tracy is a ray-tracing rendering program, working on the LINKS-l system, both of which were developed by Omura’s CG group in the Osaka University Faculty of Engineering. To render objects on a color display, Tracy needs special data files, such as cluster, shape, attribute, and staff files. Thanks to the data structure, we can change many parameters without difficulty. The kinds of parameters are as follows: fusion, shadow, shade, transmit, opacity, color(16,777,216 colors), reflection, refraction, parallel light, ambient light, camera position, target position, scale, field angle, camera angle, and so on. The meta-ball data are grouped into shape files. In a cluster file, we state the relationships of the objects which are expressed in the shape files, such as their absolute locations and the relative locational relationships among them. Using the cluster file, we can deal easily with the object as a body composed of many parts. This data format is named image score after “music score” by Omura. Just as we play musical instruments with music score, so we can play a computer with image score to make images. LIVKS-I system, a parallel pipelined multimicroprocessor system (with 64 central processing units), was developed to reduce the image generation time (15, 16). Tracy, working in the computer, sends out a data rendered image to a frame buffer via a parallel port.
2.3. Materials
The liver of an adult cod (Gadus morhua macrocephalus), fixed with 3% glutaraldehyde, was processed for conventional electron microscopy. Three hundred serial ultrathin sections at 100 nm in thickness were cut on an Ultracut ultramicrotome (Reichert-Jung), and each ribbon of 10 sections was mounted on a single hole grid with formvar film. Electron micrographs of each level were taken at magnifications of 400 and 800, using a JEM 1200 EX electron microscope.
THREE-DIMENSIONAL
RECONSTRUCTION
BY META-BALL
41
FIG. 1. Blobbiness of meta-balls. Left: fusion switch is off. Right: fusion switch is on. The right two meta-balls are fused smoothly.
FIG. 2. Outlinedataandmeta-ballsshownonacolordisplaywithMetack. Yellowlineshowsoutline of the Ito cells. Blue line, indicated with white arrows, represents that of the capillary, within which meta-balls (purple) are placed. Above the double blue line is a view of the top side of the meta-ball shown in the lower left area. To the right of the double blue line is a view of the right side. A green meta-ball, indicated with a red arrow, is identical to the green ones above and to the right of the double blue line. This shows that the meta-ball is a three-dimensional object. FIG. 3. Ito cell network in the cod liver visualized with Tracy. Since Ito cells are transparent light green, the nuclei (purple) are seen located around the blood capillary (red). Hepatocytes are not visualized so that the relationships between Ito cells and blood capillary can be observed. Ito cells and their cytoplasmic processes form a three-dimensional network among hepatocytes and around the blood capillary. x 930. FIG. 4. Transverse section of the reconstructed capillary at the same level as Fig. 2. The cut surface of the capillary is colored with white, which is well in accord with the original outline (Fig. 2).
42
TATSUMI ETAL
2.4. Chain Data Images of negative films of the electron micrographs were put onto memories on an image processing machine, NEXUS 6400 (Kashiwagi Research Corporation), through a TV camera. With this machine, we got outlines of objects of interest (the Ito cell body, its nucleus, and blood capillaries) by a digitizer and recorded the digitized data onto floppy discs as chain data. Each set of chain data consists of an identification label, one coordinate (x. y) of an initial point, and a series of digits (O-8) indicating the direction of the following points. Since the outline data of the same level have the same z coordinate value. we gathered all the data into a file, and named it after the level of the section: datal, data2, and so on. 2.5 Data Conversion a graphic editor for meta-ball, we reproduced the outlines Using “Metack,” of the objects from the chain data on a color display. We interactively entered meta-balls within the outline using a pointing device, the “mouse.” The metaballs within the outline represent the sectioned object of the level. The outlines are two-dimensional, whereas the meta-balls have three-dimensional data (Fig. 2). In this fashion, we converted the two-dimensional outline data into threedimensional meta-ball data at each level of the section. According to the distance between the consecutive sections, we set z coordinate values for the meta-balls’ radius (len) and position (mov). The meta-ball data was recorded into a shape file as an image score with Metack. Making a cluster file and setting the parameters as described above, we got images of the reconstructed objects (Figs. 3,4) with Tracy. By using moire, which shows an outline of a slice of fused meta-balls with Metack, we checked whether the fused meta-balls correctly represented the original outline in the negative film. When the moire line was not in accordance with the original outline. we modified the position: rotation. and radius of the meta-ball by using the mouse to match the moire line to the original one. Moreover, Metack can show meta-balls of serial levels at the same time. With this function, we could confirm the continuity and discontinuity of the metaballs along the serial levels, based on the electron micrographs. From the data conversion to the image rendering, all the work was done on the UNIX operating system. 3. RESULTS 3.1 Reconstruction
of Ito Cells
The reconstructed image distinctly shows the close relationship between the nucleus of the Ito cell and the blood capillary in the cod liver (Fig. 3). The Ito cell bodies and their short cytoplasmic processes surround the blood capillary to form a small mesh around it by connecting with the adjacent Ito cells also
THREE-DIMENSIONALRECONSTRUCTIONBYMETA-BALL
43
located around the capillary. In addition, Ito cells extend their slender cytoplasmic processes farther into the interparenchymal space and connect with other Ito cells at their extremities to make a large mesh among the hepatocytes. Thus, almost all Ito cells are interconnected to construct a network system. This network system looks like a skeleton in the liver tissue. The images reconstructed from the transmission electron micrographs would support a concept of hepatoskeletal system formed by the Ito cells (12). 3.2. Evaluation
of this Method
Since it is difficult to identify the Ito cell’s slender cytoplasmic process with information from a single section, we must reconsult the electron micrographs of the upper and lower level sections to check the continuity of the object. In fact, we often overlooked pin-point cross sections of the Ito cell’s processes and added their data later on. This feedback procedure is indispensable to avoid the omission of overlooked cytoplasmic processes and to attain an accurate reconstructed image. We can use a negative meta-ball having negative weight to cancel the weight of the other meta-balls at a certain area, which makes a visible part of the object invisible. Using a negative meta-ball, we can cut reconstructed objects at a certain level and make sure of the fidelity of the reconstruction by comparing the cut surface of the reconstructed object with the original outline. The outlines of a reconstructed capillary, cut with a negative meta-ball, are well in accordance with those of the original one (Figs. 2,4). So the shapes and the spatial relationships of the objects are reproduced with accuracy by this reconstruction method. 4. DISCUSSION “Meta-balls” are suitable for the modeling of complicated, branching structures with reduction of the amount of modeling data. The outline data, composed of many coordinates, are replaced with meta-balls. In the extreme case, millions of coordinates are represented by one meta-ball, which results in reducing the image generation time. If we try to replace the meta-balls of several sections with a smaller number of meta-balls, it is possible to decrease the amount of the meta-ball data further. The data conversion from two-dimensional outline data to three-dimensional meta-ball data was performed manually in this study. We are planning to automate the data conversion and minimize the number of meta-balls as a next step. Applying meta-balls in reconstruction by computer graphics, we can beautifully reproduce the complicated objects of interest. Once the chain data are converted into meta-ball data, we can stereoscopically observe the object from any direction at any magnification, and can make video movies to demonstrate the structure. These illustrations, unattainable by conventional microscopy, are very helpful to us in understanding complicated structures and deducing unknown truths.
44
TATSUMI
ET AL
These reconstructed images indicate that the Ito cells in the cod liver surround the blood capillary and extend their cytoplasmic processes into the interparenchymal space and connect with other Ito cells at the extremities of their processes. Being interconnected with each other, the Ito cells make a network system, which seems to serve as a mechanical support in the cod liver. Together with the previous findings of well developed desmosomes and intermediate filaments in the Ito cells (12), we think that this reconstructed image would strongly support a concept of hepatoskeletal system formed by the Ito cells in the cod liver. Thus, we have succeeded in making it easier to understand the complicated architecture in the cod liver, such as the morphology, the association, and the spatial distribution of the Ito cells. For this purpose, the meta-ball reconstruction method is very useful. Thanks to the structure of the image score, we can efficiently move a part of the reconstructed objects. For example, once we model the human body with the meta-ball, we can move a part of the body according to information about muscle contraction and relaxation as a simulation experiment. This trial is under way by Takaoki. Digitization of the outline of the object has to be done manually for the time being, because identification of the object is very much dependent on the researcher’s intelligence and experience, especially in the case of electron micrographs. At present it is very difficult to automate the process, so we have to wait for progress in artificial intelligence or neural net systems to solve the difficulty. Since cell outlines are obscure by light microscopy, we have to rely on the resolution power of transmission electron microscopy. But transmission electron microscopy also has weak points in the resolution. Membranous structures are favorable for reconstruction, because it is easy to trace the outline of the structure, whereas the outlines of cytoskeletal elements, such as filamentous structures, are difficult to trace. So new staining and embedding methods are needed to reconstruct these structures. This study is a challenge in developing a new method for three-dimensional reconstruction of complicated structures from serial sections. We think that this meta-ball reconstruction method is useful in morphological study and that observations on electron micrographs as well as reconstructed images are necessary to obtain accurate knowledge about structure and architecture. ACKNOWLEDGMENT We thank Links Co. for supplying the LINKS-I to us.
system used in this study and for their kind help
REFERENCES 1. NAMBA, K., CASPAR, D. L. D., AND STUBBS, G. J. Computer graphics representation of levels of organization in tabacco mosaic virus structure. Science 227, 773 (1985). 2. JIMENEZ, J., SANTISTEBAN, A., CARAZO, J. M., AND CARRASCOSA,J. L. Computer graphic display method for visualizing three-dimensional biological structure. Science 232, 1113 (1986).
THREE-DIMENSIONAL
RECONSTRUCTION
BY META-BALL
45
3. ITO, T. Cytological studies on stellate cells of Kupffer and fat storing cells in the capillary wall of the human liver. Acta Anat. Nippon. 26,42 (1951). [Japanese] 4. ITO, T., AND SHIBASAKI, S. Electron microscopic study on the hepatic sinusoidal wall and the fat-storing cells in the normal human liver. Arch. Histol. Japan. 29, 137 (1%8). 5. NAKANE, P. K. Ito’s “fat-storing cell” of the mouse liver. Anat. Rec. 145, 265 (1963). 6. WAKE, K. “Sternzellen” in the liver: Perisinusoidal cells with special reference to storage of vitamin A. Amer. J. Anat. 132, 429 (1971). 7. HIROSAWA, K., AND YAMADA, E. The localization of the vitamin A in the mouse liver as revealed by electron microscope radioautography. J. Eiectron Microsc. 22, 337 (1973). 8. NOPANITAYA, W., CARSON, J. L., GRISHAM, J. W., AND AGHAJANIAN, G. New observations on the fine structure of the liver in goldfish (Curussius auratus). Cell Tissue Res. 196, 249 (1979). 9.
10. Il. 12.
13.
14.
15’.
16.
FUJITA, H., TAMARU, T., AND MIYAGAWA, J. Fine structural characteristics of the hepatic sinusoidal walls of the goldfish (Curussius uuratus). Arch. Histol. Jup. 63, 265 (1980). SAKANO, E., AND FUJITA, H. Comparative aspects on the fine structure of the teleost liver. Okujima’s Fol. Anut. Jupon. 58, 501 (1982). TATSUMI, H., AND FUJITA, H. Fine structural aspects of the development of Ito cells (vitamin A uptake cells) in chick embryo livers. Arch. Histol. Jup. 46, 691 (1983). FUJIA, H., TATSUMI, H., BAN, T., AND TAMURA, S. Fine-structural characteristics of the liver of the cod (G&us mot-ha mucrocephulus), with special regard to the concept of a hepatoskeletal system formed by Ito cells. Cell Tissue Res. 244, 63 (1986). BLINN, J. F. A generalization of algebraic surface drawing. ACM Trans. Graphics 1,235 (1982). NISHIMURA, H., HIRAI, M., KAWATA, T., SHIRAKAWA, I., AND OMURA, K. Object modeling by distribution function and a method of image generation. Trans. Inst. Electron. Commun. Eng. Japan 68, 718 (1985). NISHIMURA, H., OHNO, H., KAWATA, T., SHIRAKAWA, I., AND OMURA, K. LINKS-l: A parallel pipelined multimicrocomputer system for image generation. In “IEEE Proc. 10th Ann. Int. Symp. on Computer Architecture,” p. 387, 1983. DEGUCHI, H., NISHIMURA, H., YOSHIMLJRA, H., KAWATA, T., SHIRAKAWA, I., AND OMURA, K. A parallel processing scheme for three-dimensional image generation. In “IEEE Proc. ISCAS’84,” p. 1285, 1984.
AND
BIOMEDICAL
A New Method Serial Sections Reconstruction
HARUYUKI
*Department
RESEARCH
23,
37-45 (l!@o)
for Three-Dimensional Reconstruction from by Computer Graphics Using “Meta-Balls”: of “Hepatoskeletal System” Formed by Ito Cells in the Cod Liver TATSUMI,* EIJI TAKAOKI,~ AND HISAO FUJITA*
KOICHI
OMURA,$
ofAnatomy, Osaka University Medical School, I-3-57Nakanoshima, 530 Japan; tMeta Corporation, Japan; and *Osaka Gukuin University,
Kiia-ku, Japan
Osaku.
Received January 4, 1989
A new method is described for three-dimensional reconstruction from serial ultrathin sections, using “meta-balls” as a primitive of object modeling in computer graphics (CG). We take advantage of the meta-ball’s blobbiness characteristic and its rendering system for reconstructing images. The two-dimensional outline data from serial sections are converted into three-dimensional meta-ball data with a graphic editor, “Metack,” by which we can check the correctness of the data conversion. Then the converted data are visualized on a color display with a CC rendering software, “Tracy.” This reconstruction method is applied in studying the spatial distribution of Ito cells (fat storing cells) in the cod liver in relation to the blood capillary. By observing the reconstructed images, we can easily understand the three-dimensional relationship between the Ito cells and the blood capillary. The Ito cells surround the blood capillary and extend their cytoplasmic processes into the interparenchymal space to make a well-developed network system. These findings would support a concept of “hepatoskeletal system” formed by Ito cells in the cod liver. Therefore, we think that the meta-ball reconstruction method is useful in morphological study. 8 tsso ’ Academic
Press, Inc.
1. INTRODUCTION In morphology, observations are the first and most essential step to study. Light and then electron microscopes were devised, bringing about great epochs in anatomy and other disciplines. They have enabled us to understand the fine structures of tissues and cells, but the method for preparing the sections itself limits comprehension of the three-dimensional structure, since the data obtained from thin sections are two-dimensional. To circumvent this problem, reconstruction by computer graphics is a suitable approach, providing us with a good understanding of the three-dimensional structure (I, 2), but reliable methods are not available to reconstruct complicated, slender, and branching structures, such as cod Ito cells (fat storing cells) from serial sections. So a need exists 37 OolO-4809/90 $3.00 Copyright 0 1!?30 by Academic Press, Inc. All rights of reproduction in any form reserved.
TATSUMI
38
ET Al..
for a method of three-dimensional reconstruction to demonstrate complicated structures. Though beauty is not always scientific, morphologists generally prefer beautiful pictures. Neat pictures often lead us to new findings and support the researcher’s conclusions well as compared with confusing pictures. In fact, good pictures are very informative as well as beautiful. Keeping this in mind, we attempted to develop a new method for three-dimensional reconstruction. This study is a first trial application of a new object modeling primitive, the “meta-ball.” in three-dimensional reconstruction from serial ultrathin sections. In 1951, Ito cells (fat storing cells) were discovered in the Disse space of the human liver by Ito (3); since then the morphology and physiology of this cell have been well studied by many researchers (4-11). Ito cells have the ability to store lipid droplets in which vitamin A is highly concentrated. In addition, the Ito cells in the cod (Gadus morhua macrocephalus) liver have thick bundles of intermediate filaments in their cytoplasm and well-developed desmosomes. by which the Ito cells are interconnected with one another (12). Although scanning electron microscopy is a powerful tool for three-dimensional morphological study, it does not reveal the internal morphology and the architecture covered by the other structures. So precise identification for the structure is difficult. By transmission electron microscopy, we can more easily identify the structure. but the micrographs of serial sections bring only fragmentary information on the three-dimensional structure. For complete understanding, it is necessary to get information about the three-dimensional structure as well as the internal structure of the same object. For the hepatic Ito cells of the cod, the threedimensional architecture must be studied as a whole in a wide area in order to make it easier to understand a concept of “hepatoskeletal system” formed by the Ito cells. Using the new reconstruction method we have developed, we made an attempt to study the three-dimensional architecture of the cod Ito cells in relation to the blood capillary. 2. METHODS
AND MATERIALS
2.1. Meta-Balls
Meta-balls (a kind of modified Blinn’s surface drawing model (13)) were developed as a primitive of object modeling in computer graphics by Omura’s CG group in Osaka University to express complicated objects with ease (14). The meta-ball is represented by a potential function MJ~,which consists of the following piecewise functions: wi(X* Y, Z) = G(rj) = d,[l - 3(v,lb,)2]
(0 I r; 5 b;/3)
= (3&‘2)[ 1 - (riibi)]’
(b,/3 cr r; 5 b;)
= 0
(bi I Tj)
111
THREE-DIMENSIONAL
RECONSTRUCTION
ri = d/(X - Ci,)* + (y -
BY META-BALL
C;,)’
+
(Z, -
Ci,)*
39
121
ci: position of the center d;: value of the center = weight of the meta-ball bi: radius. The surface of the meta-ball (meta-surface) is defined as those points where the value of this function equals some threshold amount, C: w(x, y, z) = c.
[31
Since the value of the threshold is irrelevant, we set it to some canonical value such as 1, according to Blinn (13). The meta-ball has a blobbiness characteristic if we set a fusion switch on. For example, two meta-balls, located in close vicinity less than 2b(b = radius), influence their own meta-surface, that is, invisible points become visible where the sum of the value of the functions at those points is equal to the threshold. Moving closer, the meta-balls fuse smoothly, showing the blobbiness characteristic, when the fusion switch is on (Fig. 1). This blobbiness is affected by the value of the weight (d) as well as the radius (b) of the meta-ball. On reconstructing objects from serial sections, we take advantage of this blobbiness characteristic in smoothing the surfaces of the piled meta-balls. The meta-balls data, rendered in Fig. 1, are described as follows in a shape file. This data format is called an “image score”: { Mtl
me1 { len 20.6782 20.6782 4.000000 mov 0.0000000 0.000000 0.000000 rot 0.000000 0.000000 0.0000000 wgt 3.000000
Mt2
me1 { len 20.6782 20.6782 4.000000 mov 0.0000000 -20.000000 0.000000 rot 0.000000 0.000000 0.0000000 wgt 3 .oOOooo
Mtl, Mt2: labels for the meta-balls mel: meta-ellipsoid primitive len: radius of the ellipsoid (x, y, z) mov: position of the center of the ellipsoid rot: rotation of the ellipsoid (x, y, z) wgt: weight of the meta-ball.
(x, y, z)
40 2.2 Software
TATSUMI
ET AL
and Hardware
2.2.1. Metack. Metack was programmed as a meta-ball graphic editor in order to handle meta-balls visually and interactively on a color display connected to a computer and to reproduce outline data obtained through serial sections. With Metack, we entered and edited meta-balls on the display, representing a threedimensional view (Fig. 2), and saved the meta-ball data onto a floppy or hard disc as an image score as described above. The image score is a data format, readable by a rendering software, “Tracy,” which is described in the next paragraph. Metack has a mode to make a moire, which shows an outline of a slice of the fused meta-balls at a given level. Using this mode, we checked the correctness of data conversion from outline data into meta-ball data. Metack works on the NEC PC 9801 series personal computer (8086 or 80286 CPU) and Sony NEWS workstation (68020 CPU). 2.2.2. Tracy and the LINKS-l system. Tracy is a ray-tracing rendering program, working on the LINKS-l system, both of which were developed by Omura’s CG group in the Osaka University Faculty of Engineering. To render objects on a color display, Tracy needs special data files, such as cluster, shape, attribute, and staff files. Thanks to the data structure, we can change many parameters without difficulty. The kinds of parameters are as follows: fusion, shadow, shade, transmit, opacity, color(16,777,216 colors), reflection, refraction, parallel light, ambient light, camera position, target position, scale, field angle, camera angle, and so on. The meta-ball data are grouped into shape files. In a cluster file, we state the relationships of the objects which are expressed in the shape files, such as their absolute locations and the relative locational relationships among them. Using the cluster file, we can deal easily with the object as a body composed of many parts. This data format is named image score after “music score” by Omura. Just as we play musical instruments with music score, so we can play a computer with image score to make images. LIVKS-I system, a parallel pipelined multimicroprocessor system (with 64 central processing units), was developed to reduce the image generation time (15, 16). Tracy, working in the computer, sends out a data rendered image to a frame buffer via a parallel port.
2.3. Materials
The liver of an adult cod (Gadus morhua macrocephalus), fixed with 3% glutaraldehyde, was processed for conventional electron microscopy. Three hundred serial ultrathin sections at 100 nm in thickness were cut on an Ultracut ultramicrotome (Reichert-Jung), and each ribbon of 10 sections was mounted on a single hole grid with formvar film. Electron micrographs of each level were taken at magnifications of 400 and 800, using a JEM 1200 EX electron microscope.
THREE-DIMENSIONAL
RECONSTRUCTION
BY META-BALL
41
FIG. 1. Blobbiness of meta-balls. Left: fusion switch is off. Right: fusion switch is on. The right two meta-balls are fused smoothly.
FIG. 2. Outlinedataandmeta-ballsshownonacolordisplaywithMetack. Yellowlineshowsoutline of the Ito cells. Blue line, indicated with white arrows, represents that of the capillary, within which meta-balls (purple) are placed. Above the double blue line is a view of the top side of the meta-ball shown in the lower left area. To the right of the double blue line is a view of the right side. A green meta-ball, indicated with a red arrow, is identical to the green ones above and to the right of the double blue line. This shows that the meta-ball is a three-dimensional object. FIG. 3. Ito cell network in the cod liver visualized with Tracy. Since Ito cells are transparent light green, the nuclei (purple) are seen located around the blood capillary (red). Hepatocytes are not visualized so that the relationships between Ito cells and blood capillary can be observed. Ito cells and their cytoplasmic processes form a three-dimensional network among hepatocytes and around the blood capillary. x 930. FIG. 4. Transverse section of the reconstructed capillary at the same level as Fig. 2. The cut surface of the capillary is colored with white, which is well in accord with the original outline (Fig. 2).
42
TATSUMI ETAL
2.4. Chain Data Images of negative films of the electron micrographs were put onto memories on an image processing machine, NEXUS 6400 (Kashiwagi Research Corporation), through a TV camera. With this machine, we got outlines of objects of interest (the Ito cell body, its nucleus, and blood capillaries) by a digitizer and recorded the digitized data onto floppy discs as chain data. Each set of chain data consists of an identification label, one coordinate (x. y) of an initial point, and a series of digits (O-8) indicating the direction of the following points. Since the outline data of the same level have the same z coordinate value. we gathered all the data into a file, and named it after the level of the section: datal, data2, and so on. 2.5 Data Conversion a graphic editor for meta-ball, we reproduced the outlines Using “Metack,” of the objects from the chain data on a color display. We interactively entered meta-balls within the outline using a pointing device, the “mouse.” The metaballs within the outline represent the sectioned object of the level. The outlines are two-dimensional, whereas the meta-balls have three-dimensional data (Fig. 2). In this fashion, we converted the two-dimensional outline data into threedimensional meta-ball data at each level of the section. According to the distance between the consecutive sections, we set z coordinate values for the meta-balls’ radius (len) and position (mov). The meta-ball data was recorded into a shape file as an image score with Metack. Making a cluster file and setting the parameters as described above, we got images of the reconstructed objects (Figs. 3,4) with Tracy. By using moire, which shows an outline of a slice of fused meta-balls with Metack, we checked whether the fused meta-balls correctly represented the original outline in the negative film. When the moire line was not in accordance with the original outline. we modified the position: rotation. and radius of the meta-ball by using the mouse to match the moire line to the original one. Moreover, Metack can show meta-balls of serial levels at the same time. With this function, we could confirm the continuity and discontinuity of the metaballs along the serial levels, based on the electron micrographs. From the data conversion to the image rendering, all the work was done on the UNIX operating system. 3. RESULTS 3.1 Reconstruction
of Ito Cells
The reconstructed image distinctly shows the close relationship between the nucleus of the Ito cell and the blood capillary in the cod liver (Fig. 3). The Ito cell bodies and their short cytoplasmic processes surround the blood capillary to form a small mesh around it by connecting with the adjacent Ito cells also
THREE-DIMENSIONALRECONSTRUCTIONBYMETA-BALL
43
located around the capillary. In addition, Ito cells extend their slender cytoplasmic processes farther into the interparenchymal space and connect with other Ito cells at their extremities to make a large mesh among the hepatocytes. Thus, almost all Ito cells are interconnected to construct a network system. This network system looks like a skeleton in the liver tissue. The images reconstructed from the transmission electron micrographs would support a concept of hepatoskeletal system formed by the Ito cells (12). 3.2. Evaluation
of this Method
Since it is difficult to identify the Ito cell’s slender cytoplasmic process with information from a single section, we must reconsult the electron micrographs of the upper and lower level sections to check the continuity of the object. In fact, we often overlooked pin-point cross sections of the Ito cell’s processes and added their data later on. This feedback procedure is indispensable to avoid the omission of overlooked cytoplasmic processes and to attain an accurate reconstructed image. We can use a negative meta-ball having negative weight to cancel the weight of the other meta-balls at a certain area, which makes a visible part of the object invisible. Using a negative meta-ball, we can cut reconstructed objects at a certain level and make sure of the fidelity of the reconstruction by comparing the cut surface of the reconstructed object with the original outline. The outlines of a reconstructed capillary, cut with a negative meta-ball, are well in accordance with those of the original one (Figs. 2,4). So the shapes and the spatial relationships of the objects are reproduced with accuracy by this reconstruction method. 4. DISCUSSION “Meta-balls” are suitable for the modeling of complicated, branching structures with reduction of the amount of modeling data. The outline data, composed of many coordinates, are replaced with meta-balls. In the extreme case, millions of coordinates are represented by one meta-ball, which results in reducing the image generation time. If we try to replace the meta-balls of several sections with a smaller number of meta-balls, it is possible to decrease the amount of the meta-ball data further. The data conversion from two-dimensional outline data to three-dimensional meta-ball data was performed manually in this study. We are planning to automate the data conversion and minimize the number of meta-balls as a next step. Applying meta-balls in reconstruction by computer graphics, we can beautifully reproduce the complicated objects of interest. Once the chain data are converted into meta-ball data, we can stereoscopically observe the object from any direction at any magnification, and can make video movies to demonstrate the structure. These illustrations, unattainable by conventional microscopy, are very helpful to us in understanding complicated structures and deducing unknown truths.
44
TATSUMI
ET AL
These reconstructed images indicate that the Ito cells in the cod liver surround the blood capillary and extend their cytoplasmic processes into the interparenchymal space and connect with other Ito cells at the extremities of their processes. Being interconnected with each other, the Ito cells make a network system, which seems to serve as a mechanical support in the cod liver. Together with the previous findings of well developed desmosomes and intermediate filaments in the Ito cells (12), we think that this reconstructed image would strongly support a concept of hepatoskeletal system formed by the Ito cells in the cod liver. Thus, we have succeeded in making it easier to understand the complicated architecture in the cod liver, such as the morphology, the association, and the spatial distribution of the Ito cells. For this purpose, the meta-ball reconstruction method is very useful. Thanks to the structure of the image score, we can efficiently move a part of the reconstructed objects. For example, once we model the human body with the meta-ball, we can move a part of the body according to information about muscle contraction and relaxation as a simulation experiment. This trial is under way by Takaoki. Digitization of the outline of the object has to be done manually for the time being, because identification of the object is very much dependent on the researcher’s intelligence and experience, especially in the case of electron micrographs. At present it is very difficult to automate the process, so we have to wait for progress in artificial intelligence or neural net systems to solve the difficulty. Since cell outlines are obscure by light microscopy, we have to rely on the resolution power of transmission electron microscopy. But transmission electron microscopy also has weak points in the resolution. Membranous structures are favorable for reconstruction, because it is easy to trace the outline of the structure, whereas the outlines of cytoskeletal elements, such as filamentous structures, are difficult to trace. So new staining and embedding methods are needed to reconstruct these structures. This study is a challenge in developing a new method for three-dimensional reconstruction of complicated structures from serial sections. We think that this meta-ball reconstruction method is useful in morphological study and that observations on electron micrographs as well as reconstructed images are necessary to obtain accurate knowledge about structure and architecture. ACKNOWLEDGMENT We thank Links Co. for supplying the LINKS-I to us.
system used in this study and for their kind help
REFERENCES 1. NAMBA, K., CASPAR, D. L. D., AND STUBBS, G. J. Computer graphics representation of levels of organization in tabacco mosaic virus structure. Science 227, 773 (1985). 2. JIMENEZ, J., SANTISTEBAN, A., CARAZO, J. M., AND CARRASCOSA,J. L. Computer graphic display method for visualizing three-dimensional biological structure. Science 232, 1113 (1986).
THREE-DIMENSIONAL
RECONSTRUCTION
BY META-BALL
45
3. ITO, T. Cytological studies on stellate cells of Kupffer and fat storing cells in the capillary wall of the human liver. Acta Anat. Nippon. 26,42 (1951). [Japanese] 4. ITO, T., AND SHIBASAKI, S. Electron microscopic study on the hepatic sinusoidal wall and the fat-storing cells in the normal human liver. Arch. Histol. Japan. 29, 137 (1%8). 5. NAKANE, P. K. Ito’s “fat-storing cell” of the mouse liver. Anat. Rec. 145, 265 (1963). 6. WAKE, K. “Sternzellen” in the liver: Perisinusoidal cells with special reference to storage of vitamin A. Amer. J. Anat. 132, 429 (1971). 7. HIROSAWA, K., AND YAMADA, E. The localization of the vitamin A in the mouse liver as revealed by electron microscope radioautography. J. Eiectron Microsc. 22, 337 (1973). 8. NOPANITAYA, W., CARSON, J. L., GRISHAM, J. W., AND AGHAJANIAN, G. New observations on the fine structure of the liver in goldfish (Curussius auratus). Cell Tissue Res. 196, 249 (1979). 9.
10. Il. 12.
13.
14.
15’.
16.
FUJITA, H., TAMARU, T., AND MIYAGAWA, J. Fine structural characteristics of the hepatic sinusoidal walls of the goldfish (Curussius uuratus). Arch. Histol. Jup. 63, 265 (1980). SAKANO, E., AND FUJITA, H. Comparative aspects on the fine structure of the teleost liver. Okujima’s Fol. Anut. Jupon. 58, 501 (1982). TATSUMI, H., AND FUJITA, H. Fine structural aspects of the development of Ito cells (vitamin A uptake cells) in chick embryo livers. Arch. Histol. Jup. 46, 691 (1983). FUJIA, H., TATSUMI, H., BAN, T., AND TAMURA, S. Fine-structural characteristics of the liver of the cod (G&us mot-ha mucrocephulus), with special regard to the concept of a hepatoskeletal system formed by Ito cells. Cell Tissue Res. 244, 63 (1986). BLINN, J. F. A generalization of algebraic surface drawing. ACM Trans. Graphics 1,235 (1982). NISHIMURA, H., HIRAI, M., KAWATA, T., SHIRAKAWA, I., AND OMURA, K. Object modeling by distribution function and a method of image generation. Trans. Inst. Electron. Commun. Eng. Japan 68, 718 (1985). NISHIMURA, H., OHNO, H., KAWATA, T., SHIRAKAWA, I., AND OMURA, K. LINKS-l: A parallel pipelined multimicrocomputer system for image generation. In “IEEE Proc. 10th Ann. Int. Symp. on Computer Architecture,” p. 387, 1983. DEGUCHI, H., NISHIMURA, H., YOSHIMLJRA, H., KAWATA, T., SHIRAKAWA, I., AND OMURA, K. A parallel processing scheme for three-dimensional image generation. In “IEEE Proc. ISCAS’84,” p. 1285, 1984.