Neurocomputing 26}27 (1999) 981}987
Virtual microscopy of brain tissue Brent P. Burton, Bruce H. McCormick* Scientixc Visualization Laboratory, Department of Computer Science, Texas A&M University, College Station, TX 77843-3112, USA
Abstract Virtual microscopy of brain tissue is demonstrated in a prototype automated system for the parallel tracing of neurons and their connections. The software system (1) incorporates heuristics for tissue feature extraction, (2) demonstrates dendritic arbor and "ber pathway reconstruction, and (3) frames a "nite element model of the environment in which these neurons grow and interconnect. The technology will enable the collection of an extensive database of measured neurons, leading to an enlarged knowledge base of neuron morphological models. While our work is directed to neuron and "ber tracing in brain tissue, it is applicable to other similar tissues, for instance, the reconstruction of blood vessels or bronchioles (Barillot et al., Comput. Graphics Appl. 5 (1985) 13}19). 1999 Published by Elsevier Science B.V. All rights reserved. Keywords: Virtual microscopy; Neuron tracing; 3D reconstruction; Finite element brain
1. Objectives A prototype system for massively parallel tracing of neurons and mapping their mutual connections is described. The system automates tissue scanning, digitizing, feature extraction, and neuron reconstruction. In addition, the neuronal environment is framed as a "nite element model of the embedding brain tissue. Current manual techniques for tracing neurons and mapping their mutual connections are inherently too slow to support an adequate quantitative analysis of brain morphology at the cellular and tissue levels. Additionally, these techniques typically
* Corresponding author. Tel.: #1-409-845-8870; fax: #1-409-847-8578. E-mail address:
[email protected] (B.H. McCormick) 0925-2312/99/$ } see front matter 1999 Published by Elsevier Science B.V. All rights reserved. PII: S 0 9 2 5 - 2 3 1 2 ( 9 9 ) 0 0 0 9 4 - 6
982
B.P. Burton, B.H. McCormick / Neurocomputing 26}27 (1999) 981}987
trace neuronal processes without concurrently reconstructing the cortical area or brain nucleus in which these cells develop } an environment that often dramatically distorts the cell structure. Reconstructing neurons with due regard for this environment facilitates the study of cell morphology and is essential for the description of brain pathways. 1.1. Parallel tracing of neuronal processes and xbers Interlaced sectioning and scanning of brain tissue creates a volumetric data set that represents the tissue. Each section of this set may contain tissue features, with adjacent sections typically containing neighboring features. Such neighbors are indicative of neuronal processes and "ber pathways in the three-dimensional cortical tissue. For e$cient data streaming, we cull all tissue features present in the current section, saving only pixels in an oriented bounding box immediately surrounding each feature. These localized areas of the image data, called regions of interest (ROIs), are labeled and stored. By extension, segment tracing utilizes the concept of a volume of interest (VOI). A VOI consists of a sequence of neighboring ROIs that de"ne a localized volume within the volumetric data set. VOIs are constructed by leading a programmable '#exible pipe' through the volumetric data space to intersect neighboring features. The con"guration of these pipes and their junctions constitutes the three-dimensional rete diagram of the tissue, a rough interim description (i.e., plumbing model) that is maintained until a more polished parametric model of the brain tissue is created. The rete diagram can be viewed as a Feynman diagram of the tissue's neuronal processes. The time coordinate in this world space #ows perpendicular to the plane of tissue sectioning. In this sense, neuronal processes can be traced `backwards in timea. The parallel reconstruction of these 3D networks of neuronal processes and "bers rests on very di!erent geometric modeling algorithms from those used in conventional surface-based reconstruction [2,5]. 1.2. Reconstruction of neurons and xbers Multiple dendritic and axonal processes are traced at the same time, each being updated as neighboring features are isolated. The segments of the rete diagram are checked against L-system models of neuron morphology. The neuronal processes, so modeled, create a compact representation by summarizing each segment of the rete diagram. Similarly, junctions are modeled by reference to the applicable L-system model. Each neuron, and hence segment, has a local biological time coordinate } its direction of growth. Our prior work on neuron structural representation [4] makes the disambiguation and re"nement of the plumbing model feasible. In fact, this knowledge of what segments and "bers look like, modeling the knowledge base of a trained microscopist, makes possible the automated reconstruction of neurons and their mutual connections. The forest of 3D reconstructed neurons then is placed in a neuron morphology database, and therefore can be drawn upon to build virtual environments using Exploring the Brain Forest [3] visualization software.
B.P. Burton, B.H. McCormick / Neurocomputing 26}27 (1999) 981}987
983
1.3. Framing the virtual environment: The xnite element brain Each traced neuron can be assigned to the "nite element that contains its soma. The cerebral cortex, so modeled, can be viewed as a giant `chest of drawersa (Fig. 1) where a `drawera (any selected FE or cluster of neighboring FEs) can be `openeda as a "le and its population of neurons visualized as illustrated in Fig. 2 [1]. These "nite elements therefore are used to implement a spatial data management system isomorphic to the neocortex as modeled and visualized at the cellular and tissue level.
Fig. 1. Finite element model for a piece of the neocortical shell.
Fig. 2. Finite element populated with synthetic neurons.
984
B.P. Burton, B.H. McCormick / Neurocomputing 26}27 (1999) 981}987
2. Methodology The brain tissue scanner (currently in design) works thus: From an opaque tissue block, consecutive 1 lm sections of tissue are milled o!. Each newly exposed surface is scanned at the limit of optical resolution to generate a volumetric data set of the brain tissue [6]. Starting from this volumetric data set, virtual microscopy software traces neuronal processes and "bers, and attempts to reconstruct the neuronal environment of the brain tissue. Reconstruction of neuron structure and "ber pathways has four steps: (1) section segmentation, (2) segment/"ber tracing, (3) junction and bend detection, and (4) the structural representation of neurons and their connections. Methods to embed the neurons and their connections in a "nite element model of the brain tissue have been previously described [1]. 2.1. Section segmentation Serial sections of brain tissue created by physical (or optical) sectioning are processed in real time during scanning to determine regions of interest (ROIs) that may contain a neuron feature and to quickly cull unnecessary image data. Section segmentation attempts to identify neuron structures intersected by the sectioning plane. Ideally, such structures may appear as circles, ellipses, or less completely, as arcs. However, this is not only the most important step, but also the most di$cult to execute in real time. Bounding boxes delineates ROIs. Each feature (and hence its associated ROI) is identi"ed by a characteristic shape, size, position and orientation within the section, and by the section number. Features are labeled and indexed to facilitate retrieval as neuronal processes and "bers are reconstructed. By choice of tissue stain, the image data typically contains high-contrast yet sparse features, lending itself to data compression. Identifying and saving regions of interest (ROIs) provides a concise representation of each image, yielding a highly e$cient compression by saving only the relevant image data. Saving only these ROIs e!ectively performs data compression by data culling. Data compression is required (1) to match the maximum 60 Mpixel/s data rate of the tissue scanner with the slow 5MB/s transfer rate to secondary storage, and (2) to minimize tertiary storage requirements [3]. For example, scanning physically sectioned tissue at the limit of optical resolution (0.32 lm;0.32 lm), a digital CCD camera with an image resolution of 2048;2048 covers a square 640 lm on a side. Each image, assuming a digitizing quality of 8 bits per pixel, requires 4 MB, which is sent in burst mode to the data acquisition computer. The 60 MHz digital camera, while capable of generating 15 fps, will more typically realize 8}9 burst transmissions per second. 2.2. Segment/xber tracing In the same way regions of interest are used to narrow processing to the areas that need it, segment tracing utilizes the concept of a volume of interest (VOI), which are used to guide "ber and process tracking. A VOI consists of a sequence of `closea
B.P. Burton, B.H. McCormick / Neurocomputing 26}27 (1999) 981}987
985
regions of interest, the collection of which de"nes a subset of the volumetric data set. Closeness is determined not just by the position of features in their respective sections, but is also determined by "ber directions through the 3D volume. As segments are traced through the data, they develop a local direction vector, which provides a heuristic to merge new segments with existing segments. Concentrating the segment tracing to a VOI aids the reconstruction process by limiting search. Further, by de"ning these VOIs, later visualization of the reconstructed neuron may be achieved at low computational cost by representing the volumetric data by the neuron's VOIs. 2.3. Junction and bend detection Neuron processes split (bifurcate, and occasionally trifurcate), thus creating at least two segments leading from one. During the segment tracing process, such junctions will be seen as either a place where one segment divides into two, or where two segments join into one. Both situations need to be recognized and handled properly. In the case of a bifurcation, the incoming segment is terminated and two new segments are created (Fig. 3a). Similarly in a joint, the two incoming segments are terminated
Fig. 3. Basic segment interconnections. (a) segment split, (b) segment join, (c) segment bend, (d) multisegment merger.
986
B.P. Burton, B.H. McCormick / Neurocomputing 26}27 (1999) 981}987
and a new outgoing segment is created (Fig. 3b). Alternatively, the segments being traced may indeed join, but turn out to be the same "ber with a sharp bend (Fig. 3c). After a putative intersection has been found (Fig. 3d), it may be determined that the two or more segments indeed do not intersect, but merely pass closely by each other. Such instances require looking to see where the "bers were heading prior to the putative &intersection'. The local directions of each segment can be used to determine the likelihood of a junction. Such situations require disambiguation for an accurate neuron model. To this end the reconstruction program maintains a catalog of potential ambiguities and their recommended modes of resolution. 2.4. Structural representation of neurons and segments Structural representation is the procedure for converting the assorted segments and junctions of the disambiguated and re"ned rete net (i.e., the plumbing model) into reconstructed neurons and their processes, as modeled by the parametric L-system notation of the neuron morphology modeler N## [4]. The parameterized L-system format provides an abbreviated structural description of the neurons and connections of the tissue. 2.5. Visualization of the reconstructed neurons and their connections During the reconstruction process, it is necessary to check the progress and accuracy of the reconstruction. Blending the segment and junction estimates with available 3D data allows auditing of the reconstruction process through visual comparison. Existing tools under the Visualization Toolkit [7] provide adequate visualization ability. However, custom tools have been developed on an as-needed basis to "ll gaps left by the existing tools. 2.6. Preliminary results using simulated sectioning Currently we are reconstructing neurons from synthetic volumetric data sets, generated in turn from known geometric models of neurons. Simulated sections have been generated from available neuron models. These models, generated by L-systems, are software-sectioned with visualization tools. The advantage of using such models, especially in the early software development stages, is the position of each neuronal process (or soma) is known exactly and thus the reconstruction process can be automatically checked for completeness and accuracy.
Acknowledgements This work was supported by Texas Advanced Technology Program grant 999903124 (McCormick) from the Texas Higher Education Coordinating Board.
B.P. Burton, B.H. McCormick / Neurocomputing 26}27 (1999) 981}987
987
References [1] D.A. Batte, T.S. Chow, B.H. McCormick, Finite element decomposition of human neocortex, in: J.M. Bower (Ed.), Computational Neuroscience Trends in Research, Plenum Press, New York, 1998, pp. 573}578. [2] P.V. Belichenko, A. Dalstrom, Confocal laser scanning microscopy and 3-D reconstruction of neuronal structures in human brain cortex, Neuromage 2 (1995) 201}207. [3] B.P.Burton, T.S. Chow, A. Duchowski, W. Koh, B.H. McCormick, Exploring the brain forest, Neurocomputing, (1999) to appear. [4] R.W. DeVaul, B.H. McCormick, Neuron developmental modeling and structural representation 1. An introduction to the N## language, an open stochastic L-system, Technical Report, Scienti"c Visualization Laboratory, Department of Computer Science, Texas A&M University, College Station, TX, December 4, 1996. [5] B. Geiger, Three-dimensional modeling of human organs and its application to diagnosis and surgical planning, INRIA Rapports de Recherche, No. 2105, Sophia-Antipolis, France, 1993. [6] B.H. McCormick, Design of a brain tissue scanner, Neurocomputing (1999) to appear. [7] W. Shroeder, H. Martin, B. Lorensen, The Visualization Toolkit, second ed., Prentice-Hall, Englewood Cli!s, NJ, 1998.
Brent P. Burton is a software engineer at STB Systems in Austin, Texas, performing optimization of 3D graphics drivers for hardware accelerators. He has a B.S. in computer science from Texas A&M University and anticipates completing his M.S. in computer science in December, 1999. His research interests are scienti"c visualization, and reconstruction techniques as applied to medical image data. Other interests include 3D rendering techniques and programming languages. Dr. McCormick received his B.S. and Ph.D. degrees in Physics from MIT and Harvard University, respectively. He was a professor of Computer Science and Physics at University of Illinois at UrbanaChampaign. At University of Illinois at Chicago and Texas A&M University, he was the department head of the Information Engineering and the Computer Science departments, respectively. He is a professor of Computer Science and the director of the Scienti"c Visualization Laboratory at Texas A&M University. His research areas include scienti"c visualization, brain mapping, computer graphics, and neural networks.