An interactive computer graphic system for 3-D stereoscopic reconstruction from serial sections: Analysis of metastatic growth

AN INTERACTIVE COMPUTER GRAPHIC SYSTEM FOR 3-D STEREOSCOPIC RECONSTRUCTION FROM SERIAL SECTIONS : ANALYSIS OF METASTATIC GROWTH S. General

Electric

D. CH.~I\vL.\

Company. LI,ON

Department

of Physiolqq.

M&III

Charlottesville.

Virgma,

U.S.A

CL ~5.‘; l_lnl\erslfy.

Montreal.

Quebec.

C‘anada

SIMON FK~YI\L’ALI) Hiomedlc,tl

Engineermg

I’nlt. Mc<;ill L;nwerslt?.

J. w. Alleghenq

General

Hospital.

PRO(

\lolltreal.

Quebec.

Canada

TOR

Plttsbursh.

Pennqhanls.

L1.S.A.

AbstractAn mtsractive computer graphic hystern has been developed for 3-D reconstruction of stained serial secttons. The system has been used ior 3-D reconstruction of portions of mouse lun$? contaming melanoma nodules. A strreoscoplc color picture ofa wire mesh model of the reconstructed object IS produced. Thevolume. surface area. and spherIcIt> mdeu 01 structures such as tumor nodule:, can hc computed

Interactive

computer

graphics

Three-dimensional

reconstructlon

Metastasis

Stereoscope

1. 1NTRODUCTION reconstruction of biological tissue from serial sections has been employed for over a century to study anatomical and developmental problems [I]. Early workers used laborious manual methods to produce perspective views or physical models of the organs being studied [ 11. Such manual methods are extremely tedious and consequently difficult to use in experimental situations in which one wishes to apply three-dimensional reconstruction of the same organ in different animals after experimental manipulation. To overcome these problems. recent workers have developed computer based methods of reconstruction. which have been applied to reconstruction of nervous tissue [Z 51. In experimental oncology tumor cells are often inoculated directly into the animal’s blood stream [6]. The cells lodge in the animals giving rise to tumor nodules which grow with time and eventually kill the animal. Since this process leads to a dispersed growth it is often taken as an animal model of metastatic spread in humans. Using this experimental paradigm it is obviously of great interest to study the number and sizes of growing tumor nodules in the animals’ organs at different post-inoculation times and following different experimental manipulations. There is a large literature dealing with such problems [h]. However. accurate quantitative data dealing with the number and size of growing nodules can only be obtained using three-dimensional reconstruction. Previous workers have viewed three-dimensional reconstruction as too laborious for experimental oncology and hake used approximate methods for assessing dispersed tumor growth [7. 81. In rhe following we describe the technrcal features of a computer based interactive system for the reconstruction of mouse lung containing tumor nodules from serial sections. Experimental results on the size and spatial distribution ofrumor nodules obtained using this system are being reported elsewhere

Thtxe-dimensional

PI. ;

1

Li

223

-I

A.

2.1 Orrwirw

of‘ th

THE

LUNG

SYSTEM

LL’NG system

LUNG is the computer system software developed at the Biomedical Engineering tinit, McGill University for the three-dimensional reconstruction of stained serially-sectioned mouse lung with metastatic tumor nodules. Serial sections (8pm thick) of the lung are available in the form of microscope slides. The upper left lobe of mouse lung is prepared for sectioning in accordance with standard histological procedures [9J. Prior to sectioning, three pieces of sciatic nerve are introduced into the paraffin wax block (containing the mouse lung) approximately perpendicular to the plane of sectioning (these later serve as alignment guides during computer reconstruction). After sectioning, the serial sections are placed consecutively on microscope slides and stained. The section number, indicative of its c coordinate, is marked on each slide. About 300 serial sections can be made for each lung and approximately 60 equally spaced sections are used as input to the computer system. The development of LUNG was guided by several functional requirements. Specifically. the system must: (1) accept and store the outlines ofthe lung. tumor nodules and the alignment guides in the sections ; (2) assemble all the sections of the lung together to generate a compact three-dimensional model of the lung and the tumor nodules; (3) display the lung on a cathode ray tube (CRT) or other output device at any specified viewing angle ; (4) Extract required quantitative information on the various structures in the sections; (5) Provide easy access to a novice user by means of interactive pre-programmed instructions. The software system is designed in terms of modular units with each unit meeting a specifc functional requirement. Each functional unit is logically independent of the others and operates on the data base.

LUNG is implemented on a PDP-1 l/70 minicomputer with 356 kbyte of core storage and 88 mbyte of on-line mass storage (DEC RP04). Off-line mass storage consists of IBMcompatible floppy disks. The organization of the minicomputer and its peripherals is displayed in Fig. 1. The PDP-1 l/70 runs under the RSX-11 M operating system in a real time multi-user environment. LUNG is written in the PDP-11 Fortran IV Plus language. High speed (up to 9600 baud) Tektronix 40 xx storage tube terminals are used for the display of three-dimensional pictures. Permanent copies of these pictures can be obtained on a graphic hard copier (Tektronix 4631). Colored pictures may be obtained on the digital plotter (Tektronix 4662). A Tektronix graphics software package (Plot-lo) is used to direct graphic output to the storage tube terminals. The input station consists of a Numonics digitizer interfaced to the PDP-1 l/70. The slide to be digitized is mounted on the projector. The projector lens is adjusted so that a wellfocused and magnified projection of the section is obtained on the tablet. The outline of the contour is traced using the ‘pen’ (the tip of the sliding arm). The contours are graphically displayed on the CRT as they are entered to provide immediate feedback to the user.

The data acquisition programs extract relevant data from each serial section for LIX by the alignment and display programs. The relevant information consists of the boundaries of the different structures (lungs. tumor nodules and alignment guides) in each section. The boundaries (or contours) of these structures in a section are approximated by line segments. and internally represented as the coordinates of the end-points of the line segments. All data pertaining to one lung is stored in two random access data files [lo]. One data file contains the list of coordinates representing all the contours of all the structures in the lung. The I, J’ coordinates of a contour are stored sequentially in the order digitiz.ed. The second

InteractIve

125

3-D reconstruction

by. 1. The PDP-I I.‘70 mmicomputer is shown in relation to its perlpherdl devices. The Numonics Digltlrer is connected to the PDP-Il.‘70 through a sertal interface. The Dtgttal Plotter (Tektronix 4661) can be used to obtain colored pictures. The Tektronix Hard Copier furnishes permanent copies of the storage tube displays.

data file IS a directory which stores the starting addresses of the contours in the first file. In this file. each structure is assigned one unique record that contains a sequential list of addresses ofcontours of that structure. A separate record is set aside to hold the z coordinate of sections digitized. The current configuration of the data files permits storage of up to 64 serial sections containing up to 32 different structures. A maximum of I27 points may be stored to represent an) contour.

70 generate a usable ‘stored model’ of the lung a preliminary alignment operation is necessary. The alignment operation must correct for variations in the positioning of the image of the section on the input tablet. The alignment (of successive sections) problem is classified under the general category of ‘picture matching’ among image processing techniques. A wide spectrum of algorithms are addressed to tackle the generalized problem of pictllre matching [ 1 1. 121. A simple algorithm has been developed to permit interactive alignment of consecutive sections. Two adjacent sections can be correctly aligned by matching up identical landmarks on the two sections. The landmarks are generally taken to be the contours of the alignment guides (i.e. sciatic nerve sections) in each section. The positions of corresponding landmarks on the two sections are used to compute the displacement of one section relative to another in terms of rotation and translation factors. Each of these factors is computed by applying the least squ:lres criteria to the distance between the corresponding landmarks on the two sections. Repl.:ating this procedure for the other sections in the lung results in the generation of a compact model of the lung. 2.Ja. Fmsllrtion mtl r.otution,fuc.rors. Algorithms to compute the translation and rotation factclrs arc based on the results of the following mathematical development. Lt,t Xi and Yi be the coordinates of the center of mass of the ith alignment guide in the referl:nce section, and X( and Y! represent the coordinates of the center of mass of the ith alignment guide in the section being aligned. If the section being aligned is moved by tran$Jation factors of ‘-1in the direction of the X-axis and B in the direction of the Y-axis. the resulting distance Di between the corresponding centers of mass is D,z =

If thtnre arc hi landmarks

[Xi - (X; - ,4)]2 + [yi - (y: - &I’.

being used for alignment

we wish to minimize

(1) S, where

The values

of .4 and B which minimize (3

S can be computed

by setting

is

= 0,

-- = 0

c-.-t

C’B

(31

to yield

Next. if the section being aligned is rotated by an angle rl with respect to the reference section. the distance Di between the corresponding landmarks can be expressed as: 0: = [Xi - tAY{cos;x Once again. to minimize

1’;sinsc)J’

t [Yi - (Xi sin x + Y: cos 8)12.

(5)

S. we set C‘S t-2

=o

yielding.

Alignment is a two step procedure. The factors .-I and B as computed in equation (4) determine the translation of the section. The angle CCas computed in equation (6) determines the rotation necessary to complete the alignment of the section. Alignment of sections is achieved interactively. Sections currently under alignment are displayed on the CRT to permit the user to visually inspect the aligned sections. The rotation and translation factors are displayed along with the aligned sections. Each iteration of the alignment operation computes the translation and rotation factors. Several iterations may be employed to bring two sections into perfect alignment. Our experience has shown that no more than three iterations are required to align two sections. and rotation displacements. Further iterations result in negligibly small translation indicating no further improvement in the precision of alignment.

The graphic display module of LUNG permits the viewer to examine the reconstructed lung in three dimensions. It is possible to examine the lung from different angles of view by specifying the angle of rotation of the lung about the s. J’ or z axis. A stereo-display of the wire-mesh model of the lung is presented. The 3-D picture may be previewed on-line on a storage tube terminal. Permanent colored copies of the picture are obtained by directing the output to the digital plotter (Fig. 2). A stereo picture of a three-dimensional object is generated by presenting the left and right eye view of the object on a display surface [ 131. When such a picture is viewed with ‘stereoglasses’ (or with unaided eyes after some training) a 3-D view ofthe object results. Generation of 3-D stereo pictures requires that a perspective view corresponding to each eye be drawn. Perspective view transformation matrices are used to carry out this operation [14, 15J. This module consists of programs that formulate the required transformation matrices, perspective view. rotations and translations [ 14, 151. After the X, J!, z coordinate data of the lung is approximately transformed. it is displayed graphically with the help of Plot-10 graphics package.

Interactive

219

3-D reconstruction

The volume of a tumor nodule is computed by adding together the contributions to the volume from each serial section in which a nodule appears. The contribution of a single section to the volume is the planar area of the nodule in a section multiplied by one half the sum of the distances to the adjacent serial sections. The total surface area of a tumor nodule is computed by adding up the surface area between every pair of adjoining contours [ 161. The surface between two adjoining contours is approximated by triangular tiles as shown in Fig. 3. The edges of the triangular tiles are determined hy an algorithm that connects the diagonals between points on the contours. The algorithm first connects the two closest points ((1-h) on the two contours, then connects 11to (’ if ,Jistance ac’ is shorter than (or equal to) btl or else connects h to ti. The entire surface is tri,mgulated and its surface area determined. Phe sphericity index (S.I.) of a tumor nodule is detined S.I. = J-1 3.4. ’ 2

(7)

wk ere I is the volume and .-1is the surface area of the tumor nodule. The sphericity index is a dimensionless number [17] which is maximum for a sphere (!%I. = 0.46). Although the spllerlcity index is of potential value in giving a crude measure of the deviation of histological structures from a spherical shape this index is not commonly used in quantitative histology and pathology [18. 191. ‘The error in the computation of the volume and surface area were estimated as follows.

(a) Contour

x

Contour

y

(b) f,,

b

,,,, d_

c

Cottour

x

Contour

y

FIN. ?(a) The surface between two corresponding contours A’ and Y in adjoining sections can be approximated by constructing triangular tiles asindicated. (b)The two closest points (a. h)on the two contours .‘i and Y are connected first. Next. point a is connected to c if the distance ac is less than or equal to the distance bd (resulting in the triangle ctbc) or eke point b is connected to point d (resulting in the triangle abd). The entire surface is ‘triangulated’ on the basis of this rule.

Serial section contours of a sphere (10 cm radius) were digitized. Nineteen sections spaced I cm apart were digitized. The volume and surface area of the sphere was computed by the algorithms described above. These computations of the volume and surface area were compared with theoretical computations for a sphere to arrive at the error estimate. The results were accurate to within 6”,,. Somewhat greater errors will arise with small nodules in the biological material which appear in only a few sections. and with nodules which are irregularly shaped. In Table 1 we present the quantitative data extracted from a typical lung sample. So far we have analyzed 7 samples of mouse lung. The results of our analyses are being published elsewhere [9]. 2.7. System performance

meusurements

The performance of the software system has been ticks = l/60 second). Memory Demand (in kwords count. These parameters were measured for the data module and the quantitative analysis programs. summarized in Table 2. 2.8. Resolution

measured in terms of the CPU time (in x CPU time) and I/O (Input/Output) acquisition module, picture generation The parameters and cost figures are

considerations

The Numonics digitizer used for digitization of serial sections had a resolution of0.25 mm. Since the projector magnified the sections by a factor of 27.36 it became possible to pick up features discernible to the eye at resolutions that were more than adequate for quantitative data extraction and display purposes. The display surface of the Tektronix terminal provided a screen resolution of 1024 x 780 data points. With this resolution, stereo images started getting cluttered when sections closer than about 200 microns (in the z-direction) were displayed. This limited the display to no more than about 20 uniformly spaced sections of about the 50 digitized. However, all the digitized sections are used for data extraction analysis.

3.

SUMMARY

AND

CONCLUSIONS

This paper describes an interactive computer graphics system for three-dimensional reconstruction of organs from their serial sections. The computer system permits the following operations. (1) Data acquisition of serial sections. (2) Assembly and alignment of serial sections to store a computer model of the object. (3) Three-dimensional wire-frame stereoscopic display of the reconstructed object on a storage tube terminal. (4) Computation of the volume, surface area and sphericity index of distinct features within the object. (5) Analysis of 3-D spatial distribution of structures in the organ being analyzed. The complete reconstruction and quantitative analysis of the left lung lobe from a single animal requires approximately 15 hours of computer connect time. This is sufficiently rapid that computer based three-dimensional reconstruction can be incorporated as a routine technique in experimental oncology and other areas which require detailed quantitative analysis of three-dimensional geometry.

Table Tumor No. 1 2 3

I. Quantitative

data for lung 6A ._ -.

Volume (mm”)

Diameter (mm)

Surface area (mm’)

Sphericity mdex

0.25 0.09 0.30

0.78 0.55 0.82

3.17 1.67 1.61

0.35 0.35 0.42

Interactive

3-D reconstruction

Table 7. Lung system software performance

measurements

Termmal connect time (min) I ‘0 count

1-l

1627

;

XX7 ‘77

I

PROGRAM I)rograms

Frciwald

may be obtained for details.

CPU time Is, ticks)

Memory demand tkwords x ticks)

17.10 21.51 24.17

AVAILABILITY

from the Biomedical

Engineering

Unit at McGill.

Contact

S.

~,/,~r,~u,/t~~l~/rr~rr,~t.\ WI, Mlsh to thanh Dr. R Funnell and Prof. J. L. Sibert for reading the manuscript and suggesting Improvements. We have benefited from the use of facilities in the Biomedical Engineering L’n~t at M&III Unlcersltv This research was partiallv supported by a grant from the (‘ancer Research Societ) of Montreal

REFERENCES

3. I) R. Rcddj. W J, Davts. R. B. Ohlander and D. J. Blhary. Computer analysis c,f neuronal Qructure. in ~~~rr~r~/l~/~rr Sr~rirrrn~/7~cc~hrr;q1r~\ ;,I Rrur&i,,/oql. S B Knter and C Nicholson, Ed\.. pp. 227~ 253. Springer. qew York (1973). 5 il. B~JCS) and I. Winston. A computer system for reconstructlon and display of the macrostructure ofbram [rim radiographs of seriai sections. Proc. Bloslgma ‘7% pp. 263 ‘66. Pars, April (1978). 6. I, J. F-idler. D. M. Gcrstcn and 1. R. Hart. The bl
4

.i. Mellpren. Qunntlration of metastases m expel-[mental xnlmals In ~t&~i,?lu/if~i/ ,-lix%.f.x ,J{ .1l(.f
13. I? c‘. Ahqa and S. A. C‘oons. Geometry for constructlon and display. /RR1 .S\,.si J. 7, 1X8-205 I 196X1. 15. ( I>. Barry. R A. Ellis, S. M. Graesser and G. R. Marshall. Display and manipulation m three dimensions In I’cwiwur C’om epf.\ 111C~otuptrrcv C;rqdlic .A. hl F-aiman and J. Nieverselt. Eds.. pp. 11% 153. Urbana. Linwerst! 0‘ I Ilinolz Press ( 196” 1 16. P. Fuchs. Z. M. Kedam and S. P. Uselton. Optimal surface reconstructlcln l’rom planar contours. <‘~~!?rn~~o~z ,-tr\ ( ,li,lyrl/. \/lrc~ll. 20. 603 702 (1977).

.4bout the Author SI WI I). CII \U I I recel\ed a B.Tcch. degree in electrical engineering from the Indi,m Institute of Technology. Kharagspur. lndia m 1975. and .i M.Eng. In electrical engineerin? l’rom LlcG11l Uni\ crhity m 1979. He ha> worhed for PhilIps India Ltd. aa an illumination engineer. and

232

S. D. C‘II\\\I.\.

LION GI;\~<. SI\IOY FKII\\.,\III

and J. W. PKO( I,,,<

for Washmgton Gas Llyht Co.. doing graphics software development. He is currently a Research Engineer with General Electric Co. in Charlottesville, Virginia. and is working on the design and Implementation of CAD,CAM systems. LI oh GI. \xs received a B.S. degree in chemistry from Brooklyn College in 1963 and a Ph.D. in chetmstry from the University of Chicago in 1969. Since receivmg his Ph.D. he has held appointments m the Department of Machine intelligence and PerceptIon in the Lrniverslty 01 Edinburgh. the Department of Theoretical Biology in the University of Chicago and the Department of Physics and Astronomy in the University ofRochester. Since 1975 he has been a faculty member III the Department of Physiology in McGill University where he 15 an Associate Professor. Research interests include visual perception. control of growth. and nonhnear dynamics m physiological systems.

About the Author

About the Author- -SIWW FKI.IW XLI) received the BSc. and MSc. degrees in electrical engineenng from the Technion. in Haifa, Israel in 1969 and 1974 respectively. Since 1976 he has been in the Biomedical Engineering Unit at McGill University where he is a systems engineer. He ib now involved in various applications of computers to biomedical research. .JLII.L\%W. PKOCJOK received an M.B.B.S. from London University in 1967 and ;I Ph.D. from the Chester Beatty Research Institute of London University m 1973. Since receivmg his Ph.D. he has held appointments m the Department of Pathology at Memorial University Medical School in Newfoundland and the Department of Pathology and the Cancer Research Unit at McGill University. Since 1976 he has been associated with the Radiation Oncology Division of Allegheny General Hospital in Pittsburgh where he is currently the Associate Attending Radiation Oncologist and a Senior Scientist. Research interest is mamly on processes of metastatic spread and immunological response to malignant growth. About the Authors