Measuring the human chest with structured lighting

Pattern Recognition Letters 4 (1986) 359-366 North-Holland

October 1986

Measuring the human chest with structured lighting J.R.T. LEWIS IBM UK Scientific Centre, Athelstan House, St Clement St., Winchester, United Kingdom

T. S O P W l T H Lung Function Unit, Brompton Hospital, Fulham Road, London, United Kingdom Received September 1985 Revised 20 April 1986

Abstract: Some optical techniques for the measurement of the shape of the human chest, and of how that shape changes during respiration, are reviewed. Difficulties with conventional techniques of shape recovery are discussed. A microcomputer based implementation of a new technique (SPinS) is described together with practical aspects of such a system.

Key words: Structured light, surface measurement, microcomputer, biostereometrics, human respiration.

1. Introduction

The motion of the human chest is complex. Many mechnical components are involved and damage to any of these due to injury or disease may impair breathing. In studying the respiration of such patients it is important to quantify the chest volume changes which occur. Traditional methods involve apparatus which is potentially awkward and stressful for the patient. Typically these techniques involve analysing gas flow and pressure changes during a respiratory manoeuvre. Alternative techniques are possible however. Measurement of the changes in shape and volume of the external surface of the chest provides a way of determining lung volume changes. These measurements can be made in a completely noninvasive way by projecting patterns of light onto the patient. Gourlay et al. (1984) have demonstrated the suitability o f one such technique in this particular application. There are very many optical techniques for shape Free of copyright for IBM.

recovery. Strand (1985) and Balasubramanian (1976) review of some o f these. Each technique has its own advantages and shortcomings. Three techniques which have been applied to the measurement of the shape of the human body will be examined briefly in the context of their suitability for the respiratory observations already mentioned. We shall see that none of them readily meets the requirements of a system for these observations and will examine a new method which appears to overcome the problems. Its implementation on a microcomputer will also be discussed. First it is necessary to examine the restrictions placed on the measurement system by the desire to study human respiration.

2. Restrictions on the technique

To be clinically useful the technique employed must be able to measure the surface coordinates of the human chest with reasonable precision. Typically the coordinates need to be accurate to about 1-2 mm. There must be little or no human inter-

0167-8655/86/$3.50 © 1986, Elsevier Science Publishers B.V. (North-Holland)

359

Volume 4, Number 5

PATTERN RECOGNITION LETTERS

vention required in the analysis of the images and computation of coordinates. Ideally data must be recorded all the way round the patient. A compromise solution in which data is collected from the front half of the body only is also possible. Data must be recorded in a very short time. The chest is in constant motion and recording times must be short in comparison with the time in which significant movement takes place. It must in principle be possible to extend the technique to record time sequences of data in rapid succession. Some 5 to 10 observations per second are required to allow respiratory dynamics to be studied. Finally the equipment must be reasonably inexpensive and suitable for installation within existing hospital facilities. This is a rather difficult set of requirements to fulfil. In particular the constraint on cost and the need for the system to be installed in hospitals requires that the data processing involved be carried out on a small stand-alone computer. Techniques involving sophisticated image analysis or substantial amounts of computation are effectively ruled out.

3. Moire techniques Surface shape measurements using Moire fringe techniques have become very popular in recent years. The literature is too extensive to cover here but Strand (1985) and Balasubramanian (1976) review some of it. Takasaki (1970, 1973 and 1975), amongst others, has applied Moire techniques to the study of human anatomy. Essentially, in Moire based techniques, the shadow of a grid is projected onto the object and is observed through a second grid by a camera. The effect of the surface shape of the object is to cause changes in the spatial frequency of the shadow as recorded by the camera. Since the camera views the object through another grid, beat frequencies are introduced which appear as Moire fringes in the image. The fringes correspond to depth contours on the object. The surface shape of the object can be recovered from the position of each fringe and the geometry of the projection and recording apparatus. One problem which must be overcome in all 360

October 1986

Moire systems is that of the indeterminacy of the direction of depth change. In passing from one fringe to the next, the magnitude of the depth change is completely determined but from a single observation there is no way of knowing whether the second fringe is further from or nearer to the camera. This indeterminacy is usually overcome by taking multiple observations of the object after some change in system geometry. Although Moire is suitable for a wide range of applications, particularly in industry (see for example Reid et al. (1984)), it has a number of features which make it unsuitable in the study of human respiration. The most important consideration is the need to make multiple observations after changing the system geometry in order to establish the direction of depth change. This causes difficulty in achieving the short observation time necessary when studying moving objects such as the chest. In addition, the images produced by Moire techniques are rather difficult to analyse by computer. The problems of locating and tracking the fringes are similar to those found in other techniques and indeed in other branches of image analysis. In general, extended features such as stripes or fringes are difficult to locate in images. In particular it is usually a problem to establish the continuity of such a feature across object discontinuities. Techniques for performing analysis of such features are normally very sophisticated and require a significant amount of computer resource. There are techniques for analysis of Moire patterns which avoid this kind of analysis. One such method was reported by Reid et al. (1984). Although it avoids the problem of image analysis, it requires the recording of larger numbers of images with changes of system geometry between each one. As we have seen this would be very difficult to achieve with a moving object. One final problem which Moire shares with many other techniques is that there is little prospect of being able to compress the data in the images prior to storage and processing. Although some compression is possible the amount of data involved per image is still large. This poses problems if a time sequence o f images is required, as would be the case for a study of respiration dynamics. Storage to disc on a microcomputer is too slow for the required observation rate and very few

Volume 4, Number 5

PATTERN RECOGNITION LETTERS

images could be stored in the limited amount of main memory available. It is clear that although Moire techniques are very valuable in many applications, they would pose a number of difficulties if used in the study of respiratory dynamics.

4. Rasterstereography Rasterstereography is a 'structured light' technique. It has been used by Frobin and Hierholzer (1981) in the study of the shape of the human back. A specially prepared grid of lines is projected onto the subject and recorded by a camera. The camera and projector effectively form a stereoscopic pair. By finding the image location of each intersection of the projected grid and by identifying which projected intersection this corresponds to, enough information is available to calculate the true 3D coordinates of the surface. The geometry of the apparatus must, of course, be known. Identification of the specific intersection is aided by intensifying some of the grid lines. Many of the difficulties in analysing these images are exactly the same as in Moire techniques and result from the extended nature of the features to be analysed, in this case the grid lines. Automatic analysis and identification of the grid lines has proved possible (Docter and Ensink, 1985) but requires neighbourhood searches in order to establish their continuity. The directional uncertainty of Moire techniques is not a problem with Rasterstereography so a single image is all that is required. In this respect it is possible to record data sufficiently quickly for moving objects to be studied. However, as with Moire, analysis of the image is rather complex and there is little opportunity for data compression prior to that analysis, making it difficult to record sequences of observations in rapid succession. In its current form it would be difficult to use this technique for the desired measurements.

5. Light striping Gourlay et al. (1984) reported a technique for surface shape measurement which has a number of

October 1986

similarities with Rasterstereography. The geometry of projection is different in that the camera and projector are at right angles to one another. Also, the technique was devised to allow simultaneous observation of the whole o f a patient's trunk. The method was developed in order to study respiratory dynamics and can be considered the forerunner of the SPinS technique. In essence, a set of coded, vertical stripes of light is projected onto the patient from in front and behind and is recorded by a pair of cameras placed at right angles to a line joining the projectors. By identifying and tracing out manually each stripe in the camera images, data is generated from which the position of the surface can be derived. As with Rasterstereography this is essentially a triangulation technique. Once again the problem is that of correctly identifying specific extended image features, in this case the stripes. As we have seen, automation of this process is difficult and requires significant computer resource. Indeed it was this very problem with the method which prompted development of the new technique. As with Rasterstereography, only a single image is required so that the recording time is short enough to allow studies of dynamics. However the problem of data compression once again limits the ability to record sequences o f data.

6. SPIES

In summary, the features of the 3 techniques already discussed which cause difficulties for the human respiration study are all much the same. They are due basically to the inherently complex nature of the images produced. A new technique for surface shape measurement which avoids complex images has recently been reported (Lewis and Sopwith, 1986). It will be outlined here for completeness and in the following section a microcomputer based implementation will be described. The technique has been given the acronym SPIES, standing for Spot Projection and Imaging for the Evaluation of Shape. In the SPIES technique an array of bright spots of light is projected onto the object and recorded, in the simplest arrangement, by a pair of cameras ar361

Volume 4, Number 5

PATTERN RECOGNITION LETTERS

ranged stereoscopically. The layout is shown in Figure I. If the system geometry is known and the corresponding images o f a given projected spot can be identified in the pictures from each camera it is possible to determine the 3D coordinate of the surface at one point by triangulation. Repeating this for the whole array of projected spots given a set of surface coordinates for the object. Intermediate coordinates can be estimated by interpolation. The key to the technique is the ability to identify the corresponding images of the same projected spot in the two camera pictures. This is one specific example of the so called 'correspondence problem' which must be solved in some way or other in all stereoscopic techniques. In his book 'Vision' Marr (1982) discusses the issues involved with stereoscopic matching of normal grey level images. The images in SPIES, however, are rather special and we can make use of inherent geometric constraints to solve what would otherwise be a computationally complex problem. Each camera image may contain

//\ o

S -

T _

///

?

q;

Figure 1. A structured ligth system with three optical components instead of the more conventional two. Cameras (C) are placed symmetricallyeach side of a projector (P) and view object (R). Also shown are the base plane projections (S) and (T) of object point (R). The y axis is vertical and it and the x axis lie in the base plane. Point (O) is the origin of the coordinate system. Dx and Dz are the magnitudes of the distances, in x and z respectively, of the optic centre of each camera from the origin. The broken lines are the optic axes of the cameras. The optic axis of the projector lies along the z axis. 362

October 1986

several hundred spots of each which must be paired correctly. In addition because the cameras view the object from different viewpoints, not all of the spots visible in one camera can be seen by the other. Difficulties with the apparently attractive idea of encoding the spots by shape or colour have been discussed by Lewis and Sopwith (1986). The first geometric constraint which can be employed is the familiar 'epipolar line' constraint (see for example Sandini et al. 1984). This constraint is solely due to the stereoscopic geometry. The result is that any particular point in one image has a corresponding point in the second image which lies on a particular line, the point's epipolar line. The equation of this line can be determined from the stereoscopic geometry and the first point's coordinates. The advantage of this constraint for SPIES is that only those spots, in the second image, lying on a given spot's epipolar line need be considered as its potential matches. In general, however, two problems remain. First it is unusual for a given epipolar line to pass through only one spot, thereby giving an unique match. It is much more usual for there to be several potential matches lying on the line. Some means must be found for identifying the correct match from among this, admittedly now small, set of points. Second, epipolar lines are not in general parallel to the images axes or to each other when the camera axes are convergent as they are in SPIES. This makes a search for points lying on the line potentially time consuming. A technique which greatly simplifies the search was suggested by Gourlay (1984) and has been described by Lewis and Sopwith (1986). The details need not concern us here except that to solve the problem it was necessary to introduce the idea of a 'base plane'. This is the plane z = 0 (Figure 1) and the technique involves a mathematical projection of the spots in each camera image into the base plane. On such pair of projections for the object point R is shown in Figure 1. The simplification comes from the fact that the projections of epipolar lines in the base plane are parallel to the x axis. Potential matches are therefore those points whose base plane projections have equal y coordinates. The y axis in Figure 1 is perpendicular to the page. There is one interesting practical consequence which becomes apparent f r o m this form of the epi-

Volume 4, Number 5

PATTERN RECOGNITION LETTERS

polar line constraint. If potential matches are recognised by similarity of the y coordinate of their base plane projection, it is obviously advantageous to reduce the number of potential matches which occur by chance due to different projector rays having the same y base plane coordinates. This can be achieved by using a projected array of spots in which horizontal rows are avoided. Even when the epipolar line constraint has been applied, there is still uncertainty about matching. In conventional stereoscopic techniques, some f o r m of relaxation or correlation based method would be employed at this stage to determine the correct match. Marr (1982) describes some possible approaches. Usually these techniques are computationally intensive and not suitable for use on a microcomputer. However, because SPIES employs 3 optical components, having a projector in addition to the 2 cameras, it can utilise another geometric constraint, determining uniquely almost everywhere which matches are correct whilst avoiding the complex computations usually associated with stereoscopy. In those cases where ambiguity remains it is detectable and no attempt is made to compute a coordinate. Interpolation across ambiguous points can be employed so long as they are a relatively infrequent occurrence. The additional constraint is based on knowledge of the geometry of projection of the spot array. This can be conveniently determined in an independent experiment using a plane object placed in the base plane (Lewis and Sopwith, 1986). Effectively, the equations o f lines joining the centres of each of the projected spots to the projector are determined. These lines will be referred to as projector rays. The projection geometry is used as a test of reasonableness of each potential match. Each spot lying on a given spot's epipolar line is used in computation of the 3D coordinate which it would represent if it were indeed the correct match. If this 3D point really is the correct one there must also be a projector ray which passes through it. The projector rays are examined in turn to see if any meet the condition for any potential match. In practice there is almost always only one potential match which satisfies this requirement. The projector ray is allocated to this match. There is a small probability that, due to 'malicious' alignment, an

October 1986

incorrect potential match m a y lie by chance on a projector ray. In this case it will normally be the true either that more than one potential match of the given spot satisfies a projector ray constraint, or that a match involving a different spot altogether requires the same projector ray. In either case the ambiguity is explicit. There still remains the very unlikely situation in which the match due to 'malicious' alignment is the only potential match which ies on a projector ray and that no other matches require that ray. In this case there is no explicit ambiguity. In practice this situation has not arisen and in any case it can be made detectable by suitable projection array design and by the inclusion of a surface smoothness constraint on the final 3D coordinates. The expressions for determining the 3D coordinates of a surface point are given below just to show their simplicity. They are formulated in terms of the base plane projection coordinates of the image points considered to be a correct match. A full derivation has been given elsewhere (Lewis and Sopwith, 1986).

( Dx(x1 +x2) x= \2Dx +xl-XzJ' \ xl+lJx / z=Dz(l-Y). Here (x,y,z) are the coordinates of the surface point, x~ and x 2 are the base plane projection x coordinates of the image points considered to be a match and y~ is their y base plane coordinate. Remember that this is the same for both points. The quantities Dx and Dz relate to the positions of the cameras and are shown in Figure 1.

7. Prototype implementation The feasibility of the technique has been established with a prototype apparatus constructed f r o m simple experimental equipment and using conventional film cameras (Lewis and Sopwith, 1986). An example reconstruction of a cylindrical surface is shown in Figure 2. 363

Volume 4, Number 5

PATTERN RECOGNITION LETTERS

Figure 2. Reconstructed surface from a cylindrical object. The spots represent reconstructed coordinates. They are joined by linear interpolation. Line intersections without spots are placed where no coordinate could be determined with the available data.

<3 A

RAM

CPU

I

DISK

I

DISPLAY

Figure 3. The IBM PC based prototype system. The object being observed in A. The cameras are C1 and C2. FGI and FG2 are the frame grabber cards and each has an associated TV monitor M1 and M2. P is the projector. 364

October 1986

A purpose built microcomputer based system is now nearing completion. It is outlined in Figure 3. A Kodak Carousel 35 mm slide projector is flanked by a pair of Fairchild CCD3000 solid state video cameras. These have a resolution of 380 × 488 pixels which is typical of modern devices. Solid state cameras where chosen for this system as they have superior stability and geometric precision when compared with conventional television cameras. Currently, however, their resolution is a little lower than would be ideal. An Imaging Technology PCVISION frame grabber card is used to digitise the data from each camera. These cards can capture a complete frame of data from each camera in 1/30th of a second. The cards are housed in an IBM P C / X T computer with 512K of random access memory and 20M bytes of fixed disk storage and running under the PC-DOS 3.0 operating system. Once a frame of data has been captured it can be stored on disk or processed directly. The optical components of the system are mounted on a 2 metre optical bench to allow precise positioning. The software for the system has been specially written. The ' C ' language has been used extensively with 8088 assembly language employed where necessary. Currently the software is designed to support research and development of the techniques but it is modular to allow easy incorporation of the existing functions into future applications. Figure 4 outlines the major software components. The controller module oversees system execution. The user interface allows the operator to interact with the system via a series of menus and to control both the cameras and the processing to be performed. Processing starts once images have been captured on both cameras. The first task is to extract the image plane coordinates of each spot in the picture. This is performed in AP001. A local search is performed to find all the pixels constituting a particular spot and their centre of gravity is taken as the spot's location on the image. The resulting list of spot coordinates is projected into base plane coordinates in AP002. AP005 uses the resulting coordinates and implements the epipolar line constraint to produce the list of potential matches for each spot. Finally AP006 applies the projector ray constraint to these potential matches and computes the 3D coordinates for the unam-

Volume 4, Number 5

PATTERN RECOGNITION LETTERS

October 1986

C6 ~

\ CONTROLLER right image base plane coordinates

projector ray equations

P6 -...~ C1o

I APO01

APO02

r

left image Icoordinates

APO05

left pl image base ane coordinates

P5

C5 ~,/P4

~

• C4

APO06

~

P1 ~

/ C2

list of potential

biguous points. The purpose of this apparatus, apart from demonstrating the feasibility o f using a microcomputer for these measurements, is to investigate various system geometries to find those most appropriate for specific applications. It is not possible to record data from all the way round an object with just 2 cameras. However Figure 5 shows an arrangement which could do this. In this arrangement each camera participates in two stereoscopic pairings, one with the camera to its left and one with that to its right. There are difficulties with such a system, not least o f which is the cost o f implementation. The PC based system will be used to evaluate the effectiveness o f alternative geometries which might require fewer components. It is clear that the images produced by the SPIES system are capable o f significant data reduction. Most o f the image area is blank since the spots are small in comparison with the distance between them. At least in principle, then, it should be possible to record many frames o f data within the m e m o r y space available in a typical microcomputer. Indeed a scheme using thresholding and run-length encoding applied directly to the video signals from the cameras is under consideration.

~ P3 C3

P2

matches

Figure 4. Arrangement of software modules which implement the SPIES process. They are described in the text. Arrowed lines indicate data flow, the others control flow. The resulting 3D coordinates are represented by the large arrow leaving AP006. Up to AP005 only the left image processing is shown. Processing for the right image is identical.

i ~

Figure 5. A . . . . !ble arrangement for recording the whole surface of a roughly :.~_:i.:~!ricalobject. The cameras are C I to C6 and the projectors PI to P6.

8. Concluding remarks

The requirement to observe of changes in the shape of the human chest during respiration introduces a number of restrictions which make existing surface measurement techniques difficult to employ. A new approach, the SPIES technique, is being implemented on a microcomputer based system to allow evaluation of its usefulness in a variety of situations in which surface shape measurements are required.

References Balasubramanian, N. (1976). Comparison of optical contouring methods. Photogrammetric Engineering and Remote Sensing 42, 115-120. Docter, G.J. and J. Ensink (1985). Automatic measurement of body shape by means of raster stereography. In: M. White and D. Harris, Eds., Biomechanical Measurement in Orthopaedic Practice, Clarendon Press, Oxford. Frobin, W. and E. Hierholzer (1981). Rasterstereography: a photogrammetric method for measurement of body surfaces. Photogrammetric Engineering and Remote Sensing 47, 1717-1724. Gourlay, A.R. (1984). Private communication. Gourlay, A.R., G. Kaye, D.M. Denison, A.J. Peacock and M.D.L. Morgan (1984). Analysis of an optical technique for lung function studies. Computers in Biology and Medicine 14, 47-58. 365

Volume 4, Number 5

PATTERN RECOGNITION LETTERS

Lewis, J.R.T. and T. Sopwith (1986). Three dimensional surface measurement by microcomputer. Image and Vision Computing, in press. Mart, D. (1984). Vision, Freeman, San Francisco. Reid, G.T., R.C. Rixon and H.I. Messer (1984). Absolute and comparative measurements of three dimensional shape by phase measuring moire topography. Optics and Laser Technology, 315-319. Sandini, G., M. Straforini, V. T0rre and A. Verri (1984). 3D reconstruction of silhouettes. In: A. Pugh, Ed., Proc. 4th lnternat. Conf. Robot Vision and Sensory Controls, 173-182.

366

October 1986

Strand, T.C. (1985). Optical three dimensional sensing for machine vision. Optical Engineering 24, 33-40. Takasaki, H. (1970). Moire topography. Applied Optics 9, 1457-1472. Takasaki, H. (1973). Moire topography. Applied Optics 12, 845-850. Takasaki, H. (1975). Simultaneous all-around measurement of a living body by Moire topography. Photogrammetic Engineering and Remote Sensing 41, 1527-1532.