- Email: [email protected]
A model for compression and classification of face data structures
Cornput.
Vol. 20, No. 6, pp. 863479, 1996 Copyright 0 1996 Elswier Science Ltd
& Graphics,
Pergamon
Printed in Great Britain. All rights reserved 0097-8493/96 $15.00 + 0.00
PII: soo97-8493(96)ooo57-x Technical Section
A MODEL
FOR COMPRESSION AND CLASSIFICATION FACE DATA STRUCTURES
OF
JEAN PAUL GOURRET+ and JAMAL KHAMLICHI Laboratoire d’Informatique et d’Imagerie Industrielle UniversiM de La Rochelle, Equipe Images de SynthZse, 15 Rue F. de Vaux de Fouletier, 17026 La Rochelle Cedex, France e-mail: [email protected] Abstract-In this paper we present a 3-D model with physical properties which simplifies the analysis and the synthesis of deformable faces and solids. Our model presents three relevant particularities. First, it describes the external envelope of faces with 1-D finite elements assembled with a new 3connected mesh topology. Second, the mesh deformations are analysed with a modal analysis. Because our model associates these two particularities, the number of rigid modes given by the modal analysis is equal to the number of 1-D finite elements, which is also the half of the number of Degrees of Freedom @OF). This number of rigid modes is a basic characteristic of our model. The second half of modes constitutes the nonrigid modes. Third, we use these rigid modes and the iirst nonrigid mode to synthesize a mean face named a photo-fit identikit or class, around which we synthesize face varieties by action on secondary nonrigid modes. Our physical 3-D model allows compression of face data structures because a greater number of secondary nonrigid modes can be suppressed to define a class or its varieties, and because the synthesis of varieties does not need more information storage than their classes. Our physical 3-D model allows classification of face data structures because we can associate an objective measure to each synthesized face. We can measure the deformation between a variety and its photo-fit identikit. Copyright 0 1996 Elsevier Science Ltd
1. RWRODUCTION must easilyloosenwhen a fast but rough solution is In structural mechanics the study of physical needed.The meshmay not changesize abruptly to deformations is generally achieved with the finite insurefast calculation and good accuracy. For these elementmethod [1, 21. This method is also usedin reasons,we developed a structured homogeneous
computer
graphics
for
shape
and/or
animation
mesh with
3-connected
vertices and with
links of
realism[3-51. In the 3-D space,the method defines approximately equal length. To insure thesecondiobjects
as an assemblage
of I-D,
2-D
or 3-D
elementswhich are linear, quadratic, cubic and so on. We use the finite
element
method
but
we
tions, the vertices of our model are generated on a sphere. For the modal analysis, the sphere represents a face or an unconstrainedsolid, i.e. with zero energy
intentionally restrict our model to 1-D linear finite elements(namedbars or deformablelinks) which are
of deformation, and constitutesthe initial condition of the numericalprocess.To obtain a meshon a face, 3-connected (a vertex is connected at the ends of the sphere vertices are projected on a known three bars). The 1-D finite elements cover the digitized 3-D face, or we apply forces on the sphere external envelope of the faces. This may seen to deform it. The typical method to put a grid on a regressivecomparedwith sophisticated2- and 3-D sphere is shown on Fig. l(a). The mesh follows modelsdevelopedtoday in computer graphics,but it parallel and meridian lines and creates triangular allows many advantages when we use a modal facets on the envelope. This kind of meshis often analysis for deformation calculations. With 1-D used in CAD, in image analysis, and in image finite elements3-connected,the modal analysisgives synthesisbecausethe facetsare alwaysplanar. But it a number of rigid modesequal to the number of 1-D is not suitable for our problem. Indeed, it is not finite elementsassembled to definethe face.The rigid structured because the number of neighbours for the modes combined with the first nonrigid mode two pole vertices is not the same as that for other synthesize a mean model of the face named a vertices.Near theseextraordinary points [l, 6, 71the photo-fit identikit or class, around which we can deformation calculation is not homogeneous,and a create varietiesby action on the secondarynonrigid systematic recursive subdivision of facets is difficult modes. becausethere is a loss of connectivity properties. An efficient meshmust be easily tesselatedwhen Moreover, the number of links is over-dimensioned good accuracy and/or
visual realism is needed, and
giving
an object
with
6 rigid
modes,
which
is
unsuitablefor our problem (seeSection 3). The sphere mesh is detailed in Section 2. It is a new mesh when applied to deformation calculation and measurement.
+ Author for correspondence. 863
It describes solids and faces with a
J. P. Gourret and J. Khamlichi
864 Pn
(4
(b)
Fig. 1. Mesh definition on a sphere. (a) typical, (b) chosen.
sphericaltopology and offers severaladvantages.It does not create extraordinary points and it can be tesselatedwith the sametopologic propertieswhatever the level of complexity of the face, dependingon the accuracy, time of calculation, memory size, etc. Moreover, we can associatean objective number to each synthesizedface. This number is a measureof the energyof deformation betweenthe variety and its photo-fit identikit. This approach allows sizereduction and manipulation simplification of face data structures. It is directly tied to our projects of face compressionfor transmissionin the field of multimedia, and face recognition in the field of cognitive sciences. For brevity we do not describethe representative works on face modelingwhich can be found in [8]. Among them, the works of Pentland are of interest becausehe is concerned with face recognition by modal analysis.However, his works are donein 2-D space for recognition and interactive search in a large-scaledatabase,[9, lo] or in the 3-D spacewhen facesand solidsare modelledwith the typical mesh shown in Fig. l(a) or with an assemblageof sophisticatednonlinear patches[ll]. With a typical I-D mesh or with 2-D element assemblage,it is impossibleto defme the photo-fit identikits introducedby our modelbecauseonly 6 rigid modesexist. The formulation presentedby Pentland [l l] and Nastar [12, 131usesthe formulation developedby Bathe [2]. Bathe and other researchersin the field of structural mechanicsdeveloped modal analysis to study the structure vibrations with external excitations. Modal analysis solves an eigensystemfor decouplingthe equationsof motion. Each eigenvalue and associatedeigenvectoris a vibration modeof the structure, i.e. characteristic distortion when all vertices vibrate with the samefrequency and the samephase.Pentland and Nastar usethis method for data compression.They combine vibration modes
and extract qualitative modes such as translation, rotation, scaling, shearing. But there exists only 6 rigid modes(3 translationsand 3 rotations) which do not allow the definition of the photo-fit identikits. 2. MESH DEFINITION
We define the meshshown in Fig. l(b). To obtain the mesh we start from a dodecahedron. The dodecahedronis the low level of tesselation,named level 0, which presentsthe two criteria (3-connected and links of equal length), and whose vertices are situatedon a spheresurface.The dodecahedronis a regular polyhedron. Then we construct a truncated icosahedron (named level l), which can also be termed as a semiregularpolyhedron made up of 12 pentagonsand 20 hexagons. Note that a regular polyhedron is characterizedby the samenumber of neighbours per vertex and with the same facet element.A semi-regularpolyhedron is characterized by the samenumber of neighboursper vertex and with different facet elements.An identical subdivision method is applied to transform level 1 to level 2 and more generally, to transform level n to level n+ 1. For levels n (n > 1) the sphere is made up of ‘pentagonal’ and ‘hexagonal’ facets giving a semiregular ‘polyhedron’ but with nonneeessarilyplanar facets.We put the wordspentagonal,hexagonaland polyhedron in inverted commasbecause,after ievel 1, the facetsare not planar. Moreover, the links are approximately of equal length. Each level is madeup of 12 ‘pentagons’ and of a growing number of ‘hexagons’.The ‘pentagons’ensurethe curvature of the space. The subdivisionmethod is shown on Fig. 2. For the level n (shownas a bold outline in Fig. 2), each vertex S,-,has three neighbours,Si, &, S, oriented clockwisewith respect to the facet normals and in a left hand coordinate system.The verticesdefinethree
Face data structures
865
(4 Fig. 2. Subdivision method.
edges S& (i= 1, 2, 3) from which three new edges TioTil are created [Fig. 2(a)]. The three new edges join the centroid of the ‘pentagons’ and/or ‘hexagons’ which share the vertex Se and their length is the third of the distance between centroids. The three new edges build a new ‘hexagon’ which is not situated on the sphere surface. To obtain a new ‘hexagonal’ facet of level n+ 1, we must do a radial projection of the vertices on the sphere surface. The subdivision process for a ‘pentagon’ is shown in Fig. 2(b) and the subdivision process for an ‘hexagon’ is shown in Fig. 2(c). Note that for n > 1, the links are different because the centroids joining two ‘hexagons’ and the centroids joining a ‘pentagon’ and an ‘hexagon’ are not of equal length. To define the level 0 we could start from three kinds of regular polyhedrons with three neighbours per node: the dodecahedron whose facets elements are pentagons, the cube whose facet elements are squares and the tetrahedron whose facet elements are triangles. The subdivision process applied to the three kinds of polyhedrons gives the successive structured levels shown on Fig. 3. This is evidence that the dodecahedron gives the more homogeneous mesh. The mesh insures the Euler relationship N-E + F= 2, where N is the number of vertices, E is the number of edges (bars) and F is the number of facets (‘hexagons’ or ‘pentagons’). 2.1. iUoaiz1 analysis with I-D finite elements An object in the 3-D space submitted at time t to external forces ‘R is subject to displacements ‘a. After transformations proper to the finite element method the object is modeled by an assemblage of elements made up of N interpolation vertices or nodes, and 3N equations of motion describe its behaviour:
g’Q+g’O+g’g=
‘R
(1)
‘_V is the displacement vector of size [3Nx I]
expressed at time t. The size is 3N because in the 3-D space and with a 3-D Cartesian system, a deformable body has three degrees of freedom per node, ‘0 and ‘D are the velocity and the acceleration vectors, *R is the external force vector, 4 is the stiffness matrix of size [3Nx3N] for internal elastic force calculation, _C is the damping linear matrix of size [3Nx 3iVJ, _M &the mass matrix of size [3Nx3N]. These ma&es are supposed independent of displacements. We choose 1-D elements that are elastic bars. They can be translated without deformation, compressed or stretched. Their assemblage is done through the first transformations of the graphics pipe-line as described in [8]. Usually the g, g and & matrices have a large bandwidth because the 3N DOF are coupled. Node numbering tricks can be used for bandwidth reduction of the sparse matrix [14], but the best method consists to write the displacement vector ‘.g ‘_U=$‘O
(4
Where 2 is a [3Nx3N] matrix which diagonalize & C and g giving a canonical system with each DOFdecoupled. ‘4 is the generalized displacement vector expressed in the basis of eigenvectors & where a is the ith column of 2. A system with 3N DOF has 3N free vibration modes. For each mode, displacements *_U are all oscillating with a frequency e+/2K and a phase I#Q. It is an eigenproblem with eigenvalues CL$and eigenvectors gi which gives 3N solutions:
For simplicity, we suppose that the bars have distributed mass. The mass mi is lumped node i, and we choose mi= 1 such that g identity matrix and the eigenvectors are
do not at each is the ortho-
J. P. Gourret and J. Khan&hi
Level 0
Cube
Dodecahedron
Level 1
Level 3
Fig. 3. Successive levels of subdivision starting from three regular 3-connected polyhedrons.
normal. With these assumptionswe can write the equation of motion in the spaceof eigenvectors:
1%+gc +gij= ‘3 ‘&=gT‘IJ
(4)
(5)
The matrix of eigenvaluesQz is a diagonalmatrix of size[3Nx3Nj with wf on iG diagonal, c is also a diagonalmatrix whenE is a linear combin&on of g and K.
The 3N DOF are decoupledand we can write 3N equationsin the form: ‘& +2rio,‘Gi
+wf’tii=
‘&fori==1,2,3,...,3N
(6) Each
DOF
vibrates
with
the
pulsation
w = Oir- 1 - # where & is the damping coefficient. The solution when a constant force is app4iedfrom time t = o can be written:
Face data structures ‘g=--!
867
co;cos#J1 (7) Of.
e-Cw -cos(wt
l-
+ 9)
[
Without initial velocity ( ‘ii = 0) we obtain: C#J = 0 when ti = 0
(8)
when[i # 0
(9)
More detailson matrix and vector generationfor our theoretical model are developedin [8, 151.
2.2. Distinctive features of modal analysis associated with our mesh
The modal analysis is identical to the Fourier analysiswhich decomposesa signalon the basisof complex exponential functions, and superposesall the componentsfor synthesis.With Fourier analysis the high frequency components such as 2 wi2 > wIhreshold contain little energy and can be suppressedby low pass filtering. In this way, the synthesisof the signalis doneby superpositionof the first components.For deformablefaces,we can also suppresscouples (o;, cbi) such as i> ithr&& and obtain an incompletesuperpositionof modes.In this way the calculation time for synthesisis greatly reduced becausethe size of g is reduced.The first deformation modesare rigid body modeswith null eigenvalues.
In [16] we discussa simplemodelin the 2-D space when2-D elementsand I-D elementsare used.In the 2-D spacethere are 2N DOF. When the object is modeledby 2-D elements,the numberof rigid modes equals3 (two translationsand one rotation). When it is modeled by 1-D elements,the number of rigid modesvaries between3 and 2N/2 dependingon the number of 1-D elementsin the assemblage. In the 3-D space,when 2-D elementsmodel an object envelope, there are 6 rigid modes (three translationsand three rotations). When I-D elements are used,the numberof rigid modesvariesbetween6 and 3N/2. In Fig. 4(a) we showa simpleobject made up of I-D elementsin the 3-D space.The cube is madeup of 12 1-D elements(bars) and N= 8 nodes. In the 3-D spacethere exists 3N=24 DOF and 12 rigid modesbecausethe nodesare 3-connected.Each rigid modeis a translationwithout deformation of a bar, and is associatedwith a null eigenvalue.For recoveringa 3-D object immersedin the 3-D space, the number of rigid modesmust be equal to 6, and the number of nonrigid modes is consequently 24- 6 = 18. To obtain this we can add somebars to stiffen the cube. For example, when we add a bar betweennodes 1 and 3, the eigenvaluecalculation showsthat the cube is madeup of 12- 1= 11 rigid modesand 12+ 1= 13 nonrigid modes [Fig. 4(b)]. The added bar suppresses a rigid mode which is replacedby a nonrigid mode. More generally, each bar addedsuppresses a rigid modeand replacesit by a nonrigid mode.It is necessaryto add a minimumof 6 bars to obtain the 6 rigid modes.Figure 4(c and d)
6
7, X
12MR 12MNR
(4
6MR 18 MNR
(b)
11MR 13 MNR
6fiR 18MNR
AR 11 MNR
Fig. 4. Numberof rigidmodes(MR) andnonrigidmodes (MNR) for a cubemadeup of 1-Delements in the3-D space. C&G 20:6-E
868
J. P. Gourret and J. Khamlichi
shows two possibilities. The first by addition of a strengthening bar on each facet of the cube. The second by addition of strengthening bars along internal diagonals of the cube and on its envelope. The eigenvalue calculation when we add more than 6 bars as shown in Fig. 4(e), does not modify the number of rigid modes which stays equal to 6. Consequently, the optimal number of bars is equal to 3N-6 for an object immersed in a 3-D space. It is 2N- 3 for an object immersed in the 2-D space [16]. The eigenvalue calculation for the ‘cube’ of Fig. 4(f) when the bar joining nodes 2 and 6 is suppressed gives 13 rigid modes and 11 nonrigid modes. But in this case the object is not a solid because it contains dangling faces. The internal volume is not enveloped in a closed surface and the Euler relationship N-E+ F=2 is not verified. To define a solid object Requicha[17] introduces r-sets. An r-set object is a subspaceof the 3-D Euclidean spacewhich is bounded, closed, regular and semianalytic. Bounded becauseit occupiesa finite portion of space. Closed becausethe object contains its boundary. Regular becauseit equalsthe closureof its interior. Semianalyticbecauseit can be expressedas a finite Boolean combination of sets defined by analytic functions. An r-set object can always be transformed into a boundary representation with triangular facets, and the assemblageof r-set objectsis also an r-set object provided regular boolean operators are used [17]. Following our discussionabout the cube of Fig. 4, we add to the definition of a r-set that: -for a r-set the number of rigid modesin the 3-D spacevaries between6 and 3N/2,
-the optimal mesh developed for our model is a r-set which insures 3N/2 rigid modes and a 3N/2 nonrigid modes whatever the level of discretization. To obtain a spherewith 6 rigid modeswe must add 3N/2-6 bars. A meanfor this is to triangulate the sphereas shown in Fig. l(a). But it is not an optimal method becausethe relationship E= 3N- 6 is not verified. The number E of added edgesis greater than 3N-6. Optimal or not optimal objects with 6 rigid modesare namedinfinitely rigid objects and others objects are named rigid objects because they can produce infinitesimal deformations when excited. In a 2-D space,the optimal threshold is given by E=2N-3. For example, the grid in Fig. 5(a) is made up of 40 bars and 25 nodes.It can be infinitely rigid by adding a diagonal bar for each square[Fig. 5(b)], but the number of bars doesnot give the optimal solution. The optimal solution needs the addition of 2N-3 =7 bars. Crap0 [18, 191 showedthat a rigid grid in the 2-D spacecan be made infinitely rigid and optimal when we add diagonal bars at right emplacements.Each square has a horizontal and vertical addressas shown in Fig. 5(c, d). The grid is infinitely rigid when the graph createdby diagonal bars is connected,that is whenthere existsa link for connectingany addressto any other. In the 3-D spacealso,the optimal solution does not need a complete triangulation of the surface. Our mesh[Fig. l(b)] is under-optimal and the typical mesh[Fig. l(a)] is over-optimal. Our mesh createsa rigid objectwith 3N/2 rigid modesand 3N/2 nonrigid modes.
(a)
(b)
Ll
Ll
Cl
L2
L2
c2
L3
L3
c3
L4
IA 72
c4
cc>
(d)
Fig. 5. Optimalplacement of strengthening barsin 2-D space.
Face data structures 3. RESULTS 3.1. Creation of a reference face
A referenceface can be.obtainedfrom three ways: (a) The face is obtained from a 3-D scanner.In this paper each scannedface contains 600 meridians and approximately 280nodesper meridian giving approximately 168,000nodesper face. To define the new mesh of level II we put in coincidence the center of the sphere of level n with the center of the scannedface, and we do a radial projection of the spherenodeson the face, including the planar base[Fig. 6(a)]. (b) The face is directly created by application of external forceson the nodesof the sphereof leveln (Fig. 7). (c) The face obtainedfrom (a) is partially modified by applying external forces at nodes. In the following, we study 4 headsnamedheadl, head3, head8 and head9 whoselevel 7 is shownin Fig. 6(b). We usea Gouraud shadingafter triangulation of the ‘pentagonal’and ‘hexagonal’ facets. To triangulate we define5 trianglesper ‘pentagon’and 6 triangles per ‘hexagon’ with a common vertice situated at the centroid. 3.2. Analysis of a reference face
The analysisof a referenceface of level n needs two steps.
Level 3
869
The first step is common to all referencefaces.It createsthe matrix g (M and g if needed),solvesthe eigensystemand deducesthe couples(w~,~) in the increasing order. This step is a pre-processing becauseit is sufficient to know the internal characteristicsof the deformablesystem(i.e. the sphereof level n). Matrices Q2 and g are stored directly or after compression,that is after suppressionof the high frequency couples.The time of calculation can be very long but the calculation must be done only oncefor eachlevel [111.It takes4 h on a HP9000/715 for level 3 and with a Jacobi method. To reducethe time for levelsup to 3, we presently develop these calculationson a clusterof HP9000/725workstations interconnectedwith a 100Mb/s network using the ParallelVirtual Machine (PVM) tools. The secondstep takes into account the external forces O&appliedat the verticesof the sphereat time 0. Thesecalculationsare doneduring the secondstep becausethe external forcesare not neededduring the first step. The two stepscorrespondto the solution of the homogeneous equation of motion without the right hand side,and to a particular solution obtainedwith the right hand side.In the secondstep we calculate the vector “4 with Eq. (5), then the componentsof the vector ‘_Vwith Eq. (7), then we superposethe increasing modes (eventually without the high frequencymodes)with Eq. (2).
Level 4 Fig. 6(a). Fig. 6.--continued overleaf
Level 5
J. P. Gourret and J. Khamlichi
870
Head 3
Head 1
Head 9
Head 8 Fig. 6(b).
Fig. 6. (a) Levels 3,4, 5 of the mesh for a reference face (head 9), (b) Level 7 for the reference faces under study.
Face data structures
871
Fig. 7. Application of external forces on the sphere (level 2).
The strain energy can be obtained from E = i UT_ U or from E = iaTg2 a. It is the energy of nonrigid modes(MNR) becausethe pulsationsof rigid modes(MR) are ze_ro.So, it is the strain energy relative to a rigid face Qgid madeup of rigid modes only and whose MNR are zero. The vector for visualization is given by Eq. (2):
containstwo sub-vectors&,olMR and 0 of size [3iV/ 2x l] each.The vector a,,, containsthe sub-vector alarMR of size [3N/2x 11,the first componentof the sub-vector &,a,MNR and a zero sub-vector of size [((3iV/2 - 1)x 11.We show in Fig. 8(a) the reference face a,,, (or more preciselyL& = @IL*,,), in Fig. 8(b) the rigid face &+,, and in Fig. 8(c) the class a,,,. Any modification of the components&.,MNR &id = g -@rigid (10) i ff L, or any modification of the zero sub-vector in A referenceface obtained with one of the three &,, gives a variety &,, of the face around the methods given in Section 3.1 will be noted e,O,. photo-fit identikit. A variety &or is shown in Fig. From a,,, we deducethe rigid face aiKti and the 8(d). In the following we considerfacesat level 3, that is photo-fit identikit alsonameda classof faces&,,. The vector ata, contains two sub-vectors&a,MR faces with 540 nodes, 1620DOF, 810 MR and and &MNR of size[3N/2x I] each.The vector aigid 810MNR. We first justify the suppressionof the
(a)
(b)
Cc)
Cd)
Fig. 8. Face “head 9” and level 3. (a) reference face, (b) rigid face, (c) photo-fit identikit (class), (d) a variety.
J. P. Gourret and J. Khamlichi
872
high frequency modesand the choice of the class the possibilities of our physical model and to vector noting that we can write eachcomponentof calculate the modifications of the componentswith Eq. (7). For examplewe calculate the solution ‘iii Eq. (2) as: when initial conditions are Oui= 0, Olii= 0 and 3NI2 when we apply an external force O& at time t =O. ‘Ui=C@~fCj+ 2 (11) The force is maintainedfor all time t > 0. Doing this, j=l j=(3N/2)+1 we definea variety for eachvalue of the parameterI m is the number of kept MNR. The choice of which hasthe dimensionof a time becauseour model m < 3N/2 suppresses the high frequenciesand allows is a physical model. The initial condition is the to visualize a smooth shape of the face without sphere.At t=O the sphereis deformedand take the details.In Fig. 9(a) we showthe energy versusm for appearanceof the photo-tit identikit. Then for each the referenceface ‘head9’, when m varies from 1 to parameter value At the face evolves and takes the 810.When m is about 20 (that is 830modesincluding appearanceof a variety. The successivevarieties the 810MR), the faceis visually nearby a,,, and the oscillatearound the referenceface atat with a period energy is approximately stable becausethe main T . The dampingis controlled by the coefficient &. energy variation is comprisedinto the first nonrigid W%!n &= 0 no damping occurs and when &#O the modes.Pentland et al. [9] obtain a similar curve for synthesizedvarietiesconverge towards the reference face recognition in the 2-D space. The curve face after a few periods. For this reason,we note a presentedby theseauthors givesthe recognition rate referenceface with the subscriptstat. We obtain an versusm. It is interestingto note that the recognition infinity of varieties when t evolves with arbitrary rate seemsdirectly tied to the energy measureused stepsAt. For obtaining an integer number M of for our model. We obtain similar curvesfor the four varietieswe must choosea stepAt = IL where 7,,.r xM head under study [Fig. 9(b)]. So, we choosem=20 is the maximal period and we redefinethe face vectors as: ( add a phase: @‘U’ji j
(13) with -T +,m$
u
=
x %at, 1
_ ... ustat 4
u”t”t(+,)
0
‘..
0
i=~~+2,~~~3,...,~i.~~
I
3N
(14)
(12) In Fig. 9(a), the rigid face Gigid contains the ;;i;;;;;,b~;~i2~c; 0 and thr classat,, con-
In this way ‘iii is a periodic function of period Tm0.zand the solution (whenn standsfor nAt and M standsfor T,,,,) is:
= kfaf(wj. WY are c“PP+~) we including the first MNR to define a class? Obviously the choiceof aigti to definea classwould be bad becausethe face appearsover-scaled.When with we include the first MNR the face is correctly scaled. To understandthis, we showin Fig. 10the amplitude of the 3N components of at,,. All of these componentsare approximately nil except the com- and ponent f&tot(F,) (and also with lessimportance the componentsbetweenC&l first component i&
co~;;~y;;;;;
w41) of the energy of deformation. It can be named a ‘scale factor’ and must be included in the vector d -clarS~
3.3. Synthesisof varieties For each class, the varieties are obtained by modification of the sub-vector 0 with j=U!+2 J&+3 y + m in the vG%r C&S. The 2mod&&ion’ df’the m- I= 19 MNR componentscan be done arbitrarily, but we prefer to exploit
Wj
n
27X------- -@pt2 (M
k 01
(15)
kM
(17)
Each value n givesa variety BwB and awm is the photo-fit identikit U-h. We show in Fig. El the evolution of the energy of varieties with zero damping and a period 116-10. The corr w flow chart of calculation of varieties is &own in Fig. 12. In Fig. 13we show4 classes with A&- 10 varieties. We show in Fig. 14 a graph representation of calculations. For graph simpli&ations we suppose an unrealisticmodelwith 8 DOF, that is with 4 rigid modes and 4 nonrigid modes. In this way each butterfly can be entirely outlined. The graph is composedof four stages,the first for & calculation, the secondfor D calculation, the third to obtain gnrR
Face data structures
I I
0
1
4
8
30
-
-------
jmc 810
Fig. 9. Strain energy evolution versus m. (a) for head 9, (b) for head 1, head 3, head 8 and head 9.
J. P. Gourret andJ. Khamlichi
0
Head
~mponents tis,tal 3oo 4 . i.
.,.
1 1
i.
(oo..._.........__..___............. 1 .,oo...... ................ -3oo-700. .ooo-~. -1100-~ .,300- ._ 1. ‘.I .~. I: -1500 SlO :: 0 ,520 mOdeSi ---.
c omponents 3oo I
Head3
1
3oo
i&,,
.
.,
790 70s so0 SOS *lo 015 sao ws tllOdCSi
-
arii
. ..““““..”
~
.300.
..~.
_..
.500-
Head 8
-700. -800~ -,100-
-.
-1300-15007
) 790
795
SO0
SOS
810
815
020
125
s30
i
e3oo esoo
Head9
-.300.5oo
.
-700
.._..
-9oo
-Too-~ --.-.. .@OO. .,(oo . .,300.
. _..
,....
j .((oo
I
.,.:
~.~;;i-;~:;;.i
.,300
-1sooi 0
SlO
modesj
-.
1620
~--
--
700
7 795
...
..
-.
-...--..
. *oo
..’ -. .r ,..,. _. I *es
*10 modeSi
31s
.-
* 120
s25
) s.30
Fig. 10.Amplitudeof the components (reference faces head 1, head 3, head 8 and head 9)
and gMNR dissociated,and the fourth to associate GR and ILMNR.On the graph we considerall MNR without the suppressionof y - m nonrigid modes. In the second stage for the rigid modes with pulsations Ui=O, we have “iii = ?i, and for the nonrigid modes we apply Eq. (15). So, the transformation of & to “i& can he viewedas a signal 7: applied to tilters of stepresponseh,(n) given by Eq. (15). On entry of the first stagethe signaljj is applied
to the initial sphere.Entries QYarMNR on the top of the third stage allow the creation of varieties by acting directly and arbitrarily on the nonrigid modes, but it is a blind method becauseit is very di&ult to control the final shape&. It is alsodifkult to keep a coherencebetween the varieties and their class. Moreover, this methodneedsto memo&em - 1 data items per variety. The memorization of 02- 1 data itemsis not necessarywhen we usethe relation Eq.
Face data structures
875
--.........
J. P. Gourret and J. Khamlichi
876
r--------------r t I
i I a 8 1 Process : 0______-_____--I I 1
data
MI
reference face
CJ,,osswhen n = kM r/,
II --S&l,
whenkM
(15) becausethe MNR componentsare generated calculatedwith Eq. (15) doesnot needstorage. So, from ?i. the storageneeds3N x (y + m) + m + C x (y + m) data items. The method is interestingonly when: 3.4. Advantages of the physical model The model allows the compressionof data for shapestorage and transmissionof faces and solids M>3N(if!+m)+m+C(y+m) (181 with sphericaltopology. Imagine C classesof faces 3NC with M varietiesper class,and facesdefinedwith 3N DOF. We need to memorize 3NxCxM data items or when: when _Uis directly stored for each face. With our c, 3N(y+m) +m model we memorize y+ m eigenvectors of size (19) 3NM(y+m) [3Nx l] and n? eigenvaluescommon to each class and variety. For each classwe also store the y + m or when the chosenlevel is in accordancewith: first generalizedforces?i from which we can deduce_U NI
Face data structures
, Initial sphere ,
1: 877
H[cad1
Head3
lead8
lead9
Fig. 13.Exampleof 4 classes andM= 10varietiesfor level3.
For example, the method is not interestingwhen we considerC= 10classes, M= 10varietiesper class, m=20 and 3N= 1620DOF becausethe relationsEq. (18x20) are not verified. We obtain 162,000data items for 100 faces with the direct method and 1,352,920data itemswith our method.
But it is interestingin two situations: First, when the classesare definitively set it is not necessaryto store the totality of the eigenvaluesand eigenvectorsbecausethey are neededonly to create new classes, that is to analysenew referencefaces.It is necessaryto store m- 1 eigenvaluesand m - 1 eigenvectorsof size [3Nxl] to calculate the MNR y + 2 to y + m commonto the classes and varieties. For each classit is necessaryto store the y + 1 componentsi&b*, (i = 1 to u + 1) and to store the zv m-l components Fi(i=T+2to y+m). For example, the storage for C= 10 classes, with 3N= 1620DOF, m=20 and M= 10 varieties needs 3Nx(m-l)+(m-l)+Cx(~+m) =39,099 data
878
J. P. Gourret
stage 1
stage2
and J. Khamlichi
stage4
stage3 ,’
_i M-l)
Fig.
14. Graph
representation
items comparative to 162,000 data items for the direct method. Second, the method is interesting when the best criterion for compression is not chosen in terms of size of storage but in terms of transmission of information. Imagine a transmission of objects with spherical topology. The source machine and the destination machine (the server and the client for
of faces.
computer scientists) know all the eigenvectors and all the eigenvalues associated with a level of discretization of objects. In this case we need only to transmit the y + M first generalized forces Fi per class. That is C x (y-t m) data items to transmit information concerning C classes of h4 varieties each. For C= 10, M= 10, 3N= 1620,and m=20, we have to compare 8300 data items to the 162,000given by the direct
1 l-2x+m 3N 'M
Compressionratio 4
~ 3’4’5’6’7’6’9 Fig.
15. Compression
ratio
versus the number
M of varieties
(3N=
1620, m= 20).
Face data structures
method. That is a compressionratio of 94.8%. We show in Fig. 15 the compressionratio versus the number A4 of varieties. 4. CONCLUSION
AND
FUTURE
WORKS
We presenteda physicalmodelbasedon deformation calculations.The model can be applied to the classification and to the compressionof face and solid data structures.It can also be applied to the transmissionof facesand solidswith a good ratio of transmission.The modeldefinesfacesand solidswith 1-D finite elementsassembledwith a new mesh topology, analysethe meshwith a modal analysis, and synthesizeclassesand varieties of objects.The synthesisof photo-fit identikits is a basiccharacteristic of our modeland cannot be obtainedfrom other models. With this project of modelling we want to cover two fields. Face and solid compressionfor transmission in the field of multimedia. Face recognition in the field of cognitive sciences. In the field of cognitive scienceswe want to usethe modeland the computer graphics transformations as tools for modelling natural systems[20]. Very often, neuroscientistsare interested by the deformation of objects. We envisageto create new cognitive testsby comparing resultsgiven by the model and experimentalmeasurements presently under development. This approach would allow comparisonof mental imagery (with figurative sights)with 3-D imagesproducedby physicallaws.We could for examplepresenta photofit identikit and a variety and ask for expertsto give a likeness ratio (identical, good likeness, poor likeness, different) for comparing their subjective measurements to the objective measurements given by the model.
819
4. Pentland,A. andWilliams,J., Goodvibrations:modal dynamics for graphics and animation. Computer Graphics, SIGGRAPH‘89, 1989,pp. 215-222. 5. Celniker, G. and Gossard, D., Deformable curve and surface finite elements for free-form shape design. Compufer Graphics, SIGGRAPH ‘91, 1991,pp. 257266. 6. Doo, D. and Sabin, M., Behavior of recursive division surfaces near extraordinary points. Computer-aided Design, 1978, 10(6), 356-360. I. Catmull, E. and Clark, J., RecursivelygeneratedBSpline surfaces on arbitrary topological meshes. Computer-aided Design, 1978, 10(6), 350-354. 8. Khamlichi, J., Modelisation de deformations d’images tridimcnsionnelles. Application aux structures de donnees de visages. Thesis, University of La Rochelle, Janv., 1995. 9. Pentland, A., Moghaddam, B. and Stamer, T., Viewedbased and modular eigenspaces for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, TR 245, Seattle, WA, 1994. 10. Moghaddam, B. and Pentland, A., Face recognition using view-based and modular eigenspaces, automated systemsfor theidentification of humans. In Proceedings of SPZE 2277, TR-301, July, 1994. 11. Pentland, A. and Sclaroff, S., Closed-form solutions for physically based shape modeling and recognition. Pattern Analysis and Machine Intelligence, 1991, 3(7), 115-129. 12. Nastar, C. and Ayache, N., Fast segmentation, tracking, and analysis of deformable objects. Rapport de
recherche INRIA 1783,1992.
13. Nastar, C., Analytical computation of the free vibration modes: application to non rigid motion analysis and animation in 3-D images. Rapport de rccherche INRIA 1935, 1993. 14. Cuthill, E. and McKee, J. M., Reducing the bandwidth of sparse matrices. In Proceedings of ACM, 1969, pp. 151-172. 15. Gourret, J. P., Modelisation d’images fixes et animees, manuels informatiques. MASSON. Paris. 1994. 16. Gourret, J. P. and Khamlichi J.; Three-dimensional image synthesis and modelling of physically deformable objects using a finite element model. In Application to Image Analysis, Fourth Eurographics Animation and Simulation Workshop, eds A. Luciani, D. Thalmann, Barcclone, 1993, pp. 121-133. REFERENCES 17. Requicha, A., Representation for rigid solid. Comput1. Zienkiewicz, 0. C., The Finite Element Method. ing Survey, 1980, 12(4), 437464. McGraw-Hill NewYork. 1982. 18. Crapo, H., On the generic rigidity of plane framework. 2. Bathe, K. J., .Finite Element Procedures in Engineering Raooort de Recherche INRIA 1278. 1990. Analysis. Prentice-Hall, Englewood Cliffs, 1982. 19. Crapo, H., Invariant-theoretic methods in scene analy3. Gourret,J. P., Modeling3D contactand deformations sis and structural mechanics. Rapport de Recherche using 6nite element theory in synthetic human tactile INRIA 1143, 1989. perception. In Course Notes On Synthetic Actors, eds 20. Gourret J. P. and Rostain J. C., L’infographie, un outil D. Thalmann et al.. SIGGRAPH 1988,1988,pp. 222pour les sciences cognitives. Recueil du colloquc sur le 230. neuromimetisme. AIDRI, Lyon, pp. 233-231, 1994.
Vol. 20, No. 6, pp. 863479, 1996 Copyright 0 1996 Elswier Science Ltd
& Graphics,
Pergamon
Printed in Great Britain. All rights reserved 0097-8493/96 $15.00 + 0.00
PII: soo97-8493(96)ooo57-x Technical Section
A MODEL
FOR COMPRESSION AND CLASSIFICATION FACE DATA STRUCTURES
OF
JEAN PAUL GOURRET+ and JAMAL KHAMLICHI Laboratoire d’Informatique et d’Imagerie Industrielle UniversiM de La Rochelle, Equipe Images de SynthZse, 15 Rue F. de Vaux de Fouletier, 17026 La Rochelle Cedex, France e-mail: [email protected] Abstract-In this paper we present a 3-D model with physical properties which simplifies the analysis and the synthesis of deformable faces and solids. Our model presents three relevant particularities. First, it describes the external envelope of faces with 1-D finite elements assembled with a new 3connected mesh topology. Second, the mesh deformations are analysed with a modal analysis. Because our model associates these two particularities, the number of rigid modes given by the modal analysis is equal to the number of 1-D finite elements, which is also the half of the number of Degrees of Freedom @OF). This number of rigid modes is a basic characteristic of our model. The second half of modes constitutes the nonrigid modes. Third, we use these rigid modes and the iirst nonrigid mode to synthesize a mean face named a photo-fit identikit or class, around which we synthesize face varieties by action on secondary nonrigid modes. Our physical 3-D model allows compression of face data structures because a greater number of secondary nonrigid modes can be suppressed to define a class or its varieties, and because the synthesis of varieties does not need more information storage than their classes. Our physical 3-D model allows classification of face data structures because we can associate an objective measure to each synthesized face. We can measure the deformation between a variety and its photo-fit identikit. Copyright 0 1996 Elsevier Science Ltd
1. RWRODUCTION must easilyloosenwhen a fast but rough solution is In structural mechanics the study of physical needed.The meshmay not changesize abruptly to deformations is generally achieved with the finite insurefast calculation and good accuracy. For these elementmethod [1, 21. This method is also usedin reasons,we developed a structured homogeneous
computer
graphics
for
shape
and/or
animation
mesh with
3-connected
vertices and with
links of
realism[3-51. In the 3-D space,the method defines approximately equal length. To insure thesecondiobjects
as an assemblage
of I-D,
2-D
or 3-D
elementswhich are linear, quadratic, cubic and so on. We use the finite
element
method
but
we
tions, the vertices of our model are generated on a sphere. For the modal analysis, the sphere represents a face or an unconstrainedsolid, i.e. with zero energy
intentionally restrict our model to 1-D linear finite elements(namedbars or deformablelinks) which are
of deformation, and constitutesthe initial condition of the numericalprocess.To obtain a meshon a face, 3-connected (a vertex is connected at the ends of the sphere vertices are projected on a known three bars). The 1-D finite elements cover the digitized 3-D face, or we apply forces on the sphere external envelope of the faces. This may seen to deform it. The typical method to put a grid on a regressivecomparedwith sophisticated2- and 3-D sphere is shown on Fig. l(a). The mesh follows modelsdevelopedtoday in computer graphics,but it parallel and meridian lines and creates triangular allows many advantages when we use a modal facets on the envelope. This kind of meshis often analysis for deformation calculations. With 1-D used in CAD, in image analysis, and in image finite elements3-connected,the modal analysisgives synthesisbecausethe facetsare alwaysplanar. But it a number of rigid modesequal to the number of 1-D is not suitable for our problem. Indeed, it is not finite elementsassembled to definethe face.The rigid structured because the number of neighbours for the modes combined with the first nonrigid mode two pole vertices is not the same as that for other synthesize a mean model of the face named a vertices.Near theseextraordinary points [l, 6, 71the photo-fit identikit or class, around which we can deformation calculation is not homogeneous,and a create varietiesby action on the secondarynonrigid systematic recursive subdivision of facets is difficult modes. becausethere is a loss of connectivity properties. An efficient meshmust be easily tesselatedwhen Moreover, the number of links is over-dimensioned good accuracy and/or
visual realism is needed, and
giving
an object
with
6 rigid
modes,
which
is
unsuitablefor our problem (seeSection 3). The sphere mesh is detailed in Section 2. It is a new mesh when applied to deformation calculation and measurement.
+ Author for correspondence. 863
It describes solids and faces with a
J. P. Gourret and J. Khamlichi
864 Pn
(4
(b)
Fig. 1. Mesh definition on a sphere. (a) typical, (b) chosen.
sphericaltopology and offers severaladvantages.It does not create extraordinary points and it can be tesselatedwith the sametopologic propertieswhatever the level of complexity of the face, dependingon the accuracy, time of calculation, memory size, etc. Moreover, we can associatean objective number to each synthesizedface. This number is a measureof the energyof deformation betweenthe variety and its photo-fit identikit. This approach allows sizereduction and manipulation simplification of face data structures. It is directly tied to our projects of face compressionfor transmissionin the field of multimedia, and face recognition in the field of cognitive sciences. For brevity we do not describethe representative works on face modelingwhich can be found in [8]. Among them, the works of Pentland are of interest becausehe is concerned with face recognition by modal analysis.However, his works are donein 2-D space for recognition and interactive search in a large-scaledatabase,[9, lo] or in the 3-D spacewhen facesand solidsare modelledwith the typical mesh shown in Fig. l(a) or with an assemblageof sophisticatednonlinear patches[ll]. With a typical I-D mesh or with 2-D element assemblage,it is impossibleto defme the photo-fit identikits introducedby our modelbecauseonly 6 rigid modesexist. The formulation presentedby Pentland [l l] and Nastar [12, 131usesthe formulation developedby Bathe [2]. Bathe and other researchersin the field of structural mechanicsdeveloped modal analysis to study the structure vibrations with external excitations. Modal analysis solves an eigensystemfor decouplingthe equationsof motion. Each eigenvalue and associatedeigenvectoris a vibration modeof the structure, i.e. characteristic distortion when all vertices vibrate with the samefrequency and the samephase.Pentland and Nastar usethis method for data compression.They combine vibration modes
and extract qualitative modes such as translation, rotation, scaling, shearing. But there exists only 6 rigid modes(3 translationsand 3 rotations) which do not allow the definition of the photo-fit identikits. 2. MESH DEFINITION
We define the meshshown in Fig. l(b). To obtain the mesh we start from a dodecahedron. The dodecahedronis the low level of tesselation,named level 0, which presentsthe two criteria (3-connected and links of equal length), and whose vertices are situatedon a spheresurface.The dodecahedronis a regular polyhedron. Then we construct a truncated icosahedron (named level l), which can also be termed as a semiregularpolyhedron made up of 12 pentagonsand 20 hexagons. Note that a regular polyhedron is characterizedby the samenumber of neighbours per vertex and with the same facet element.A semi-regularpolyhedron is characterized by the samenumber of neighboursper vertex and with different facet elements.An identical subdivision method is applied to transform level 1 to level 2 and more generally, to transform level n to level n+ 1. For levels n (n > 1) the sphere is made up of ‘pentagonal’ and ‘hexagonal’ facets giving a semiregular ‘polyhedron’ but with nonneeessarilyplanar facets.We put the wordspentagonal,hexagonaland polyhedron in inverted commasbecause,after ievel 1, the facetsare not planar. Moreover, the links are approximately of equal length. Each level is madeup of 12 ‘pentagons’ and of a growing number of ‘hexagons’.The ‘pentagons’ensurethe curvature of the space. The subdivisionmethod is shown on Fig. 2. For the level n (shownas a bold outline in Fig. 2), each vertex S,-,has three neighbours,Si, &, S, oriented clockwisewith respect to the facet normals and in a left hand coordinate system.The verticesdefinethree
Face data structures
865
(4 Fig. 2. Subdivision method.
edges S& (i= 1, 2, 3) from which three new edges TioTil are created [Fig. 2(a)]. The three new edges join the centroid of the ‘pentagons’ and/or ‘hexagons’ which share the vertex Se and their length is the third of the distance between centroids. The three new edges build a new ‘hexagon’ which is not situated on the sphere surface. To obtain a new ‘hexagonal’ facet of level n+ 1, we must do a radial projection of the vertices on the sphere surface. The subdivision process for a ‘pentagon’ is shown in Fig. 2(b) and the subdivision process for an ‘hexagon’ is shown in Fig. 2(c). Note that for n > 1, the links are different because the centroids joining two ‘hexagons’ and the centroids joining a ‘pentagon’ and an ‘hexagon’ are not of equal length. To define the level 0 we could start from three kinds of regular polyhedrons with three neighbours per node: the dodecahedron whose facets elements are pentagons, the cube whose facet elements are squares and the tetrahedron whose facet elements are triangles. The subdivision process applied to the three kinds of polyhedrons gives the successive structured levels shown on Fig. 3. This is evidence that the dodecahedron gives the more homogeneous mesh. The mesh insures the Euler relationship N-E + F= 2, where N is the number of vertices, E is the number of edges (bars) and F is the number of facets (‘hexagons’ or ‘pentagons’). 2.1. iUoaiz1 analysis with I-D finite elements An object in the 3-D space submitted at time t to external forces ‘R is subject to displacements ‘a. After transformations proper to the finite element method the object is modeled by an assemblage of elements made up of N interpolation vertices or nodes, and 3N equations of motion describe its behaviour:
g’Q+g’O+g’g=
‘R
(1)
‘_V is the displacement vector of size [3Nx I]
expressed at time t. The size is 3N because in the 3-D space and with a 3-D Cartesian system, a deformable body has three degrees of freedom per node, ‘0 and ‘D are the velocity and the acceleration vectors, *R is the external force vector, 4 is the stiffness matrix of size [3Nx3N] for internal elastic force calculation, _C is the damping linear matrix of size [3Nx 3iVJ, _M &the mass matrix of size [3Nx3N]. These ma&es are supposed independent of displacements. We choose 1-D elements that are elastic bars. They can be translated without deformation, compressed or stretched. Their assemblage is done through the first transformations of the graphics pipe-line as described in [8]. Usually the g, g and & matrices have a large bandwidth because the 3N DOF are coupled. Node numbering tricks can be used for bandwidth reduction of the sparse matrix [14], but the best method consists to write the displacement vector ‘.g ‘_U=$‘O
(4
Where 2 is a [3Nx3N] matrix which diagonalize & C and g giving a canonical system with each DOFdecoupled. ‘4 is the generalized displacement vector expressed in the basis of eigenvectors & where a is the ith column of 2. A system with 3N DOF has 3N free vibration modes. For each mode, displacements *_U are all oscillating with a frequency e+/2K and a phase I#Q. It is an eigenproblem with eigenvalues CL$and eigenvectors gi which gives 3N solutions:
For simplicity, we suppose that the bars have distributed mass. The mass mi is lumped node i, and we choose mi= 1 such that g identity matrix and the eigenvectors are
do not at each is the ortho-
J. P. Gourret and J. Khan&hi
Level 0
Cube
Dodecahedron
Level 1
Level 3
Fig. 3. Successive levels of subdivision starting from three regular 3-connected polyhedrons.
normal. With these assumptionswe can write the equation of motion in the spaceof eigenvectors:
1%+gc +gij= ‘3 ‘&=gT‘IJ
(4)
(5)
The matrix of eigenvaluesQz is a diagonalmatrix of size[3Nx3Nj with wf on iG diagonal, c is also a diagonalmatrix whenE is a linear combin&on of g and K.
The 3N DOF are decoupledand we can write 3N equationsin the form: ‘& +2rio,‘Gi
+wf’tii=
‘&fori==1,2,3,...,3N
(6) Each
DOF
vibrates
with
the
pulsation
w = Oir- 1 - # where & is the damping coefficient. The solution when a constant force is app4iedfrom time t = o can be written:
Face data structures ‘g=--!
867
co;cos#J1 (7) Of.
e-Cw -cos(wt
l-
+ 9)
[
Without initial velocity ( ‘ii = 0) we obtain: C#J = 0 when ti = 0
(8)
when[i # 0
(9)
More detailson matrix and vector generationfor our theoretical model are developedin [8, 151.
2.2. Distinctive features of modal analysis associated with our mesh
The modal analysis is identical to the Fourier analysiswhich decomposesa signalon the basisof complex exponential functions, and superposesall the componentsfor synthesis.With Fourier analysis the high frequency components such as 2 wi2 > wIhreshold contain little energy and can be suppressedby low pass filtering. In this way, the synthesisof the signalis doneby superpositionof the first components.For deformablefaces,we can also suppresscouples (o;, cbi) such as i> ithr&& and obtain an incompletesuperpositionof modes.In this way the calculation time for synthesisis greatly reduced becausethe size of g is reduced.The first deformation modesare rigid body modeswith null eigenvalues.
In [16] we discussa simplemodelin the 2-D space when2-D elementsand I-D elementsare used.In the 2-D spacethere are 2N DOF. When the object is modeledby 2-D elements,the numberof rigid modes equals3 (two translationsand one rotation). When it is modeled by 1-D elements,the number of rigid modesvaries between3 and 2N/2 dependingon the number of 1-D elementsin the assemblage. In the 3-D space,when 2-D elementsmodel an object envelope, there are 6 rigid modes (three translationsand three rotations). When I-D elements are used,the numberof rigid modesvariesbetween6 and 3N/2. In Fig. 4(a) we showa simpleobject made up of I-D elementsin the 3-D space.The cube is madeup of 12 1-D elements(bars) and N= 8 nodes. In the 3-D spacethere exists 3N=24 DOF and 12 rigid modesbecausethe nodesare 3-connected.Each rigid modeis a translationwithout deformation of a bar, and is associatedwith a null eigenvalue.For recoveringa 3-D object immersedin the 3-D space, the number of rigid modesmust be equal to 6, and the number of nonrigid modes is consequently 24- 6 = 18. To obtain this we can add somebars to stiffen the cube. For example, when we add a bar betweennodes 1 and 3, the eigenvaluecalculation showsthat the cube is madeup of 12- 1= 11 rigid modesand 12+ 1= 13 nonrigid modes [Fig. 4(b)]. The added bar suppresses a rigid mode which is replacedby a nonrigid mode. More generally, each bar addedsuppresses a rigid modeand replacesit by a nonrigid mode.It is necessaryto add a minimumof 6 bars to obtain the 6 rigid modes.Figure 4(c and d)
6
7, X
12MR 12MNR
(4
6MR 18 MNR
(b)
11MR 13 MNR
6fiR 18MNR
AR 11 MNR
Fig. 4. Numberof rigidmodes(MR) andnonrigidmodes (MNR) for a cubemadeup of 1-Delements in the3-D space. C&G 20:6-E
868
J. P. Gourret and J. Khamlichi
shows two possibilities. The first by addition of a strengthening bar on each facet of the cube. The second by addition of strengthening bars along internal diagonals of the cube and on its envelope. The eigenvalue calculation when we add more than 6 bars as shown in Fig. 4(e), does not modify the number of rigid modes which stays equal to 6. Consequently, the optimal number of bars is equal to 3N-6 for an object immersed in a 3-D space. It is 2N- 3 for an object immersed in the 2-D space [16]. The eigenvalue calculation for the ‘cube’ of Fig. 4(f) when the bar joining nodes 2 and 6 is suppressed gives 13 rigid modes and 11 nonrigid modes. But in this case the object is not a solid because it contains dangling faces. The internal volume is not enveloped in a closed surface and the Euler relationship N-E+ F=2 is not verified. To define a solid object Requicha[17] introduces r-sets. An r-set object is a subspaceof the 3-D Euclidean spacewhich is bounded, closed, regular and semianalytic. Bounded becauseit occupiesa finite portion of space. Closed becausethe object contains its boundary. Regular becauseit equalsthe closureof its interior. Semianalyticbecauseit can be expressedas a finite Boolean combination of sets defined by analytic functions. An r-set object can always be transformed into a boundary representation with triangular facets, and the assemblageof r-set objectsis also an r-set object provided regular boolean operators are used [17]. Following our discussionabout the cube of Fig. 4, we add to the definition of a r-set that: -for a r-set the number of rigid modesin the 3-D spacevaries between6 and 3N/2,
-the optimal mesh developed for our model is a r-set which insures 3N/2 rigid modes and a 3N/2 nonrigid modes whatever the level of discretization. To obtain a spherewith 6 rigid modeswe must add 3N/2-6 bars. A meanfor this is to triangulate the sphereas shown in Fig. l(a). But it is not an optimal method becausethe relationship E= 3N- 6 is not verified. The number E of added edgesis greater than 3N-6. Optimal or not optimal objects with 6 rigid modesare namedinfinitely rigid objects and others objects are named rigid objects because they can produce infinitesimal deformations when excited. In a 2-D space,the optimal threshold is given by E=2N-3. For example, the grid in Fig. 5(a) is made up of 40 bars and 25 nodes.It can be infinitely rigid by adding a diagonal bar for each square[Fig. 5(b)], but the number of bars doesnot give the optimal solution. The optimal solution needs the addition of 2N-3 =7 bars. Crap0 [18, 191 showedthat a rigid grid in the 2-D spacecan be made infinitely rigid and optimal when we add diagonal bars at right emplacements.Each square has a horizontal and vertical addressas shown in Fig. 5(c, d). The grid is infinitely rigid when the graph createdby diagonal bars is connected,that is whenthere existsa link for connectingany addressto any other. In the 3-D spacealso,the optimal solution does not need a complete triangulation of the surface. Our mesh[Fig. l(b)] is under-optimal and the typical mesh[Fig. l(a)] is over-optimal. Our mesh createsa rigid objectwith 3N/2 rigid modesand 3N/2 nonrigid modes.
(a)
(b)
Ll
Ll
Cl
L2
L2
c2
L3
L3
c3
L4
IA 72
c4
cc>
(d)
Fig. 5. Optimalplacement of strengthening barsin 2-D space.
Face data structures 3. RESULTS 3.1. Creation of a reference face
A referenceface can be.obtainedfrom three ways: (a) The face is obtained from a 3-D scanner.In this paper each scannedface contains 600 meridians and approximately 280nodesper meridian giving approximately 168,000nodesper face. To define the new mesh of level II we put in coincidence the center of the sphere of level n with the center of the scannedface, and we do a radial projection of the spherenodeson the face, including the planar base[Fig. 6(a)]. (b) The face is directly created by application of external forceson the nodesof the sphereof leveln (Fig. 7). (c) The face obtainedfrom (a) is partially modified by applying external forces at nodes. In the following, we study 4 headsnamedheadl, head3, head8 and head9 whoselevel 7 is shownin Fig. 6(b). We usea Gouraud shadingafter triangulation of the ‘pentagonal’and ‘hexagonal’ facets. To triangulate we define5 trianglesper ‘pentagon’and 6 triangles per ‘hexagon’ with a common vertice situated at the centroid. 3.2. Analysis of a reference face
The analysisof a referenceface of level n needs two steps.
Level 3
869
The first step is common to all referencefaces.It createsthe matrix g (M and g if needed),solvesthe eigensystemand deducesthe couples(w~,~) in the increasing order. This step is a pre-processing becauseit is sufficient to know the internal characteristicsof the deformablesystem(i.e. the sphereof level n). Matrices Q2 and g are stored directly or after compression,that is after suppressionof the high frequency couples.The time of calculation can be very long but the calculation must be done only oncefor eachlevel [111.It takes4 h on a HP9000/715 for level 3 and with a Jacobi method. To reducethe time for levelsup to 3, we presently develop these calculationson a clusterof HP9000/725workstations interconnectedwith a 100Mb/s network using the ParallelVirtual Machine (PVM) tools. The secondstep takes into account the external forces O&appliedat the verticesof the sphereat time 0. Thesecalculationsare doneduring the secondstep becausethe external forcesare not neededduring the first step. The two stepscorrespondto the solution of the homogeneous equation of motion without the right hand side,and to a particular solution obtainedwith the right hand side.In the secondstep we calculate the vector “4 with Eq. (5), then the componentsof the vector ‘_Vwith Eq. (7), then we superposethe increasing modes (eventually without the high frequencymodes)with Eq. (2).
Level 4 Fig. 6(a). Fig. 6.--continued overleaf
Level 5
J. P. Gourret and J. Khamlichi
870
Head 3
Head 1
Head 9
Head 8 Fig. 6(b).
Fig. 6. (a) Levels 3,4, 5 of the mesh for a reference face (head 9), (b) Level 7 for the reference faces under study.
Face data structures
871
Fig. 7. Application of external forces on the sphere (level 2).
The strain energy can be obtained from E = i UT_ U or from E = iaTg2 a. It is the energy of nonrigid modes(MNR) becausethe pulsationsof rigid modes(MR) are ze_ro.So, it is the strain energy relative to a rigid face Qgid madeup of rigid modes only and whose MNR are zero. The vector for visualization is given by Eq. (2):
containstwo sub-vectors&,olMR and 0 of size [3iV/ 2x l] each.The vector a,,, containsthe sub-vector alarMR of size [3N/2x 11,the first componentof the sub-vector &,a,MNR and a zero sub-vector of size [((3iV/2 - 1)x 11.We show in Fig. 8(a) the reference face a,,, (or more preciselyL& = @IL*,,), in Fig. 8(b) the rigid face &+,, and in Fig. 8(c) the class a,,,. Any modification of the components&.,MNR &id = g -@rigid (10) i ff L, or any modification of the zero sub-vector in A referenceface obtained with one of the three &,, gives a variety &,, of the face around the methods given in Section 3.1 will be noted e,O,. photo-fit identikit. A variety &or is shown in Fig. From a,,, we deducethe rigid face aiKti and the 8(d). In the following we considerfacesat level 3, that is photo-fit identikit alsonameda classof faces&,,. The vector ata, contains two sub-vectors&a,MR faces with 540 nodes, 1620DOF, 810 MR and and &MNR of size[3N/2x I] each.The vector aigid 810MNR. We first justify the suppressionof the
(a)
(b)
Cc)
Cd)
Fig. 8. Face “head 9” and level 3. (a) reference face, (b) rigid face, (c) photo-fit identikit (class), (d) a variety.
J. P. Gourret and J. Khamlichi
872
high frequency modesand the choice of the class the possibilities of our physical model and to vector noting that we can write eachcomponentof calculate the modifications of the componentswith Eq. (7). For examplewe calculate the solution ‘iii Eq. (2) as: when initial conditions are Oui= 0, Olii= 0 and 3NI2 when we apply an external force O& at time t =O. ‘Ui=C@~fCj+ 2 (11) The force is maintainedfor all time t > 0. Doing this, j=l j=(3N/2)+1 we definea variety for eachvalue of the parameterI m is the number of kept MNR. The choice of which hasthe dimensionof a time becauseour model m < 3N/2 suppresses the high frequenciesand allows is a physical model. The initial condition is the to visualize a smooth shape of the face without sphere.At t=O the sphereis deformedand take the details.In Fig. 9(a) we showthe energy versusm for appearanceof the photo-tit identikit. Then for each the referenceface ‘head9’, when m varies from 1 to parameter value At the face evolves and takes the 810.When m is about 20 (that is 830modesincluding appearanceof a variety. The successivevarieties the 810MR), the faceis visually nearby a,,, and the oscillatearound the referenceface atat with a period energy is approximately stable becausethe main T . The dampingis controlled by the coefficient &. energy variation is comprisedinto the first nonrigid W%!n &= 0 no damping occurs and when &#O the modes.Pentland et al. [9] obtain a similar curve for synthesizedvarietiesconverge towards the reference face recognition in the 2-D space. The curve face after a few periods. For this reason,we note a presentedby theseauthors givesthe recognition rate referenceface with the subscriptstat. We obtain an versusm. It is interestingto note that the recognition infinity of varieties when t evolves with arbitrary rate seemsdirectly tied to the energy measureused stepsAt. For obtaining an integer number M of for our model. We obtain similar curvesfor the four varietieswe must choosea stepAt = IL where 7,,.r xM head under study [Fig. 9(b)]. So, we choosem=20 is the maximal period and we redefinethe face vectors as: ( add a phase: @‘U’ji j
(13) with -T +,m$
u
=
x %at, 1
_ ... ustat 4
u”t”t(+,)
0
‘..
0
i=~~+2,~~~3,...,~i.~~
I
3N
(14)
(12) In Fig. 9(a), the rigid face Gigid contains the ;;i;;;;;,b~;~i2~c; 0 and thr classat,, con-
In this way ‘iii is a periodic function of period Tm0.zand the solution (whenn standsfor nAt and M standsfor T,,,,) is:
= kfaf(wj. WY are c“PP+~) we including the first MNR to define a class? Obviously the choiceof aigti to definea classwould be bad becausethe face appearsover-scaled.When with we include the first MNR the face is correctly scaled. To understandthis, we showin Fig. 10the amplitude of the 3N components of at,,. All of these componentsare approximately nil except the com- and ponent f&tot(F,) (and also with lessimportance the componentsbetweenC&l first component i&
co~;;~y;;;;;
w41) of the energy of deformation. It can be named a ‘scale factor’ and must be included in the vector d -clarS~
3.3. Synthesisof varieties For each class, the varieties are obtained by modification of the sub-vector 0 with j=U!+2 J&+3 y + m in the vG%r C&S. The 2mod&&ion’ df’the m- I= 19 MNR componentscan be done arbitrarily, but we prefer to exploit
Wj
n
27X------- -@pt2 (M
k 01
(15)
kM
(17)
Each value n givesa variety BwB and awm is the photo-fit identikit U-h. We show in Fig. El the evolution of the energy of varieties with zero damping and a period 116-10. The corr w flow chart of calculation of varieties is &own in Fig. 12. In Fig. 13we show4 classes with A&- 10 varieties. We show in Fig. 14 a graph representation of calculations. For graph simpli&ations we suppose an unrealisticmodelwith 8 DOF, that is with 4 rigid modes and 4 nonrigid modes. In this way each butterfly can be entirely outlined. The graph is composedof four stages,the first for & calculation, the secondfor D calculation, the third to obtain gnrR
Face data structures
I I
0
1
4
8
30
-
-------
jmc 810
Fig. 9. Strain energy evolution versus m. (a) for head 9, (b) for head 1, head 3, head 8 and head 9.
J. P. Gourret andJ. Khamlichi
0
Head
~mponents tis,tal 3oo 4 . i.
.,.
1 1
i.
(oo..._.........__..___............. 1 .,oo...... ................ -3oo-700. .ooo-~. -1100-~ .,300- ._ 1. ‘.I .~. I: -1500 SlO :: 0 ,520 mOdeSi ---.
c omponents 3oo I
Head3
1
3oo
i&,,
.
.,
790 70s so0 SOS *lo 015 sao ws tllOdCSi
-
arii
. ..““““..”
~
.300.
..~.
_..
.500-
Head 8
-700. -800~ -,100-
-.
-1300-15007
) 790
795
SO0
SOS
810
815
020
125
s30
i
e3oo esoo
Head9
-.300.5oo
.
-700
.._..
-9oo
-Too-~ --.-.. .@OO. .,(oo . .,300.
. _..
,....
j .((oo
I
.,.:
~.~;;i-;~:;;.i
.,300
-1sooi 0
SlO
modesj
-.
1620
~--
--
700
7 795
...
..
-.
-...--..
. *oo
..’ -. .r ,..,. _. I *es
*10 modeSi
31s
.-
* 120
s25
) s.30
Fig. 10.Amplitudeof the components (reference faces head 1, head 3, head 8 and head 9)
and gMNR dissociated,and the fourth to associate GR and ILMNR.On the graph we considerall MNR without the suppressionof y - m nonrigid modes. In the second stage for the rigid modes with pulsations Ui=O, we have “iii = ?i, and for the nonrigid modes we apply Eq. (15). So, the transformation of & to “i& can he viewedas a signal 7: applied to tilters of stepresponseh,(n) given by Eq. (15). On entry of the first stagethe signaljj is applied
to the initial sphere.Entries QYarMNR on the top of the third stage allow the creation of varieties by acting directly and arbitrarily on the nonrigid modes, but it is a blind method becauseit is very di&ult to control the final shape&. It is alsodifkult to keep a coherencebetween the varieties and their class. Moreover, this methodneedsto memo&em - 1 data items per variety. The memorization of 02- 1 data itemsis not necessarywhen we usethe relation Eq.
Face data structures
875
--.........
J. P. Gourret and J. Khamlichi
876
r--------------r t I
i I a 8 1 Process : 0______-_____--I I 1
data
MI
reference face
CJ,,osswhen n = kM r/,
II --S&l,
whenkM
(15) becausethe MNR componentsare generated calculatedwith Eq. (15) doesnot needstorage. So, from ?i. the storageneeds3N x (y + m) + m + C x (y + m) data items. The method is interestingonly when: 3.4. Advantages of the physical model The model allows the compressionof data for shapestorage and transmissionof faces and solids M>3N(if!+m)+m+C(y+m) (181 with sphericaltopology. Imagine C classesof faces 3NC with M varietiesper class,and facesdefinedwith 3N DOF. We need to memorize 3NxCxM data items or when: when _Uis directly stored for each face. With our c, 3N(y+m) +m model we memorize y+ m eigenvectors of size (19) 3NM(y+m) [3Nx l] and n? eigenvaluescommon to each class and variety. For each classwe also store the y + m or when the chosenlevel is in accordancewith: first generalizedforces?i from which we can deduce_U NI
Face data structures
, Initial sphere ,
1: 877
H[cad1
Head3
lead8
lead9
Fig. 13.Exampleof 4 classes andM= 10varietiesfor level3.
For example, the method is not interestingwhen we considerC= 10classes, M= 10varietiesper class, m=20 and 3N= 1620DOF becausethe relationsEq. (18x20) are not verified. We obtain 162,000data items for 100 faces with the direct method and 1,352,920data itemswith our method.
But it is interestingin two situations: First, when the classesare definitively set it is not necessaryto store the totality of the eigenvaluesand eigenvectorsbecausethey are neededonly to create new classes, that is to analysenew referencefaces.It is necessaryto store m- 1 eigenvaluesand m - 1 eigenvectorsof size [3Nxl] to calculate the MNR y + 2 to y + m commonto the classes and varieties. For each classit is necessaryto store the y + 1 componentsi&b*, (i = 1 to u + 1) and to store the zv m-l components Fi(i=T+2to y+m). For example, the storage for C= 10 classes, with 3N= 1620DOF, m=20 and M= 10 varieties needs 3Nx(m-l)+(m-l)+Cx(~+m) =39,099 data
878
J. P. Gourret
stage 1
stage2
and J. Khamlichi
stage4
stage3 ,’
_i M-l)
Fig.
14. Graph
representation
items comparative to 162,000 data items for the direct method. Second, the method is interesting when the best criterion for compression is not chosen in terms of size of storage but in terms of transmission of information. Imagine a transmission of objects with spherical topology. The source machine and the destination machine (the server and the client for
of faces.
computer scientists) know all the eigenvectors and all the eigenvalues associated with a level of discretization of objects. In this case we need only to transmit the y + M first generalized forces Fi per class. That is C x (y-t m) data items to transmit information concerning C classes of h4 varieties each. For C= 10, M= 10, 3N= 1620,and m=20, we have to compare 8300 data items to the 162,000given by the direct
1 l-2x+m 3N 'M
Compressionratio 4
~ 3’4’5’6’7’6’9 Fig.
15. Compression
ratio
versus the number
M of varieties
(3N=
1620, m= 20).
Face data structures
method. That is a compressionratio of 94.8%. We show in Fig. 15 the compressionratio versus the number A4 of varieties. 4. CONCLUSION
AND
FUTURE
WORKS
We presenteda physicalmodelbasedon deformation calculations.The model can be applied to the classification and to the compressionof face and solid data structures.It can also be applied to the transmissionof facesand solidswith a good ratio of transmission.The modeldefinesfacesand solidswith 1-D finite elementsassembledwith a new mesh topology, analysethe meshwith a modal analysis, and synthesizeclassesand varieties of objects.The synthesisof photo-fit identikits is a basiccharacteristic of our modeland cannot be obtainedfrom other models. With this project of modelling we want to cover two fields. Face and solid compressionfor transmission in the field of multimedia. Face recognition in the field of cognitive sciences. In the field of cognitive scienceswe want to usethe modeland the computer graphics transformations as tools for modelling natural systems[20]. Very often, neuroscientistsare interested by the deformation of objects. We envisageto create new cognitive testsby comparing resultsgiven by the model and experimentalmeasurements presently under development. This approach would allow comparisonof mental imagery (with figurative sights)with 3-D imagesproducedby physicallaws.We could for examplepresenta photofit identikit and a variety and ask for expertsto give a likeness ratio (identical, good likeness, poor likeness, different) for comparing their subjective measurements to the objective measurements given by the model.
819
4. Pentland,A. andWilliams,J., Goodvibrations:modal dynamics for graphics and animation. Computer Graphics, SIGGRAPH‘89, 1989,pp. 215-222. 5. Celniker, G. and Gossard, D., Deformable curve and surface finite elements for free-form shape design. Compufer Graphics, SIGGRAPH ‘91, 1991,pp. 257266. 6. Doo, D. and Sabin, M., Behavior of recursive division surfaces near extraordinary points. Computer-aided Design, 1978, 10(6), 356-360. I. Catmull, E. and Clark, J., RecursivelygeneratedBSpline surfaces on arbitrary topological meshes. Computer-aided Design, 1978, 10(6), 350-354. 8. Khamlichi, J., Modelisation de deformations d’images tridimcnsionnelles. Application aux structures de donnees de visages. Thesis, University of La Rochelle, Janv., 1995. 9. Pentland, A., Moghaddam, B. and Stamer, T., Viewedbased and modular eigenspaces for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, TR 245, Seattle, WA, 1994. 10. Moghaddam, B. and Pentland, A., Face recognition using view-based and modular eigenspaces, automated systemsfor theidentification of humans. In Proceedings of SPZE 2277, TR-301, July, 1994. 11. Pentland, A. and Sclaroff, S., Closed-form solutions for physically based shape modeling and recognition. Pattern Analysis and Machine Intelligence, 1991, 3(7), 115-129. 12. Nastar, C. and Ayache, N., Fast segmentation, tracking, and analysis of deformable objects. Rapport de
recherche INRIA 1783,1992.
13. Nastar, C., Analytical computation of the free vibration modes: application to non rigid motion analysis and animation in 3-D images. Rapport de rccherche INRIA 1935, 1993. 14. Cuthill, E. and McKee, J. M., Reducing the bandwidth of sparse matrices. In Proceedings of ACM, 1969, pp. 151-172. 15. Gourret, J. P., Modelisation d’images fixes et animees, manuels informatiques. MASSON. Paris. 1994. 16. Gourret, J. P. and Khamlichi J.; Three-dimensional image synthesis and modelling of physically deformable objects using a finite element model. In Application to Image Analysis, Fourth Eurographics Animation and Simulation Workshop, eds A. Luciani, D. Thalmann, Barcclone, 1993, pp. 121-133. REFERENCES 17. Requicha, A., Representation for rigid solid. Comput1. Zienkiewicz, 0. C., The Finite Element Method. ing Survey, 1980, 12(4), 437464. McGraw-Hill NewYork. 1982. 18. Crapo, H., On the generic rigidity of plane framework. 2. Bathe, K. J., .Finite Element Procedures in Engineering Raooort de Recherche INRIA 1278. 1990. Analysis. Prentice-Hall, Englewood Cliffs, 1982. 19. Crapo, H., Invariant-theoretic methods in scene analy3. Gourret,J. P., Modeling3D contactand deformations sis and structural mechanics. Rapport de Recherche using 6nite element theory in synthetic human tactile INRIA 1143, 1989. perception. In Course Notes On Synthetic Actors, eds 20. Gourret J. P. and Rostain J. C., L’infographie, un outil D. Thalmann et al.. SIGGRAPH 1988,1988,pp. 222pour les sciences cognitives. Recueil du colloquc sur le 230. neuromimetisme. AIDRI, Lyon, pp. 233-231, 1994.