- Email: [email protected]
3D mug shot—3D head models from photogrammetry for forensic identification
Forensic Science International 300 (2019) 6–12
Contents lists available at ScienceDirect
Forensic Science International journal homepage: www.elsevier.com/locate/forsciint
3D mug shot—3D head models from photogrammetry for forensic identification Anja Leipnera,* , Zuzana Obertováa , Martin Wermutha , Michael Thalib , Thomas Ottikera , Till Sieberthb a b
Zurich Forensic Science Institute, Zeughausstrasse 11, CH-8004 Zurich, Switzerland Institute of Forensic Medicine, University of Zurich, Winterthurerstrasse 190/52, CH-8057 Zurich, Switzerland
A R T I C L E I N F O
A B S T R A C T
Article history: Available online 22 April 2019
No human face is like another, not even in monozygotic twins, which makes the face one of the most individualizing characteristic. It is for this reason that the human face is commonly used for identification purposes and police officers take portrait photographs of arrested persons, so-called mug shots. The disadvantage of these 2D mug shots is that the perspective, in which they are taken (usually frontal and lateral-right, left or both), cannot be changed after acquisition, thus limiting a potential comparison between a mug shot and surveillance footage or other visual recordings. Documenting a face in 3D would reduce this problem as it allows adjusting the perspective of the face for image comparisons depending on the needs of the investigator. We have developed a 3D mug shot system containing 26 digital single-lens reflex cameras arranged semi-circularly in a 200 arc with a 1.46 m radius around a height-adjustable chair. We generated photogrammetric models of a test person’s face captured by the mug shot system using three different focal lengths settings as well as 3D models of the same face with GOM Atos Triple Scan and Artec Space Spider. The 3D models were then analysed regarding the visibility of detailed morphological features in different regions of the face compared to 2D mug shots. Our results showed that our 3D mug shot system with its photogrammetric documentation generates 3D models with comparable surface quality to Artec-generated models, or even better quality, compared to GOM-generated models. The results of the morphological assessment were affected by the focal length and availability of texture information. In conclusion, the 3D mug shot system is a fast and efficient tool to generate 3D models of the face and may be used in addition to 2D photographs for the purpose of visual forensic identification based on images. © 2019 Elsevier B.V. All rights reserved.
Keywords: Photography Photogrammetry Forensic documentation 3D modelling 3D reconstruction Person identification
1. Introduction No human face is like another, not even in monozygotic twins, which makes the face one of the most individualizing characteristic between humans [1,2]. It is for this reason that the human face is commonly used for identification purposes and police officers take portrait photographs of arrested persons, so-called mug shots [3]. In 1888 Alphonse Bertillon standardized the photographic documentation of criminals by acquiring images from frontal and profile views, which has not significantly changed since then [3]. The disadvantage of these 2D mug shots is that the perspective, in which they are taken, cannot be changed
* Corresponding author. E-mail address: [email protected] (A. Leipner). https://doi.org/10.1016/j.forsciint.2019.04.015 0379-0738/© 2019 Elsevier B.V. All rights reserved.
after acquisition, thus limiting a potential comparison between the mug shot and oblique visual recordings like usually acquired by surveillance cameras. The comparison process can be facilitated by acquiring additional photographs of the face. The photographs can be taken in a comparable perspective. But there is no guaranty to achieve the correct viewing angle. Furthermore, the lighting situation and shadows can variably influence the visibility of morphological facial features captured on the images and thus the subsequent comparison [4,5]. Documenting a face in 3D would reduce these problems as it allows adjusting the perspective of the face for image comparisons depending on the needs of the investigator. Current methods to document a face in 3D include laser scanning, structured light scanning [6,8] or single camera photogrammetry [5,9,10]. However, these methods might not offer colour and texture information or the quality of the 3D model is inadequate. The methods also
A. Leipner et al. / Forensic Science International 300 (2019) 6–12
differ in the geometric resolution of the generated 3D model. Another considerable factor is the time required to perform the 3D documentation of participant human subject, which may be prolonged and the subject is required to sit still during the procedure. Multi-camera systems are a fast 3D documentation method, which overcomes this issue due to their swift scanning procedure and therefore is often used for real time 3D documentation [11–13]. The method presented by Leipner et al. [14] is also based on a multi-camera device and is used for forensic documentation of body height and for medical examination [15]. As the system has been initially developed to document the whole body of a subject, detailed information of the face is only visible as texture information in the resulting 3D model but not as a 3D surface. In this paper, we introduce and test a 3D mug shot system for facial images specifically developed for the purpose of forensic identification. 2. Method 2.1. System setup Based on the experience with the multi-camera system for the documentation of the whole body [14], we have developed a 3D mug shot system for facial images. The system contains 26 digital single-lens reflex (DSLR) cameras arranged semi-circularly in 200 arc with a 1.46 m radius around a height-adjustable chair (Fig. 1). The cameras are of type Canon EOS 80D (Canon Inc., Tokyo, Japan). For photogrammetric documentation, 23 of the cameras are equipped with Canon EF-S 18–135 mm f/3.5–5.6 IS USM lenses offering flexibility in the prototype phase to test several focal lengths. The remaining three cameras are equipped with a Canon EF-S 60 mm f/2.8 Macro USM lenses. The semi-circular frame consists of two rows with 10 DSLRs each with 21 between the individual cameras. The first row of cameras is 1.67 m above ground level and the second row is 0.74 m above ground level. Further, three DSLRs are placed centred approximately 0.26 m under the lower row with the focus on the subject’s chin area. The three DSLRs with the fixed lenses are placed centred, right and left of the head position in 1.17 m above ground for standard 2D mug shots (Fig. 1). The 23 cameras for the photogrammetric documentation are in portrait orientation and the three 2D mug shot cameras are in landscape orientation. The position and alignment of the DSLRs was established based on the area of interest for the
7
3D mug shot and the requirements of at least 60% image overlap for photogrammetric documentation and needs to be done only once [16]. The cameras are connected to a workstation, and the camera release is a synchronized remote release. The data download is performed via USB. For setting the camera parameters a workstation with the software Smart Shooter 3 (Kuvacode Oy, Kerava, Finland) is used. Smart Shooter 3 provides an overview over all cameras simultaneously and allows adjusting the camera settings such as ISO, shutter speed and aperture, individually for each camera or for all connected cameras together. Two LED tubes on top and bottom of the frame are used to illuminate the scene. The LED tubes are continuously dimmable and have up to 4000 lm light output per metre with a colour temperature of about 4250 K. This was considered in the white balance settings of the cameras. The frame and the height of the camera positions are fixed. Therefore a height-adjustable chair is used to compensate for the body height variation among subjects. The chair is located in the middle of the semi-circular device with a distance of approximately 1.46 m from the cameras. By using a cross-line laser pointing to the right side of the head, the chair is adjusted until the head of the subject is positioned in the defined height of 1.20 m. The method is used in order to ensure a standardized documentation process and to allow the documentation of subjects with a body height between 1.50 m and 2.05 m without any further adjustments. The head of the subject is oriented according to the Frankfort horizontal plane as is the case for standard 2D mug shots [17]. Reference measurements are required to scale to 3D data within Agisoft PhotoScan. A stable ruler attached to a necklace offers flexibility independent of the size of the subject and the used focal length. The ruler is hung around the subject’s neck before documentation and is located at the base of the neck (Fig. 3). 2.2. Camera settings For close range photogrammetric face documentation in an indoor environment with external fixed permanent lighting, the camera parameters have to be chosen carefully. The ISO value should be set in a way that the images do not contain significant noise. We found that for our setup, the ISO value of 500 provided sufficient images. For the f-stop the short focusing distance required a closed aperture, which however was limited due to the
Fig. 1. The prototype setup of the 3D mug shot system. The cameras for the standard 2D mug shot are highlighted.
8
A. Leipner et al. / Forensic Science International 300 (2019) 6–12
dark indoor environment. We found with an aperture f/9 an acceptable depth of field for different head dimensions and changing head positions. Furthermore, all cameras were focused once on the face of a reference person and position. The restricting condition for the shutter speed is the subject to be photographed. All settings are always depended on the lighting conditions. To prevent motion blur caused by the human jitter the shutter speed was set to 1/60. Photogrammetric recommendations demand scene filling images. Three different focal lengths were tested: 50 mm, 85 mm and 135 mm. The acquired images are subsequently processed in photogrammetric software to generate a 3D model of the face. 2.3. 3D models The photogrammetric images were transferred to Agisoft PhotoScan Professional (Version 1.4.0 build 5650) and processed to true-to-scale 3D model. For the three data sets the calculations was performed with the following settings: - split camera groups (as 23 different cameras are used, the three mug shot cameras are neglected for the photogrammetry), - alignment of the photos with high accuracy, - setting of scale markers and calculating camera correction, - build dense cloud with high quality, - build mesh with high face count using dense cloud as source data, - build texture with a texture size of 8192 8192 pixels.
visible or not visible at all (for example, the ears in the frontal view or philtrum in the lateral view); in these cases the features were not assessed. Due to the flashing lights in GOM and Artec the subjects eyes are closed during the scanning procedure causing that not all features in the eye regions can be analysed. The assessment was undertaken by a forensic anthropologist experienced in facial image comparisons. The images based on 3D models with and without texture information were compared separately. The GOM-generated 3D model has no texture information and thus was not taken into account for the comparisons of images with texture. The images based on 3D models with texture information were compared also with respect to the visibility of skin discolorations or other skin surface changes. All generated 3D models were compared to high-quality 2D digital photographs (frontal, right and left lateral oriented in Frankfort Horizontal) being considered as the golden standard for the assessed features. For comparison the 3D models were imported into 3 ds Max 2017 (Autodesk Inc., San Rafael, California, USA) as OBJ-files. This was done because the software is already used for parallel projections and superimpositions and to achieve scanner software independent illumination and visualization of the models. In 3 ds Max we generated rendered images (frontal, right and left lateral oriented in Frankfort Horizontal) with and without the texture information. For the image comparison the images were turned to greyscale. The curve gradient histogram was adjusted by excluding missing colour information (in all cases the images became brighter). 3. Results
The resulting 3D models of the test person were compared with regard to surface and texture quality. 2.4. Comparison models The GOM Atos Triple Scan (GOM GmbH, Braunschweig, Germany) and the Artec Space Spider (Artec 3D, Luxemburg, Luxembourg) are two structured light scanners with high accuracy. It is for this reason that both scanners were previously used for forensic 3D face documentation. For this study we also used both to survey the test subject’s face to compare them with the photogrammetric 3D head models. One model without texture information was created with the GOM Atos Triple Scan, and two with the Artec Space Spider, one with the other without texture information both based on the same scan (Table 2). 2.5. Comparison of visible morphological features of the captured faces A predefined list of morphological features (Table 1, [17]) was used to compare the visibility of these features on the 2D images derived from the 3D models against the 2D digital photographs. For the evaluation the features were grouped by facial regions. Depending on the facial orientation, some features were not fully
3.1. Technical comparisons Three textured 3D models of the head with three different focal length settings were generated (Fig. 2). The resolution of the 3D models during the different calculation steps is shown in Table 2. In the photographs taken with 50 mm focal length, the head and neck of the test person fill the format up to a maximum of oneninth (Fig. 3). The generated 3D model of the head consists of 291.370 faces and includes the head and neck. During the data processing, the shoulder area, which was also documented in the images and therefore part in the created 3D model, was removed. The parts of the face – eyes, nose, mouth – and ears are visible. The 3D surface model exhibits visible inaccuracies in the area of the tip of the nose. The texture map of the model shows fine characteristics of the face. At the transition from the chin to the neck, the skin colour gets darker. Light reflections are visible in the eyes and on the tip of the nose. In the photographs taken with 85 mm focal length, the head and neck of the test person fill the format up to a maximum of onefourth (Fig. 3). The generated 3D model of the head consists of 690.951 faces and includes the head and the neck. In this model, the shoulder and chest region were also removed during the
Table 1 The list of predefined morphological features of the face used for the comparisons. Region/feature complex
Feature
Eye
Upper eyelid, upper lid margin, medial canthus, caruncle lacrimalis, form of the palpebral ligament, pupil, iris, lower lid margin, lower eyelid, lateral canthus, eyebrow Bridge, tip, septum, nostril, nasal wings, alar groove Philtrum, upper (normal skin) lip, vermilion border upper lip, upper (vermilion) lip, lower (vermilion) lip, vermilion border lower lip, fissure Helix, scaphoid fossa, antihelix, triangular fossa, crus helicis, tragus, incisura intertragica, antitragus, concha, lobe Transverse frontal lines, vertical glabellar lines, transverse nasal lines, superior orbital groove, lateral orbital lines, inferior orbital groove, nasolabial folds, buccomandibular groove, perioral lines, oromental groove, mandibular folds, mentolabial groove, mental fovea
Nose Mouth Ear Lines and grooves
A. Leipner et al. / Forensic Science International 300 (2019) 6–12
9
Table 2 Characteristics of the 3D models. 3D model
50 mm
85 mm
135 mm
GOM
Artec without texture
Artec with texture
Tie points Dense cloud points Mesh faces
9445 3,777,648 291,370
11,522 6,073,109 690,951
16,824 9,910,391 1,705,321
544,071
4,244,574
804,961
Fig. 2. Overview of the 3D models. The first and second row show the PhotoScan models in 50 mm (A, D), 85 mm (B, E) and 135 mm (C, F). The third row shows GOM (G), Artec without texture information (H) and Artec with texture information (I).
processing. The parts of the face – eyes, nose, mouth – and ears are clearly visible. The texture shows also fine characteristics of the face. At the transition from the chin to the neck, the skin colour gets darker, like with the 50 mm model, and light reflections appear in
the eyes and on the tip of the nose just as they do for the 135 mm focal length model. In the photographs taken with 135 mm focal length the head and neck of the test person fill the format up to a maximum of
10
A. Leipner et al. / Forensic Science International 300 (2019) 6–12
Fig. 3. Example photograph of camera 1_3 taken with 50 mm (A), 85 mm (B) and 135 mm (C) focal length.
almost one-half (Fig. 3), and a model consisting of 1.7 million faces was created. The generated 3D model did not contain the shoulder or chest region so that no cropping had to be performed. The parts of the face – eyes, nose, mouth – and ears are clearly visible and fine details of the face are visible in the texture map. Furthermore, three different 3D models were generated using two types of structured light scanners (Fig. 2). The 3D model generated with the GOM ATOS Triple Scan consists of 544,071 faces (Table 2). The parts of the face – eyes, nose, mouth – and ears are clearly structured. Facial hair (eyebrows and eyelashes) as well as head hair was not recorded with the scanner. Due to the flashing light of the projector the eyes of the test person are closed. The surface of the 3D model appears rough and there are some errors in the surface of the left side of the nose, mouth region and chin. There are small holes in the regions of the mouth, chin, neck and ears, which were not captured by the scanner. The 3D model has no texture. With the Artec Space Spider two different 3D models were generated based on the same scan. The 3D model without texture has a resolution of 4,244,574 faces (Tables 2). The 3D model with texture information has a resolution of 804,961 faces because of a system required reduction (Table 2). The 3D information of both models is the same, but finer structures appear in the surface of the model without texture. In the 3D models, the parts of the face – eyes, nose, mouth – and ears are clearly structured. Facial hair (eyebrows and eyelashes) as well as head hair were not scanned. The eyes are closed due to the flashing light of the projector. There are errors in the surface structure at the chin and the neck, resulting in small holes like with the GOM Atos scanner. The concha of the ears is not visible. The 3D model with reduced resolution is textured and shows fine characteristics of the face. Light reflections are visible on the nose, the chin and the forehead. The texture information appears darker compared with the photogrammetric models.
3.2. Morphological comparisons The comparisons are summarized in the Tables 3–8 providing an overview about the number of features assessed in the given region and visible in the images extracted from the individual 3D models. In the PhotoScan models, more features were assessed than in GOM and Artec. The reason for this is the scanning process, since the test person needs to be captured with the eyes closed for GOM and Artec scanning. In the frontal view images based on 3D models without texture information (Table 3), the image from Artec-generated model achieved best results as 70% of the present features were visible. The PhotoScan model with 135 mm focal length was second best with 34% visible features. The lowest number of features was visible in the PhotoScan model with 50 mm focal length. In the right lateral view images based on 3D models without texture information (Table 4), the GOM-generated model showed 41% visible features of the 29 present. In this model, more features were visible in the nose and mouth region compared to the other models. As with the frontal view, the lowest number of visible features was recorded in the PhotoScan model with 50 mm focal length. In the left lateral view images based on 3D models without texture information (Table 5), the Artec model achieved best results with 48%, followed by GOM and PhotoScan 135 mm with 17% and 16%. Only two features were visible in the PhotoScan models with 50 and 85 mm focal length. For all models, except for the Artec-generated model, fewer features were visible on the left than on the right facial profiles. In the frontal view images based on 3D models with texture information (Table 6), PhotoScan 135 mm and Artec both achieved 77% visible features. Compared to the results of Table 3 there were significantly more features visible with texture information. For
Table 3 The visibility of morphological features of the face captured in the frontal view using models without texture (the numerator is the number of visible features, the denominator the number of assessable features). Region/features
50 mm
85 mm
135 mm
GOM
Artec
Eye Nose Mouth Ear Lines and grooves Total
0/11 0/6 2/7 0/1 3/13 5/38 (13%)
0/11 0/6 1/7 0/1 6/13 7/38 (18%)
2/11 3/6 3/7 0/1 5/13 13/38 (34%)
0/3 0/6 4/7 1/1 4/13 9/30 (30%)
2/3 4/6 5/7 1/1 9/13 21/30 (70%)
A. Leipner et al. / Forensic Science International 300 (2019) 6–12 Table 4 The visibility of morphological features of the face captured in the right lateral view using models without texture (the numerator is the number of visible features, the denominator the number of assessable features).
11
Table 8 The visibility of morphological features of the face captured in the left lateral view using models without texture (the numerator is the number of visible features, the denominator the number of assessable features).
Region/features
50 mm
85 mm
135 mm
GOM
Artec
Region/features
50 mm
85 mm
135 mm
Artec
Eye Nose Mouth Ear Lines and grooves Total
0/3 1/6 1/6 0/10 2/6
0/3 2/6 3/6 0/10 2/6
0/3 2/6 4/6 0/10 2/6
0/1 3/6 5/6 0/10 4/6
0/1 2/6 3/6 0/10 5/6
0/3 1/6 3/6 2/10 2/6 0/1
0/3 2/6 2/6 1/10 2/6 0/1
1/3 1/6 3/6 3/10 3/6 1/1
0/1 4/6 5/6 0/10 4/6 1/1
4/31 (13%)
7/31 (23%)
8/31 (26%)
12/29 (41%)
10/29 (34%)
Eye Nose Mouth Ear Lines and grooves Skin discolorations/ changes Total
8/32 (25%)
7/32 (22%)
12/32 (38%)
14/30 (47%)
Table 5 The visibility of morphological features of the face captured in the left lateral view using models without texture (the numerator is the number of visible features, the denominator the number of assessable features). Region/features
50 mm
85 mm
135 mm
GOM
Artec
Eye Nose Mouth Ear Lines and grooves Total
0/3 1/6 1/6 0/10 0/6
0/3 1/6 0/6 0/10 1/6
0/3 1/6 1/6 0/10 3/6
0/1 2/6 0/6 0/10 3/6
0/1 4/6 5/6 0/10 5/6
2/31 (6%)
2/31 (6%)
5/31 (16%)
5/29 (17%)
14/29 (48%)
Table 6 The visibility of morphological features of the face captured in the frontal view using models with texture (the numerator is the number of visible features, the denominator the number of assessable features). Region/features
50 mm
85 mm
135 mm
Artec
Eye Nose Mouth Ear Lines and grooves Skin discolorations/ changes Total
8/11 0/6 6/7 1/1 12/13 0/1
8/11 1/6 6/7 0/1 13/13 1/1
9/11 2/6 7/7 0/1 11/13 1/1
2/3 4/6 6/7 1/1 10/13 1/1
27/39 (69%)
29/39 (74%)
30/39 (77%)
24/31 (77%)
Table 7 The visibility of morphological features of the face captured in the right lateral view using models without texture (the numerator is the number of visible features, the denominator the number of assessable features). Region/Features
50 mm
85 mm
135 mm
Artec
Eye Nose Mouth Ear Lines and grooves Skin discolorations/ changes Total
1/3 1/6 5/6 5/10 5/6 0/1
2/3 2/6 5/6 4/10 5/6 1/1
1/3 2/6 5/6 4/10 6/6 1/1
0/1 3/6 5/6 2/10 4/6 1/1
17/32 (53%)
19/32 (59%)
19/32 (59%)
15/30 (50%)
example for the PhotoScan with 50 mm the number of visible features increased from five to 27 features. In the right lateral view images based on 3D models with texture information (Table 7), the PhotoScan models with 85 mm and 135 mm achieved best results with 59% visibility of features. In the left lateral view images based on 3D models with texture information (Table 8), the Artec model achieved best results with 47% visibility.
In general, the Artec-generated models achieved very good results in the visibility of the features, with or without texture information. Taking only the models with texture information into account, the comparison with the gold standard showed similar results in the visibility of features for Artec and PhotoScan 135 mm. For all PhotoScan models the results were better with texture information. 4. Discussion We have developed a fast 3D mug shot system for the documentation of human faces. The documentation itself only takes a split second. The system also includes the standard 2D mug shots. The resulting 3D models accurately represent the geometry of the face including detailed colour and texture information. With the 3D model the orientation of the face can be adjusted in rotation, location and distance accordingly to the reference images of a surveillance camera. Furthermore, the virtual parameters of the camera, such as focal length can be adjusted according to the surveillance camera allowing comparison between surveillance footage and the virtual camera image. The results of the comparison show that the 3D mug shot system with its photogrammetric documentation generates 3D models with surface quality comparable to other methods like surface scans with an Artec Space Spider, or even better quality, compared to GOM Atos Triple Scan. The comparison of the photogrammetric models with the gold standard (2D digital photographs of high quality) showed that there was an improvement of the PhotoScan models from 50 mm to 135 mm. The photogrammetric 3D models created with a 135 mm focal length lead to a more detailed surface structure and reduced errors due to the format filling photographs compared to images with 50 mm focal length. The format filling images enable Agisoft PhotoScan to generate the 3D model with more details, which is visible in the larger dense cloud. The rendered 2D images for the evaluation of the 3D data were generated with 3 ds Max. In general, the quality of the visualized data in the scanner software is better than in 3 ds Max, especially with Agisoft PhotoScan due to restrictions in 3 ds Max regarding the texture quality. Furthermore, the lighting in 3 ds Max caused shadows, which in turn caused reduced visibility of some features but can also be adjusted dependent on the case. However, shadows which are already part of the texture cannot be prevented and disturb the evaluation and should therefore be avoided by illuminating the subject during the scan procedure. A closer evaluation also showed that the lighting conditions during the photogrammetric documentation were not even on both sides of the face. The resulting 3D models are less illuminated on the left side of the face while the right side appears brighter. This is due to
12
A. Leipner et al. / Forensic Science International 300 (2019) 6–12
the room, which has windows on the right side adding additional light to the right side of the face. This – together with the lighting in 3 ds Max – may have caused the differences in the visibility of features observed between the right and left lateral profile images. The side differences in feature visibility were observed for all, except for the Artec-generated models. In general, ear features were rarely visible in all 3D models. The photographs, as gold standard, capture more details in this region. Given that ear characteristics belong to the most stable and individualizing traits, the low visibility of the features in the 3D models requires adjustments to the lighting and scanning setup to achieve the desired quality for forensic identification purposes. The scan process of GOM and Artec can be improved in this area by manually adjusting the scanning position and generating more information of the ear features. For the 3D mug shot system it has to be evaluated, if the improvement of the light conditions or the addition of one or more cameras would solve this issue. There are several advantages of the photogrammetric documentation compared to structured light scanners. It is possible to document head and facial hair within the images and therefore in the texture map. Additionally, information about the hair colour at the time of the documentation is available. Photogrammetry enables the documentation of a person with eyes open, in difference to structured light scanners (independent of the system) where open eyes are not advisable. The flickering of the light might cause epileptic seizures and the person is incapable to keep the eyes open during the whole documentation period, and opening and closing of the eyes would cause movement and thus generate errors in the scan data [7]. The 3D documentation process, independent of the system used, requires the cooperation of the person to be documented. The scanning with a structured light scanner, even with expert operators, takes about five minutes, or might take up to ten minutes depending on the system, assuming a smooth process without interruptions. This is a long period of time during which the subject has to sit still. Furthermore, advanced training and experience is required to align misaligned scans and to analyse the scan data during the scanning procedure to prevent holes in the surface. In comparison, the 3D mug shot system operates within a fraction of a second preventing any disturbance and the fixed camera setup allows even untrained operators to perform a documentation procedure. If necessary, several shots can be made directly one after another and the immediately available photos allow the operator to check for errors right after the acquisition. Independently from the system used, glossy skin, three-day stubble and unruly beards have negative influence on the 3D model quality. Glossy skin may be reduced with makeup towels. Head hair should generally be moved away from the face region, for example with a headband, most importantly to be able to capture detailed surface information of the ears. Moreover, in any of the systems, it is not possible to document glasses, which should be removed before the documentation procedure. The combination of 3D mug shots and standard 2D mug shots requires the correct orientation of the head of the person to be documented. The exact positioning is also important with the used focal length of 135 mm. The focal length generates format filling photos, therefore the space around the head is small. With the help of an instructions manual and the cross-line laser, a standardized procedure can be guaranteed. The settings of the cameras for the photogrammetric documentation (ISO 500, f-stop f/9 and shutter speed 1/60) are caused by the dark lighting conditions. As the setup is still a prototype, there are no covers over the frame of the cameras. There is also no white panel behind the chair. Both, the cover and the rear panel, will ensure that the light will become softer and better illuminate the subject. This will optimize the lighting conditions and camera
settings can be improved accordingly, resulting in a further improvement of the 3D surface structure. 5. Conclusion In conclusion, the 3D mug shot system is a fast and efficient tool to generate 3D models of the human head and face. The major advantage of the system is that the generated facial 3D models can be adjusted in head orientation to the perspective captured on the comparative surveillance footage, thus improving the conditions for image comparisons between the mug shot and the person of interest. With the current limitations regarding the visibility of features on the PhotoScan models, the 3D mug shot system may be used as a complementary tool along with the traditional 2D mug shots for the purpose of visual forensic identification. CRediT authorship contribution statement Anja Leipner: Conceptualization, Data curation, Investigation, Methodology, Project administration, Visualization, Writing original draft, Writing - review & editing. Zuzana Obertová: Formal analysis, Methodology, Validation, Writing - review & editing. Martin Wermuth: Conceptualization, Software. Michael Thali: Funding acquisition, Resources, Supervision. Thomas Ottiker: Funding acquisition, Resources, Supervision. Till Sieberth: Methodology, Project administration, Validation, Writing review & editing. References [1] U. Park, A.K. Jain, Face matching and retrieval using soft biometrics, IEEE Trans. Inf. Forensics Secur. 5 (September (3)) (2010) 406–415. [2] S. Biswas, K.W. Bowyer, P.J. Flynn, A study of face recognition of identical twins by humans, 2011 IEEE International Workshop on Information Forensics and Security (2011) 1–6. [3] M. Maguire, The birth of biometric security, Anthropol. Today 25 (April (2)) (2009) 9–14. [4] A.K. Jain, B. Klare, U. Park, Face recognition: some challenges in forensics, Face and Gesture 2011 (2011) 726–733. [5] C. Hong Liu, C.A. Collin, A.M. Burton, A. Chaudhuri, Lighting direction affects recognition of untextured faces in photographic positive and negative, Vision Res. 39 (December (24)) (1999) 4003–4009. [6] U. Buck, S. Naether, K. Kreutz, M. Thali, Geometric facial comparisons in speedcheck photographs, Int. J. Leg. Med. 125 (November (6)) (2011) 785–790. [7] A. Modabber, et al., Evaluation of the accuracy of a mobile and a stationary system for three-dimensional facial scanning, J. Craniomaxillofac. Surg. 44 (October (10)) (2016) 1719–1724. [8] J.J. Secher, T.A. Darvann, E.M. Pinholt, Accuracy and reproducibility of the DAVID SLS-2 scanner in three-dimensional facial imaging, J. Craniomaxillofac. Surg. 45 (October (10)) (2017) 1662–1670. [9] D. Smeets, P. Claes, D. Vandermeulen, J.G. Clement, Objective 3D face recognition: evolution, approaches and challenges, Forensic Sci. Int. 201 (Septenber (1–3)) (2010) 125–132. [10] S.M. Weinberg, N.M. Scott, K. Neiswanger, C.A. Brandon, M.L. Marazita, Digital three-dimensional photogrammetry: evaluation of anthropometric precision and accuracy using a genex 3D camera system, Cleft Palate Craniofac. J. 41 (September (5)) (2004) 507–518. [11] J. Schneider, T. Läbe, W. Förstner, Incremental real-time bundle adjustment for multi-camera systems with points at infinity, ISPRS: Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XL-1 (August (W2)) (2013) 355–360. [12] A. Malian, A. Azizi, MEDPHOS: a new photogrammetric system for medical measurement, International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 35 (January) (2004). [13] I. Detchev, M. Mazaheri, S. Rondeel, A. Habib, Calibration of multi-camera photogrammetric systems, ISPRS: Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XL–1 (November) (2014) 101–108. [14] A. Leipner, R. Baumeister, M.J. Thali, M. Braun, E. Dobler, L.C. Ebert, Multicamera system for 3D forensic documentation, Forensic Sci. Int. 261 (April) (2016) 123–128. [15] R. Michienzi, S. Meier, L.C. Ebert, R.M. Martinez, T. Sieberth, Comparison of forensic photo-documentation to a photogrammetric solution using the multi-camera system ‘Botscan’, Forensic Sci. Int. 288 (July) (2018) 46–52. [16] Agisoft PhotoScan User Manual – Professional Edition, Version 1.4,” p. 127. [17] K. Taylor, Forensic Art and Illustration, CRC Press LLC, 2001.
Contents lists available at ScienceDirect
Forensic Science International journal homepage: www.elsevier.com/locate/forsciint
3D mug shot—3D head models from photogrammetry for forensic identification Anja Leipnera,* , Zuzana Obertováa , Martin Wermutha , Michael Thalib , Thomas Ottikera , Till Sieberthb a b
Zurich Forensic Science Institute, Zeughausstrasse 11, CH-8004 Zurich, Switzerland Institute of Forensic Medicine, University of Zurich, Winterthurerstrasse 190/52, CH-8057 Zurich, Switzerland
A R T I C L E I N F O
A B S T R A C T
Article history: Available online 22 April 2019
No human face is like another, not even in monozygotic twins, which makes the face one of the most individualizing characteristic. It is for this reason that the human face is commonly used for identification purposes and police officers take portrait photographs of arrested persons, so-called mug shots. The disadvantage of these 2D mug shots is that the perspective, in which they are taken (usually frontal and lateral-right, left or both), cannot be changed after acquisition, thus limiting a potential comparison between a mug shot and surveillance footage or other visual recordings. Documenting a face in 3D would reduce this problem as it allows adjusting the perspective of the face for image comparisons depending on the needs of the investigator. We have developed a 3D mug shot system containing 26 digital single-lens reflex cameras arranged semi-circularly in a 200 arc with a 1.46 m radius around a height-adjustable chair. We generated photogrammetric models of a test person’s face captured by the mug shot system using three different focal lengths settings as well as 3D models of the same face with GOM Atos Triple Scan and Artec Space Spider. The 3D models were then analysed regarding the visibility of detailed morphological features in different regions of the face compared to 2D mug shots. Our results showed that our 3D mug shot system with its photogrammetric documentation generates 3D models with comparable surface quality to Artec-generated models, or even better quality, compared to GOM-generated models. The results of the morphological assessment were affected by the focal length and availability of texture information. In conclusion, the 3D mug shot system is a fast and efficient tool to generate 3D models of the face and may be used in addition to 2D photographs for the purpose of visual forensic identification based on images. © 2019 Elsevier B.V. All rights reserved.
Keywords: Photography Photogrammetry Forensic documentation 3D modelling 3D reconstruction Person identification
1. Introduction No human face is like another, not even in monozygotic twins, which makes the face one of the most individualizing characteristic between humans [1,2]. It is for this reason that the human face is commonly used for identification purposes and police officers take portrait photographs of arrested persons, so-called mug shots [3]. In 1888 Alphonse Bertillon standardized the photographic documentation of criminals by acquiring images from frontal and profile views, which has not significantly changed since then [3]. The disadvantage of these 2D mug shots is that the perspective, in which they are taken, cannot be changed
* Corresponding author. E-mail address: [email protected] (A. Leipner). https://doi.org/10.1016/j.forsciint.2019.04.015 0379-0738/© 2019 Elsevier B.V. All rights reserved.
after acquisition, thus limiting a potential comparison between the mug shot and oblique visual recordings like usually acquired by surveillance cameras. The comparison process can be facilitated by acquiring additional photographs of the face. The photographs can be taken in a comparable perspective. But there is no guaranty to achieve the correct viewing angle. Furthermore, the lighting situation and shadows can variably influence the visibility of morphological facial features captured on the images and thus the subsequent comparison [4,5]. Documenting a face in 3D would reduce these problems as it allows adjusting the perspective of the face for image comparisons depending on the needs of the investigator. Current methods to document a face in 3D include laser scanning, structured light scanning [6,8] or single camera photogrammetry [5,9,10]. However, these methods might not offer colour and texture information or the quality of the 3D model is inadequate. The methods also
A. Leipner et al. / Forensic Science International 300 (2019) 6–12
differ in the geometric resolution of the generated 3D model. Another considerable factor is the time required to perform the 3D documentation of participant human subject, which may be prolonged and the subject is required to sit still during the procedure. Multi-camera systems are a fast 3D documentation method, which overcomes this issue due to their swift scanning procedure and therefore is often used for real time 3D documentation [11–13]. The method presented by Leipner et al. [14] is also based on a multi-camera device and is used for forensic documentation of body height and for medical examination [15]. As the system has been initially developed to document the whole body of a subject, detailed information of the face is only visible as texture information in the resulting 3D model but not as a 3D surface. In this paper, we introduce and test a 3D mug shot system for facial images specifically developed for the purpose of forensic identification. 2. Method 2.1. System setup Based on the experience with the multi-camera system for the documentation of the whole body [14], we have developed a 3D mug shot system for facial images. The system contains 26 digital single-lens reflex (DSLR) cameras arranged semi-circularly in 200 arc with a 1.46 m radius around a height-adjustable chair (Fig. 1). The cameras are of type Canon EOS 80D (Canon Inc., Tokyo, Japan). For photogrammetric documentation, 23 of the cameras are equipped with Canon EF-S 18–135 mm f/3.5–5.6 IS USM lenses offering flexibility in the prototype phase to test several focal lengths. The remaining three cameras are equipped with a Canon EF-S 60 mm f/2.8 Macro USM lenses. The semi-circular frame consists of two rows with 10 DSLRs each with 21 between the individual cameras. The first row of cameras is 1.67 m above ground level and the second row is 0.74 m above ground level. Further, three DSLRs are placed centred approximately 0.26 m under the lower row with the focus on the subject’s chin area. The three DSLRs with the fixed lenses are placed centred, right and left of the head position in 1.17 m above ground for standard 2D mug shots (Fig. 1). The 23 cameras for the photogrammetric documentation are in portrait orientation and the three 2D mug shot cameras are in landscape orientation. The position and alignment of the DSLRs was established based on the area of interest for the
7
3D mug shot and the requirements of at least 60% image overlap for photogrammetric documentation and needs to be done only once [16]. The cameras are connected to a workstation, and the camera release is a synchronized remote release. The data download is performed via USB. For setting the camera parameters a workstation with the software Smart Shooter 3 (Kuvacode Oy, Kerava, Finland) is used. Smart Shooter 3 provides an overview over all cameras simultaneously and allows adjusting the camera settings such as ISO, shutter speed and aperture, individually for each camera or for all connected cameras together. Two LED tubes on top and bottom of the frame are used to illuminate the scene. The LED tubes are continuously dimmable and have up to 4000 lm light output per metre with a colour temperature of about 4250 K. This was considered in the white balance settings of the cameras. The frame and the height of the camera positions are fixed. Therefore a height-adjustable chair is used to compensate for the body height variation among subjects. The chair is located in the middle of the semi-circular device with a distance of approximately 1.46 m from the cameras. By using a cross-line laser pointing to the right side of the head, the chair is adjusted until the head of the subject is positioned in the defined height of 1.20 m. The method is used in order to ensure a standardized documentation process and to allow the documentation of subjects with a body height between 1.50 m and 2.05 m without any further adjustments. The head of the subject is oriented according to the Frankfort horizontal plane as is the case for standard 2D mug shots [17]. Reference measurements are required to scale to 3D data within Agisoft PhotoScan. A stable ruler attached to a necklace offers flexibility independent of the size of the subject and the used focal length. The ruler is hung around the subject’s neck before documentation and is located at the base of the neck (Fig. 3). 2.2. Camera settings For close range photogrammetric face documentation in an indoor environment with external fixed permanent lighting, the camera parameters have to be chosen carefully. The ISO value should be set in a way that the images do not contain significant noise. We found that for our setup, the ISO value of 500 provided sufficient images. For the f-stop the short focusing distance required a closed aperture, which however was limited due to the
Fig. 1. The prototype setup of the 3D mug shot system. The cameras for the standard 2D mug shot are highlighted.
8
A. Leipner et al. / Forensic Science International 300 (2019) 6–12
dark indoor environment. We found with an aperture f/9 an acceptable depth of field for different head dimensions and changing head positions. Furthermore, all cameras were focused once on the face of a reference person and position. The restricting condition for the shutter speed is the subject to be photographed. All settings are always depended on the lighting conditions. To prevent motion blur caused by the human jitter the shutter speed was set to 1/60. Photogrammetric recommendations demand scene filling images. Three different focal lengths were tested: 50 mm, 85 mm and 135 mm. The acquired images are subsequently processed in photogrammetric software to generate a 3D model of the face. 2.3. 3D models The photogrammetric images were transferred to Agisoft PhotoScan Professional (Version 1.4.0 build 5650) and processed to true-to-scale 3D model. For the three data sets the calculations was performed with the following settings: - split camera groups (as 23 different cameras are used, the three mug shot cameras are neglected for the photogrammetry), - alignment of the photos with high accuracy, - setting of scale markers and calculating camera correction, - build dense cloud with high quality, - build mesh with high face count using dense cloud as source data, - build texture with a texture size of 8192 8192 pixels.
visible or not visible at all (for example, the ears in the frontal view or philtrum in the lateral view); in these cases the features were not assessed. Due to the flashing lights in GOM and Artec the subjects eyes are closed during the scanning procedure causing that not all features in the eye regions can be analysed. The assessment was undertaken by a forensic anthropologist experienced in facial image comparisons. The images based on 3D models with and without texture information were compared separately. The GOM-generated 3D model has no texture information and thus was not taken into account for the comparisons of images with texture. The images based on 3D models with texture information were compared also with respect to the visibility of skin discolorations or other skin surface changes. All generated 3D models were compared to high-quality 2D digital photographs (frontal, right and left lateral oriented in Frankfort Horizontal) being considered as the golden standard for the assessed features. For comparison the 3D models were imported into 3 ds Max 2017 (Autodesk Inc., San Rafael, California, USA) as OBJ-files. This was done because the software is already used for parallel projections and superimpositions and to achieve scanner software independent illumination and visualization of the models. In 3 ds Max we generated rendered images (frontal, right and left lateral oriented in Frankfort Horizontal) with and without the texture information. For the image comparison the images were turned to greyscale. The curve gradient histogram was adjusted by excluding missing colour information (in all cases the images became brighter). 3. Results
The resulting 3D models of the test person were compared with regard to surface and texture quality. 2.4. Comparison models The GOM Atos Triple Scan (GOM GmbH, Braunschweig, Germany) and the Artec Space Spider (Artec 3D, Luxemburg, Luxembourg) are two structured light scanners with high accuracy. It is for this reason that both scanners were previously used for forensic 3D face documentation. For this study we also used both to survey the test subject’s face to compare them with the photogrammetric 3D head models. One model without texture information was created with the GOM Atos Triple Scan, and two with the Artec Space Spider, one with the other without texture information both based on the same scan (Table 2). 2.5. Comparison of visible morphological features of the captured faces A predefined list of morphological features (Table 1, [17]) was used to compare the visibility of these features on the 2D images derived from the 3D models against the 2D digital photographs. For the evaluation the features were grouped by facial regions. Depending on the facial orientation, some features were not fully
3.1. Technical comparisons Three textured 3D models of the head with three different focal length settings were generated (Fig. 2). The resolution of the 3D models during the different calculation steps is shown in Table 2. In the photographs taken with 50 mm focal length, the head and neck of the test person fill the format up to a maximum of oneninth (Fig. 3). The generated 3D model of the head consists of 291.370 faces and includes the head and neck. During the data processing, the shoulder area, which was also documented in the images and therefore part in the created 3D model, was removed. The parts of the face – eyes, nose, mouth – and ears are visible. The 3D surface model exhibits visible inaccuracies in the area of the tip of the nose. The texture map of the model shows fine characteristics of the face. At the transition from the chin to the neck, the skin colour gets darker. Light reflections are visible in the eyes and on the tip of the nose. In the photographs taken with 85 mm focal length, the head and neck of the test person fill the format up to a maximum of onefourth (Fig. 3). The generated 3D model of the head consists of 690.951 faces and includes the head and the neck. In this model, the shoulder and chest region were also removed during the
Table 1 The list of predefined morphological features of the face used for the comparisons. Region/feature complex
Feature
Eye
Upper eyelid, upper lid margin, medial canthus, caruncle lacrimalis, form of the palpebral ligament, pupil, iris, lower lid margin, lower eyelid, lateral canthus, eyebrow Bridge, tip, septum, nostril, nasal wings, alar groove Philtrum, upper (normal skin) lip, vermilion border upper lip, upper (vermilion) lip, lower (vermilion) lip, vermilion border lower lip, fissure Helix, scaphoid fossa, antihelix, triangular fossa, crus helicis, tragus, incisura intertragica, antitragus, concha, lobe Transverse frontal lines, vertical glabellar lines, transverse nasal lines, superior orbital groove, lateral orbital lines, inferior orbital groove, nasolabial folds, buccomandibular groove, perioral lines, oromental groove, mandibular folds, mentolabial groove, mental fovea
Nose Mouth Ear Lines and grooves
A. Leipner et al. / Forensic Science International 300 (2019) 6–12
9
Table 2 Characteristics of the 3D models. 3D model
50 mm
85 mm
135 mm
GOM
Artec without texture
Artec with texture
Tie points Dense cloud points Mesh faces
9445 3,777,648 291,370
11,522 6,073,109 690,951
16,824 9,910,391 1,705,321
544,071
4,244,574
804,961
Fig. 2. Overview of the 3D models. The first and second row show the PhotoScan models in 50 mm (A, D), 85 mm (B, E) and 135 mm (C, F). The third row shows GOM (G), Artec without texture information (H) and Artec with texture information (I).
processing. The parts of the face – eyes, nose, mouth – and ears are clearly visible. The texture shows also fine characteristics of the face. At the transition from the chin to the neck, the skin colour gets darker, like with the 50 mm model, and light reflections appear in
the eyes and on the tip of the nose just as they do for the 135 mm focal length model. In the photographs taken with 135 mm focal length the head and neck of the test person fill the format up to a maximum of
10
A. Leipner et al. / Forensic Science International 300 (2019) 6–12
Fig. 3. Example photograph of camera 1_3 taken with 50 mm (A), 85 mm (B) and 135 mm (C) focal length.
almost one-half (Fig. 3), and a model consisting of 1.7 million faces was created. The generated 3D model did not contain the shoulder or chest region so that no cropping had to be performed. The parts of the face – eyes, nose, mouth – and ears are clearly visible and fine details of the face are visible in the texture map. Furthermore, three different 3D models were generated using two types of structured light scanners (Fig. 2). The 3D model generated with the GOM ATOS Triple Scan consists of 544,071 faces (Table 2). The parts of the face – eyes, nose, mouth – and ears are clearly structured. Facial hair (eyebrows and eyelashes) as well as head hair was not recorded with the scanner. Due to the flashing light of the projector the eyes of the test person are closed. The surface of the 3D model appears rough and there are some errors in the surface of the left side of the nose, mouth region and chin. There are small holes in the regions of the mouth, chin, neck and ears, which were not captured by the scanner. The 3D model has no texture. With the Artec Space Spider two different 3D models were generated based on the same scan. The 3D model without texture has a resolution of 4,244,574 faces (Tables 2). The 3D model with texture information has a resolution of 804,961 faces because of a system required reduction (Table 2). The 3D information of both models is the same, but finer structures appear in the surface of the model without texture. In the 3D models, the parts of the face – eyes, nose, mouth – and ears are clearly structured. Facial hair (eyebrows and eyelashes) as well as head hair were not scanned. The eyes are closed due to the flashing light of the projector. There are errors in the surface structure at the chin and the neck, resulting in small holes like with the GOM Atos scanner. The concha of the ears is not visible. The 3D model with reduced resolution is textured and shows fine characteristics of the face. Light reflections are visible on the nose, the chin and the forehead. The texture information appears darker compared with the photogrammetric models.
3.2. Morphological comparisons The comparisons are summarized in the Tables 3–8 providing an overview about the number of features assessed in the given region and visible in the images extracted from the individual 3D models. In the PhotoScan models, more features were assessed than in GOM and Artec. The reason for this is the scanning process, since the test person needs to be captured with the eyes closed for GOM and Artec scanning. In the frontal view images based on 3D models without texture information (Table 3), the image from Artec-generated model achieved best results as 70% of the present features were visible. The PhotoScan model with 135 mm focal length was second best with 34% visible features. The lowest number of features was visible in the PhotoScan model with 50 mm focal length. In the right lateral view images based on 3D models without texture information (Table 4), the GOM-generated model showed 41% visible features of the 29 present. In this model, more features were visible in the nose and mouth region compared to the other models. As with the frontal view, the lowest number of visible features was recorded in the PhotoScan model with 50 mm focal length. In the left lateral view images based on 3D models without texture information (Table 5), the Artec model achieved best results with 48%, followed by GOM and PhotoScan 135 mm with 17% and 16%. Only two features were visible in the PhotoScan models with 50 and 85 mm focal length. For all models, except for the Artec-generated model, fewer features were visible on the left than on the right facial profiles. In the frontal view images based on 3D models with texture information (Table 6), PhotoScan 135 mm and Artec both achieved 77% visible features. Compared to the results of Table 3 there were significantly more features visible with texture information. For
Table 3 The visibility of morphological features of the face captured in the frontal view using models without texture (the numerator is the number of visible features, the denominator the number of assessable features). Region/features
50 mm
85 mm
135 mm
GOM
Artec
Eye Nose Mouth Ear Lines and grooves Total
0/11 0/6 2/7 0/1 3/13 5/38 (13%)
0/11 0/6 1/7 0/1 6/13 7/38 (18%)
2/11 3/6 3/7 0/1 5/13 13/38 (34%)
0/3 0/6 4/7 1/1 4/13 9/30 (30%)
2/3 4/6 5/7 1/1 9/13 21/30 (70%)
A. Leipner et al. / Forensic Science International 300 (2019) 6–12 Table 4 The visibility of morphological features of the face captured in the right lateral view using models without texture (the numerator is the number of visible features, the denominator the number of assessable features).
11
Table 8 The visibility of morphological features of the face captured in the left lateral view using models without texture (the numerator is the number of visible features, the denominator the number of assessable features).
Region/features
50 mm
85 mm
135 mm
GOM
Artec
Region/features
50 mm
85 mm
135 mm
Artec
Eye Nose Mouth Ear Lines and grooves Total
0/3 1/6 1/6 0/10 2/6
0/3 2/6 3/6 0/10 2/6
0/3 2/6 4/6 0/10 2/6
0/1 3/6 5/6 0/10 4/6
0/1 2/6 3/6 0/10 5/6
0/3 1/6 3/6 2/10 2/6 0/1
0/3 2/6 2/6 1/10 2/6 0/1
1/3 1/6 3/6 3/10 3/6 1/1
0/1 4/6 5/6 0/10 4/6 1/1
4/31 (13%)
7/31 (23%)
8/31 (26%)
12/29 (41%)
10/29 (34%)
Eye Nose Mouth Ear Lines and grooves Skin discolorations/ changes Total
8/32 (25%)
7/32 (22%)
12/32 (38%)
14/30 (47%)
Table 5 The visibility of morphological features of the face captured in the left lateral view using models without texture (the numerator is the number of visible features, the denominator the number of assessable features). Region/features
50 mm
85 mm
135 mm
GOM
Artec
Eye Nose Mouth Ear Lines and grooves Total
0/3 1/6 1/6 0/10 0/6
0/3 1/6 0/6 0/10 1/6
0/3 1/6 1/6 0/10 3/6
0/1 2/6 0/6 0/10 3/6
0/1 4/6 5/6 0/10 5/6
2/31 (6%)
2/31 (6%)
5/31 (16%)
5/29 (17%)
14/29 (48%)
Table 6 The visibility of morphological features of the face captured in the frontal view using models with texture (the numerator is the number of visible features, the denominator the number of assessable features). Region/features
50 mm
85 mm
135 mm
Artec
Eye Nose Mouth Ear Lines and grooves Skin discolorations/ changes Total
8/11 0/6 6/7 1/1 12/13 0/1
8/11 1/6 6/7 0/1 13/13 1/1
9/11 2/6 7/7 0/1 11/13 1/1
2/3 4/6 6/7 1/1 10/13 1/1
27/39 (69%)
29/39 (74%)
30/39 (77%)
24/31 (77%)
Table 7 The visibility of morphological features of the face captured in the right lateral view using models without texture (the numerator is the number of visible features, the denominator the number of assessable features). Region/Features
50 mm
85 mm
135 mm
Artec
Eye Nose Mouth Ear Lines and grooves Skin discolorations/ changes Total
1/3 1/6 5/6 5/10 5/6 0/1
2/3 2/6 5/6 4/10 5/6 1/1
1/3 2/6 5/6 4/10 6/6 1/1
0/1 3/6 5/6 2/10 4/6 1/1
17/32 (53%)
19/32 (59%)
19/32 (59%)
15/30 (50%)
example for the PhotoScan with 50 mm the number of visible features increased from five to 27 features. In the right lateral view images based on 3D models with texture information (Table 7), the PhotoScan models with 85 mm and 135 mm achieved best results with 59% visibility of features. In the left lateral view images based on 3D models with texture information (Table 8), the Artec model achieved best results with 47% visibility.
In general, the Artec-generated models achieved very good results in the visibility of the features, with or without texture information. Taking only the models with texture information into account, the comparison with the gold standard showed similar results in the visibility of features for Artec and PhotoScan 135 mm. For all PhotoScan models the results were better with texture information. 4. Discussion We have developed a fast 3D mug shot system for the documentation of human faces. The documentation itself only takes a split second. The system also includes the standard 2D mug shots. The resulting 3D models accurately represent the geometry of the face including detailed colour and texture information. With the 3D model the orientation of the face can be adjusted in rotation, location and distance accordingly to the reference images of a surveillance camera. Furthermore, the virtual parameters of the camera, such as focal length can be adjusted according to the surveillance camera allowing comparison between surveillance footage and the virtual camera image. The results of the comparison show that the 3D mug shot system with its photogrammetric documentation generates 3D models with surface quality comparable to other methods like surface scans with an Artec Space Spider, or even better quality, compared to GOM Atos Triple Scan. The comparison of the photogrammetric models with the gold standard (2D digital photographs of high quality) showed that there was an improvement of the PhotoScan models from 50 mm to 135 mm. The photogrammetric 3D models created with a 135 mm focal length lead to a more detailed surface structure and reduced errors due to the format filling photographs compared to images with 50 mm focal length. The format filling images enable Agisoft PhotoScan to generate the 3D model with more details, which is visible in the larger dense cloud. The rendered 2D images for the evaluation of the 3D data were generated with 3 ds Max. In general, the quality of the visualized data in the scanner software is better than in 3 ds Max, especially with Agisoft PhotoScan due to restrictions in 3 ds Max regarding the texture quality. Furthermore, the lighting in 3 ds Max caused shadows, which in turn caused reduced visibility of some features but can also be adjusted dependent on the case. However, shadows which are already part of the texture cannot be prevented and disturb the evaluation and should therefore be avoided by illuminating the subject during the scan procedure. A closer evaluation also showed that the lighting conditions during the photogrammetric documentation were not even on both sides of the face. The resulting 3D models are less illuminated on the left side of the face while the right side appears brighter. This is due to
12
A. Leipner et al. / Forensic Science International 300 (2019) 6–12
the room, which has windows on the right side adding additional light to the right side of the face. This – together with the lighting in 3 ds Max – may have caused the differences in the visibility of features observed between the right and left lateral profile images. The side differences in feature visibility were observed for all, except for the Artec-generated models. In general, ear features were rarely visible in all 3D models. The photographs, as gold standard, capture more details in this region. Given that ear characteristics belong to the most stable and individualizing traits, the low visibility of the features in the 3D models requires adjustments to the lighting and scanning setup to achieve the desired quality for forensic identification purposes. The scan process of GOM and Artec can be improved in this area by manually adjusting the scanning position and generating more information of the ear features. For the 3D mug shot system it has to be evaluated, if the improvement of the light conditions or the addition of one or more cameras would solve this issue. There are several advantages of the photogrammetric documentation compared to structured light scanners. It is possible to document head and facial hair within the images and therefore in the texture map. Additionally, information about the hair colour at the time of the documentation is available. Photogrammetry enables the documentation of a person with eyes open, in difference to structured light scanners (independent of the system) where open eyes are not advisable. The flickering of the light might cause epileptic seizures and the person is incapable to keep the eyes open during the whole documentation period, and opening and closing of the eyes would cause movement and thus generate errors in the scan data [7]. The 3D documentation process, independent of the system used, requires the cooperation of the person to be documented. The scanning with a structured light scanner, even with expert operators, takes about five minutes, or might take up to ten minutes depending on the system, assuming a smooth process without interruptions. This is a long period of time during which the subject has to sit still. Furthermore, advanced training and experience is required to align misaligned scans and to analyse the scan data during the scanning procedure to prevent holes in the surface. In comparison, the 3D mug shot system operates within a fraction of a second preventing any disturbance and the fixed camera setup allows even untrained operators to perform a documentation procedure. If necessary, several shots can be made directly one after another and the immediately available photos allow the operator to check for errors right after the acquisition. Independently from the system used, glossy skin, three-day stubble and unruly beards have negative influence on the 3D model quality. Glossy skin may be reduced with makeup towels. Head hair should generally be moved away from the face region, for example with a headband, most importantly to be able to capture detailed surface information of the ears. Moreover, in any of the systems, it is not possible to document glasses, which should be removed before the documentation procedure. The combination of 3D mug shots and standard 2D mug shots requires the correct orientation of the head of the person to be documented. The exact positioning is also important with the used focal length of 135 mm. The focal length generates format filling photos, therefore the space around the head is small. With the help of an instructions manual and the cross-line laser, a standardized procedure can be guaranteed. The settings of the cameras for the photogrammetric documentation (ISO 500, f-stop f/9 and shutter speed 1/60) are caused by the dark lighting conditions. As the setup is still a prototype, there are no covers over the frame of the cameras. There is also no white panel behind the chair. Both, the cover and the rear panel, will ensure that the light will become softer and better illuminate the subject. This will optimize the lighting conditions and camera
settings can be improved accordingly, resulting in a further improvement of the 3D surface structure. 5. Conclusion In conclusion, the 3D mug shot system is a fast and efficient tool to generate 3D models of the human head and face. The major advantage of the system is that the generated facial 3D models can be adjusted in head orientation to the perspective captured on the comparative surveillance footage, thus improving the conditions for image comparisons between the mug shot and the person of interest. With the current limitations regarding the visibility of features on the PhotoScan models, the 3D mug shot system may be used as a complementary tool along with the traditional 2D mug shots for the purpose of visual forensic identification. CRediT authorship contribution statement Anja Leipner: Conceptualization, Data curation, Investigation, Methodology, Project administration, Visualization, Writing original draft, Writing - review & editing. Zuzana Obertová: Formal analysis, Methodology, Validation, Writing - review & editing. Martin Wermuth: Conceptualization, Software. Michael Thali: Funding acquisition, Resources, Supervision. Thomas Ottiker: Funding acquisition, Resources, Supervision. Till Sieberth: Methodology, Project administration, Validation, Writing review & editing. References [1] U. Park, A.K. Jain, Face matching and retrieval using soft biometrics, IEEE Trans. Inf. Forensics Secur. 5 (September (3)) (2010) 406–415. [2] S. Biswas, K.W. Bowyer, P.J. Flynn, A study of face recognition of identical twins by humans, 2011 IEEE International Workshop on Information Forensics and Security (2011) 1–6. [3] M. Maguire, The birth of biometric security, Anthropol. Today 25 (April (2)) (2009) 9–14. [4] A.K. Jain, B. Klare, U. Park, Face recognition: some challenges in forensics, Face and Gesture 2011 (2011) 726–733. [5] C. Hong Liu, C.A. Collin, A.M. Burton, A. Chaudhuri, Lighting direction affects recognition of untextured faces in photographic positive and negative, Vision Res. 39 (December (24)) (1999) 4003–4009. [6] U. Buck, S. Naether, K. Kreutz, M. Thali, Geometric facial comparisons in speedcheck photographs, Int. J. Leg. Med. 125 (November (6)) (2011) 785–790. [7] A. Modabber, et al., Evaluation of the accuracy of a mobile and a stationary system for three-dimensional facial scanning, J. Craniomaxillofac. Surg. 44 (October (10)) (2016) 1719–1724. [8] J.J. Secher, T.A. Darvann, E.M. Pinholt, Accuracy and reproducibility of the DAVID SLS-2 scanner in three-dimensional facial imaging, J. Craniomaxillofac. Surg. 45 (October (10)) (2017) 1662–1670. [9] D. Smeets, P. Claes, D. Vandermeulen, J.G. Clement, Objective 3D face recognition: evolution, approaches and challenges, Forensic Sci. Int. 201 (Septenber (1–3)) (2010) 125–132. [10] S.M. Weinberg, N.M. Scott, K. Neiswanger, C.A. Brandon, M.L. Marazita, Digital three-dimensional photogrammetry: evaluation of anthropometric precision and accuracy using a genex 3D camera system, Cleft Palate Craniofac. J. 41 (September (5)) (2004) 507–518. [11] J. Schneider, T. Läbe, W. Förstner, Incremental real-time bundle adjustment for multi-camera systems with points at infinity, ISPRS: Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XL-1 (August (W2)) (2013) 355–360. [12] A. Malian, A. Azizi, MEDPHOS: a new photogrammetric system for medical measurement, International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 35 (January) (2004). [13] I. Detchev, M. Mazaheri, S. Rondeel, A. Habib, Calibration of multi-camera photogrammetric systems, ISPRS: Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XL–1 (November) (2014) 101–108. [14] A. Leipner, R. Baumeister, M.J. Thali, M. Braun, E. Dobler, L.C. Ebert, Multicamera system for 3D forensic documentation, Forensic Sci. Int. 261 (April) (2016) 123–128. [15] R. Michienzi, S. Meier, L.C. Ebert, R.M. Martinez, T. Sieberth, Comparison of forensic photo-documentation to a photogrammetric solution using the multi-camera system ‘Botscan’, Forensic Sci. Int. 288 (July) (2018) 46–52. [16] Agisoft PhotoScan User Manual – Professional Edition, Version 1.4,” p. 127. [17] K. Taylor, Forensic Art and Illustration, CRC Press LLC, 2001.