Deloitte biometric test results

FEATURE the biometric system. Recording of time began with the test subject being two metres away from the data acquisition position and ended when the system reported successful completion of the transaction.

NPL/Deloitte biometric test results

Highlights of results

This section describes high-level results achieved through testing two face recognition and two iris recognition systems as well as the comparison of two new iris recognition algorithms. The findings are ordered alphabetically by manufacturer.

Currently there are no formal certification schemes for biometric system performance. Without an independent and scientifically credible evaluation, it is very difficult for prospective customers to assess the relative merits of different biometric systems. This especially hinders the adoption of newer and more innovative systems, where there is no history of previous use and which cannot be tested using extant biometric databases which only exist for established technologies. In 2000, the National Physical Laboratory (NPL) tested the performance of a variety of biometric systems for the UK Government Biometrics Working Group. The resulting report has often been cited as indicative of biometric system performance. Nevertheless, over the last few years, biometric technology has advanced significantly through innovations in biometric sensors and algorithms, as well as the development of standards for biometric data interchange and testing. To evaluate the effects of such developments and to address the described market needs, a Joint Industry Project was launched in 2005 under the Measurement for Innovation programme of the UK Department of Trade and Industry (DTI). The project was funded by DTI and industry partners that included product suppliers, integrators, universities, a consultancy and the Home Office. Testing was conducted by NPL and Deloitte UK.

Objectives of the test

The project focussed primarily on the evaluation of innovations in face and iris technologies; in particular, the use of skin texture and 3D imaging in facial recognition, as well as the use of alternative algorithms and hardware developments for iris recognition. Core objectives of the project were: • to provide an independent and scientific evaluation of innovative biometric technologies; • to apply the standards for biometric systems performance testing currently being developed and provide feedback to enhance these standards; and • to make progress towards establishing schemes for performance certification of biometric systems. Biometric Technology Today • November/December 2006

This summary highlights key improvements in performance, demonstrates the importance of image quality, and discusses implications for optimising performance for different types of applications.

Methodology

The test scenario was one of positive verification with co-operative nonhabituated users. The tests were conducted with 210 volunteers that were provided by NPL and Deloitte, over a five-month period. The average time separation between enrolment and verification was three months. Tests were conducted on two sites: NPL and Deloitte premises, with laboratory and office conditions, respectively. To avoid the potential for bias, the order of use of the systems was randomised for each visit. Acquired images were saved in standard format. The offline data analysis resulted in >600 genuine attempts and >120,000 impostor attempts. Emerging international standards for biometric system evaluation (meanwhile partly finalised as ISO19795) were applied wherever possible during this project. Complete transaction timings have been recorded, including time spent for the average test subject to get into position in front of Performance indicator Enrolment transaction time Recognition transaction time Failure to enrol False non-match rate

A4Vision: Vision Access The Vision Access system performs 3D facial image capture and recognition, based on the principle of structured lighting. In essence, the system projects a near-infrared light pattern of known structure at the subject’s face. Then a video camera captures the reflected light patterns distorted by the subject’s facial geometry. Based on the structure of the initial light patterns, it is possible to reconstruct the surface that caused the distortion in the captured reflections. The device can create a 3D mask of the face from a variety of angles and as the method is based on infra-red light it can operate in the dark. Test subjects were required to remove their spectacles during all transactions. As indicated in Table 1, for most users the system was remarkably fast and accurate in both authentication and real-time identification modes. Vision Access was able to distinguish between identical twins in identification mode, even though the twins were able to authenticate as each other. However, enrolment issues were encountered with a few people with beards and with a test subject who was not tall enough (body height: 1.45 metres) for the lowest setting of the camera stand. The latest version of the A4Vision system has been modified to deal with the issues noted. Identix: G6 The unique feature of the G6 system is that it combines traditional 2D face recognition with skin texture analysis to deliver higher accuracy rates. The skin analysis element looks for uniqueness in texture and randomly formed

Measurement 48 seconds 07 seconds 5 in 210 (2.4%) ~3% at a false match rate of 0.0001

Table 1: Key performance measurements for Vision Access by A4Vision

7

FEATURE features on the skin surface. This requires high resolution images; during our tests, we used a Canon PowerShot G6 camera (7MP) with an external flash (Canon Macro Twin Light, MT-24EX). Test subjects were not required to remove spectacles for enrolment or verification. As Table 2 indicates, the system delivered impressive performance but at the cost of time. The system was also capable of distinguishing between identical twins. In addition to requiring an operator for both enrolment and verification, the Identix software has several levels of in-built image quality controls. This ensures consistent, quality input, which is essential for image processing. Indeed, the system requested the image to be re-taken in 11.7% of all transactions (157 in 1346), due to poor focus or other image error, sometimes not visible to the human eye. Whilst these controls enhance input quality and hence overall recognition performance, having to re-take images requires even more time than the average transaction timings displayed in Table 2. Provided the G6 system has high quality images to work with, the skin texture analysis it uses is very powerful. Figure 1 demonstrates that the G6 recognition algorithms are quite robust even when missing key facial geometry factors. However, the time-resistance of skin texture is not yet clear. LG: IrisAccess iCam4100 The iCam4000 series is a two-eye iris camera with a much improved user interface when compared to earlier models. Image acquisition is under system control and occurs automatically once the user is positioned correctly; the camera gives both visual and verbal feedback to the user to ease the process of positioning. Whilst the iCam4000 is an iris-only device, the iCam4100 also features a keypad and a smart card reader to accommodate a wide array of application scenarios. Test subjects were asked to remove spectacles for enrolment but not for recognition attempts. Enrolment A

Performance indicator Enrolment transaction time Recognition transaction time Failure to enrol

Measurement 76 seconds 33 seconds 0

False non-match rate and false match rate (For images captured in accordance with protocol, i.e. subjects in focus, facing camera with neutral expression.)

0

Table 2: Key performance measurements for the Identix G6 system

Performance indicator Enrolment transaction time Recognition transaction time Failure to enrol False non-match rate

Measurement 13 seconds 07 seconds 0 (Note: One person could not enrol both eyes simultaneously) ~1.4% at a false match rate of 0.001%

Table 3: Key performance measurements for LG IrisAccess – iCam 4000

Performance indicator Enrolment transaction time Recognition transaction time. (Recognition transaction time includes an exhaustive search of the user database as the system runs in identification mode by default.) Failure to enrol

Measurement 73 seconds

False non-match rate

~3% at a false match rate of 0.001%

16 seconds 1 in 210

Table 4: Key performance measurements for SecuriMetrics – PIER 2.3

As expected, the system proved to be both very quick and accurate (see Table 3). There were no issues noted during enrolment, except for one test subject with a divergent squint whose eyes could not be enrolled simultaneously. The fully automated image acquisition function worked well even if occasionally the system failed to capture a good image of one or both irises; such cases included subjects moving or blinking when images were taken. As the recognition function was carried out on both eyes simultaneously (not on either eye), these cases resulted in false non-matches.

Verification A1 & A2

SecuriMetrics: PIER 2.3 PIER stands for Portable Iris Enrolment and Recognition, describing the main feature of the system: full mobility. This handheld device has been developed for and is mainly used by US military forces to control access in areas of conflict such as Iraq and Afghanistan. The device can communicate with a central database using wireless networks, be connected to a computer or operate as a stand-alone device, storing enrolment details for up to 200,000 individuals (both eyes) locally. Image acquisition occurs under operator control and the software also has in-built Enrolment B

Verification B1

Figure 1: Skin texture analysis makes it possible to recognise subjects based on input images that usually would have been rejected by a traditional 2D system

8

Biometric Technology Today • November/December 2006

FEATURE image quality controls to ensure consistent, quality input. Test subjects were asked to remove their spectacles for enrolment but not for recognition. Operating the device requires skill, especially if the subject wears glasses as light reflections must not obstruct the iris. Once the operator is trained however, image acquisition is relatively easy and recognition is quick and reliable. When examining the results displayed in Table 4, it must be considered that the system runs in identification mode by default. This means that an exhaustive search of the entire database is carried out for each recognition attempt. In fact, the system searches the database every time a new enrolment is made to make sure that an iris can only be registered once, i.e. to screen for multiple identities.

SmartSensors versus Cambridge University iris algorithms

The performance of two new iris algorithms was also tested using the images captured by both the iCam4100 and PIER 2.3 devices. One of the algorithms tested was supplied by SmartSensors, based on the work of Don Monroe at the University of Bath. The other set of algorithms tested were the latest iris recognition software by John Daugman from Cambridge University, that were initially developed for the currently running Iris Challenge Evaluation (ICE) organised by NIST. In the following figures, these new algorithms are referred to as ‘Bath’ and ‘Cambridge’, respectively. The objective was to compare performance and to benchmark these new developments against the established iris algorithm ‘Iris2pi’, also based on algorithms by John Daugman from Cambridge University, which is used by default by both iris devices tested. During the evaluations, both multialgorithm fusion and multi-instance fusion have been considered. When using both left and right iris images, in the same manner as the LGIrisAccess iCam4100 application, the new algorithms performed at a level comparable with the established Iris2pi algorithm, as shown in Figure 2. Figure 3 shows the distribution of comparison scores for the SmartSensors and Cambridge algorithms. When investigating the false non-matches recorded, it was noted that the error cases seem to be uncorrelated so that a fusion of the two algorithms could potentially lead to better overall performance. Biometric Technology Today • November/December 2006

Figure 2: Overall comparison of the performance of Bath, Cambridge and Iris2pi algorithms for images acquired by LG IrisAccess iCam4100 and SecuriMetrics PIER 2.3 devices

Conclusions

Biometric technologies have developed significantly since the first NPL trials, both in terms of hardware and software. However, most products on the market have been developed specifically for a defined application scenario so that direct comparisons between different products are not meaningful unless all system requirements are known. It is clear that face recognition has experienced strong technical advances in the last couple of years. In these recent trials, the Identix G6 system displayed perfect matching performance, but at the cost of transaction time. The tested 3D face recognition technology by A4 also performed very well both in terms of speed and accuracy, outperforming all except iris systems analysed in 2000/2001. However,

further testing is required to assess the effects of “template ageing” over longer periods (i.e. years rather than months) for these innovative approaches such as skin texture and 3D imaging. Iris recognition has also seen significant improvements since the last NPL trial. However, iris recognition has been a strong technical performer ever since its invention; hence the greatest improvements are mainly related to user friendliness and breadth of application, as opposed to performance. The LG system tested is quick, intuitivelydesigned and hence easy to use, whilst the SecuriMetrics device impresses with superb image quality and versatility. Furthermore, new iris algorithms are now appearing on the market to compete with the original encoding and matching methods, for which patent rights are expiring. Our results show

Figure 3: Score distributions in comparison between Bath (SmartSensors) and latest Cambridge algorithms

9

SURVEY that a multi-algorithm fusion could enhance overall performance in certain cases. Finally, the quality of input data has emerged as a key concern for all biometric devices in the coming years. Almost all errors recorded in these trials could be attributed to poor quality input data. Therefore, appropriate operator skills and About the authors Bori Toth is responsible for Deloitte’s Biometric Research and Advisory service offering. Deloitte provides fully vendorindependent advice on biometric strategy development, systems selection and design, customised training and workshops as well as biometric security auditing to organisations within both the public and private sectors. Alongside coordinating client service delivery, Bori also leads Deloitte’s involvement in biometric testing and standardisation. Deloitte UK is an active member of the European Biometrics Forum and the CEN/ISSS Biometrics Focus Group.

quality control are indeed as important as the characteristics of the capture device and the processing algorithms. Different systems take different approaches to ensure good quality input: some devices take several images whilst others focus on taking one high quality image. Either way, our findings confirm that input quality is essential Toth can be contacted at [email protected] or +44 207 303 2289. For more information, please visit www.deloitte.co.uk/biometrics. Dr Tony Mansfield’s biometric expertise is focussed on performance and security. His involvement in biometrics commenced in 1996, when he was Technical Manager of the EC’s BioTest project, seeking to establish and promote objective methodologies for assessing the performance and security of biometric systems. This work has continued through his involvement in the UK Government Biometrics Working Group, and more recently the work towards international standards.

Government initiatives – Part 2 This is the second part of our focus on Government projects. It gives a brief overview of five of the highest profile projects today, including the UK’s national ID card project which is soon to move into its procurement phase and where the project’s infrastructure design has recently been changed to a less centralised design. As described in last month’s focus on government projects, these dominate the biometric landscape at present and look set to do so in the short to medium term. There are obviously dozens of government projects, such as the USA’s registered traveller programme, ePassports issuance and so on. However, this survey takes a look at five of the world’s main projects either up and running with a biometric element at 10

their core, or one’s that are on the drawing board but will be significant opportunities for the industry. Seafarer ID One of the issues considered crucial for improving maritime security is ensuring that seafarers have documents enabling their “positive verifiable identification”. The seafarer ID card, specified by the

to ensuring good performance, proving that guidelines for input data, such as the passport photo guidelines issued by the UK Identity and Passport Service, are indeed necessary. In fact, there may even be merit in storing images at a higher resolution to enable the future use of technologies such as skin texture. He has run and assisted with many evaluations of biometric system performance for both industry and government. These range from scenario evaluations run in-house at the National Physical Laboratory to overarching performance reviews of systems such as Privium at Amsterdam Schiphol airport and SmartGate at Sydney airport. He is Principal UK expert for the ISO standards work on performance testing of biometrics and is editor of the ISO/IEC 19795-1 standard on Biometric Performance Testing and Reporting – Part 1: Principles and Framework. He also contributes to the standards on technical interfaces, data interchange formats, application profiles and biometric security evaluation. He can be contacted at [email protected] or Tel: +44 208 943 7029. Fore more information, please visit www.npl.co.uk.

International Labor Organization (ILO), set sail in 2005, claiming to be one of the first global biometric identification systems. An initial trio of approved suppliers to provide technology for the card was announced and a number of testing rounds. There are now eight countries that have registered as being ratified with ILO, but Btt understands that there are others that are already pursuing the issuance of Seafarer Identity Documents (SIDs) without being formally ratified, such as Russia, Indonesia, Philippines and so on. From the initial list of three approved suppliers there are now nine approved products from nine vendors. More may be added in due course – as long as they can meet the requirements of Convention No. 185 and the ILO SID-0002 standard. New biometric tests are slated to begin in the Spring and will focus on mobile, rather than desktop, barcode and fingerprint reading devices, which would be useful products for the port/ship environment. One of the first countries to issue SIDs is Nigeria, although Pakistan is believed to also be issuing the documents. In Nigeria, Cogent Systems’ fingerprint technology was integrated by 3M Corporation for use in its programme. Biometric Technology Today • November/December 2006