- Email: [email protected]
The reference ballistic imaging database revisited
Forensic Science International 248 (2015) 82–87
Contents lists available at ScienceDirect
Forensic Science International journal homepage: www.elsevier.com/locate/forsciint
The reference ballistic imaging database revisited Jan De Ceuster *, Sylvain Dujardin Nationaal Instituut voor Criminalistiek en Criminologie/Institut Nationale de Criminalistique et de Criminologie (NICC/INCC), Vilvoordsesteenweg 100, B-1120 Brussels, Belgium
A R T I C L E I N F O
A B S T R A C T
Article history: Received 3 June 2014 Received in revised form 25 November 2014 Accepted 27 November 2014 Available online 9 December 2014
A reference ballistic image database (RBID) contains images of cartridge cases fired in firearms that are in circulation: a ballistic fingerprint database. The performance of an RBID was investigated a decade ago by De Kinder et al. using IBIS1 HeritageTM technology. The results of that study were published in this journal, issue 214. Since then, technologies have evolved quite significantly and novel apparatus have become available on the market. The current research article investigates the efficiency of another automated ballistic imaging system, Evofinder1 using the same database as used by De Kinder et al. The results demonstrate a significant increase in correlation efficiency: 38% of all matches were on first position of the Evofinder correlation list in comparison to IBIS1 HeritageTM where only 19% were on the first position. Average correlation times are comparable to the IBIS1 HeritageTM system. While Evofinder1 demonstrates specific improvement for mutually correlating different ammunition brands, ammunition dependence of the markings is still strongly influencing the correlation result because the markings may vary considerably. As a consequence a great deal of potential hits (36%) was still far down in the correlation lists (positions 31 and lower). The large database was used to examine the probability of finding a match as a function of correlation list verification. As an example, the RBID study on Evofinder1 demonstrates that to find at least 90% of all potential matches, at least 43% of the items in the database need to be compared on screen and this for breech face markings and firing pin impression separately. These results, although a clear improvement to the original RBID study, indicate that the implementation of such a database should still not be considered nowadays. ß 2014 Elsevier Ireland Ltd. All rights reserved.
Keywords: Ballistic database Evofinder Automated comparison Correlation Firearms identification Quality control
1. Introduction The reference ballistic imaging database is a concept that received interest in the beginning of last decade. The general idea was to build a database of spent ammunition, especially cartridge cases, from newly manufactured firearms prior to them being sold on the market. In the event that such a firearm is used in crime, and unrecovered by the police, it could be identified from the cartridge cases that are found on the scene of the crime. This could potentially lead to the owner of the firearm or at least give a strong investigative lead for law enforcement agencies. The feasibility of such a database was examined by Jan De Kinder et al. in 2003 with the tools available at the time, being an Integraded Ballistics Identification System (IBIS1) HeritageTM imaging system [1]. Their study has shown that: ‘‘a reference ballistics image database of
* Corresponding author. Tel.: +32 2 2400500; fax: +32 2 2434625. E-mail address: [email protected] (J. De Ceuster). http://dx.doi.org/10.1016/j.forsciint.2014.11.025 0379-0738/ß 2014 Elsevier Ireland Ltd. All rights reserved.
new guns is currently fraught with too many difficulties to be an effective and efficient law enforcement tool.’’ In the years following the aforementioned study the automated ballistic imaging tools have seen drastic improvements of their capabilities and performances. The market became more wide spread as different tools become available on the market, each with their own approach of ballistic imaging and correlation. This has certainly triggered the competition between companies to develop a better product. This is obviously beneficial for the forensic laboratories that keep open case files to which more and more items will be added as time goes by. Still, automated pre-screening systems have inherent parameters that will always have their influence on the result they come up with. These parameters were extensively studied by Nennstiel and Rahm for the IBIS Heritage correlator [2,3]. The state-of-the-art technology offers improvements in image resolution to sub 5 mm, capturing three-dimensional information (topography) of the markings on the object, the semi-automatic selection of the relevant marking areas (specifically for bullets), enhanced correlation efficiencies and improved manipulation
J. De Ceuster, S. Dujardin / Forensic Science International 248 (2015) 82–87
possibilities during the on screen comparison process. Gerules et al. [4] have recently written a survey article of current image processing techniques and statistics dealing with the uniqueness of markings on ballistic evidence. But have the novel technologies evolved up to the point where the idea of a reference ballistic imaging database can be reconsidered? Our institute acquired an Evofinder1 system, developed by ScannBI Technology. To understand the system’s parameters and performance, it was considered that it should be evaluated with a large database. The cartridge cases from the original RBID study seemed to be an ideal tool as they consists of firearms of the same class and different ammunition brands. As such, this database can be regarded a worst case scenario tool that will give good insight in the limitations of a system and of the resolvability in on screen comparisons of poorly reproduced markings (specifically between different ammunition brands). The Sig Sauer cartridge cases of the RBID study were kindly made available to us by Mr. Fred Tulleners (current affiliation: California Criminalistics Institute – former affiliation: University of California, Davis California). Hence we were able to re-assess the concept of the RBID and at the same time develop a tool to check the system’s performance and build a quality control database for future use. 2. Materials and methods The goal of our study is two-fold. Firstly we re-evaluate the feasibility of an RBID with novel technology through a direct comparison with the correlation results that were produced at the time of the original RBID study. Secondly we learn about automated system performance and thus practical application in case work by exploiting the results of this large database study. 2.1. Preliminary remarks A big part of the discussion in this article deals with the comparison of results obtained with Evofinder1 and IBIS1 HeritageTM. The reader has to be aware that both systems are very different. They are developed in a different era, make use of other computer infrastructure and correlate different input. It could be objected that a comparison does not make sense for these reasons. It is the goal of the article, however, to evaluate the concept of the RBID and not to compare the systems that were used. The systems under examination had the following technical parameters. IBIS1: Windows NT 4.00, Intel1 Pentium III 750 MHz, i440 Bx chipset with 256 MB RAM, data acquisition station and signature analysis station unit version 3.4.167 (system acquired in 2002) – Evofinder1: Windows7 32bit, Intel1 CoreTM i5 2.80 GHz processor with 4 GB RAM, software version 5.4 (system acquired in 2010). Both systems have got improvements/upgrades in the meantime. Therefore, the results do not reflect a nowadays situation yet rather an evolution in time. 2.2. RBID preparations and methodology The reader is referred to the article by De Kinder et al. to learn about the preparation of the samples as well as the discussion on the results. Wherever useful, the results of the 2003 RBID study will be quoted. The raw data of that study was available to us for exploitation. The cartridge cases were scanned with Evofinder1 by trained staff. Particular care was taken in the orientation of the cartridge cases in the scanner. It was chosen to orient the extractor mark in the 12 o’clock position. The total time per cartridge case that was
83
needed to complete the whole procedure was about 5 min. This includes fixing the cartridge case in the holder and inserting it in the scanner, running a scan, fine-tuning the firing pin position and size and saving the data with the necessary administrative data. This time is equivalent to what was necessary with the IBIS1 HeritageTM system. The ejector mark was not taken into consideration for this study nor was it for the original study. From the original 600 reference cartridge cases, we had 592 at our disposal. The differences in size of the databases are negligible. The questioned items consisted of 189 specimens (three were missing from the original test). As we did not have the whole set, we will have to verify whether or not a comparison with the results from IBIS1 HeritageTM is justified. After the database was completely set up, correlations were performed between the questioned items and the reference items. Just like IBIS1 HeritageTM, Evofinder1 organizes the items in order of best matching on breechface markings and firing pin impression separately. For each trace a correlation factor is given that ranges between zero and one. Finding the match in the database was not done blindly by the examiners; the study was primarily focused on system performance not on examiner’s interpretation. The known match was searched in both of the correlation lists for breech face markings and firing pin impression. The position (ranking) in the correlation list, the correlation coefficient as well as the correlation time were denoted. The questioned items consisted of 189 cartridges of six different brands of which one brand (Remington–Peters (R–P)) was the same as the reference cartridges. We followed the analytical approach by De Kinder to enable comparisons with his results. If the ranking position of an item in the correlation list was between 1 and 30, the true ranking position was denoted as such. If it was further down the list it was attributed the arbitrary position ‘‘31’’. The marking that returned the highest ranking position was chosen: this was either breech face (BF) or firing pin impression (FPI). Histograms are made of the occurrence of matches as function of the ranking position in the correlation list. A direct comparison was performed with the data from the IBIS1 HeritageTM system. It was examined how the size of the database has an influence on the correlation time as well as on the position of a matching pair in the correlation list. As the database was built over the course of several days (weeks), it was easy to increase the database size by filtering on this date and varying the date range stepwise. 2.3. System performance preparations and methodology Running correlations of multiple questioned items against a large database will result in a good indicator of the overall system performance. The data was organized in the following way. A table was constructed with columns: questioned item no., matching reference no., ranking position BF, ranking position FPI. The best ranking position was determined: this was either BF or FPI. Contrary to the previously detailed methodology, the original best ranking position was kept (positions > 30 were not denoted as ‘‘31’’). Next, a histogram was created from the occurrence (number of matches) as a function of best ranking position. This could be transformed to a probability curve using cumulative percentage. The performance of the system on the data of the RBID yields interesting information that can be taken a step further to actual case work (no RBID). Under the circumstances of normal case work, the database is filled with images of open cases (open case file). With every comparison, it is unknown if a match is present in the database. Databases (correlation lists) can be screened completely if the examiner has the time or could be screened up until a certain
J. De Ceuster, S. Dujardin / Forensic Science International 248 (2015) 82–87
84
point, defined by in-house protocols. A test of the system with a large database, such as the RBID, can aid in fine-tuning these protocols (e.g. understanding detection probability limits). The idea of determining such a break condition is reviewed.
about an automated system’s performance and the confidence that can be attributed to a routine database search in open case files. The fact that its behaviour is linear is important and can be exploited to extrapolate/interpolate to larger/smaller databases.
3. Results and discussion
3.2. RBID performance The reader is also referred to the preliminary remarks in Section
3.1. Linearity check: ranking position as a function of database size 2. The more items are added to a database, the more the noise level grows. It stands to reason that an evident match will remain to be classified quite high in the correlation list. However, if the database consists of many items bearing the same class characteristics and/or if the questioned item has only few individual markings it will become difficult for a match to stand out in the correlation list, especially with different ammunition brands that result in partially different markings by the same firearm. We have observed a clear linear behaviour between ranking position of a matching pair and size of the database. The slope of these lines is dependent on the quality and quantity of the individual markings. Therefore, the graph in Fig. 1 (for the pair EV141 (questioned) and SS315 (reference)) is an example just for the matter of demonstrating the linearity but the slope is not to be generalized. The database under examination consisted of firearms of the same class, so quite homogeneous. When the database grows, every item that is added has therefore approximately an equivalent chance of positioning itself either above or below the matching pair in the correlation list. If the database would have been more heterogeneous, these chances would be variable and hence a linear behaviour is maybe not to be expected. The linear behaviour is also applicable for the firing pin impression. For the RBID database, the correlation efficiency of Evofinder was somewhat better for the FPI as opposed to the BF but this is not to be generalized as it depends strongly on the firearms markings that are present on the samples. The linearity can be used to check if a comparison between the results of this study with a somewhat smaller RBID (590 items) against the original study consisting of 600 items can be done. Our approach was to classify matching pairs up until correlation list position 30 according to their true position. Anything below will get position ‘‘31’’ by default. For a match on position 30 for a database of 590 items the slope of the line is equal to 30/590 = 0.05. In a database of 600 items, the same pair will be on theoretical position 30.6 meaning just a negligible decrease (less than 1 position) and thus a justification for a direct comparison. As will be shown further in this article, the ranking position as a function of the database size can yield interesting information
BF Ranking position
350
The total sample size consists of 189 cartridge cases all of which are correlated against the database of 590 reference items. The 189 correlation results are subdivided in six groups by ammunition brand. These smaller groups are statistically less significant and variability could occur. As such, the results are only partially representative but obviously the same is true for the results obtained with the original RBID study. 3.2.1. Correlations with the same brand of ammunition Sometimes different brands of ammunition are marked very differently during the firing process so ammunition brand influences the correlation result. In principle it is the cartridge case build material and hardness as well as manufacturing tolerances that lie at the basis of these differences [5,6]. It is logical that an automated system cannot make interpretations about these differences. Automated systems are as objective as can be and the correlation result takes into consideration both similarities and differences. As such, the correlation result is always correct. But differences in markings can be so important that a match will be low in the correlation list, just because of that. It is only at the stage where the examiner is proceeding with the comparisons, that differences in markings can sometimes be interpreted as being irrelevant when they originate from different ammunition brands. At the point where the examiner goes in depth on the (on screen) comparisons he could make interpretations about the observed similarities and differences, and focus on the markings that he estimates to be more reproducible, based on his training and experience. But this is where the subjectivity sets in again. The correlation between two cartridge cases of the same brand is the closest scenario to review the reproducibility of the markings from one shot to another. It gives a first indicator of the system’s performance. From Fig. 2 one can learn two interesting things. One is the number of matches that were found on the first position in the correlation list and the other is the number of matches that are positioned lower than 30. Evofinder1 positions 18 out of 31 matches on the first position. In that respect it compares very well to the IBIS1 HeritageTM. Evofinder1 positions 3 out of 31 matches at position 31 or lower. IBIS1 HeritageTM did worse with 8 out of 31. Generally, the graph demonstrates an improvement of the Evofinder1 system as opposed to the IBIS1 system: 90% vs. 74% within the first 30 positions of the correlation list for cartridge cases of the same brand.
300 250 200 150 100 50 0 0
50
100
150
200
250
300
350
400
450
500
550
600
650
database size Fig. 1. Graph representing the Evofinder1 breechface ranking position of the pair EV141-SS315 as a function of size of the RBID. The graph cannot be generalized as the ranking position strongly depends on the presence and reproducibility of the markings. However, the behaviour is linear, which is general for all items.
3.2.2. Correlations with different brands of ammunition It is well known that ammunition brand will have a severe influence on the correlation efficiencies. This was one of the main aspects that influenced the outcome of the RBID study in 2003. The following graphs (Figs. 2–7) show the histograms of the best ranking order obtained for Winchester, Speer, Wolf, Federal and CCI cartridge cases in correlation against R–P cartridge cases. Again each graph compares the occurrence of a match as a function of ranking position for the Evofinder1 vs. the IBIS1 HeritageTM system. Again it is specifically the occurrence of matches on the first position of the correlation list as well as the number of matches that situate themselves at position 31 or lower that reveal the increase in correlation efficiency of Evofinder1 as opposed to IBIS1
J. De Ceuster, S. Dujardin / Forensic Science International 248 (2015) 82–87
85
Remington-Peters Occurrence (number)
20 IBIS 15
Evofinder
10
5
0 1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Best Ranking Order Fig. 2. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 590-gun RBID. Both the reference cartridge cases and the questioned items were of the same brand (Remington–Peters). Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
HeritageTM. Fig. 8 shows the cumulative percentage of matches as a function of best ranking position for the whole study (189 comparisons). It shows that for Evofinder1 in comparison with IBIS1 HeritageTM there is a doubling of efficiency for discerning a match and positioning it on the first position of the correlation list (best ranking position). 42% of all matches on the first two positions in the correlation list and 64% on the first 30 positions is not a bad result. Still, after that, the efficiency will diminish as one goes through the correlation list. As a result, and this will be shown further in this paper, one would have to go through a lot more on screen comparisons to get the correct item.
Winchester
20 Occurrence (number)
3.2.3. Correlation time as a function of DB size Evofinder1 has the option of extending the correlation time for more in depth calculation. This can be chosen by the operator. For the results presented in this section, correlation times were always maximized. The more entries to compare with, the longer the correlation time. By performing the 189 comparisons it was observed that not every item took the same time to correlate. The behaviour, however, is clearly linear (see Fig. 9) with slope factor between 0.19 (for the fastest) and 0.33 for the slowest with 0.25 on the average.
IBIS 15
Evofinder
10
5
0 1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Best Ranking Order
Fig. 3. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 591-gun RBID. The reference cartridge cases were of the Remington–Peters brand while the questioned items were of the Winchester brand. Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
Speer
20 Occurrence (number)
Fig. 5. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 590-gun RBID. The reference cartridge cases were of the Remington–Peters brand while the questioned items were of the Wolf brand. Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
IBIS 15
Evofinder
10
5
0 1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Best Ranking Order
Fig. 4. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 591-gun RBID. The reference cartridge cases were of the Remington–Peters brand while the questioned items were of the Speer brand. Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
Fig. 6. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 591-gun RBID. The reference cartridge cases were of the Remington–Peters brand while the questioned items were of the Federal brand. Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
J. De Ceuster, S. Dujardin / Forensic Science International 248 (2015) 82–87
Fig. 7. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 591-gun RBID. The reference cartridge cases were of the Remington–Peters brand while the questioned items were of the CCI brand. Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
Cumulative % of hits vs best ranking position 100% 90% 80%
% of hits
70% 60% 50% 40% 30% 20% 10% 0% 1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
IBIS
Evofinder
Fig. 8. Graph representing the cumulative percentage of matches that are in the correlation list as a function of best ranking position (up until position 30) for either firing pin or breech face as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 592-gun RBID.
From the original RBID results with IBIS1 the authors obtained a linear behaviour as well but there was an offset of about 25 s that was not explained by the authors. Using the average correlation time and extrapolating, it would take Evofinder1 about 40.9 min to compare 1 item against 10,000 items in a database. This time is comparable to with what was found for IBIS1 (45.9 min). From that point of view there has not been massive increase in efficiency. However, all depends on the
speed of the computer. And logically, the data that needs to be compared by Evofinder1 is likely a lot more complex. Where IBIS1 based itself on two-dimensional images, Evofinder takes into consideration topographical information. Newer technologies may decrease the calculation time significantly. For instance, the possibility exists to spread calculations on different computers in a network or on graphical card CPUs. In principle, running a correlation does not require human interference. Correlation duration is therefore not a real issue except when an answer is really urgent. When multiple correlations against a large database need to be performed, they can be run overnight and the results be evaluated the next day. 3.3. System performance Using the results of the test, the performance of the automated system could be thoroughly evaluated. A probability curve [7] for finding a match as a function of browsing down the correlation list was constructed to aid in this process (see Fig. 10). The probability curve rises quickly after which it gradually flattens out. The explanation is simple. As an automated system can be regarded as a sorting machine, the cartridge cases that resemble best (potential matches) will be positioned high in the ranking. As one browses through the correlation list the noise level quickly increases. After a certain point where the probability curve nearly flattens, the probability for lesser evident matches to position them on position x or a lot further down the ranking become quite comparable especially with low reproducible markings such as e.g. on different brands of ammunition. Following Rahm [7], the ideal examiner is someone who manually compares with each individual item in the open case file and eventually finds the match if there is one. The probability curve of an ideal examiner is therefore essentially a straight line. For each cartridge case that is compared there is an equal chance that it is or is not a match. The ideal examiner’s efficiency will never fail as opposed to an automated system that is very efficient for evident matches yet less efficient for non-evident matches. Rahm suggests using a break condition for hit list verifications in practical case work. As soon as the automated system’s efficiency drops below that of the ideal examiner, the chance to find the correct item
Probability of finding a match vs ranking position for the RBID database 70
250
100% 90%
60
80% 50 Frequency
correlation time (s)
200
150
100
70% y = 0.1369ln(x) + 0.1475 60%
40
50% 30
40% 30%
20
20%
50
0 0
10
10% 0%
0 100
200 fastest
300 400 500 database size (# items) slowest average IBIS RBID study
600
700
Fig. 9. Graph representing the time needed for the Evofinder1 to correlate a questioned item against a database for which the size is increasingly growing (xaxis). The study demonstrated that there is no fixed time for all objects but rather there was slight spread. The figure therefore shows the graphs for the fastest, the slowest and the average time need. As a reference, the data from the study performed by IBIS1 HeritageTM is given (reproduced from the study by De Kinder).
Accumulative percentage (probability)
86
1
51
101
151
201 251 301 351 401 451 501 551 Best Ranking Position Frequency Cumulative % Log. (Cumulative %)
Fig. 10. Graph representing the Evofinder1 probability curve for having a match in the correlation list of a 592-gun RBID as a function of best ranking position (either breech face or firing pin impression). The data could be best fitted with a logarithmic function. The straight line represents the ideal examiner’s efficiency. The left y-axis represents the frequency (count) of matches as function of the best ranking position; the right y-axis represents the accumulative percentage
J. De Ceuster, S. Dujardin / Forensic Science International 248 (2015) 82–87
becomes at that point lower than the chance that an examiner would have without sorting help of a machine, thus performing arbitrary selection. This occurs when the slope of the probability curve of the system becomes equal to or less than the slope of the ideal examiner’s efficiency. Studies by Rahm, based on real case work with the Evofinder database containing several hundreds of cartridge cases have resulted in low break conditions for correlation list verification (e.g. the 11th position for 9 mm Luger, see Table 3 in [7]). He concludes, however, that it is overall recommendable to go for the first 20. Yet, it is clear that in such a way potential matches, even though they are less evident, will be missed. If no matches were discovered by verifying the correlation list only until the break condition, how certain can one be that there really was no match present in the database? How does the system perform when it comes to discerning a match from other non-relevant items? From a scientific point of view these are interesting questions to be asked. The constructed RBID database can help in understanding. This information can be derived from the probability curve of the system. It has to be kept in mind, however, that the test that is described in this article was tough for a number of reasons. Firstly, there are the different brands of ammunition that really lower the score. In case work one will always try to test fire ammunition from the same brands and with properties close to what is being used by the criminals. In addition, it is possible to scan several cartridge cases (different brands) fired with the same firearm in order to appreciate the inter-variability of markings that can occur. It indicates how well the other item can be found under optimal conditions. This procedure gives a clue of how far the correlation list must be verified by the operator. Secondly, the database consists of firearms of the same brand, with the same class characteristics, a situation unlikely to occur in real open case files. In other words, the detection probability limits that are attributed here to a certain break condition represent a worst case scenario situation. The data points were fitted with the function proposed by Rahm in [7] (point 3.1). The best fit was obtained with parameters a = 0.82, b = 9.81 and c = 0.00041. The R2 value for this fit was 0.95. Alternatively, the data was fitted with a simple logarithmic function y = d + e ln(x). Even though this is scientifically not completely the correct function to be used, it fitted even better than the aforementioned function. With parameters d = 0.1369 and e = 0.1475 the R2 value was 0.98. For the database of 592 specimen, the slope of the ideal examiner’s efficiency curve is 1/592 = 0.0017. The fit with the function by Rahm has the same slope at a ranking position of 65; with the logarithmic function this is 81. Using these points as break condition means that respectively 74% (Rahm) and 75% (logarithmic) of all potential matches will be found, leaving respectively 26% and 25% potentially unfound. One could work the other way around as well. A laboratory could decide to set a confidence level of e.g. 90% for a specific correlation task. This means that many more items in the correlation list need to be verified. In the case of the RBID with 592 references, cartridge cases up to position 255 need to be reviewed. It should be reminded that this graph represents best
87
ranking position either on BF or FPI. This means that verifications need to be done on both traces independently meaning 2 255 onscreen verifications. 255 specimens in a database of 592 represent 43% of the complete database that needs to be verified. As ranking position behaves linear with database size, such a comparison work could eventually be considered for smaller databases but not for large ones as this is a lot of work and not efficient for any laboratory that is trying to cope with backlogs. For time reasons these laboratories sometimes set a limit to the amount of verifications they perform e.g. by looking only at the first 30 items. Obviously, this will downsize the confidence level, independent of database size. This could be done under the perspective that in any way not all firearm crimes will be solved with ballistic image databases. In the RBID database that was constructed for the purpose of the study, only looking at the first 30 items will amount to 64% certainty to find the match. The database could be used to verify system’s performance during case work. Also, when the system would receive a massive update on the correlation part, this could be checked.
4. Conclusions The RBID study, initially performed by De Kinder et al., was repeated using novel technology. The Evofinder1 system has demonstrated an important improvement in automated ballistic imaging equipment. No doubt this is also valid for other state-of-the-art equipment that is available on the market nowadays. Nevertheless the idea of a reference ballistic imaging database remains utopic for now. The main aspects limiting this idea are two-fold. One is the nature of the markings, which reproducibility is predominantly influenced by the cartridge case and primer hardness. The other is that for large databases one has to verify a great deal of the potential matches in the correlation list in order to have a certain confidence level. The system’s performance on a large database such as the RBID is a good indicator for its behaviour during daily work with the open case file.
References [1] J. De Kinder, F. Tulleners, H. Thiebaut, Reference ballistic imaging database performance, Forensic Sci. Int. 140 (2004) 207–215. [2] R. Nennstiel, J. Rahm, A parameter study regarding the IBISTM correlator, J. Forensic Sci. 51 (2006) 18–23. [3] R. Nennstiel, J. Rahm, An experience report regarding the performance of the IBISTM correlator, J. Forensic Sci. 51 (2006) 24–30. [4] G. Gerules, S.K. Bhatia, D.E. Jackson, A survey of image processing techniques and statistics for ballistic specimens in forensic science, Sci. Justice 53 (2013) 236–250. [5] J.J. Davis, Primer cup properties & how the affect identification, AFTE J. 42 (2010) 3–22. [6] P. De Smet, R. Hermsen, B. van Leuven, J. De Kinder, K. Hoffmann, Experimental evaluation of the impact of seating depth variations on observed marks on primers, Forensic Sci. Int. 179 (2008) 163–171. [7] J. Rahm, Evaluation of an electronic comparison system and implementation of a quantitative effectiveness criterion, Forensic Sci. Int. 214 (2012) 173–177.
Contents lists available at ScienceDirect
Forensic Science International journal homepage: www.elsevier.com/locate/forsciint
The reference ballistic imaging database revisited Jan De Ceuster *, Sylvain Dujardin Nationaal Instituut voor Criminalistiek en Criminologie/Institut Nationale de Criminalistique et de Criminologie (NICC/INCC), Vilvoordsesteenweg 100, B-1120 Brussels, Belgium
A R T I C L E I N F O
A B S T R A C T
Article history: Received 3 June 2014 Received in revised form 25 November 2014 Accepted 27 November 2014 Available online 9 December 2014
A reference ballistic image database (RBID) contains images of cartridge cases fired in firearms that are in circulation: a ballistic fingerprint database. The performance of an RBID was investigated a decade ago by De Kinder et al. using IBIS1 HeritageTM technology. The results of that study were published in this journal, issue 214. Since then, technologies have evolved quite significantly and novel apparatus have become available on the market. The current research article investigates the efficiency of another automated ballistic imaging system, Evofinder1 using the same database as used by De Kinder et al. The results demonstrate a significant increase in correlation efficiency: 38% of all matches were on first position of the Evofinder correlation list in comparison to IBIS1 HeritageTM where only 19% were on the first position. Average correlation times are comparable to the IBIS1 HeritageTM system. While Evofinder1 demonstrates specific improvement for mutually correlating different ammunition brands, ammunition dependence of the markings is still strongly influencing the correlation result because the markings may vary considerably. As a consequence a great deal of potential hits (36%) was still far down in the correlation lists (positions 31 and lower). The large database was used to examine the probability of finding a match as a function of correlation list verification. As an example, the RBID study on Evofinder1 demonstrates that to find at least 90% of all potential matches, at least 43% of the items in the database need to be compared on screen and this for breech face markings and firing pin impression separately. These results, although a clear improvement to the original RBID study, indicate that the implementation of such a database should still not be considered nowadays. ß 2014 Elsevier Ireland Ltd. All rights reserved.
Keywords: Ballistic database Evofinder Automated comparison Correlation Firearms identification Quality control
1. Introduction The reference ballistic imaging database is a concept that received interest in the beginning of last decade. The general idea was to build a database of spent ammunition, especially cartridge cases, from newly manufactured firearms prior to them being sold on the market. In the event that such a firearm is used in crime, and unrecovered by the police, it could be identified from the cartridge cases that are found on the scene of the crime. This could potentially lead to the owner of the firearm or at least give a strong investigative lead for law enforcement agencies. The feasibility of such a database was examined by Jan De Kinder et al. in 2003 with the tools available at the time, being an Integraded Ballistics Identification System (IBIS1) HeritageTM imaging system [1]. Their study has shown that: ‘‘a reference ballistics image database of
* Corresponding author. Tel.: +32 2 2400500; fax: +32 2 2434625. E-mail address: [email protected] (J. De Ceuster). http://dx.doi.org/10.1016/j.forsciint.2014.11.025 0379-0738/ß 2014 Elsevier Ireland Ltd. All rights reserved.
new guns is currently fraught with too many difficulties to be an effective and efficient law enforcement tool.’’ In the years following the aforementioned study the automated ballistic imaging tools have seen drastic improvements of their capabilities and performances. The market became more wide spread as different tools become available on the market, each with their own approach of ballistic imaging and correlation. This has certainly triggered the competition between companies to develop a better product. This is obviously beneficial for the forensic laboratories that keep open case files to which more and more items will be added as time goes by. Still, automated pre-screening systems have inherent parameters that will always have their influence on the result they come up with. These parameters were extensively studied by Nennstiel and Rahm for the IBIS Heritage correlator [2,3]. The state-of-the-art technology offers improvements in image resolution to sub 5 mm, capturing three-dimensional information (topography) of the markings on the object, the semi-automatic selection of the relevant marking areas (specifically for bullets), enhanced correlation efficiencies and improved manipulation
J. De Ceuster, S. Dujardin / Forensic Science International 248 (2015) 82–87
possibilities during the on screen comparison process. Gerules et al. [4] have recently written a survey article of current image processing techniques and statistics dealing with the uniqueness of markings on ballistic evidence. But have the novel technologies evolved up to the point where the idea of a reference ballistic imaging database can be reconsidered? Our institute acquired an Evofinder1 system, developed by ScannBI Technology. To understand the system’s parameters and performance, it was considered that it should be evaluated with a large database. The cartridge cases from the original RBID study seemed to be an ideal tool as they consists of firearms of the same class and different ammunition brands. As such, this database can be regarded a worst case scenario tool that will give good insight in the limitations of a system and of the resolvability in on screen comparisons of poorly reproduced markings (specifically between different ammunition brands). The Sig Sauer cartridge cases of the RBID study were kindly made available to us by Mr. Fred Tulleners (current affiliation: California Criminalistics Institute – former affiliation: University of California, Davis California). Hence we were able to re-assess the concept of the RBID and at the same time develop a tool to check the system’s performance and build a quality control database for future use. 2. Materials and methods The goal of our study is two-fold. Firstly we re-evaluate the feasibility of an RBID with novel technology through a direct comparison with the correlation results that were produced at the time of the original RBID study. Secondly we learn about automated system performance and thus practical application in case work by exploiting the results of this large database study. 2.1. Preliminary remarks A big part of the discussion in this article deals with the comparison of results obtained with Evofinder1 and IBIS1 HeritageTM. The reader has to be aware that both systems are very different. They are developed in a different era, make use of other computer infrastructure and correlate different input. It could be objected that a comparison does not make sense for these reasons. It is the goal of the article, however, to evaluate the concept of the RBID and not to compare the systems that were used. The systems under examination had the following technical parameters. IBIS1: Windows NT 4.00, Intel1 Pentium III 750 MHz, i440 Bx chipset with 256 MB RAM, data acquisition station and signature analysis station unit version 3.4.167 (system acquired in 2002) – Evofinder1: Windows7 32bit, Intel1 CoreTM i5 2.80 GHz processor with 4 GB RAM, software version 5.4 (system acquired in 2010). Both systems have got improvements/upgrades in the meantime. Therefore, the results do not reflect a nowadays situation yet rather an evolution in time. 2.2. RBID preparations and methodology The reader is referred to the article by De Kinder et al. to learn about the preparation of the samples as well as the discussion on the results. Wherever useful, the results of the 2003 RBID study will be quoted. The raw data of that study was available to us for exploitation. The cartridge cases were scanned with Evofinder1 by trained staff. Particular care was taken in the orientation of the cartridge cases in the scanner. It was chosen to orient the extractor mark in the 12 o’clock position. The total time per cartridge case that was
83
needed to complete the whole procedure was about 5 min. This includes fixing the cartridge case in the holder and inserting it in the scanner, running a scan, fine-tuning the firing pin position and size and saving the data with the necessary administrative data. This time is equivalent to what was necessary with the IBIS1 HeritageTM system. The ejector mark was not taken into consideration for this study nor was it for the original study. From the original 600 reference cartridge cases, we had 592 at our disposal. The differences in size of the databases are negligible. The questioned items consisted of 189 specimens (three were missing from the original test). As we did not have the whole set, we will have to verify whether or not a comparison with the results from IBIS1 HeritageTM is justified. After the database was completely set up, correlations were performed between the questioned items and the reference items. Just like IBIS1 HeritageTM, Evofinder1 organizes the items in order of best matching on breechface markings and firing pin impression separately. For each trace a correlation factor is given that ranges between zero and one. Finding the match in the database was not done blindly by the examiners; the study was primarily focused on system performance not on examiner’s interpretation. The known match was searched in both of the correlation lists for breech face markings and firing pin impression. The position (ranking) in the correlation list, the correlation coefficient as well as the correlation time were denoted. The questioned items consisted of 189 cartridges of six different brands of which one brand (Remington–Peters (R–P)) was the same as the reference cartridges. We followed the analytical approach by De Kinder to enable comparisons with his results. If the ranking position of an item in the correlation list was between 1 and 30, the true ranking position was denoted as such. If it was further down the list it was attributed the arbitrary position ‘‘31’’. The marking that returned the highest ranking position was chosen: this was either breech face (BF) or firing pin impression (FPI). Histograms are made of the occurrence of matches as function of the ranking position in the correlation list. A direct comparison was performed with the data from the IBIS1 HeritageTM system. It was examined how the size of the database has an influence on the correlation time as well as on the position of a matching pair in the correlation list. As the database was built over the course of several days (weeks), it was easy to increase the database size by filtering on this date and varying the date range stepwise. 2.3. System performance preparations and methodology Running correlations of multiple questioned items against a large database will result in a good indicator of the overall system performance. The data was organized in the following way. A table was constructed with columns: questioned item no., matching reference no., ranking position BF, ranking position FPI. The best ranking position was determined: this was either BF or FPI. Contrary to the previously detailed methodology, the original best ranking position was kept (positions > 30 were not denoted as ‘‘31’’). Next, a histogram was created from the occurrence (number of matches) as a function of best ranking position. This could be transformed to a probability curve using cumulative percentage. The performance of the system on the data of the RBID yields interesting information that can be taken a step further to actual case work (no RBID). Under the circumstances of normal case work, the database is filled with images of open cases (open case file). With every comparison, it is unknown if a match is present in the database. Databases (correlation lists) can be screened completely if the examiner has the time or could be screened up until a certain
J. De Ceuster, S. Dujardin / Forensic Science International 248 (2015) 82–87
84
point, defined by in-house protocols. A test of the system with a large database, such as the RBID, can aid in fine-tuning these protocols (e.g. understanding detection probability limits). The idea of determining such a break condition is reviewed.
about an automated system’s performance and the confidence that can be attributed to a routine database search in open case files. The fact that its behaviour is linear is important and can be exploited to extrapolate/interpolate to larger/smaller databases.
3. Results and discussion
3.2. RBID performance The reader is also referred to the preliminary remarks in Section
3.1. Linearity check: ranking position as a function of database size 2. The more items are added to a database, the more the noise level grows. It stands to reason that an evident match will remain to be classified quite high in the correlation list. However, if the database consists of many items bearing the same class characteristics and/or if the questioned item has only few individual markings it will become difficult for a match to stand out in the correlation list, especially with different ammunition brands that result in partially different markings by the same firearm. We have observed a clear linear behaviour between ranking position of a matching pair and size of the database. The slope of these lines is dependent on the quality and quantity of the individual markings. Therefore, the graph in Fig. 1 (for the pair EV141 (questioned) and SS315 (reference)) is an example just for the matter of demonstrating the linearity but the slope is not to be generalized. The database under examination consisted of firearms of the same class, so quite homogeneous. When the database grows, every item that is added has therefore approximately an equivalent chance of positioning itself either above or below the matching pair in the correlation list. If the database would have been more heterogeneous, these chances would be variable and hence a linear behaviour is maybe not to be expected. The linear behaviour is also applicable for the firing pin impression. For the RBID database, the correlation efficiency of Evofinder was somewhat better for the FPI as opposed to the BF but this is not to be generalized as it depends strongly on the firearms markings that are present on the samples. The linearity can be used to check if a comparison between the results of this study with a somewhat smaller RBID (590 items) against the original study consisting of 600 items can be done. Our approach was to classify matching pairs up until correlation list position 30 according to their true position. Anything below will get position ‘‘31’’ by default. For a match on position 30 for a database of 590 items the slope of the line is equal to 30/590 = 0.05. In a database of 600 items, the same pair will be on theoretical position 30.6 meaning just a negligible decrease (less than 1 position) and thus a justification for a direct comparison. As will be shown further in this article, the ranking position as a function of the database size can yield interesting information
BF Ranking position
350
The total sample size consists of 189 cartridge cases all of which are correlated against the database of 590 reference items. The 189 correlation results are subdivided in six groups by ammunition brand. These smaller groups are statistically less significant and variability could occur. As such, the results are only partially representative but obviously the same is true for the results obtained with the original RBID study. 3.2.1. Correlations with the same brand of ammunition Sometimes different brands of ammunition are marked very differently during the firing process so ammunition brand influences the correlation result. In principle it is the cartridge case build material and hardness as well as manufacturing tolerances that lie at the basis of these differences [5,6]. It is logical that an automated system cannot make interpretations about these differences. Automated systems are as objective as can be and the correlation result takes into consideration both similarities and differences. As such, the correlation result is always correct. But differences in markings can be so important that a match will be low in the correlation list, just because of that. It is only at the stage where the examiner is proceeding with the comparisons, that differences in markings can sometimes be interpreted as being irrelevant when they originate from different ammunition brands. At the point where the examiner goes in depth on the (on screen) comparisons he could make interpretations about the observed similarities and differences, and focus on the markings that he estimates to be more reproducible, based on his training and experience. But this is where the subjectivity sets in again. The correlation between two cartridge cases of the same brand is the closest scenario to review the reproducibility of the markings from one shot to another. It gives a first indicator of the system’s performance. From Fig. 2 one can learn two interesting things. One is the number of matches that were found on the first position in the correlation list and the other is the number of matches that are positioned lower than 30. Evofinder1 positions 18 out of 31 matches on the first position. In that respect it compares very well to the IBIS1 HeritageTM. Evofinder1 positions 3 out of 31 matches at position 31 or lower. IBIS1 HeritageTM did worse with 8 out of 31. Generally, the graph demonstrates an improvement of the Evofinder1 system as opposed to the IBIS1 system: 90% vs. 74% within the first 30 positions of the correlation list for cartridge cases of the same brand.
300 250 200 150 100 50 0 0
50
100
150
200
250
300
350
400
450
500
550
600
650
database size Fig. 1. Graph representing the Evofinder1 breechface ranking position of the pair EV141-SS315 as a function of size of the RBID. The graph cannot be generalized as the ranking position strongly depends on the presence and reproducibility of the markings. However, the behaviour is linear, which is general for all items.
3.2.2. Correlations with different brands of ammunition It is well known that ammunition brand will have a severe influence on the correlation efficiencies. This was one of the main aspects that influenced the outcome of the RBID study in 2003. The following graphs (Figs. 2–7) show the histograms of the best ranking order obtained for Winchester, Speer, Wolf, Federal and CCI cartridge cases in correlation against R–P cartridge cases. Again each graph compares the occurrence of a match as a function of ranking position for the Evofinder1 vs. the IBIS1 HeritageTM system. Again it is specifically the occurrence of matches on the first position of the correlation list as well as the number of matches that situate themselves at position 31 or lower that reveal the increase in correlation efficiency of Evofinder1 as opposed to IBIS1
J. De Ceuster, S. Dujardin / Forensic Science International 248 (2015) 82–87
85
Remington-Peters Occurrence (number)
20 IBIS 15
Evofinder
10
5
0 1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Best Ranking Order Fig. 2. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 590-gun RBID. Both the reference cartridge cases and the questioned items were of the same brand (Remington–Peters). Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
HeritageTM. Fig. 8 shows the cumulative percentage of matches as a function of best ranking position for the whole study (189 comparisons). It shows that for Evofinder1 in comparison with IBIS1 HeritageTM there is a doubling of efficiency for discerning a match and positioning it on the first position of the correlation list (best ranking position). 42% of all matches on the first two positions in the correlation list and 64% on the first 30 positions is not a bad result. Still, after that, the efficiency will diminish as one goes through the correlation list. As a result, and this will be shown further in this paper, one would have to go through a lot more on screen comparisons to get the correct item.
Winchester
20 Occurrence (number)
3.2.3. Correlation time as a function of DB size Evofinder1 has the option of extending the correlation time for more in depth calculation. This can be chosen by the operator. For the results presented in this section, correlation times were always maximized. The more entries to compare with, the longer the correlation time. By performing the 189 comparisons it was observed that not every item took the same time to correlate. The behaviour, however, is clearly linear (see Fig. 9) with slope factor between 0.19 (for the fastest) and 0.33 for the slowest with 0.25 on the average.
IBIS 15
Evofinder
10
5
0 1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Best Ranking Order
Fig. 3. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 591-gun RBID. The reference cartridge cases were of the Remington–Peters brand while the questioned items were of the Winchester brand. Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
Speer
20 Occurrence (number)
Fig. 5. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 590-gun RBID. The reference cartridge cases were of the Remington–Peters brand while the questioned items were of the Wolf brand. Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
IBIS 15
Evofinder
10
5
0 1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Best Ranking Order
Fig. 4. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 591-gun RBID. The reference cartridge cases were of the Remington–Peters brand while the questioned items were of the Speer brand. Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
Fig. 6. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 591-gun RBID. The reference cartridge cases were of the Remington–Peters brand while the questioned items were of the Federal brand. Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
J. De Ceuster, S. Dujardin / Forensic Science International 248 (2015) 82–87
Fig. 7. Histogram representing the best ranking order for either firing pin or breech face correlations, as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 591-gun RBID. The reference cartridge cases were of the Remington–Peters brand while the questioned items were of the CCI brand. Ranking order ‘‘31’’ represents all items ranked at position 31 or lower.
Cumulative % of hits vs best ranking position 100% 90% 80%
% of hits
70% 60% 50% 40% 30% 20% 10% 0% 1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
IBIS
Evofinder
Fig. 8. Graph representing the cumulative percentage of matches that are in the correlation list as a function of best ranking position (up until position 30) for either firing pin or breech face as provided by Evofinder1 and IBIS1 HeritageTM (reproduction from original data from De Kinder) for a 592-gun RBID.
From the original RBID results with IBIS1 the authors obtained a linear behaviour as well but there was an offset of about 25 s that was not explained by the authors. Using the average correlation time and extrapolating, it would take Evofinder1 about 40.9 min to compare 1 item against 10,000 items in a database. This time is comparable to with what was found for IBIS1 (45.9 min). From that point of view there has not been massive increase in efficiency. However, all depends on the
speed of the computer. And logically, the data that needs to be compared by Evofinder1 is likely a lot more complex. Where IBIS1 based itself on two-dimensional images, Evofinder takes into consideration topographical information. Newer technologies may decrease the calculation time significantly. For instance, the possibility exists to spread calculations on different computers in a network or on graphical card CPUs. In principle, running a correlation does not require human interference. Correlation duration is therefore not a real issue except when an answer is really urgent. When multiple correlations against a large database need to be performed, they can be run overnight and the results be evaluated the next day. 3.3. System performance Using the results of the test, the performance of the automated system could be thoroughly evaluated. A probability curve [7] for finding a match as a function of browsing down the correlation list was constructed to aid in this process (see Fig. 10). The probability curve rises quickly after which it gradually flattens out. The explanation is simple. As an automated system can be regarded as a sorting machine, the cartridge cases that resemble best (potential matches) will be positioned high in the ranking. As one browses through the correlation list the noise level quickly increases. After a certain point where the probability curve nearly flattens, the probability for lesser evident matches to position them on position x or a lot further down the ranking become quite comparable especially with low reproducible markings such as e.g. on different brands of ammunition. Following Rahm [7], the ideal examiner is someone who manually compares with each individual item in the open case file and eventually finds the match if there is one. The probability curve of an ideal examiner is therefore essentially a straight line. For each cartridge case that is compared there is an equal chance that it is or is not a match. The ideal examiner’s efficiency will never fail as opposed to an automated system that is very efficient for evident matches yet less efficient for non-evident matches. Rahm suggests using a break condition for hit list verifications in practical case work. As soon as the automated system’s efficiency drops below that of the ideal examiner, the chance to find the correct item
Probability of finding a match vs ranking position for the RBID database 70
250
100% 90%
60
80% 50 Frequency
correlation time (s)
200
150
100
70% y = 0.1369ln(x) + 0.1475 60%
40
50% 30
40% 30%
20
20%
50
0 0
10
10% 0%
0 100
200 fastest
300 400 500 database size (# items) slowest average IBIS RBID study
600
700
Fig. 9. Graph representing the time needed for the Evofinder1 to correlate a questioned item against a database for which the size is increasingly growing (xaxis). The study demonstrated that there is no fixed time for all objects but rather there was slight spread. The figure therefore shows the graphs for the fastest, the slowest and the average time need. As a reference, the data from the study performed by IBIS1 HeritageTM is given (reproduced from the study by De Kinder).
Accumulative percentage (probability)
86
1
51
101
151
201 251 301 351 401 451 501 551 Best Ranking Position Frequency Cumulative % Log. (Cumulative %)
Fig. 10. Graph representing the Evofinder1 probability curve for having a match in the correlation list of a 592-gun RBID as a function of best ranking position (either breech face or firing pin impression). The data could be best fitted with a logarithmic function. The straight line represents the ideal examiner’s efficiency. The left y-axis represents the frequency (count) of matches as function of the best ranking position; the right y-axis represents the accumulative percentage
J. De Ceuster, S. Dujardin / Forensic Science International 248 (2015) 82–87
becomes at that point lower than the chance that an examiner would have without sorting help of a machine, thus performing arbitrary selection. This occurs when the slope of the probability curve of the system becomes equal to or less than the slope of the ideal examiner’s efficiency. Studies by Rahm, based on real case work with the Evofinder database containing several hundreds of cartridge cases have resulted in low break conditions for correlation list verification (e.g. the 11th position for 9 mm Luger, see Table 3 in [7]). He concludes, however, that it is overall recommendable to go for the first 20. Yet, it is clear that in such a way potential matches, even though they are less evident, will be missed. If no matches were discovered by verifying the correlation list only until the break condition, how certain can one be that there really was no match present in the database? How does the system perform when it comes to discerning a match from other non-relevant items? From a scientific point of view these are interesting questions to be asked. The constructed RBID database can help in understanding. This information can be derived from the probability curve of the system. It has to be kept in mind, however, that the test that is described in this article was tough for a number of reasons. Firstly, there are the different brands of ammunition that really lower the score. In case work one will always try to test fire ammunition from the same brands and with properties close to what is being used by the criminals. In addition, it is possible to scan several cartridge cases (different brands) fired with the same firearm in order to appreciate the inter-variability of markings that can occur. It indicates how well the other item can be found under optimal conditions. This procedure gives a clue of how far the correlation list must be verified by the operator. Secondly, the database consists of firearms of the same brand, with the same class characteristics, a situation unlikely to occur in real open case files. In other words, the detection probability limits that are attributed here to a certain break condition represent a worst case scenario situation. The data points were fitted with the function proposed by Rahm in [7] (point 3.1). The best fit was obtained with parameters a = 0.82, b = 9.81 and c = 0.00041. The R2 value for this fit was 0.95. Alternatively, the data was fitted with a simple logarithmic function y = d + e ln(x). Even though this is scientifically not completely the correct function to be used, it fitted even better than the aforementioned function. With parameters d = 0.1369 and e = 0.1475 the R2 value was 0.98. For the database of 592 specimen, the slope of the ideal examiner’s efficiency curve is 1/592 = 0.0017. The fit with the function by Rahm has the same slope at a ranking position of 65; with the logarithmic function this is 81. Using these points as break condition means that respectively 74% (Rahm) and 75% (logarithmic) of all potential matches will be found, leaving respectively 26% and 25% potentially unfound. One could work the other way around as well. A laboratory could decide to set a confidence level of e.g. 90% for a specific correlation task. This means that many more items in the correlation list need to be verified. In the case of the RBID with 592 references, cartridge cases up to position 255 need to be reviewed. It should be reminded that this graph represents best
87
ranking position either on BF or FPI. This means that verifications need to be done on both traces independently meaning 2 255 onscreen verifications. 255 specimens in a database of 592 represent 43% of the complete database that needs to be verified. As ranking position behaves linear with database size, such a comparison work could eventually be considered for smaller databases but not for large ones as this is a lot of work and not efficient for any laboratory that is trying to cope with backlogs. For time reasons these laboratories sometimes set a limit to the amount of verifications they perform e.g. by looking only at the first 30 items. Obviously, this will downsize the confidence level, independent of database size. This could be done under the perspective that in any way not all firearm crimes will be solved with ballistic image databases. In the RBID database that was constructed for the purpose of the study, only looking at the first 30 items will amount to 64% certainty to find the match. The database could be used to verify system’s performance during case work. Also, when the system would receive a massive update on the correlation part, this could be checked.
4. Conclusions The RBID study, initially performed by De Kinder et al., was repeated using novel technology. The Evofinder1 system has demonstrated an important improvement in automated ballistic imaging equipment. No doubt this is also valid for other state-of-the-art equipment that is available on the market nowadays. Nevertheless the idea of a reference ballistic imaging database remains utopic for now. The main aspects limiting this idea are two-fold. One is the nature of the markings, which reproducibility is predominantly influenced by the cartridge case and primer hardness. The other is that for large databases one has to verify a great deal of the potential matches in the correlation list in order to have a certain confidence level. The system’s performance on a large database such as the RBID is a good indicator for its behaviour during daily work with the open case file.
References [1] J. De Kinder, F. Tulleners, H. Thiebaut, Reference ballistic imaging database performance, Forensic Sci. Int. 140 (2004) 207–215. [2] R. Nennstiel, J. Rahm, A parameter study regarding the IBISTM correlator, J. Forensic Sci. 51 (2006) 18–23. [3] R. Nennstiel, J. Rahm, An experience report regarding the performance of the IBISTM correlator, J. Forensic Sci. 51 (2006) 24–30. [4] G. Gerules, S.K. Bhatia, D.E. Jackson, A survey of image processing techniques and statistics for ballistic specimens in forensic science, Sci. Justice 53 (2013) 236–250. [5] J.J. Davis, Primer cup properties & how the affect identification, AFTE J. 42 (2010) 3–22. [6] P. De Smet, R. Hermsen, B. van Leuven, J. De Kinder, K. Hoffmann, Experimental evaluation of the impact of seating depth variations on observed marks on primers, Forensic Sci. Int. 179 (2008) 163–171. [7] J. Rahm, Evaluation of an electronic comparison system and implementation of a quantitative effectiveness criterion, Forensic Sci. Int. 214 (2012) 173–177.