- Email: [email protected]
On the use of the likelihood ratio for forensic evaluation: Response to Fenton et al.
SCIJUS-00436; No of Pages 3 Science and Justice xxx (2014) xxx–xxx
Contents lists available at ScienceDirect
Science and Justice journal homepage: www.elsevier.com/locate/scijus
Letter to the editor
On the use of the likelihood ratio for forensic evaluation: Response to Fenton et al. Alex Biedermann a,c,⁎, Tacha Hicks a, Franco Taroni a, Christophe Champod a, Colin Aitken b a b c
University of Lausanne, Ecole des sciences criminelles, Institut de Police Scientifique, 1015 Lausanne-Dorigny, Switzerland University of Edinburgh, School of Mathematics and Maxwell Institute, Edinburgh, EH9 3JZ, United Kingdom Università Ca' Foscari Venezia, Department of Economics, Cannaregio 873, 30121 Venice, Italy
a r t i c l e Available online xxxx
i n f o
a b s t r a c t This letter to the Editor comments on the article When ‘neutral’ evidence still has probative value (with implications from the Barry George Case) by N. Fenton et al. [[1], in press]. © 2014 Forensic Science Society. Published by Elsevier Ireland Ltd. All rights reserved.
Dear Editor, With the title of their paper, When ‘neutral’ evidence still has probative value (with implications from the Barry George Case), Fenton et al. ([1], in press) chose a formulation that has the potential to sound attractive – in the sense of paradox – to a wide readership, which makes it worthwhile to look at it more closely. We wish to discuss three points: first, the different inferences which can arise from different levels of propositions; second, the lack of a need for the propositions to be exhaustive and third the inference which may be drawn from a likelihood ratio of 1. A main point raised in [1] is that an expression of probative value for a forensic result based on a likelihood ratio, given source-level propositions, may be very different from an expression of probative value given another pair of propositions (e.g., given crime level propositions). The paper claims to point out the full impact of this issue for the first time, and to demonstrate the use of Bayesian networks as a way to clarify the distinction between different propositional levels. The discussion is driven by an emphasis on requiring propositions in the forensic evaluation to be both mutually exclusive and exhaustive. In this letter, we seek to draw the readers' attention to the fact that none of these points of discussion are unprecedented topics, and that there is no need to fear paradoxical situations. We will do so by placing an emphasis on several fundamental works published previously in this and other journals, several of which are not quoted in [1]. Twenty years ago, Ian Evett [2] presented in this journal (then running under the title Journal of the Forensic Science Society (JFSS)) a
⁎ Corresponding author at: University of Lausanne, Ecole des sciences criminelles, Institut de Police Scientifique, 1015 Lausanne-Dorigny, Switzerland. E-mail addresses: [email protected] (A. Biedermann), [email protected] (T. Hicks), [email protected] (F. Taroni), [email protected] (C. Champod), [email protected] (C. Aitken).
likelihood ratio development (for cases in which material is recovered on a crime scene) that coherently incorporates distinct propositional levels, in particular the main propositions referring to an offence, and propositions now typically known as source-level propositions (referred to in [2] as intermediate association propositions). This development proceeds from source to crime level propositions through the consideration of so-called association propositions. These association propositions express the relevance of the trace material for the case at hand (i.e., the proposition according to which the trace material has been left by the offender, or one of the offenders, as the case may be). The form of the likelihood ratio for ultimate propositions (offence level) is shown to depend on various factors, such as the rarity of the corresponding analytical features or the probability that, given that the stain is not case-related, it was left by the suspect for innocent reasons. Moreover, sensitivity analysis on these factors demonstrate that the likelihood ratio may change (substantially), even directionally (i.e., support for one proposition may run into support of the specified alternative) if the level of proposition changes. The likelihood ratio development in [2] has later been reconsidered in the same journal (now Science & Justice) for the context of shoemarks found on a crime scene [3], drawing again a distinction between inference about source and crime level propositions, and emphasising – amongst other factors – the role of relevance for the case at hand. In 1998, the whole topic received a full treatment in this journal by the seminal paper by Cook et al. [4], extending the discussion to the so-called activity level propositions (formerly association level propositions). The essential concept of hierarchy of propositions and its relationship with the value of DNA findings has also been explored by Evett et al. in 2002, already taking advantage of Bayesian networks [5]. As an aside, note that propositions “defendant was/was not at the crime scene” ([1], abstract), referred to by the authors as source level propositions, are not source level propositions in the sense of the above mentioned hierarchy because they do not say anything about a potential source, nor about the actual material or trace for which the source is in dispute.
http://dx.doi.org/10.1016/j.scijus.2014.04.001 1355-0306/© 2014 Forensic Science Society. Published by Elsevier Ireland Ltd. All rights reserved.
Please cite this article as: A. Biedermann, et al., On the use of the likelihood ratio for forensic evaluation: Response to Fenton et al., Sci. Justice (2014), http://dx.doi.org/10.1016/j.scijus.2014.04.001
2
A. Biedermann et al. / Science and Justice xxx (2014) xxx–xxx
From the body of literature quoted above, followed by many others subsequently, it became clear for the forensic community at large that, given that the likelihood ratio is a ratio between two conditional probabilities, it is natural that the probative value may be different for one pair of propositions and other conditioning information, than for another pair. This is important but not a novelty. Also, in principle, there is nothing worrying in this because different pairs of propositions refer to different issues. The very fact that the likelihood ratio may change due to differences in situations is what makes it actually useful. Indeed, there would be little interest in the likelihood ratio if it could not be sensitive to differences in situations. In turn, this presents an incentive for both scientists and all other participants in the legal process to take the task of evaluation very seriously. For the same reason, it is also good practice – widely implemented and recommended (e.g., [6,7]) – for scientists to note in their written reports that their assessment is based on the information and propositions presented to them at a given point in time, and that any changes in the framework or circumstances (notably changing propositions) may require a review of the scientist's evaluation of the probative value. There is no problem whatsoever in such a reconsideration: it does not intend to suggest that the previous evaluation was wrong. The point is that it represents a different evaluation because the conditioning (information) is different. This crucial importance of conditioning information in forensic evaluation is one of the reasons why it has been retained as one of the three principles of interpretation (e.g., [8]). More generally, the question of how to frame propositions appropriately for reasoning in forensic science applications based on likelihood ratios is a challenging one, not least because its discussion in applied settings may be disturbed by so-called (prescriptive) explanations [9], which provide a complete account of the scientist's results, but that cannot be discriminated on the basis of these results. Here, again, it is a consideration of the framework of circumstances which represents the key to resolving the issue: evaluation is not about invoking any kind of potential explanation, but about invoking one that makes sense in the context of the case; that is, explanations may well provide a good account of the scientist's findings, but they may be fanciful, speculative or even a statement of the obvious, none of which is a quality required for propositions. The latter have a logical relationship with the agreed stated framework of circumstances. The choice of propositions is a crucial issue. To draw yet another distinction with respect to explanations, it is instructive to mention that explanations need not be mutually exclusive, where propositions need to be. Stated otherwise, propositions need to be mutually exclusive (i.e., they cannot both be true), as Fenton at al. [1] rightly mention, but this is an entirely elementary requirement, and uncontested throughout the relevant probabilistic literature in the field. Another question, the second point of discussion, is whether the propositions of interest need be exhaustive. Exhaustive propositions are not needed for the application of the so-called odds form of Bayes' Theorem. We use the term ‘odds’ advisedly since if the propositions are not exhaustive then the ratio of their probabilities are not odds in the technical sense of the word. This form of Bayes' Theorem states that the posterior ‘odds’ in favour of one proposition over another, posterior to evidence, is the product of the likelihood ratio and the prior ‘odds’ in favour of that proposition over the other. For example, consider a paternity analysis in relation to the fatherhood of a child. DNA profiling results are available. The two propositions against which the DNA profiling results are to be evaluated are, first, that the father of the child is one particular putative father and, second, that the father of the child is a man unrelated to the putative father. These are not exhaustive propositions. There are still possible fathers amongst the relatives of the putative father. Of course, these relatives may not be ‘in the frame’ so to speak, but the existence of that proposition means that the other two are not exhaustive propositions. An unconditional requirement of propositions to be exhaustive may encounter practical limitations and is not necessary for the proper
administration of justice, contrary to what Fenton et al. [1] suggest. Although it is straightforward to obtain a mutually exclusive and exhaustive alternative by the negation of the first proposition, it is not advisable, as such an alternative proposition may be unhelpful in practical terms. The reason for this is that it may not be feasible to assign a probability for the scientific findings given such a proposition. For forensic practice it is thus relevant to ask if the evaluator is able to specify the denominator of the likelihood ratio with respect to a relevant population as defined by the alternative proposition. The price for too stringent a requirement for exhaustiveness should not lead to an inferential impasse. Besides, statistical literature, too, has pointed out that it may be necessary to make an informed judgement in deciding which of all the possible propositions should be entertained at all (e.g., [10]). Instead of an absolute exhaustiveness, practice can proceed with an acceptable coverage, that is the omission of no relevant proposition. Probabilistic evaluation in the context of DNA mixtures provides a telling illustration for this. For a given DNA mixture, propositions may cover the suspect plus one unknown person, the suspect plus two unknown persons etc. Clearly, the enumeration can be potentially large but we can also see that quite soon, both the component probabilities for the profiling results given propositions supposing high numbers of contributors, as well as the prior probabilities of these propositions, become increasingly smaller, and hence of weaker influence on the overall evaluation. Some propositions are effectively discarded because they are considered as unrealistic. This has led to the view that propositions are relevant for consideration when they contribute substantially to prior and likelihood [11,12]. Even for unmixed DNA traces, the question of how to define the propositional framework sensibly has been a point of attention for scientists. Buckleton and Triggs [13] have argued, for example, that in cases where the suspect has a non-excluded relative, the alternative proposition should not be restricted to only an unknown and unrelated person. Instead, multiple propositions should be considered, including non-excluded relatives (i.e., cousins, sibling, etc.), a point also made elsewhere in legal literature (e.g., [14]). Note further that this position with respect to propositional exhaustiveness has also been considered in this journal previously through the use of Bayesian networks by Buckleton et al. [15]. This group of authors used an illustration in the context of DNA profiling results when it is felt that the sole alternative, ‘an unknown person, unrelated to the suspect, is the source of the crime stain’, is unsatisfactory in the light of the circumstances given by the case. Since the formal likelihood ratio development for multiple propositions may involve some technical detail [16], the authors in [15] proposed generic Bayesian network structures that allow one to regroup propositions according to the particular needs at hand, and assign likelihood ratios accordingly. Another matter of concern is the claim that a likelihood ratio of 1 has probative value. Comments by Royall ([17], p. 8) are pertinent here. “The law of likelihood applies to pairs of hypotheses,1 telling when a given set of observations is evidence for one versus the other: hypothesis A is better supported than B if A implies a greater probability for the observations than B does. This law represents a concept of evidence that is essentially relative, one that does not apply to a single hypothesis taken alone. Thus it explains how observations should be interpreted as evidence for A vis-a-vis B, but it makes no mention of how those observations should be interpreted as evidence in relation to A alone.” Royall then goes on to give an example where given evidence X = x, the probability of A given X = x is less than the probability of A before there was knowledge of the value of X [Pr(A|X = x) b Pr(A)] and the probability of B given X = x is less than the probability of B before there was knowledge of the value of X [Pr(B|X = x) b Pr(B)] yet 1 Royall uses the word ‘hypothesis’ where we use ‘proposition’; in this context the two are interchangeable.
Please cite this article as: A. Biedermann, et al., On the use of the likelihood ratio for forensic evaluation: Response to Fenton et al., Sci. Justice (2014), http://dx.doi.org/10.1016/j.scijus.2014.04.001
A. Biedermann et al. / Science and Justice xxx (2014) xxx–xxx
the ratio of Pr(A|X = x) to Pr(B|X = x) is twice the ratio of Pr(A) to Pr(B). The observation is not evidence supporting A taken alone — it is evidence supporting A over B. The explanation lies in the existence of another proposition C and is given in Appendix A. If A and B were exhaustive then this situation would not arise. For mutually exclusive and exhaustive propositions, an LR N 1 implies a posterior probability of the proposition in the numerator greater than the prior probability of the proposition in the numerator. However, as explained above, it is not necessary for the propositions to be exhaustive. It should also be noted that if the propositions are exhaustive and the LR = 1 then the posterior probability of the proposition in the numerator equals the prior probability of the proposition in the numerator. If the propositions are not exhaustive and the LR = 1 then all that can be said is that the posterior ‘odds’ equals the prior ‘odds’; the posterior probability of the proposition in the numerator can be greater than, equal to or less than the prior probability of the proposition in the numerator. Thus, the particular claim made by Fenton et al. [1] in that a likelihood ratio of 1 for a given couple of propositions can still have significant2 probative value, is based on the indubitable result that it is possible for the posterior probability of the prosecution proposition to be greater than its prior probability with evidence for which the likelihood ratio is 1. In such a situation the result that the posterior probability for the defence proposition increases by proportionally the same amount is dismissed in [1] as essentially irrelevant. This increase in the posterior probability for the defence proposition may be irrelevant in a situation where the defence proposition itself is not relevant for consideration of the prosecution proposition or the evidence under consideration is itself not relevant for assessment of the propositions. The specification of propositions is a difficult topic, as Fenton et al. comment. However it is disingenuous when considering a likelihood ratio of 1 to concentrate only on the posterior probability of the prosecution proposition. The posterior probability of the defence proposition has changed by the same amount. Thus the evidence has no probative value relative to these propositions. It may have probative value relative to other unstated propositions but that is not relevant to this argument. In summary, we conclude that neither the concern of exhaustiveness, nor the ways to deal with this issue using graphical probability models, represent original topics. Scientists who care about their approach to forensic evaluation can find much material on the subject in existing literature, amongst which Science & Justice is a primary source (e.g., examples quoted above). By exposing known concepts in the context of a highly mediated case, Fenton et al.'s [1] critical exposition bears the risk of leaving the unfortunate impression that there is an inherent problem with probability theory as a framework with which to deal with uncertainty. A suggestion that the findings in relation to gunshot residue should have been weighted differently by the scientists arises from a confusion between the role assigned to them and the role assigned to the court. In the context of the propositions put to the scientist, the conclusion stating that the value to be assigned to the findings was ‘neutral’ – that is a likelihood ratio of 1 – was fully justified. Had the propositions been different, the assessment of the findings may have required to be revised, but there is nothing new here and more importantly, there is nothing in the framework of circumstances of that particular case that suggests
3
that such a change is needed. Discussion of such a case in abstracto only creates confusion that is not helpful for a proper administration of justice. Appendix A This example is taken from Royall ([17], p. 8). An observation that supports A over B can reduce the probabilities of both hypotheses. Suppose there is one other hypothesis C and that a priori PrðAÞ ¼ PrðBÞ ¼ Pr ðC Þ ¼ 13 and these hypotheses are mutually exclusive and exhaustive; hypotheses A and B are exclusive but not exhaustive. Suppose also that there is evidence X that takes a value x, such that PrðAjxÞ ¼ 16; PrðBjxÞ ¼ 121 and PrðCjxÞ ¼ 34 . The effect of X is to reduce the probability of A and of B whilst increasing the probability of C. Observation X = x also doubles the probability of A relative to B; i.e., Pr(A|X = x) b Pr(A) and Pr(B|X = x) b Pr(B), yet PrðAjX ¼ xÞ PrðAÞ ¼2 : PrðBjX ¼ xÞ PrðBÞ The observation is not evidence supporting A taken alone — it is evidence supporting A over B. References [1] N. Fenton, D. Berger, D. Lagnado, M. Neil, A. Hsu, When ‘neutral’ evidence still has probative value (with implications from the Barry George Case), Sci. Justice (2014). http://dx.doi.org/10.1016/j.scijus.2013.07.002 (in press). [2] I.W. Evett, Establishing the evidential value of a small quantity of material found at a crime scene, J. Forensic Sci. Soc. 33 (1993) 83–86. [3] I.W. Evett, J.A. Lambert, J.S. Buckleton, A Bayesian approach to interpreting footwear marks in forensic casework, Sci. Justice 38 (1998) 241–247. [4] R. Cook, I.W. Evett, G. Jackson, P.J. Jones, J.A. Lambert, A hierarchy of propositions: deciding which level to address in casework, Sci. Justice 38 (1998) 231–239. [5] I.W. Evett, P.D. Gill, G. Jackson, J. Whitaker, C. Champod, Interpreting small quantities of DNA: the hierarchy of propositions and the use of Bayesian networks, J. Forensic Sci. 47 (2002) 520–530. [6] I.W. Evett, G. Jackson, J.A. Lambert, S. McCrossan, The impact of the principles of evidence interpretation and the structure and content of statements, Sci. Justice 40 (2000) 233–239. [7] Association of Forensic Science Providers, Standards for the formulation of evaluative forensic science expert opinion, Sci. Justice 49 (2009) 161–164. [8] I.W. Evett, B.S. Weir, Interpreting DNA Evidence, Sinauer Associates Inc., Sunderland, 1998. [9] I.W. Evett, G. Jackson, J.A. Lambert, More on the hierarchy of propositions: exploring the distinction between explanations and propositions, Sci. Justice 40 (2000) 3–10. [10] I.J. Good, Good Thinking: The Foundations of Probability and Its Applications, University of Minnesota Press, Minneapolis, 1983. [11] D.J. Balding, Weight-of-Evidence for Forensic DNA Profiles, John Wiley & Sons, Chichester, 2005. [12] J.S. Buckleton, J.M. Curran, P. Gill, Towards understanding the effect of uncertainty in the number of contributors to DNA stains, Forensic Sci. Int. Genet. 1 (2007) 20–28. [13] J.S. Buckleton, C.M. Triggs, Relatedness and DNA: are we taking it seriously enough? Forensic Sci. Int. 152 (2005) 115–119. [14] R.O. Lempert, Some caveats concerning DNA as criminal identification evidence: with thanks to the Reverend Bayes, Cardozo Law Rev. 13 (1991) 303–341. [15] J.S. Buckleton, C.M. Triggs, C. Champod, An extended likelihood ratio framework for interpreting evidence, Sci. Justice 46 (2006) 69–78. [16] C.G.G. Aitken, F. Taroni, Statistics and the Evaluation of Evidence for Forensic Scientists, 2nd Edition John Wiley & Sons, Chichester, 2004. [17] R.M. Royall, Statistical Evidence: A Likelihood Paradigm, Chapman & Hall, London, 1997
2 The word ‘significant’ as used in [1] is ambiguous because, strictly speaking, ‘significant’ is a technical word in statistics, with a well-defined meaning, but which is not that to which reference is actually made in [1].
Please cite this article as: A. Biedermann, et al., On the use of the likelihood ratio for forensic evaluation: Response to Fenton et al., Sci. Justice (2014), http://dx.doi.org/10.1016/j.scijus.2014.04.001
Contents lists available at ScienceDirect
Science and Justice journal homepage: www.elsevier.com/locate/scijus
Letter to the editor
On the use of the likelihood ratio for forensic evaluation: Response to Fenton et al. Alex Biedermann a,c,⁎, Tacha Hicks a, Franco Taroni a, Christophe Champod a, Colin Aitken b a b c
University of Lausanne, Ecole des sciences criminelles, Institut de Police Scientifique, 1015 Lausanne-Dorigny, Switzerland University of Edinburgh, School of Mathematics and Maxwell Institute, Edinburgh, EH9 3JZ, United Kingdom Università Ca' Foscari Venezia, Department of Economics, Cannaregio 873, 30121 Venice, Italy
a r t i c l e Available online xxxx
i n f o
a b s t r a c t This letter to the Editor comments on the article When ‘neutral’ evidence still has probative value (with implications from the Barry George Case) by N. Fenton et al. [[1], in press]. © 2014 Forensic Science Society. Published by Elsevier Ireland Ltd. All rights reserved.
Dear Editor, With the title of their paper, When ‘neutral’ evidence still has probative value (with implications from the Barry George Case), Fenton et al. ([1], in press) chose a formulation that has the potential to sound attractive – in the sense of paradox – to a wide readership, which makes it worthwhile to look at it more closely. We wish to discuss three points: first, the different inferences which can arise from different levels of propositions; second, the lack of a need for the propositions to be exhaustive and third the inference which may be drawn from a likelihood ratio of 1. A main point raised in [1] is that an expression of probative value for a forensic result based on a likelihood ratio, given source-level propositions, may be very different from an expression of probative value given another pair of propositions (e.g., given crime level propositions). The paper claims to point out the full impact of this issue for the first time, and to demonstrate the use of Bayesian networks as a way to clarify the distinction between different propositional levels. The discussion is driven by an emphasis on requiring propositions in the forensic evaluation to be both mutually exclusive and exhaustive. In this letter, we seek to draw the readers' attention to the fact that none of these points of discussion are unprecedented topics, and that there is no need to fear paradoxical situations. We will do so by placing an emphasis on several fundamental works published previously in this and other journals, several of which are not quoted in [1]. Twenty years ago, Ian Evett [2] presented in this journal (then running under the title Journal of the Forensic Science Society (JFSS)) a
⁎ Corresponding author at: University of Lausanne, Ecole des sciences criminelles, Institut de Police Scientifique, 1015 Lausanne-Dorigny, Switzerland. E-mail addresses: [email protected] (A. Biedermann), [email protected] (T. Hicks), [email protected] (F. Taroni), [email protected] (C. Champod), [email protected] (C. Aitken).
likelihood ratio development (for cases in which material is recovered on a crime scene) that coherently incorporates distinct propositional levels, in particular the main propositions referring to an offence, and propositions now typically known as source-level propositions (referred to in [2] as intermediate association propositions). This development proceeds from source to crime level propositions through the consideration of so-called association propositions. These association propositions express the relevance of the trace material for the case at hand (i.e., the proposition according to which the trace material has been left by the offender, or one of the offenders, as the case may be). The form of the likelihood ratio for ultimate propositions (offence level) is shown to depend on various factors, such as the rarity of the corresponding analytical features or the probability that, given that the stain is not case-related, it was left by the suspect for innocent reasons. Moreover, sensitivity analysis on these factors demonstrate that the likelihood ratio may change (substantially), even directionally (i.e., support for one proposition may run into support of the specified alternative) if the level of proposition changes. The likelihood ratio development in [2] has later been reconsidered in the same journal (now Science & Justice) for the context of shoemarks found on a crime scene [3], drawing again a distinction between inference about source and crime level propositions, and emphasising – amongst other factors – the role of relevance for the case at hand. In 1998, the whole topic received a full treatment in this journal by the seminal paper by Cook et al. [4], extending the discussion to the so-called activity level propositions (formerly association level propositions). The essential concept of hierarchy of propositions and its relationship with the value of DNA findings has also been explored by Evett et al. in 2002, already taking advantage of Bayesian networks [5]. As an aside, note that propositions “defendant was/was not at the crime scene” ([1], abstract), referred to by the authors as source level propositions, are not source level propositions in the sense of the above mentioned hierarchy because they do not say anything about a potential source, nor about the actual material or trace for which the source is in dispute.
http://dx.doi.org/10.1016/j.scijus.2014.04.001 1355-0306/© 2014 Forensic Science Society. Published by Elsevier Ireland Ltd. All rights reserved.
Please cite this article as: A. Biedermann, et al., On the use of the likelihood ratio for forensic evaluation: Response to Fenton et al., Sci. Justice (2014), http://dx.doi.org/10.1016/j.scijus.2014.04.001
2
A. Biedermann et al. / Science and Justice xxx (2014) xxx–xxx
From the body of literature quoted above, followed by many others subsequently, it became clear for the forensic community at large that, given that the likelihood ratio is a ratio between two conditional probabilities, it is natural that the probative value may be different for one pair of propositions and other conditioning information, than for another pair. This is important but not a novelty. Also, in principle, there is nothing worrying in this because different pairs of propositions refer to different issues. The very fact that the likelihood ratio may change due to differences in situations is what makes it actually useful. Indeed, there would be little interest in the likelihood ratio if it could not be sensitive to differences in situations. In turn, this presents an incentive for both scientists and all other participants in the legal process to take the task of evaluation very seriously. For the same reason, it is also good practice – widely implemented and recommended (e.g., [6,7]) – for scientists to note in their written reports that their assessment is based on the information and propositions presented to them at a given point in time, and that any changes in the framework or circumstances (notably changing propositions) may require a review of the scientist's evaluation of the probative value. There is no problem whatsoever in such a reconsideration: it does not intend to suggest that the previous evaluation was wrong. The point is that it represents a different evaluation because the conditioning (information) is different. This crucial importance of conditioning information in forensic evaluation is one of the reasons why it has been retained as one of the three principles of interpretation (e.g., [8]). More generally, the question of how to frame propositions appropriately for reasoning in forensic science applications based on likelihood ratios is a challenging one, not least because its discussion in applied settings may be disturbed by so-called (prescriptive) explanations [9], which provide a complete account of the scientist's results, but that cannot be discriminated on the basis of these results. Here, again, it is a consideration of the framework of circumstances which represents the key to resolving the issue: evaluation is not about invoking any kind of potential explanation, but about invoking one that makes sense in the context of the case; that is, explanations may well provide a good account of the scientist's findings, but they may be fanciful, speculative or even a statement of the obvious, none of which is a quality required for propositions. The latter have a logical relationship with the agreed stated framework of circumstances. The choice of propositions is a crucial issue. To draw yet another distinction with respect to explanations, it is instructive to mention that explanations need not be mutually exclusive, where propositions need to be. Stated otherwise, propositions need to be mutually exclusive (i.e., they cannot both be true), as Fenton at al. [1] rightly mention, but this is an entirely elementary requirement, and uncontested throughout the relevant probabilistic literature in the field. Another question, the second point of discussion, is whether the propositions of interest need be exhaustive. Exhaustive propositions are not needed for the application of the so-called odds form of Bayes' Theorem. We use the term ‘odds’ advisedly since if the propositions are not exhaustive then the ratio of their probabilities are not odds in the technical sense of the word. This form of Bayes' Theorem states that the posterior ‘odds’ in favour of one proposition over another, posterior to evidence, is the product of the likelihood ratio and the prior ‘odds’ in favour of that proposition over the other. For example, consider a paternity analysis in relation to the fatherhood of a child. DNA profiling results are available. The two propositions against which the DNA profiling results are to be evaluated are, first, that the father of the child is one particular putative father and, second, that the father of the child is a man unrelated to the putative father. These are not exhaustive propositions. There are still possible fathers amongst the relatives of the putative father. Of course, these relatives may not be ‘in the frame’ so to speak, but the existence of that proposition means that the other two are not exhaustive propositions. An unconditional requirement of propositions to be exhaustive may encounter practical limitations and is not necessary for the proper
administration of justice, contrary to what Fenton et al. [1] suggest. Although it is straightforward to obtain a mutually exclusive and exhaustive alternative by the negation of the first proposition, it is not advisable, as such an alternative proposition may be unhelpful in practical terms. The reason for this is that it may not be feasible to assign a probability for the scientific findings given such a proposition. For forensic practice it is thus relevant to ask if the evaluator is able to specify the denominator of the likelihood ratio with respect to a relevant population as defined by the alternative proposition. The price for too stringent a requirement for exhaustiveness should not lead to an inferential impasse. Besides, statistical literature, too, has pointed out that it may be necessary to make an informed judgement in deciding which of all the possible propositions should be entertained at all (e.g., [10]). Instead of an absolute exhaustiveness, practice can proceed with an acceptable coverage, that is the omission of no relevant proposition. Probabilistic evaluation in the context of DNA mixtures provides a telling illustration for this. For a given DNA mixture, propositions may cover the suspect plus one unknown person, the suspect plus two unknown persons etc. Clearly, the enumeration can be potentially large but we can also see that quite soon, both the component probabilities for the profiling results given propositions supposing high numbers of contributors, as well as the prior probabilities of these propositions, become increasingly smaller, and hence of weaker influence on the overall evaluation. Some propositions are effectively discarded because they are considered as unrealistic. This has led to the view that propositions are relevant for consideration when they contribute substantially to prior and likelihood [11,12]. Even for unmixed DNA traces, the question of how to define the propositional framework sensibly has been a point of attention for scientists. Buckleton and Triggs [13] have argued, for example, that in cases where the suspect has a non-excluded relative, the alternative proposition should not be restricted to only an unknown and unrelated person. Instead, multiple propositions should be considered, including non-excluded relatives (i.e., cousins, sibling, etc.), a point also made elsewhere in legal literature (e.g., [14]). Note further that this position with respect to propositional exhaustiveness has also been considered in this journal previously through the use of Bayesian networks by Buckleton et al. [15]. This group of authors used an illustration in the context of DNA profiling results when it is felt that the sole alternative, ‘an unknown person, unrelated to the suspect, is the source of the crime stain’, is unsatisfactory in the light of the circumstances given by the case. Since the formal likelihood ratio development for multiple propositions may involve some technical detail [16], the authors in [15] proposed generic Bayesian network structures that allow one to regroup propositions according to the particular needs at hand, and assign likelihood ratios accordingly. Another matter of concern is the claim that a likelihood ratio of 1 has probative value. Comments by Royall ([17], p. 8) are pertinent here. “The law of likelihood applies to pairs of hypotheses,1 telling when a given set of observations is evidence for one versus the other: hypothesis A is better supported than B if A implies a greater probability for the observations than B does. This law represents a concept of evidence that is essentially relative, one that does not apply to a single hypothesis taken alone. Thus it explains how observations should be interpreted as evidence for A vis-a-vis B, but it makes no mention of how those observations should be interpreted as evidence in relation to A alone.” Royall then goes on to give an example where given evidence X = x, the probability of A given X = x is less than the probability of A before there was knowledge of the value of X [Pr(A|X = x) b Pr(A)] and the probability of B given X = x is less than the probability of B before there was knowledge of the value of X [Pr(B|X = x) b Pr(B)] yet 1 Royall uses the word ‘hypothesis’ where we use ‘proposition’; in this context the two are interchangeable.
Please cite this article as: A. Biedermann, et al., On the use of the likelihood ratio for forensic evaluation: Response to Fenton et al., Sci. Justice (2014), http://dx.doi.org/10.1016/j.scijus.2014.04.001
A. Biedermann et al. / Science and Justice xxx (2014) xxx–xxx
the ratio of Pr(A|X = x) to Pr(B|X = x) is twice the ratio of Pr(A) to Pr(B). The observation is not evidence supporting A taken alone — it is evidence supporting A over B. The explanation lies in the existence of another proposition C and is given in Appendix A. If A and B were exhaustive then this situation would not arise. For mutually exclusive and exhaustive propositions, an LR N 1 implies a posterior probability of the proposition in the numerator greater than the prior probability of the proposition in the numerator. However, as explained above, it is not necessary for the propositions to be exhaustive. It should also be noted that if the propositions are exhaustive and the LR = 1 then the posterior probability of the proposition in the numerator equals the prior probability of the proposition in the numerator. If the propositions are not exhaustive and the LR = 1 then all that can be said is that the posterior ‘odds’ equals the prior ‘odds’; the posterior probability of the proposition in the numerator can be greater than, equal to or less than the prior probability of the proposition in the numerator. Thus, the particular claim made by Fenton et al. [1] in that a likelihood ratio of 1 for a given couple of propositions can still have significant2 probative value, is based on the indubitable result that it is possible for the posterior probability of the prosecution proposition to be greater than its prior probability with evidence for which the likelihood ratio is 1. In such a situation the result that the posterior probability for the defence proposition increases by proportionally the same amount is dismissed in [1] as essentially irrelevant. This increase in the posterior probability for the defence proposition may be irrelevant in a situation where the defence proposition itself is not relevant for consideration of the prosecution proposition or the evidence under consideration is itself not relevant for assessment of the propositions. The specification of propositions is a difficult topic, as Fenton et al. comment. However it is disingenuous when considering a likelihood ratio of 1 to concentrate only on the posterior probability of the prosecution proposition. The posterior probability of the defence proposition has changed by the same amount. Thus the evidence has no probative value relative to these propositions. It may have probative value relative to other unstated propositions but that is not relevant to this argument. In summary, we conclude that neither the concern of exhaustiveness, nor the ways to deal with this issue using graphical probability models, represent original topics. Scientists who care about their approach to forensic evaluation can find much material on the subject in existing literature, amongst which Science & Justice is a primary source (e.g., examples quoted above). By exposing known concepts in the context of a highly mediated case, Fenton et al.'s [1] critical exposition bears the risk of leaving the unfortunate impression that there is an inherent problem with probability theory as a framework with which to deal with uncertainty. A suggestion that the findings in relation to gunshot residue should have been weighted differently by the scientists arises from a confusion between the role assigned to them and the role assigned to the court. In the context of the propositions put to the scientist, the conclusion stating that the value to be assigned to the findings was ‘neutral’ – that is a likelihood ratio of 1 – was fully justified. Had the propositions been different, the assessment of the findings may have required to be revised, but there is nothing new here and more importantly, there is nothing in the framework of circumstances of that particular case that suggests
3
that such a change is needed. Discussion of such a case in abstracto only creates confusion that is not helpful for a proper administration of justice. Appendix A This example is taken from Royall ([17], p. 8). An observation that supports A over B can reduce the probabilities of both hypotheses. Suppose there is one other hypothesis C and that a priori PrðAÞ ¼ PrðBÞ ¼ Pr ðC Þ ¼ 13 and these hypotheses are mutually exclusive and exhaustive; hypotheses A and B are exclusive but not exhaustive. Suppose also that there is evidence X that takes a value x, such that PrðAjxÞ ¼ 16; PrðBjxÞ ¼ 121 and PrðCjxÞ ¼ 34 . The effect of X is to reduce the probability of A and of B whilst increasing the probability of C. Observation X = x also doubles the probability of A relative to B; i.e., Pr(A|X = x) b Pr(A) and Pr(B|X = x) b Pr(B), yet PrðAjX ¼ xÞ PrðAÞ ¼2 : PrðBjX ¼ xÞ PrðBÞ The observation is not evidence supporting A taken alone — it is evidence supporting A over B. References [1] N. Fenton, D. Berger, D. Lagnado, M. Neil, A. Hsu, When ‘neutral’ evidence still has probative value (with implications from the Barry George Case), Sci. Justice (2014). http://dx.doi.org/10.1016/j.scijus.2013.07.002 (in press). [2] I.W. Evett, Establishing the evidential value of a small quantity of material found at a crime scene, J. Forensic Sci. Soc. 33 (1993) 83–86. [3] I.W. Evett, J.A. Lambert, J.S. Buckleton, A Bayesian approach to interpreting footwear marks in forensic casework, Sci. Justice 38 (1998) 241–247. [4] R. Cook, I.W. Evett, G. Jackson, P.J. Jones, J.A. Lambert, A hierarchy of propositions: deciding which level to address in casework, Sci. Justice 38 (1998) 231–239. [5] I.W. Evett, P.D. Gill, G. Jackson, J. Whitaker, C. Champod, Interpreting small quantities of DNA: the hierarchy of propositions and the use of Bayesian networks, J. Forensic Sci. 47 (2002) 520–530. [6] I.W. Evett, G. Jackson, J.A. Lambert, S. McCrossan, The impact of the principles of evidence interpretation and the structure and content of statements, Sci. Justice 40 (2000) 233–239. [7] Association of Forensic Science Providers, Standards for the formulation of evaluative forensic science expert opinion, Sci. Justice 49 (2009) 161–164. [8] I.W. Evett, B.S. Weir, Interpreting DNA Evidence, Sinauer Associates Inc., Sunderland, 1998. [9] I.W. Evett, G. Jackson, J.A. Lambert, More on the hierarchy of propositions: exploring the distinction between explanations and propositions, Sci. Justice 40 (2000) 3–10. [10] I.J. Good, Good Thinking: The Foundations of Probability and Its Applications, University of Minnesota Press, Minneapolis, 1983. [11] D.J. Balding, Weight-of-Evidence for Forensic DNA Profiles, John Wiley & Sons, Chichester, 2005. [12] J.S. Buckleton, J.M. Curran, P. Gill, Towards understanding the effect of uncertainty in the number of contributors to DNA stains, Forensic Sci. Int. Genet. 1 (2007) 20–28. [13] J.S. Buckleton, C.M. Triggs, Relatedness and DNA: are we taking it seriously enough? Forensic Sci. Int. 152 (2005) 115–119. [14] R.O. Lempert, Some caveats concerning DNA as criminal identification evidence: with thanks to the Reverend Bayes, Cardozo Law Rev. 13 (1991) 303–341. [15] J.S. Buckleton, C.M. Triggs, C. Champod, An extended likelihood ratio framework for interpreting evidence, Sci. Justice 46 (2006) 69–78. [16] C.G.G. Aitken, F. Taroni, Statistics and the Evaluation of Evidence for Forensic Scientists, 2nd Edition John Wiley & Sons, Chichester, 2004. [17] R.M. Royall, Statistical Evidence: A Likelihood Paradigm, Chapman & Hall, London, 1997
2 The word ‘significant’ as used in [1] is ambiguous because, strictly speaking, ‘significant’ is a technical word in statistics, with a well-defined meaning, but which is not that to which reference is actually made in [1].
Please cite this article as: A. Biedermann, et al., On the use of the likelihood ratio for forensic evaluation: Response to Fenton et al., Sci. Justice (2014), http://dx.doi.org/10.1016/j.scijus.2014.04.001