- Email: [email protected]
The scientist and the Scales of Justice
PROCEEDINGS OF THE 40th ANNIVERSARY MEETING OF THE FORENSIC SCIENCE SOCIETY
The scientist and the Scales of Justice G JACKSON Forensic Science Service, Washington Hall, Euxton, Chorley PR7 6HJ, United Kingdom
I believe that forensic science is in the middle of a very exciting period - a period of great opportunity and great challenge for the profession. We have developed some of our technologies to the point of extremely high discrimination and sensitivity. We can obtain more and more information from ever-smaller quantities of material. For me, this trend emphasises still further the importance of asking the question - 'what do the results mean in this particular case?'. In addition, the opening-up of the market for forensic science, the advent of direct charging and the drive to provide value-for-money services increases the spotlight on what we do and why we do it. The role of forensic science in the Criminal Justice System (CJS) is being questioned. Our contribution and our future impact continue to be clarified and developed. Inevitably, changes in the demands placed upon us require changes in the way we operate. I would like to share with you some thoughts on these issues.
I would like particularly to focus on one question: 'how does a scientist's evidence affect the scales of justice?' The symbol of the scales of justice is a familiar icon but I had not previously given much thought to the implications for forensic science. If the scales are a good analogy of the process of justice, what is our contribution as forensic scientists? I believe this question goes to the very heart of our role within that system. I would like to begin by exploring the contribution we as scientists have made in the past and then discuss what contribution we should be making in the future. I must emphasise that these are my personal views, and they are not necessarily original. My own thinking on these issues has been helped a great deal by working on a project, Case Assessment and Interpretation (CAI), that has been running formally in the Forensic Science Service for two years. A small team, currently Ian Evett, Jim Lambert, Stella McCrossan and myself, has been leading the project. A main element of the project has involved the team working closely with operational colleagues on the interpretation of a wide variety of case and evidence types. It is this experience, together with my own contact with customers over the years, that has helped me to see more clearly the needs of the CJS, to work through some of the underlying issues and to begin to crystallise some views. Our work has been reported in a series of papers in Science & Justice [l-31. I am not going to consider, in this presentation, evidence
that is interpreted as conclusive: physical fits; footwear marks; toolmarks; and handwriting. I would like to concentrate on the 'greyer' areas that do present us with challenges in interpretation of the findings. Thinking firstly about how forensic science affects the scales of justice, if we view the two sides of the scales as representing, respectively, the prosecution and the defence positions, any piece of evidence, whether it be provided by a scientist, a medical expert, a police officer or a 'lay' person, will be added to one or other of the pans. The impact of the weight of the evidence will depend on the prior position of the pans. For example, it may be that the scales are heavily tilted to one side prior to the scientific evidence. The scientific evidence may simply be adding yet more weight to that side. On the other hand, if the scales are tilted to the other side, the impact of the same scientific evidence may be to counteract totally the weight of the prior evidence -the balance of the pans would be reversed. There will be other situations where, if the evidence is perceived as not helping to make progress with the issue, then the prior balance of the pans will not be affected at all. (Good has written extensively on this topic, e.g., [4]). If this is a valid view of how forensic science affects the scales, the key question then is - 'how is the weight of the scientific evidence to be evaluated?' Furthermore, do we, as a profession, have an agreed, common framework, or model, for the process of evaluation of that weight? Looking back at the contribution we have made in the past, I think it's fair to say that the history of forensic science over the last sixty years has been mainly the history of what could be called 'prosecution-commissioned' science. I don't use that phrase in any pejorative or critical way - simply to observe that the vast majority of forensic science has been carried out at the instigation of the police or prosecution. I think one consequence of this is that, perhaps inevitably, scientists working in this environment may have received sparse, or one-sided, information on the circumstances of the case. This again is not critical of the police or prosecution, but it is more of a reflection of the way the system worked. For example, in the early phase of the investigation, information may well be sparse or vague because of the lack of knowledge of what has happened, and information may well change as the investigation and subsequent prosecution progresses.
Science & Justice 2000; 40(2): 81-85 O The Forensic Science Society 2000
81
The scientist and the Scales of Justice I suspect that one main reason for scientists receiving sparse information may have been our customers believing we could answer their questions in isolation of any background information: 'Science has all the answers'. I think also that we have promoted, possibly unknowingly, an image of authority: 'Trust me, I'm a scientist, I know what you need in this case, I can give you an answer to your question'. I know some of us believe that we do not need any background information to evaluate our findings. This may well be the case when addressing certain types of question but there may be a problem with this approach when addressing other types of question. And there is a risk that we could be viewed as practising a 'black art'. I like a phrase used by Franco Taroni and Colin Aitken in 1998, 'the aura of 'mystic infallibility' associated with scientific evidence has to be carefully considered' [ 5 ] .
Another problem with these phrases is that there may be a perception of bias. It can be argued that phrases such as 'could have come from' and 'consistent with', imply a biased evaluation because the scientist appears to be considering only a prosecution view. For example, a set of findings may be 'consistent with' X having driven a stolen vehicle when, in fact, they may be 'more consistent with' an alternative scenario. Some of us will also be aware of the problems associated with use of phrases of the kind '. ..likely to have originated from.. . ', 'likely to have been in contact... ' and '...highly probable he has handled this material... '. Are these phrases still in common usage? If so, do we expose the reasoning behind the conclusion and is that reasoning robust? As evaluators of findings and as communicators, we should be aware of the potential pitfalls when using these phrases.
Turning now to the issue of 'one-sided' information, I suspect that any of us who have worked for both prosecution and defence will have experienced a difference in how we have been commissioned by the two sides. Submission forms from the police could be quite 'thin' and presented the allegations as the police saw them. It wasn't untypical to see this sort of submission twenty years ago. On the other hand, submissions from the defence generally were, and still are, a lot 'heavier', including an enormous amount of background information, usually in the form of statements. This is not to say that all of the information was useful to us - perhaps one of the most time-consuming tasks was that of reading through the information to glean the essentials like, for example, what the defendant would say. This difference in submission suggests there was a difference in the way we were being commissioned and could therefore have directed a different approach to the two types of work. I am not suggesting that the prosecution side knowingly presented us with partial, biased or edited information but maybe they were not aware of the sort of information that we needed in order to offer a fuller analysis and opinion.
Another feature of statements from the past is that of brevity. When I think back to some of the statements that I produced twenty years ago, many were no more than a few lines. Usually they were the ones that were seen as 'negative': 'I have found nothing of signzjicance on the clothing of Mr X'. I might have been a bit more expansive in some cases: 'No matching fibres have been found on the car seat tapings'; or 'Nothing of apparent significance has been found to connect the clothing of Mr Y with the scene'. But, again, that was one-sided. That left the evaluation of the findings, i.e. the absence of transferred material, to others in the CJS who, because of limited or no knowledge and understanding of that evidence type, would be more poorly placed to make the evaluation. To highlight this issue, I would like to refer to one particular high-profile murder case, dealt with by one of my colleagues, that involved a violent attack on the victim. Two people were convicted for the murder. The case became a cause ckl2bre and featured in one of the campaigning programmes on television. When I watched the programme, not knowing that there had been some forensic science involvement, I saw my colleague's name appear on an image of a CJA statement shown in the title sequence. I wondered, apprehensively, what criticism was about to be levelled at forensic science in general and at my colleague in particular. But I should never have doubted my colleague, for the issue was not about the science, it was more concerned with the other evidence in the case. Some time later, I discussed the case with my colleague. He told me that, from an examination of the scene, he had had very high expectations of finding blood on the clothing of the true assailant(s). He had, in fact, found no blood, matching the victim, on the clothing of the suspects. Instead of saying in his statement something like 'I have found nothing of significance', he had explained his thinking and his analysis of the findings. He said that, in his view, given the circumstances of the case, either the suspects were not the people who had attacked the victim or, if they were the true assailants, this was not the clothing they
Still staying in the past and exploring our contribution, I think it is informative to take a look at what we used to write in our statements. For many years, we have been using phrases such as '...could have come from.. . ', '...consistent with coming from.. . ' and '...entirelyconsistent with... '. However, there are problems with these phrases (G Craddock, P Lamb and A Moffat, personal communication 1989; and J Satterthwaite and J Lambert, personal communication, 1989). One problem was that of the range of perceptions by different people of the strength of these phrases. Taking the simple phrase 'could have' as an example, we have used that particular phrase when reporting an evidence type that had a relative frequency of 1 in 3 as well as for an evidence type that had a relative frequency of 1 in 3 million. That cannot be right.
Science & Justice 2000; 40(2): 81-85
G JACKSON
were wearing at the time. Now this may not sound earthshattering today but, in the 1980s, when this case was being examined, this was unusual reporting. I do not believe many practitioners at that time would have phrased it like that. Today, we spend a lot of time training new recruits in the evaluation and reporting of all findings irrespective of whether the findings are viewed as 'positive' or 'negative'.
A further observation on our contribution is linked to this issue. I believe our style of operation used to be what could be called 'findings-led'. We would carry out the examinations, recover material and carry out analyses. We would look at the results of the tests and try to rationalise our findings by constructing an explanatory scenario: if we did not find a lot of matching glass we would postulate that he must have been further away from the breaking window than was first thought. Or if we found no semen on a complainant's swabs, we would postulate either that he must not have ejaculated or that all the semen had been lost. I suggest that we should look critically at this process of rationalisation to ensure we are acting impartially. I suspect also that we may have been influenced, in an implicit way, by the information that was provided by the customer although many of us may have believed that we worked in isolation, were not influenced by the case circumstances and, indeed, did not need to know any background.
A hypothetical murder scene illustrates the issue:
A man leaves his wife at their home in the evening to visit a friend. He returns a few hours later to find there has been a forced entry through a rear ground floor window and, on entering the house, he finds the body of his wife lying in an upstairs room. He finds a screwdriver outside on the ground below the forced window. He cannot identify the screwdriver because it is not his, and it was not there when he left that evening. A Scenes of Crime Officer takes a cast of the mark on the forced window and submits it, along with the screwdriver, for examination. On comparing the cast with test marks made with the screwdriver, the scientist finds a very good correspondence between the marks but it is not, in the scientist's view, a conclusive match. How is this to be reported? It could be argued that, from a common sense point of view, it would be coy not to express a view that it was this screwdriver, given both the circumstances and the very good comparison results. It may seem eminently reasonable to conclude that this is the right screwdriver. However, let us assume that the screwdriver was not found at the premises, it was one of 200 screwdrivers that had been taken from a population of potential perpetrators from around the country. The police did not indicate that there was any prior evidence for any of the screwdrivers or suspects. Would the scientist be justified in still forming an opinion that it was this screwdriver? I am not implying that it is necessarily wrong to form such a conclusion but that, Science & Justice 2000; 40(2): 81-85
in the former situation, we should be explicit in the information we are taking into account when forming opinions. If we are explicit, then double-counting, by both the scientist and the non-scientist, of the evidence of the presence of the screwdriver at the scene would be avoided. And finally on statements, I think we have had a tendency to concentrate on providing analytical findings. By that I mean we reported the results of our analytical tests and we left the interpretation, or evaluation, of those results to others in the system. I believe that is a clear abrogation of our responsibilities for who else is better placed to evaluate test results? If we do not evaluate results, how do others evaluate them: what framework, what knowledge and what understanding do they have to help them evaluate the evidence in a robust, reliable way? Still on analysis, how did we handle analytical results that gave us some similarities and some differences when two samples were compared? Were we tempted to report these results as 'inconclusive', or even as 'no evidence'? For example, how would we report fibres that were a very good match on white light microscopy but gave us a worrying comparison on fluorescence? Or paint that was good on inorganic analysis but gave us slightly different infrared or PGC traces. I'm sure you can think of other examples. What did we do in those situations? Did we write them off? Did we 'fall off the cliff' and report an exclusion 'this paint fragment has not comefrom that source '? Or were we more equivocal and report 'inconclusive'? Or did we report an inclusion; 'it could have come from that source'? What would be a more robust way of evaluating those results, of dealing with both similarities and differences? Before I move away from the past, I think it is useful briefly to consider the type of cases we used to examine, particularly now viewed with the benefit of hindsight of our experiences over the last 20 years. I will use just one type of case as an example. I can recall a time when we used to examine what were called vehicle-vehicle road traffic accident cases. They could be really good cases. Considerable force was usually involved in the collision and there was a very good chance that material would have been transferred in both directions. The transferred materials were really interesting to examine, both microscopically and with analytical instruments. We could obtain good discrimination. We also had good databases on paint colours, paint layer structures and lens markings. If we were lucky, we might find a physical fit. Wonderful cases! But, were they good value for the customer? How much did it cost us to examine those cases? What was the benefit, in terms of the overall objectives of the police force, in doing those cases? What resources did we apply to them compared with examining rape, robbery and assault cases? I'm sure the owners of the 'innocent' vehicle would be delighted with the powerful scientific evidence but was it really a cost-effective service and a wise
The scientist and the Scales of Justice
use of resources? I think there is a real risk that providers of scientific services can become science and technology-led instead of being driven by the objective of meeting the customers' real needs. Furthermore, customers may become beguiled by impressive technology and 'big' numbers and think that that will solve all their problems. We have a duty to work closely with our customers to harness new developments to meet their real needs in a cost-effective way. I must not leave you with an impression that I am criticising everything from our past because I most certainly am not. We have made, I believe, a significant and rapidly increasing contribution to crime detection, and possibly crime deterrence, over the years. However, we must learn from our collective experience, continue to question critically what we do and improve, in a continuous way, our service to customers. I would like now to return to the main question and to explore how our contribution as scientists could develop. I believe that currently there is a growing recognition that, as forensic scientists, we operate at different levels within a complex Criminal Justice System. The system has distinct phases, each of which may require a different type of service from forensic scientists. We have to accommodate: - working with investigators to give them useful information prior to arrest of a suspect; - working with investigators to determine whether or not there is evidence to support charging a suspect; - working with prosecutors to evaluate the strength of the scientific evidence for or against a defendant; - working with a defence team to test the validity of scientific evidence; - working - with the courts to help them reach a verdict. We bring a scientific approach to the CJS and we know that science and the law are uneasy but necessary partners. How should we therefore support the operation of the Criminal Justice System? What should be our core values, our guiding principles as a profession? I would like to share with you what I think is a good list. It is only my list, and I'm sure you will have your own - we may agree or disagree but therein lies a healthy debate. Provide sustainable, robust evidence By that I mean the scientist should provide information and opinion that is based on sound principles of interpretation: logical; balanced; and supported by a body of data and a degree of understanding of that particular evidence type. If these criteria are met, the evidence will stand the test of time and will be less open to challenge - .years later. Be expert Following on from the first requirement, it follows that to be an 'expert' in a particular domain requires that person to be fully aware of, and understand, the body of information within that domain. The body of information should be extensive. It should be accessible, and preferably in the 84
public arena. It should be testable. An expert should have a high degree of understanding of both the principles and the data in order to evaluate the 'grey' areas of scientific findings. Be transparent By this I mean explaining in a clear and explicit way what we have done, why we have done it and how we have arrived at our conclusions. We need to expose the reasoning, the rationale, behind our work. In principle, our statements should guide our customers through the process of our evaluation. We should provide them with the information that we have used in order to lead them to draw the same conclusion as we have drawn. We should move away from practising the 'black arts'. Act impartially The scientist should look at both sides of an argument. If it is the role of a forensic scientist to test a hypothesis, and evaluate findings, then it can only be done if an alternative is also considered. This raises the question of whether scientists operate in a different manner when commissioned by the two opposing sides - and, if they do, should they? We need to consider how alternative hypotheses are formulated. In the absence of a defence version, how does the prosecution-commissioned scientist begin to formulate an alternative? For the defence-commissioned scientist, there may be difficulties in formulating plausible, alternative hypotheses. Add value By this I mean forensic science should be used in a costeffective way to help resolve specific issues in a case. There should be a focused, directed use of resources to help make progress with the matters that are of relevance and importance to the customer and for which forensic science can actually provide useful information. Wherever possible, scientists should be addressing questions that are a stage advanced beyond the simply analytical. Despite the positive changes that the impact of direct charging has made in this respect, I am sure there is still more progress to be made.
If these constitute a reasonable set of guiding principles, a useful framework for forensic science, how do we make it happen? I believe one very good way to achieve this is through scientists focusing on an evaluation of a likelihood ratio (LR) for the findings. For those of you who may not be too familiar with the concept, I would like to define briefly, in words not symbols, the likelihood ratio and the three key principles that flow from it. My working definition of the likelihood ratio in words is: The probability of the observations given that the proposition were true (and given the truth of the relevant background information) divided by the probability of the observations given that the alternative proposition were true (and given the truth of the relevant background information). Science & Justice 2000; 40(2):81-85
G JACKSON
The three principles for interpretation that flow from this are: 1. We need to be made aware of the relevant background information in each case. The background information provides a framework within which the scientist operates and conditions hislher evaluation. There have to be some basic 'givens' or assumptions and we have to state these clearly to our readers, along with a proviso that if any of these assumptions change, we need to reappraise our interpretation. 2. We need to consider at least two competing propositions or views. Frequently, the prosecution view is relatively clearly defined; the defence view, on the other hand, may be less so. To be clear about the propositions that we are addressing gives us a real focus and a clear purpose to the work that we do and allows our customers also to be clear about what it is we are trying to test. 3. We need to appraise how likely it is that we would observe the results, including both similarities and differences, if the prosecution proposition were true and if the defence view were true. This raises the issue of what knowledge and understanding do we have of that evidence type? What data is there to support interpretation? Do we need more? What research do we need to do in order to be able to interpret robustly the results of a new technique? Applying these three principles to casework, I believe meets the requirements I set out earlier. Exploring them again but in reverse order: 'Add value' A prior appraisal of, firstly, the issues in the case and, secondly, the likely magnitude of the LR, helps both the scientist and the customer arrive at a joint decision on the most cost-effective strategy. There are obvious benefits in this approach in complex cases in which there are choices of items to examine and tests to employ. There should be a clear understanding and agreement on the issues on which the scientist can actually make an impact in a case. 'Act impartially ' Clearly, the LR requires us to consider two competing propositions - I cannot think of a better way in which to demonstrate an impartial approach. 'Be transparent' We can achieve this by explaining our reasoning, our rationale for the case, not only in our statements but also in our conversations with customers and when we give evidence in court. Our customers will be able to see clearly what we have done, why we have done it and how we have arrived at our conclusion. We should set out the background information that we have taken into account and the propositions that we will be testing. We should make clear our expectations and explain the technicalities. We should explain how the findings 'fit', or don't 'fit', the expectations and we should give a bottom line. 'Provide sustainable, robust evidence and be expert'
The LR approach is certainly logical and robust. In an explicit way, the scientist is directed to challenge the Science & Justice 2000; 40(2): 81-85
available data and to use it in an expert way. If there is limited data, this should be reflected in the evaluation of the strength of the evidence. The approach not only helps guard against overstating the value of evidence but also assists obtaining maximum value from the evidence. If we have good databases, then the LR approach helps in the evaluation of both similarities and differences in analytical results and avoids the risk of 'falling-off-the-cliff'. Finally, to return to the main theme, and to answer the question of 'how does a forensic scientist affect the scales of justice?' In my view, the answer is: 'with an evaluation of a likelihood ratio'. The magnitude of the LR is a good assessment of the size of the weight - that is added to one or other of the pans of the scales. I am not implying that a scientist should provide a magic number for the value of the evidence - that would require a degree of precision to the evaluation of the LR that is just not possible in many situations. Rather, the scientist should evaluate broadly the magnitude for the LR and translate that into a verbal equivalent. We have found that the most important part of adopting an LR approach to evaluation is that of improving the clarity of thinking - we should not, in the first instance, become too focused on the 'numbers'. There is a lot of work still to do on various aspects of the LR approach, including: - explaining more fully the LR approach to our customers in the CJS; - developing with them effective ways of presenting our evidence; - agreeing a verbal scale; - applying the approach across all evidence types; - acquiring more background data where necessary; - developing applications in investigative work. Having said that, I believe there is a robustness in this approach - a logical, balanced way of appraising scientific findings. I believe it is the framework to take forensic science forward into the next century. And you may have noticed I have not used the word Bayes anywhere in this presentation! References: 1. Cook R, Evett IW, Jackson G, Jones PJ and Lambert JA. A model for case assessment and interpretation. Science & Justice 1998; 38: 151-156 2. Cook R, Evett IW, Jackson G, Jones PJ and Lambert JA. A hierarchy of propositions: deciding which to address in forensic science. Science &Justice 1998; 38: 231-239 3. Evett IW, Jackson G and Lambert JA. More on the hierarchy of propositions: exploring the distinction between explanations and propositions. Science &Justice 2000: 40: 3-10. 4. Good IJ. Weight of evidence and the Bayesian likelihood ratio. In: Aitken CGG and Stoney DA, eds. The use of statistics in forensic science. Chichester: Ellis Horwood, 1991. 5. Taroni F and Aitken CGG. Probabilistic reasoning in the law. Part 1: assessment of probabilities and explanation of the value of DNA evidence. Science & Justice 1998; 38(3): 165-177.
The scientist and the Scales of Justice G JACKSON Forensic Science Service, Washington Hall, Euxton, Chorley PR7 6HJ, United Kingdom
I believe that forensic science is in the middle of a very exciting period - a period of great opportunity and great challenge for the profession. We have developed some of our technologies to the point of extremely high discrimination and sensitivity. We can obtain more and more information from ever-smaller quantities of material. For me, this trend emphasises still further the importance of asking the question - 'what do the results mean in this particular case?'. In addition, the opening-up of the market for forensic science, the advent of direct charging and the drive to provide value-for-money services increases the spotlight on what we do and why we do it. The role of forensic science in the Criminal Justice System (CJS) is being questioned. Our contribution and our future impact continue to be clarified and developed. Inevitably, changes in the demands placed upon us require changes in the way we operate. I would like to share with you some thoughts on these issues.
I would like particularly to focus on one question: 'how does a scientist's evidence affect the scales of justice?' The symbol of the scales of justice is a familiar icon but I had not previously given much thought to the implications for forensic science. If the scales are a good analogy of the process of justice, what is our contribution as forensic scientists? I believe this question goes to the very heart of our role within that system. I would like to begin by exploring the contribution we as scientists have made in the past and then discuss what contribution we should be making in the future. I must emphasise that these are my personal views, and they are not necessarily original. My own thinking on these issues has been helped a great deal by working on a project, Case Assessment and Interpretation (CAI), that has been running formally in the Forensic Science Service for two years. A small team, currently Ian Evett, Jim Lambert, Stella McCrossan and myself, has been leading the project. A main element of the project has involved the team working closely with operational colleagues on the interpretation of a wide variety of case and evidence types. It is this experience, together with my own contact with customers over the years, that has helped me to see more clearly the needs of the CJS, to work through some of the underlying issues and to begin to crystallise some views. Our work has been reported in a series of papers in Science & Justice [l-31. I am not going to consider, in this presentation, evidence
that is interpreted as conclusive: physical fits; footwear marks; toolmarks; and handwriting. I would like to concentrate on the 'greyer' areas that do present us with challenges in interpretation of the findings. Thinking firstly about how forensic science affects the scales of justice, if we view the two sides of the scales as representing, respectively, the prosecution and the defence positions, any piece of evidence, whether it be provided by a scientist, a medical expert, a police officer or a 'lay' person, will be added to one or other of the pans. The impact of the weight of the evidence will depend on the prior position of the pans. For example, it may be that the scales are heavily tilted to one side prior to the scientific evidence. The scientific evidence may simply be adding yet more weight to that side. On the other hand, if the scales are tilted to the other side, the impact of the same scientific evidence may be to counteract totally the weight of the prior evidence -the balance of the pans would be reversed. There will be other situations where, if the evidence is perceived as not helping to make progress with the issue, then the prior balance of the pans will not be affected at all. (Good has written extensively on this topic, e.g., [4]). If this is a valid view of how forensic science affects the scales, the key question then is - 'how is the weight of the scientific evidence to be evaluated?' Furthermore, do we, as a profession, have an agreed, common framework, or model, for the process of evaluation of that weight? Looking back at the contribution we have made in the past, I think it's fair to say that the history of forensic science over the last sixty years has been mainly the history of what could be called 'prosecution-commissioned' science. I don't use that phrase in any pejorative or critical way - simply to observe that the vast majority of forensic science has been carried out at the instigation of the police or prosecution. I think one consequence of this is that, perhaps inevitably, scientists working in this environment may have received sparse, or one-sided, information on the circumstances of the case. This again is not critical of the police or prosecution, but it is more of a reflection of the way the system worked. For example, in the early phase of the investigation, information may well be sparse or vague because of the lack of knowledge of what has happened, and information may well change as the investigation and subsequent prosecution progresses.
Science & Justice 2000; 40(2): 81-85 O The Forensic Science Society 2000
81
The scientist and the Scales of Justice I suspect that one main reason for scientists receiving sparse information may have been our customers believing we could answer their questions in isolation of any background information: 'Science has all the answers'. I think also that we have promoted, possibly unknowingly, an image of authority: 'Trust me, I'm a scientist, I know what you need in this case, I can give you an answer to your question'. I know some of us believe that we do not need any background information to evaluate our findings. This may well be the case when addressing certain types of question but there may be a problem with this approach when addressing other types of question. And there is a risk that we could be viewed as practising a 'black art'. I like a phrase used by Franco Taroni and Colin Aitken in 1998, 'the aura of 'mystic infallibility' associated with scientific evidence has to be carefully considered' [ 5 ] .
Another problem with these phrases is that there may be a perception of bias. It can be argued that phrases such as 'could have come from' and 'consistent with', imply a biased evaluation because the scientist appears to be considering only a prosecution view. For example, a set of findings may be 'consistent with' X having driven a stolen vehicle when, in fact, they may be 'more consistent with' an alternative scenario. Some of us will also be aware of the problems associated with use of phrases of the kind '. ..likely to have originated from.. . ', 'likely to have been in contact... ' and '...highly probable he has handled this material... '. Are these phrases still in common usage? If so, do we expose the reasoning behind the conclusion and is that reasoning robust? As evaluators of findings and as communicators, we should be aware of the potential pitfalls when using these phrases.
Turning now to the issue of 'one-sided' information, I suspect that any of us who have worked for both prosecution and defence will have experienced a difference in how we have been commissioned by the two sides. Submission forms from the police could be quite 'thin' and presented the allegations as the police saw them. It wasn't untypical to see this sort of submission twenty years ago. On the other hand, submissions from the defence generally were, and still are, a lot 'heavier', including an enormous amount of background information, usually in the form of statements. This is not to say that all of the information was useful to us - perhaps one of the most time-consuming tasks was that of reading through the information to glean the essentials like, for example, what the defendant would say. This difference in submission suggests there was a difference in the way we were being commissioned and could therefore have directed a different approach to the two types of work. I am not suggesting that the prosecution side knowingly presented us with partial, biased or edited information but maybe they were not aware of the sort of information that we needed in order to offer a fuller analysis and opinion.
Another feature of statements from the past is that of brevity. When I think back to some of the statements that I produced twenty years ago, many were no more than a few lines. Usually they were the ones that were seen as 'negative': 'I have found nothing of signzjicance on the clothing of Mr X'. I might have been a bit more expansive in some cases: 'No matching fibres have been found on the car seat tapings'; or 'Nothing of apparent significance has been found to connect the clothing of Mr Y with the scene'. But, again, that was one-sided. That left the evaluation of the findings, i.e. the absence of transferred material, to others in the CJS who, because of limited or no knowledge and understanding of that evidence type, would be more poorly placed to make the evaluation. To highlight this issue, I would like to refer to one particular high-profile murder case, dealt with by one of my colleagues, that involved a violent attack on the victim. Two people were convicted for the murder. The case became a cause ckl2bre and featured in one of the campaigning programmes on television. When I watched the programme, not knowing that there had been some forensic science involvement, I saw my colleague's name appear on an image of a CJA statement shown in the title sequence. I wondered, apprehensively, what criticism was about to be levelled at forensic science in general and at my colleague in particular. But I should never have doubted my colleague, for the issue was not about the science, it was more concerned with the other evidence in the case. Some time later, I discussed the case with my colleague. He told me that, from an examination of the scene, he had had very high expectations of finding blood on the clothing of the true assailant(s). He had, in fact, found no blood, matching the victim, on the clothing of the suspects. Instead of saying in his statement something like 'I have found nothing of significance', he had explained his thinking and his analysis of the findings. He said that, in his view, given the circumstances of the case, either the suspects were not the people who had attacked the victim or, if they were the true assailants, this was not the clothing they
Still staying in the past and exploring our contribution, I think it is informative to take a look at what we used to write in our statements. For many years, we have been using phrases such as '...could have come from.. . ', '...consistent with coming from.. . ' and '...entirelyconsistent with... '. However, there are problems with these phrases (G Craddock, P Lamb and A Moffat, personal communication 1989; and J Satterthwaite and J Lambert, personal communication, 1989). One problem was that of the range of perceptions by different people of the strength of these phrases. Taking the simple phrase 'could have' as an example, we have used that particular phrase when reporting an evidence type that had a relative frequency of 1 in 3 as well as for an evidence type that had a relative frequency of 1 in 3 million. That cannot be right.
Science & Justice 2000; 40(2): 81-85
G JACKSON
were wearing at the time. Now this may not sound earthshattering today but, in the 1980s, when this case was being examined, this was unusual reporting. I do not believe many practitioners at that time would have phrased it like that. Today, we spend a lot of time training new recruits in the evaluation and reporting of all findings irrespective of whether the findings are viewed as 'positive' or 'negative'.
A further observation on our contribution is linked to this issue. I believe our style of operation used to be what could be called 'findings-led'. We would carry out the examinations, recover material and carry out analyses. We would look at the results of the tests and try to rationalise our findings by constructing an explanatory scenario: if we did not find a lot of matching glass we would postulate that he must have been further away from the breaking window than was first thought. Or if we found no semen on a complainant's swabs, we would postulate either that he must not have ejaculated or that all the semen had been lost. I suggest that we should look critically at this process of rationalisation to ensure we are acting impartially. I suspect also that we may have been influenced, in an implicit way, by the information that was provided by the customer although many of us may have believed that we worked in isolation, were not influenced by the case circumstances and, indeed, did not need to know any background.
A hypothetical murder scene illustrates the issue:
A man leaves his wife at their home in the evening to visit a friend. He returns a few hours later to find there has been a forced entry through a rear ground floor window and, on entering the house, he finds the body of his wife lying in an upstairs room. He finds a screwdriver outside on the ground below the forced window. He cannot identify the screwdriver because it is not his, and it was not there when he left that evening. A Scenes of Crime Officer takes a cast of the mark on the forced window and submits it, along with the screwdriver, for examination. On comparing the cast with test marks made with the screwdriver, the scientist finds a very good correspondence between the marks but it is not, in the scientist's view, a conclusive match. How is this to be reported? It could be argued that, from a common sense point of view, it would be coy not to express a view that it was this screwdriver, given both the circumstances and the very good comparison results. It may seem eminently reasonable to conclude that this is the right screwdriver. However, let us assume that the screwdriver was not found at the premises, it was one of 200 screwdrivers that had been taken from a population of potential perpetrators from around the country. The police did not indicate that there was any prior evidence for any of the screwdrivers or suspects. Would the scientist be justified in still forming an opinion that it was this screwdriver? I am not implying that it is necessarily wrong to form such a conclusion but that, Science & Justice 2000; 40(2): 81-85
in the former situation, we should be explicit in the information we are taking into account when forming opinions. If we are explicit, then double-counting, by both the scientist and the non-scientist, of the evidence of the presence of the screwdriver at the scene would be avoided. And finally on statements, I think we have had a tendency to concentrate on providing analytical findings. By that I mean we reported the results of our analytical tests and we left the interpretation, or evaluation, of those results to others in the system. I believe that is a clear abrogation of our responsibilities for who else is better placed to evaluate test results? If we do not evaluate results, how do others evaluate them: what framework, what knowledge and what understanding do they have to help them evaluate the evidence in a robust, reliable way? Still on analysis, how did we handle analytical results that gave us some similarities and some differences when two samples were compared? Were we tempted to report these results as 'inconclusive', or even as 'no evidence'? For example, how would we report fibres that were a very good match on white light microscopy but gave us a worrying comparison on fluorescence? Or paint that was good on inorganic analysis but gave us slightly different infrared or PGC traces. I'm sure you can think of other examples. What did we do in those situations? Did we write them off? Did we 'fall off the cliff' and report an exclusion 'this paint fragment has not comefrom that source '? Or were we more equivocal and report 'inconclusive'? Or did we report an inclusion; 'it could have come from that source'? What would be a more robust way of evaluating those results, of dealing with both similarities and differences? Before I move away from the past, I think it is useful briefly to consider the type of cases we used to examine, particularly now viewed with the benefit of hindsight of our experiences over the last 20 years. I will use just one type of case as an example. I can recall a time when we used to examine what were called vehicle-vehicle road traffic accident cases. They could be really good cases. Considerable force was usually involved in the collision and there was a very good chance that material would have been transferred in both directions. The transferred materials were really interesting to examine, both microscopically and with analytical instruments. We could obtain good discrimination. We also had good databases on paint colours, paint layer structures and lens markings. If we were lucky, we might find a physical fit. Wonderful cases! But, were they good value for the customer? How much did it cost us to examine those cases? What was the benefit, in terms of the overall objectives of the police force, in doing those cases? What resources did we apply to them compared with examining rape, robbery and assault cases? I'm sure the owners of the 'innocent' vehicle would be delighted with the powerful scientific evidence but was it really a cost-effective service and a wise
The scientist and the Scales of Justice
use of resources? I think there is a real risk that providers of scientific services can become science and technology-led instead of being driven by the objective of meeting the customers' real needs. Furthermore, customers may become beguiled by impressive technology and 'big' numbers and think that that will solve all their problems. We have a duty to work closely with our customers to harness new developments to meet their real needs in a cost-effective way. I must not leave you with an impression that I am criticising everything from our past because I most certainly am not. We have made, I believe, a significant and rapidly increasing contribution to crime detection, and possibly crime deterrence, over the years. However, we must learn from our collective experience, continue to question critically what we do and improve, in a continuous way, our service to customers. I would like now to return to the main question and to explore how our contribution as scientists could develop. I believe that currently there is a growing recognition that, as forensic scientists, we operate at different levels within a complex Criminal Justice System. The system has distinct phases, each of which may require a different type of service from forensic scientists. We have to accommodate: - working with investigators to give them useful information prior to arrest of a suspect; - working with investigators to determine whether or not there is evidence to support charging a suspect; - working with prosecutors to evaluate the strength of the scientific evidence for or against a defendant; - working with a defence team to test the validity of scientific evidence; - working - with the courts to help them reach a verdict. We bring a scientific approach to the CJS and we know that science and the law are uneasy but necessary partners. How should we therefore support the operation of the Criminal Justice System? What should be our core values, our guiding principles as a profession? I would like to share with you what I think is a good list. It is only my list, and I'm sure you will have your own - we may agree or disagree but therein lies a healthy debate. Provide sustainable, robust evidence By that I mean the scientist should provide information and opinion that is based on sound principles of interpretation: logical; balanced; and supported by a body of data and a degree of understanding of that particular evidence type. If these criteria are met, the evidence will stand the test of time and will be less open to challenge - .years later. Be expert Following on from the first requirement, it follows that to be an 'expert' in a particular domain requires that person to be fully aware of, and understand, the body of information within that domain. The body of information should be extensive. It should be accessible, and preferably in the 84
public arena. It should be testable. An expert should have a high degree of understanding of both the principles and the data in order to evaluate the 'grey' areas of scientific findings. Be transparent By this I mean explaining in a clear and explicit way what we have done, why we have done it and how we have arrived at our conclusions. We need to expose the reasoning, the rationale, behind our work. In principle, our statements should guide our customers through the process of our evaluation. We should provide them with the information that we have used in order to lead them to draw the same conclusion as we have drawn. We should move away from practising the 'black arts'. Act impartially The scientist should look at both sides of an argument. If it is the role of a forensic scientist to test a hypothesis, and evaluate findings, then it can only be done if an alternative is also considered. This raises the question of whether scientists operate in a different manner when commissioned by the two opposing sides - and, if they do, should they? We need to consider how alternative hypotheses are formulated. In the absence of a defence version, how does the prosecution-commissioned scientist begin to formulate an alternative? For the defence-commissioned scientist, there may be difficulties in formulating plausible, alternative hypotheses. Add value By this I mean forensic science should be used in a costeffective way to help resolve specific issues in a case. There should be a focused, directed use of resources to help make progress with the matters that are of relevance and importance to the customer and for which forensic science can actually provide useful information. Wherever possible, scientists should be addressing questions that are a stage advanced beyond the simply analytical. Despite the positive changes that the impact of direct charging has made in this respect, I am sure there is still more progress to be made.
If these constitute a reasonable set of guiding principles, a useful framework for forensic science, how do we make it happen? I believe one very good way to achieve this is through scientists focusing on an evaluation of a likelihood ratio (LR) for the findings. For those of you who may not be too familiar with the concept, I would like to define briefly, in words not symbols, the likelihood ratio and the three key principles that flow from it. My working definition of the likelihood ratio in words is: The probability of the observations given that the proposition were true (and given the truth of the relevant background information) divided by the probability of the observations given that the alternative proposition were true (and given the truth of the relevant background information). Science & Justice 2000; 40(2):81-85
G JACKSON
The three principles for interpretation that flow from this are: 1. We need to be made aware of the relevant background information in each case. The background information provides a framework within which the scientist operates and conditions hislher evaluation. There have to be some basic 'givens' or assumptions and we have to state these clearly to our readers, along with a proviso that if any of these assumptions change, we need to reappraise our interpretation. 2. We need to consider at least two competing propositions or views. Frequently, the prosecution view is relatively clearly defined; the defence view, on the other hand, may be less so. To be clear about the propositions that we are addressing gives us a real focus and a clear purpose to the work that we do and allows our customers also to be clear about what it is we are trying to test. 3. We need to appraise how likely it is that we would observe the results, including both similarities and differences, if the prosecution proposition were true and if the defence view were true. This raises the issue of what knowledge and understanding do we have of that evidence type? What data is there to support interpretation? Do we need more? What research do we need to do in order to be able to interpret robustly the results of a new technique? Applying these three principles to casework, I believe meets the requirements I set out earlier. Exploring them again but in reverse order: 'Add value' A prior appraisal of, firstly, the issues in the case and, secondly, the likely magnitude of the LR, helps both the scientist and the customer arrive at a joint decision on the most cost-effective strategy. There are obvious benefits in this approach in complex cases in which there are choices of items to examine and tests to employ. There should be a clear understanding and agreement on the issues on which the scientist can actually make an impact in a case. 'Act impartially ' Clearly, the LR requires us to consider two competing propositions - I cannot think of a better way in which to demonstrate an impartial approach. 'Be transparent' We can achieve this by explaining our reasoning, our rationale for the case, not only in our statements but also in our conversations with customers and when we give evidence in court. Our customers will be able to see clearly what we have done, why we have done it and how we have arrived at our conclusion. We should set out the background information that we have taken into account and the propositions that we will be testing. We should make clear our expectations and explain the technicalities. We should explain how the findings 'fit', or don't 'fit', the expectations and we should give a bottom line. 'Provide sustainable, robust evidence and be expert'
The LR approach is certainly logical and robust. In an explicit way, the scientist is directed to challenge the Science & Justice 2000; 40(2): 81-85
available data and to use it in an expert way. If there is limited data, this should be reflected in the evaluation of the strength of the evidence. The approach not only helps guard against overstating the value of evidence but also assists obtaining maximum value from the evidence. If we have good databases, then the LR approach helps in the evaluation of both similarities and differences in analytical results and avoids the risk of 'falling-off-the-cliff'. Finally, to return to the main theme, and to answer the question of 'how does a forensic scientist affect the scales of justice?' In my view, the answer is: 'with an evaluation of a likelihood ratio'. The magnitude of the LR is a good assessment of the size of the weight - that is added to one or other of the pans of the scales. I am not implying that a scientist should provide a magic number for the value of the evidence - that would require a degree of precision to the evaluation of the LR that is just not possible in many situations. Rather, the scientist should evaluate broadly the magnitude for the LR and translate that into a verbal equivalent. We have found that the most important part of adopting an LR approach to evaluation is that of improving the clarity of thinking - we should not, in the first instance, become too focused on the 'numbers'. There is a lot of work still to do on various aspects of the LR approach, including: - explaining more fully the LR approach to our customers in the CJS; - developing with them effective ways of presenting our evidence; - agreeing a verbal scale; - applying the approach across all evidence types; - acquiring more background data where necessary; - developing applications in investigative work. Having said that, I believe there is a robustness in this approach - a logical, balanced way of appraising scientific findings. I believe it is the framework to take forensic science forward into the next century. And you may have noticed I have not used the word Bayes anywhere in this presentation! References: 1. Cook R, Evett IW, Jackson G, Jones PJ and Lambert JA. A model for case assessment and interpretation. Science & Justice 1998; 38: 151-156 2. Cook R, Evett IW, Jackson G, Jones PJ and Lambert JA. A hierarchy of propositions: deciding which to address in forensic science. Science &Justice 1998; 38: 231-239 3. Evett IW, Jackson G and Lambert JA. More on the hierarchy of propositions: exploring the distinction between explanations and propositions. Science &Justice 2000: 40: 3-10. 4. Good IJ. Weight of evidence and the Bayesian likelihood ratio. In: Aitken CGG and Stoney DA, eds. The use of statistics in forensic science. Chichester: Ellis Horwood, 1991. 5. Taroni F and Aitken CGG. Probabilistic reasoning in the law. Part 1: assessment of probabilities and explanation of the value of DNA evidence. Science & Justice 1998; 38(3): 165-177.