- Email: [email protected]
Clinical trials and RIRO
The last word
Clinical trials and RIRO John A Lee Department of Pathology, University of Sheffield Medical School, Beech Hill Road, Sheffield S10 2RX, UK
Karl Popper knew a thing or two about science and the scientific method. But, being primarily a philosopher, he was also interested in the nature of knowledge. When he was wearing that hat, he had things to say about science that deserve to be more widely known. For instance, he characterised science as ‘the art of systematic oversimplification’. As a pathologist, I know exactly what he meant. A lot of my day is spent producing diagnoses, especially for patients with malignant disease. My reports trigger specific clinical action, sometimes depending on a single word or number in the report. Many of these reports are eventually distilled into the tickboxes of clinical trial forms, supposedly designed to support the next generation of advances in treatment. I find features of this process disturbing. Many doctors now place too much faith both in randomised controlled trials (RCTs) and in the ability of pathologists to tell them ‘the diagnosis’. Let me explain. Both RCTs and histopathological diagnosis can be powerful methods for informing clinical activity, but we often forget that this achievable standard may be very different from the theoretically perfect gold standard we originally had in mind. In fact, it may be so different as to be of Well, that’s simplified you a bit! little practical use at all. Take RCTs – They are generally thought of as sources of precise, unambiguous guidance for clinical practice. The results are increasingly generalised into ‘evidence-based’ guidelines and protocols which, in turn, give their users the comfort of being seen to provide ‘best-practice’ care. Of course, the desire for certainty in clinical practice is perfectly understandable, not least because of the modern vogue for aggressive questioning of clinical decision-making. But the reality is that trial-based guidelines have proved to be just as fallible as other reasonable sources of information. After all, trials are carried out on selected, unrepresentative populations of patients. In many cases, and for obvious reasons, there is inadequate experimental control for important variables such as other diseases and treatments, which leads to considerable heterogeneity among the participating patients. And most trials are carried out by motivated, well-resourced staff, under circumstances that are not those of routine clinical practice. I could go on. Even if we leave aside the point that statistical output 128
from RCTs just doesn't translate directly into the information needed to treat an individual patient, there are at least two other important issues. First, in the comfort provided by guidelines there is a great temptation to pigeonhole patients and stop thinking. Many will recognise that situation – you know that nominal best practice has been followed, and yet you can't help feeling that the patient's best interests have not been served. Second, the mounting incentives to carry out more and better clinical trials, and write up increasingly detailed guidelines, are a great distraction from sorting out logistics and carefully applying what we already know. Most doctors feel that their patients’ interests are better served by good traditional medicine-based-on-evidence than by fashionable RCT-led evidence-based medicine. Wellconducted RCTs are one way of obtaining unbiased information about treatment effects, and they can be helpful for informing clinical decisions, but they are a systematic oversimplification. Ironically, there is still no evidence to show that evidence-based medicine is any better than careful, traditional review by experienced and knowledgeable clinicians. Pathological diagnosis makes the point perfectly. Widely perceived as a gold-standard laboratory ‘test’, the pathology report is in fact nothing more (or less) than the professional opinion of one experienced clinician. For many lesions, pathologists can all agree on the answer, but in plenty of cases, especially for malignant disease, tick-boxes don’t tell the half of it. Even expert panels regularly produce answers that fall outside any reasonable definition of consensus, not just for ‘difficult topics’ such as lymphoma or sarcoma, but also in what you may think of as obvious diagnoses, such as lobular versus ductal breast carcinoma. And then there are all the suppressed caveats and comments that the tick-boxes never reveal. We do our best, but you can see that, in fact, even the best trials can’t avoid depending absolutely on the traditional gold standard. Unfortunately, this is often just not good enough for real experimental rigour. No matter how strong the statistics, there’s no getting away from RIRO – rubbish in, rubbish out. Time to reaffirm our professional standing, rely on guidelines a little less, and start thinking clearly again about what medicine is. THE LANCET Oncology Vol 1 October 2000
For personal use only. Not to be reproduced without permission of The Lancet.
Clinical trials and RIRO John A Lee Department of Pathology, University of Sheffield Medical School, Beech Hill Road, Sheffield S10 2RX, UK
Karl Popper knew a thing or two about science and the scientific method. But, being primarily a philosopher, he was also interested in the nature of knowledge. When he was wearing that hat, he had things to say about science that deserve to be more widely known. For instance, he characterised science as ‘the art of systematic oversimplification’. As a pathologist, I know exactly what he meant. A lot of my day is spent producing diagnoses, especially for patients with malignant disease. My reports trigger specific clinical action, sometimes depending on a single word or number in the report. Many of these reports are eventually distilled into the tickboxes of clinical trial forms, supposedly designed to support the next generation of advances in treatment. I find features of this process disturbing. Many doctors now place too much faith both in randomised controlled trials (RCTs) and in the ability of pathologists to tell them ‘the diagnosis’. Let me explain. Both RCTs and histopathological diagnosis can be powerful methods for informing clinical activity, but we often forget that this achievable standard may be very different from the theoretically perfect gold standard we originally had in mind. In fact, it may be so different as to be of Well, that’s simplified you a bit! little practical use at all. Take RCTs – They are generally thought of as sources of precise, unambiguous guidance for clinical practice. The results are increasingly generalised into ‘evidence-based’ guidelines and protocols which, in turn, give their users the comfort of being seen to provide ‘best-practice’ care. Of course, the desire for certainty in clinical practice is perfectly understandable, not least because of the modern vogue for aggressive questioning of clinical decision-making. But the reality is that trial-based guidelines have proved to be just as fallible as other reasonable sources of information. After all, trials are carried out on selected, unrepresentative populations of patients. In many cases, and for obvious reasons, there is inadequate experimental control for important variables such as other diseases and treatments, which leads to considerable heterogeneity among the participating patients. And most trials are carried out by motivated, well-resourced staff, under circumstances that are not those of routine clinical practice. I could go on. Even if we leave aside the point that statistical output 128
from RCTs just doesn't translate directly into the information needed to treat an individual patient, there are at least two other important issues. First, in the comfort provided by guidelines there is a great temptation to pigeonhole patients and stop thinking. Many will recognise that situation – you know that nominal best practice has been followed, and yet you can't help feeling that the patient's best interests have not been served. Second, the mounting incentives to carry out more and better clinical trials, and write up increasingly detailed guidelines, are a great distraction from sorting out logistics and carefully applying what we already know. Most doctors feel that their patients’ interests are better served by good traditional medicine-based-on-evidence than by fashionable RCT-led evidence-based medicine. Wellconducted RCTs are one way of obtaining unbiased information about treatment effects, and they can be helpful for informing clinical decisions, but they are a systematic oversimplification. Ironically, there is still no evidence to show that evidence-based medicine is any better than careful, traditional review by experienced and knowledgeable clinicians. Pathological diagnosis makes the point perfectly. Widely perceived as a gold-standard laboratory ‘test’, the pathology report is in fact nothing more (or less) than the professional opinion of one experienced clinician. For many lesions, pathologists can all agree on the answer, but in plenty of cases, especially for malignant disease, tick-boxes don’t tell the half of it. Even expert panels regularly produce answers that fall outside any reasonable definition of consensus, not just for ‘difficult topics’ such as lymphoma or sarcoma, but also in what you may think of as obvious diagnoses, such as lobular versus ductal breast carcinoma. And then there are all the suppressed caveats and comments that the tick-boxes never reveal. We do our best, but you can see that, in fact, even the best trials can’t avoid depending absolutely on the traditional gold standard. Unfortunately, this is often just not good enough for real experimental rigour. No matter how strong the statistics, there’s no getting away from RIRO – rubbish in, rubbish out. Time to reaffirm our professional standing, rely on guidelines a little less, and start thinking clearly again about what medicine is. THE LANCET Oncology Vol 1 October 2000
For personal use only. Not to be reproduced without permission of The Lancet.