- Email: [email protected]
Allocation concealment continues to be misunderstood
Journal of Clinical Epidemiology 63 (2010) 468e470
LETTERS TO THE EDITOR Allocation concealment continues to be misunderstood
To the Editor: Foley et al. [1] lamented the fact that allocation concealment is described inadequately in two-thirds of the studies, but the flip side of this is the solace that one can take in knowing that at least allocation concealment is described adequately in one-third of the studies. Of course, Foley et al. [1] also pointed out that allocation concealment continues to be misunderstood by many investigators, and we would like to echo this sentiment. In fact, it seems to be true even more than Foley et al. [1] themselves recognize, and there is a particular irony here, a point to which we will return shortly. Foley et al. [1] considered the reporting of allocation concealment to be adequate ‘‘if the authors clearly reported a mechanism through which it could be reasonably ascertained that the investigators had no foreknowledge of the treatment assignments.’’ Now we return to the aforementioned irony. By highlighting that the lack of adequate reporting persists even in the post-CONSORT era, Foley et al. [1] correctly imply that knowledge of the relevant literature should be a prerequisite to research. But they then go on to show their own lack of knowledge of the relevant literature when they inexplicably use a definition of allocation concealment that has been shown to be badly flawed and has been improved on with a more accurate definition. This is especially problematic, because the legacy of a publication includes not only its conclusions, but also its methodology [2]. When the methodology is flawed, it sets a dangerous precedent. Hence, the record needs to be set straight, even when the right conclusions are found. In this case, the issue is what it takes to establish the concealment of future allocations. Is it enough to follow the Schulz and Grimes [3] definition? The definition used specifically does not consider the method used to generate the randomization sequence but rather concerns itself only with how well it is then guarded. This definition therefore ignores what is known about allocation concealment, specifically that there are two threats, including both (1) direct observation of the allocation sequence and (2) prediction of future allocations based on knowledge of the past ones [4] ([5]; Sections 2.5e2.6). The second mechanism requires knowledge only of the previous allocations and possibly the restrictions used to generate the
DOI of original article: 10.1016/j.jclinepi.2009.09.005. 0895-4356/10/$ e see front matter Published by Elsevier Inc.
allocation sequence (such as permuted blocks), but it most certainly does not require the direct observation of the allocation sequence. In other words, ruling out the first mechanism in no way addresses the second mechanism, and the popular definition of allocation concealment ‘‘plays on the inability of the general public (and, sadly, even researchers who should know better) to distinguish between necessity and sufficiency’’ [2]; yes, it is necessary to address the first mechanism; no, this is not sufficient when there is another one to consider too. The bottom line, then, is a picture much less optimistic than the one painted by Foley et al. [1]. In fact one-third of the trials met only an appalling low standard of pseudo allocation concealment, but it is highly doubtful that even 10% reported and/ or operated with true allocation concealment. Of course, we have no way to know for sure, because Foley et al. [1] did not enumerate the trials considered and the description of allocation concealment in each. What we do know for sure is that allocation concealment remains misunderstood even by those who would try to explain it to others. We know that the relevant literature continues to be ignored when it is convenient for authors to do so. And we know that the vast majority of trials cannot meet even so low a standard as the wink and the nod required, along with keeping a straight face, when reporting that the trial had or did or was blessed by allocation concealment. We know that in the future we all need to do better. Patients are depending on us to do so. Vance W. Berger* Biometry Research Group, National Cancer Institute Bethesda, MD, USA
Anh-Chi Do Biometry Research Group, National Cancer Institute Bethesda, MD, USA The College of New Jersey, Ewing, NJ, USA
* Corresponding author. Tel.: þ1 301 435 5303; fax: þ1 301 402 0816 E-mail address: [email protected] (V.W. Berger)
References [1] Foley NC, Zettler L, Salter KL, Bhogal SK, Teasell RW, Speechley M. In a review of stroke rehabilitation sftudies, concealed allocation was under reported. J Clin Epidemiol 2009;62:766e70. [2] Berger VW. Right conclusion, wrong method. Eur J Contracept Reprod Health Care 2009. [3] Schulz KF, Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet 2002;359:614e8.
Letters to the Editor / Journal of Clinical Epidemiology 63 (2010) 468e470 [4] Berger VW. Allocation concealment and blinding: when ignorance is bliss (letter). Med J Aust 2005;183:165. [5] Berger VW. Selection bias and covariate imbalances in randomized clinical trials. Chichester, UK: John Wiley & Sons; 2005. doi: 10.1016/j.jclinepi.2009.09.004
Concealed allocation: an under-reported and misunderstood component of trial methodology in stroke rehabilitation - reply
In reply: We would like to thank Drs. Berger and Do [1] for making us aware of Berger’s work, which includes approaches for the prevention, detection, and statistical adjustment of the effects of third-order residual selection bias because of lack of concealed allocation (CA) in randomized trials [2]. Having now acquainted ourselves with his work, we believe that our two groups probably share many values and beliefs. At a general level, we share a profound respect for the scientific method, including the necessity of observing proper methodology. More specifically, we share an interest in the problematic issue of CA in clinical trials. And finally, we would likely agree that CA is frequently misunderstood by authors and, by implication, editors and reviewers. With so much in common, it is puzzling that Berger and Do would adopt such a noncollegial tone in their letter, particularly when we seem to agree on our main pointdthat CA is adequately described in a minority of articles in stroke rehabilitation. Because some of Berger and Do’s hyperbole may itself set a ‘‘dangerous precedent,’’ we welcome this opportunity to ‘‘set the record’’ even straighter. First, we take exception to the claim that we ignored his work because it was convenient to do so. Although we plead guilty to having never heard of him, we maintain our innocence against the more serious charge of knowingly ignoring his work. Second, we disagree with the claim that we ourselves do not understand CA. As we noted in our article, ‘‘authors often failed to distinguish between the generation of the randomization schedule and the implementation of the schedule.’’ ([3]; emphasis in original). This clearly signals our understanding of the twin aspects of CA raised by Berger and Do [1]: ‘‘(1) direct observation of the allocation sequence and (2) prediction of future allocations based on knowledge of the past ones.’’ When we became interested in CA around 2004, we faced the pragmatic need for a reliable operational definition of CA (as reported by trial authors) that could be used by abstractors performing systematic reviews of large numbers of studies. It is true that Shulz and Grimes’ definition only considers whether or not CA was attempted, and the clarity with which the techniques to guard the allocation
469
sequence was reported. It does not distinguish among methods of generating the sequence that may differ in terms of the ability to predict future allocations based on knowledge of past ones. For the record, in our study, 54% of the trials that met our minimum criteria for adequately reporting CA used some form of restricted randomization procedure, thus allowing for the possibility of both prediction and subversion of future assignments. Drs. Berger and Do suggest that ‘‘it is highly doubtful that even 10% [of trials] reported and/or operated with true allocation concealment’’ [1]. Again, we are principally in agreement: if the adequate description of CA is under one third, the adequate implementation could only be lower, and possibly much lower. Following the work of Kunz and Oxman [4], however, we maintain that the major watershed in the introduction of bias is among trials where CA was attempted (using SNOSE [sequentially numbered, opaque, sealed envelopes] or central registration) and those where the trialists either misunderstood the issue or ignored it completely. So we disagree that Shulz and Grimes’ definition is ‘‘badly flawed,’’ in fact, it has the highly desired scientific properties of being an immediately usable and highly reliable measure of the necessary, if not of the sufficient [3]. Had any of the authors of the 165 articles we reviewed reported the results of the BergereExner test, we would have assiduously set about learning more about it. As it turns out, none did. We do believe that the BergereExner test has much to recommend it. However, because this test requires access to record-level data including accession numbers, restrictions, allocations, and responses [2], it does not lend itself to systematic reviews if none of the original authors used it. We do endorse independent evaluations of the BergereExner test. As systematic reviewers, we also strongly agree with Berger that journals and authors should provide more details to enable the appraisal of selection bias [2]. So although Berger has written thoughtfully for those who design, analyze, and audit RCTs, we could find no practical way to apply his insights to the systematic review of published trial results that do not report his or a similar test. The closest Berger comes to providing an operational definition of whether CA was actually maintained or not is in Chapter 3 of his book [2]. He could only definitively rule out subversion in the six articles that reported the BergereExner test, which, as pointed out above, none of our trial authors did. In the remaining 30 articles, however, his method consists of examining baseline data for systematic imbalances in covariates (i.e., prognostic indicators that are consistently more prevalent in one group than the other). Probably the major prognostic indicator of rehabilitation outcome is stroke severity, for which reason rehabilitation trialists often use the method of restriction (i.e., by studying patients at only one level of severity.) Berger’s method would not apply here as the major prognostic variable is a constant by design. Nonetheless, we agree that the examination of systematic baseline imbalances can be an
LETTERS TO THE EDITOR Allocation concealment continues to be misunderstood
To the Editor: Foley et al. [1] lamented the fact that allocation concealment is described inadequately in two-thirds of the studies, but the flip side of this is the solace that one can take in knowing that at least allocation concealment is described adequately in one-third of the studies. Of course, Foley et al. [1] also pointed out that allocation concealment continues to be misunderstood by many investigators, and we would like to echo this sentiment. In fact, it seems to be true even more than Foley et al. [1] themselves recognize, and there is a particular irony here, a point to which we will return shortly. Foley et al. [1] considered the reporting of allocation concealment to be adequate ‘‘if the authors clearly reported a mechanism through which it could be reasonably ascertained that the investigators had no foreknowledge of the treatment assignments.’’ Now we return to the aforementioned irony. By highlighting that the lack of adequate reporting persists even in the post-CONSORT era, Foley et al. [1] correctly imply that knowledge of the relevant literature should be a prerequisite to research. But they then go on to show their own lack of knowledge of the relevant literature when they inexplicably use a definition of allocation concealment that has been shown to be badly flawed and has been improved on with a more accurate definition. This is especially problematic, because the legacy of a publication includes not only its conclusions, but also its methodology [2]. When the methodology is flawed, it sets a dangerous precedent. Hence, the record needs to be set straight, even when the right conclusions are found. In this case, the issue is what it takes to establish the concealment of future allocations. Is it enough to follow the Schulz and Grimes [3] definition? The definition used specifically does not consider the method used to generate the randomization sequence but rather concerns itself only with how well it is then guarded. This definition therefore ignores what is known about allocation concealment, specifically that there are two threats, including both (1) direct observation of the allocation sequence and (2) prediction of future allocations based on knowledge of the past ones [4] ([5]; Sections 2.5e2.6). The second mechanism requires knowledge only of the previous allocations and possibly the restrictions used to generate the
DOI of original article: 10.1016/j.jclinepi.2009.09.005. 0895-4356/10/$ e see front matter Published by Elsevier Inc.
allocation sequence (such as permuted blocks), but it most certainly does not require the direct observation of the allocation sequence. In other words, ruling out the first mechanism in no way addresses the second mechanism, and the popular definition of allocation concealment ‘‘plays on the inability of the general public (and, sadly, even researchers who should know better) to distinguish between necessity and sufficiency’’ [2]; yes, it is necessary to address the first mechanism; no, this is not sufficient when there is another one to consider too. The bottom line, then, is a picture much less optimistic than the one painted by Foley et al. [1]. In fact one-third of the trials met only an appalling low standard of pseudo allocation concealment, but it is highly doubtful that even 10% reported and/ or operated with true allocation concealment. Of course, we have no way to know for sure, because Foley et al. [1] did not enumerate the trials considered and the description of allocation concealment in each. What we do know for sure is that allocation concealment remains misunderstood even by those who would try to explain it to others. We know that the relevant literature continues to be ignored when it is convenient for authors to do so. And we know that the vast majority of trials cannot meet even so low a standard as the wink and the nod required, along with keeping a straight face, when reporting that the trial had or did or was blessed by allocation concealment. We know that in the future we all need to do better. Patients are depending on us to do so. Vance W. Berger* Biometry Research Group, National Cancer Institute Bethesda, MD, USA
Anh-Chi Do Biometry Research Group, National Cancer Institute Bethesda, MD, USA The College of New Jersey, Ewing, NJ, USA
* Corresponding author. Tel.: þ1 301 435 5303; fax: þ1 301 402 0816 E-mail address: [email protected] (V.W. Berger)
References [1] Foley NC, Zettler L, Salter KL, Bhogal SK, Teasell RW, Speechley M. In a review of stroke rehabilitation sftudies, concealed allocation was under reported. J Clin Epidemiol 2009;62:766e70. [2] Berger VW. Right conclusion, wrong method. Eur J Contracept Reprod Health Care 2009. [3] Schulz KF, Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet 2002;359:614e8.
Letters to the Editor / Journal of Clinical Epidemiology 63 (2010) 468e470 [4] Berger VW. Allocation concealment and blinding: when ignorance is bliss (letter). Med J Aust 2005;183:165. [5] Berger VW. Selection bias and covariate imbalances in randomized clinical trials. Chichester, UK: John Wiley & Sons; 2005. doi: 10.1016/j.jclinepi.2009.09.004
Concealed allocation: an under-reported and misunderstood component of trial methodology in stroke rehabilitation - reply
In reply: We would like to thank Drs. Berger and Do [1] for making us aware of Berger’s work, which includes approaches for the prevention, detection, and statistical adjustment of the effects of third-order residual selection bias because of lack of concealed allocation (CA) in randomized trials [2]. Having now acquainted ourselves with his work, we believe that our two groups probably share many values and beliefs. At a general level, we share a profound respect for the scientific method, including the necessity of observing proper methodology. More specifically, we share an interest in the problematic issue of CA in clinical trials. And finally, we would likely agree that CA is frequently misunderstood by authors and, by implication, editors and reviewers. With so much in common, it is puzzling that Berger and Do would adopt such a noncollegial tone in their letter, particularly when we seem to agree on our main pointdthat CA is adequately described in a minority of articles in stroke rehabilitation. Because some of Berger and Do’s hyperbole may itself set a ‘‘dangerous precedent,’’ we welcome this opportunity to ‘‘set the record’’ even straighter. First, we take exception to the claim that we ignored his work because it was convenient to do so. Although we plead guilty to having never heard of him, we maintain our innocence against the more serious charge of knowingly ignoring his work. Second, we disagree with the claim that we ourselves do not understand CA. As we noted in our article, ‘‘authors often failed to distinguish between the generation of the randomization schedule and the implementation of the schedule.’’ ([3]; emphasis in original). This clearly signals our understanding of the twin aspects of CA raised by Berger and Do [1]: ‘‘(1) direct observation of the allocation sequence and (2) prediction of future allocations based on knowledge of the past ones.’’ When we became interested in CA around 2004, we faced the pragmatic need for a reliable operational definition of CA (as reported by trial authors) that could be used by abstractors performing systematic reviews of large numbers of studies. It is true that Shulz and Grimes’ definition only considers whether or not CA was attempted, and the clarity with which the techniques to guard the allocation
469
sequence was reported. It does not distinguish among methods of generating the sequence that may differ in terms of the ability to predict future allocations based on knowledge of past ones. For the record, in our study, 54% of the trials that met our minimum criteria for adequately reporting CA used some form of restricted randomization procedure, thus allowing for the possibility of both prediction and subversion of future assignments. Drs. Berger and Do suggest that ‘‘it is highly doubtful that even 10% [of trials] reported and/or operated with true allocation concealment’’ [1]. Again, we are principally in agreement: if the adequate description of CA is under one third, the adequate implementation could only be lower, and possibly much lower. Following the work of Kunz and Oxman [4], however, we maintain that the major watershed in the introduction of bias is among trials where CA was attempted (using SNOSE [sequentially numbered, opaque, sealed envelopes] or central registration) and those where the trialists either misunderstood the issue or ignored it completely. So we disagree that Shulz and Grimes’ definition is ‘‘badly flawed,’’ in fact, it has the highly desired scientific properties of being an immediately usable and highly reliable measure of the necessary, if not of the sufficient [3]. Had any of the authors of the 165 articles we reviewed reported the results of the BergereExner test, we would have assiduously set about learning more about it. As it turns out, none did. We do believe that the BergereExner test has much to recommend it. However, because this test requires access to record-level data including accession numbers, restrictions, allocations, and responses [2], it does not lend itself to systematic reviews if none of the original authors used it. We do endorse independent evaluations of the BergereExner test. As systematic reviewers, we also strongly agree with Berger that journals and authors should provide more details to enable the appraisal of selection bias [2]. So although Berger has written thoughtfully for those who design, analyze, and audit RCTs, we could find no practical way to apply his insights to the systematic review of published trial results that do not report his or a similar test. The closest Berger comes to providing an operational definition of whether CA was actually maintained or not is in Chapter 3 of his book [2]. He could only definitively rule out subversion in the six articles that reported the BergereExner test, which, as pointed out above, none of our trial authors did. In the remaining 30 articles, however, his method consists of examining baseline data for systematic imbalances in covariates (i.e., prognostic indicators that are consistently more prevalent in one group than the other). Probably the major prognostic indicator of rehabilitation outcome is stroke severity, for which reason rehabilitation trialists often use the method of restriction (i.e., by studying patients at only one level of severity.) Berger’s method would not apply here as the major prognostic variable is a constant by design. Nonetheless, we agree that the examination of systematic baseline imbalances can be an