- Email: [email protected]
Evaluating complementary therapies for use in the National Health Service: ‘Horses for courses’. Part 2: Alternative research strategies
Complementary Therapiesin Medicine (1997)5, 94-98
© PearsonProfessionalLtd 1997
Evaluating complementary therapies for use in the National Health Service: 'Horses for courses'. Part 2: Alternative research strategies K. J. Thomas, M. J. F i t t e r Medical Care Research Unit, University of Sheffield and Foundation for Traditional Chinese Medicine, York, UK SUMMARY. The classical randomized controlled trial (RCT) is still held up by many researchers as the gold standard design for evaluating the effectiveness of a clinical intervention, despite its limitations. This has the consequence of polarizing the complementary medicine research community into those who wish to force research into this model, and those who reject the RCT as impractical. In this, the second of two papers which assess the usefulness of specific research designs for evaluating the provision of complementary therapies in the National Health Service (NHS), we describe two alternative RCT designs: the partial randomization (randomized block) design; and a pragmatic design with prior randomization. The strengths and weaknesses of each design are discussed. We conclude that they should be considered as serious alternatives to the 'old favourite' in the 'race' to evaluate the potential contribution of complementary therapies to the NHS.
PARTIAL RANDOMIZATION (RANDOMIZED BLOCK) DESIGN
INTRODUCTION Our aim in this paper is to introduce practical research designs that may be appropriate for evaluating the potential benefit of introducing complementary therapies within the British National Health Service (NHS). In our previous paper, 1we explored a number of substantial problems with using what we described as a 'classical RCT' design, i.e. a randomized controlled clinical trial, for evaluating complex interventions, and identified a set of design challenges that arise from these problems. We wish to retain the element of randomization in the research design. However, we wish to use randomization in a way that maximizes the proportion of the population that is included in the evaluation and builds on patients' preferences and choices as much as possible. In addition, we wish to leave practitioners free to give individualized treatments as appropriate. In effect, we want to evaluate the therapy delivered under as near to 'normal' service conditions as is possible. This paper describes two existing trial designs that appear to satisfy these challenges, and we will argue tlaat they should be considered seriously for evaluating the use of complementary therapies within the NHS.
The first approach that we wish to consider is partial randomization or randomized block design. This derives from a methodology developed by Brewin & Bradley2 and is designed explicitly to take patient preferences into account (Fig. 1). Following recruitment, patients are asked if they have a clear preference for one of the two treatments (or services) on offer. If they do, they are assigned to it (groups A and D). If not, they are randomly assigned to either the new service (group B) or the normal service (group C). This design meets the design challenge to maximize the proportion of population included, as well as that of building on patient preferences and choice. However, rather than providing a simple comparison of the 'new' with 'normal' service, it provides data to address two separate research questions. Firstly, the partial randomization design can begin to unravel the effect of patient preferences on outcome. For the new service, this is a comparison of outcomes for groups A vs B. For normal service, it is a comparison of outcomes for groups C vs D. However, preference could be associated with other confounding variables, such as gender, social class or education, which also may be predictive of outcome. Secondly, the partial randomization design can provide data on the comparative effectiveness of the two services: by comparing outcomes for groups A vs
Kate Thomas, S C H A R R , Medical Care Research Unit, University o f Sheffield, Regent Court, 30 Regent Street, Sheffield S1 4DA, U K . Mike Fitter, Foundation for Traditional Chinese Medicine, 124 A c o m b Road, York YO2 4EY, U K . 94
Evaluating complementary therapies for use in the NHS: Part 2
S
ncesrltl \
Prefer new service
No preference Randomize
I I e"
service
II I
service
Normal service
Prefer normal service
I EFFECT OF PREFERENCE = A vs B & C vs D
95
~
Normal service
@
I
COMPARATIVE EFFECTIVENESS = A vs D & B vs C ]
Fig. 1 Partial randomizationdesign.
D, one can evaluate comparative effectiveness when patients do have a preference. By comparing outcomes for groups B vs C, one can evaluate comparative effectiveness when patients do not have a preference.
A PRAGMATIC DESIGN WITH PRIOR RANDOMIZATION
In a seminal paper, Schwartz & Lellouch 3 outlined a distinction between 'explanatory' and 'pragmatic' clinical trials. Their motivation, as is ours, was to clarify a confusion in the literature between two types of trial. They distinguished between the trial that evaluates treatment efficacy by comparing two groups which are alike in all respects other than the presence or absence of a single 'active intervention' (i.e. the classical RCT, also called the 'explanatory' trial because it aims to provide evidence to help explain the mechanisms of action) and the trial that evaluates treatment effectiveness by separately optimizing the conditions under which each treatment is provided (called the 'pragmatic' trial because it aims to produce clinically optimal results in each treatment arm rather than make each arm identical in all respects except for the active intervention). A crucial distinction between the approaches is that the explanatory trial takes place under 'laboratory' conditions to aid scientific understanding, and the pragmatic trial takes place under 'normal' conditions (i.e. treatment as provided by a competent clinician) to aid decisionmaking. The pragmatic trial design is therefore the appropriate design for evaluating the effectiveness of an in situ service. In the Medical Research Council chiropractic trial, Meade and his colleagues used a pragmatic approach, with each patient's treatment being decided at the discretion of the chiropractor or hospital team. Their stated rationale for not under-
taking a more proscriptive or 'fastidious' trial supports our arguments, 'the effectiveness of treatment in day-to-day practice is of most interest to patients as well as doctors and therapists'? We need to introduce one further idea before we can outline the second design option. This idea was developed by Zelen 5 in the USA as a response to ethical objections that had been raised to a classical RCT. Essentially, his proposal was that the target population of patients should be randomized prior to recruitment. This is in contrast to the conventional design in which patients are recruited first and are then randomized to one or other treatment arms. Zelen's aim was for patients to know what they were consenting to at the point at which they gave consent. We believe this design is particularly suited to the evaluation of a new service that is being introduced on a trial basis. The design is illustrated in Figure 2. As with the previous design, this approach meets the design challenges of maximizing the proportion of the population included and building on patient preference. It may be used to compare a new service with what would normally be offered to the patient in the target group if the new service were not available. The target group of patients for the trial is identified by an agreed set of criteria. Once identified, patients are randomly assigned to a service arm (new service or normal service). The patients in the group assigned to the new service are told that a research trial of the new service is taking place to learn if it is effective and to aid decision-making as to whether it should be routinely available. They may choose to accept care via the new service or to decline and receive the normal service. The other group is informed that a study is taking place to examine the effectiveness of current treatment for their condition. They are invited to participate and to provide certain information to help in the assessment (some of which
96
Complementary Therapies in Medicine
Offer newservice
Identify groups
~
service
I
Randomize
Monitor normal service ] NN~
Normal@ service
UPTAKE = Avs A+B J
I
I COMPARATIVE EFFECTIVENESS = A+B vs C I
Fig. 2 Pragmaticdesignwith prior randomization.
may be over and above what would be recorded in routine care). This design therefore results in three groups of patients: those who accept the offer of the new service (group A); those who decline the offer and receive normal management (group B); and those who are simply monitored receiving normal management (group C). The design provides useful information relating to two separate aspects of the new service. Firstly, it provides information on the likely uptake. This is simply the proportion of patients who accept the offer (A) divided by the total number offered the new service (A+B). If the uptake is low, this may be regarded as evidence against introducing the new service. If the uptake is very high, supply issues may need to be considered. Secondly, it provides information on the effectiveness of the new service in comparison with normal management. This comparative effectiveness is determined by comparing the outcomes for those offered the new service (A+B) with those not offered it (C). It may appear that one should be comparing groups A and C, but this would be wrong. In a pragmatic trial, we are evaluating the benefits of offering the new service to the whole population, not just those who choose to use it. These results will give purchasers the best information to make the decision as to whether it should be offered as part of routine care.
STRENGTHS AND WEAKNESSES
A potential weakness of partial randomization (Fig. 1) is that it does not provide a fully pragmatic comparison of two services, but only a comparison under the conditions of preference or no preference, i.e. it is not possible to combine groups A and B (or C and D) and equate this with the service as it would be provided outside the trial conditions. For this reason,
it is not strictly a pragmatic trial. If no patients express a preference and groups A and D reduce to zero, then the design reduces to a classical RCT. If all patients have a preference (and groups B and C reduce to zero) then the design reduces to a simple comparison of two services when patients make their own choice (a comparative study rather than an RCT since there is no randomization). Thus the viability of this design depends on a sufficient number of patients having a preference and a sufficient number of patients having no preference. This cannot be guaranteed and would clearly need assessing in a feasibility study. We have a concern that this design requires sufficient patients to have no preference. With respect to complementary therapies, it may be that with adequate information, the vast majority of patients would have a preference - and that it is 'healthy' for the patient to have a preference. However, a strength of partial randomization is that it appears to be particularly useful for evaluating the comparative effectiveness of two existing health care services. That is, in a situation where two therapies are each potential treatments of choice, both are available, and some patients are likely to have a strong preference for one or the other. The design has been used recently in a clinical trial comparing the effectiveness of medical abortion with surgical vacuum aspiration. 6 The feasibility study showed that there would be approximately equal numbers of women in each of the four arms of the trial. Half the women expressed a preference and half did not. One of the benefits of the study was the provision of information to aid those women who expressed no initial preference to make a choice. In contrast, a pragmatic trial with prior randomization (Fig. 2) seems particularly appropriate for evaluating a new service that is under consideration but is not yet routinely available. Based on the results of a feasibility study, a pilot service can be set up to
Evaluating complementary therapies for use in the NHS: Part 2 provide for the number of patients who are likely to accept the offer if they are randomly assigned to the 'offer new service' group. Those who are not offered the new service would continue to receive the normal service. A potential weakness of this design is that the process of monitoring normal service may in some way influence the outcome, particularly if, for ethical reasons, patients are told that a trial is taking place of a new service that is not (at this stage) available to them. Greater recruitment difficulties may therefore be anticipated for this arm of the trial. The decision as to whether or not to inform patients who will receive the normal service that there is a trial of a new service taking place is an important one, for which there are no easy answers. Strictly speaking, since they are receiving the normal service and are not affected by the trial, other than possibly by increased monitoring, they do not need to know. Ethically speaking, however, it may be preferable to inform them that the data they provide will contribute to the evaluation of the new service, but that such a service is not on offer to them at this stage. Indeed, the Royal College of Physicians recommends that, in a trial with post-randomization consent, patients should normally be informed about all the treatments being evaluated2 In his paper, Zelen5 considers the appropriate power statistics that enable a researcher to determine the necessary sample sizes for the study. These are different from those which apply to a conventional design, because of the different method of randomization. He concludes that this new design may be more efficient (and therefore require a smaller study population) than the conventional randomization design if the proportion of patients who agree to participate is higher than the proportion in the conventional design, where several may choose not to be randomly assigned a treatment.
DISCUSSION Our purpose in writing this paper was to explore and describe specific research designs available for evaluating the provision of complementary therapies in the NHS. The classical RCT, as we have described it, is still held up as the gold standard design by many researchers, despite its limitations. This has the consequence of polarizing the complementary medicine research community into those who wish to force research into this model, and those who reject the RCT as impractical. This creates problems for those wishing to undertake evaluative work who feel strongly that a classical RCT is not appropriate for evaluating complementary therapies, and who feel doomed to have their findings disregarded if they do not adhere to an overly restrictive set of methodological assumptions, but who wish to draw upon the strengths of other types of RCT design. Having
97
discussed some key problems associated with a classical RCT design and having developed a series of design preferences, we sought to apply these considerations by describing two alternative RCT designs which combine methodological rigour with a concern to take into account patient preferences. It is now necessary to consider the potential usefulness of these two designs for evaluating the effectiveness of a complex (e.g. complementary) therapy, provided under normal service conditions. The first important advantage of these pragmatic designs is that they provide the practitioners with the opportunity to offer what they consider to be the best clinical practice, individualized for each patient. This is in contrast with an explanatory design which requires that a predetermined treatment, or range of treatments, be specified. Although not unique to complementary therapies, this freedom to treat each patient uniquely and to vary treatment over time is particularly important for most of its practitioners. Secondly, these designs avoid the problem of finding suitable control group conditions, since this is provided by the presence of the normal clinical management arm of the trial. Once again, normal clinical management does not need to be identical for all patients. Each practitioner who is recruiting patients into the trial makes his or her own decision, in consultation with each patient and on the basis of current accepted practice, as to what will be offered to patients randomized into this arm. Since practitioner and patient blinding are not required to address the 'problem' of patient preference bias, this presents no impediment to the relationship, another consideration of particular importance to complementary therapists. In essence, the pragmatic design draws a boundary around the service and only requires control of the conditions of entry and exit. That is, randomization with explicit inclusion and exclusion criteria is important for recruitment, and outcome measures are important prior to and after a course of treatment. However, there is no attempt to manipulate conditions inside the boundary. For this reason, such designs are sometimes known as 'black-box' trials? We have identified some concerns about the generalizability of findings from a classical RCT to actual clinical practice. In pragmatic trials, the generalizability of findings raises a different issue. The design is well suited to application within normal clinical practice since, as nearly as possible, it evaluates just this. However, if the new service does appear to be more cost-effective, because pragmatic trials do not identify a specific active intervention that constitutes the explanation for any effect that is found, there can be doubt about what it is that works. But the key issue is reproducibility. It must be possible to reproduce a positive result elsewhere if the trial is to be of any value. To enable this, a comprehensive description of the service must be available. The study will also require several
98
Complementary Therapies in Medicine
practitioners to be involved in the delivery of care if it is to be considered a service evaluation. The purpose of this paper is not to go into details on collecting appropriate information and measuring outcomes. However, the study should include both clinical and economic measures, to enable costeffectiveness to be calculated and to assess whether the new service is safe. The issues discussed in this paper are not specific to the debate about evaluation in complementary medicine: they are current in health services research, but the issues are particularly pertinent to complementary medicine. The pragmatic evaluation of a service provides a research design which leaves the practitioner free to give the treatment of choice on a case-by-case basis, and is conducive to a comprehensive and holistic evaluation of outcomes. In addition, this approach does not require a separation of specific and non-specific effects of treatment and it can readily accommodate the importance of the therapeutic relationship. Finally, entering this methodological debate places research in complementary medicine into the mainstream of contemporary health services research. 9,1° In the 'race' to evaluate the potential contribution of complementary therapies to the NHS, it is important that complementary medicine researchers do not adopt a restrictive understanding of the RCT. The consequence of restrictive thinking is either that the RCT is rejected as inappropriate or that researchers remain blinkered by the 'old favourite' and run the risk of backing the wrong 'horse.' If, however, the contemporary and evolving character of the RCT is
recognized, it can be harnessed for the benefit of complementary medicine research.
ACKNOWLEDGEMENTS This paper is based on a presentation to the Second Annual Symposium on Complementary Health Care, University of Exeter, December 1995. The development of the ideas has benefited from discussions with several people. We would particularly like to mention the contributions of Mark Lankshear, Hugh MacPherson and Jon Nicholl.
REFERENCES
1. Fitter MJ, Thomas KJ. Evaluating complementary therapies for use in the NHS: 'horses for courses'. Part I: the design challenge. Compl Ther Med 1997; 5: 88-91. 2. Brewin C, Bradley C. Patient preferences and randomised clinical trials. BMJ 1989; 299: 313-315. 3. Swartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutic trials. J Chron Dis 1967; 20: 637-648. 4. Meade T, Dyer S, Browne W, Townsend J, Frank A. Low back pain of mechanical origin: randomised comparison of chiropractic and hospital outpatient treatment. BMJ 1990; 300: 1431-1437. 5. Zelen M. A new design for randomized clinical trials. N Engl J Med 1979; 300: 1242-1245. 6. Henshaw R, Naji S, Russell I, Templeton A. Comparison of medical abortion with surgical vacuum aspiration: women's preferences and acceptability of treatment. BMJ 1993; 307: 714-717. 7. Royal College of Physicians. Guidelines on the practice of ethics committees in medical research involving human subjects 2nd edn. London: Royal College of Physicians, 1990. 8. St George D. Towards a research and development strategy for complementary medicine. Homoeopath 1994; 54: 254-256. 9. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996; 312: 1215-1218. 10. Pringle M, Churchill R. Randomised controlled trials in general practice; gold standard or fool's gold? BMJ 1995; 311: i382-1383.
© PearsonProfessionalLtd 1997
Evaluating complementary therapies for use in the National Health Service: 'Horses for courses'. Part 2: Alternative research strategies K. J. Thomas, M. J. F i t t e r Medical Care Research Unit, University of Sheffield and Foundation for Traditional Chinese Medicine, York, UK SUMMARY. The classical randomized controlled trial (RCT) is still held up by many researchers as the gold standard design for evaluating the effectiveness of a clinical intervention, despite its limitations. This has the consequence of polarizing the complementary medicine research community into those who wish to force research into this model, and those who reject the RCT as impractical. In this, the second of two papers which assess the usefulness of specific research designs for evaluating the provision of complementary therapies in the National Health Service (NHS), we describe two alternative RCT designs: the partial randomization (randomized block) design; and a pragmatic design with prior randomization. The strengths and weaknesses of each design are discussed. We conclude that they should be considered as serious alternatives to the 'old favourite' in the 'race' to evaluate the potential contribution of complementary therapies to the NHS.
PARTIAL RANDOMIZATION (RANDOMIZED BLOCK) DESIGN
INTRODUCTION Our aim in this paper is to introduce practical research designs that may be appropriate for evaluating the potential benefit of introducing complementary therapies within the British National Health Service (NHS). In our previous paper, 1we explored a number of substantial problems with using what we described as a 'classical RCT' design, i.e. a randomized controlled clinical trial, for evaluating complex interventions, and identified a set of design challenges that arise from these problems. We wish to retain the element of randomization in the research design. However, we wish to use randomization in a way that maximizes the proportion of the population that is included in the evaluation and builds on patients' preferences and choices as much as possible. In addition, we wish to leave practitioners free to give individualized treatments as appropriate. In effect, we want to evaluate the therapy delivered under as near to 'normal' service conditions as is possible. This paper describes two existing trial designs that appear to satisfy these challenges, and we will argue tlaat they should be considered seriously for evaluating the use of complementary therapies within the NHS.
The first approach that we wish to consider is partial randomization or randomized block design. This derives from a methodology developed by Brewin & Bradley2 and is designed explicitly to take patient preferences into account (Fig. 1). Following recruitment, patients are asked if they have a clear preference for one of the two treatments (or services) on offer. If they do, they are assigned to it (groups A and D). If not, they are randomly assigned to either the new service (group B) or the normal service (group C). This design meets the design challenge to maximize the proportion of population included, as well as that of building on patient preferences and choice. However, rather than providing a simple comparison of the 'new' with 'normal' service, it provides data to address two separate research questions. Firstly, the partial randomization design can begin to unravel the effect of patient preferences on outcome. For the new service, this is a comparison of outcomes for groups A vs B. For normal service, it is a comparison of outcomes for groups C vs D. However, preference could be associated with other confounding variables, such as gender, social class or education, which also may be predictive of outcome. Secondly, the partial randomization design can provide data on the comparative effectiveness of the two services: by comparing outcomes for groups A vs
Kate Thomas, S C H A R R , Medical Care Research Unit, University o f Sheffield, Regent Court, 30 Regent Street, Sheffield S1 4DA, U K . Mike Fitter, Foundation for Traditional Chinese Medicine, 124 A c o m b Road, York YO2 4EY, U K . 94
Evaluating complementary therapies for use in the NHS: Part 2
S
ncesrltl \
Prefer new service
No preference Randomize
I I e"
service
II I
service
Normal service
Prefer normal service
I EFFECT OF PREFERENCE = A vs B & C vs D
95
~
Normal service
@
I
COMPARATIVE EFFECTIVENESS = A vs D & B vs C ]
Fig. 1 Partial randomizationdesign.
D, one can evaluate comparative effectiveness when patients do have a preference. By comparing outcomes for groups B vs C, one can evaluate comparative effectiveness when patients do not have a preference.
A PRAGMATIC DESIGN WITH PRIOR RANDOMIZATION
In a seminal paper, Schwartz & Lellouch 3 outlined a distinction between 'explanatory' and 'pragmatic' clinical trials. Their motivation, as is ours, was to clarify a confusion in the literature between two types of trial. They distinguished between the trial that evaluates treatment efficacy by comparing two groups which are alike in all respects other than the presence or absence of a single 'active intervention' (i.e. the classical RCT, also called the 'explanatory' trial because it aims to provide evidence to help explain the mechanisms of action) and the trial that evaluates treatment effectiveness by separately optimizing the conditions under which each treatment is provided (called the 'pragmatic' trial because it aims to produce clinically optimal results in each treatment arm rather than make each arm identical in all respects except for the active intervention). A crucial distinction between the approaches is that the explanatory trial takes place under 'laboratory' conditions to aid scientific understanding, and the pragmatic trial takes place under 'normal' conditions (i.e. treatment as provided by a competent clinician) to aid decisionmaking. The pragmatic trial design is therefore the appropriate design for evaluating the effectiveness of an in situ service. In the Medical Research Council chiropractic trial, Meade and his colleagues used a pragmatic approach, with each patient's treatment being decided at the discretion of the chiropractor or hospital team. Their stated rationale for not under-
taking a more proscriptive or 'fastidious' trial supports our arguments, 'the effectiveness of treatment in day-to-day practice is of most interest to patients as well as doctors and therapists'? We need to introduce one further idea before we can outline the second design option. This idea was developed by Zelen 5 in the USA as a response to ethical objections that had been raised to a classical RCT. Essentially, his proposal was that the target population of patients should be randomized prior to recruitment. This is in contrast to the conventional design in which patients are recruited first and are then randomized to one or other treatment arms. Zelen's aim was for patients to know what they were consenting to at the point at which they gave consent. We believe this design is particularly suited to the evaluation of a new service that is being introduced on a trial basis. The design is illustrated in Figure 2. As with the previous design, this approach meets the design challenges of maximizing the proportion of the population included and building on patient preference. It may be used to compare a new service with what would normally be offered to the patient in the target group if the new service were not available. The target group of patients for the trial is identified by an agreed set of criteria. Once identified, patients are randomly assigned to a service arm (new service or normal service). The patients in the group assigned to the new service are told that a research trial of the new service is taking place to learn if it is effective and to aid decision-making as to whether it should be routinely available. They may choose to accept care via the new service or to decline and receive the normal service. The other group is informed that a study is taking place to examine the effectiveness of current treatment for their condition. They are invited to participate and to provide certain information to help in the assessment (some of which
96
Complementary Therapies in Medicine
Offer newservice
Identify groups
~
service
I
Randomize
Monitor normal service ] NN~
Normal@ service
UPTAKE = Avs A+B J
I
I COMPARATIVE EFFECTIVENESS = A+B vs C I
Fig. 2 Pragmaticdesignwith prior randomization.
may be over and above what would be recorded in routine care). This design therefore results in three groups of patients: those who accept the offer of the new service (group A); those who decline the offer and receive normal management (group B); and those who are simply monitored receiving normal management (group C). The design provides useful information relating to two separate aspects of the new service. Firstly, it provides information on the likely uptake. This is simply the proportion of patients who accept the offer (A) divided by the total number offered the new service (A+B). If the uptake is low, this may be regarded as evidence against introducing the new service. If the uptake is very high, supply issues may need to be considered. Secondly, it provides information on the effectiveness of the new service in comparison with normal management. This comparative effectiveness is determined by comparing the outcomes for those offered the new service (A+B) with those not offered it (C). It may appear that one should be comparing groups A and C, but this would be wrong. In a pragmatic trial, we are evaluating the benefits of offering the new service to the whole population, not just those who choose to use it. These results will give purchasers the best information to make the decision as to whether it should be offered as part of routine care.
STRENGTHS AND WEAKNESSES
A potential weakness of partial randomization (Fig. 1) is that it does not provide a fully pragmatic comparison of two services, but only a comparison under the conditions of preference or no preference, i.e. it is not possible to combine groups A and B (or C and D) and equate this with the service as it would be provided outside the trial conditions. For this reason,
it is not strictly a pragmatic trial. If no patients express a preference and groups A and D reduce to zero, then the design reduces to a classical RCT. If all patients have a preference (and groups B and C reduce to zero) then the design reduces to a simple comparison of two services when patients make their own choice (a comparative study rather than an RCT since there is no randomization). Thus the viability of this design depends on a sufficient number of patients having a preference and a sufficient number of patients having no preference. This cannot be guaranteed and would clearly need assessing in a feasibility study. We have a concern that this design requires sufficient patients to have no preference. With respect to complementary therapies, it may be that with adequate information, the vast majority of patients would have a preference - and that it is 'healthy' for the patient to have a preference. However, a strength of partial randomization is that it appears to be particularly useful for evaluating the comparative effectiveness of two existing health care services. That is, in a situation where two therapies are each potential treatments of choice, both are available, and some patients are likely to have a strong preference for one or the other. The design has been used recently in a clinical trial comparing the effectiveness of medical abortion with surgical vacuum aspiration. 6 The feasibility study showed that there would be approximately equal numbers of women in each of the four arms of the trial. Half the women expressed a preference and half did not. One of the benefits of the study was the provision of information to aid those women who expressed no initial preference to make a choice. In contrast, a pragmatic trial with prior randomization (Fig. 2) seems particularly appropriate for evaluating a new service that is under consideration but is not yet routinely available. Based on the results of a feasibility study, a pilot service can be set up to
Evaluating complementary therapies for use in the NHS: Part 2 provide for the number of patients who are likely to accept the offer if they are randomly assigned to the 'offer new service' group. Those who are not offered the new service would continue to receive the normal service. A potential weakness of this design is that the process of monitoring normal service may in some way influence the outcome, particularly if, for ethical reasons, patients are told that a trial is taking place of a new service that is not (at this stage) available to them. Greater recruitment difficulties may therefore be anticipated for this arm of the trial. The decision as to whether or not to inform patients who will receive the normal service that there is a trial of a new service taking place is an important one, for which there are no easy answers. Strictly speaking, since they are receiving the normal service and are not affected by the trial, other than possibly by increased monitoring, they do not need to know. Ethically speaking, however, it may be preferable to inform them that the data they provide will contribute to the evaluation of the new service, but that such a service is not on offer to them at this stage. Indeed, the Royal College of Physicians recommends that, in a trial with post-randomization consent, patients should normally be informed about all the treatments being evaluated2 In his paper, Zelen5 considers the appropriate power statistics that enable a researcher to determine the necessary sample sizes for the study. These are different from those which apply to a conventional design, because of the different method of randomization. He concludes that this new design may be more efficient (and therefore require a smaller study population) than the conventional randomization design if the proportion of patients who agree to participate is higher than the proportion in the conventional design, where several may choose not to be randomly assigned a treatment.
DISCUSSION Our purpose in writing this paper was to explore and describe specific research designs available for evaluating the provision of complementary therapies in the NHS. The classical RCT, as we have described it, is still held up as the gold standard design by many researchers, despite its limitations. This has the consequence of polarizing the complementary medicine research community into those who wish to force research into this model, and those who reject the RCT as impractical. This creates problems for those wishing to undertake evaluative work who feel strongly that a classical RCT is not appropriate for evaluating complementary therapies, and who feel doomed to have their findings disregarded if they do not adhere to an overly restrictive set of methodological assumptions, but who wish to draw upon the strengths of other types of RCT design. Having
97
discussed some key problems associated with a classical RCT design and having developed a series of design preferences, we sought to apply these considerations by describing two alternative RCT designs which combine methodological rigour with a concern to take into account patient preferences. It is now necessary to consider the potential usefulness of these two designs for evaluating the effectiveness of a complex (e.g. complementary) therapy, provided under normal service conditions. The first important advantage of these pragmatic designs is that they provide the practitioners with the opportunity to offer what they consider to be the best clinical practice, individualized for each patient. This is in contrast with an explanatory design which requires that a predetermined treatment, or range of treatments, be specified. Although not unique to complementary therapies, this freedom to treat each patient uniquely and to vary treatment over time is particularly important for most of its practitioners. Secondly, these designs avoid the problem of finding suitable control group conditions, since this is provided by the presence of the normal clinical management arm of the trial. Once again, normal clinical management does not need to be identical for all patients. Each practitioner who is recruiting patients into the trial makes his or her own decision, in consultation with each patient and on the basis of current accepted practice, as to what will be offered to patients randomized into this arm. Since practitioner and patient blinding are not required to address the 'problem' of patient preference bias, this presents no impediment to the relationship, another consideration of particular importance to complementary therapists. In essence, the pragmatic design draws a boundary around the service and only requires control of the conditions of entry and exit. That is, randomization with explicit inclusion and exclusion criteria is important for recruitment, and outcome measures are important prior to and after a course of treatment. However, there is no attempt to manipulate conditions inside the boundary. For this reason, such designs are sometimes known as 'black-box' trials? We have identified some concerns about the generalizability of findings from a classical RCT to actual clinical practice. In pragmatic trials, the generalizability of findings raises a different issue. The design is well suited to application within normal clinical practice since, as nearly as possible, it evaluates just this. However, if the new service does appear to be more cost-effective, because pragmatic trials do not identify a specific active intervention that constitutes the explanation for any effect that is found, there can be doubt about what it is that works. But the key issue is reproducibility. It must be possible to reproduce a positive result elsewhere if the trial is to be of any value. To enable this, a comprehensive description of the service must be available. The study will also require several
98
Complementary Therapies in Medicine
practitioners to be involved in the delivery of care if it is to be considered a service evaluation. The purpose of this paper is not to go into details on collecting appropriate information and measuring outcomes. However, the study should include both clinical and economic measures, to enable costeffectiveness to be calculated and to assess whether the new service is safe. The issues discussed in this paper are not specific to the debate about evaluation in complementary medicine: they are current in health services research, but the issues are particularly pertinent to complementary medicine. The pragmatic evaluation of a service provides a research design which leaves the practitioner free to give the treatment of choice on a case-by-case basis, and is conducive to a comprehensive and holistic evaluation of outcomes. In addition, this approach does not require a separation of specific and non-specific effects of treatment and it can readily accommodate the importance of the therapeutic relationship. Finally, entering this methodological debate places research in complementary medicine into the mainstream of contemporary health services research. 9,1° In the 'race' to evaluate the potential contribution of complementary therapies to the NHS, it is important that complementary medicine researchers do not adopt a restrictive understanding of the RCT. The consequence of restrictive thinking is either that the RCT is rejected as inappropriate or that researchers remain blinkered by the 'old favourite' and run the risk of backing the wrong 'horse.' If, however, the contemporary and evolving character of the RCT is
recognized, it can be harnessed for the benefit of complementary medicine research.
ACKNOWLEDGEMENTS This paper is based on a presentation to the Second Annual Symposium on Complementary Health Care, University of Exeter, December 1995. The development of the ideas has benefited from discussions with several people. We would particularly like to mention the contributions of Mark Lankshear, Hugh MacPherson and Jon Nicholl.
REFERENCES
1. Fitter MJ, Thomas KJ. Evaluating complementary therapies for use in the NHS: 'horses for courses'. Part I: the design challenge. Compl Ther Med 1997; 5: 88-91. 2. Brewin C, Bradley C. Patient preferences and randomised clinical trials. BMJ 1989; 299: 313-315. 3. Swartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutic trials. J Chron Dis 1967; 20: 637-648. 4. Meade T, Dyer S, Browne W, Townsend J, Frank A. Low back pain of mechanical origin: randomised comparison of chiropractic and hospital outpatient treatment. BMJ 1990; 300: 1431-1437. 5. Zelen M. A new design for randomized clinical trials. N Engl J Med 1979; 300: 1242-1245. 6. Henshaw R, Naji S, Russell I, Templeton A. Comparison of medical abortion with surgical vacuum aspiration: women's preferences and acceptability of treatment. BMJ 1993; 307: 714-717. 7. Royal College of Physicians. Guidelines on the practice of ethics committees in medical research involving human subjects 2nd edn. London: Royal College of Physicians, 1990. 8. St George D. Towards a research and development strategy for complementary medicine. Homoeopath 1994; 54: 254-256. 9. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996; 312: 1215-1218. 10. Pringle M, Churchill R. Randomised controlled trials in general practice; gold standard or fool's gold? BMJ 1995; 311: i382-1383.