- Email: [email protected]
Issues in the planning and conduct of non-randomised studies
Injury, Int. J. Care Injured (2006) 37, 340—348
www.elsevier.com/locate/injury
Issues in the planning and conduct of non-randomised studies ´ a,*, Beate Hanson a, Branko Kopjar b Laurent Audige a
AO Clinical Investigation and Documentation, AO Center, Clavadelerstrasse, 7270 Davos Platz, Switzerland b Department of Health Services, University of Washington, H682 Health Sciences Center, Box 357660, Seattle, WA, USA
KEYWORDS Orthopaedics; Clinical research; Study designs; Non-randomised studies; Case-series; Cohort study
Summary This paper discusses topics related to the planning and implementation of non-randomised clinical studies in orthopaedics. A well-conducted case-series is appropriate to demonstrate the safety of a surgical intervention. The case-series design involves the provision of a defined intervention to a group of patients with the ultimate objective of describing the final outcome, including such occurrences as complications. There is no alternative procedure serving as a control. The key aspects are to ensure enrolment of all eligible patients and to obtain a sufficiently large sample size to allow precise and valid estimation of complication risks. Targeted complications should be clearly defined and fully documented during a pre-defined follow-up period. Loss to follow-up should be minimised. Comparative studies are required to demonstrate treatment effectiveness. If a randomised controlled trial (RCT) is not feasible, an observational design such as a cohort or a case-control study should be considered. In observational designs, the treatment decision is made by the surgeons. In a case-control study, patients are selected based on their outcomes and their treatment or exposure status is recorded retrospectively. In a cohort study, groups of patients are selected based on their treatment and are followed for outcomes. There are numerous variations. Data can be collected prospectively or retrospectively; comparison groups may be concurrent or non-concurrent, or studied at different locations. The optimal design is tailored to clinical questions and research settings, while keeping in mind the respective methodological strengths and weaknesses of available options. The strength of the observational study is its proximity to daily clinical practice. The limitations are the possibility of numerous biases and confounding factors. Despite many challenges to
* Corresponding author at: AO Clinical Investigation and Documentation, AO Foundation, Stettbachstrasse 10, CH-8600 Du ¨bendorf, Switzerland. Tel.: +41 44 200 2462; fax: +41 44 200 2460. E-mail address: [email protected] (L. Audige ´). 0020–1383/$ — see front matter # 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.injury.2006.01.026
Issues in the planning and conduct of non-randomised studies
341
the internal validity of non-randomised studies in orthopaedics surgery, it is possible to use such designs in order to provide reasonably valid answers to clinically important questions. # 2006 Elsevier Ltd. All rights reserved.
Introduction There is a growing interest in applying evidencebased approaches in orthopaedic surgery.5,12 Evidence-based orthopaedics is a process by which treatment decisions are made based on the combination of clinical experience, best available evidence from clinical studies, and patient preferences. Despite numerous challenges to the validity of clinical studies in orthopaedic surgery, it is still possible to draw reasonably valid conclusions about the safety and effectiveness of orthopaedic treatment from them. Documentation on the effectiveness of surgical interventions is best derived from well designed and conducted randomised controlled trials (RCT), and systematic reviews and meta-analyses of such trials.10,25 The RCT is considered a gold standard design because it is based on the randomisation of patients to treatment groups and their concurrent prospective comparison. The principles of randomisation and the conduction of RCTs are presented in a companion article in this issue of the Journal.6 There are genuine difficulties in conducting RCTs in orthopaedics,14 and in surgery in general.20,24,27,28 When RCT is not feasible, investigators should consider alternative non-randomised research designs. In the following we describe key methodological issues related to the planning and implementation of non-randomised clinical studies, and the threats to the validity of their findings.
Study design algorithm A design algorithm for the clinical study of surgical intervention is presented in Fig. 1. The figure is a simplified design classification tree that highlights the most important design characteristics. In the following, we often use the term ‘‘investigation’’ in a broad sense without pointing to one particular design. In defining the right design, the first major characteristic to define is whether the intervention of interest is to be compared with one or more other interventions. If there is no comparison group involved, the study is a ‘‘non-comparative investigation’’. Non-comparative investigations are also referred to as ‘‘descriptive studies’’ because their main purpose is to describe experiences with new or
complex treatment approaches. They are useful to document the safety of the intervention, and their results are used to generate hypotheses on treatment effectiveness. Descriptive studies provide a bridge between the developmental approaches and clinical research and, later, clinical practice. The most common clinical descriptive design is the ‘‘case series’’. A case-series describes common clinical cases. Other terms can be found, i.e. handling test, documentation series. For example, based on a case-series of 62 patients, Schutz et al.26 described the clinical and radiographic results following fixation of distal femoral fractures with the Less Invasive Stabilisation System (LISS). The outcomes were documented prospectively for 12 months. Case-series should be distinguished from ‘‘case reports’’ that usually describe unusual fractures, disorders or rare complications and include a limited number of patients (usually less than 10). In a literature review of concurrent ipsilateral fractures of the hip and femoral shaft, Alho3 identified 59 reports including 34 reports with less than 10 patients. Comparative investigations are also called ‘‘analytical studies’’ because their main objective is to make inferences about the effectiveness of two or more interventions. They are used to test specific research hypotheses. If the interventions are clearly defined and standardised, then the investigation is more experimental in nature and the term ‘‘trial’’ is used. Randomised trials usually compare welldefined and controlled interventions and are thus termed ‘‘Randomised Controlled Trials’’ (RCT). The degree to which surgical interventions are controlled varies between the trials. The terminology
Figure 1 Overview of the main study designs available to orthopaedic surgeons.
342 ‘‘experimental studies’’ is often used for controlled trials to underline their similarity with laboratory experiments. In most cases however surgical interventions are no longer in an experimental phase when there are evaluated in ‘‘controlled trials’’. In non-randomised controlled trials the design resembles that of RCT in all aspects, except for the random allocation to the treatment groups. Investigations in orthopaedic surgery often use data that are obtained from medical records or are collected prospectively without intervening in the treatment decision process. In these studies, the choice of treatment is decided according to the standard clinical approach, that is, the interaction between the patient, physician and other factors, rather than being enforced by a study protocol as in clinical trials. If surgeons are allowed to decide on the type of treatment approach, study investigators have no control over the interventions and only observe what surgeons are doing. Such study designs are called ‘‘observational studies’’. The advantage of this approach is that the study results are closer to those obtained in routine clinical practice. At the same time, these studies are significantly limited in the extent to which they may reach valid conclusions about treatment effectiveness. Two main observational study designs are cohort study and casecontrol study. Cohort studies (synonymous with incidence studies, longitudinal studies, follow-up studies) follow groups of patients over time in order to observe and compare the occurrence (incidence) or measurement of defined outcomes. The most common cohort design is the comparison of two concurrent treatment groups from the same clinics. There are however many variations that can be considerably influenced by the clinical questions, availability of the data, ethics and logistical issues. Information about patients, exposures and outcomes can be collected prospectively or retrospectively. The latter is based on a retroactive review of historical data from medical records and other sources of information. For instance, Grassi et al.13 compared non-operative treatment and intramedullary fixation of midclavicular fractures in a retrospective review of 40 patients in each group. Both groups were treated during the same period (so-called ‘‘concurrent’’), but in other studies they may have been treated in different periods (so-called ‘‘non-concurrent’’). When surgeons are convinced about the benefit of a new treatment, it may be difficult to obtain a concurrent control group. In such cases historical data may be obtained for the control group. Finally, study groups may also be documented by
L. Audige ´ et al. different clinicians and different clinics. A group of clinicians or clinics may apply one type of treatment and the other group applies the control intervention. For example, Papaloizos et al.22 have compared return-to-work outcomes for patients treated operatively and non-operatively for non-displaced scaphoid fractures. Data on the operative treatments were collected at the Geneva University Hospital and data on the non-operative treatments were collected from the Berne hospital and the Geneva University hospital whereby these patients were treated prior to the operative group. Studying groups at different places and at different times is inherently weaker as the findings can be attributed to time, clinic and surgeon-related factors in addition to the interventions under investigation. The strength of the cohort study depends upon the availability of a direct comparison of two treatments in similar groups. Finally, comparing outcomes of an intervention with what has been previously published in the literature (this is sometimes referred to as ‘‘literature controls’’) is a comparison of limited validity. The major limitation of cohort studies is that the patient groups being compared may differ in baseline characteristics important for the outcome. These characteristics, which may influence the outcome, are usually referred to as confounders (see later section). This may result in a situation in which differences in the outcomes are attributed to the effect of treatment, when such differences are actually the result of another factor (confounder). To reduce the impact of confounding researchers obtain information on confounding variables and use various design and statistical approaches in order to adjust their findings for the effects of confounding. Cohort studies and non-randomised controlled trials have similar designs and differ mainly in regard to the extent of control of the studied intervention. The term ‘‘cohort study’’ is commonly used to describe one or the other. Case-control studies compare patients with some definitive outcomes to a control group of patients without such outcomes. Information about exposures (e.g. treatment) is then collected retrospectively. Case-control studies can be very useful to investigate rare outcomes such as the assessment of treatment failures or occurrence of rare complications but they are infrequent in the investigation of treatment effectiveness. Most commonly they are used to investigate risk factors for the occurrence of fractures in epidemiological studies. In a recent case-control study, investigators interviewed 448 patients with a proximal femoral fracture (the cases) and 2023 persons over 45 years
Issues in the planning and conduct of non-randomised studies without such a fracture (the controls) to document a range of potential risk factors.8 They concluded that prevention of falls among frail, osteoporotic persons would reduce the incidence of proximal humeral fractures. Case-control studies are of short duration because of their retrospective nature. However, they are prone to different types of biases, in particular those related to the choice of the control group and the limitations of retrospective data collection.
Case-series are suitable to document the safety of an intervention The principal study objective may be to demonstrate the safety of a particular treatment. The study should demonstrate that serious complications and adverse events do not occur, or occur with a low and acceptable frequency. Safety can reasonably well be detected without recourse to a control group. Usually, a well-conducted case-series is appropriate. ‘‘Well-conducted’’ means that investigators ensure that random and systematic sources of error are limited and that the results are a valid representation of the target patient population. Important steps for conducting a case-series are as follows.
Specify the clinical question in the study objective All clinical studies should have stated objectives. For the case-series, the clinical question to be answered should include the target population defined by inclusion and exclusion criteria, the treatment to be applied, and the outcome(s) to be evaluated. For example, adult patients with unstable trochanteric fractures are treated with a Proximal Femoral Nail A to document treatment complications (Fig. 2).
343
Document completeness of patient inclusion One of the most important methodological aspects for a case-series is its external validity. Investigators should ensure that the results of a case-series apply to the target population defined by the inclusion and exclusion criteria and that no eligible patients are ignored. The completeness of patient inclusion into the study can probably be improved by a prospective case enrolment procedure rather than relying solely on retrospective case reviews. Whenever possible, measures should be implemented at each study site to check patient lists and identify all potentially eligible patients. This is often best achieved by onsite study personnel such as a study nurse. We recommend that the systematic prospective documentation of all eligible patients including a minimum of anonymous information should be used in all prospective case-series studies. This information allows exploration of differences between the patients receiving the studied intervention and those who are not, as well as exploring differences between the participating and non-participating patients. In a controlled approach, all patients defined by the inclusion criteria should receive the targeted intervention. In an observational approach, however, surgeons are allowed to choose alternative treatments. In such situations, the true target population is defined by the specific indications for receiving the studied intervention. These indication factors should be objectively defined and recorded as optimally as possible since they are likely to differ between interventions and could also differ between clinics and surgeons. For instance, Littenberg et al.18 reviewed cases series of closed tibia shaft fractures. They showed that series immobilised in a cast included 24—58% fractures localised in the distal aspect of the tibia (median 42%). This proportion ranged from 47% to 85% (median 66%) in series treated by open reduction and internal fixation, and was only 0% and 20% in two series treated with intramedullary nails.
Sample size
Figure 2 An example of a case-series in orthopaedic surgery. PFNA, Proximal Femoral Nail A; DHS, dynamic hip screw.
Theoretically, the only way to establish the truth is to document the experiences of all eligible patients in the population. For obvious reasons this is not feasible. Fortunately, it is also not necessary as reasonably valid and precise estimates of population values can be derived from samples of patients. The sample has to be of sufficient size to limit the effect of random variations on the precision of the estimates. To estimate the required sample size investigators must have a rough idea about the frequency
344 of outcomes and the desired level of precision. For example, in the study of complications following the treatment of trochanteric fractures with a Proximal Femoral Nail A, the major concern was the occurrence of cutting out and iatrogenic fractures. The study aimed to demonstrate with 95% statistical confidence that the frequency of such a serious complication does not exceed 2.5 occurrences per 100 patients. The sample size required to test the above hypothesis is 146 patients. This example shows that case-series with small sample sizes do not provide enough confidence in risk estimates and that large sample sizes are required to evaluate infrequent events such are rare but severe complications. In a recent study, 28 patients with 29 complex proximal humeral fractures were treated with the Locking Proximal Humerus Plate.11 Among other complications, the authors reported one case of plate breakage and no cases of non-union. The corresponding risk of plate breakage is 3.4% (95% confidence intervals 0.08—17.8%) and that of nonunion 0% (95% confidence interval 0—12.0%). Wide confidence intervals in this example illustrate that small case-series have major limitations in estimating the true risk of rare complications.
All complications and adverse events should be documented The outcomes to be evaluated should be clearly defined in advance so that there is no ambiguity in defining the cases in which the outcomes occur. The study protocol should provide a list of expected complications, most commonly distinguished as intra- and post-operative. Some complications are well known and defined (e.g. wound infection following the criteria of the Centers for Disease Control and Prevention1), but other types of complications often remain poorly defined. When there is a lack of a widely agreed definition (e.g. there is no uniform definition of delayed healing and fracture nonunion), investigators must provide and justify the definition used in their study. The timing of the occurrence of complications is also important, and the study protocol should specify the length of follow-up. For example, patients with trochanteric fractures may be followed-up for 6 months or for 12 months after fracture fixation with an intramedullary nail. The difference between the 6 and 12 months window would not have a major impact on the validity of the risk estimate for the occurrence of cutting out (a complication expected in some of these patients) as the vast majority of such complications occur in the first 3—4 months. However, the differential follow-up time would have an impact on the validity of the
L. Audige ´ et al. estimates of occurrence of late fracture of the femur. Case-series are retrospective when data about patients and outcomes are already collected when the study is initiated, such as when they are based on medical record reviews. They are prospective when patients are followed-up after the initiation of the study to collect required data. Well-conducted prospective case-series have advantages including the possibility to have an active investigator control over the completeness of registration and documentation of events, something that is not integrated into the retrospective approach.
Data analyses and reporting of the findings Treatment complications are commonly reported by tabulation and classification according to their level of severity (‘‘mild’’, ‘‘moderate’’, ‘‘serious/severe’’) and in their relation to the treatment (‘‘definite’’, ‘‘probable’’, ‘‘possible’’, ‘‘not related’’). The occurrence of a definite treatment-related serious/severe complication may lead to discontinuation of the study. For less serious events, the observed number of complications (even in the case of absence) have limited value without careful consideration of the number of patients ‘‘at risk’’ at any specific time following the treatment. The commonly used term ‘‘complication rates’’ is, in most cases, in fact the cumulative number of complications up to a specific point in time. With no patient loss during the followup, treatment complication risks are simply calculated as the number of complication events relative to the number of patients treated. For instance, Tennent et al.29 examined the clinical records of 47 patients with 51 displaced intra-articular calcaneus fractures treated by open reduction and internal fixation and followed-up for a minimum of 2 years. The estimated risk of infection was 22% (11/51). The variability in follow-up time from 25 to 74 months has almost no relevance since infections occurred in the immediate post-operative period. For late complications however the period of follow-up is critical, such as for the investigation of late ipsilateral fractures of the femur that occur around 5—6 months following hip fracture surgery.23 Lost to follow-up due to mortality is usually high in this population so the estimated risk of late fracture needs to be adjusted for patient loss. The patients who died could have sustained a re-fracture if they had not been lost. It is therefore important to record the follow-up time for each patient as well as the timing of occurrence of treatment complications. If the period of observation is known for each patient, including those lost to follow-up, and if the sample size is large enough, it is possible to apply more advanced statistical methods
Issues in the planning and conduct of non-randomised studies
345
and derive more correct estimates of the risk of treatment complications.
Other purposes of the case-series In addition to safety issues, case-series are used to provide descriptive information about other outcomes such are healing, pain, function, and patient satisfaction. Common descriptive statistics can be used.15 These data are used to generate hypotheses about the effectiveness of the intervention, and support sample size calculation for comparative studies aiming at verifying these hypotheses. It is important to emphasise that such observations serve only to generate hypotheses and that effectiveness of treatments can only be validly established in comparative studies.
Comparative studies are needed to document the effectiveness of an intervention Often the objective of the investigation is to test the hypothesis that patients do ‘‘better’’ or ‘‘worse’’ with the investigated treatment. The wording of this sort of hypothesis implies a comparison with a control treatment, ideally in a RCT design. When RCT is not feasible, non-randomised comparative designs should be considered. In the following we address the main methodological issues of non-randomised designs with a specific focus on the cohort study. Cohort studies are more exposed to biases than RCTs. The investigators should be aware of these potential biases and be sure to do whatever they can to limit their effects on the validity of the findings. For further presentation of the major sources of bias see Audige ´ et al.4 The main steps to be ensured in nonrandomised comparative designs are described below.
Specify the clinical question in the study objective The formulation of the clinical question for a cohort study is similar to that for the RCT. The question should include the target population defined by inclusion and exclusion criteria, the intervention to be applied, the control intervention under comparison, and the outcome(s) to be assessed. For example, adult patients with an isolated fracture of the scaphoid and treated with a screw fixation may be compared to similar patients treated with a cast with regard to return to work (Fig. 3).
Figure 3 surgery.
An example of a cohort study in orthopedic
Enrolled patients should be representative of the target population A trial sample should be representative of the population to which the conclusions will be generalised. While this was already discussed for the case-series, it also applies to comparative studies. A systematic difference in characteristics between patients who are selected for study and those who are not would lead to a form of selection bias.2 This affects external validity but a study with a nonrepresentative sample can still have good internal validity and lead to valid conclusions on treatment effectiveness. Patients included in RCTs (i.e. eligible and participating) tend to have a different prognosis from patients identified from clinical databases.7 Participation in non-randomised prospective studies however may also be limited because of the need to obtain the patient’s informed consent. For instance, imagine that study participants were younger than non-participants; if age influenced the effectiveness of treatment, results would be biased and not fully apply to the target patient population. We therefore encourage investigators to keep a ‘‘clinical journal’’ of eligible patients thus increasing understanding of the reasons for non-participation in comparative surgical studies. This would help not only to better describe external validity of the study but also to develop strategies for improving patient recruitment. The participation rate of eligible patients should be considered in prospective studies so that a realistic enrolment period is determined in a study plan.
Differences in prognostic factors should be recorded Allocation bias occurs when the allocation of patients to intervention groups is related to the prognostic factors for the study outcomes. Allocation bias is a form of selection bias, the situation in which treatment apparently appears as effective
346 due to the ways in which the patients were selected to the groups rather than due to the true effect of the treatment. This bias occurs in RCT when the randomisation process is violated, for example, when the randomisation sequence is not concealed before patient inclusion.6 In non-randomised studies, treatment allocation is not concealed but decided by the surgeon, and probably results in a group imbalance with regard to baseline and prognostic factors because of their influence on treatment choice. Therefore, non-randomised studies are prone to selection bias. For example, Connor et al.9 compared early (within 1 week from the time of injury) versus delayed fixation of pelvic ring fractures with regard to pulmonary complications. The 28 patients in the delayed fixation group tended to be older (35 years versus 32 years), had a higher Injury Severity Score (23.2 versus 18.8) and a higher proportion of head injuries (42.9% versus 25.4%) than the other group of 71 patients. They stayed in the Intensive Care Unit significantly longer (13.3 days versus 6.1 days). Any observed differences in outcome between these groups may be related (fully or partly) to the differences in the baseline factors. In observational studies, the factors associated with treatment choice may be few or numerous, objective or subjective, conscious or unconscious, but investigators should attempt to document them and understand the decision process. Should one or several of these factors be prognostic for the study outcome(s), observed outcome differences between groups will be confounded. Therefore, in planning the study, investigators should develop a list of relevant prognostic factors after in-depth literature review and discussion with experienced clinicians, including those involved in the study. The information about these important prognostic characteristics should then be recorded at baseline examination and later used to adjust findings for the effect of confounding.
The intended interventions should be distinguished from unplanned treatment changes In RCTs, the application of the intention-to-treat principle implies that the patients are analysed in the group to which they were randomised.4 This concept is easy to understand in the context of a RCT because ‘‘intended’’ treatments are defined in the trial protocol and by the randomised allocation. In non-randomised studies, the ‘‘intended’’ treatments are less easy to define. Planned treatments, such as the use of external fixators or temporary conservative immobilisation, planned nail dynamisation or implant removal, should be considered as
L. Audige ´ et al. ‘‘intended’’ treatments. The treatment plan (not only the initial treatment) that is determined after the diagnosis should be recorded. Any changes in this plan caused by unforeseen circumstances, such as technical problems during the operation, failure of fracture reduction or the occurrence of complications, should be recorded as ‘‘unintended’’. This is usually referred to as cross-over and is common in trials of surgical interventions. When conducting the analyses according to the Intention-To-Treat (ITT) principles, treatment groups should be formed on the basis of the planned (intended) treatments. For instance, in a study of Locking Compression Plate fixation of distal radius fractures we included patients treated within 10 days post-injury, but in an ITTanalysis excluded patients who received a LCP because of the failure of primary non-operative treatment. Most often however, non-randomised treatment groups are formed by investigators on the basis of what patients actually received rather than what was planned for them (so-called ‘‘astreated’’ analysis). This is particularly the case in retrospective observational studies since the historical data only document what treatment patients received while the information on planned treatments is lacking.
All patients and treatment groups should be followed similarly and examined at defined clinically relevant time points Recording bias will occur when the documentation of outcome(s) is not standardised between patients and intervention groups. Recording bias belongs to a larger group of information biases. For instance, if the control group is examined earlier (say at 5 months) than the intervention group (say at 6 months), it is likely that the healing status or functional outcome will be biased in favour of the intervention group. Prospective studies usually do not suffer from recording bias. However, recording bias is often present in retrospective studies. In such situations, we recommend that investigators define an acceptable ‘‘follow-up time’’ for each outcome and analyse the data accordingly. In prospective studies, a tolerable time deviation from the predetermined follow-up time should be set and monitored. For instance, the healing status of tibia shaft fractures may be examined at 6 months (180 days) with a tolerated interval of 165—195 days. Any nontolerated deviation in follow-up time should be examined for its potential effect bias on the results, e.g. if a delayed healing is reported at 150 days (but reported as a ‘‘6-month’’ follow-up). Lost-to-follow-up bias (also called attrition bias) occurs when patients are lost during the follow-up
Issues in the planning and conduct of non-randomised studies for reasons that are directly or indirectly related to the intervention received and is one of the trial outcomes. One example is patients with healed fractures who may not feel a need to come back for the final examination. This is an important topic for all clinical investigations, including RCTs. Every attempt should be made to minimise a ‘‘lost-to-follow-up’’. The study protocol should describe the steps taken to limit the risk of losing patients, and the study should document the reasons for patient loss. The extent to which patient attrition can bias the results should be examined for each study.
347
adequately adjusted for due to a lack of data or limitations in the statistical methods. For this reason, residual confounding always remains an alternative explanation for the effects seen in the observational studies. This is a limitation of observational studies that it is not possible to overcome satisfactorily and therefore there will always be less confidence in the findings from observational studies compared to those derived from well-conducted RCTs.
Final words Imbalances between the treatment groups in baseline characteristics should be controlled for in the data analyses Confounding bias occurs when intervention groups differ in a prognostic factor that is not being taken into account in the analysis. A prognostic factor for the study outcome is a confounding factor only if it differs between the treatment groups. In RCTs, this is minimised by the randomisation. In non-randomised studies, the baseline differences occur due to the nature of the study (Fig. 4). Non-randomised studies must control for confounding factors and apply appropriate advanced statistical methods in order to adjust for the differences in the baseline prognostic factors.19,21 For instance, in the study of Connor et al.9 mentioned earlier, 28.6% of patients (8/28) with delayed fixation and 12.7% of patients (9/71) with early fixation suffered pulmonary complications (unadjusted Risk Ratio = 0.44). After adjustment for baseline differences in ISS the risk of pulmonary complications was statistically significantly lower in the early fixation group (adjusted Risk Ratio = 0.49). However, it is important to keep in mind that not all differences can be
We have described some important methodological issues in the planning and execution of non-randomised orthopaedic studies. Selecting the right study design depends on the type of clinical question and other factors such as expected frequency of the outcome(s), patient volumes and patient willingness to participate, etc. Current clinical research methods are applied to ensure the validity of the results both externally (in relation to the broader patient population) and internally (in relation to the validity of between group comparisons). Many other important issues in designing and running a clinical trial were omitted in this text for reasons of space. We encourage readers to consult additional literature on the topic.5,16,17 In addition to considering issues described in this text, it is always wise to share a study protocol with colleagues and professional methodologists. Only in shared efforts can the best possible design be found for a specific research question and setting. Investigators should not be discouraged if the ideal study design is not feasible, but make the best methodological compromises and assess how these compromises can influence the validity of the results.
References
Figure 4 studies.
Prognostic and confounding factors in clinical
1. http://www.cdc.gov/ncidod/hip/NNIS/NosInfDefinitions.pdf. 2. http://www.cochrane.org/resources/glossary.htm. 3. Alho A. Concurrent ipsilateral fractures of the hip and femoral shaft: a meta-analysis of 659 cases. Acta Orthop Scand 1996;67:19—28. 4. Audige ´ L, Hanson B, Bhandari M, Schemitsch E. Interpretation of data and analysis of surgical trials. Techn Orthop 2004;19:1—8. 5. Barbier O, Hoogmartens M. Evidence-based medicine in orthopaedics. Acta Orthop Belg 2004;70:91—7. 6. Bhandari M, Giannoudis P, Pape HC. Issues in the planning and conduct of randomized trials. Injury 2005; same issue. 7. Britton A, McKee M, Black N, et al. Threats to applicability of randomised trials: exclusions and selective participation. J Health Serv Res Policy 1999;4:112—21.
348 8. Chu SP, Kelsey JL, Keegan TH, et al. Risk factors for proximal humerus fracture. Am J Epidemiol 2004;160:360—7. 9. Connor GS, McGwin Jr G, MacLennan PA, et al. Early versus delayed fixation of pelvic ring fractures. Am Surg 2003;69: 1019—23. 10. Egger M, Smith GD, O’Rourke K. Rationale, potentials, and promise of systematic reviews. In: Egger M, Smith GD, Altman DG, editors. Systematic reviews in health care–— meta-analysis in context. 2nd ed., London: BMJ Publishing; 2001. p. 3—19. 11. Fankhauser F, Boldin C, Schippinger G, et al. A new locking plate for unstable fractures of the proximal humerus. Clin Orthop 2005;176—81. 12. Goldhahn S, Audige L, Helfet DL, Hanson B. Pathways to evidence-based knowledge in orthopaedic surgery: an international survey of AO course participants. Int Orthop 2005;29:59—64. 13. Grassi FA, Tajana MS, D’Angelo F. Management of midclavicular fractures: comparison between nonoperative treatment and open intramedullary fixation in 80 patients. J Trauma 2001;50:1096—100. 14. Griffin D. Randomisation in orthopaedic surgical studies–—are there special problems which prevent the acquisition of randomised evidence of clinical effectiveness in orthopaedic surgery? Master Thesis. Department of Public Health and Primary Care, Faculty of Medicine, University of Cambridge; 2000. 15. Griffin D, Audige L. Common statistical methods in orthopaedic clinical studies. Clin Orthop 2003;413:70—9. 16. Gurwitz JH, Sykora K, Mamdani M, et al. Reader’s guide to critical appraisal of cohort studies. 1. Role and design. BMJ 2005;330:895—7. 17. Hartz A, Marsh JL. Methodologic issues in observational studies. Clin Orthop 2003;33—42. 18. Littenberg B, Weinstein LP, McCarren M, et al. Closed fractures of the tibial shaft. A meta-analysis of three methods of treatment [see comments]. J Bone Joint Surg Am 1998;80: 174—83.
L. Audige ´ et al. 19. Mamdani M, Sykora K, Li P, et al. Reader’s guide to critical appraisal of cohort studies. 2. Assessing potential for confounding. BMJ 2005;330:960—2. 20. McCulloch P, Taylor I, Sasako M, et al. Randomised trials in surgery: problems and possible solutions. BMJ 2002;324: 1448—51. 21. Normand SL, Sykora K, Li P, et al. Readers guide to critical appraisal of cohort studies. 3. Analytical strategies to reduce confounding. BMJ 2005;330:1021—3. 22. Papaloizos MY, Fusetti C, Christen T, et al. Minimally invasive fixation versus conservative treatment of undisplaced scaphoid fractures: a cost-effectiveness study. J Hand Surg [Br] 2004;29:116—9. 23. Robinson CM, Adams CI, Craig M, et al. Implant-related fractures of the femur following hip fracture surgery. J Bone Joint Surg Am 2002;84-A:1116—22. 24. Rudicel S, Esdaile J. The randomized clinical trial in orthopaedics: obligation or option? J Bone Joint Surg [Am] 1985;67:1284—93. 25. Sackett DL, Straus SE, Richardson WS, et al. Evidence-based medicine. How to practice and teach EBM, 2nd ed., Edinburgh, London: Churchill Livingstone; 2000. 26. Schutz M, Muller M, Regazzoni P, et al. Use of the less invasive stabilization system (LISS) in patients with distal femoral (AO33) fractures: a prospective multicenter study. Arch Orthop Trauma Surg 2005. 27. Solomon MJ, McLeod RS. Should we be performing more randomized controlled trials evaluating surgical operations? Surgery 1995;118:459—67. 28. Solomon MJ, McLeod RS. Surgery and the randomised controlled trial: past, present and future [see comments]. Med J Aust 1998;169:380—3. 29. Tennent TD, Calder PR, Salisbury RD, et al. The operative management of displaced intra-articular fractures of the calcaneum: a two-centre study using a defined protocol. Injury 2001;32:491—6.
www.elsevier.com/locate/injury
Issues in the planning and conduct of non-randomised studies ´ a,*, Beate Hanson a, Branko Kopjar b Laurent Audige a
AO Clinical Investigation and Documentation, AO Center, Clavadelerstrasse, 7270 Davos Platz, Switzerland b Department of Health Services, University of Washington, H682 Health Sciences Center, Box 357660, Seattle, WA, USA
KEYWORDS Orthopaedics; Clinical research; Study designs; Non-randomised studies; Case-series; Cohort study
Summary This paper discusses topics related to the planning and implementation of non-randomised clinical studies in orthopaedics. A well-conducted case-series is appropriate to demonstrate the safety of a surgical intervention. The case-series design involves the provision of a defined intervention to a group of patients with the ultimate objective of describing the final outcome, including such occurrences as complications. There is no alternative procedure serving as a control. The key aspects are to ensure enrolment of all eligible patients and to obtain a sufficiently large sample size to allow precise and valid estimation of complication risks. Targeted complications should be clearly defined and fully documented during a pre-defined follow-up period. Loss to follow-up should be minimised. Comparative studies are required to demonstrate treatment effectiveness. If a randomised controlled trial (RCT) is not feasible, an observational design such as a cohort or a case-control study should be considered. In observational designs, the treatment decision is made by the surgeons. In a case-control study, patients are selected based on their outcomes and their treatment or exposure status is recorded retrospectively. In a cohort study, groups of patients are selected based on their treatment and are followed for outcomes. There are numerous variations. Data can be collected prospectively or retrospectively; comparison groups may be concurrent or non-concurrent, or studied at different locations. The optimal design is tailored to clinical questions and research settings, while keeping in mind the respective methodological strengths and weaknesses of available options. The strength of the observational study is its proximity to daily clinical practice. The limitations are the possibility of numerous biases and confounding factors. Despite many challenges to
* Corresponding author at: AO Clinical Investigation and Documentation, AO Foundation, Stettbachstrasse 10, CH-8600 Du ¨bendorf, Switzerland. Tel.: +41 44 200 2462; fax: +41 44 200 2460. E-mail address: [email protected] (L. Audige ´). 0020–1383/$ — see front matter # 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.injury.2006.01.026
Issues in the planning and conduct of non-randomised studies
341
the internal validity of non-randomised studies in orthopaedics surgery, it is possible to use such designs in order to provide reasonably valid answers to clinically important questions. # 2006 Elsevier Ltd. All rights reserved.
Introduction There is a growing interest in applying evidencebased approaches in orthopaedic surgery.5,12 Evidence-based orthopaedics is a process by which treatment decisions are made based on the combination of clinical experience, best available evidence from clinical studies, and patient preferences. Despite numerous challenges to the validity of clinical studies in orthopaedic surgery, it is still possible to draw reasonably valid conclusions about the safety and effectiveness of orthopaedic treatment from them. Documentation on the effectiveness of surgical interventions is best derived from well designed and conducted randomised controlled trials (RCT), and systematic reviews and meta-analyses of such trials.10,25 The RCT is considered a gold standard design because it is based on the randomisation of patients to treatment groups and their concurrent prospective comparison. The principles of randomisation and the conduction of RCTs are presented in a companion article in this issue of the Journal.6 There are genuine difficulties in conducting RCTs in orthopaedics,14 and in surgery in general.20,24,27,28 When RCT is not feasible, investigators should consider alternative non-randomised research designs. In the following we describe key methodological issues related to the planning and implementation of non-randomised clinical studies, and the threats to the validity of their findings.
Study design algorithm A design algorithm for the clinical study of surgical intervention is presented in Fig. 1. The figure is a simplified design classification tree that highlights the most important design characteristics. In the following, we often use the term ‘‘investigation’’ in a broad sense without pointing to one particular design. In defining the right design, the first major characteristic to define is whether the intervention of interest is to be compared with one or more other interventions. If there is no comparison group involved, the study is a ‘‘non-comparative investigation’’. Non-comparative investigations are also referred to as ‘‘descriptive studies’’ because their main purpose is to describe experiences with new or
complex treatment approaches. They are useful to document the safety of the intervention, and their results are used to generate hypotheses on treatment effectiveness. Descriptive studies provide a bridge between the developmental approaches and clinical research and, later, clinical practice. The most common clinical descriptive design is the ‘‘case series’’. A case-series describes common clinical cases. Other terms can be found, i.e. handling test, documentation series. For example, based on a case-series of 62 patients, Schutz et al.26 described the clinical and radiographic results following fixation of distal femoral fractures with the Less Invasive Stabilisation System (LISS). The outcomes were documented prospectively for 12 months. Case-series should be distinguished from ‘‘case reports’’ that usually describe unusual fractures, disorders or rare complications and include a limited number of patients (usually less than 10). In a literature review of concurrent ipsilateral fractures of the hip and femoral shaft, Alho3 identified 59 reports including 34 reports with less than 10 patients. Comparative investigations are also called ‘‘analytical studies’’ because their main objective is to make inferences about the effectiveness of two or more interventions. They are used to test specific research hypotheses. If the interventions are clearly defined and standardised, then the investigation is more experimental in nature and the term ‘‘trial’’ is used. Randomised trials usually compare welldefined and controlled interventions and are thus termed ‘‘Randomised Controlled Trials’’ (RCT). The degree to which surgical interventions are controlled varies between the trials. The terminology
Figure 1 Overview of the main study designs available to orthopaedic surgeons.
342 ‘‘experimental studies’’ is often used for controlled trials to underline their similarity with laboratory experiments. In most cases however surgical interventions are no longer in an experimental phase when there are evaluated in ‘‘controlled trials’’. In non-randomised controlled trials the design resembles that of RCT in all aspects, except for the random allocation to the treatment groups. Investigations in orthopaedic surgery often use data that are obtained from medical records or are collected prospectively without intervening in the treatment decision process. In these studies, the choice of treatment is decided according to the standard clinical approach, that is, the interaction between the patient, physician and other factors, rather than being enforced by a study protocol as in clinical trials. If surgeons are allowed to decide on the type of treatment approach, study investigators have no control over the interventions and only observe what surgeons are doing. Such study designs are called ‘‘observational studies’’. The advantage of this approach is that the study results are closer to those obtained in routine clinical practice. At the same time, these studies are significantly limited in the extent to which they may reach valid conclusions about treatment effectiveness. Two main observational study designs are cohort study and casecontrol study. Cohort studies (synonymous with incidence studies, longitudinal studies, follow-up studies) follow groups of patients over time in order to observe and compare the occurrence (incidence) or measurement of defined outcomes. The most common cohort design is the comparison of two concurrent treatment groups from the same clinics. There are however many variations that can be considerably influenced by the clinical questions, availability of the data, ethics and logistical issues. Information about patients, exposures and outcomes can be collected prospectively or retrospectively. The latter is based on a retroactive review of historical data from medical records and other sources of information. For instance, Grassi et al.13 compared non-operative treatment and intramedullary fixation of midclavicular fractures in a retrospective review of 40 patients in each group. Both groups were treated during the same period (so-called ‘‘concurrent’’), but in other studies they may have been treated in different periods (so-called ‘‘non-concurrent’’). When surgeons are convinced about the benefit of a new treatment, it may be difficult to obtain a concurrent control group. In such cases historical data may be obtained for the control group. Finally, study groups may also be documented by
L. Audige ´ et al. different clinicians and different clinics. A group of clinicians or clinics may apply one type of treatment and the other group applies the control intervention. For example, Papaloizos et al.22 have compared return-to-work outcomes for patients treated operatively and non-operatively for non-displaced scaphoid fractures. Data on the operative treatments were collected at the Geneva University Hospital and data on the non-operative treatments were collected from the Berne hospital and the Geneva University hospital whereby these patients were treated prior to the operative group. Studying groups at different places and at different times is inherently weaker as the findings can be attributed to time, clinic and surgeon-related factors in addition to the interventions under investigation. The strength of the cohort study depends upon the availability of a direct comparison of two treatments in similar groups. Finally, comparing outcomes of an intervention with what has been previously published in the literature (this is sometimes referred to as ‘‘literature controls’’) is a comparison of limited validity. The major limitation of cohort studies is that the patient groups being compared may differ in baseline characteristics important for the outcome. These characteristics, which may influence the outcome, are usually referred to as confounders (see later section). This may result in a situation in which differences in the outcomes are attributed to the effect of treatment, when such differences are actually the result of another factor (confounder). To reduce the impact of confounding researchers obtain information on confounding variables and use various design and statistical approaches in order to adjust their findings for the effects of confounding. Cohort studies and non-randomised controlled trials have similar designs and differ mainly in regard to the extent of control of the studied intervention. The term ‘‘cohort study’’ is commonly used to describe one or the other. Case-control studies compare patients with some definitive outcomes to a control group of patients without such outcomes. Information about exposures (e.g. treatment) is then collected retrospectively. Case-control studies can be very useful to investigate rare outcomes such as the assessment of treatment failures or occurrence of rare complications but they are infrequent in the investigation of treatment effectiveness. Most commonly they are used to investigate risk factors for the occurrence of fractures in epidemiological studies. In a recent case-control study, investigators interviewed 448 patients with a proximal femoral fracture (the cases) and 2023 persons over 45 years
Issues in the planning and conduct of non-randomised studies without such a fracture (the controls) to document a range of potential risk factors.8 They concluded that prevention of falls among frail, osteoporotic persons would reduce the incidence of proximal humeral fractures. Case-control studies are of short duration because of their retrospective nature. However, they are prone to different types of biases, in particular those related to the choice of the control group and the limitations of retrospective data collection.
Case-series are suitable to document the safety of an intervention The principal study objective may be to demonstrate the safety of a particular treatment. The study should demonstrate that serious complications and adverse events do not occur, or occur with a low and acceptable frequency. Safety can reasonably well be detected without recourse to a control group. Usually, a well-conducted case-series is appropriate. ‘‘Well-conducted’’ means that investigators ensure that random and systematic sources of error are limited and that the results are a valid representation of the target patient population. Important steps for conducting a case-series are as follows.
Specify the clinical question in the study objective All clinical studies should have stated objectives. For the case-series, the clinical question to be answered should include the target population defined by inclusion and exclusion criteria, the treatment to be applied, and the outcome(s) to be evaluated. For example, adult patients with unstable trochanteric fractures are treated with a Proximal Femoral Nail A to document treatment complications (Fig. 2).
343
Document completeness of patient inclusion One of the most important methodological aspects for a case-series is its external validity. Investigators should ensure that the results of a case-series apply to the target population defined by the inclusion and exclusion criteria and that no eligible patients are ignored. The completeness of patient inclusion into the study can probably be improved by a prospective case enrolment procedure rather than relying solely on retrospective case reviews. Whenever possible, measures should be implemented at each study site to check patient lists and identify all potentially eligible patients. This is often best achieved by onsite study personnel such as a study nurse. We recommend that the systematic prospective documentation of all eligible patients including a minimum of anonymous information should be used in all prospective case-series studies. This information allows exploration of differences between the patients receiving the studied intervention and those who are not, as well as exploring differences between the participating and non-participating patients. In a controlled approach, all patients defined by the inclusion criteria should receive the targeted intervention. In an observational approach, however, surgeons are allowed to choose alternative treatments. In such situations, the true target population is defined by the specific indications for receiving the studied intervention. These indication factors should be objectively defined and recorded as optimally as possible since they are likely to differ between interventions and could also differ between clinics and surgeons. For instance, Littenberg et al.18 reviewed cases series of closed tibia shaft fractures. They showed that series immobilised in a cast included 24—58% fractures localised in the distal aspect of the tibia (median 42%). This proportion ranged from 47% to 85% (median 66%) in series treated by open reduction and internal fixation, and was only 0% and 20% in two series treated with intramedullary nails.
Sample size
Figure 2 An example of a case-series in orthopaedic surgery. PFNA, Proximal Femoral Nail A; DHS, dynamic hip screw.
Theoretically, the only way to establish the truth is to document the experiences of all eligible patients in the population. For obvious reasons this is not feasible. Fortunately, it is also not necessary as reasonably valid and precise estimates of population values can be derived from samples of patients. The sample has to be of sufficient size to limit the effect of random variations on the precision of the estimates. To estimate the required sample size investigators must have a rough idea about the frequency
344 of outcomes and the desired level of precision. For example, in the study of complications following the treatment of trochanteric fractures with a Proximal Femoral Nail A, the major concern was the occurrence of cutting out and iatrogenic fractures. The study aimed to demonstrate with 95% statistical confidence that the frequency of such a serious complication does not exceed 2.5 occurrences per 100 patients. The sample size required to test the above hypothesis is 146 patients. This example shows that case-series with small sample sizes do not provide enough confidence in risk estimates and that large sample sizes are required to evaluate infrequent events such are rare but severe complications. In a recent study, 28 patients with 29 complex proximal humeral fractures were treated with the Locking Proximal Humerus Plate.11 Among other complications, the authors reported one case of plate breakage and no cases of non-union. The corresponding risk of plate breakage is 3.4% (95% confidence intervals 0.08—17.8%) and that of nonunion 0% (95% confidence interval 0—12.0%). Wide confidence intervals in this example illustrate that small case-series have major limitations in estimating the true risk of rare complications.
All complications and adverse events should be documented The outcomes to be evaluated should be clearly defined in advance so that there is no ambiguity in defining the cases in which the outcomes occur. The study protocol should provide a list of expected complications, most commonly distinguished as intra- and post-operative. Some complications are well known and defined (e.g. wound infection following the criteria of the Centers for Disease Control and Prevention1), but other types of complications often remain poorly defined. When there is a lack of a widely agreed definition (e.g. there is no uniform definition of delayed healing and fracture nonunion), investigators must provide and justify the definition used in their study. The timing of the occurrence of complications is also important, and the study protocol should specify the length of follow-up. For example, patients with trochanteric fractures may be followed-up for 6 months or for 12 months after fracture fixation with an intramedullary nail. The difference between the 6 and 12 months window would not have a major impact on the validity of the risk estimate for the occurrence of cutting out (a complication expected in some of these patients) as the vast majority of such complications occur in the first 3—4 months. However, the differential follow-up time would have an impact on the validity of the
L. Audige ´ et al. estimates of occurrence of late fracture of the femur. Case-series are retrospective when data about patients and outcomes are already collected when the study is initiated, such as when they are based on medical record reviews. They are prospective when patients are followed-up after the initiation of the study to collect required data. Well-conducted prospective case-series have advantages including the possibility to have an active investigator control over the completeness of registration and documentation of events, something that is not integrated into the retrospective approach.
Data analyses and reporting of the findings Treatment complications are commonly reported by tabulation and classification according to their level of severity (‘‘mild’’, ‘‘moderate’’, ‘‘serious/severe’’) and in their relation to the treatment (‘‘definite’’, ‘‘probable’’, ‘‘possible’’, ‘‘not related’’). The occurrence of a definite treatment-related serious/severe complication may lead to discontinuation of the study. For less serious events, the observed number of complications (even in the case of absence) have limited value without careful consideration of the number of patients ‘‘at risk’’ at any specific time following the treatment. The commonly used term ‘‘complication rates’’ is, in most cases, in fact the cumulative number of complications up to a specific point in time. With no patient loss during the followup, treatment complication risks are simply calculated as the number of complication events relative to the number of patients treated. For instance, Tennent et al.29 examined the clinical records of 47 patients with 51 displaced intra-articular calcaneus fractures treated by open reduction and internal fixation and followed-up for a minimum of 2 years. The estimated risk of infection was 22% (11/51). The variability in follow-up time from 25 to 74 months has almost no relevance since infections occurred in the immediate post-operative period. For late complications however the period of follow-up is critical, such as for the investigation of late ipsilateral fractures of the femur that occur around 5—6 months following hip fracture surgery.23 Lost to follow-up due to mortality is usually high in this population so the estimated risk of late fracture needs to be adjusted for patient loss. The patients who died could have sustained a re-fracture if they had not been lost. It is therefore important to record the follow-up time for each patient as well as the timing of occurrence of treatment complications. If the period of observation is known for each patient, including those lost to follow-up, and if the sample size is large enough, it is possible to apply more advanced statistical methods
Issues in the planning and conduct of non-randomised studies
345
and derive more correct estimates of the risk of treatment complications.
Other purposes of the case-series In addition to safety issues, case-series are used to provide descriptive information about other outcomes such are healing, pain, function, and patient satisfaction. Common descriptive statistics can be used.15 These data are used to generate hypotheses about the effectiveness of the intervention, and support sample size calculation for comparative studies aiming at verifying these hypotheses. It is important to emphasise that such observations serve only to generate hypotheses and that effectiveness of treatments can only be validly established in comparative studies.
Comparative studies are needed to document the effectiveness of an intervention Often the objective of the investigation is to test the hypothesis that patients do ‘‘better’’ or ‘‘worse’’ with the investigated treatment. The wording of this sort of hypothesis implies a comparison with a control treatment, ideally in a RCT design. When RCT is not feasible, non-randomised comparative designs should be considered. In the following we address the main methodological issues of non-randomised designs with a specific focus on the cohort study. Cohort studies are more exposed to biases than RCTs. The investigators should be aware of these potential biases and be sure to do whatever they can to limit their effects on the validity of the findings. For further presentation of the major sources of bias see Audige ´ et al.4 The main steps to be ensured in nonrandomised comparative designs are described below.
Specify the clinical question in the study objective The formulation of the clinical question for a cohort study is similar to that for the RCT. The question should include the target population defined by inclusion and exclusion criteria, the intervention to be applied, the control intervention under comparison, and the outcome(s) to be assessed. For example, adult patients with an isolated fracture of the scaphoid and treated with a screw fixation may be compared to similar patients treated with a cast with regard to return to work (Fig. 3).
Figure 3 surgery.
An example of a cohort study in orthopedic
Enrolled patients should be representative of the target population A trial sample should be representative of the population to which the conclusions will be generalised. While this was already discussed for the case-series, it also applies to comparative studies. A systematic difference in characteristics between patients who are selected for study and those who are not would lead to a form of selection bias.2 This affects external validity but a study with a nonrepresentative sample can still have good internal validity and lead to valid conclusions on treatment effectiveness. Patients included in RCTs (i.e. eligible and participating) tend to have a different prognosis from patients identified from clinical databases.7 Participation in non-randomised prospective studies however may also be limited because of the need to obtain the patient’s informed consent. For instance, imagine that study participants were younger than non-participants; if age influenced the effectiveness of treatment, results would be biased and not fully apply to the target patient population. We therefore encourage investigators to keep a ‘‘clinical journal’’ of eligible patients thus increasing understanding of the reasons for non-participation in comparative surgical studies. This would help not only to better describe external validity of the study but also to develop strategies for improving patient recruitment. The participation rate of eligible patients should be considered in prospective studies so that a realistic enrolment period is determined in a study plan.
Differences in prognostic factors should be recorded Allocation bias occurs when the allocation of patients to intervention groups is related to the prognostic factors for the study outcomes. Allocation bias is a form of selection bias, the situation in which treatment apparently appears as effective
346 due to the ways in which the patients were selected to the groups rather than due to the true effect of the treatment. This bias occurs in RCT when the randomisation process is violated, for example, when the randomisation sequence is not concealed before patient inclusion.6 In non-randomised studies, treatment allocation is not concealed but decided by the surgeon, and probably results in a group imbalance with regard to baseline and prognostic factors because of their influence on treatment choice. Therefore, non-randomised studies are prone to selection bias. For example, Connor et al.9 compared early (within 1 week from the time of injury) versus delayed fixation of pelvic ring fractures with regard to pulmonary complications. The 28 patients in the delayed fixation group tended to be older (35 years versus 32 years), had a higher Injury Severity Score (23.2 versus 18.8) and a higher proportion of head injuries (42.9% versus 25.4%) than the other group of 71 patients. They stayed in the Intensive Care Unit significantly longer (13.3 days versus 6.1 days). Any observed differences in outcome between these groups may be related (fully or partly) to the differences in the baseline factors. In observational studies, the factors associated with treatment choice may be few or numerous, objective or subjective, conscious or unconscious, but investigators should attempt to document them and understand the decision process. Should one or several of these factors be prognostic for the study outcome(s), observed outcome differences between groups will be confounded. Therefore, in planning the study, investigators should develop a list of relevant prognostic factors after in-depth literature review and discussion with experienced clinicians, including those involved in the study. The information about these important prognostic characteristics should then be recorded at baseline examination and later used to adjust findings for the effect of confounding.
The intended interventions should be distinguished from unplanned treatment changes In RCTs, the application of the intention-to-treat principle implies that the patients are analysed in the group to which they were randomised.4 This concept is easy to understand in the context of a RCT because ‘‘intended’’ treatments are defined in the trial protocol and by the randomised allocation. In non-randomised studies, the ‘‘intended’’ treatments are less easy to define. Planned treatments, such as the use of external fixators or temporary conservative immobilisation, planned nail dynamisation or implant removal, should be considered as
L. Audige ´ et al. ‘‘intended’’ treatments. The treatment plan (not only the initial treatment) that is determined after the diagnosis should be recorded. Any changes in this plan caused by unforeseen circumstances, such as technical problems during the operation, failure of fracture reduction or the occurrence of complications, should be recorded as ‘‘unintended’’. This is usually referred to as cross-over and is common in trials of surgical interventions. When conducting the analyses according to the Intention-To-Treat (ITT) principles, treatment groups should be formed on the basis of the planned (intended) treatments. For instance, in a study of Locking Compression Plate fixation of distal radius fractures we included patients treated within 10 days post-injury, but in an ITTanalysis excluded patients who received a LCP because of the failure of primary non-operative treatment. Most often however, non-randomised treatment groups are formed by investigators on the basis of what patients actually received rather than what was planned for them (so-called ‘‘astreated’’ analysis). This is particularly the case in retrospective observational studies since the historical data only document what treatment patients received while the information on planned treatments is lacking.
All patients and treatment groups should be followed similarly and examined at defined clinically relevant time points Recording bias will occur when the documentation of outcome(s) is not standardised between patients and intervention groups. Recording bias belongs to a larger group of information biases. For instance, if the control group is examined earlier (say at 5 months) than the intervention group (say at 6 months), it is likely that the healing status or functional outcome will be biased in favour of the intervention group. Prospective studies usually do not suffer from recording bias. However, recording bias is often present in retrospective studies. In such situations, we recommend that investigators define an acceptable ‘‘follow-up time’’ for each outcome and analyse the data accordingly. In prospective studies, a tolerable time deviation from the predetermined follow-up time should be set and monitored. For instance, the healing status of tibia shaft fractures may be examined at 6 months (180 days) with a tolerated interval of 165—195 days. Any nontolerated deviation in follow-up time should be examined for its potential effect bias on the results, e.g. if a delayed healing is reported at 150 days (but reported as a ‘‘6-month’’ follow-up). Lost-to-follow-up bias (also called attrition bias) occurs when patients are lost during the follow-up
Issues in the planning and conduct of non-randomised studies for reasons that are directly or indirectly related to the intervention received and is one of the trial outcomes. One example is patients with healed fractures who may not feel a need to come back for the final examination. This is an important topic for all clinical investigations, including RCTs. Every attempt should be made to minimise a ‘‘lost-to-follow-up’’. The study protocol should describe the steps taken to limit the risk of losing patients, and the study should document the reasons for patient loss. The extent to which patient attrition can bias the results should be examined for each study.
347
adequately adjusted for due to a lack of data or limitations in the statistical methods. For this reason, residual confounding always remains an alternative explanation for the effects seen in the observational studies. This is a limitation of observational studies that it is not possible to overcome satisfactorily and therefore there will always be less confidence in the findings from observational studies compared to those derived from well-conducted RCTs.
Final words Imbalances between the treatment groups in baseline characteristics should be controlled for in the data analyses Confounding bias occurs when intervention groups differ in a prognostic factor that is not being taken into account in the analysis. A prognostic factor for the study outcome is a confounding factor only if it differs between the treatment groups. In RCTs, this is minimised by the randomisation. In non-randomised studies, the baseline differences occur due to the nature of the study (Fig. 4). Non-randomised studies must control for confounding factors and apply appropriate advanced statistical methods in order to adjust for the differences in the baseline prognostic factors.19,21 For instance, in the study of Connor et al.9 mentioned earlier, 28.6% of patients (8/28) with delayed fixation and 12.7% of patients (9/71) with early fixation suffered pulmonary complications (unadjusted Risk Ratio = 0.44). After adjustment for baseline differences in ISS the risk of pulmonary complications was statistically significantly lower in the early fixation group (adjusted Risk Ratio = 0.49). However, it is important to keep in mind that not all differences can be
We have described some important methodological issues in the planning and execution of non-randomised orthopaedic studies. Selecting the right study design depends on the type of clinical question and other factors such as expected frequency of the outcome(s), patient volumes and patient willingness to participate, etc. Current clinical research methods are applied to ensure the validity of the results both externally (in relation to the broader patient population) and internally (in relation to the validity of between group comparisons). Many other important issues in designing and running a clinical trial were omitted in this text for reasons of space. We encourage readers to consult additional literature on the topic.5,16,17 In addition to considering issues described in this text, it is always wise to share a study protocol with colleagues and professional methodologists. Only in shared efforts can the best possible design be found for a specific research question and setting. Investigators should not be discouraged if the ideal study design is not feasible, but make the best methodological compromises and assess how these compromises can influence the validity of the results.
References
Figure 4 studies.
Prognostic and confounding factors in clinical
1. http://www.cdc.gov/ncidod/hip/NNIS/NosInfDefinitions.pdf. 2. http://www.cochrane.org/resources/glossary.htm. 3. Alho A. Concurrent ipsilateral fractures of the hip and femoral shaft: a meta-analysis of 659 cases. Acta Orthop Scand 1996;67:19—28. 4. Audige ´ L, Hanson B, Bhandari M, Schemitsch E. Interpretation of data and analysis of surgical trials. Techn Orthop 2004;19:1—8. 5. Barbier O, Hoogmartens M. Evidence-based medicine in orthopaedics. Acta Orthop Belg 2004;70:91—7. 6. Bhandari M, Giannoudis P, Pape HC. Issues in the planning and conduct of randomized trials. Injury 2005; same issue. 7. Britton A, McKee M, Black N, et al. Threats to applicability of randomised trials: exclusions and selective participation. J Health Serv Res Policy 1999;4:112—21.
348 8. Chu SP, Kelsey JL, Keegan TH, et al. Risk factors for proximal humerus fracture. Am J Epidemiol 2004;160:360—7. 9. Connor GS, McGwin Jr G, MacLennan PA, et al. Early versus delayed fixation of pelvic ring fractures. Am Surg 2003;69: 1019—23. 10. Egger M, Smith GD, O’Rourke K. Rationale, potentials, and promise of systematic reviews. In: Egger M, Smith GD, Altman DG, editors. Systematic reviews in health care–— meta-analysis in context. 2nd ed., London: BMJ Publishing; 2001. p. 3—19. 11. Fankhauser F, Boldin C, Schippinger G, et al. A new locking plate for unstable fractures of the proximal humerus. Clin Orthop 2005;176—81. 12. Goldhahn S, Audige L, Helfet DL, Hanson B. Pathways to evidence-based knowledge in orthopaedic surgery: an international survey of AO course participants. Int Orthop 2005;29:59—64. 13. Grassi FA, Tajana MS, D’Angelo F. Management of midclavicular fractures: comparison between nonoperative treatment and open intramedullary fixation in 80 patients. J Trauma 2001;50:1096—100. 14. Griffin D. Randomisation in orthopaedic surgical studies–—are there special problems which prevent the acquisition of randomised evidence of clinical effectiveness in orthopaedic surgery? Master Thesis. Department of Public Health and Primary Care, Faculty of Medicine, University of Cambridge; 2000. 15. Griffin D, Audige L. Common statistical methods in orthopaedic clinical studies. Clin Orthop 2003;413:70—9. 16. Gurwitz JH, Sykora K, Mamdani M, et al. Reader’s guide to critical appraisal of cohort studies. 1. Role and design. BMJ 2005;330:895—7. 17. Hartz A, Marsh JL. Methodologic issues in observational studies. Clin Orthop 2003;33—42. 18. Littenberg B, Weinstein LP, McCarren M, et al. Closed fractures of the tibial shaft. A meta-analysis of three methods of treatment [see comments]. J Bone Joint Surg Am 1998;80: 174—83.
L. Audige ´ et al. 19. Mamdani M, Sykora K, Li P, et al. Reader’s guide to critical appraisal of cohort studies. 2. Assessing potential for confounding. BMJ 2005;330:960—2. 20. McCulloch P, Taylor I, Sasako M, et al. Randomised trials in surgery: problems and possible solutions. BMJ 2002;324: 1448—51. 21. Normand SL, Sykora K, Li P, et al. Readers guide to critical appraisal of cohort studies. 3. Analytical strategies to reduce confounding. BMJ 2005;330:1021—3. 22. Papaloizos MY, Fusetti C, Christen T, et al. Minimally invasive fixation versus conservative treatment of undisplaced scaphoid fractures: a cost-effectiveness study. J Hand Surg [Br] 2004;29:116—9. 23. Robinson CM, Adams CI, Craig M, et al. Implant-related fractures of the femur following hip fracture surgery. J Bone Joint Surg Am 2002;84-A:1116—22. 24. Rudicel S, Esdaile J. The randomized clinical trial in orthopaedics: obligation or option? J Bone Joint Surg [Am] 1985;67:1284—93. 25. Sackett DL, Straus SE, Richardson WS, et al. Evidence-based medicine. How to practice and teach EBM, 2nd ed., Edinburgh, London: Churchill Livingstone; 2000. 26. Schutz M, Muller M, Regazzoni P, et al. Use of the less invasive stabilization system (LISS) in patients with distal femoral (AO33) fractures: a prospective multicenter study. Arch Orthop Trauma Surg 2005. 27. Solomon MJ, McLeod RS. Should we be performing more randomized controlled trials evaluating surgical operations? Surgery 1995;118:459—67. 28. Solomon MJ, McLeod RS. Surgery and the randomised controlled trial: past, present and future [see comments]. Med J Aust 1998;169:380—3. 29. Tennent TD, Calder PR, Salisbury RD, et al. The operative management of displaced intra-articular fractures of the calcaneum: a two-centre study using a defined protocol. Injury 2001;32:491—6.