- Email: [email protected]
Noncompliance in lifestyle intervention studies: the instrumental variable method provides insight into the bias
Journal of Clinical Epidemiology 63 (2010) 900e906
Noncompliance in lifestyle intervention studies: the instrumental variable method provides insight into the bias Emmy M. Hertogha, A. Jantine Schuitb,c, Petra H.M. Peetersa, Evelyn M. Monninkhofa,* a
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, STR 6.131, PO Box 85500, 3508 GA, Utrecht, The Netherlands b National Institute for Public Health and the Environment, Bilthoven, The Netherlands c Institute for Health Science, VU University Amsterdam, The Netherlands Accepted 29 October 2009
Abstract Objective: In lifestyle intervention trials, participants of the control group often change their behavior despite the request to maintain their usual lifestyle pattern. These changes in the control group and changes in addition to the intended in the intervention group can lead to undesirable confounding effects. Study Design and Setting: We address several considerations for study design to prevent noncompliance or minimize its effects. Furthermore, we demonstrate how the instrumental variable method can give insight into the extent of bias introduced by noncompliance in randomized trials, within the context of the Sex Hormones and Physical Exercise study. Results: Noncompliance can be prevented by measures taken in the design phase of a study, for example, limited duration of the study, clear recommendations, power calculation, intensity of the intervention, involvement of the control group, waiting-list control group, and single-consent design nested within an observational study. When nevertheless noncompliance does occur, the instrumental variable method estimates the intervention effect of treatment among the compliers. Conclusion: Noncompliance can seriously affect validity of lifestyle trial results. Its occurrence should be prevented by taking measures during the design phase of a study. The instrumental variable method can give insight into confounding by noncompliance in randomized trials. Ó 2010 Published by Elsevier Inc. Keywords: Noncompliance; Lifestyle intervention study; Instrumental variable method; Behavioral changes; Physical activity; Dietary habits
1. Introduction In a randomized controlled trial (RCT), randomization serves to ensure that all confounding factors are evenly distributed between the intervention and control groups. In theory, this will result in two equal groups and all outcome differences between the groups can be attributed to the intervention. To obtain an optimal intervention effect, researchers usually focus on the intervention group and whether they are compliant with the imposed protocol. However, in lifestyle intervention trials, often the control group also changes their behavior despite the request to maintain their usual lifestyle pattern [1]. This phenomenon of subjects in the control group adopting the intervention behavior has extensively been described by van Sluijs et al. [2]. Unexpected changes in the behavior of the control group are probably
* Corresponding author. Tel.: þ31-88-755-9379; fax: þ31-88-7555480. E-mail address: [email protected] (E.M. Monninkhof). 0895-4356/$ e see front matter Ó 2010 Published by Elsevier Inc. doi: 10.1016/j.jclinepi.2009.10.007
caused by the effect of study participation (the Hawthorne effect) and the effect of measurements. People willing to participate in a lifestyle intervention study are thought to be more aware of their lifestyle behavior and the subsequent health implications and have the desire to change their behavior [3,4]. The physical examinations and questionnaires used during the study can even raise their awareness. Studies have shown that awareness and feedback on personal lifestyle patterns are important for behavioral change and that measurements can be an effective intervention method to improve physical activity levels [2,5]. Changes toward a healthier lifestyle should be encouraged in the general population. However, when studying the effect of an intervention within an RCT, this kind of noncompliance can jeopardize the results and should, therefore, be monitored carefully. Lifestyle factors such as physical activity, dietary intake, smoking, and alcohol consumption have related determinants [eg, self-efficacy (ie, belief of competence), social norms] and occur in particular patterns [6,7]. Because these associated lifestyle factors are often related to the outcome of interest, trial
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
What is new? Noncompliance in lifestyle intervention trials can seriously affect validity and should be monitored carefully. This article outlines several measures that can be taken during the design phase of a lifestyle intervention trial to prevent noncompliance or minimize its effects. The instrumental variable method can give insight into the extent of bias introduced by noncompliance in randomized controlled trials.
results might also be affected by changes in these factors (which should also be judged as noncompliance). Controlling for associated factors might not be desirable, especially when these factors are intermediates in the causal pathway. RCTs are often used to investigate the effect of lifestyle interventions. However, because of the risk of noncompliance, this is not always an ideal method. Because better alternatives are lacking, the best option at present is to prevent noncompliance as much as possible and to gain insight into the extent of bias that is introduced by its occurrence. In this article, we will illustrate noncompliance within the Sex Hormones and Physical Exercise (SHAPE) study [8]. We will address considerations for study design that can prevent noncompliance as much as possible. Furthermore, we will introduce the instrumental variable method [9,10]. This method can give insight into the extent of bias introduced by noncompliance in RCTs by estimating the intervention effect among the people who comply with the treatment.
2. Illustrative study: the SHAPE study The SHAPE study is an RCT designed to assess the effect of a 12-month physical activity intervention on sex hormone levels known to be associated with breast cancer risk in postmenopausal women [8,11]. The study included 189 healthy low-active postmenopausal women aged between 50 and 69 years. Participants were assigned to the exercise intervention group (n 5 96) or control group (n 5 93). Randomization was blocked on two categories of waist circumference: !92 and >92 cm. The intervention consisted of a combined endurance and strength training program over a period of 12 months. Twice a week, participants of the exercise group gathered for a group session of 1 hour supervised by a qualified sports instructor, with an intensity classified as 5.5 metabolic equivalent (MET, ie, the ratio of the work metabolic rate to the resting metabolic rate). Furthermore, once a week, participants conducted an individual session of brisk walking or cycling
901
for at least 30 minutes. The control group was requested to retain their habitual physical activity pattern. Both intervention and control groups were asked to maintain their usual food intake. Lifestyle factors were recorded at baseline and after 12 months. Habitual physical activity was measured with the Modified Baecke Questionnaire [12]. This questionnaire includes questions about household activities, sports, and leisure time activities over the past year. All items result in a separate score that incorporates activity duration, frequency, and an intensity code based on energy expenditure. Summing the household score, sport score, and leisure time activity score results in a continuous overall unitless activity score. In addition, we calculated the MET-hours per week spent on at least moderate intensive activities (>4 MET), by coding the sports and leisure time activities reported in the questionnaire according to the Ainsworth compendium of physical activities, a coding scheme for classifying physical activity by rate of energy expenditure [13,14]. Food intake was measured with a food frequency questionnaire [15,16]. Smoking behavior, although known to be associated with physical activity, was not measured in this study because smoking in the past 12 months was an exclusion criterion and, therefore, not relevant for this study population. Compliance with the exercise prescription was defined as taking part in at least 70% of the group sessions, which equaled an increase of at least 7.7 MET-hours per week (1 hour 2 times per week 5.5 MET 70%). 2.1. SHAPE study: results and compliance At baseline, women in the intervention and control groups were similar with respect to age (59 years) and habitual physical activity level (Modified Baecke Questionnaire score of 8.5 and 4.6 MET-hours per week spent on at least moderate intensive activities). Despite randomization, differences were present for body composition, education, and some items on the food frequency questionnaire (total energy and carbohydrate intake) (Table 1). During the study, six women (one from the intervention group and five from the control group) dropped out because of personal problems or time constraints. In the intervention group, 63% of the women adhered to at least 70% of the group sessions. Seventeen women stopped attending the group sessions before the end of the study (nine because of medical reasons, three because of personal problems, three because of time constraints, one because of lack of motivation, and one without a specified reason). Because holidays were an important reason for absence at group sessions, women were asked to compensate these missed sessions by performing moderate intensive activities at least three times a week during their holidays. In the control group, 11 women reported that they had not maintained their habitual lifestyle pattern (five started
902
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
Table 1 Baseline characteristics Characteristics Age (yr), mean (SD) Body mass index (kg/m2), mean (SD) Body fat (%), mean (SD) Systolic blood pressure (mm Hg), mean (SD) Education, n (%) Primary school Technical/professional school Secondary school Academic education Food intakea, mean (SD)/ median (range) Total energy (kJ/d) Protein (g/d) Total fat (g/d) Saturated fat (g/d) Cholesterol (mg/d) Carbohydrate (g/d) Alcohol (g/d) Dietary fiber (g/d) Fat (% of energy) Protein (% of energy) Carbohydrate (% of energy) Alcohol (% of energy) Physical activity, median (range) Modified Baecke Questionnaire score Household score Sport score Leisure score MET-hours per weekb
Intervention group (n 5 96)
Control group (n 5 93)
58.9 (4.6) 26.6 (2.9)
58.4 (4.2) 27.3 (3.6)
39.8 (4.5) 132.5 (17.9)
40.9 (5.8) 133.4 (17.8)
5 (5.2) 29 (30.2)
5 (5.4) 29 (31.2)
38 (39.6) 24 (25.0)
20 (21.5) 39 (41.9)
7,817.9 72.3 75.4 28.4 180.5 203.6 7.5 22.2 35.2 16.0 44.5
(1,946.2) 8,096.5 (1,788.0) (15.1) 72.3 (15.3) (25.7) 75.9 (22.9) (11.3) 29.2 (10.5) (65.4) 179.7 (72.8) (51.8) 221.3 (57.7) (0.0e53.9) 5.3 (0.0e75.2) (5.5) 23.6 (6.8) (5.1) 34.4 (5.7) (2.5) 15.3 (2.1) (6.1) 46.4 (6.0)
2.6 (0.0e21.3)
2.0 (0.0e33.2)
8.3 (0.5e27.1)
8.8 (0.8e32.5)
2.3 0.0 5.3 4.9
2.2 0.0 5.5 4.3
(0.5e3.0) (0.0e10.7) (0.0e24.7) (0.0e120.0)
(0.8e3.0) (0.0e4.1) (0.0e29.6) (0.0e70.7)
Abbreviations: SD, standard deviation; MET, metabolic equivalent. a Information on food intake is missing for one woman in the intervention group. b MET-hours per week spent on at least moderate intensive (>4 MET) activities.
exercising, four went on a strict diet, one started both exercising and a diet, and one started smoking). Table 2 shows the absolute changes in food intake and habitual physical activity adjusted for baseline differences in both groups. Food intake in the intervention group did not change remarkably. However, in the control group, overall food intake declined considerably. Even when we excluded women who reported starting a strict diet, the declines in food intake remained (data not shown). The physical activity level in the intervention group rose as a result of the exercise program. Furthermore, women in the intervention group increased their time spent on leisure time activities even more than that was proposed by the study protocol. Although to a lesser extent, the control group also became more physically active, especially by increasing their time spent on leisure time activities (Table 2).
The increased level of physical activity in the control group showed that motivation and awareness about the beneficial effects of physical activity could also induce some behavioral change. Although this effect is challenging our trial results, its impact on public health should not be ignored. Overall, 74.5% of women in the intervention group and 17.6% of women in the control group complied with the exercise prescription of >7.7 MET-hours per week of at least moderate intensive activities.
3. Considerations for study design The best way of dealing with noncompliance is to prevent its occurrence. Here, we will discuss several measures that can be taken during the design phase of a study. 3.1. Duration of the study The duration of a study should be as short as possible to reach better compliance in both the intervention and control groups. The study should be long enough to draw valid conclusions regarding the study question but no longer than necessary to avoid substantial noncompliance. Most people who voluntarily participate in a lifestyle intervention study are motivated to change their behavior [4]. As a result, randomization into the control group can be disappointing and may lead to poor compliance [18]. Maintaining habitual behavioral patterns in the control group will be much easier when the duration of the study is short. In lifestyle intervention studies, it is often seen that motivation and subsequent results are more favorable in the beginning of the study. Within the 1-year SHAPE study, effects on weight, waist circumference, and body fat were larger at 4 months than at 12 months [19]. Researchers should consider whether a longer duration of the study will result in substantially larger effects. 3.2. Clear recommendations Adherence can be improved by giving very clear recommendations about the expected behavior of the participants, not only at start but also throughout the study period. Understanding of the study goals will lead to better compliance and is ethically required. Researchers should not restrict recommendations to the lifestyle factor under study but also pay attention to associated lifestyle factors. The SHAPE study, for instance, might have benefited from recommendations on maintaining habitual food intake, especially in the control group. However, by extensively instructing the control group and by performing intermediate measurements, awareness may be raised, which subsequently may induce change of behavior. For a further discussion of this topic, see Section 3.6. 3.3. Power One of the problems caused by noncompliance is the difficulty to detect significant differences between the
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
903
Table 2 Mean baseline values and absolute changes in the intervention and control groups, calculated by linear regression analysis and adjusted for baseline differencesa Intervention group
Control group
Characteristics
Baseline (n 5 94/93)b,c
Absolute change (95% CI)
Baseline (n 5 87)b
Absolute change (95% CI)
Total energy (kJ/d) Protein (g/d) Total fat (g/d) Saturated fat (g/d) Cholesterol (mg/d) Carbohydrate (g/d) Alcohol (g/d) Dietary fiber (g/d) Fat (% of energy) Protein (% of energy) Carbohydrate (% of energy) Alcohol (% of energy)
7,801.2 72.1 75.2 28.4 179.7 203.4 11.5 22.1 35.1 16.03 44.6 4.23
26.6 0.7 0.9 1.2 0.3 3.4 1.3 0.3 0.4 0.03 1.0 0.40
8,078.5 72.0 75.8 29.1 180.4 220.4 10.4 23.6 34.4 15.31 46.3 3.92
445.1 3.7 3.0 2.3 12.9 13.2 1.4 0.7 0.6 0.01 0.3 0.36
9.74 2.22 0.98 6.55
6.17 0.03 3.74 2.41
9.67 2.22 0.84 6.60
1.51 0.03 0.26 1.21
Modified Baecke Questionnaire score Household score Sport score Leisure score
(97.0, 43.8) (1.5, 0.1) (2.0, 0.1) (1.7, 0.8) (3.6, 4.2) (1.2, 5.7) (1.8, 0.7) (0.0, 0.7) (0.8, 0.1) (0.12, 0.18) (0.7, 1.2) (0.63, 0.17) (5.76, (0.02, (3.61, (1.95,
6.58) 0.03) 3.87) 2.87)
(515.5, 374.8) (4.5, 2.9) (4.0, 1.9) (2.8, 1.8) (16.9, 9.0) (15.5, 11.0) (2.0, 0.9) (1.1, 0.4) (0.2, 0.9) (0.16, 0.15) (0.5, 0.0) (0.59, 0.12) (1.10, (0.03, (0.13, (0.75,
1.93) 0.04) 0.39) 1.66)
Abbreviation: CI, confidence interval. a Because linear regression analysis is used to calculate changes, we present means instead of medians. Absolute changes were calculated by linear regression analysis, with the difference between measurements at baseline and 12 months as the dependent variable and the group of randomization and the measurement at baseline as independent variables. By adjusting for baseline differences, regression to the mean will be prevented and will not be falsely interpreted as an intervention effect [17]. b Total numbers are lower than in those in Table 1 because only cases with complete information were included in the linear regression analysis. c Information on food intake is missing for one woman in the intervention group.
intervention and control groups. During the design phase of a study, this attrition of the main intervention effect should be taken into account when performing power calculations. 3.4. Intensity of the intervention To overcome the Hawthorne effect and the effect of measurements, the intervention must be intensive enough to induce a measurable effect. The SHAPE study did not find favorable effects on sex hormone levels by the exercise intervention [11]. This might partly be caused by the fact that women in the control group changed their food intake and slightly their physical activity level. A more intensive intervention could have resulted in a larger contrast between the intervention and control groups, despite noncompliance in the control group.
showed indeed no increase of physical activity in the control group, besides participation in the stretching sessions. However, this approach did not prevent changes of associated lifestyle factors. In the control group, there was a decrease in total energy intake of 511 kJ/d [21], comparable to the SHAPE study. Another option is to consider a waiting-list control group, that is, participants in the control group will also receive the intervention but after a specified waiting period. Usually, a waiting-list control group is used for ethical reasons, because withholding participants from a specific intervention may be considered unethical. In case of a lifestyle intervention study with limited duration, a waiting-list control group might avoid noncompliance by postponing behavioral changes until the study period has ended.
3.5. Involvement of control group
3.6. Single-consent design nested within an observational study
Participants of a lifestyle intervention study are willing to change their behavior and often prefer to be randomized into the intervention group. To prevent the control group from making changes in their lifestyle behavior, an option is to make them feel more involved. As an example, in the trial of McTiernan et al. [20], who also examined the effect of a 1-year exercise program on endogenous sex hormone levels, control participants were offered a light stretching program. This might have demotivated them to change their exercise behavior in other ways. This study
The ideal situation in a lifestyle intervention study would be a control group that is unaware of the presence of a trial and, subsequently, unaware of the study goals and other treatment options. The prerandomization design or the Zelen design [22], in which participants of the control group are not asked for consent and the trial is not mentioned [23,24], can provide this situation. However, this design needs some adaptations in case of a lifestyle trial, because participants of both the intervention and control groups should be screened for inclusion and exclusion
904
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
criteria, and lifestyle behavior should be measured during the study, which will reveal the presence of a study. Campbell et al. [25] modified the prerandomization design into a single-consent design nested within an observational study to overcome these problems. In their study to assess the effectiveness of a combined physical and behavioral intervention for knee joint osteoarthritis, consent for followup was obtained from all participants before randomization within the context of an observational study on arthritis. After randomization, the control group was not informed about the existence of the trial and consent for the experimental treatment was obtained only from the intervention group. This design can also be applied to lifestyle intervention studies in the general population. However, selective decline of allocation should be handled with caution. People declining participation in the intervention group are likely to have specific characteristics. Therefore, either reallocating them to the control group or excluding them from the study might lead to inequality of groups. Finally, contact between participants of the intervention and control groups should be avoided to make sure that the existence of the intervention program remains unknown to the control group.
4. Statistical considerations In the next paragraph, we will discuss the instrumental variable method that can give insight into the extent of bias introduced by noncompliance in RCTs. Our objective is to give a comprehensible nontechnical description of this method and an example of its application. More profound background about the statistical methods can be found in publications of Angrist et al. [26], Brookhart et al. [27], Dunn et al. [28], Greenland [9], Hernan and Robins [29], Martens et al. [10], Rassen et al. [30,31], and others. Because the sex hormone data of the SHAPE study were not normally distributed and baseline differences were present, these data are too complicated for an illustrative example. Therefore, we chose to assess the effect of the SHAPE exercise program on systolic blood pressure. 4.1. Instrumental variable method The most often used analysis of RCTs is intention-totreat. In this analysis, the outcomes of participants assigned to the intervention and control groups are compared, irrespective of their actual compliance. In case of a substantial level of noncompliance, this will lead to a biased estimate Instrumental variable
Determinant
Outcome
randomization
physical activity
systolic blood pressure
Fig. 1. Illustration of instrumental variable method. The instrumental variable should be related to the outcome only by means of the determinant.
of the intervention effect. The approach often used to estimate the maximal treatment effect is the as-treated or perprotocol analysis. However, these methods are prone to selection bias because compliers and noncompliers may have different characteristics [32]. The method of instrumental variables can adjust for the effect of noncompliance in RCTs [9,10]. It estimates the effect of treatment among the compliers. The use of instrumental variables is well known in econometrics; however, its use in medical research is still limited. The instrumental variable method estimates the causal effect of the determinant on the outcome (outcome 5 a þ b determinant). This determinant is explained by the relationship between the determinant and another variable, the instrumental variable (determinant 5 g þ d instrumental variable). The instrumental variable should be related to the outcome only by means of the determinant. No direct relationship or indirect relationship through other variables should be present (Fig. 1). In case of an RCT, the instrumental variable is the random allocation to the intervention of interest. In its simplest form, the instrumental variable estimate can be obtained by dividing the intention-to-treat estimate by the difference between proportions of participants who comply with the treatment in the intervention and control groups [9,10]. Therefore, this method can only be applied when measurements of compliance are available for both the intervention and control groups. Furthermore, one should assume that there are no defiers who will always take the opposite treatment to that assigned (ie, the monotonicity assumption) [30,32]. The instrumental variable method has, for instance, been applied in the field of smoking cessation and its effect on birth weight and weight gain [33,34]. Because randomization to quitting or continuing to smoke is unethical, participants were randomized to a behavioral program encouraging smoking cessation or to a control group without intervention. Both groups in this study were prone to noncompliance, that is, some participants in the intervention group continued smoking and some participants in the control group stopped smoking. By dividing the intention-to-treat estimate by the difference in probabilities of quitting across the intervention and control groups, the effect of actual quitting of smoking was estimated. In a study of Mock et al. [35], the effect of exercise on fatigue in breast cancer patients was investigated with an RCT. No treatment effect was observed using the conventional intention-to-treat analysis, in part, because of noncompliance. However, when estimating the effect adjusted for noncompliance using the method of instrumental variables, the exercise intervention was found to be effective in reducing fatigue. The validity of the instrumental variable method rests on several important assumptions [9,10]. First, the instrumental variable should be causally associated with the determinant. In case of an RCT, this means that randomization should lead to different exposures in the intervention and control groups. Second, the relationship between the
X=1 X=0.745
X=0.176 X=0
physical activity compliance
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
randomization
-4.4 mm Hg
systolic blood pressure
control
intervention
-2.5 mm Hg
Fig. 2. The instrumental variable method applied to the Sex Hormones and Physical Exercise study. The solid regression line represents the intentionto-treat effect of physical activity on systolic blood pressure; the dotted line represents the instrumental variable estimate, that is, the extrapolation of the effect estimate when all participants would have complied with the protocol.
instrumental variable and the determinant should not be confounded by other variables. In case of an RCT, the randomization procedure (instrumental variable) is fully controlled by the researcher and, therefore, not confounded. Third, the most crucial assumption is that the instrumental variable should not influence the outcome, neither directly nor indirectly by its relationship with other variables. In case of an RCT, this means that randomization should not affect the outcome besides through the allocated exposure of interest. Consequently, when applying the instrumental variable method to an RCT, the first two assumptions are automatically fulfilled. Violation of the third assumption can often not be completely ruled out. 4.2. Application to the SHAPE study Exercise might prevent the development of hypertension and can lower blood pressure in normotensive and hypertensive adults. Meta-analyses have shown training-mediated decreases in systolic blood pressure, irrespective of baseline blood pressure, between 3.4 and 4.7 mm Hg [36]. In the SHAPE study, mean systolic blood pressure at baseline was 133.0 mm Hg (intervention group 132.5 mm Hg; control group 133.4 mm Hg). After 12 months of study participation, systolic blood pressure decreased to 128.8 mm Hg in the intervention group and 131.3 mm Hg in the control group. The intention-to-treat effect of participation in the intervention group is 2.5 mm Hg (95% confidence interval [CI]: 8.0, 3.0); per-protocol analysis yields an effect of 2.9 mm Hg (95% CI: 9.2, 3.3). In the intervention group, 74.5% complied with the exercise prescription of an increase of at least 7.7 MET-hours per week of moderate intensive activities, and in the control group, 17.6% reached this level of exercise. The instrumental variable estimation is 2.5/(0.745 0.176) 5 4.4 mm Hg, indicating that actually exercising changes average systolic blood
905
pressure by 4.4 mm Hg (95% CI: 14.1, 5.2), assuming that the effect is constant among the entire population (ie, effect homogeneity). These results are in accordance with those of meta-analyses. Figure 2 illustrates the instrumental variable method [10]. The solid regression line represents the intention-totreat effect of physical activity on systolic blood pressure; the dotted line represents the instrumental variable estimate, that is, the extrapolation of the effect estimate when all participants would have complied with the protocol. In the SHAPE study, we also observed changes in dietary intake, besides changes in physical activity. Energy restriction might reduce blood pressure indirectly because of weight loss [37]. Therefore, the third assumption of the instrumental variable method, that is, offering an exercise program has no effect on systolic blood pressure other than by means of changing physical activity, is problematic in the SHAPE study. Correction for change in energy intake, however, is difficult because of its dual character. Besides its direct effect on blood pressure, it is also a mediator in the relation between physical activity and blood pressure. An increase in physical activity can lead to a negative energy balance, resulting in an increased energy demand. Because energy restriction is thought to reduce blood pressure via weight loss and body weight did not change remarkably during the study and was almost equal in both groups, we assume that there was no large bias introduced. However, because the third assumption cannot be statistically verified, possible disturbing effects should always be considered carefully. Any violation of the assumptions will result in amplification of residual bias. 5. Conclusions Noncompliance in the intervention and control groups can seriously affect trial results, leading to spurious conclusions. Researchers should focus on preventing its occurrence by taking measures during the design phase of a study. When noncompliance does occur, it should be monitored carefully to take it into account in the analysis. The instrumental variable method can be used to get insight into the extent of bias introduced by noncompliance in RCTs. Although this method can serve as a valuable tool, it should be applied with caution and the underlying assumptions should be considered carefully.
Acknowledgments The authors gratefully acknowledge Edwin Martens and Wiebe Pestman (Centre for Biostatistics, Utrecht University, The Netherlands) for their statistical consultation. The authors also thank Jos Twisk (Institute for Health Science, VU University Amsterdam, The Netherlands) for his valuable comments regarding considerations for study design. The trial is sponsored by the Dutch Cancer Society (project number UU 2003-2793).
906
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
References [1] Napolitano MA, Whiteley JA, Papandonatos G, Dutton G, Farrell NC, Albrecht A, et al. Outcomes from the women’s wellness project: a community-focused physical activity trial for women. Prev Med 2006;43:447e53. [2] van Sluijs EM, van Poppel MN, Twisk JW, van Mechelen W. Physical activity measurements affected participants’ behavior in a randomized controlled trial. J Clin Epidemiol 2006;59:404e11. [3] Lerman Y, Shemer J. Epidemiologic characteristics of participants and nonparticipants in health-promotion programs. J Occup Environ Med 1996;38:535e8. [4] Lakerveld J, Ijzelenberg W, van Tulder MW, Hellemans IM, Rauwerda JA, van Rossum AC, et al. Motives for (not) participating in a lifestyle intervention trial. BMC Med Res Methodol 2008;8:17. [5] Proper KI, van der Beek AJ, Hildebrandt VH, Twisk JW, van Mechelen W. Short term effect of feedback on fitness and health measurements on self reported appraisal of the stage of change. Br J Sports Med 2003;37:529e34. [6] Ma J, Betts NM, Hampl JS. Clustering of lifestyle behaviors: the relationship between cigarette smoking, alcohol consumption, and dietary intake. Am J Health Promot 2000;15:107e17. [7] Schuit AJ, van Loon AJ, Tijhuis M, Ocke M. Clustering of lifestyle risk factors in a general adult population. Prev Med 2002;35:219e24. [8] Monninkhof EM, Peeters PH, Schuit AJ. Design of the sex hormones and physical exercise (SHAPE) study. BMC Public Health 2007;7:232. [9] Greenland S. An introduction to instrumental variables for epidemiologists. Int J Epidemiol 2000;29:722e9. [10] Martens EP, Pestman WR, de Boer A, Belitser SV, Klungel OH. Instrumental variables: application and limitations. Epidemiology 2006;17:260e7. [11] Monninkhof EM, Velthuis MJ, Peeters PH, Twisk JW, Schuit AJ. Effect of exercise on postmenopausal sex hormone levels and role of body fat: a randomized controlled trial. J Clin Oncol 2009;27:4492e9. [12] Voorrips LE, Ravelli AC, Dongelmans PC, Deurenberg P, van Staveren WA. A physical activity questionnaire for the elderly. Med Sci Sports Exerc 1991;23:974e9. [13] Ainsworth BE, Haskell WL, Leon AS, Jacobs DR Jr, Montoye HJ, Sallis JF, et al. Compendium of physical activities: classification of energy costs of human physical activities. Med Sci Sports Exerc 1993;25:71e80. [14] Ainsworth BE, Haskell WL, Whitt MC, Irwin ML, Swartz AM, Strath SJ, et al. Compendium of physical activities: an update of activity codes and MET intensities. Med Sci Sports Exerc 2000;32: S498e504. [15] Feunekes GI, van Staveren WA, De Vries JH, Burema J, Hautvast JG. Relative and biomarker-based validity of a food-frequency questionnaire estimating intake of fats and cholesterol. Am J Clin Nutr 1993;58:489e96. [16] Feunekes IJ, van Staveren WA, Graveland F, De Vos J, Burema J. Reproducibility of a semiquantitative food frequency questionnaire to assess the intake of fats and cholesterol in The Netherlands. Int J Food Sci Nutr 1995;46:117e23. [17] Twisk JWR. Applied longitudinal data analysis for epidemiology: a practical guide. Cambridge, UK: Cambridge University Press; 2003. [18] Torgerson DJ, Sibbald B. Understanding controlled trials. What is a patient preference trial? BMJ 1998;316:360.
[19] Velthuis MJ, Schuit AJ, Peeters PH, Monninkhof EM. Exercise program affects body composition but not weight in postmenopausal women. Menopause 2009;16:777e84. [20] McTiernan A, Ulrich CM, Yancey D, Slate S, Nakamura H, Oestreicher N, et al. The Physical Activity for Total Health (PATH) study: rationale and design. Med Sci Sports Exerc 1999;31:1307e12. [21] Rhew I, Yasui Y, Sorensen B, Ulrich CM, Neuhouser ML, Tworoger SS, et al. Effects of an exercise intervention on other health behaviors in overweight/obese post-menopausal women. Contemp Clin Trials 2007;28:472e81. [22] Zelen M. A new design for randomized clinical trials. N Engl J Med 1979;300:1242e5. [23] Homer CS. Using the Zelen design in randomized controlled trials: debates and controversies. J Adv Nurs 2002;38:200e7. [24] Torgerson DJ, Roland M. What is Zelen’s design? BMJ 1998;316: 606. [25] Campbell R, Peters T, Grant C, Quilty B, Dieppe P. Adapting the randomized consent (Zelen) design for trials of behavioural interventions for chronic disease: feasibility study. J Health Serv Res Policy 2005;10: 220e5. [26] Angrist JD, Imbens GW, Rubin DB. Identification of causal effects using instrumental variables. J Am Stat Assoc 1996;91:444e55. [27] Brookhart MA, Wang PS, Solomon DH, Schneeweiss S. Evaluating short-term drug effects using a physician-specific prescribing preference as an instrumental variable. Epidemiology 2006;17:268e75. [28] Dunn G, Maracy M, Tomenson B. Estimating treatment effects from randomized clinical trials with noncompliance and loss to follow-up: the role of instrumental variable methods. Stat Methods Med Res 2005;14:369e95. [29] Hernan MA, Robins JM. Instruments for causal inference: an epidemiologist’s dream? Epidemiology 2006;17:360e72. [30] Rassen JA, Brookhart MA, Glynn RJ, Mittleman MA, Schneeweiss S. Instrumental variables I: instrumental variables exploit natural variation in nonexperimental data to estimate causal relationships. J Clin Epidemiol 2009;62:1226e32. [31] Rassen JA, Brookhart MA, Glynn RJ, Mittleman MA, Schneeweiss S. Instrumental variables II: instrumental variable application-in 25 variations, the physician prescribing preference generally was strong and reduced covariate imbalance. J Clin Epidemiol 2009;62:1233e41. [32] Little RJ, Long Q, Lin X. A comparison of methods for estimating the causal effect of a treatment in randomized clinical trials subject to noncompliance. Biometrics 2009;65:640e9. [33] Permutt T, Hebel JR. Simultaneous-equation estimation in a clinical trial of the effect of smoking on birth weight. Biometrics 1989;45:619e22. [34] Eisenberg D, Quinn BC. Estimating the effect of smoking cessation on weight gain: an instrumental variable approach. Health Serv Res 2006;41:2255e66. [35] Mock V, Frangakis C, Davidson NE, Ropka ME, Pickett M, Poniatowski B, et al. Exercise manages fatigue during breast cancer treatment: a randomized controlled trial. Psychooncology 2005;14: 464e77. [36] Pescatello LS, Franklin BA, Fagard R, Farquhar WB, Kelley GA, Ray CA. American College of Sports Medicine position stand. Exercise and hypertension. Med Sci Sports Exerc 2004;36:533e53. [37] Neter JE, Stam BE, Kok FJ, Grobbee DE, Geleijnse JM. Influence of weight reduction on blood pressure: a meta-analysis of randomized controlled trials. Hypertension 2003;42:878e84.
Noncompliance in lifestyle intervention studies: the instrumental variable method provides insight into the bias Emmy M. Hertogha, A. Jantine Schuitb,c, Petra H.M. Peetersa, Evelyn M. Monninkhofa,* a
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, STR 6.131, PO Box 85500, 3508 GA, Utrecht, The Netherlands b National Institute for Public Health and the Environment, Bilthoven, The Netherlands c Institute for Health Science, VU University Amsterdam, The Netherlands Accepted 29 October 2009
Abstract Objective: In lifestyle intervention trials, participants of the control group often change their behavior despite the request to maintain their usual lifestyle pattern. These changes in the control group and changes in addition to the intended in the intervention group can lead to undesirable confounding effects. Study Design and Setting: We address several considerations for study design to prevent noncompliance or minimize its effects. Furthermore, we demonstrate how the instrumental variable method can give insight into the extent of bias introduced by noncompliance in randomized trials, within the context of the Sex Hormones and Physical Exercise study. Results: Noncompliance can be prevented by measures taken in the design phase of a study, for example, limited duration of the study, clear recommendations, power calculation, intensity of the intervention, involvement of the control group, waiting-list control group, and single-consent design nested within an observational study. When nevertheless noncompliance does occur, the instrumental variable method estimates the intervention effect of treatment among the compliers. Conclusion: Noncompliance can seriously affect validity of lifestyle trial results. Its occurrence should be prevented by taking measures during the design phase of a study. The instrumental variable method can give insight into confounding by noncompliance in randomized trials. Ó 2010 Published by Elsevier Inc. Keywords: Noncompliance; Lifestyle intervention study; Instrumental variable method; Behavioral changes; Physical activity; Dietary habits
1. Introduction In a randomized controlled trial (RCT), randomization serves to ensure that all confounding factors are evenly distributed between the intervention and control groups. In theory, this will result in two equal groups and all outcome differences between the groups can be attributed to the intervention. To obtain an optimal intervention effect, researchers usually focus on the intervention group and whether they are compliant with the imposed protocol. However, in lifestyle intervention trials, often the control group also changes their behavior despite the request to maintain their usual lifestyle pattern [1]. This phenomenon of subjects in the control group adopting the intervention behavior has extensively been described by van Sluijs et al. [2]. Unexpected changes in the behavior of the control group are probably
* Corresponding author. Tel.: þ31-88-755-9379; fax: þ31-88-7555480. E-mail address: [email protected] (E.M. Monninkhof). 0895-4356/$ e see front matter Ó 2010 Published by Elsevier Inc. doi: 10.1016/j.jclinepi.2009.10.007
caused by the effect of study participation (the Hawthorne effect) and the effect of measurements. People willing to participate in a lifestyle intervention study are thought to be more aware of their lifestyle behavior and the subsequent health implications and have the desire to change their behavior [3,4]. The physical examinations and questionnaires used during the study can even raise their awareness. Studies have shown that awareness and feedback on personal lifestyle patterns are important for behavioral change and that measurements can be an effective intervention method to improve physical activity levels [2,5]. Changes toward a healthier lifestyle should be encouraged in the general population. However, when studying the effect of an intervention within an RCT, this kind of noncompliance can jeopardize the results and should, therefore, be monitored carefully. Lifestyle factors such as physical activity, dietary intake, smoking, and alcohol consumption have related determinants [eg, self-efficacy (ie, belief of competence), social norms] and occur in particular patterns [6,7]. Because these associated lifestyle factors are often related to the outcome of interest, trial
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
What is new? Noncompliance in lifestyle intervention trials can seriously affect validity and should be monitored carefully. This article outlines several measures that can be taken during the design phase of a lifestyle intervention trial to prevent noncompliance or minimize its effects. The instrumental variable method can give insight into the extent of bias introduced by noncompliance in randomized controlled trials.
results might also be affected by changes in these factors (which should also be judged as noncompliance). Controlling for associated factors might not be desirable, especially when these factors are intermediates in the causal pathway. RCTs are often used to investigate the effect of lifestyle interventions. However, because of the risk of noncompliance, this is not always an ideal method. Because better alternatives are lacking, the best option at present is to prevent noncompliance as much as possible and to gain insight into the extent of bias that is introduced by its occurrence. In this article, we will illustrate noncompliance within the Sex Hormones and Physical Exercise (SHAPE) study [8]. We will address considerations for study design that can prevent noncompliance as much as possible. Furthermore, we will introduce the instrumental variable method [9,10]. This method can give insight into the extent of bias introduced by noncompliance in RCTs by estimating the intervention effect among the people who comply with the treatment.
2. Illustrative study: the SHAPE study The SHAPE study is an RCT designed to assess the effect of a 12-month physical activity intervention on sex hormone levels known to be associated with breast cancer risk in postmenopausal women [8,11]. The study included 189 healthy low-active postmenopausal women aged between 50 and 69 years. Participants were assigned to the exercise intervention group (n 5 96) or control group (n 5 93). Randomization was blocked on two categories of waist circumference: !92 and >92 cm. The intervention consisted of a combined endurance and strength training program over a period of 12 months. Twice a week, participants of the exercise group gathered for a group session of 1 hour supervised by a qualified sports instructor, with an intensity classified as 5.5 metabolic equivalent (MET, ie, the ratio of the work metabolic rate to the resting metabolic rate). Furthermore, once a week, participants conducted an individual session of brisk walking or cycling
901
for at least 30 minutes. The control group was requested to retain their habitual physical activity pattern. Both intervention and control groups were asked to maintain their usual food intake. Lifestyle factors were recorded at baseline and after 12 months. Habitual physical activity was measured with the Modified Baecke Questionnaire [12]. This questionnaire includes questions about household activities, sports, and leisure time activities over the past year. All items result in a separate score that incorporates activity duration, frequency, and an intensity code based on energy expenditure. Summing the household score, sport score, and leisure time activity score results in a continuous overall unitless activity score. In addition, we calculated the MET-hours per week spent on at least moderate intensive activities (>4 MET), by coding the sports and leisure time activities reported in the questionnaire according to the Ainsworth compendium of physical activities, a coding scheme for classifying physical activity by rate of energy expenditure [13,14]. Food intake was measured with a food frequency questionnaire [15,16]. Smoking behavior, although known to be associated with physical activity, was not measured in this study because smoking in the past 12 months was an exclusion criterion and, therefore, not relevant for this study population. Compliance with the exercise prescription was defined as taking part in at least 70% of the group sessions, which equaled an increase of at least 7.7 MET-hours per week (1 hour 2 times per week 5.5 MET 70%). 2.1. SHAPE study: results and compliance At baseline, women in the intervention and control groups were similar with respect to age (59 years) and habitual physical activity level (Modified Baecke Questionnaire score of 8.5 and 4.6 MET-hours per week spent on at least moderate intensive activities). Despite randomization, differences were present for body composition, education, and some items on the food frequency questionnaire (total energy and carbohydrate intake) (Table 1). During the study, six women (one from the intervention group and five from the control group) dropped out because of personal problems or time constraints. In the intervention group, 63% of the women adhered to at least 70% of the group sessions. Seventeen women stopped attending the group sessions before the end of the study (nine because of medical reasons, three because of personal problems, three because of time constraints, one because of lack of motivation, and one without a specified reason). Because holidays were an important reason for absence at group sessions, women were asked to compensate these missed sessions by performing moderate intensive activities at least three times a week during their holidays. In the control group, 11 women reported that they had not maintained their habitual lifestyle pattern (five started
902
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
Table 1 Baseline characteristics Characteristics Age (yr), mean (SD) Body mass index (kg/m2), mean (SD) Body fat (%), mean (SD) Systolic blood pressure (mm Hg), mean (SD) Education, n (%) Primary school Technical/professional school Secondary school Academic education Food intakea, mean (SD)/ median (range) Total energy (kJ/d) Protein (g/d) Total fat (g/d) Saturated fat (g/d) Cholesterol (mg/d) Carbohydrate (g/d) Alcohol (g/d) Dietary fiber (g/d) Fat (% of energy) Protein (% of energy) Carbohydrate (% of energy) Alcohol (% of energy) Physical activity, median (range) Modified Baecke Questionnaire score Household score Sport score Leisure score MET-hours per weekb
Intervention group (n 5 96)
Control group (n 5 93)
58.9 (4.6) 26.6 (2.9)
58.4 (4.2) 27.3 (3.6)
39.8 (4.5) 132.5 (17.9)
40.9 (5.8) 133.4 (17.8)
5 (5.2) 29 (30.2)
5 (5.4) 29 (31.2)
38 (39.6) 24 (25.0)
20 (21.5) 39 (41.9)
7,817.9 72.3 75.4 28.4 180.5 203.6 7.5 22.2 35.2 16.0 44.5
(1,946.2) 8,096.5 (1,788.0) (15.1) 72.3 (15.3) (25.7) 75.9 (22.9) (11.3) 29.2 (10.5) (65.4) 179.7 (72.8) (51.8) 221.3 (57.7) (0.0e53.9) 5.3 (0.0e75.2) (5.5) 23.6 (6.8) (5.1) 34.4 (5.7) (2.5) 15.3 (2.1) (6.1) 46.4 (6.0)
2.6 (0.0e21.3)
2.0 (0.0e33.2)
8.3 (0.5e27.1)
8.8 (0.8e32.5)
2.3 0.0 5.3 4.9
2.2 0.0 5.5 4.3
(0.5e3.0) (0.0e10.7) (0.0e24.7) (0.0e120.0)
(0.8e3.0) (0.0e4.1) (0.0e29.6) (0.0e70.7)
Abbreviations: SD, standard deviation; MET, metabolic equivalent. a Information on food intake is missing for one woman in the intervention group. b MET-hours per week spent on at least moderate intensive (>4 MET) activities.
exercising, four went on a strict diet, one started both exercising and a diet, and one started smoking). Table 2 shows the absolute changes in food intake and habitual physical activity adjusted for baseline differences in both groups. Food intake in the intervention group did not change remarkably. However, in the control group, overall food intake declined considerably. Even when we excluded women who reported starting a strict diet, the declines in food intake remained (data not shown). The physical activity level in the intervention group rose as a result of the exercise program. Furthermore, women in the intervention group increased their time spent on leisure time activities even more than that was proposed by the study protocol. Although to a lesser extent, the control group also became more physically active, especially by increasing their time spent on leisure time activities (Table 2).
The increased level of physical activity in the control group showed that motivation and awareness about the beneficial effects of physical activity could also induce some behavioral change. Although this effect is challenging our trial results, its impact on public health should not be ignored. Overall, 74.5% of women in the intervention group and 17.6% of women in the control group complied with the exercise prescription of >7.7 MET-hours per week of at least moderate intensive activities.
3. Considerations for study design The best way of dealing with noncompliance is to prevent its occurrence. Here, we will discuss several measures that can be taken during the design phase of a study. 3.1. Duration of the study The duration of a study should be as short as possible to reach better compliance in both the intervention and control groups. The study should be long enough to draw valid conclusions regarding the study question but no longer than necessary to avoid substantial noncompliance. Most people who voluntarily participate in a lifestyle intervention study are motivated to change their behavior [4]. As a result, randomization into the control group can be disappointing and may lead to poor compliance [18]. Maintaining habitual behavioral patterns in the control group will be much easier when the duration of the study is short. In lifestyle intervention studies, it is often seen that motivation and subsequent results are more favorable in the beginning of the study. Within the 1-year SHAPE study, effects on weight, waist circumference, and body fat were larger at 4 months than at 12 months [19]. Researchers should consider whether a longer duration of the study will result in substantially larger effects. 3.2. Clear recommendations Adherence can be improved by giving very clear recommendations about the expected behavior of the participants, not only at start but also throughout the study period. Understanding of the study goals will lead to better compliance and is ethically required. Researchers should not restrict recommendations to the lifestyle factor under study but also pay attention to associated lifestyle factors. The SHAPE study, for instance, might have benefited from recommendations on maintaining habitual food intake, especially in the control group. However, by extensively instructing the control group and by performing intermediate measurements, awareness may be raised, which subsequently may induce change of behavior. For a further discussion of this topic, see Section 3.6. 3.3. Power One of the problems caused by noncompliance is the difficulty to detect significant differences between the
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
903
Table 2 Mean baseline values and absolute changes in the intervention and control groups, calculated by linear regression analysis and adjusted for baseline differencesa Intervention group
Control group
Characteristics
Baseline (n 5 94/93)b,c
Absolute change (95% CI)
Baseline (n 5 87)b
Absolute change (95% CI)
Total energy (kJ/d) Protein (g/d) Total fat (g/d) Saturated fat (g/d) Cholesterol (mg/d) Carbohydrate (g/d) Alcohol (g/d) Dietary fiber (g/d) Fat (% of energy) Protein (% of energy) Carbohydrate (% of energy) Alcohol (% of energy)
7,801.2 72.1 75.2 28.4 179.7 203.4 11.5 22.1 35.1 16.03 44.6 4.23
26.6 0.7 0.9 1.2 0.3 3.4 1.3 0.3 0.4 0.03 1.0 0.40
8,078.5 72.0 75.8 29.1 180.4 220.4 10.4 23.6 34.4 15.31 46.3 3.92
445.1 3.7 3.0 2.3 12.9 13.2 1.4 0.7 0.6 0.01 0.3 0.36
9.74 2.22 0.98 6.55
6.17 0.03 3.74 2.41
9.67 2.22 0.84 6.60
1.51 0.03 0.26 1.21
Modified Baecke Questionnaire score Household score Sport score Leisure score
(97.0, 43.8) (1.5, 0.1) (2.0, 0.1) (1.7, 0.8) (3.6, 4.2) (1.2, 5.7) (1.8, 0.7) (0.0, 0.7) (0.8, 0.1) (0.12, 0.18) (0.7, 1.2) (0.63, 0.17) (5.76, (0.02, (3.61, (1.95,
6.58) 0.03) 3.87) 2.87)
(515.5, 374.8) (4.5, 2.9) (4.0, 1.9) (2.8, 1.8) (16.9, 9.0) (15.5, 11.0) (2.0, 0.9) (1.1, 0.4) (0.2, 0.9) (0.16, 0.15) (0.5, 0.0) (0.59, 0.12) (1.10, (0.03, (0.13, (0.75,
1.93) 0.04) 0.39) 1.66)
Abbreviation: CI, confidence interval. a Because linear regression analysis is used to calculate changes, we present means instead of medians. Absolute changes were calculated by linear regression analysis, with the difference between measurements at baseline and 12 months as the dependent variable and the group of randomization and the measurement at baseline as independent variables. By adjusting for baseline differences, regression to the mean will be prevented and will not be falsely interpreted as an intervention effect [17]. b Total numbers are lower than in those in Table 1 because only cases with complete information were included in the linear regression analysis. c Information on food intake is missing for one woman in the intervention group.
intervention and control groups. During the design phase of a study, this attrition of the main intervention effect should be taken into account when performing power calculations. 3.4. Intensity of the intervention To overcome the Hawthorne effect and the effect of measurements, the intervention must be intensive enough to induce a measurable effect. The SHAPE study did not find favorable effects on sex hormone levels by the exercise intervention [11]. This might partly be caused by the fact that women in the control group changed their food intake and slightly their physical activity level. A more intensive intervention could have resulted in a larger contrast between the intervention and control groups, despite noncompliance in the control group.
showed indeed no increase of physical activity in the control group, besides participation in the stretching sessions. However, this approach did not prevent changes of associated lifestyle factors. In the control group, there was a decrease in total energy intake of 511 kJ/d [21], comparable to the SHAPE study. Another option is to consider a waiting-list control group, that is, participants in the control group will also receive the intervention but after a specified waiting period. Usually, a waiting-list control group is used for ethical reasons, because withholding participants from a specific intervention may be considered unethical. In case of a lifestyle intervention study with limited duration, a waiting-list control group might avoid noncompliance by postponing behavioral changes until the study period has ended.
3.5. Involvement of control group
3.6. Single-consent design nested within an observational study
Participants of a lifestyle intervention study are willing to change their behavior and often prefer to be randomized into the intervention group. To prevent the control group from making changes in their lifestyle behavior, an option is to make them feel more involved. As an example, in the trial of McTiernan et al. [20], who also examined the effect of a 1-year exercise program on endogenous sex hormone levels, control participants were offered a light stretching program. This might have demotivated them to change their exercise behavior in other ways. This study
The ideal situation in a lifestyle intervention study would be a control group that is unaware of the presence of a trial and, subsequently, unaware of the study goals and other treatment options. The prerandomization design or the Zelen design [22], in which participants of the control group are not asked for consent and the trial is not mentioned [23,24], can provide this situation. However, this design needs some adaptations in case of a lifestyle trial, because participants of both the intervention and control groups should be screened for inclusion and exclusion
904
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
criteria, and lifestyle behavior should be measured during the study, which will reveal the presence of a study. Campbell et al. [25] modified the prerandomization design into a single-consent design nested within an observational study to overcome these problems. In their study to assess the effectiveness of a combined physical and behavioral intervention for knee joint osteoarthritis, consent for followup was obtained from all participants before randomization within the context of an observational study on arthritis. After randomization, the control group was not informed about the existence of the trial and consent for the experimental treatment was obtained only from the intervention group. This design can also be applied to lifestyle intervention studies in the general population. However, selective decline of allocation should be handled with caution. People declining participation in the intervention group are likely to have specific characteristics. Therefore, either reallocating them to the control group or excluding them from the study might lead to inequality of groups. Finally, contact between participants of the intervention and control groups should be avoided to make sure that the existence of the intervention program remains unknown to the control group.
4. Statistical considerations In the next paragraph, we will discuss the instrumental variable method that can give insight into the extent of bias introduced by noncompliance in RCTs. Our objective is to give a comprehensible nontechnical description of this method and an example of its application. More profound background about the statistical methods can be found in publications of Angrist et al. [26], Brookhart et al. [27], Dunn et al. [28], Greenland [9], Hernan and Robins [29], Martens et al. [10], Rassen et al. [30,31], and others. Because the sex hormone data of the SHAPE study were not normally distributed and baseline differences were present, these data are too complicated for an illustrative example. Therefore, we chose to assess the effect of the SHAPE exercise program on systolic blood pressure. 4.1. Instrumental variable method The most often used analysis of RCTs is intention-totreat. In this analysis, the outcomes of participants assigned to the intervention and control groups are compared, irrespective of their actual compliance. In case of a substantial level of noncompliance, this will lead to a biased estimate Instrumental variable
Determinant
Outcome
randomization
physical activity
systolic blood pressure
Fig. 1. Illustration of instrumental variable method. The instrumental variable should be related to the outcome only by means of the determinant.
of the intervention effect. The approach often used to estimate the maximal treatment effect is the as-treated or perprotocol analysis. However, these methods are prone to selection bias because compliers and noncompliers may have different characteristics [32]. The method of instrumental variables can adjust for the effect of noncompliance in RCTs [9,10]. It estimates the effect of treatment among the compliers. The use of instrumental variables is well known in econometrics; however, its use in medical research is still limited. The instrumental variable method estimates the causal effect of the determinant on the outcome (outcome 5 a þ b determinant). This determinant is explained by the relationship between the determinant and another variable, the instrumental variable (determinant 5 g þ d instrumental variable). The instrumental variable should be related to the outcome only by means of the determinant. No direct relationship or indirect relationship through other variables should be present (Fig. 1). In case of an RCT, the instrumental variable is the random allocation to the intervention of interest. In its simplest form, the instrumental variable estimate can be obtained by dividing the intention-to-treat estimate by the difference between proportions of participants who comply with the treatment in the intervention and control groups [9,10]. Therefore, this method can only be applied when measurements of compliance are available for both the intervention and control groups. Furthermore, one should assume that there are no defiers who will always take the opposite treatment to that assigned (ie, the monotonicity assumption) [30,32]. The instrumental variable method has, for instance, been applied in the field of smoking cessation and its effect on birth weight and weight gain [33,34]. Because randomization to quitting or continuing to smoke is unethical, participants were randomized to a behavioral program encouraging smoking cessation or to a control group without intervention. Both groups in this study were prone to noncompliance, that is, some participants in the intervention group continued smoking and some participants in the control group stopped smoking. By dividing the intention-to-treat estimate by the difference in probabilities of quitting across the intervention and control groups, the effect of actual quitting of smoking was estimated. In a study of Mock et al. [35], the effect of exercise on fatigue in breast cancer patients was investigated with an RCT. No treatment effect was observed using the conventional intention-to-treat analysis, in part, because of noncompliance. However, when estimating the effect adjusted for noncompliance using the method of instrumental variables, the exercise intervention was found to be effective in reducing fatigue. The validity of the instrumental variable method rests on several important assumptions [9,10]. First, the instrumental variable should be causally associated with the determinant. In case of an RCT, this means that randomization should lead to different exposures in the intervention and control groups. Second, the relationship between the
X=1 X=0.745
X=0.176 X=0
physical activity compliance
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
randomization
-4.4 mm Hg
systolic blood pressure
control
intervention
-2.5 mm Hg
Fig. 2. The instrumental variable method applied to the Sex Hormones and Physical Exercise study. The solid regression line represents the intentionto-treat effect of physical activity on systolic blood pressure; the dotted line represents the instrumental variable estimate, that is, the extrapolation of the effect estimate when all participants would have complied with the protocol.
instrumental variable and the determinant should not be confounded by other variables. In case of an RCT, the randomization procedure (instrumental variable) is fully controlled by the researcher and, therefore, not confounded. Third, the most crucial assumption is that the instrumental variable should not influence the outcome, neither directly nor indirectly by its relationship with other variables. In case of an RCT, this means that randomization should not affect the outcome besides through the allocated exposure of interest. Consequently, when applying the instrumental variable method to an RCT, the first two assumptions are automatically fulfilled. Violation of the third assumption can often not be completely ruled out. 4.2. Application to the SHAPE study Exercise might prevent the development of hypertension and can lower blood pressure in normotensive and hypertensive adults. Meta-analyses have shown training-mediated decreases in systolic blood pressure, irrespective of baseline blood pressure, between 3.4 and 4.7 mm Hg [36]. In the SHAPE study, mean systolic blood pressure at baseline was 133.0 mm Hg (intervention group 132.5 mm Hg; control group 133.4 mm Hg). After 12 months of study participation, systolic blood pressure decreased to 128.8 mm Hg in the intervention group and 131.3 mm Hg in the control group. The intention-to-treat effect of participation in the intervention group is 2.5 mm Hg (95% confidence interval [CI]: 8.0, 3.0); per-protocol analysis yields an effect of 2.9 mm Hg (95% CI: 9.2, 3.3). In the intervention group, 74.5% complied with the exercise prescription of an increase of at least 7.7 MET-hours per week of moderate intensive activities, and in the control group, 17.6% reached this level of exercise. The instrumental variable estimation is 2.5/(0.745 0.176) 5 4.4 mm Hg, indicating that actually exercising changes average systolic blood
905
pressure by 4.4 mm Hg (95% CI: 14.1, 5.2), assuming that the effect is constant among the entire population (ie, effect homogeneity). These results are in accordance with those of meta-analyses. Figure 2 illustrates the instrumental variable method [10]. The solid regression line represents the intention-totreat effect of physical activity on systolic blood pressure; the dotted line represents the instrumental variable estimate, that is, the extrapolation of the effect estimate when all participants would have complied with the protocol. In the SHAPE study, we also observed changes in dietary intake, besides changes in physical activity. Energy restriction might reduce blood pressure indirectly because of weight loss [37]. Therefore, the third assumption of the instrumental variable method, that is, offering an exercise program has no effect on systolic blood pressure other than by means of changing physical activity, is problematic in the SHAPE study. Correction for change in energy intake, however, is difficult because of its dual character. Besides its direct effect on blood pressure, it is also a mediator in the relation between physical activity and blood pressure. An increase in physical activity can lead to a negative energy balance, resulting in an increased energy demand. Because energy restriction is thought to reduce blood pressure via weight loss and body weight did not change remarkably during the study and was almost equal in both groups, we assume that there was no large bias introduced. However, because the third assumption cannot be statistically verified, possible disturbing effects should always be considered carefully. Any violation of the assumptions will result in amplification of residual bias. 5. Conclusions Noncompliance in the intervention and control groups can seriously affect trial results, leading to spurious conclusions. Researchers should focus on preventing its occurrence by taking measures during the design phase of a study. When noncompliance does occur, it should be monitored carefully to take it into account in the analysis. The instrumental variable method can be used to get insight into the extent of bias introduced by noncompliance in RCTs. Although this method can serve as a valuable tool, it should be applied with caution and the underlying assumptions should be considered carefully.
Acknowledgments The authors gratefully acknowledge Edwin Martens and Wiebe Pestman (Centre for Biostatistics, Utrecht University, The Netherlands) for their statistical consultation. The authors also thank Jos Twisk (Institute for Health Science, VU University Amsterdam, The Netherlands) for his valuable comments regarding considerations for study design. The trial is sponsored by the Dutch Cancer Society (project number UU 2003-2793).
906
E.M. Hertogh et al. / Journal of Clinical Epidemiology 63 (2010) 900e906
References [1] Napolitano MA, Whiteley JA, Papandonatos G, Dutton G, Farrell NC, Albrecht A, et al. Outcomes from the women’s wellness project: a community-focused physical activity trial for women. Prev Med 2006;43:447e53. [2] van Sluijs EM, van Poppel MN, Twisk JW, van Mechelen W. Physical activity measurements affected participants’ behavior in a randomized controlled trial. J Clin Epidemiol 2006;59:404e11. [3] Lerman Y, Shemer J. Epidemiologic characteristics of participants and nonparticipants in health-promotion programs. J Occup Environ Med 1996;38:535e8. [4] Lakerveld J, Ijzelenberg W, van Tulder MW, Hellemans IM, Rauwerda JA, van Rossum AC, et al. Motives for (not) participating in a lifestyle intervention trial. BMC Med Res Methodol 2008;8:17. [5] Proper KI, van der Beek AJ, Hildebrandt VH, Twisk JW, van Mechelen W. Short term effect of feedback on fitness and health measurements on self reported appraisal of the stage of change. Br J Sports Med 2003;37:529e34. [6] Ma J, Betts NM, Hampl JS. Clustering of lifestyle behaviors: the relationship between cigarette smoking, alcohol consumption, and dietary intake. Am J Health Promot 2000;15:107e17. [7] Schuit AJ, van Loon AJ, Tijhuis M, Ocke M. Clustering of lifestyle risk factors in a general adult population. Prev Med 2002;35:219e24. [8] Monninkhof EM, Peeters PH, Schuit AJ. Design of the sex hormones and physical exercise (SHAPE) study. BMC Public Health 2007;7:232. [9] Greenland S. An introduction to instrumental variables for epidemiologists. Int J Epidemiol 2000;29:722e9. [10] Martens EP, Pestman WR, de Boer A, Belitser SV, Klungel OH. Instrumental variables: application and limitations. Epidemiology 2006;17:260e7. [11] Monninkhof EM, Velthuis MJ, Peeters PH, Twisk JW, Schuit AJ. Effect of exercise on postmenopausal sex hormone levels and role of body fat: a randomized controlled trial. J Clin Oncol 2009;27:4492e9. [12] Voorrips LE, Ravelli AC, Dongelmans PC, Deurenberg P, van Staveren WA. A physical activity questionnaire for the elderly. Med Sci Sports Exerc 1991;23:974e9. [13] Ainsworth BE, Haskell WL, Leon AS, Jacobs DR Jr, Montoye HJ, Sallis JF, et al. Compendium of physical activities: classification of energy costs of human physical activities. Med Sci Sports Exerc 1993;25:71e80. [14] Ainsworth BE, Haskell WL, Whitt MC, Irwin ML, Swartz AM, Strath SJ, et al. Compendium of physical activities: an update of activity codes and MET intensities. Med Sci Sports Exerc 2000;32: S498e504. [15] Feunekes GI, van Staveren WA, De Vries JH, Burema J, Hautvast JG. Relative and biomarker-based validity of a food-frequency questionnaire estimating intake of fats and cholesterol. Am J Clin Nutr 1993;58:489e96. [16] Feunekes IJ, van Staveren WA, Graveland F, De Vos J, Burema J. Reproducibility of a semiquantitative food frequency questionnaire to assess the intake of fats and cholesterol in The Netherlands. Int J Food Sci Nutr 1995;46:117e23. [17] Twisk JWR. Applied longitudinal data analysis for epidemiology: a practical guide. Cambridge, UK: Cambridge University Press; 2003. [18] Torgerson DJ, Sibbald B. Understanding controlled trials. What is a patient preference trial? BMJ 1998;316:360.
[19] Velthuis MJ, Schuit AJ, Peeters PH, Monninkhof EM. Exercise program affects body composition but not weight in postmenopausal women. Menopause 2009;16:777e84. [20] McTiernan A, Ulrich CM, Yancey D, Slate S, Nakamura H, Oestreicher N, et al. The Physical Activity for Total Health (PATH) study: rationale and design. Med Sci Sports Exerc 1999;31:1307e12. [21] Rhew I, Yasui Y, Sorensen B, Ulrich CM, Neuhouser ML, Tworoger SS, et al. Effects of an exercise intervention on other health behaviors in overweight/obese post-menopausal women. Contemp Clin Trials 2007;28:472e81. [22] Zelen M. A new design for randomized clinical trials. N Engl J Med 1979;300:1242e5. [23] Homer CS. Using the Zelen design in randomized controlled trials: debates and controversies. J Adv Nurs 2002;38:200e7. [24] Torgerson DJ, Roland M. What is Zelen’s design? BMJ 1998;316: 606. [25] Campbell R, Peters T, Grant C, Quilty B, Dieppe P. Adapting the randomized consent (Zelen) design for trials of behavioural interventions for chronic disease: feasibility study. J Health Serv Res Policy 2005;10: 220e5. [26] Angrist JD, Imbens GW, Rubin DB. Identification of causal effects using instrumental variables. J Am Stat Assoc 1996;91:444e55. [27] Brookhart MA, Wang PS, Solomon DH, Schneeweiss S. Evaluating short-term drug effects using a physician-specific prescribing preference as an instrumental variable. Epidemiology 2006;17:268e75. [28] Dunn G, Maracy M, Tomenson B. Estimating treatment effects from randomized clinical trials with noncompliance and loss to follow-up: the role of instrumental variable methods. Stat Methods Med Res 2005;14:369e95. [29] Hernan MA, Robins JM. Instruments for causal inference: an epidemiologist’s dream? Epidemiology 2006;17:360e72. [30] Rassen JA, Brookhart MA, Glynn RJ, Mittleman MA, Schneeweiss S. Instrumental variables I: instrumental variables exploit natural variation in nonexperimental data to estimate causal relationships. J Clin Epidemiol 2009;62:1226e32. [31] Rassen JA, Brookhart MA, Glynn RJ, Mittleman MA, Schneeweiss S. Instrumental variables II: instrumental variable application-in 25 variations, the physician prescribing preference generally was strong and reduced covariate imbalance. J Clin Epidemiol 2009;62:1233e41. [32] Little RJ, Long Q, Lin X. A comparison of methods for estimating the causal effect of a treatment in randomized clinical trials subject to noncompliance. Biometrics 2009;65:640e9. [33] Permutt T, Hebel JR. Simultaneous-equation estimation in a clinical trial of the effect of smoking on birth weight. Biometrics 1989;45:619e22. [34] Eisenberg D, Quinn BC. Estimating the effect of smoking cessation on weight gain: an instrumental variable approach. Health Serv Res 2006;41:2255e66. [35] Mock V, Frangakis C, Davidson NE, Ropka ME, Pickett M, Poniatowski B, et al. Exercise manages fatigue during breast cancer treatment: a randomized controlled trial. Psychooncology 2005;14: 464e77. [36] Pescatello LS, Franklin BA, Fagard R, Farquhar WB, Kelley GA, Ray CA. American College of Sports Medicine position stand. Exercise and hypertension. Med Sci Sports Exerc 2004;36:533e53. [37] Neter JE, Stam BE, Kok FJ, Grobbee DE, Geleijnse JM. Influence of weight reduction on blood pressure: a meta-analysis of randomized controlled trials. Hypertension 2003;42:878e84.