- Email: [email protected]
Methodologic Issues in the Conduct and Interpretation of Pediatric Effectiveness Research
Methodologic Issues in the Conduct and Interpretation of Pediatric Effectiveness Research Sherrie H. Kaplan, PhD, MPH; Sheldon Greenfield, MD; Grace A. Connolly, JD; Sarah E. Barlow, MD, MPH; Richard Grand, MD Effectiveness research represents a number of methodologic challenges not shared with randomized, controlled clinical trials. This practice-based research attempts to translate clinical practices to a wide variety of different practice settings and situations and to diverse patient subgroups. However, because study designs most often used in the conduct of effectiveness research limit the ability to establish firm causal links between medical care and outcomes, it is important to address key methodologic features to generate sound, useable findings. Such features include selection of appropriate outcome measures (with a priori hypotheses linking care to the outcomes chosen), specification of appropriate primary sampling unit, specification of unit of analysis, establishment of appropriate comparison groups, and case-mix adjustment. Conduct of this type of research in pediatrics presents a number of unique methodologic concerns that either do not apply in adult medicine or are particularly acute in pediatrics. To alert policy makers and funders to the unique aspects of pediatric health services research and to provide guidance for the conduct and interpretation of pediatric effectiveness studies, we have organized and described the methodologic issues associated with the specific type of pediatric care under study (eg, specific disease-prevention, ‘‘bundled’’ care for chronic disease, care for problems with social etiologies, etc). We conclude with a summary of the methodologic steps that are critical to the conduct of sound effectiveness research in pediatrics. Ambulatory Pediatrics 2001;1:63 70
E
fficacy studies are most commonly conducted under ideal conditions and are typically designed as randomized, controlled trials (RCTs). RCTs have the advantage of examining the impact of highly defined, standardized, and often narrow interventions that are under tight investigator control. In order to make accurate causal inferences, they are usually restricted to homogeneous, often newly diagnosed patients, with unusually high or study-inflated levels of adherence to the intervention. Because they can involve withholding of services, testing of interventions that have more than negligible risk and typically eliminate patients with other medical or social problems, many children are excluded from randomized trials. In contrast, effectiveness research is conducted in highly varied practice settings; typically employs nonexperimental designs; attempts to include the ‘‘real world’’ range of patients, providers, and health care arrangements that are likely to be involved in the receipt and delivery of the care under study; and often takes the total context, rather than discrete elements of care, into account. More
recently, effectiveness research has evolved to include a great range of outcomes of care (ie, beyond clinical end points) and broader and more inclusive definitions of ‘‘care’’ (ie, beyond drugs and procedures to include health care delivery systems, physician specialties, practice guidelines, quality improvement initiatives, transition to other care venues, etc). As pointed out by Forrest et al,1 Halfon et al,2 and Lieu et al,3 this evolution has heightened the need for greater methodologic rigor and analytic sophistication in the conduct of this type of research. Absence of attention to these design and methodologic issues can compromise the use and interpretation of effectiveness research to guide clinical practice.3–11 This paper is intended to highlight the unique features of effectiveness research in pediatrics and to provide a general overview of the methodologic issues involved in the conduct and interpretation of these studies. BACKGROUND Effectiveness research in pediatrics must address 3 basic issues that, different from adults, uniquely disadvantage its application among children. First, generalizability, the principal strength of effectiveness research, is more difficult to ensure in pediatrics because of the smaller numbers of children with some of the acute and most of the chronic diseases (with rare exceptions) at any specific institution or care setting. Second and related to generalizability is the broadening and changing definition of pediatric care to include problems with social etiologies (domestic violence, guns in the household, automobile accidents, teenage pregnancy, school safety, school performance, etc) and problems with genetic origins. As the
From the Primary Care Outcomes Research Institute (Dr Kaplan, Dr Greenfield, and Ms Connolly), Tufts University School of Medicine, Boston, Mass; the Department of Pediatrics (Dr Barlow), Saint Louis University, St Louis, Mo; and the Department of Pediatrics (Dr Grand), New England Medical Center, Boston, Mass. Address correspondence to Sherrie H. Kaplan, PhD, MPH, Primary Care Outcomes Research Institute, Tufts University School of Medicine, 750 Washington St, Box 345, Boston, MA 02111 (e-mail: [email protected]). Received for publication May 4, 2000; accepted September 26, 2000. AMBULATORY PEDIATRICS Copyright q 2001 by Ambulatory Pediatric Association
63
Volume 1, Number 1 January-February 2001
64
Kaplan et al
definition of care for these problems broadens, so does the number and type of providers involved in its delivery. When an intervention involves more than a specific drug or procedure and involves more than one provider or provider type (eg, endocrinologists, dietitians, and nurse educators) to deliver the care under consideration (ie, effective juvenile diabetes care), the precise specification of the care being studied and who is providing that care become serious considerations. Further, measuring the effect of medical care versus broader social factors (eg, inadequate community services, poverty, etc) on the outcomes of care for problems with social etiologies becomes problematic. Studies of effectiveness of pediatric care can vary from the assessment of narrow and very specific interventions (such as immunizations, drugs, procedures, or diagnostic tests) to ‘‘bundled’’ or aggregated elements of care for diseases or problems (eg, all diabetes care) to the care rendered by institutions or under varying health care arrangements (eg, care for children with chronic diseases managed under health maintenance organizations versus fee-for-service arrangements). The methodologic issues involved in the conduct of sound effectiveness research, such as choice of appropriate outcome measures, primary sampling unit, unit of analysis, target of inference, and appropriate comparison groups, vary depending on the nature and breadth of the intervention under study. Finally, and perhaps most importantly, because effectiveness research does not have the luxury of considerable investigator control over or standardization of the interventions under study, the specification of a priori hypotheses, with particular attention to the anticipated amount of variance in outcomes explained by medical care (vs other exogenous factors) is critical. Absence of the specification of such hypotheses can easily lead to spurious interpretation of results of effectiveness studies. Recent methodologic advances and practical insights, some of which have been uncovered in effectiveness research among adults, provide an opportunity to address these obstacles in the conduct of effectiveness research among children. This paper discusses the following issues and steps needed to overcome them in sound studies of effective pediatric care: 1. 2. 3. 4. 5. 6.
Concise definition of the care to be evaluated. Choice of appropriate outcomes of care. Specification of a primary sampling unit. Specification of a unit of analysis. Specification of an appropriate comparison group. Definition of appropriate adjusters/covariates.
These issues are discussed in relation to the specification of the nature and type of pediatric care under study. DEFINITION OF PEDIATRIC CARE BEING EVALUATED The nature of pediatric care is changing. As evidence builds that risk factors for serious diseases in adults, such as cardiovascular diseases, may begin in childhood,12–15 screening for these diseases will increasingly become part of appropriate pediatric care.16 With advances in gene
AMBULATORY PEDIATRICS
therapy,17,18 problems with genetic origins may also soon be treated as part of routine pediatric care. As problems with social etiologies, such as domestic violence, teenage pregnancy, traffic accidents, school performance, guns in households, etc, become increasingly medicalized,19–24 providers are being asked to add these problems to routine clinical screening. Identification of ‘‘high-risk’’ families25–30 (eg, those with compromised household safety) for the purpose of preventing and managing a spectrum of health problems is also likely to become a feature of pediatric care. Effective management for these problems often includes non–health care personnel. Defining the boundaries of care for these problems (eg, which providers’ care is to be included in the assessment of effectiveness, how delivery systems or health care arrangements directly affect these problems, etc) will be critical first steps in conducting effectiveness studies. Because this first step (ie, defining the care under study) has implications for the methodologic and design considerations outlined above, we have attempted to create a taxonomy of types of pediatric care (or ‘‘interventions’’) and to examine each methodologic issue in the context of the care delivered (see Table). We have organized varying types of pediatric care into seven major categories: 1) care aimed at primary prevention; 2) assessment of drugs, procedures, diagnostic tests, etc; 3) ‘‘bundled’’ (eg, chronic disease) care; 4) care for problems with social etiologies; 5) individual provider’s care; 6) care of groups of providers (eg, generalists vs specialists); and 7) care provided under different delivery systems, different health care arrangements, etc. Primary prevention includes such care as immunizations, cholesterol screening, lead levels, hepatitis screening, genetic screening, etc. Drugs, procedures, and diagnostic tests include the range of discrete elements of care from assessment of the effectiveness of individual drugs and diagnostic tests through complex procedures such as stem cell transplantation. ‘‘Bundled’’ care (eg, all care for specific chronic diseases) includes all elements of care or management provided for a specific disease or condition. Such care is often provided within organizational units, such as cancer units for longitudinal follow-up of stem cell transplantation or diabetes clinics for management of the multiple aspects of diabetes care. Care for problems with social etiologies, such as domestic violence, teenage pregnancy, guns in households, etc, can include screening for these problems and referrals to and management by a variety of social agencies. Care by individual providers includes all elements of technical and interpersonal care delivered by an individual practitioner. Groups of providers can include specialty groups, physician networks, etc. Delivery systems or institutions can include hospitals, integrated delivery systems, health maintenance organizations, traditional fee-for-service, Medicaid managed care programs, etc. The progressive broadening of the definition of care under study from very narrow and relatively standardized, such as provision of immunizations, to all care provided
n Critical
n No treatment or stan- n No treatment or stan- n Defined gradients/ dard practice dard practice levels of care
n Not necessary
Comparison group
Case-mix adjustment
n Critical
n Patient
n Patient
Unit of analysis
QOL 5 Quality of Life
n Defined gradients/ levels of care
n Patient
n Patient
Primary sampling unit
n Critical
n Patient n Family n Provider
n Patient n Family n Provider n Patient n Family n Provider
n Patient n Family n Provider
n Broad n Broad n Aggregated n Aggregated n Medical/general ver- n Inclusive sus disease specific QOL
n Narrow n Medical n Short time window
Narrow Medical Long time window Large samples
n n n n
n Broad n Less standardized n Weighted elements
Care for Problems With Social Etiologies
Nature of outcomes
n Broad n Less standardized n Weighted elements
‘‘Bundled’’ (eg, Chronic Disease)
n Narrow n Standardized
Drugs, Procedures, Diagnostic Tests, etc.
n Narrow n Standardized
Primary Disease Prevention
Intervention Type
Nature of intervention/ care
Methodologic/Design Issues
Methodologic Issues for Varying Levels of Pediatric Interventions
n Broad n Unstandardized
Groups of Providers (eg, Generalists vs Specialists)
n Broadest n Least standardized
n Critical
n Other providers
n Patient n Family n Provider
n Provider
Patient Family Provider Group
n Critical
n Other groups
n n n n
n Provider
Patient Family Plan System/institution
n Needed
n Other systems, institutions, plans
n n n n
n System n Institution n Health plan
n Broad n Broad n Broadest n Aggregated n Aggregated n Aggregated n Medical/general ver- n Medical/general ver- n More general, less sus disease specifsus disease specifdisease specific ic QOL ic QOL
n Broad n Less standardized n Weighted elements
Individual Provider’s Care
Delivery System (Institution) or Health Care Arrangement (eg, HMO)
AMBULATORY PEDIATRICS Methodologic Issues in Pediatric Effectiveness Research 65
66
Kaplan et al
AMBULATORY PEDIATRICS
Relationship of medical care for type 1 diabetes to health-related quality of life.
to children managed under different health care delivery systems has implications for all other methodologic issues, including outcomes, unit of analysis, sampling, appropriate comparison groups, etc. CHOOSING APPROPRIATE OUTCOMES OF CARE Arguably the most important steps in conducting methodologically sound pediatric effectiveness research are the choice of appropriate outcomes measures and the specification of the nature and magnitude of the hypothesized relationship of the outcome to effective medical care. Such directional hypotheses drive the choice among mortality, physiologic measures, disease-specific measures (such as symptom experience, episodes, and self-reported disease control), disease-specific dysfunction (including bothersomeness and disease-related functional impairment), disease-related days lost from school, general health status and well-being, and overall quality of life. As noted by others,31–34 development of measures of many of these concepts among children has lagged substantially behind that for adults. Because longitudinality is a key feature of effectiveness research, sensitivity of outcomes measures to variations in medical care, assessed over a feasible time period, is key to the choice of appropriate pediatric outcome measures. Depending on the nature of the problem under study, these measures will each explain different proportions of the variation in care. For example, a successful immunization program would be expected to explain the majority of the variation in the absence of the target disease. Optimal prescription of insulin among children with diabetes may explain only a modest proportion of the variance in glycemic control (because of poor compliance, variable exercise, diet, etc). Traditionally, the majority of pediatric care has been organized to prevent or treat disease, not specifically to improve quality of life. As we and others have noted,4,35 the progression from biologic or physiologic markers to general health status and well-being and overall quality of life increases the probability that characteristics of the individual and the environment (vs effective medical care) will explain increasing proportions of the variation in outcomes. For example, as noted in diabetes, optimal use of insulin may have modest effects on glucose control but
lesser or even negative effects on outcomes such as functional status and quality of life that are more sensitive to effects of family, school, and environment (Figure). The time window needed to observe changes in outcomes related to effective care can also affect the choice of appropriate outcomes measures. For primary prevention including immunizations, cholesterol and lead screening, etc, the ideal outcomes, in this case the absence of the target disease, could require long time windows and very large sample sizes to study appropriately. As the time interval between care and outcomes lengthens, the relationship of patient or environmental characteristics to the outcomes becomes greater (ie, the proportion of variance explained by medical care becomes small). For such care, performance of the service itself (eg, proportion of children immunized or screening) in some cases has to serve as an outcome proxy for effective care. In contrast, outcomes of care that involve discrete elements, such as drugs, procedures, diagnostic tests, etc, are more feasibly measured, tend to be more narrow, are medical in nature (eg, improved pulmonary function, lower hemoglobin A1c levels, etc), and usually involve abbreviated and feasible time windows. Use of more global measures of health, such as functional status and quality of life, is warranted only if a plausible directional hypothesis can be expressed linking variations in care to clinically meaningful variations in these outcomes.4 Studies not designed with these general measures as the primary end point often lack sufficient power to detect clinically meaningful differences. Studies of effective pediatric care that is aggregated or ‘‘bundled,’’ such as that provided for a chronic disease (eg, asthma or juvenile diabetes) over a discrete time period (eg, 1 year) require a broader array of outcomes, including both clinical measures (eg, pulmonary function, glycated hemoglobin, etc) as well as disease-specific dysfunction and potentially general health status. Although more narrow clinical end points could be used to assess this care, the inclusion of a broader array of outcomes allows for assessment of the collective impact of the range of providers who address the diverse and diffuse aspects of disease management (eg, interpersonal care for children and parents, motivation and encouragement to adhere to management plans, etc). Using multiple outcomes includ-
AMBULATORY PEDIATRICS
ing clinical end points, disease-specific outcome measures, and general health status in the case of ‘‘bundled’’ care where possible and plausible, allows for the most comprehensive approach to the assessment of effectiveness of this type of care. A good example of the study of effectiveness of the multiple elements of care for a single condition was done for bronchiolitis by Perlstein et al.36 This study summarized the evidence from randomized trials and observational research that in the aggregate defined effective care for children hospitalized with bronchiolitis. The relationship of this care to outcomes (eg, recovery from the acute illness) had been examined in the literature synthesized by the investigators. Assessment of the effectiveness of care for problems with social etiologies (eg, domestic violence, guns in the household, traffic accidents, etc) represents distinct challenges for studies of pediatric care. The appropriate outcomes of such care are often determined more by patient and environmental characteristics than by effective pediatric interventions,37 especially because certain outcomes may require long time windows in which factors other than medical care intervene. In addition, documentation of these outcomes, especially for long-term stability over time, requires adequate data sources and standardized monitoring, both difficult to ensure. For example, observing for the effectiveness of counseling by pediatricians for appropriate car seat or seat belt use for young children or against drinking and driving for adolescents, may require not only cessation of the discrete behavior but also include a broader constellation of risky behaviors. Currently available general health status and health-related quality of life measures might not be relevant or sufficiently sensitive for the evaluation of such care. More attention is needed to develop appropriate outcomes of this type of pediatric care to provide a comprehensive assessment of its effectiveness. Although some measures of family functioning are available,38–40 development and testing of measures that will be sensitive to variations in effectiveness of this type of pediatric care are needed. Finally, since the target of such care is often not limited to the individual child, integration of children’s and families’ outcomes may be needed. Aggregation of outcomes measures for individual patients is somewhat controversial and challenging4; aggregation of measures across multiple individuals (eg, children and families) for the purposes of assessing effective medical care is even more complex and has not received sufficient methodologic attention. Assessment of the effectiveness of care by individual pediatricians or other pediatric care providers and groups of providers, is most commonly done for specific diseases or management problems. How to choose appropriate outcomes for such care, whether and how to aggregate specific outcome measures to create an aggregate or composite outcome score to generate ‘‘profiles,’’ or summary measures to gauge effectiveness of care delivered by the group are rarely addressed. An example that highlights these issues is reflected in the Provider Recognition Program of the American Diabetes Association.41 This program profiles both individual and groups of practicing pe-
Methodologic Issues in Pediatric Effectiveness Research
67
diatricians to reward performance of effective diabetes care, using standardized process and outcome measures. It has begun to address the complex methodologic issues involved in conducting fair and accurate comparisons of individual or groups of pediatric providers, including adequate sampling, appropriate adjustment, etc. However, the creation of sound ‘‘profiling’’ techniques and methods for evaluating the effectiveness of individual provider’s care remains in its infancy. The effectiveness of care by delivery systems or under varying health care arrangements is most commonly evaluated using general health status outcomes measures.33–35,42 However, these measures, as noted earlier, are often substantially influenced by variables other than effective medical care. Development of methods for aggregation across disease-specific measures may provide greater sensitivity to variations in pediatric care. Percentage of Variance in Outcomes Explained by Pediatric Care The first criterion for choosing outcomes measures to evaluate effectiveness of pediatric care should be a causal, directionally hypothesized relationship to that care. A second related criterion involves the plausibility of attributing a reasonable amount of variation in outcomes to medical care. For example, assume that outcomes could be used to evaluate effective pediatric care for asthma in children and are causally arrayed in a manner similar to those illustrated for juvenile diabetes in the Figure. Let us assume further that effective medical care (prescribing of antiasthma medications, counseling regarding reduction of triggers, etc.) explains 50% of the variance in asthmarelated events, with the residual accounted for by patient and environmental characteristics. In a cascading fashion, if reduced asthma events further explain 50% of the variance in asthma-related symptoms, which in turn explain 50% of the variance in energy and school performance, which then explains 50% of the improvement in general health perceptions, which in turn explains 50% of the improvement in overall quality of life, then the amount of variation in overall quality of life explained by effective pediatric care would be (0.5)6 5 1.56%. This estimate may be conservative, since effective pediatric asthma care could affect quality of life via other causal routes. However, it may also be an overestimation, since the assumed percent of variance explained at each level was set at 50%, a reasonably high estimate. Target of the Intervention and Outcome Measures For some types of pediatric interventions, the child is not the sole or even the primary target of the intervention. If the parents, family, some combination, or even the community43–45 is the focus of the intervention under study, then the outcomes measures chosen must evaluate variations in the health of those individuals or groups. As noted above and by others,32 measures that evaluate the health outcomes of families (even those that explore the variation within or methods for aggregation of individual family members) have not received substantial scientific atten-
68
Kaplan et al
tion. Assessment of the health of communities was the original purpose for which general health status measures were designed. However, examining the sensitivity of these measures to variations in pediatric care has not been done. Further measures development and testing for application in which the target of the interventions delivered by pediatric providers is not limited to the individual child are needed. Outstanding research has been done that operates outside the traditional pediatric care delivery system. Research such as that done by Robinson et al46,47 that is conducted in the community setting (eg, schools) has shown that such interventions can have a direct effect on children’s health outcomes (eg, body mass index and cardiovascular risk factors). However, the link between such interventions and effective pediatric care remains tenuous. Although the successful results of studies such as those by Robinson et al and others could be moved into the pediatrician’s office and studied for effectiveness in that venue, such research remains to be done. In summary, as we have noted elsewhere,4 the same guidelines apply to the use of outcomes to evaluate effectiveness of pediatric care as apply to causal inference in any research effort. First, observed differences in outcomes measures should be large enough to be clinically convincing, especially when measures of health-related quality of life are used. Such measures are often not clinically intuitive. For these outcomes measures, a modest ‘‘effect size,’’ or the ratio of an observed difference to the standard deviation, ranging from 0.40 to 0.60, is often considered meaningful. Second, a progressive positive relationship should be observed between care and outcomes (ie, increasingly better care leads to increasingly improved outcomes). Third, temporal relationships must exist between care and outcomes (ie, care precedes changes in outcomes). Fourth, observed differences in outcomes must have a clinically plausible relationship to effective care (as specified in directional, a priori hypotheses). In addition, the issues in choosing among outcomes measures to assess the effectiveness of pediatric care include: 1) the target of care (child, parent, and family) must be specified and measures chosen appropriately; 2) if parents’ health or health-related quality of life is a target outcome, hypothesized relationship between care and the child’s and parent’s health must both be specified; 3) because the validity of proxy reports of children’s health is currently in doubt, children’s self-reported measures should be used where possible; 4) measures that will vary in relation to effective medical care over a time window should be chosen; 5) outcome measures should be used for problems with social etiologies that not only reflect modification of the discrete behavior at a single, short-term time interval but assess the degree to which constellations of related risky behaviors are reduced over time. IDENTIFYING THE PRIMARY SAMPLING UNIT The target and scope of inference about effective care can range from individual children and families to all institutions or health care delivery systems that share com-
AMBULATORY PEDIATRICS
mon features. Specifying the target of inference is key to the choice of the primary sampling unit, as illustrated by the CATCH study48 in which both the school and the individual child were studied. In that study, the target of the intervention and the primary sampling unit were the school, the level in which its effectiveness was demonstrated. As shown in the Table, primary sampling units increasingly broaden and vary with pediatric intervention type. Studies, for example, attempting to attribute variations in care and its outcomes to individual providers, must sample providers (and patients/families within providers). Studies wishing to attribute effectiveness of care to different types of delivery systems (institutions, health care arrangements, etc) must sample systems, arrangements, etc and patients within those systems to make accurate and credible inferences about those systems. DETERMINING THE UNIT OF ANALYSIS The unit of analysis for effectiveness studies may or may not be the same as the primary sampling unit. Patient level analyses can be performed in studies in which the provider was the primary sampling unit. Hierarchical modeling techniques49 are often used in studies with nested designs (eg, children within families, patients with practices or institutions) to account for the nonrandom clustering of patient characteristics by individual providers or provider groups. Pediatric effectiveness studies comparing health care systems or institutions should also take the nonrandom distribution of patient or provider characteristics into account when the health care system is the unit of analysis. IDENTIFYING APPROPRIATE COMPARISON GROUPS Choice of the most appropriate outcome measures, sampling strategies, and analytic methods will not overcome one of the often most difficult to address study design issues—identifying a comparison group. Interpreting results of effectiveness studies is often handicapped by the absence of such groups. Because these studies most commonly are conducted with nonexperimental designs, choice of a comparison group becomes critical to the interpretation of study results. This group is particularly important when a group under study (eg, a health plan, a group of providers, a type of treatment) has been intentionally selected because it represents an extreme of care (ie, most or potentially least effective) in which regression to the mean over time is of concern. CASE-MIX ADJUSTMENT The importance of case-mix adjustment is most critical when the care being compared is likely to be delivered to very heterogeneous groups of patients. For example, casemix adjustment for comparisons involving hospitals or health care management arrangements may be less important because the heterogeneity within the institution or organization reduces differences between groups attributable to characteristics other than care. That is, the within group variation in these patient and provider characteris-
AMBULATORY PEDIATRICS
Methodologic Issues in Pediatric Effectiveness Research
69
tics is greater than the between group variation. Such is not the case, however, when comparing individual physicians or groups of physicians. In this case, patients with certain characteristics may ‘‘cluster’’ or be more represented in one practice or group than another. In a recent study of physician practices, adjustment for case-mix, along with individual practice variations, using hierarchical modeling techniques, eliminated observed differences between generalists and specialists seeing patients with diabetes.41 Therefore, for pediatric effectiveness studies, adequate case-mix adjustment is key to the interpretation of findings.
through schools, this type of study could be mounted through, or involve, school settings, as illustrated by Robinson et al.46,47 Performed well, pediatric effectiveness studies have the potential to provide the credible evidence for efforts to improve the quality of pediatric care. Randomized trials have benefited from the kind of methodologic attention that is only now being brought to the conduct of effectiveness studies.
SUMMARY
REFERENCE
Effectiveness research is messy, complicated, and frustrating. It can lead to more uncertainty and controversy regarding interpretation and generalizability (ostensibly its strength) than RCTs. However, with careful attention to the kinds of methodologic considerations noted above, this kind of research can bridge the gap between efficacy studies and real-world practice to promote effective pediatric care. Certain requirements must be met to conduct methodologically sound, practically useful effectiveness in research in pediatrics. First, development of well-tested outcome measures in pediatrics has lagged far behind that for adults. More attention to the development, testing and use of children’s self-reported health status measures, particularly young children, is needed. Better measures and methods for approaching the assessment of the health status of families are also needed. Second, case-mix adjustment measures that go beyond diagnoses (such as those available in and applied to claims data50,51) and include multiple, aggregate clinical severity variables must also be developed for children.52 Third, more methodologic attention is needed to ensure that the target of inference (ie, whose care is being evaluated) is specified at the earliest design stages of effectiveness studies in order to choose the appropriate sampling units, comparison groups, and analytic methods. Clustering effects and the nonrandom distribution of children in health care settings are critical elements in the correct interpretation of links between care and outcomes. Finally, sound effectiveness research requires the specification of directional a priori hypotheses linking care to outcomes. Especially for more general outcomes, such as health related quality of life, the amount of variance to be explained by effective care must be specified at the outset of the study. The absence of such hypotheses specifying the precise relationship of effective care to these outcomes has led to substantial confusion and difficulties in interpretation of studies in both pediatrics and adult medicine where health related quality of life has been used as an outcome.4 To understand how care versus child, family, or environmental characteristics affects such measures over time, long-term longitudinal observational studies, such as those done among adults53–55 are needed. Although such studies represent unique methodologic challenges, because most children are relatively easily identifiable
1. Forrest CB, Simpson L, Clancy C. Child health services research. Challenges and opportunities. JAMA. 1997;277:1787– 1793. 2. Halfon N, Schuster M, Valentine W, McGlynn E. Improving quality of healthcare for children: implementing the results of AHSR research agenda conference. Health Serv Res. 1998;33: 955–976. 3. Lieu TA, Newman TB. Issues in studying the effectiveness of health services for children. Health Serv Res. 1998;33:1041– 1058. 4. Kaplan SH, Kravitz RL, Greenfield S. A critique of current uses of health status for the assessment of treatment effectiveness and quality of care. Med Care. 2000;38(9 suppl):II-184–II-191. 5. Bergman DA. Evidence-based guidelines and critical pathways for quality improvement, principles, and perspectives. Pediatrics. 1999;103:225–232. 6. Audet AM, Greenfield S, Field M. Medical practice guidelines: current activities and future directions. Ann Intern Med. 1990; 113:709–714. 7. Greenfield S. The distinction between guidelines and performance measures: why doctors comply with the latter and not the former. Med Care. In press. 8. Woolf SH, Battista RN, Anderson GM, et al. Assessing the clinical effectiveness of preventive maneuvers: analytic principles and systemic methods in reviewing evidence and developing clinical practice recommendations. J Clin Epidemiol. 1990;43: 891–905. 9. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluation. Lancet. 1993;342:1317–1322. 10. Bauchner H, Simpson L. Specific issues related to developing, disseminating, and implementing paediatric practice guidelines for physicians, patients, families, and other stakeholders. Health Serv Res. 1998;33:1161–1177. 11. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993;342:1317–1322. 12. Johnston FE. Health implications of childhood obesity. Ann Intern Med. 1985;103:1068–1072. 13. Must A, Jacques PF, Dallal GE, et al. Long-term mortality of overweight adolescents. A follow-up of the Harvard Growth Study of 1922 to 1935. N Engl J Med. 1992;327:1350–1355. 14. Gunnell DJ, Frankel SJ, Nanchahal K, et al. Childhood obesity and adult cardiovascular mortality: a 57-y follow-up study based on the Boyd Orr cohort. Am J Clin Nutr. 1998;67:1111–1118. 15. Sinaiko AR, Donahue RP, Jacobs DR Jr, Prineas RJ. Relation of weight and rate of increase in weight during childhood and adolescence to body size, blood pressure, fasting insulin, and lipids in young adults. The Minneapolis Children’s Blood Pressure Study. Circulation. 1999;99:1471–1476. 16. Labarthe DR. Prevention of cardiovascular risk factors in the first place. Prev Med. 1999;29:S72–S78. 17. Ledley FD. Gene therapy in pediatric medicine. Adv Pediatr. 1996;43:1–25.
ACKNOWLEDGMENT This work was supported by a grant from the Agency for Healthcare Research and Quality.
70
Kaplan et al
18. Lien YH, Lai LW. Entering the gene therapy era. J Formos Med Assoc. 1999;98:718–721. 19. Barkin S, Ryan G, Gelberg L. What pediatricians can do to further youth violence prevention—a qualitative study. Inj Prev. 1999;5:53–58. 20. Oatis PJ, Fenn Buderer NM, Cummings P, Fleitz R. Pediatric practice based evaluation of the steps to prevent firearm injury program. Inj Prev. 1999;5:48–52. 21. High PC, LaGasse L, Becker S, et al. Literacy promotion in primary care pediatrics: can we make a difference? Pediatrics. 2000;105:927–934. 22. Anderson C. Violence in television commercials during nonviolent programming. The 1996 Major League Baseball playoffs. JAMA. 1997;278:1045–1046. 23. Teran-Santos J, Jimenez-Gomez A, Cordero-Guevara J. The association between sleep apnea and the risk of traffic accidents. Cooperative Group Burgos-Santander. N Engl J Med. 1999;340: 847–851. 24. Carter DM, Felice ME, Rosoff J, et al. When children have children: the teen pregnancy predicament. Am J Prev Med. 1994;10:108–113. 25. Grossman DC, Rivara FP. Injury control in childhood. Pediatr Clin North Am. 1992;39:471–485. 26. Alexander DE, Gwyther RE. Alcoholism in adolescents and their families. Family-focused assessment and management. Pediatr Clin North Am. 1995;42:217–234. 27. Casey PH, Bradley RH. The impact of the home environment on children’s development: clinical relevance for the pediatrician. J Dev Behav Pediatr. 1982;3:146–152. 28. Guyer B, Hughart N, Strobino D, et al. Assessing the impact of pediatric-based development services on infants, families, and clinicians: challenges to evaluating the Health Steps Program. Pediatrics. 2000;105:E33. 29. Horwitz SM, Morgenstern H, DiPietro L, Morrison CL. Determinants of pediatric injuries. Am J Dis Child. 1988;142:605– 611. 30. Schmitt BD. Seven deadly sins of childhood: advising parents about difficult developmental phases. Child Abuse Negl. 1987; 11:421–432. 31. Christakis DA. Methodologic challenges in pediatric outcomes research. Ambul Pediatr. 2001;1:59–62. 32. Starfield B, Riley AW, Green BF, et al. The adolescent child health and illness profile. A population-based measure of health. Med Care. 1995;33:553–566. 33. Vivier PM, Bernier JA, Starfield B. Current approaches to measuring health outcomes in pediatric research. Curr Opin Pediatr. 1994;6:530–537. 34. Starfield B, Bergner M, Ensminger M, et al. Adolescent health status measurement: development of the child health and illness profile. Pediatrics. 1993;91:430–435. 35. Wilson IB. Clinical understanding and clinical implications of response shift. Soc Sci Med. 1999;48:1577–1588. 36. Perlstein PH, Kotagal UR, Bolling C, et al. Evaluation of an evidence-based guideline for bronchiolitis. Pediatrics. 1999; 104:1334–1341. 37. Robinson TN, Chen HL, Killen JD. Television and music video exposure and risk of adolescent alcohol abuse. Pediatrics. 1998; 102:E54.
AMBULATORY PEDIATRICS 38. Halvorsen JG. Self-report family assessment instruments: an evaluative review. Fam Pract Res J. 1991;11:21–55. 39. Roelofse R, Middleton MR. The family functioning in adolescence questionnaire: a measure of psychosocial family health during adolescence. J Adolesc. 1985;8:33–45. 40. Alexander BB, Johnson SB, Carter RL. A psychometric study of the family adaptability and cohesion evaluation scales. J Abnorm Child Psychol. 1984;12:199–207. 41. Greenfield S, Kahn R, Ninomiya J, Griffith JL. Evaluating practice differences in the delivery of diabetes care: effects of specialty, patient characteristics and individual practice variation. Ann Intern Med. In press. 42. McHorney CA, Ware JE Jr, Lu JF, Sherbourne CD. The MOS 36-item Short-Form Health Survey (SF-36): III. Tests of data quality, scaling assumptions, and reliability across diverse patient groups. Med Care. 1994;32:40–66. 43. Margolis PA, Lannon CM, Stevens R, et al. Linking clinical and public health approaches to improve access to health care for socially disadvantaged mothers and children. A feasibility study. Arch Pediatr Adolesc Med. 1996;150:815–821. 44. Waterman SH, Hill LL, Robyn B, et al. A model immunization demonstration for preschoolers in an inner-city barrio, San Diego, California, 1992–1994. Am J Prev Med. 1996;12:8–13. 45. Gielen AC, Wilson ME, Faden RR, et al. In-home injury prevention practices for infants and toddlers: the role of parental beliefs, barriers, and housing quality. Health Educ Q. 1995;22: 85–95. 46. Killen JD, Robinson TN, Telch MJ, et al. The Stanford Adolescent Heart Health Program. Health Educ Q. 1989;16:263– 283. 47. Robinson TN. Reducing children’s television viewing to prevent obesity: Randomized controlled trial. JAMA. 1999;282:1561– 1567. 48. Resnicow K, Robinson TN, Frank E. Advances and future directions for school-based health promotion research: commentary on the CATCH intervention trial. Prev Med. 1996;25:378– 383. 49. Bryk AS, Raudenbush SW, eds. Hierarchical Linear Models. Beverly Hills, Calif: Sage Publications; 1992. 50. Weiner JP, Garfield BH, Powe NR, et al. Ambulatory care practice variation within a Medicaid population. Health Serv Res. 1996;30:751–770. 51. Tayne SM, Schwartz RM. An evaluation of pediatric-modified diagnosis-related groups. Health Care Finan Rev. 1993;15:51– 70. 52. Barlow SE, Kaplan SH, Parsons SK, et al. Pilot studies of severity scale development in pediatric chronic disease. In press. 53. Christen WG, Gaziano JM, Hennekens CH. Design of physicians’ health study II—a randomized trial of beta-carotene, vitamins E and C, and multivitamins, in prevention of cancer, cardiovascular disease, and eye disease, and review of results of completed trails. Ann Epidemiol. 2000;10:125–134. 54. Futterman LG, Lemberg L. The Framingham Heart Study: a pivotal legacy of the last millennium. Am Heart J. 2000;9:147– 151. 55. Fine JT, Colditz GA, Coakley EH, et al. A prospective study of weight change and health-related quality of life in women. JAMA. 1999;282:2136–2142.
E
fficacy studies are most commonly conducted under ideal conditions and are typically designed as randomized, controlled trials (RCTs). RCTs have the advantage of examining the impact of highly defined, standardized, and often narrow interventions that are under tight investigator control. In order to make accurate causal inferences, they are usually restricted to homogeneous, often newly diagnosed patients, with unusually high or study-inflated levels of adherence to the intervention. Because they can involve withholding of services, testing of interventions that have more than negligible risk and typically eliminate patients with other medical or social problems, many children are excluded from randomized trials. In contrast, effectiveness research is conducted in highly varied practice settings; typically employs nonexperimental designs; attempts to include the ‘‘real world’’ range of patients, providers, and health care arrangements that are likely to be involved in the receipt and delivery of the care under study; and often takes the total context, rather than discrete elements of care, into account. More
recently, effectiveness research has evolved to include a great range of outcomes of care (ie, beyond clinical end points) and broader and more inclusive definitions of ‘‘care’’ (ie, beyond drugs and procedures to include health care delivery systems, physician specialties, practice guidelines, quality improvement initiatives, transition to other care venues, etc). As pointed out by Forrest et al,1 Halfon et al,2 and Lieu et al,3 this evolution has heightened the need for greater methodologic rigor and analytic sophistication in the conduct of this type of research. Absence of attention to these design and methodologic issues can compromise the use and interpretation of effectiveness research to guide clinical practice.3–11 This paper is intended to highlight the unique features of effectiveness research in pediatrics and to provide a general overview of the methodologic issues involved in the conduct and interpretation of these studies. BACKGROUND Effectiveness research in pediatrics must address 3 basic issues that, different from adults, uniquely disadvantage its application among children. First, generalizability, the principal strength of effectiveness research, is more difficult to ensure in pediatrics because of the smaller numbers of children with some of the acute and most of the chronic diseases (with rare exceptions) at any specific institution or care setting. Second and related to generalizability is the broadening and changing definition of pediatric care to include problems with social etiologies (domestic violence, guns in the household, automobile accidents, teenage pregnancy, school safety, school performance, etc) and problems with genetic origins. As the
From the Primary Care Outcomes Research Institute (Dr Kaplan, Dr Greenfield, and Ms Connolly), Tufts University School of Medicine, Boston, Mass; the Department of Pediatrics (Dr Barlow), Saint Louis University, St Louis, Mo; and the Department of Pediatrics (Dr Grand), New England Medical Center, Boston, Mass. Address correspondence to Sherrie H. Kaplan, PhD, MPH, Primary Care Outcomes Research Institute, Tufts University School of Medicine, 750 Washington St, Box 345, Boston, MA 02111 (e-mail: [email protected]). Received for publication May 4, 2000; accepted September 26, 2000. AMBULATORY PEDIATRICS Copyright q 2001 by Ambulatory Pediatric Association
63
Volume 1, Number 1 January-February 2001
64
Kaplan et al
definition of care for these problems broadens, so does the number and type of providers involved in its delivery. When an intervention involves more than a specific drug or procedure and involves more than one provider or provider type (eg, endocrinologists, dietitians, and nurse educators) to deliver the care under consideration (ie, effective juvenile diabetes care), the precise specification of the care being studied and who is providing that care become serious considerations. Further, measuring the effect of medical care versus broader social factors (eg, inadequate community services, poverty, etc) on the outcomes of care for problems with social etiologies becomes problematic. Studies of effectiveness of pediatric care can vary from the assessment of narrow and very specific interventions (such as immunizations, drugs, procedures, or diagnostic tests) to ‘‘bundled’’ or aggregated elements of care for diseases or problems (eg, all diabetes care) to the care rendered by institutions or under varying health care arrangements (eg, care for children with chronic diseases managed under health maintenance organizations versus fee-for-service arrangements). The methodologic issues involved in the conduct of sound effectiveness research, such as choice of appropriate outcome measures, primary sampling unit, unit of analysis, target of inference, and appropriate comparison groups, vary depending on the nature and breadth of the intervention under study. Finally, and perhaps most importantly, because effectiveness research does not have the luxury of considerable investigator control over or standardization of the interventions under study, the specification of a priori hypotheses, with particular attention to the anticipated amount of variance in outcomes explained by medical care (vs other exogenous factors) is critical. Absence of the specification of such hypotheses can easily lead to spurious interpretation of results of effectiveness studies. Recent methodologic advances and practical insights, some of which have been uncovered in effectiveness research among adults, provide an opportunity to address these obstacles in the conduct of effectiveness research among children. This paper discusses the following issues and steps needed to overcome them in sound studies of effective pediatric care: 1. 2. 3. 4. 5. 6.
Concise definition of the care to be evaluated. Choice of appropriate outcomes of care. Specification of a primary sampling unit. Specification of a unit of analysis. Specification of an appropriate comparison group. Definition of appropriate adjusters/covariates.
These issues are discussed in relation to the specification of the nature and type of pediatric care under study. DEFINITION OF PEDIATRIC CARE BEING EVALUATED The nature of pediatric care is changing. As evidence builds that risk factors for serious diseases in adults, such as cardiovascular diseases, may begin in childhood,12–15 screening for these diseases will increasingly become part of appropriate pediatric care.16 With advances in gene
AMBULATORY PEDIATRICS
therapy,17,18 problems with genetic origins may also soon be treated as part of routine pediatric care. As problems with social etiologies, such as domestic violence, teenage pregnancy, traffic accidents, school performance, guns in households, etc, become increasingly medicalized,19–24 providers are being asked to add these problems to routine clinical screening. Identification of ‘‘high-risk’’ families25–30 (eg, those with compromised household safety) for the purpose of preventing and managing a spectrum of health problems is also likely to become a feature of pediatric care. Effective management for these problems often includes non–health care personnel. Defining the boundaries of care for these problems (eg, which providers’ care is to be included in the assessment of effectiveness, how delivery systems or health care arrangements directly affect these problems, etc) will be critical first steps in conducting effectiveness studies. Because this first step (ie, defining the care under study) has implications for the methodologic and design considerations outlined above, we have attempted to create a taxonomy of types of pediatric care (or ‘‘interventions’’) and to examine each methodologic issue in the context of the care delivered (see Table). We have organized varying types of pediatric care into seven major categories: 1) care aimed at primary prevention; 2) assessment of drugs, procedures, diagnostic tests, etc; 3) ‘‘bundled’’ (eg, chronic disease) care; 4) care for problems with social etiologies; 5) individual provider’s care; 6) care of groups of providers (eg, generalists vs specialists); and 7) care provided under different delivery systems, different health care arrangements, etc. Primary prevention includes such care as immunizations, cholesterol screening, lead levels, hepatitis screening, genetic screening, etc. Drugs, procedures, and diagnostic tests include the range of discrete elements of care from assessment of the effectiveness of individual drugs and diagnostic tests through complex procedures such as stem cell transplantation. ‘‘Bundled’’ care (eg, all care for specific chronic diseases) includes all elements of care or management provided for a specific disease or condition. Such care is often provided within organizational units, such as cancer units for longitudinal follow-up of stem cell transplantation or diabetes clinics for management of the multiple aspects of diabetes care. Care for problems with social etiologies, such as domestic violence, teenage pregnancy, guns in households, etc, can include screening for these problems and referrals to and management by a variety of social agencies. Care by individual providers includes all elements of technical and interpersonal care delivered by an individual practitioner. Groups of providers can include specialty groups, physician networks, etc. Delivery systems or institutions can include hospitals, integrated delivery systems, health maintenance organizations, traditional fee-for-service, Medicaid managed care programs, etc. The progressive broadening of the definition of care under study from very narrow and relatively standardized, such as provision of immunizations, to all care provided
n Critical
n No treatment or stan- n No treatment or stan- n Defined gradients/ dard practice dard practice levels of care
n Not necessary
Comparison group
Case-mix adjustment
n Critical
n Patient
n Patient
Unit of analysis
QOL 5 Quality of Life
n Defined gradients/ levels of care
n Patient
n Patient
Primary sampling unit
n Critical
n Patient n Family n Provider
n Patient n Family n Provider n Patient n Family n Provider
n Patient n Family n Provider
n Broad n Broad n Aggregated n Aggregated n Medical/general ver- n Inclusive sus disease specific QOL
n Narrow n Medical n Short time window
Narrow Medical Long time window Large samples
n n n n
n Broad n Less standardized n Weighted elements
Care for Problems With Social Etiologies
Nature of outcomes
n Broad n Less standardized n Weighted elements
‘‘Bundled’’ (eg, Chronic Disease)
n Narrow n Standardized
Drugs, Procedures, Diagnostic Tests, etc.
n Narrow n Standardized
Primary Disease Prevention
Intervention Type
Nature of intervention/ care
Methodologic/Design Issues
Methodologic Issues for Varying Levels of Pediatric Interventions
n Broad n Unstandardized
Groups of Providers (eg, Generalists vs Specialists)
n Broadest n Least standardized
n Critical
n Other providers
n Patient n Family n Provider
n Provider
Patient Family Provider Group
n Critical
n Other groups
n n n n
n Provider
Patient Family Plan System/institution
n Needed
n Other systems, institutions, plans
n n n n
n System n Institution n Health plan
n Broad n Broad n Broadest n Aggregated n Aggregated n Aggregated n Medical/general ver- n Medical/general ver- n More general, less sus disease specifsus disease specifdisease specific ic QOL ic QOL
n Broad n Less standardized n Weighted elements
Individual Provider’s Care
Delivery System (Institution) or Health Care Arrangement (eg, HMO)
AMBULATORY PEDIATRICS Methodologic Issues in Pediatric Effectiveness Research 65
66
Kaplan et al
AMBULATORY PEDIATRICS
Relationship of medical care for type 1 diabetes to health-related quality of life.
to children managed under different health care delivery systems has implications for all other methodologic issues, including outcomes, unit of analysis, sampling, appropriate comparison groups, etc. CHOOSING APPROPRIATE OUTCOMES OF CARE Arguably the most important steps in conducting methodologically sound pediatric effectiveness research are the choice of appropriate outcomes measures and the specification of the nature and magnitude of the hypothesized relationship of the outcome to effective medical care. Such directional hypotheses drive the choice among mortality, physiologic measures, disease-specific measures (such as symptom experience, episodes, and self-reported disease control), disease-specific dysfunction (including bothersomeness and disease-related functional impairment), disease-related days lost from school, general health status and well-being, and overall quality of life. As noted by others,31–34 development of measures of many of these concepts among children has lagged substantially behind that for adults. Because longitudinality is a key feature of effectiveness research, sensitivity of outcomes measures to variations in medical care, assessed over a feasible time period, is key to the choice of appropriate pediatric outcome measures. Depending on the nature of the problem under study, these measures will each explain different proportions of the variation in care. For example, a successful immunization program would be expected to explain the majority of the variation in the absence of the target disease. Optimal prescription of insulin among children with diabetes may explain only a modest proportion of the variance in glycemic control (because of poor compliance, variable exercise, diet, etc). Traditionally, the majority of pediatric care has been organized to prevent or treat disease, not specifically to improve quality of life. As we and others have noted,4,35 the progression from biologic or physiologic markers to general health status and well-being and overall quality of life increases the probability that characteristics of the individual and the environment (vs effective medical care) will explain increasing proportions of the variation in outcomes. For example, as noted in diabetes, optimal use of insulin may have modest effects on glucose control but
lesser or even negative effects on outcomes such as functional status and quality of life that are more sensitive to effects of family, school, and environment (Figure). The time window needed to observe changes in outcomes related to effective care can also affect the choice of appropriate outcomes measures. For primary prevention including immunizations, cholesterol and lead screening, etc, the ideal outcomes, in this case the absence of the target disease, could require long time windows and very large sample sizes to study appropriately. As the time interval between care and outcomes lengthens, the relationship of patient or environmental characteristics to the outcomes becomes greater (ie, the proportion of variance explained by medical care becomes small). For such care, performance of the service itself (eg, proportion of children immunized or screening) in some cases has to serve as an outcome proxy for effective care. In contrast, outcomes of care that involve discrete elements, such as drugs, procedures, diagnostic tests, etc, are more feasibly measured, tend to be more narrow, are medical in nature (eg, improved pulmonary function, lower hemoglobin A1c levels, etc), and usually involve abbreviated and feasible time windows. Use of more global measures of health, such as functional status and quality of life, is warranted only if a plausible directional hypothesis can be expressed linking variations in care to clinically meaningful variations in these outcomes.4 Studies not designed with these general measures as the primary end point often lack sufficient power to detect clinically meaningful differences. Studies of effective pediatric care that is aggregated or ‘‘bundled,’’ such as that provided for a chronic disease (eg, asthma or juvenile diabetes) over a discrete time period (eg, 1 year) require a broader array of outcomes, including both clinical measures (eg, pulmonary function, glycated hemoglobin, etc) as well as disease-specific dysfunction and potentially general health status. Although more narrow clinical end points could be used to assess this care, the inclusion of a broader array of outcomes allows for assessment of the collective impact of the range of providers who address the diverse and diffuse aspects of disease management (eg, interpersonal care for children and parents, motivation and encouragement to adhere to management plans, etc). Using multiple outcomes includ-
AMBULATORY PEDIATRICS
ing clinical end points, disease-specific outcome measures, and general health status in the case of ‘‘bundled’’ care where possible and plausible, allows for the most comprehensive approach to the assessment of effectiveness of this type of care. A good example of the study of effectiveness of the multiple elements of care for a single condition was done for bronchiolitis by Perlstein et al.36 This study summarized the evidence from randomized trials and observational research that in the aggregate defined effective care for children hospitalized with bronchiolitis. The relationship of this care to outcomes (eg, recovery from the acute illness) had been examined in the literature synthesized by the investigators. Assessment of the effectiveness of care for problems with social etiologies (eg, domestic violence, guns in the household, traffic accidents, etc) represents distinct challenges for studies of pediatric care. The appropriate outcomes of such care are often determined more by patient and environmental characteristics than by effective pediatric interventions,37 especially because certain outcomes may require long time windows in which factors other than medical care intervene. In addition, documentation of these outcomes, especially for long-term stability over time, requires adequate data sources and standardized monitoring, both difficult to ensure. For example, observing for the effectiveness of counseling by pediatricians for appropriate car seat or seat belt use for young children or against drinking and driving for adolescents, may require not only cessation of the discrete behavior but also include a broader constellation of risky behaviors. Currently available general health status and health-related quality of life measures might not be relevant or sufficiently sensitive for the evaluation of such care. More attention is needed to develop appropriate outcomes of this type of pediatric care to provide a comprehensive assessment of its effectiveness. Although some measures of family functioning are available,38–40 development and testing of measures that will be sensitive to variations in effectiveness of this type of pediatric care are needed. Finally, since the target of such care is often not limited to the individual child, integration of children’s and families’ outcomes may be needed. Aggregation of outcomes measures for individual patients is somewhat controversial and challenging4; aggregation of measures across multiple individuals (eg, children and families) for the purposes of assessing effective medical care is even more complex and has not received sufficient methodologic attention. Assessment of the effectiveness of care by individual pediatricians or other pediatric care providers and groups of providers, is most commonly done for specific diseases or management problems. How to choose appropriate outcomes for such care, whether and how to aggregate specific outcome measures to create an aggregate or composite outcome score to generate ‘‘profiles,’’ or summary measures to gauge effectiveness of care delivered by the group are rarely addressed. An example that highlights these issues is reflected in the Provider Recognition Program of the American Diabetes Association.41 This program profiles both individual and groups of practicing pe-
Methodologic Issues in Pediatric Effectiveness Research
67
diatricians to reward performance of effective diabetes care, using standardized process and outcome measures. It has begun to address the complex methodologic issues involved in conducting fair and accurate comparisons of individual or groups of pediatric providers, including adequate sampling, appropriate adjustment, etc. However, the creation of sound ‘‘profiling’’ techniques and methods for evaluating the effectiveness of individual provider’s care remains in its infancy. The effectiveness of care by delivery systems or under varying health care arrangements is most commonly evaluated using general health status outcomes measures.33–35,42 However, these measures, as noted earlier, are often substantially influenced by variables other than effective medical care. Development of methods for aggregation across disease-specific measures may provide greater sensitivity to variations in pediatric care. Percentage of Variance in Outcomes Explained by Pediatric Care The first criterion for choosing outcomes measures to evaluate effectiveness of pediatric care should be a causal, directionally hypothesized relationship to that care. A second related criterion involves the plausibility of attributing a reasonable amount of variation in outcomes to medical care. For example, assume that outcomes could be used to evaluate effective pediatric care for asthma in children and are causally arrayed in a manner similar to those illustrated for juvenile diabetes in the Figure. Let us assume further that effective medical care (prescribing of antiasthma medications, counseling regarding reduction of triggers, etc.) explains 50% of the variance in asthmarelated events, with the residual accounted for by patient and environmental characteristics. In a cascading fashion, if reduced asthma events further explain 50% of the variance in asthma-related symptoms, which in turn explain 50% of the variance in energy and school performance, which then explains 50% of the improvement in general health perceptions, which in turn explains 50% of the improvement in overall quality of life, then the amount of variation in overall quality of life explained by effective pediatric care would be (0.5)6 5 1.56%. This estimate may be conservative, since effective pediatric asthma care could affect quality of life via other causal routes. However, it may also be an overestimation, since the assumed percent of variance explained at each level was set at 50%, a reasonably high estimate. Target of the Intervention and Outcome Measures For some types of pediatric interventions, the child is not the sole or even the primary target of the intervention. If the parents, family, some combination, or even the community43–45 is the focus of the intervention under study, then the outcomes measures chosen must evaluate variations in the health of those individuals or groups. As noted above and by others,32 measures that evaluate the health outcomes of families (even those that explore the variation within or methods for aggregation of individual family members) have not received substantial scientific atten-
68
Kaplan et al
tion. Assessment of the health of communities was the original purpose for which general health status measures were designed. However, examining the sensitivity of these measures to variations in pediatric care has not been done. Further measures development and testing for application in which the target of the interventions delivered by pediatric providers is not limited to the individual child are needed. Outstanding research has been done that operates outside the traditional pediatric care delivery system. Research such as that done by Robinson et al46,47 that is conducted in the community setting (eg, schools) has shown that such interventions can have a direct effect on children’s health outcomes (eg, body mass index and cardiovascular risk factors). However, the link between such interventions and effective pediatric care remains tenuous. Although the successful results of studies such as those by Robinson et al and others could be moved into the pediatrician’s office and studied for effectiveness in that venue, such research remains to be done. In summary, as we have noted elsewhere,4 the same guidelines apply to the use of outcomes to evaluate effectiveness of pediatric care as apply to causal inference in any research effort. First, observed differences in outcomes measures should be large enough to be clinically convincing, especially when measures of health-related quality of life are used. Such measures are often not clinically intuitive. For these outcomes measures, a modest ‘‘effect size,’’ or the ratio of an observed difference to the standard deviation, ranging from 0.40 to 0.60, is often considered meaningful. Second, a progressive positive relationship should be observed between care and outcomes (ie, increasingly better care leads to increasingly improved outcomes). Third, temporal relationships must exist between care and outcomes (ie, care precedes changes in outcomes). Fourth, observed differences in outcomes must have a clinically plausible relationship to effective care (as specified in directional, a priori hypotheses). In addition, the issues in choosing among outcomes measures to assess the effectiveness of pediatric care include: 1) the target of care (child, parent, and family) must be specified and measures chosen appropriately; 2) if parents’ health or health-related quality of life is a target outcome, hypothesized relationship between care and the child’s and parent’s health must both be specified; 3) because the validity of proxy reports of children’s health is currently in doubt, children’s self-reported measures should be used where possible; 4) measures that will vary in relation to effective medical care over a time window should be chosen; 5) outcome measures should be used for problems with social etiologies that not only reflect modification of the discrete behavior at a single, short-term time interval but assess the degree to which constellations of related risky behaviors are reduced over time. IDENTIFYING THE PRIMARY SAMPLING UNIT The target and scope of inference about effective care can range from individual children and families to all institutions or health care delivery systems that share com-
AMBULATORY PEDIATRICS
mon features. Specifying the target of inference is key to the choice of the primary sampling unit, as illustrated by the CATCH study48 in which both the school and the individual child were studied. In that study, the target of the intervention and the primary sampling unit were the school, the level in which its effectiveness was demonstrated. As shown in the Table, primary sampling units increasingly broaden and vary with pediatric intervention type. Studies, for example, attempting to attribute variations in care and its outcomes to individual providers, must sample providers (and patients/families within providers). Studies wishing to attribute effectiveness of care to different types of delivery systems (institutions, health care arrangements, etc) must sample systems, arrangements, etc and patients within those systems to make accurate and credible inferences about those systems. DETERMINING THE UNIT OF ANALYSIS The unit of analysis for effectiveness studies may or may not be the same as the primary sampling unit. Patient level analyses can be performed in studies in which the provider was the primary sampling unit. Hierarchical modeling techniques49 are often used in studies with nested designs (eg, children within families, patients with practices or institutions) to account for the nonrandom clustering of patient characteristics by individual providers or provider groups. Pediatric effectiveness studies comparing health care systems or institutions should also take the nonrandom distribution of patient or provider characteristics into account when the health care system is the unit of analysis. IDENTIFYING APPROPRIATE COMPARISON GROUPS Choice of the most appropriate outcome measures, sampling strategies, and analytic methods will not overcome one of the often most difficult to address study design issues—identifying a comparison group. Interpreting results of effectiveness studies is often handicapped by the absence of such groups. Because these studies most commonly are conducted with nonexperimental designs, choice of a comparison group becomes critical to the interpretation of study results. This group is particularly important when a group under study (eg, a health plan, a group of providers, a type of treatment) has been intentionally selected because it represents an extreme of care (ie, most or potentially least effective) in which regression to the mean over time is of concern. CASE-MIX ADJUSTMENT The importance of case-mix adjustment is most critical when the care being compared is likely to be delivered to very heterogeneous groups of patients. For example, casemix adjustment for comparisons involving hospitals or health care management arrangements may be less important because the heterogeneity within the institution or organization reduces differences between groups attributable to characteristics other than care. That is, the within group variation in these patient and provider characteris-
AMBULATORY PEDIATRICS
Methodologic Issues in Pediatric Effectiveness Research
69
tics is greater than the between group variation. Such is not the case, however, when comparing individual physicians or groups of physicians. In this case, patients with certain characteristics may ‘‘cluster’’ or be more represented in one practice or group than another. In a recent study of physician practices, adjustment for case-mix, along with individual practice variations, using hierarchical modeling techniques, eliminated observed differences between generalists and specialists seeing patients with diabetes.41 Therefore, for pediatric effectiveness studies, adequate case-mix adjustment is key to the interpretation of findings.
through schools, this type of study could be mounted through, or involve, school settings, as illustrated by Robinson et al.46,47 Performed well, pediatric effectiveness studies have the potential to provide the credible evidence for efforts to improve the quality of pediatric care. Randomized trials have benefited from the kind of methodologic attention that is only now being brought to the conduct of effectiveness studies.
SUMMARY
REFERENCE
Effectiveness research is messy, complicated, and frustrating. It can lead to more uncertainty and controversy regarding interpretation and generalizability (ostensibly its strength) than RCTs. However, with careful attention to the kinds of methodologic considerations noted above, this kind of research can bridge the gap between efficacy studies and real-world practice to promote effective pediatric care. Certain requirements must be met to conduct methodologically sound, practically useful effectiveness in research in pediatrics. First, development of well-tested outcome measures in pediatrics has lagged far behind that for adults. More attention to the development, testing and use of children’s self-reported health status measures, particularly young children, is needed. Better measures and methods for approaching the assessment of the health status of families are also needed. Second, case-mix adjustment measures that go beyond diagnoses (such as those available in and applied to claims data50,51) and include multiple, aggregate clinical severity variables must also be developed for children.52 Third, more methodologic attention is needed to ensure that the target of inference (ie, whose care is being evaluated) is specified at the earliest design stages of effectiveness studies in order to choose the appropriate sampling units, comparison groups, and analytic methods. Clustering effects and the nonrandom distribution of children in health care settings are critical elements in the correct interpretation of links between care and outcomes. Finally, sound effectiveness research requires the specification of directional a priori hypotheses linking care to outcomes. Especially for more general outcomes, such as health related quality of life, the amount of variance to be explained by effective care must be specified at the outset of the study. The absence of such hypotheses specifying the precise relationship of effective care to these outcomes has led to substantial confusion and difficulties in interpretation of studies in both pediatrics and adult medicine where health related quality of life has been used as an outcome.4 To understand how care versus child, family, or environmental characteristics affects such measures over time, long-term longitudinal observational studies, such as those done among adults53–55 are needed. Although such studies represent unique methodologic challenges, because most children are relatively easily identifiable
1. Forrest CB, Simpson L, Clancy C. Child health services research. Challenges and opportunities. JAMA. 1997;277:1787– 1793. 2. Halfon N, Schuster M, Valentine W, McGlynn E. Improving quality of healthcare for children: implementing the results of AHSR research agenda conference. Health Serv Res. 1998;33: 955–976. 3. Lieu TA, Newman TB. Issues in studying the effectiveness of health services for children. Health Serv Res. 1998;33:1041– 1058. 4. Kaplan SH, Kravitz RL, Greenfield S. A critique of current uses of health status for the assessment of treatment effectiveness and quality of care. Med Care. 2000;38(9 suppl):II-184–II-191. 5. Bergman DA. Evidence-based guidelines and critical pathways for quality improvement, principles, and perspectives. Pediatrics. 1999;103:225–232. 6. Audet AM, Greenfield S, Field M. Medical practice guidelines: current activities and future directions. Ann Intern Med. 1990; 113:709–714. 7. Greenfield S. The distinction between guidelines and performance measures: why doctors comply with the latter and not the former. Med Care. In press. 8. Woolf SH, Battista RN, Anderson GM, et al. Assessing the clinical effectiveness of preventive maneuvers: analytic principles and systemic methods in reviewing evidence and developing clinical practice recommendations. J Clin Epidemiol. 1990;43: 891–905. 9. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluation. Lancet. 1993;342:1317–1322. 10. Bauchner H, Simpson L. Specific issues related to developing, disseminating, and implementing paediatric practice guidelines for physicians, patients, families, and other stakeholders. Health Serv Res. 1998;33:1161–1177. 11. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993;342:1317–1322. 12. Johnston FE. Health implications of childhood obesity. Ann Intern Med. 1985;103:1068–1072. 13. Must A, Jacques PF, Dallal GE, et al. Long-term mortality of overweight adolescents. A follow-up of the Harvard Growth Study of 1922 to 1935. N Engl J Med. 1992;327:1350–1355. 14. Gunnell DJ, Frankel SJ, Nanchahal K, et al. Childhood obesity and adult cardiovascular mortality: a 57-y follow-up study based on the Boyd Orr cohort. Am J Clin Nutr. 1998;67:1111–1118. 15. Sinaiko AR, Donahue RP, Jacobs DR Jr, Prineas RJ. Relation of weight and rate of increase in weight during childhood and adolescence to body size, blood pressure, fasting insulin, and lipids in young adults. The Minneapolis Children’s Blood Pressure Study. Circulation. 1999;99:1471–1476. 16. Labarthe DR. Prevention of cardiovascular risk factors in the first place. Prev Med. 1999;29:S72–S78. 17. Ledley FD. Gene therapy in pediatric medicine. Adv Pediatr. 1996;43:1–25.
ACKNOWLEDGMENT This work was supported by a grant from the Agency for Healthcare Research and Quality.
70
Kaplan et al
18. Lien YH, Lai LW. Entering the gene therapy era. J Formos Med Assoc. 1999;98:718–721. 19. Barkin S, Ryan G, Gelberg L. What pediatricians can do to further youth violence prevention—a qualitative study. Inj Prev. 1999;5:53–58. 20. Oatis PJ, Fenn Buderer NM, Cummings P, Fleitz R. Pediatric practice based evaluation of the steps to prevent firearm injury program. Inj Prev. 1999;5:48–52. 21. High PC, LaGasse L, Becker S, et al. Literacy promotion in primary care pediatrics: can we make a difference? Pediatrics. 2000;105:927–934. 22. Anderson C. Violence in television commercials during nonviolent programming. The 1996 Major League Baseball playoffs. JAMA. 1997;278:1045–1046. 23. Teran-Santos J, Jimenez-Gomez A, Cordero-Guevara J. The association between sleep apnea and the risk of traffic accidents. Cooperative Group Burgos-Santander. N Engl J Med. 1999;340: 847–851. 24. Carter DM, Felice ME, Rosoff J, et al. When children have children: the teen pregnancy predicament. Am J Prev Med. 1994;10:108–113. 25. Grossman DC, Rivara FP. Injury control in childhood. Pediatr Clin North Am. 1992;39:471–485. 26. Alexander DE, Gwyther RE. Alcoholism in adolescents and their families. Family-focused assessment and management. Pediatr Clin North Am. 1995;42:217–234. 27. Casey PH, Bradley RH. The impact of the home environment on children’s development: clinical relevance for the pediatrician. J Dev Behav Pediatr. 1982;3:146–152. 28. Guyer B, Hughart N, Strobino D, et al. Assessing the impact of pediatric-based development services on infants, families, and clinicians: challenges to evaluating the Health Steps Program. Pediatrics. 2000;105:E33. 29. Horwitz SM, Morgenstern H, DiPietro L, Morrison CL. Determinants of pediatric injuries. Am J Dis Child. 1988;142:605– 611. 30. Schmitt BD. Seven deadly sins of childhood: advising parents about difficult developmental phases. Child Abuse Negl. 1987; 11:421–432. 31. Christakis DA. Methodologic challenges in pediatric outcomes research. Ambul Pediatr. 2001;1:59–62. 32. Starfield B, Riley AW, Green BF, et al. The adolescent child health and illness profile. A population-based measure of health. Med Care. 1995;33:553–566. 33. Vivier PM, Bernier JA, Starfield B. Current approaches to measuring health outcomes in pediatric research. Curr Opin Pediatr. 1994;6:530–537. 34. Starfield B, Bergner M, Ensminger M, et al. Adolescent health status measurement: development of the child health and illness profile. Pediatrics. 1993;91:430–435. 35. Wilson IB. Clinical understanding and clinical implications of response shift. Soc Sci Med. 1999;48:1577–1588. 36. Perlstein PH, Kotagal UR, Bolling C, et al. Evaluation of an evidence-based guideline for bronchiolitis. Pediatrics. 1999; 104:1334–1341. 37. Robinson TN, Chen HL, Killen JD. Television and music video exposure and risk of adolescent alcohol abuse. Pediatrics. 1998; 102:E54.
AMBULATORY PEDIATRICS 38. Halvorsen JG. Self-report family assessment instruments: an evaluative review. Fam Pract Res J. 1991;11:21–55. 39. Roelofse R, Middleton MR. The family functioning in adolescence questionnaire: a measure of psychosocial family health during adolescence. J Adolesc. 1985;8:33–45. 40. Alexander BB, Johnson SB, Carter RL. A psychometric study of the family adaptability and cohesion evaluation scales. J Abnorm Child Psychol. 1984;12:199–207. 41. Greenfield S, Kahn R, Ninomiya J, Griffith JL. Evaluating practice differences in the delivery of diabetes care: effects of specialty, patient characteristics and individual practice variation. Ann Intern Med. In press. 42. McHorney CA, Ware JE Jr, Lu JF, Sherbourne CD. The MOS 36-item Short-Form Health Survey (SF-36): III. Tests of data quality, scaling assumptions, and reliability across diverse patient groups. Med Care. 1994;32:40–66. 43. Margolis PA, Lannon CM, Stevens R, et al. Linking clinical and public health approaches to improve access to health care for socially disadvantaged mothers and children. A feasibility study. Arch Pediatr Adolesc Med. 1996;150:815–821. 44. Waterman SH, Hill LL, Robyn B, et al. A model immunization demonstration for preschoolers in an inner-city barrio, San Diego, California, 1992–1994. Am J Prev Med. 1996;12:8–13. 45. Gielen AC, Wilson ME, Faden RR, et al. In-home injury prevention practices for infants and toddlers: the role of parental beliefs, barriers, and housing quality. Health Educ Q. 1995;22: 85–95. 46. Killen JD, Robinson TN, Telch MJ, et al. The Stanford Adolescent Heart Health Program. Health Educ Q. 1989;16:263– 283. 47. Robinson TN. Reducing children’s television viewing to prevent obesity: Randomized controlled trial. JAMA. 1999;282:1561– 1567. 48. Resnicow K, Robinson TN, Frank E. Advances and future directions for school-based health promotion research: commentary on the CATCH intervention trial. Prev Med. 1996;25:378– 383. 49. Bryk AS, Raudenbush SW, eds. Hierarchical Linear Models. Beverly Hills, Calif: Sage Publications; 1992. 50. Weiner JP, Garfield BH, Powe NR, et al. Ambulatory care practice variation within a Medicaid population. Health Serv Res. 1996;30:751–770. 51. Tayne SM, Schwartz RM. An evaluation of pediatric-modified diagnosis-related groups. Health Care Finan Rev. 1993;15:51– 70. 52. Barlow SE, Kaplan SH, Parsons SK, et al. Pilot studies of severity scale development in pediatric chronic disease. In press. 53. Christen WG, Gaziano JM, Hennekens CH. Design of physicians’ health study II—a randomized trial of beta-carotene, vitamins E and C, and multivitamins, in prevention of cancer, cardiovascular disease, and eye disease, and review of results of completed trails. Ann Epidemiol. 2000;10:125–134. 54. Futterman LG, Lemberg L. The Framingham Heart Study: a pivotal legacy of the last millennium. Am Heart J. 2000;9:147– 151. 55. Fine JT, Colditz GA, Coakley EH, et al. A prospective study of weight change and health-related quality of life in women. JAMA. 1999;282:2136–2142.