Methods in Public Health Services and Systems Research

Methods in Public Health Services and Systems Research A Systematic Review Jenine K. Harris, PhD, Kate E. Beatty, MPH, Colleen Barbero, MPPA, Alex F. Howard, MPH, Robin A. Cheskin, BA, Robert M. Shapiro II, MALS, Glen P. Mays, PhD, MPH Context: Public Health Services and Systems Research (PHSSR) is concerned with evaluating the organization, fınancing, and delivery of public health services and their impact on public health. The strength of the current PHSSR evidence is somewhat dependent on the methods used to examine the fıeld. Methods used in PHSSR articles, reports, and other documents were reviewed to assess their methodologic strengths and challenges in light of PHSSR goals. Evidence acquisition: A total of 364 documents from the PHSSR library met the inclusion criteria as empirical and based in the U.S. After additional exclusions, 327 of these were analyzed. Evidence synthesis: A detailed codebook was used to classify articles in terms of (1) study design; (2) sampling; (3) instrumentation; (4) data collection; (5) data analysis; and (6) study validity. Inter-coder reliability was assessed for the codebook; once it was found reliable, the available empirical documents were coded.

Conclusions: Although there has been a dramatic increase in the amount of published PHSSR recently, methods used remain primarily cross-sectional and descriptive. Moreover, although appropriate for exploratory and foundational work in a new fıeld, these approaches are limiting progress toward some PHSSR goals. Recommendations are given to advance and strengthen the methods used in PHSSR to better meet the goals and challenges facing the fıeld. (Am J Prev Med 2012;42(5S1):S42–S57) © 2012 American Journal of Preventive Medicine

Context

P

ublic health services and systems research (PHSSR) is a multidisciplinary fıeld of study concerned with evaluating the “organization, fınancing, and delivery of public health services and the impact of these services on public health.”1,2 PHSSR brings together a wide variety of research areas and theoretic and methodologic traditions.1,3–5 Historically, the development of PHSSR has been slow, with progress not always apparent; however, development has begun to speed up over the last few decades. In 1988, the influential IOM Future of Public Health report6 From the George Warren Brown School of Social Work (Harris, Barbero, Cheskin), Washington University in St. Louis, the School of Public Health (Beatty), Saint Louis University, St. Louis, Missouri; the College of Public Health (Howard), the Medical Center Library (Shapiro), University of Kentucky, Lexington, Kentucky; and the University of Arkansas for Medical Sciences (Mays), Little Rock, Arkansas Address correspondence to: Jenine K. Harris, PhD, Assistant Professor, George Warren Brown School of Social Work, Campus Box 1196, Washington University in St. Louis, St. Louis MO 63130. E-mail: jharris@ brownschool.wustl.edu. 0749-3797/$36.00 doi: 10.1016/j.amepre.2012.01.028

S42 Am J Prev Med 2012;42(5S1):S42–S57

called for the development of the evidence base in PHSSR. In 1990, the DHHS decennial publication, Healthy People, set the goal that by 2000 at least 90% of the population would be served by a public health department that effectively carries out the IOM’s core functions.2,7 In direct response, the CDC and the National Association of County and City Health Offıcials (NACCHO) began to research strategies for guidelines and self-assessment tools to measure how well public health agencies carried out core functions.2 Between 2001 and 2009, the U.S. government invested more than $10 billion in new funds to support public health activities.2 In the midst of these 10 years of public health investment, CDC introduced its fırst PHSSR agenda (in 2003) and the Robert Wood Johnson Foundation (RWJF) began to support efforts to convene representatives from states working on PHSSR (in 2006).2 In 2007, the RWJF awarded the University of Kentucky Research Foundation more than $2.8 million for “Creating a resource center for public health systems and services research.” During this time of growth, six overarching goals for PHSSR were defıned: (1) determine how public health

© 2012 American Journal of Preventive Medicine • Published by Elsevier Inc.

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

agency structure affects performance; (2) defıne and quantify dimensions of public health systems, including interorganizational relationships; (3) explore the relationship between performance and health outcomes; (4) defıne the characteristics of high-performing local, state, and federal public health agencies; (5) explore the relationship between social determinants of health and system performance; and (6) evaluate the costs of achieving and maintaining acceptable/optimal levels of performance.8 Meeting each of the six goals requires different methodologic strategies. The fırst, third, and fıfth goals, for example, describe causal relationships. These goals require research designs and analytic strategies that allow for hypothesis testing relating outcomes to explanatory variables. In contrast, the second and fourth goals could be addressed using primarily descriptive statistics and qualitative information. Finally, the sixth goal might benefıt from an economic analytic strategy such as cost-effectiveness or cost– benefıt analysis. Despite the focus and investment in PHSSR, as recently as 2009 the fıeld was still considered underdeveloped.2,6 The adequate development of PHSSR is, in part, dependent on the methods used to understand the fıeld. The goal of the current review is to examine the research designs and analytic strategies used in PHSSR over the past 3 decades.

Evidence Acquisition Data Source The data source was the PHSSR library developed by the University of Kentucky Center for Public Health Services and Systems Research (CPHSSR). The PHSSR library incorporates a subset of works from the NLM’s Health Services and Sciences Research Resources database (HSSRR)3; literature from other relevant databases; and grey literature primarily from the New York Academy of Medicine’s Grey Literature Report. Beginning in 2006, a team of librarians and researchers at CPHSSR regularly search these sources and identify literature for inclusion/exclusion; the search methods and inclusion/exclusion criteria used to develop the PHSSR library have been detailed in Scutchfıeld et al.3 As of October 2010, when documents to date were collected for this study, the library included 781 documents.

Study Eligibility Criteria Because this review focused on PHSSR methods, original empirical studies where information was derived from data (quantitative or qualitative) were sought, rather than reviews, theoretic, or editorial pieces. Abstracts or summaries for all but four of the 781 documents were obtained (n⫽777; 99.5%) and each was coded for inclusion or exclusion by two coders independently. A document was included if it was empirical and based in the U.S. Sixteen percent (n⫽125) required resolution by a third coder. A total of 364 (46.8%) articles met the inclusion criteria; 333 (91.5%) were used for the coding (see Appendix). May 2012

781 PHSSR documents in the library 777 abstracts/summaries found 364 included data analysis and were U.S.-based

S43

4 documents not found 413 excluded 386 did not include data analysis 27 were international 31 not available

333 full documents obtained 6 did not include data analysis 327 included data analysis

Figure 1. Flow chart showing article inclusion into the systematic review of public health services and systems research (PHSSR)

The coding process included two steps: (1) codebook development and (2) coding of articles.

Codebook Development and Reliability The codebook was developed based on three sources: (1) a review9 of methods in research from the top journal in the fıeld of community psychology; (2) a paper from AcademyHealth10 summarizing 66 methods-focused articles in PHSSR; and (3) discussion among the team members. Five PHSSR articles were selected from the data and coded by all six coders to test the initial version of the codebook. Following in-depth discussions of the coding for these fıve articles, the codebook was refıned. The fınal codebook included fıve sections: (1) general characteristics; (2) data collection and analytic strategy; (3) sampling; (4) research design; and (5) instrumentation and validity (Appendix A, available online at www. ajpmonline.org). Before coding all articles, reliability testing was conducted to ensure consistent classifıcation. Using a random start, a systematic sample of 23 articles (7% of the 333 articles) was coded for reliability. Percentage agreement among the six trained coders ranged from 66% to 95% across the 23 articles, with a mean percentage agreement of 80%. To account for the proportion of agreement that may have happened by chance, a modifıed form of the kappa statistic was calculated to account for multiple coders and multiple items.11 The intraclass correlation coeffıcient (ICC) across all 23 articles was good, bordering on excellent (ICC⫽0.73) and ranged from 0.53 to 0.92 for each article.11 Eleven articles had excellent agreement among coders (ICC⬎0.75), whereas 12 had good agreement (0.4 ⱕ ICC ⱕ 0.75). The overall kappa for the data set was 0.51. According to Landis and Koch,12 this kappa represents a moderate amount of agreement among coders beyond what would have happened by chance. Given acceptable reliability, each of the 333 articles was coded individually by one of the six coders. Of the 333 articles, six were subsequently excluded for not fıtting the inclusion criteria of being original empirical studies, leaving a sample size of 327 for data analysis (41.9% of the original 781 articles; 89.8% of those coded for inclusion). Figure 1 shows this process. The distribution of these 327 PHSSR articles showed increases in publication volume that appear to coincide with the release of the influential papers (e.g., Future of Public Health) and the PHSSR funding and organizational efforts described in the background section above (Figure 2).

S44

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

Data Analysis Descriptive statistics and graphs were used to examine the data. Data quality checking was conducted during the early stages of analysis whenever unexpected values were obtained. For example, two quantitative studies13,14 were coded as having used focus groups to collect data. Because focus groups are used most often in qualitative research, these studies were reviewed. In these two cases, focus groups were used to Figure 2. Increase in the publication of empirical and non-empirical public health services vote on topics and record and systems research (PHSSR) documents over time quantitative results, so the categorization of these studies non from two irreconcilable perspectives.15,16 In the was not a coding or data entry error. A few minor corrections were context of PHSSR, mixed methods approaches have been made where data were improperly coded or recorded.

Evidence Synthesis The following sections describe the use of research designs and sampling frames, data collection and analysis methods, sample size, power, and validity in 327 empirical PHSSR studies.

Public Health Services and Systems Research Study Designs The most commonly used quantitative research design in PHSSR was cross-sectional, whereas case studies were the most frequently used qualitative design. Two-hundred sixty documents (79.5%) were journal articles; 48 (14.7%) were government reports; 18 (5.5%) were nongovernment reports; and one document (0.3%) was a book. Of these 327 documents, 224 (68.5%) used quantitative research designs; 45 (13.8%) used qualitative designs; and 58 (17.7%) used a mixed methods research designs. A majority of the 282 quantitative or mixed methods studies were cross-sectional (n⫽228; 80.9%), followed by longitudinal (n⫽29; 10.3%); case– control (n⫽9; 3.2%); quasiexperimental (n⫽7; 2.5%); and experimental (n⫽7; 2.5%) (Figure 3). Of the 228 cross-sectional designs, 59 (25.9%) were repeated cross-sectional studies (cross-sectional data collected at two or more time points). The majority (n⫽46; 52.9%) of qualitative or mixed studies (n⫽87) used a case-study approach (Figure 4), and 53 (60.8%) collected data at a single time period. Seventy-three (83.9%) of qualitative or mixed studies used primary data. Fifty-eight (17.7%) of the 327 studies in this sample used a mixed methods research design. Mixed methods approaches have been praised for providing complementary information and criticized for attempting to draw conclusions about a single phenome-

used for many purposes. For example, Wheeler’s 2007 article17 examined the influence of a new smokefree hospital policy on consumer behavior through focus groups, indepth interviews, and a cross-sectional survey. Another mixed methods study examined the roles and funding structures of local health departments through in-depth interviews, administrative data, and annual reports.18

Public Health Services and Systems Research Sampling Strategies Nonprobability samples were the most frequently employed type of sample in PHSSR. Of the 291 studies reporting on sampling frame, probability samples were used by 68 (23.4%) studies and 69 (23.7%) used population samples, whereas nonprobability samples were found in 154 (52.9%) of the included studies. Thirty-eight (11.6%) studies did not report what sort of sampling frame was used. The sampling strategy varied by study design (Figure 5). Although often used in studies with nonprobability sampling, inferential statistics are based on the assumption of a known probability for each observation.19 Nonprobability samples do not have this quality.20 To account for a nonprobability sample, researchers can conduct power analysis to determine a sample size that would approximate the results expected with a probability sample.19,20 Of the 132 nonprobability quantitative studies, 79 (59.8%) used one or more inferential statistical methods; fıve of these (6.3%) discussed power calculations.

Public Health Services and Systems Research Data Collection Strategies The most commonly used data collection strategy in PHSSR was written surveys or questionnaires developed for the purposes of the study. Data were collected using www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

Figure 3. Quantitative research designs used in the quantitative (n⫽224) and mixed methods (n⫽58) studies

many strategies; the most frequent method was written survey or questionnaire (n⫽145; 44.3%). Researchers often created their own surveys, tailored to answering their specifıc research questions. For example, Abarca and colleagues21 surveyed Florida county health departments on an annual basis to assess community capacity, using a web-based survey developed as part of the Comprehensive Assessment, Strategic Success initiative. The next most commonly used data collection method involved extracting administrative data (n⫽77; 23.5%) from existing databases. Administrative data were defıned as computerized records gathered for some administrative purpose (e.g., birth records, death records, hospital discharge fıles). The least common data collection methods were focus groups (n⫽20; 6.1%) and observations (n⫽20; 6.1%). Distribution of data collection strategies is shown in Table 1. Public health services and systems research studies used primary data only (n⫽200; 61.2%); secondary data only (n⫽116; 35.5%); or both (n⫽11; 3.4%). Primary data are collected specifıcally for the reported study, whereas secondary data are not collected specifıcally for the study at hand.

S45

Figure 5. Sampling strategy and type of research design (n⫽284)

Public Health Services and Systems Research Data Characteristics A majority of PHSSR studies collected quantitative data at the individual level at a single point in time. Most PHSSR studies collected quantitative data (n⫽224; 68.5%), followed by mixed (n⫽58; 17.7%). Strictly qualitative data were collected in only 45 (13.8%) of the studies. The majority of studies collected data on individuals (n⫽198; 60.6%); Figure 6). The second most common level of data collection was local health department (n⫽62; 19.0%) followed closely by community groups (n⫽48; 14.7%). More than one third of the studies (n⫽21; 33.9%) of local health departments used secondary data, which were likely to have come from the NACCHO Profıle Study of Local Health Departments.22 Surveys of local health departments have been conducted by NACCHO on a regular basis (1990, 1992, 1996, 2005, 2008, and 2010), and it makes the data available for public health professionals and others. Findings from studies using NACCHO data often address the PHSSR goals related to health system structure and performance. For example, an Table 1. Data collection method used in 327 PHSSR studies, n (%) Mixed Qualitative Quantitative methods Administrative data

4 (8)

61 (25)

12 (13)

Focus group

10 (20)

2 (1)

8 (9)

Interview

28 (56)

65 (27)

30 (32)

Observation

3 (6)

10 (4)

7 (8)

Survey

5 (10)

104 (43)

36 (39)

242 (100)

93 (100)

Total

Figure 4. Qualitative designs used in the qualitative (n⫽45) and mixed methods (n⫽58) studies May 2012

50 (100)

PHSSR, public health services and systems research

S46

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

nonzero effect size. The power of a statistical test is determined by three factors: number of observations (sample size); the size of the effect in the population; and the alpha level (␣). Power increases with a larger effect size, larger sample size, and more lenient alpha level.25 Of the 282 quantitative or mixed studies, 5.7% (n⫽16) mentioned power. Of those studies that mentioned power, 37.5% (n⫽6) reported adequate power.

Reliability and Validity of Measures in Public Health Services and Systems Research

Figure 6. Percentage of studies collecting data at different levels (n⫽327)

early study23 using the NACCHO data examined core occupations in local health departments and found that numerous vacancies across the country resulting in reduced LHD effectiveness in terms of responding to urgent health threats. Another NACCHO study24 found that LHD performance was higher in LHDs that include full-time leadership and diverse funding sources. Information related to poverty, education, racial, and ethnic composition can all be found at the community level using census data. The other fıve levels of data collection—state, program or project, state health department, country, and region—were each used in fewer than 10% of studies (nⱕ25). Most studies (n⫽281; 85.9%) used a single level of data, whereas 46 studies (14.1%) collected data at two or more levels. Of the 46 studies collecting data at multiple levels, 24 (52.1%) were quantitative; four (8.7%) were qualitative; and 18 (39.1%) were mixed methods. Finally, most studies (n⫽210; 64.2%) collected data at one time point. Data were collected at two time points by 12.5% (n⫽41) of the studies and at three or more time points by 23.2% (n⫽76) of the studies.

Power and Sample Size in Public Health Services and Systems Research Studies Although most studies reported sample size and response rate, few reported on power. Of the included studies, 87.5% of the studies (n⫽286) reported the sample size. More than one third (37.0%; n⫽121) of the studies reported a response rate. Of these studies, the mean response rate was 75.6%. The lowest response rate reported was 3.8%. The highest response rate reported was 100%, reported by 12 studies. Power is defıned as the probability of not making a type II error (1–␤), or the probability of detecting an existing

Nearly one third of PHSSR studies reported developing new instruments; however, few reported testing the instruments or using validated or reliable existing instruments. Instrument quality directly affects data quality and study results. Instrument reliability is one statistical measure of how reproducible the data are from a given survey instrument.26 Instrument validity refers to the extent an instrument measures what it intends to measure (p. 33). Although sometimes appropriate in qualitative research, reliability and validity testing as conceptualized here are primarily applicable to quantitative studies. Of the coded studies, 32.7% (n⫽107) reported developing an instrument; of these studies, 38.3% (n⫽41) reported validity and reliability testing of their newly developed instrument. Fifteen studies (4.6%) reported testing the validity or reliability of a new instrument, and 10.7% of studies (n⫽35) reported using validated and reliable instruments. More than half of studies did not report on instrumentation (n⫽170; 52.0%). This may, in part, be due to widespread use of secondary data from sources like the NACCHO Profıle Study, which has not been tested for reliability or validity.

Data Analysis in Public Health Services and Systems Research Nearly all studies collecting quantitative data presented descriptive statistics, and more than half also conducted inferential statistics, whereas many of the qualitative studies used thematic or content analysis strategies. Descriptive statistics were reported for 262 of the 282 (92.9%) quantitative studies. Overall, 60.2% (n⫽170) of quantitative studies used inferential methods. Standard inferential methods including chi-square, t test, correlation, ANOVA, and regression were used in a majority of inferential studies (n⫽130, 76.5%), whereas inferential methods associated with measurement (e.g., factor analysis) were used in 28 of the 170 inferential studies (16.5%). Systems methods (e.g., network analysis; n⫽3; 1.1%); mapping or spatial analysis (e.g., use of GIS; n⫽5; 1.8%); and multilevel modeling (n⫽4; 1.4%) were used infrequently. Of the quantitative and mixed studies (n⫽282) collecting data at more than one level (n⫽43; www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

15.2%) or using a longitudinal study design (n⫽28; 9.9%), one study used structural equation modeling or multilevel modeling. Of the 103 studies using qualitative or mixed methods, the two most common qualitative data analysis strategies were thematic analysis with 51.5% (n⫽53) and content analysis with 46.6% (n⫽48). Thematic analysis identifıes emerging themes, or patterns found in information. For example, Beitsch and colleagues27 used thematic analysis for a study on establishing a national voluntary public health accreditation program; they reported common themes from a review of state applications to the MultiState Learning Collaborative on Performance and Capacity Assessment or Accreditation of Public Health Departments. Few qualitative studies included diagrams (n⫽7; 6.8%) or typologies (n⫽4; 3.9%).

Validity in Public Health Services and Systems Research Studies Few PHSSR studies discussed external or internal validity. External validity measures how well a study translates to others outside the study population, whereas internal validity measures whether a study provides accurate unbiased estimates of the phenomena it purports to measure. Generalizability is the main characteristic of external validity. A study that is generalizable has attempted to reduce sources of error variance to obtain results that go beyond the study sample and apply to a larger group or population.28 –30 Of the 282 quantitative or mixed methods studies, 53 (18.8%) reported on generalizability of fındings. Six of the 53 (11.3%) reported that fındings were generalizable, whereas 47 (88.7%) reported that fındings were limited or not generalizable beyond the study. Representativeness refers to how well a sample represents the population and is another component of external validity.30 Sixty-six (23.4%) of the quantitative or mixed methods studies reported on the representativeness of their sample. Twenty-two (32.4%) of these studies reported being representative, whereas 44 studies (64.7%) reported limited or no representativeness. Of studies using nonprobability samples (n⫽170), which are more prone to selection bias, only 22.9% of studies (n⫽39) reported on the representativeness of the sample, and only 8.8% (n⫽15) reported that their studies were representative of the population. There are many biases that threaten internal validity; this review identifıed three of the most common: selfreport bias, nonresponse bias, and recall bias. Self-report bias is the propensity for research participants to respond to researchers in ways that are desirable.31 Nonresponse bias occurs when those who are part of the sample but do not participate are systematically different from participants.32 Recall bias is the result of inaccurate recall of past May 2012

S47

exposures or events and is affected by characteristics of the event and respondent.33 Of the 282 quantitative and mixed methods studies, 31 (11.0%) reported self-report bias; fıve (1.8%) reported recall bias; and 23 (8.2%) reported nonresponse bias. Non-experimental research designs are most susceptible to threats to internal validity given their lack of control over study conditions and lack of ability to assess cause and effect.34 These three types of bias were primarily discussed in the CDC’s Morbidity and Mortality Weekly Reports (MMWRs) included in this review. Each MMWR typically included a standard limitations section that discussed representativeness, generalizability, and the potential for bias in the Editorial Note following the main text of the study.35,36 A few non-MMWR articles such as the one by Avery and colleagues37 also addressed limitations. However, aside from the MMWR, attention to problems with internal validity was uncommon in the articles reviewed. The distribution of threats to internal validity by study design (cross-sectional, case–control, longitudinal, quasiexperimental, experimental) was examined. Of the 31 studies reporting self-report bias, 29 were cross-sectional and two were case–control. Of the fıve reporting recall bias, four were cross-sectional and one was case–control. Of the 23 reporting nonresponse bias, all were crosssectional. The majority of threats to internal validity were reported in cross-sectional studies, with 48 of the 228 (21.1%) cross-sectional studies reporting any of the three common threats to internal validity. Three case–control studies reported threats to internal validity.

Limitations As with many systematic reviews, the data source is one of the primary limitations for the current study. There are two possible weaknesses of the library produced by the CPHSSR. The fırst is that there may be relevant items that have not been included. This limitation speaks to a larger challenge in PHSSR, which is the lack of a consistent term that represents PHSSR in large databases. CPHSSR staff has advocated for the inclusion of a “public health services and systems research” medical subject heading (MeSH) term, yet no such term exists to date. This, in part, may be due to the lack of consistency in terms used to describe the fıeld.5 The second weakness is that there are items included in the library that are not PHSSR. This challenge is related to the purpose of the library; the library is meant to serve the needs of PHSSR researchers and therefore contains some items that may not be PHSSR but that may be useful for PHSSR. Additionally, overall intercoder reliability was found to be moderate; however, good or excellent reliability scores were seen for most of the articles during reliability testing. Finally,

S48

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

although the codebook was extensive, it was likely not comprehensive. For example, collecting additional data that would allow for more specifıc conclusions, such as identifying which studies were appropriate for conducting power and effect size calculations, would strengthen the fındings. In addition, there may be important characteristics of the methods utilized that were not identifıed.

Discussion In the past few years, the PHSSR library has experienced rapid growth, including a substantial increase in the volume of empirical PHSSR being disseminated through reports and journal articles (Figure 2). The empirical PHSSR examined here varied from surveillance studies of influenza and other infectious and chronic diseases,36,38 – 40 to quantitative and qualitative studies examining how public health agencies, workforce, and educational programs rate according to standards,41– 47 to research on how public policy influences health and how PHSSR can influence policy.48 –50 Although the volume of research seems to be catching up to the need for evidence in PHSSR, questions remain about the quality of the methods being used and therefore the quality of evidence being accumulated. The studies examined in this review demonstrated widespread use of individual-level data, cross-sectional designs, and nonprobability samples, and limited use of inferential and complex statistics. These characteristics present barriers to meeting several of the PHSSR research goals.8 For example, although the surveillance studies (mostly from the CDC) were based on Behavioral Risk Factor Surveillance Survey data, which is collected using a probability sample, most of these studies presented only descriptive information rather than taking advantage of the ability to make inferences based on a large national probability sample. In addition, many of the studies examining public health standards, such as the 10 Essential Public Health Services, also used primarily descriptive statistics, with a few exceptions such as Mays and colleagues’43 use of factor analysis to identify dimensions of performance in local public health systems. Finally, the limited use of non–individual level data pointed to a disconnect between PHSSR goals, which primarily focused on the public health system, and PHSSR data sources. Equally importantly, these same study characteristics may limit PHSSR in providing much needed externally valid, generalizable information to the fıeld. For example, much of the work on public policy and public health in PHSSR was qualitative and therefore not generalizable. It has been recommended that more PHSSR policy studies attempt to link public policy to measurable quantitative outcomes, which, if carefully executed, could provide

generalizable information on health outcomes associated with particular policies.50 Additional challenges to the quality of the evidence might be found in the lack of validity and reliability studies examining the many new instruments being developed across the fıeld, and the lack of power analyses reported, which may hinder the contributions of negative study results. Therefore, the following recommendations are designed to increase the ability of PHSSR to meet many of its goals and challenges: 1. Increase the use of systems-level data, study designs, sampling frames, and analytic strategies that can better answer the complex questions facing PHSSR. Research questions in PHSSR that focus on causal relationships, such as the relationship between system structure and performance, would benefıt from additional use of study designs, sampling frames, and analytic strategies that are representative, generalizable, and can capture change over time. Some of these qualities can be addressed by simply taking advantage of the unique opportunities that exist in PHSSR to conduct natural experiments and use creative quasi-experimental designs (e.g., regression-discontinuity) as public health systems grow and change. In addition, if PHSSR adopts study designs and sampling frames that collect representative information, and uses inferential statistics where appropriate, the number of studies that meet the criteria for external validity will increase. External validity in PHSSR studies may be especially important when trying to build successful public health systems. For example, identifying characteristics of successful local or state health departments is most useful when this information can be generalized to the larger population of health departments. Finally, efforts are currently underway to collect, harmonize, and validate PHSSR-specifıc large national data sets (www.publichealthsystems.org/ cphssr/MembershipResources/1411/SynopticAnalysis), which may aid in improving the availability and use of PHSSR data that goes beyond the individual level. 2. Report study power where appropriate when negative results are found. Understanding whether the relationship examined in a study with negative results is not signifıcant, or whether the study just did not have enough power, will increase the strength of the evidence in PHSSR. Studies examining important questions but having insuffıcient power might be revisited. Conversely, those studies with adequate power that did not fınd a signifıcant effect could provide useful information about what does not work, and, depending on the strength and accumulation of evidence, may not need to be replicated, saving resources. www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

3. Increase the testing of validity and reliability of existing (e.g., NACCHO Profıle Study) and new PHSSR instruments, along with increasing the use of already validated instruments. Like calculating and reporting study power, testing and reporting on the validity and reliability of instruments will strengthen the evidence and allow PHSSR to become more effıcient through the accumulation and use of consistent and valid measures. Although this study is the largest systematic review of PHSSR methods to date, many of these recommendations are not new for PHSSR. These suggestions echo some of the fındings from prior work in 2009 by Scutchfıeld and colleagues3 and the recent AcademyHealth paper “A Needs Assessment for Data and Methods in Public Health Systems Research.”3,10 The consistent message across these three papers seems clear: It is time for PHSSR researchers to purposefully adopt research strategies that will improve methodologic strength and sophistication of PHSSR in order to best answer the big questions facing the fıeld. Publication of this article was supported by a grant from the Robert Wood Johnson Foundation. The project team would like to thank Margaret Hower for her help in the initial codebook development and Kari Lindberg for assisting with data entry. No fınancial disclosures were reported by the authors of this paper.

References 1. Mays GP, Halverson PK, Scutchfıeld FD. Behind the curve? What we know and need to learn from public health systems research. J Public Health Manag Pract 2003;9(3):179 – 82. 2. Scutchfıeld FD, Mays GP, Lurie N. Applying health services research to public health practice: an emerging priority. Health Serv Res 2009;44(5 Pt 2):1775– 87. 3. Scutchfıeld FD, Lawhorn N, Ingram R, Pérez DJ, Brewer R, Bhandari M. Public health systems and services research: dataset development, dissemination, and use. Public Health Rep 2009;124(3):372–7. 4. Savoia E, Massin-Short SB, Rodday AM, Aaron LA, Higdon MA, Stoto MA. Public health systems research in emergency preparedness: a review of the literature. Am J Prev Med 2009;37(2):150 – 6. 5. Harris JK, Beatty KE, Lecy JD, Cyr JM, Shapiro RM II. Mapping the multidisciplinary fıeld of public health services and systems research. Am J Prev Med 2011;41(1):105–11. 6. IOM. Committee for the Study of the Future of Public Health editor. The future of public health. Washington DC: National Academy Press, 1998. 7. DHHS. Healthy People 2000. www.healthypeople.gov/publications/. 8. Lenaway D, Halverson P, Sotnikov S, Tilson H, Corso L, Millington W. Public health systems research: setting a national agenda. Am J Public Health 2006;96(3):410 –3. 9. Luke DA. Getting the big picture in community science: methods that capture context. Am J Community Psychol 2005;35(3-4):185–200.

May 2012

S49

10. Holve E, Papa K, Arnold S, Ix M. A needs assessment for data and methods in public health systems research. AcademyHealth 2010. 11. Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull 1971;76(5):378 –97. 12. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159 –74. 13. Baird JR, Carlson KJ. National Public Health Performance Standards assessment: fırst steps in strengthening North Dakota’s public health system. J Public Health Manag Pract 2005;11(5):422–7. 14. Scutchfıeld FD, Beaulieu J, Ireson C, Buege A. Public health competencies required by managed care organizations. J Public Health Manag Pract 2002;8(5):22–9. 15. Razum O, Gerharus A. Methodological triangulation in public health research—advancement or mirage? [Editorial]. Trop Med Int Health 1999;4(4):243– 4. 16. Sale JEM, Lohfeld LH, Brazil K. Revisiting the quantitative-qualitative debate: implications for mixed-methods research. Qual Quant 2002; 36(1):43–53. 17. Wheeler JG, Pulley L, Felix HC, et al. Impact of a smoke-free hospital campus policy on employee and consumer behavior. Public Health Rep 2007;122(6):744 –52. 18. Potter MA, Fitzpatrick T. State funding for local public health: observations from six case studies. J Public Health Manag Pract 2007;13(2): 163– 8. 19. LoBiondo-Wood G, Haber J. Nursing research: methods and critical appraisal for evidence-based practice. 7th ed. St. Louis MO: Mosby/ Elsevier, 2010. 20. Babbie ER. The practice of social research. 12th ed. Belmont CA: Wadsworth Cengage, 2010. 21. Abarca C, Grigg CM, Steele JA, Osgood L, Keating H. Building and measuring infrastructure and capacity for community health assessment and health improvement planning in Florida. J Public Health Manag Pract 2009;15(1):54 – 8. 22. National Association of County and City Health Offıcials. 2005 National Profıle of Local Health Departments. 2006. 23. Gerzoff RB, Brown CK, Baker EL. Full-time employees of U.S. local health departments, 1992–1993. J Public Health Manag Pract 1999;5(3):1–9. 24. Handler AS, Turnock BJ. Local health department effectiveness in addressing the core functions of public health: essential ingredients. J Public Health Policy 1996;17(4):460 – 83. 25. Hallahan M, Rosenthal R. Statistical power: concepts, procedures, and applications. Behav Res Ther 1996;34(5-6):489 –99. 26. Litwin MS. How to measure survey reliability and validity. Beverly Hills CA: Sage Publications, 1995. 27. Beitsch LM, Mays G, Corso L, Chang C, Brewer R. States gathering momentum: promising strategies for accreditation and assessment activities in multistate learning collaborative applicant states. J Public Health Manag Pract 2007;13(4):364 –73. 28. Cronbach LJ, Gleser GC, Nanda H, Rajaratnam N. The dependability of behavioral measurements: theory of generalizability for scores and profıles. New York: Wiley, 1972. 29. Streiner DL, Norman RN. Health measurement scales: a practical guide to their development and use. 3rd ed. Oxford, UK: Oxford University Press, 2003. 30. Rea L, Parker R. Selecting a representative sample. 3rd ed. San Francisco CA: Jossey-Bass, 2005:114 –27. 31. Donaldson SI, Grant-Valone EJ. Understanding self-report bias in organizational behavior research. J Business Psychol 2002;17(2): 245– 60. 32. Aschengrau A, Seage GR III. Bias. Sudbury MA: Jones and Bartlett, 2003;251–79. 33. Coughlin SS. Recall bias in epidemiologic studies. J Clin Epidemiol 1990;43(1):87–91.

S50

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

34. Szklo M, Nieto FJ. Basic study designs in analytical epidemiology. Sudbury MA: Jones and Bartlett, 2007. 35. CDC. Influenza vaccination coverage among children aged 6 months–18 years— eight immunization information system sentinel sites, U.S., 2008 – 09 influenza season. MMWR Morb Mortal Wkly Rep 2009;58(38):1059 – 62. 36. CDC. State-specifıc prevalence of obesity among adults—U.S., 2007. MMWR Morb Mortal Wkly Rep 2008;57(28):765– 8. 37. Avery GH, Wholey DR, Christianson JB. Physician evaluations of care management practices in Medicaid programs. Am J Manag Care 2005;11(3):156 – 64. 38. CDC. Update: influenza activity—U.S., September 28, 2008-April 4, 2009, and composition of the 2009 –10 influenza vaccine. MMWR Morb Mortal Wkly Rep 2009;58(14):369 –74. 39. CDC. State-specifıc influenza vaccination coverage among adults— U.S., 2006 – 07 influenza season. MMWR Morb Mortal Wkly Rep 2008;57(38):1033–9. 40. CDC. Impact of expanded newborn screening—U.S., 2006. MMWR Morb Mortal Wkly Rep 2008;57(37(:1012–5. 41. Honore PA, Schlechte T. State public health agency expenditures: categorizing and comparing to performance levels. J Public Health Manag Pract 2007;13(2):156 – 62. 42. Joly BM, O’Rourke K, Tilson HH, Leonard JF. Use of national public health performance standards to assess Maine’s diabetes system. J Public Health Manag Pract 2007;13(1):68 –71. 43. Mays GP, McHugh MC, Shim K, et al. Identifying dimensions of performance in local public health systems: results from the National Public Health Performance Standards Program. J Public Health Manag Pract 2004;10(3):193–203. 44. Purcell JM. Recruiting the future public health workforce: an analysis of prospect communication among accredited Schools of Public Health. J Community Health 2009;34(3):216 –21. 45. Scutchfıeld FD, Quimson S, Williams SJ, Hofstetter R. The demand for doctorally prepared public health personnel in institutions of higher education. Am J Prev Med 1988;4(5):298 –301. 46. CDC. Assessment of epidemiology capacity in State Health Departments— U.S., 2009. MMWR Morb Mortal Wkly Rep 2009;58(49):1373–7. 47. CDC. Status of state electronic disease surveillance systems—U.S., 2007. MMWR Morb Mortal Wkly Rep 2009;58(29):804 –7. 48. Anderson LM, Brownson RC, Fullilove MT, et al. Evidence-based public health policy and practice: promises and limits. Am J Prev Med 2005;28(5S):226 –30. 49. Stachenko S. Challenges and opportunities for surveillance data to inform public health policy on chronic non-communicable diseases: Canadian perspectives. Public Health 2008;122(10):1038 – 41. 50. Berkowitz B, Ivory J, Morris T. Rural public health: policy and research opportunities. J Rural Health 2002;18S:186 –96.

Appendix: Full List of Studies in the Review Abarca C, Grigg CM, Steele JA, Osgood L, Keating H. Building and measuring infrastructure and capacity for community health assessment and health improvement planning in Florida. J Public Health Manag Pract 2009;15(1):54 – 8. Ablah E, Tinius AM, Horn L, Williams C, Gebbie KM. Community health centers and emergency preparedness: an assessment of competencies and training needs. J Community Health 2008;33(4):241–7. Agee B, Gimbel RW. Assessing the legal and ethical preparedness of master of public health graduates. Am J Public Health 2009;99(8):1505–9. Ahluwalia IB, Bolen J. Lack of health insurance coverage among workingage adults, evidence from the Behavioral Risk Factor Surveillance System, 1993–2006. J Community Health 2008;33(5):293– 6. Aiello AE, Coulborn RM, Perez V, Larson EL. Effect of hand hygiene on infectious disease risk in the community setting: a meta-analysis. Am J Public Health 2008;98(8):1372– 81.

Alejos A, Weingartner A, Scharff DP, et al. Ensuring the success of local public health workforce assessments: using a participatory-based research approach with a rural population. Public Health 2008; 122(12):1447–55. Alexander LK, Dail K, Horney JA, et al. Partnering to meet training needs: a communicable-disease continuing education course for public health nurses in North Carolina. Public Health Rep 2008;123(S2): 36 – 43. Asch SM, Stoto M, Mendes M, et al. A review of instruments assessing public health preparedness. Public Health Rep 2005;120(5):532– 42. Avery GH, Wholey DR, Christianson JB. Physician evaluations of care management practices in Medicaid programs. Am J Manag Care 2005;11(3):156 – 64. Axnick NW, Katz M, Schiffer C, Johnson W, Cross F. Survey of city/county public health agencies to determine the development, use, and effect of program performance standards. Am J Public Health 1986;76(6): 692– 4. Baird JR, Carlson KJ. National public health performance standards assessment: fırst steps in strengthening North Dakota’s public health system. J Public Health Manag Pract 2005;11(5):422–7. Baker EL. Pilot study of public health workforce competency, agency capacity and performance. University of North Carolina at Chapel Hill School of Public Health, North Carolina Institute for Public Health. Health Policy and Administration Baker EL, Blumenstock JS, Jensen J, Morris RD, Moulton AD. Building the legal foundation for an effective public health system. J Law Med Ethics 2002;30(3S):48 –51. Baldwin LM, Hollow WB, Casey S, et al. Access to specialty health care for rural American Indians in two states. J Rural Health 2008;24(3): 269 –78. Bara D, Mcphillips-Tangum C, Wild EL, Mann MY. Integrating child health information systems in public health agencies. J Public Health Manag Pract 2009;15(6):451– 8. Baron S, Sinclair R, Payne-Sturges D, et al. Partnerships for environmental and occupational justice: contributions to research, capacity and public health. Am J Public Health 2009;99(S3):S517–S525. Basta NE, Edwards SE, Schulte J. Assessing public health department employees’ willingness to report to work during an influenza pandemic. J Public Health Manag Pract 2009;15(5):375– 83. Bastida E, Brown HS 3rd, Pagan JA. Persistent disparities in the use of health care along the U.S.-Mexico border: an ecological perspective. Am J Public Health 2008;98(11):1987–95. Baumbach J, Mueller M, Smelser C, Albanese B, Sewell CM. Enhancement of influenza surveillance with aggregate rapid influenza test results: New Mexico, 2003–2007. Am J Public Health 2009;99(S2): S372–S377. Bazzoli GJ. Public-private collaboration in health and human service delivery: evidence from community partnerships. Milbank Q 1997;75(4): 533– 61. Beaulieu J, Scutchfıeld FD. Assessment of validity of the National Public Health Performance Standards: the local public health performance assessment instrument. Public Health Rep 2002;117(1):28 –36. Beaulieu J, Scutchfıeld FD, Kelly AV. Content and criterion validity evaluation of National Public Health Performance Standards measurement instruments. Public Health Rep 2003;118(6):508 –17. Beaulieu JE, Scutchfıeld FD, Kelly AV. Recommendations from testing of the National Public Health Performance Standards instruments. J Public Health Manag Pract 2003;9(3):188 –98. Beckett AB, Scutchfıeld FD, Pfeifle W, Hill R, Ingram RC. The forgotten instrument: analysis of the National Public Health Performance Standards Program governance instrument. J Public Health Manag Pract 2008;14(4):E17–E22. Befort CA, Orr S, Davis A, Ely A, Steiger K. Perspectives on research among Kansas County health department administrators. J Public Health Manag Pract 2009;15(3):E9 –E15.

www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 Beitsch LM, Brooks RG, Grigg M, Menachemi N. Structure and functions of state public health agencies. Am J Public Health 2006;96(1):167–72. Beitsch LM, Grigg M, Menachemi N, Brooks RG. Roles of local public health agencies within the state public health system. J Public Health Manag Pract 2006;12(3):232– 41. Beitsch LM, Mays G, Corso L, Chang C, Brewer R. States gathering momentum: promising strategies for accreditation and assessment activities in multistate learning collaborative applicant states. J Public Health Manag Pract 2007;13(4):364 –73. Bekemeier B. Credentialing for public health nurses: personally valued . . . but not well recognized. Public Health Nurs 2007;24(5): 439 – 48. Bekemeier B. Nurses’ utilization and perception of the community/public health nursing credential. Am J Public Health 2009;99(5): 944 –9. Bekemeier B, Riley CM, Padgett SM, Berkowitz B. Making the case: leveraging resources toward public health system improvement in Turning Point states. J Public Health Manag Pract 2007;13(6): 649 –54. Bernet PM. Local public health agency funding: money begets money. J Public Health Manag Pract 2007;13(2):188 –93. Bernstein AB, Gauthier AK. Defıning competition in markets: why and how? Health Serv Res 1998;33(5 Pt 2):1421–38. Birkhead GS, Davies J, Miner K, Lemmings J, Koo D. Developing competencies for applied epidemiology: from process to product. Public Health Rep 2008;123(S 1):67–118. Blendon RJ, Buhr T, Cassidy EF, et al. Disparities in health: perspectives of a multi-ethnic, multi-racial America. Health Aff (Millwood) 2007; 26(5):1437– 47. Blendon RJ, Buhr T, Cassidy EF, et al. Disparities in physician care: experiences and perceptions of a multi-ethnic America. Health Aff (Millwood) 2008;27(2):507–17. Boulton ML, Lemmings J, Beck AJ. Assessment of epidemiology capacity in state health departments, 2001–2006. J Public Health Manag Pract 2009;15(4):328 –36. Brooks RG, Beitsch LM, Street P, Chukmaitov A. Aligning public health fınancing with essential public health service functions and National Public Health Performance Standards. J Public Health Manag Pract 2009;15(4):299 –306. Brosnan CA, Brosnan P, Therrell BL, et al. A comparative cost analysis of newborn screening for classic congenital adrenal hyperplasia in Texas. Public Health Rep 1998;113(2):170 – 8. Broz D, Levin EC, Mucha AP, et al. Lessons learned from Chicago’s emergency response to mass evacuations caused by Hurricane Katrina. Am J Public Health 2009;99(8):1496 –504. Bryant-Stephens T, Kurian C, Guo R, Zhao H. Impact of a household environmental intervention delivered by lay health workers on asthma symptom control in urban, disadvantaged children with asthma. Am J Public Health 2009;99(S3):S657–S665. Buehler JW, Whitney EA, Berkelman RL. Business and public health collaboration for emergency preparedness in Georgia: a case study. BMC Public Health 2006;6:285. Burckhardt CS, Anderson KL. The Quality of Life Scale (QOLS): reliability, validity, and utilization. Health Qual Life Outcomes 2003;1:60. Burns LR, Lee JA. Hospital purchasing alliances: utilization, services, and performance. Health Care Manage Rev 2008;33(3):203–15. Butler J, Tews D, Raevsky C, et al. Accreditation/Performance assessment on-site reviews in Michigan, Missouri, North Carolina, and Washington: implications for States and an evolving national model. J Public Health Manag Pract 2007;13(4):395– 403. Cahn MA, Auston I, Selden CR, et al. The Partners in Information Access for the Public Health Workforce: a collaboration to improve and protect the public’s health, 1995–2006. J Med Libr Assoc 2007;95(3):301–9. Cantreill J, Hung D, Fahs MC, Shelley D. Purchasing patterns and smoking behaviors after a large tobacco tax increase: a study of Chinese Americans living in New York City. Public Health Rep 2008;123(2):135– 46.

May 2012

S51

CDC. Impact of expanded newborn screening—U.S., 2006. MMWR Morb Mortal Wkly Rep 2008;57(37):1012–5. CDC. National, state, and local area vaccination coverage among children aged 19-35 months—U.S., 2007. MMWR Morb Mortal Wkly Rep 2008;57(35):961– 6. CDC. State-specifıc influenza vaccination coverage among adults—U.S., 2006 – 07 influenza season. MMWR Morb Mortal Wkly Rep 2008; 57(38):1033–9. CDC. Use of enhanced surveillance for hepatitis C virus infection to detect a cluster among young injection-drug users—New York, November 2004-April 2007. MMWR Morb Mortal Wkly Rep 2008;57(19): 517–21. CDC. State-specifıc prevalence of obesity among adults—U.S., 2007. MMWR Morb Mortal Wkly Rep. 2008 57;28(765– 8. CDC. Subpopulation estimates from the HIV incidence surveillance system—U.S., 2006. MMWR Morb Mortal Wkly Rep 2008; 57(36):985–9. CDC. Impact of seasonal influenza-related school closures on families— Southeastern Kentucky, February 2008. MMWR Morb Mortal Wkly Rep 2009;58(50):1405–9. CDC. Reduced hospitalizations for acute myocardial infarction after implementation of a smoke-free ordinance—City of Pueblo, Colorado, 2002–2006. MMWR Morb Mortal Wkly Rep 2009;57(51): 1373–7. CDC. Status of state electronic disease surveillance systems—U.S., 2007. MMWR Morb Mortal Wkly Rep 2009;58((29):804 –7. CDC. 2009 pandemic influenza A (H1N1) virus infections—Chicago, Illinois, April-July 2009. MMWR Morb Mortal Wkly Rep 2009; 58(33):913– 8. CDC. Adult blood lead epidemiology and surveillance—U.S., 2005–2007. MMWR Morb Mortal Wkly Rep 2009;58(14):365–9. CDC. Deaths related to 2009 pandemic influenza A (H1N1) among American Indian/Alaska Natives—12 states, 2009. MMWR Morb Mortal Wkly Rep 2009;58(48):1341– 4. CDC. Effectiveness of 2008 – 09 trivalent influenza vaccine against 2009 pandemic influenza A (H1N1)—U.S., May-June 2009. MMWR Morb Mortal Wkly Rep 2009;58(44)::1241–5. CDC. Influenza vaccination coverage among children aged 6 –23 months— U.S., 2007– 08 influenza season. MMWR Morb Mortal Wkly Rep 2009;58(38):1063– 6. CDC. Influenza vaccination coverage among children aged 6 months–18 years— eight immunization information system sentinel sites, U.S., 2008 – 09 influenza season. MMWR Morb Mortal Wkly Rep 2009; 58(38):1059 – 62. CDC. Influenza vaccination coverage among children and adults—U.S., 2008 – 09 influenza season. MMWR Morb Mortal Wkly Rep 2009; 58(39):1091–5. CDC. National, state, and local area vaccination coverage among adolescents aged 13–17 years—U.S., 2008. MMWR Morb Mortal Wkly Rep 2009;58(36):997–1001. CDC. National, state, and local area vaccination coverage among children aged 19-35 months—U.S., 2008. MMWR Morb Mortal Wkly Rep 2009;58(33):921– 6. CDC. Norovirus outbreaks on three college campuses—California, Michigan, and Wisconsin, 2008. MMWR Morb Mortal Wkly Rep 2009;58(39):1095–100. CDC. Performance of rapid influenza diagnostic tests during two school outbreaks of 2009 pandemic influenza A (H1N1) virus infection— Connecticut, 2009. MMWR Morb Mortal Wkly Rep 2009; 58(37):1029 –32. CDC. Preliminary FoodNet Data on the incidence of infection with pathogens transmitted commonly through food—10 States, 2008. MMWR Morb Mortal Wkly Rep 2009;58(13):333–7. CDC. Reduction in rotavirus after vaccine introduction—U.S., 2000 –2009. MMWR Morb Mortal Wkly Rep 2009;58(41):1146 –9.

S52

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

CDC. State-specifıc prevalence and trends in adult cigarette smoking— U.S., 1998 –2007. MMWR Morb Mortal Wkly Rep 2009;58(9):221– 6. CDC. State-specifıc secondhand smoke exposure and current cigarette smoking among adults—U.S., 2008. MMWR Morb Mortal Wkly Rep 2009;58(44):1232–5. CDC. Surveillance for pediatric deaths associated with 2009 pandemic influenza A (H1N1) virus infection—U.S., April-August 2009. MMWR Morb Mortal Wkly Rep 2009;58(34):941–7. CDC. Update: influenza activity—U.S., August 30-October 31, 2009. MMWR Morb Mortal Wkly Rep 2009;58(44):1236 – 41. CDC. Update: influenza activity—U.S., April-August 2009. MMWR Morb Mortal Wkly Rep 2009;58(36):1009 –12. CDC. Update: influenza activity—U.S., September 28, 2008-April 4, 2009, and composition of the 2009 –10 influenza vaccine. MMWR Morb Mortal Wkly Rep 2009;58(14):369 –74. CDC. Assessment of epidemiology capacity in State Health Departments— U.S., 2009. MMWR Morb Mortal Wkly Rep 2009;58(49):1373–7. CDC. Intent to receive influenza A (H1N1) 2009 monovalent and seasonal influenza vaccines—two counties, North Carolina, August 2009. MMWR Morb Mortal Wkly Rep 2009;58(50):1401–5. CDC. Assessment of epidemiology capacity in State Health Departments— U.S., 2009. MMWR Morb Mortal Wkly Rep 2009;58(49):1373–7. CDC. Impact of expanded newborn screening—U.S., 2006. MMWR Morb Mortal Wkly Rep 2008;57(37):1012–5. CDC. Influenza vaccination coverage among children aged 6 months–18 years— eight immunization information system sentinel sites, U.S., 2008 – 09 influenza season. MMWR Morb Mortal Wkly Rep 2009; 58(38):1059 – 62. CDC. Influenza vaccination coverage among children aged 6 –23 months— U.S., 2007– 08 influenza season. MMWR Morb Mortal Wkly Rep 2009;58(38):1063– 6. CDC. Influenza vaccination coverage among children and adults—U.S., 2008 – 09 influenza season. MMWR Morb Mortal Wkly Rep 2009; 58(39):1091–5. CDC. Laboratory surveillance for wild and vaccine-derived polioviruses— worldwide, January 2008-June 2009. MMWR Morb Mortal Wkly Rep 2009;58(34):950 – 4. CDC. National laboratory inventories for wild poliovirus containment— Western Pacifıc region, 2008. MMWR Morb Mortal Wkly Rep 2009;58(35):975– 8. CDC. National, state, and local area vaccination coverage among adolescents aged 13–17 years—U.S., 2008. MMWR Morb Mortal Wkly Rep 2009;58(36):997–1001. CDC. National, state, and local area vaccination coverage among children aged 19-35 months—U.S., 2007. MMWR Morb Mortal Wkly Rep 2008;57(35):961– 6. CDC. National, state, and local area vaccination coverage among children aged 19-35 months—U.S., 2008. MMWR Morb Mortal Wkly Rep 2009;58(33):921– 6. CDC. Progress in immunization information systems—U.S., 2008. MMWR Morb Mortal Wkly Rep 2010;59(5):133–5. CDC. Reduced hospitalizations for acute myocardial infarction after implementation of a smoke-free ordinance–City of Pueblo, Colorado, 2002– 2006. MMWR Morb Mortal Wkly Rep 2009;57(51):1373–7. CDC. Reduction in rotavirus after vaccine introduction—U.S., 2000 –2009. MMWR Morb Mortal Wkly Rep 2009;58(41):1146 –9. CDC. State-specifıc influenza vaccination coverage among adults—U.S., 2006 – 07 influenza season. MMWR Morb Mortal Wkly Rep 2008; 57(38):1033–9. CDC. State-specifıc prevalence of obesity among adults—U.S., 2007. MMWR Morb Mortal Wkly Rep 2008;57(28):765– 8. CDC. State-specifıc secondhand smoke exposure and current cigarette smoking among adults—U.S., 2008. MMWR Morb Mortal Wkly Rep 2009;58(44):1232–5. CDC. Status of state electronic disease surveillance systems—U.S., 2007. MMWR Morb Mortal Wkly Rep 2009;58(29):804 –7.

CDC. Subpopulation estimates from the HIV incidence surveillance system—U.S., 2006. MMWR Morb Mortal Wkly Rep 2008; 57(36):985–9. CDC. Surveillance for pediatric deaths associated with 2009 pandemic influenza A (H1N1) virus infection–U.S., April-August 2009. MMWR Morb Mortal Wkly Rep 2009;58(34):941–7. Chan B, Feldman R, Manning WG. The effects of group size and group economic factors on collaboration: a study of the fınancial performance of rural hospitals in consortia. Health Serv Res 1999;34(1 Pt 1):9 –31. Chandler T, Qureshi K, Gebbie KM, Morse SS. Teaching emergency preparedness to public health workers: use of blended learning in webbased training. Public Health Rep 2008;123(5):676 – 80. Cheadle A, Hsu C, Schwartz PM, et al. Involving local health departments in community health partnerships: evaluation results from the partnership for the public’s health initiative. J Urban Health 2008; 85(2):162–77. Chen LS, Kwok OM, Goodson P. U.S. health educators’ likelihood of adopting genomic competencies into health promotion. Am J Public Health 2008;98(9):1651–7. Chenoweth D, Estes C, Lee C. The economic cost of environmental factors among North Carolina children living in substandard housing. Am J Public Health 2009;99(S3):S666 –S674. Chesson HW, Harrison P, Scotton CR, Varghese B. Does funding for HIV and sexually transmitted disease prevention matter? Evidence from panel data. Eval Rev 2005;29(1):3–23. Chintapalli S, Goodman M, Allen M, et al. Assessment of a commercial searchable population directory as a means of selecting controls for case-control studies. Public Health Rep 2009;124(3):378 – 83. Cioffı JP, Lichtveld MY, Tilson H. A research agenda for public health workforce development. J Public Health Manag Pract 2004; 10(3):186 –92. Clark NM, Lachance L, Doctor LJ, et al. Policy and system change and community coalitions: outcomes from allies against asthma. Am J Public Health 2010;100(5):904 –12. Colon-Ramos U, Atienza AA, Weber D, Taylor M, Uy C, Yaroch A. Practicing what they preach: health behaviors of those who provide health advice to extensive social networks. J Health Commun 2009; 14(2):119 –30. Cosby AG, Bowser DM. The health of the Delta Region: a story of increasing disparities. J Health Hum Serv Adm 2008;31(1):58 –71. Costich JF, Honore PA, Scutchfıeld FD. Public health fınancial management needs: report of a national survey. J Public Health Manag Pract 2009;15(4):307–10. Costich JF, Scutchfıeld FD. Public health preparedness and response capacity inventory validity study. J Public Health Manag Pract 2004; 10(3):225–33. Curry CW, De AK, Ikeda RM, Thacker SB. Health burden and funding at the Centers for Disease Control and Prevention. Am J Prev Med 2006;30(3):269 –76. Dausey DJ, Chandra A, Schaefer AG, et al. Measuring the performance of telephone-based disease surveillance systems in local health departments. Am J Public Health 2008;98(9):1706 –11. Davis MV, Cannon MM, Corso L, Lenaway D, Baker EL. Incentives to encourage participation in the national public health accreditation model: a systematic investigation. Am J Public Health 2009; 99(9):1705–11. Davis MV, Macdonald PD, Cline JS, Baker EL. Evaluation of public health response to hurricanes fınds North Carolina better prepared for public health emergencies. Public Health Rep 2007;122(1):17–26. Declercq E, Caldwell K, Hobbs SH, Guyer B. The changing pattern of doctoral education in public health from 1985 to 2006 and the challenge of doctoral training for practice and leadership. Am J Public Health 2008;98(9):1565–9. Defriese GH, Hetherington JS, Brooks EF, et al. The program implications of administrative relationships between local health departments and state and local government. Am J Public Health 1981;71(10):1109 –15.

www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 Delpierre C, Lauwers-Cances V, Datta GD, Berkman L, Lang T. Impact of social position on the effect of cardiovascular risk factors on self-rated health. Am J Public Health 2009;99(7):1278 – 84. Derose SF, Asch SM, Fielding JE, Schuster MA. Developing quality indicators for local health departments—Experience in Los Angeles County. Am J Prev Med 2003;25(4):347–57. Downey LH, Ireson CL, Scutchfıeld FD. The use of photovoice as a method of facilitating deliberation. Health Promot Pract 2008;2008/03/07. Driscoll D, Rojas Smith L, Sotnikov S, et al. An instrument for assessing public health system performance: validity in rural settings. J Rural Health 2006;22(3):254 –9. Duncan AR, Priest PC, Jennings LC, Brunton CR, Baker MG. Screening for influenza infection in international airline travelers. Am J Public Health 2009;99(S2):S360 –S362. Dymova N, Hanumara RC, Enander RT, Gagnon RN. Use of the global test statistic as a performance measurement in a reanalysis of environmental health data. Am J Public Health 2009;99(10):1739 – 41. Edgar M, Mayer JP, Scharff DP. Construct validity of the core competencies for public health professionals. J Public Health Manag Pract 2009; 15(4):E7–E16. Edwards JC, Kang J, Silenas R. Promoting regional disaster preparedness among rural hospitals. J Rural Health 2008;24(3):321–5. Ellis BH, Bannister WM, Cox JK, et al. Utilization of the propensity score method: an exploratory comparison of proxy-completed to selfcompleted responses in the Medicare Health Outcomes Survey. Health Qual Life Outcomes 2003;1:47. Ellison JH. National Public Health Performance Standards: are they a means of evaluating the local public health system? J Public Health Manag Pract 2005;11(5):433– 6. Eng TR. Prevention of sexually transmitted diseases—a model for overcoming barriers between managed care and public health. Am J Prev Med 1999;16(1):60 –9. Erwin PC. A self-assessment process for accreditation preparedness: a practical example for local health departments. J Public Health Manag Pract 2009;15(6):503– 8. Erwin PC, Sheeler L, Lott JM. A shot in the rear, not a shot in the dark: application of a mass clinic framework in a public health emergency. Public Health Rep 2009;124(2):212– 6. Farhang L, Bhatia R, Scully CC, Corburn J, Gaydos M, Malekafzali S. Creating tools for healthy development: case study of San Francisco’s Eastern Neighborhoods Community Health Impact Assessment. J Public Health Manag Pract 2008;14(3):255– 65. Fielding S, Fayers PM, Mcdonald A, Mcpherson G, Campbell MK. Simple imputation methods were inadequate for missing not at random (MNAR) quality of life data. Health Qual Life Outcomes 2008;6:57. Fielding S, Fayers PM, Ramsay CR. Investigating the missing data mechanism in quality of life outcomes: a comparison of approaches. Health Qual Life Outcomes 2009;7:57. Findley SE, Irigoyen M, Sanchez M, et al. Effectiveness of a community coalition for improving child vaccination rates in New York City. Am J Public Health 2008;98(11):1959 – 62. Ford EW, Wells R, Bailey B. Sustainable network advantages: a game theoretic approach to community-based health care coalitions. Health Care Manag Rev 2004;29(2):159 – 69. Foster-Fishman PG, Pierce SJ, Van Egeren LA. Who participates and why: building a process model of citizen participation. Health Educ Behav 2009;36(3):550 – 69. Fox M, Curriero F, Kulbicki K, Resnick B, Burke T. Evaluating the community health legacy of WWI chemical weapons testing. J Community Health 2010;35(1):93–103. Garcia R, Fenwick C. Social science, equal justice, and public health policy: lessons from Los Angeles. J Public Health Policy 2009;30(S1):S26 –S32. George A, Springer C, Haughton B. Retirement intentions of the public health nutrition workforce. J Public Health Manag Pract 2009; 15(2):127–34.

May 2012

S53

Gerzoff RB, Brown CK, Baker EL. Full-time employees of U.S. local health departments, 1992–1993. J Public Health Manag Pract 1999;5(3):1–9. Gerzoff RB, Gordon RL, Richards TB. Recent changes in local health department spending. J Public Health Policy 1996;17(2):170 – 80. Ginsburg M, Goold SD, Danis M. (De)constructing “basic” benefıts: citizens defıne the limits of coverage. Health Aff (Millwood) 2006; 25(6):1648 –55. Gizzi C, Klementiev A, Britt J, Cruz-Uribe F. The use of assessment in promoting secondhand smoke policy in a local health jurisdiction. J Public Health Manag Pract 2009;15(1):41– 6. Goes JB, Zhan C. The effects of hospital-physician integration strategies on hospital fınancial performance. Health Serv Res 1995;30(4):507–30. Goldman RH, Cohen AP, Sheahan F. Using seminar blogs to enhance student participation and learning in public health school classes. Am J Public Health 2008;98(9):1658 – 63. Goldsmith LJ, Ricketts TC. Proposed changes to designations of medically underserved populations and health professional shortage areas: effects on rural areas. J Rural Health 1999;15(1):44 –54. Goodman RM. A construct for building the capacity of community-based initiatives in racial and ethnic communities: a qualitative cross-case analysis. J Public Health Manag Pract 2009;15(2):E1–E8. Goody B. Defıning rural hospital markets. Health Serv Res 1993; 28(2):183–200. Goold SD, Fessler D, Moyer CA. A measure of trust in insurers. Health Serv Res 2006;41(1):58 –78. Goold SD, Klipp G. Managed care members talk about trust. Soc Sci Med 2002;54(6):879 – 88. Gordon RL, Baker EL, Roper WL, Omenn GS. Prevention and the reforming U.S. health care system: changing roles and responsibilities for public health. Annu Rev Public Health 1996;17:489 –509. Gordon RL, Gerzoff RB, Richards TB. Determinants of U.S. local health department expenditures, 1992 through 1993. Am J Public Health 1997;87(1):91–5. Graham JP, Leibler JH, Price LB. The animal-human interface and infectious disease in industrial food animal production: rethinking biosecurity and biocontainment. Public Health Rep 2008;123(3):282–99. Griffın SO, Jones KA, Lockwood S, Mosca NG, Honore PA. Impact of increasing Medicaid dental reimbursement and implementing school sealant programs on sealant prevalence. J Public Health Manag Pract 2007;13(2):202– 6. Gwynn RC, Garg RK, Kerker BD, Frieden TR, Thorpe LE. Contributions of a local health examination survey to the surveillance of chronic and infectious diseases in New York City. Am J Public Health 2009; 99(1):152–9. Haffer SC. Using multiple survey vendors to collect health outcomes information: how accurate are the data? Health Qual Life Outcomes 2003;1:6. Hajat A, Cilenti D, Harrison LM, et al. What predicts local public health agency performance improvement? A pilot study in North Carolina J Public Health Manag Pract 2009;15(2):E22–33. Halverson PK, Mays GP, Kaluzny AD. Working together? Organizational and market determinants of collaboration between public health and medical care providers. Am J Public Health 2000;90(12):1913– 6. Handler AS, Turnock BJ. Local health department effectiveness in addressing the core functions of public health: essential ingredients. J Public Health Policy 1996;17(4):460 – 83. Hanusaik N, O’loughlin JL, Kishchuk N, Eyles J, Robinson K, Cameron R. Building the backbone for organisational research in public health systems: development of measures of organisational capacity for chronic disease prevention. J Epidemiol Community Health 2007; 61(8):742–9. Harden A, Garcia J, Oliver S, et al. Applying systematic review methods to studies of people’s views: an example from public health research. J Epidemiol Community Health 2004;58(9):794 – 800.

S54

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

Harrison KM, Kajese T, Hall HI, Song R. Risk factor redistribution of the national HIV/AIDS surveillance data: an alternative approach. Public Health Rep 2008;123(5):618 –27. Helmkamp JC, Aitken ME, Lawrence BA. ATV and bicycle deaths and associated costs in the U.S., 2000 –2005. Public Health Rep 2009; 124(3):409 –18. Hempstead K. The accuracy of a death certifıcate checkbox for diabetes: early results from New Jersey. Public Health Rep 2009;124(5):726 –32. Hendryx M, Ahern MM. Mortality in Appalachian coal mining regions: the value of statistical life lost. Public Health Rep 2009;124(4):541–50. Hessler BJ, Soper P, Bondy J, Hanes P, Davidson A. Assessing the relationship between health information exchanges and public health agencies. J Public Health Manag Pract 2009;15(5):416 –24. Holtgrave DR, Kates J. HIV incidence and CDC’s HIV prevention budget: an exploratory correlational analysis. Am J Prev Med 2007;32(1):63–7. Honoré PA, Simoes EJ, Jones WJ, Moonesinghe R. Practices in public health fınance: an investigation of jurisdiction funding patterns and performance. J Public Health Manag Pract 2004;10(5):444 –50. Honoré PA, Simoes EJ, Moonesinghe R, Kirbey HC, Renner M. Applying principles for outcomes-based contracting in a public health program. J Public Health Manag Pract 2004;10(5):451–7. Honoré PA, Simoes EJ, Moonesinghe R, Wang X, Brown L. Evaluating the ecological association of casino industry economic development on community health status: a natural experiment in the Mississippi Delta region. J Public Health Manag Pract 2007;13(2):214 –22. Horney JA. Evaluation of the Certifıcate in Community Preparedness and Disaster Management Program at the University of North Carolina Gillings School of Global Public Health. Public Health Rep 2009; 124(4):610 – 6. Hosler AS. Retail food availability, obesity, and cigarette smoking in rural communities. J Rural Health 2009;25(2):203–10. Hurley R, Grossman J, Lake T, Casalino L. A longitudinal perspective on health plan-provider risk contracting. Health Aff (Millwood) 2002; 21(4):144 –53. Hurley RE, Grossman JM, Strunk BC. Medicare contracting risk/Medicare risk contracting: a life-cycle view from twelve markets. Health Serv Res 2003;38(1 Pt 2):395– 417. Hwang J, Mcclintock S, Kachur SP, Slutsker L, Arguin P. Comparison of national malaria surveillance system with the national notifıable diseases surveillance system in the U.S. J Public Health Manag Pract 2009;15(4):345–51. Jacobson PD. Measuring the value of public health systems. University of Michigan School of Public Health, Department of Health Management and Policy, School of Public Health, 2006. Jacobson PD, Neumann PJ. A framework to measure the value of public health services. Health Serv Res 2009;44(5):1880 –96. Janosky JE, Laird SB, Sun Q. Content and context of a research registry for community-based research. J Community Health 2008;33(4):270 – 8. Jewell CJ, Bero LA. ”Developing good taste in evidence”: facilitators of and hindrances to evidence-informed health policymaking in state government. Milbank Q 2008;86(2):177–208. Jones M, O’carroll P, Thompson J, D’ambrosio L. Assessing regional public health preparedness: a new tool for considering cross-border issues. J Public Health Manag Pract 2008;14(5):E15–22. Kanarek N, Fitzek B, Su SC, Brower M, Jia H. County lung cancer mortality: a decision tree model for control and prevention. J Public Health Manag Pract 2008;14(4):E1–9. Kanarek N, Stanley J, Bialek R. Local public health agency performance and community health status. J Public Health Manag Practice 2006; 12(6):522–7. Kazda MJ, Beel ER, Villegas D, Martinez JG, Patel N, Migala W. Methodological complexities and the use of GIS in conducting a community needs assessment of a large U.S. municipality. J Community Health 2009;34(3):210 –5. Keane C, Marx J, Ricci E. Local health departments’ mission to the uninsured. J Public Health Policy 2003;24(2):130 – 49.

Kennedy VC. Public health workforce employment in U.S. public and private sectors. J Public Health Manag Pract 2009;15(3):E1– 8. Kerby DS, Brand MW, Johnson DL, Ghouri FS. Self-assessment in the measurement of public health workforce preparedness for bioterrorism or other public health disasters. Public Health Rep 2005;120(2):186 –91. Kessel A, Green J, Pinder R, Wilkinson P, Grundy C, Lachowycz K. Multidisciplinary research in public health: a case study of research on access to green space. Public Health 2009;123(1):32– 8. Kinner K, Pellegrini C. Expenditures for public health: assessing historical and prospective trends. Am J Public Health 2009;99(10):1780 –91. Kohr JM, Strack RW, Newton-Ward M, Cooke CH. The use of programme planning and social marketing models by a state public health agency: a case study. Public Health 2008;122(3):300 – 6. Kohrs FP, Mainous AG. Is health status related to residence in medically underserved areas? Evidence and implications for policy. J Rural Health 1996;12(3):218 –24. Kramer MR, Hogue CR. Place matters: variation in the black/white very preterm birth rate across U.S. metropolitan areas, 2002–2004. Public Health Rep 2008;123(5):576 – 85. Kruger DJ, Shirey L, Morrel-Samuels S, Skorcz S, Brady J. Using a communitybased health survey as a tool for informing local health policy. J Public Health Manag Pract 2009;15(1):47–53. Lacar ES, Soto X, Riley WJ. Adolescent obesity in a low-income Mexican American district in South Texas. Arch Pediatr Adolesc Med 2000; 154(8):837– 40. Lauer J, Kastner J, Nutsch A. Primary care physicians and pandemic influenza: an appraisal of the 1918 experience and an assessment of contemporary planning. J Public Health Manag Pract 2008;14(4):379 – 86. Lee CJ. Does the internet displace health professionals? J Health Commun 2008;13(5):450 – 64. Leep C, Beitsch LM, Gorenflo G, Solomon J, Brooks RG. Quality improvement in local health departments: progress, pitfalls, and potential. J Public Health Manag Pract 2009;15(6):494 –502. Lempa M, Goodman RM, Rice J, Becker AB. Development of scales measuring the capacity of community-based initiatives. Health Educ Behav 2008;35(3):298 –315. Leon K, Mcdonald MC, Moore B, Rust G. Disparities in influenza treatment among disabled Medicaid patients in Georgia. Am J Public Health 2009;99(S2):S378 –S382. Li F, Harmer P, Cardinal BJ, Bosworth M, Johnson-Shelton D. Obesity and the built environment: does the density of neighborhood fast-food outlets matter? Am J Health Promot 2009;23(3):203–9. Linas BP, Zheng H, Losina E, Walensky RP, Freedberg KA. Assessing the impact of federal HIV prevention spending on HIV testing and awareness. Am J Public Health 2006;96(6):1038 – 43. Marshall H, Ryan P, Roberton D, Street J, Watson M. Pandemic influenza and community preparedness. Am J Public Health 2009;99(S2): S365–S371. Martin EG, Pollack HA, Paltiel AD. Fact, fıction, and fairness: resource allocation under the Ryan White CARE Act. Health Aff (Millwood) 2006;25(4):1103–12. Mays GP, Halverson P, Miller CA. Assessing the performance of local public health systems: a survey of state health agency efforts. J Public Health Manag Pract 1998;4(4):63–78. Mays GP, Halverson PK, Baker EL, Stevens R, Vann JJ. Availability and perceived effectiveness of public health activities in the nation’s most populous communities. Am J Public Health 2004;94(6):1019 –26. Mays GP, Halverson PK, Stevens R. The contributions of managed care plans to public health practice: evidence from the nation’s largest local health departments. Public Health Rep 2001;116:50 – 67. Mays GP, Hesketh HA, Ammerman AS, Stockmyer CK, Johnson TL, Bayne-Smith M. Integrating preventive health services within community health centers: lessons from WISEWOMAN. J Womens Health 2004;13(5):607–15.

www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 Mays GP, Hurley RE, Grossman JM. An empty toolbox? Changes in health plans’ approaches for managing costs and care. Health Serv Res 2003; 38(1 Pt 2):375–93. Mays GP, Mchugh MC, Shim K, et al. Getting what you pay for: public health spending and the performance of essential public health services. J Public Health Manag Pract 2004;10(5):435– 43. Mays GP, Mchugh MC, Shim K, et al. Identifying dimensions of performance in local public health systems: results from the National Public Health Performance Standards Program. J Public Health Manag Pract 2004;10(3):193–203. Mays GP, Mchugh MC, Shim K, et al. Institutional and economic determinants of public health system performance. Am J Public Health 2006;96(3):523–31. Mays GP, Scutchfıeld FD, Bhandari MW, Smith SA. Understanding the organization of public health delivery systems: an empirical typology. Milbank Q 2010;88(1):81–111. Mays GP, Smith SA. Geographic variation in public health spending: correlates and consequences. Health Serv Res 2009;44(5):1796 – 817. Mccann PJC. Agency discretion and public health service delivery. Health Serv Res 2009;44(5):1897–908. Mccarthy WJ, Mistry R, Lu Y, Patel M, Zheng H, Dietsch B. Density of tobacco retailers near schools: effects on tobacco use among students. Am J Public Health 2009;99(11):2006 –13. Mcclellan CS. Utilizing a national performance standards local public health assessment instrument in a community assessment process: the Clarendon County Turning Point Initiative. J Public Health Manag Pract 2005;11(5):428 –32. Mccue MJ, Mccall N, Hurley RE, Wyttenbach M, White M. Financial performance and participation in Medicaid and Medi-Cal managed care. Health Care Financ Rev 2001;23(2):69 – 81. Mcdonnell KA, Strobino DM, Baldwin KM, Grason H, Misra DP. Comparison of FIMR programs with other perinatal systems initiatives. Matern Child Health J 2004;8(4):231– 8. Mcgrath MM, Fullilove RE, Kaufman MR, Wallace R, Fullilove MT. The limits of collaboration: a qualitative study of community ethical review of environmental health research. Am J Public Health 2009;99(8): 1510 – 4. Mckinney MM, Morrissey JP, Kaluzny AD. Interorganizational exchanges as performance markers in a community cancer network. Health Serv Res 1993;28(4):459 –78. Meit M, Ettaro L, Hamlin BN, Piya B. Rural public health fınancing: implications for community health promotion initiatives. J Public Health Manag Pract 2009;15(3):210 –5. Merrill J, Keeling J, Gebbie K. Toward standardized, comparable public health systems data: a taxonomic description of essential public health work. Health Serv Res 2009;44(5):1818 – 41. Merrill J, Meier BM, Keeling J, Jia H, Gebbie KM. Examination of the relationship between public health statute modernization and local public health system performance. J Public Health Manag Pract 2009;15(4):292– 8. Miller CA, Moore KS, Richards TB. The impact of critical events of the 1980s on core functions for a selected group of local health departments. Public Health Rep 1993;108(6):695–700. Miller CA, Moore KS, Richards TB, McKaig C. A screening survey to assess local public health performance. Public Health Rep 1994;109(5): 659 – 64. Miller CA, Moore KS, Richards TB, Monk JD. A proposed method for assessing the performance of local public health functions and practices. Am J Public Health 1994;84(11):1743–9. Miriti MK, Billah K, Weinbaum C, et al. Economic benefıts of hepatitis B vaccination at sexually transmitted disease clinics in the U.S. Public Health Rep 2008;123(4):504 –13. Misra DP, Grason H, Liao M, Strobino DM, Mcdonnell KA, Allston AA. The nationwide evaluation of fetal and infant mortality review (FIMR) programs: development and implementation of recommendations and

May 2012

S55

conduct of essential maternal and child health services by FIMR programs. Matern Child Health J]?2004; 8/sb:volume-nr4):217–29. Molinari C, Morlock L, Alexander J, Lyles CA. Hospital board effectiveness: relationships between governing board composition and hospital fınancial viability. Health Serv Res 1993;28(3):358 –77. Moore J. Studying an ill-defıned workforce: public health workforce research. J Public Health Manag Pract 2009;15(6S):S48 –S53. Morrissey J, Calloway M, Johnsen M, Ullman M. Service system performance and integration: a baseline profıle of the ACCESS demonstration sites: access to community care and effective services and supports. Psychiatr Serv 1997;48(3):374 – 80. National Association of County and City Health Offıcials. 2005 national profıle of local health departments. Washington DC: NACCHO, 2006. Nelson JC, Bittner RC, Bounds L, et al. Compliance with multiple-dose vaccine schedules among older children, adolescents, and adults: results from a vaccine safety datalink study. Am J Public Health 2009;99 Suppl 2:S389 –97. Nelson JC, Rashid H, Galvin VG, Essien JD, Levine LM. Public/private partners. Key factors in creating a strategic alliance for community health. Am J Prev Med 1999;16(3S):94 –102. Neumann PJ, Jacobson PD, Palmer JA. Measuring the value of public health systems: the disconnect between health economists and public health practitioners. Am J Public Health 2008;98(12):2173– 80. Noonan VK, Kopec JA, Noreau L, et al. Comparing the content of participation instruments using the international classifıcation of functioning, disability and health. Health Qual Life Outcomes 2009;7:93. Odierna DH, Schmidt LA. The effects of failing to include hard-to-reach respondents in longitudinal surveys. Am J Public Health 2009;99(8): 1515–21. Olden PC, Clement DG. The prevalence of hospital health promotion and disease prevention services: good news, bad news, and policy implications. Milbank Quarterly 2000;78(1):115– 46, iii–iv. Oliva G, Rienks J, Chavez GF. Evaluating a program to build data capacity for core public health functions in local maternal child and adolescent health programs in California. Matern Child Health J 2007;11(1):1–10. Orians C, Rose S, Hubbard B, et al. Strengthening the capacity of local health agencies through community-based assessment and planning. Public Health Rep 2009;124(6):875– 82. Padget SM, Bekemeier B, Berkowitz B. Collaborative partnerships at the state level: promoting systems changes in public health infrastructure. J Public Health Manag Pract 2004;10(3):251–7. Padgett SM, Bekemeier B, Berkowitz B. Building sustainable public health systems change at the state level. J Public Health Manag Pract 2005;11(2):109 –15. Pals SL, Murray DM, Alfano CM, Shadish WR, Hannan PJ, Baker WL. Individually randomized group treatment trials: a critical appraisal of frequently used design and analytic approaches. Am J Public Health 2008;98(8):1418 –24. Patel AS, Powell TA, Woolard CD. Assessment of applied epidemiology competencies among the Virginia Department of Health workforce. Public Health Rep 2008;123(S1):119 –27. Pati S, Danagoulian S. Immigrant children’s reliance on public health insurance in the wake of immigration reform. Am J Public Health 2008;98(11):2004 –10. Pearcy JN, Keppel KG. Monitoring change in health disparity. J Public Health Manag Pract 2008;14(5):481– 6. Pearson J, Windsor R, El-Mohandes A, Perry DC. Evaluation of the immediate impact of the Washington, D.C., smoke-free indoor air policy on bar employee environmental tobacco smoke exposure. Public Health Rep 2009;124(S1):134 – 42. Peck BM, Asch DA, Goold SD. Measuring patient expectations: does the instrument affect satisfaction or expectations? Med Care 2001;39(1): 100 – 8. Pillay R. The skills gap in hospital management in the South African public health sector. J Public Health Manag Pract 2008;14(5):E8 –14.

S56

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

Pinto RM. Community perspectives on factors that influence collaboration in public health research. Health Educ Behav 2009;36(5):930 – 47. Potter MA, Fitzpatrick T. State funding for local public health: observations from six case studies. J Public Health Manag Pract 2007;13(2):163– 8. Prentice RL, Anderson GL. The women’s health initiative: lessons learned. Annu Rev Public Health 2008;29:131–50. Purcell JM. Recruiting the future public health workforce: an analysis of prospect communication among accredited Schools of Public Health. J Community Health 2009;34(3):216 –21. Ransom PE, Wei Y, Stellman SD. Community capacity for cancer prevention. J Health Hum Serv Adm 2009;32(1):5–29. Reed J, Pavletic D, Devlin L, Davis MV, Beitsch LM, Baker EL. Piloting a state health department accreditation model: the North Carolina experience. J Public Health Manag Pract 2009;15(2):85–95. Resnick BA, Zablotsky J, Janus ER, Maggy B, Burke TA. An examination of environmental public health organizational and workforce confıgurations in the Northeast/Mid-Atlantic U.S.: how do we determine if these confıgurations impact performance? J Public Health Manag Pract 2009;15(6):509 –17. Riley W, Brewer R. Review and analysis of quality improvement techniques in police departments: application for public health. J Public Health Manag Pract 2009;15(2):139 – 49. Riley W, Parsons H, Mccoy K, Burns D, Anderson D, Lee S, Sainfort F. Introducing quality improvement methods into local public health departments: structured evaluation of a statewide pilot project. Health Serv Res 2009;44(5):1863–79. Rohan AM, Booske BC, Remington PL. Using the Wisconsin County Health Rankings to catalyze community health improvement. J Public Health Manag Pract 2009;15(1):24 –32. Rosenfeld LA, Fox CE, Kerr D, et al. Use of computer modeling for emergency preparedness functions by local and state health offıcials: a needs assessment. J Public Health Manag Pract 2009;15(2):96 –104. Rosenheck R, Morrissey J, Lam J, et al. Service delivery and community: social capital, service systems integration, and outcomes among homeless persons with severe mental illness. Health Serv Res 2001;36(4): 691–710. Ruth BJ, Sisco S, Wyatt J, Bethke C, Bachman SS, Piper TM. Public health and social work: training dual professionals for the contemporary workplace. Public Health Rep 2008;123(S2):71–7. Savoia E, Massin-Short SB, Rodday AM, Aaron LA, Higdon MA, Stoto MA. Public health systems research in emergency preparedness: a review of the literature. Am J Prev Med 2009;37(2):150 – 6. Savoia E, Testa MA, Biddinger PD, et al. Assessing public health capabilities during emergency preparedness tabletop exercises: reliability and validity of a measurement tool. Public Health Rep 2009;124(1):138 – 48. Schauffler HH, Chapman SA. Health promotion and managed care: Surveys of California’s health plans and population. Am J Prev Med 1998;14(3):161–7. Schlesinger M. Paradigms lost: The persisting search for community in U.S. health policy. J Health Politics Policy and Law 1997;22(4):937–92. Schlesinger M, Gray BH, Gusmano M. A broader vision for managed care, part 3: The scope and determinants of community benefıts. Health Aff 2004;23(3):210 –21. Schlitt JJ, Juszczak LJ, Eichner NH. Current status of state policies that support school-based health centers. Public Health Rep 2008;123(6): 731– 8. Scutchfıeld FD, Beaulieu J, Ireson C, Buege A. Public health competencies required by managed care organizations. J Public Health Manag Pract 2002;8(5):22–9. Scutchfıeld FD, Beversdof CA, Hiltabiddle SE, Violante T. A survey of state health department compliance with the recommendations of the Institute of Medicine report, The Future of Public Health. J Public Health Policy 1997;18(1):13–29. Scutchfıeld FD, Hall L, Ireson CL. The public and public health organizations: issues for community engagement in public health. Health Policy 2006;77(1):76 – 85.

Scutchfıeld FD, Hiltabiddle SE, Rawding N, Violante T. Compliance with the recommendations of the Institute of Medicine report, The Future of Public Health: a survey of local health departments. J Public Health Policy 1997;18(2):155– 66. Scutchfıeld FD, Knight EA, Kelly AV, Bhandari MW, Vasilescu IP. Local public health agency capacity and its relationship to public health system performance. J Public Health Manag Pract 2004;10(3):204 –15. Scutchfıeld FD, Spain C, Pointer DD, Hafey JM. The public health leadership institute: leadership training for state and local health offıcers. J Public Health Policy 1995;16(3):304 –23. Scutchfıeld FD, Zuniga De Nuncio ML, Bush RA, Fainstein SH, Larocco MA, Anvar N. The presence of total quality management and continuous quality improvement processes in California public health clinics. J Public Health Manag Pract 1997;3(3):57– 60. Seagle HM, Moore JB, Dubose KD. An assessment of the walkability of two school neighborhoods in Greenville, North Carolina. J Public Health Manag Pract 2008;14(3):e1– 8. Shebl F, Poppell CE, Zhan M, et al. Measuring health behaviors and landline telephones: potential coverage bias in a low-income, rural population. Public Health Rep 2009;124(4):495–502. Shores KA, West ST. The relationship between built park environments and physical activity in four park locations. J Public Health Manag Pract 2008;14(3):e9 –16. Siegel C, Haugland G, Chambers ED. Performance measures and their benchmarks for assessing organizational cultural competency in behavioral health care service delivery. Admin Policy Ment Health 2003; 31(2):141–70. Skiles MP, Scott KD, Young C. When the cat is out of the bag: a case study in public health rationing in Oregon during the 2004 –2005 influenza vaccine shortage. J Public Health Manag Pract 2008;14(5):464 –70. Smith PJ, Singleton JA. Vaccination coverage estimates for selected counties: achievement of Healthy People 2010 goals and association with indices of access to care, economic conditions, and demographic composition. Public Health Rep 2008;123(2):155–72. Spice C, Snyder K. Reviewing self-reported impacts of community health assessment in local health jurisdictions. J Public Health Manag Pract 2009;15(1):18 –23. Spinello EF, Fischbach R. Using a Web-based simulation as a problembased learning experience: perceived and actual performance of undergraduate public health students. Public Health Rep 2008;123(S2): 78 – 84. Stoto MA. Regionalization in local public health systems: variation in rationale, implementation, and impact on public health preparedness. Public Health Rep 2008;123(4):441–9. Stoto MA, Straus SG, Bohn C, Irani P. A Web-based tool for assessing and improving the usefulness of community health assessments. J Public Health Manag Pract 2009;15(1):10 –7. Strobino DM, Baldwin KM, Grason H, et al. The relation of FIMR programs and other perinatal systems initiatives with maternal and child health activities in the community. Matern Child Health J 2004;8(4):239 – 49. Studnicki J, Gipson LS, Berndt DJ, et al. Special healthcare taxing districts: association with population health status. Am J Prev Med 2007; 32(2):116 –23. Subramanyan GS, Yokoe DS, Sharnprapai S, Nardell E, Mccray E, Platt R. Using automated pharmacy records to assess the management of tuberculosis. Emerg Infect Dis 1999;5(6):788 –91. Thacker SB, Brownson RC. Practicing epidemiology: how competent are we? Public Health Rep 2008;123(S1):4 –5. Thacker SB, Stroup DF, Carande-Kulis V, Marks JS, Roy K, Gerberding JL. Measuring the public’s health. Public Health Rep 2006;121(1):14 –22. Thomas JC, Macdonald PD, Wenink E. Ethical decision making in a crisis: a case study of ethics in public health emergencies. J Public Health Manag Pract 2009;15(2):E16 –21.

www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 Toomey TL, Chen V, Forster JL, Van Coevering P, Lenk KM. Do cigarette prices vary by brand, neighborhood, and store characteristics? Public Health Rep 2009;124(4):535– 40. Tremain B, Davis M, Joly B, Edgar M, Kushion ML, Schmidt R. Evaluation as a critical factor of success in local public health accreditation programs. J Public Health Manag Pract 2007;13(4):404 –9. Trepka MJ, Martin P, Mavunda K, Rodriguez D, Zhang G, Brown C. A pilot asthma incidence surveillance system and case defınition: lessons learned. Public Health Rep 2009;124(2):267–79. Trust for America’s Health. Shortchanging America’s health 2006: a stateby-state look at how federal public health dollars are spent. Washington DC: Trust for America’s Health, 2006. Tsakos G, Bernabe E, O’brien K, Sheiham A, De Oliveira C. Comparison of the self-administered and interviewer-administered modes of the childOIDP. Health Qual Life Outcomes 2008;6:40. Turnock BJ, Handler A, Hall W, Potsic S, Nalluri R, Vaughn EH. Local health department effectiveness in addressing the core functions of public health. Public Health Rep 1994;109(5):653– 8. Turnock BJ, Handler AS, Miller CA. Core function-related local public health practice effectiveness. J Public Health Manag Pract 1998; 4(5):26 –32. Tutor RP, Zarate MA, Loury S. Pesticide exposure surveillance and prevention skills of staff in eastern North Carolina health departments. J Public Health Manag Pract 2008;14(3):299 –310. Uscher-Pines L, Farrell CL, Cattani J, et al. A survey of usage protocols of syndromic surveillance systems by state public health departments in the U.S. J Public Health Manag Pract 2009;15(5):432– 8. Van Meijgaard J, Fielding JE, Kominski GF. Assessing and forecasting population health: integrating knowledge and beliefs in a comprehensive framework. Public Health Rep 2009;124(6):778 – 89. Waters HR, Foldes SS, Alesci NL, Samet J. The economic impact of exposure to secondhand smoke in Minnesota. Am J Public Health 2009;99(4):754 –9. Weiler P, Boggess J, Eastman E, Pomer B. The implementation of model standards in local health departments. American J Public Health 1982;72(11):1230 –7. Weiner BJ, Shortell SM, Alexander J. Promoting clinical involvement in hospital quality improvement efforts: the effects of top management, board, and physician leadership. Health Serv Res 1997;32(4): 491–510. Wells R, Lemak CH, D’aunno TA. Factors associated with interorganizational relationships among outpatient drug treatment organizations 1990 –2000. Health Serv Res 2005;40(5):1356 –78. Wheeler JG, Pulley L, Felix HC, et al. Impact of a smoke-free hospital campus policy on employee and consumer behavior. Public Health Rep 2007;122(6):744 –52. Whittingham J, Ruiter RA, Zimbile F, Kok G. Experimental pretesting of public health campaigns: a case study. J Health Commun 2008; 13(3):216 –29. Wholey DR, Gregg W, Moscovice I. Public health systems: a social networks perspective. Health Services Res 2009;44(5):1842– 62. Wilczynski NL, Haynes RB, Lavis JN, Ramkissoonsingh R, Arnold-Oatley AE. Optimal search strategies for detecting health services research studies in MEDLINE. CMAJ 2004;171(10):1179 – 85. Williams JC. State of emergency preparedness of Kentucky’s rural public health workforce: assessing its ability to identify community health problems. Public Health Rep 2008;123(2):178 – 88. Wing C. Effects of written informed consent requirements on HIV testing rates: evidence from a natural experiment. Am J Public Health 2009;99(6):1087–92. Wing S, Horton RA, Muhammad N, Grant GR, Tajik M, Thu K. Integrating epidemiology, education, and organizing for environmental justice:

May 2012

S57

community health effects of industrial hog operations. Am J Public Health 2008;98(8):1390 –7. Wingo PA, Kulkarni A, Borrud LG, Mcdonald JA, Villalobos SA, Green DC. Health disparities among Mexican American women aged 15-44 years: National Health and Nutrition Examination Survey, 1999 –2004. Am J Public Health 2009;99(7):1300 –7. Wray RJ, Becker SM, Henderson N, et al. Communicating with the public about emerging health threats: lessons from the Pre-Event Message Development Project. Am J Public Health 2008;98(12):2214 –22. Yawn BP, Fryer GE, Phillips RL, Dovey SM, Lanier D, Green LA. Using the ecology model to describe the impact of asthma on patterns of health care. BMC Pulm Med 2005;5:7. Yeh SCJ, Wan TTH, Huang CH, Lo YY. Ambulatory care visits and physician satisfaction: From medical directors’ perspectives. Health Care Manag Rev 2007;32(3):236 – 44. Yoo BK, Kasajima M, Fiscella K, Bennett NM, Phelps CE, Szilagyi PG. Effects of an ongoing epidemic on the annual influenza vaccination rate and vaccination timing among the Medicare elderly: 2000 –2005. Am J Public Health 2009;99(S2):S383–S388. Young G, Beekun RI, Ginn GO. Governing board structure, business strategy, and performance of acute care hospitals: a contingency perspective. Health Serv Res 1992;27(4):543– 64. Yu SM, Huang ZJ, Kogan MD. State-level health care access and use among children in US immigrant families. Am J Public Health 2008; 98(11):1996 –2003. Zahner SJ. Memoranda of understanding between Medicaid MCOs and public health departments. Manag Care 2001;10(9):47–52. Zahner SJ. Evaluation of mandated memoranda of understanding between local health departments and Medicaid managed care plans. J Public Health Manag Pract 2002;8(5):11–21. Zahner SJ. Local public health system partnerships. Public Health Rep 2005;120(1):76 – 83. Zahner SJ. Partnerships for learning population-based public health nursing: web-delivered continuing education for public health nurse preceptors. Public Health Nurs 2006;23(6):547–54. Zahner SJ, Corrado SM. Local health department partnerships with faith-based organizations. J Public Health Manag Pract 2004; 10(3):258 – 65. Zahner SJ, Gredig QN. Improving public health nursing education: recommendations of local public health nurses. Public Health Nurs 2005;22(5):445–50. Zahner SJ, Gredig QN. Public health nursing practice change and recommendations for improvement. Public Health Nurs 2005;22(5):422– 8. Zahner SJ, Kaiser B, Kapelke-Dale J. Local partnerships for community assessment and planning. J Public Health Manag Pract 2005;11(5): 460 – 4. Zahner SJ, Vandermause R. Local health department performance: compliance with state statutes and rules. J Public Health Manag Pract 2003;9(1):25–34. Zaric GS, Brandeau ML. Optimal investment in a portfolio of HIV prevention programs. Med Decis Making 2001;21(5):391– 408. Zuniga MA, Carrillo-Zuniga G, Seol YH, Fos PJ. Multi-criteria assessment of county public health capability disparities. J Health Hum Serv Adm 2009;32(3):238 –58.

Appendix Supplementary data Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.amepre.2012.01.028.