Theory-based evaluation of a comprehensive Latino education initiative: An interactive evaluation approach

ARTICLE IN PRESS

Evaluation and Program Planning 30 (2007) 267–281 www.elsevier.com/locate/evalprogplan

Theory-based evaluation of a comprehensive Latino education initiative: An interactive evaluation approach Teresa M. Nesmana,, Catherine Batscheb, Mario Hernandezc a

Louis de la Parte Florida Mental Health Institute, MHC 2428, 13301 Bruce B. Downs Blvd., University of South Florida, Tampa, FL 33612, USA Louis de la Parte Florida Mental Health Institute, MHC 1115, 13301 Bruce B. Downs Blvd., University of South Florida, Tampa, FL 33612, USA c Louis de la Parte Florida Mental Health Institute, MHC 2437, 13301 Bruce B. Downs Blvd., University of South Florida, Tampa, FL 33612, USA

b

Received 7 February 2007; received in revised form 4 April 2007; accepted 4 April 2007

Abstract Latino student access to higher education has received significant national attention in recent years. This article describes a theorybased evaluation approach used with ENLACE of Hillsborough, a 5-year project funded by the W.K. Kellogg Foundation for the purpose of increasing Latino student graduation from high school and college. Theory-based evaluation guided planning, implementation as well as evaluation through the process of developing consensus on the Latino population of focus, adoption of culturally appropriate principles and values to guide the project, and identification of strategies to reach, engage, and impact outcomes for Latino students and their families. The approach included interactive development of logic models that focused the scope of interventions and guided evaluation designs for addressing three stages of the initiative. Challenges and opportunities created by the approach are discussed, as well as ways in which the initiative impacted Latino students and collaborating educational institutions. r 2007 Elsevier Ltd. All rights reserved. Keywords: Theory-based evaluation; Logic models; Theories of change; Latino education; Interactive evaluation

1. Introduction Latino student access to higher education has received significant national attention for more than 20 years (President’s Advisory Commission on Educational Excellence for Hispanic Americans, 1996, 2003; The Toma´s Rivera Policy Institute & National Hispanic Caucus of State Legislators, 2003). While numerous reports have been issued at the national level, the challenge remains to identify strategies that will be effective in working with a heterogeneous group of individuals whose needs vary by nation of origin, level of acculturation, and geographic location in the US (Hernandez & Nesman, 2004; Nieto, 2000; Torres-Guzman, & Martinez Thorne, 2000). This challenge was addressed by a nation-wide initiative known as ENLACE: Engaging Latino Communities for EducaCorresponding author. Tel.: +813 974 7417; fax: +813 974 7563.

E-mail addresses: [email protected] (T.M. Nesman), [email protected] (C. Batsche), [email protected] (M. Hernandez). 0149-7189/$ - see front matter r 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.evalprogplan.2007.04.004

tion sponsored by the W.K. Kellogg Foundation. Through ENLACE, 13 sites in 7 states were charged with forming community partnerships to develop, implement and evaluate programs that would increase Latino student access to higher education. As a condition of funding, each site was required to devote a minimum of 8–10% of the budget to program evaluation. This requirement had the positive effect of actively engaging the evaluation team with the proposal writers in the early conceptual stages of the project and produced an evaluation design that was responsive to implementer information needs (Patton, 1997). The ENLACE initiative funded in Hillsborough County, Florida, (hereafter referred to as ENLACE-HC) adopted a theorybased evaluation approach (Chen, 1990; Patton, 1997) that articulated the local theory of change and used logic models to guide the planning, implementation, and evaluation of a 5-year program (Hernandez & Hodges, 2003; W.K. Kellogg Foundation, 2000). Although much has been written about theory-based evaluation, few practical examples have been published to

ARTICLE IN PRESS 268

T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

share the knowledge gained and the lessons learned (Donaldson & Gooler, 2003). Such case studies can be helpful in exhibiting the impact of theory-based evaluation on planning and implementation of community-based initiatives that involved partnerships with multiple stakeholders. The ENLACE-HC case study described in this paper provides an example of the usefulness of theorybased evaluation in guiding a comprehensive initiative targeting Latino students and families, and the educational systems that serve them. 2. Background ENLACE-HC was based on a cultural–ecological approach for working across individual, family, institutional, and community levels (Garbarino, 1992; Szapocznik & Kurtines, 1993). The underlying assumptions were that ENLACE-HC needed to utilize culturally competent practices that build on Latino cultural strengths, and identify and incorporate community resources in ways that promote sustainability (Cross, Bazron, Dennis, & Isaacs, 1989; Gutierrez-Mayka & Contreras-Neira, 1998; Hernandez & Isaacs, 1998; Kretzman & McKnight, 1993). Cultural competence was defined as behaviors, attitudes, and policies within a system that enable people to work effectively in cross-cultural situations (Cross et al., 1989). Based on these beliefs, ENLACE-HC implemented a comprehensive community initiative with school-based interventions and prevention strategies, public education activities, and collaborative efforts across multiple sectors of the community. A theory-based evaluation approach was chosen because of its efficacy with comprehensive community initiatives (CCI) (Connell, Kubisch, Schorr, & Weiss, 1995). CCI’s attempt to create social change that is rooted in the community, carried out in a participatory manner, and informed by ongoing research and experience (Connell et al., 1995). ENLACE-HC was designed as a CCI that would bring together educational institutions and other stakeholders in a community-wide effort to increase the educational success of Latino students. Similar community-based approaches have been found to be effective in identifying needs, strengths, and solutions for Latino communities (Batsche, Hernandez, & Montenegro, 1999; Gutierrez-Mayka & Contreras-Neira, 1998; Kretzman & McKnight, 1993; Lucas, 1997; Nieto, 2002). Theory-based evaluation is helpful in surfacing the underlying beliefs and assumptions stakeholders have about strategies that will lead to desired changes (Hernandez, Hodges, & Worthington, 2000; Weiss, 1995). A theory-based approach is useful for linking evaluation to the factors implementers want or need to know to implement a comprehensive initiative. Data collection can also be linked to the beliefs implementers have about the effectiveness of strategies articulated in the theory of change (Chen, 1990; Donaldson & Gooler, 2003; Julian, Jones, & Deyo, 1995; Lipsey & Pollard, 1989; Patton, 1980;

Shadish, 1987; Weiss, 1995). The approach can also provide direction to the evaluation team for collecting data that will best track key components of the initiative (Donaldson & Gooler, 2003; Hernandez & Hodges, 2001; Julian, 1997), report outcomes in the context of environments in which they are produced, and synthesize findings across multiple program levels (Carvalho & White, 2004; Hernandez & Hodges, 2003; Yang, Shen, Cao, & Warfield, 2004). Logic models were chosen as a means for articulating the ENLACE-HC theory of change (Connell, Aber, & Walker, 1995; Julian et al., 1995; W.K. Kellogg Foundation, 2000). Logic models can portray linkages between issues to be addressed by an initiative, the activities to be implemented, the expected outcomes of such efforts, and the desired ultimate impact (Julian et al., 1995; Linney & Wandersman, 1991; W.K. Kellogg Foundation, 2000). Logic models can also serve as a roadmap for implementers to move from ideas to action by putting components together into a visual framework (Hernandez & Hodges, 2003). Theory-based logic models have been particularly useful for coordinating data collection in comprehensive systems of care where evaluation information is used to inform service delivery decisions (Hernandez & Hodges, 2001). The process of creating logic models in comprehensive community initiatives has contributed to the development of collaborative relationships among community participants by helping them build consensus on goals and strategies and serving as a focal point for reflection over the life of the project (Connell, Aber, & Walker, 1995). The following case example describes the theory-based planning, implementation, and evaluation process used for ENLACE-HC. 3. ENLACE: A case example ENLACE was initiated in 1998 when the W.K. Kellogg Foundation issued a request for proposals from Hispanic Serving Institutions and other higher education institutions serving large populations of Latino students. These institutions were considered to be core resources for stimulating community efforts to strengthen educational opportunities for Latino youth (W.K. Kellogg Foundation, 1998). The University of South Florida was awarded a 1-year planning grant in 1999 to address the educational needs of Latino students and families in Hillsborough County. The planning process targeted the entire county, which included the city of Tampa and surrounding municipalities. In 1999, the overall population of Hillsborough County was 939,000, with 18% classified as Latino (US Census Bureau, 1999). Tampa has been a home for a large number of immigrants from various Latin American, Caribbean, and European countries since the early 1900s. As part of the planning activities, an ENLACE Coalition was formed with four core partners: the University of South Florida (USF), Hillsborough Community College (HCC), the School District of Hillsborough County

ARTICLE IN PRESS T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

(SDHC), and the Hispanic Services Council (HSC). The ENLACE Coalition established a goal of initiating a ‘‘movement’’ that would increase community capacity to serve Latino students through the joint definition of problems and barriers, the development of shared agendas and goals, the leveraging of funds, and the adoption of definitions of success that were meaningful to all partners (ENLACE, 2000). The planning process culminated in the development of a strategic plan to implement ENLACEHC over a 4-year period from 2000 to 2004. 3.1. Program planning and theory development During the planning year, the ENLACE Coalition formed a Concilio (council) that was charged with the responsibility of developing the strategic plan that would guide the 4-year implementation phase. The Concilio included representatives from local educational institutions, the public and private sectors, parents, and students from high school, community college, and university levels. The main role of the Concilio was to review the information collected through a needs assessment process and ensure that this information was reflected in the ENLACE-HC implementation proposal. In the early stages of planning, Concilio members reviewed information obtained from recently conducted needs assessment activities such as a study of Latino student dropout (Latino Coalition, 1999) and a study of educational barriers for Latino students who migrate for farmwork (Martinez & Cranston-Gingras, 1996). For example, the Concilio learned that Latino students comprised 18% of the school population, but had disproportionately high rates of dropout (20%), truancy (28%) and non-promotion (25%) from one grade to another (School District of Hillsborough County, 1999). Latino dropout percentages were highest in eighth (25%), ninth (21%), and tenth (21%) grades (School District of Hillsborough County, 1999). The ENLACE Concilio also conducted focus groups with students, parents, school personnel, and community members to further understand the needs of the Latino population in the county. The following areas emerged as the most critical needs: lack of access to financial aid; limited transition support services for students at the precollege and college levels; lack of awareness/appreciation of Latino cultures among educators and service providers; few Latino role models for college attendance and professional careers; limited parental involvement in and understanding of college preparation, college entrance, and financial aid processes. In addition, focus group participants emphasized the need for increased communication among students, families, and service providers about educational processes, and increased cultural and linguistic competence in these efforts. In the last 5 weeks of the planning phase, the typically productive Concilio began to lose focus. Each participant came to the table wanting to serve a different population

269

and proposed a different set of strategies to meet the needs of these populations. Efforts to establish priorities for the proposal often became bogged down with discussions about the variety of needs that had been identified for Latino students and families. Each issue was considered to be so critical to the educational success of students that it was difficult for Concilio members to coalesce around a focused scope of work. The decision-making process was also affected by various views on what ‘‘Kellogg wants’’ in the proposal versus what ‘‘we need’’ in Hillsborough County (it was eventually clarified that Kellogg ‘‘wanted’’ whatever the community wanted as long as it led to the intended outcomes in a manner that strengthened the community partnerships). The Concilio was close to impasse when one of the evaluators went to the chalkboard and began to sketch out a logic model based on the discussion, needs assessment findings, and objectives of the funding agency. As the visual representation emerged, the dialogue became more focused, commonalities and linkages were discovered, and the strategic plan began to take shape. The resulting logic model clarified the ENLACE-HC theory of change in a way that all stakeholders could see how their ideas fit with the larger picture and each was able to commit to its implementation. 3.2. The ENLACE-HC theory of change The ENLACE-HC logic model (Fig. 1) developed by the Concilio portrayed the components of the theory of change by linking together the issues to be addressed, implementation strategies, guiding principles, and desired outcomes (Hernandez, 2000; Linney & Wandersman, 1991; W.K. Kellogg Foundation, 2000). An underlying theme was the focus on community strengths through varying intensities of interventions within ecological levels of individual, family, institution, and community and the development of a culture of opportunity for Latino students (Calderon, 1998; Garbarino, 1992; Kretzman & McKnight, 1993). Each component is summarized in the following sections. 3.2.1. Context The primary target population was first-generation in college Latino students at the middle school, high school and college levels. The first column of the logic model delineated the issues that would be addressed based on the outcomes of the needs assessment activities. Issues to be addressed for Latino students and families were the demystification of college processes, the limited number of Latinos in leadership roles, and the limited collaboration among local institutional partners when dealing with Latino populations. Additional issues included the high drop-out rate and low graduation rate of Latino students, cultural and linguistic biases in standardized testing, and the perception that Latino students would face cultural isolation on college campuses. Financial aid and

ARTICLE IN PRESS 270

T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

Fig. 1. The ENLACE of Hillsborough logic model illustrates each component of the theory of change developed by the ENLACE coalition during the planning year.

scholarship acquisition were among the most critical issues to be addressed. 3.2.2. Guiding principles The Concilio agreed on a set of guiding principles which emerged from the underlying theory of embedded ecological processes, community strengths, and cultural competence. The primary guiding principle of ENLACE-HC was that all activities and programs were to be culturally competent. Secondary principles were that all strategies were to be family focused, linked to the community, and based on the strengths and needs of students. In addition, activities were to be developed collaboratively and services were to be provided in a holistic manner that matched Latino student needs and assets. 3.2.3. Implementation strategies The ENLACE-HC logic model portrayed the overall strategy of building on formal partnerships among the Coalition members and informal partnerships established through the ENLACE Concilio consisting of community advisory groups, families, students, business, and community organizations. Together these partnerships were expected to increase school and university engagement with the community and create sustainable systemic change. Specific implementation activities for students were aimed at critical transition points in the educational pipeline (Lucas, 1997; Roderick, 1993), i.e., middle school

to high school, high school to college, 2–4-year college, and undergraduate to graduate education. 3.2.4. Outcomes The third component of the logic model listed the longterm outcomes the strategies were expected to have with the primary target population. The outcomes chosen by ENLACE-HC included (1) increasing the number of Latino students transitioning through the educational pipeline and (2) increasing college enrollment and graduation of Latino students. 3.2.5. Evaluation The role of evaluation was shown in the logic model as a two-way arrow to illustrate its function as one of linking issues, strategies and outcomes through data collection and feedback to implementers. The evaluation team was charged with monitoring activities to determine if they were effective at addressing the needs listed in the logic model, utilized the guiding principles in approaches and methods, and building on existing partnerships in a way that engaged the community and created systemic change. 3.2.6. Theory of change Based on the discussion for creating the logic model, the Concilio reached consensus on the theory of change under which ENLACE-HC would operate. The ENLACE-HC theory of change was articulated in the following vision statement:

ARTICLE IN PRESS T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

The vision of ENLACE is to enhance the academic success of Latino students and to enable them to achieve their full potential as role models, mentors, and community leaders. This vision will be achieved by building a strong alliance within the community to develop, enhance, and expand educational programs and transition services leading to post-secondary and graduate education; by promoting sustainable advocacy and support through community outreach, leadership development, and systemic change; and by developing a culture of opportunity for Latinos. The logic model portrayed key components of the vision and linked them to ENLACE-HC strategies in a way that portrayed the theory of change. The agreed upon theory of change for ENLACE-HC was that identified issues of concern would best be addressed by a culturally competent approach based on partnerships among educational institutions and community partners. These partnerships were expected to provide leadership in engaging the community, families, institutional administration, and faculty in supporting Latino students at critical transition points so that they progressed successfully through the educational pipeline and were able to obtain a higher education. 4. ENLACE-HC interventions Having articulated the ENLACE-HC vision and theory of change, the Concilio was able to focus its attention on the identification of interventions needed for each of the target populations. A variety of programs were developed in four main strategic areas: community and family outreach,

271

leadership development, policy analysis/advocacy and critical transition points (see Fig. 2). Community outreach activities included various forms of media communications with local television, radio, and newspapers that communicated college access information in the Spanish language. Family Support Specialists were hired by the Hispanic Services Council to address family issues that affected the educational progress of Latino students. Leadership development activities included the establishment of student Concilios at the local colleges and student/parent leadership training. Policy analysis and advocacy activities focused on two key educational policies affecting college admission and scholarship eligibility of minority students as well as several issues related to undocumented students. Critical transition points in the educational pipeline emerged as the primary focal points for intervention. An inventory of existing programs and services was developed and gaps in services were identified. An initial assumption of the Concilio was that several new interventions would need to be developed. However, the inventory showed that each institution already offered a large number of programs and services that Latinos were not using. As a result of this analysis, services were designed that would link students with existing services as they transitioned from one point to the next, and new programs were to be developed only when there was a gap.

5. ENLACE-HC theory-based evaluation approach Using the ENLACE-HC logic model as a guide, the evaluation employed an embedded case study design, that

Fig. 2. ENLACE implementation strategies are linked to specific focus areas and programs.

ARTICLE IN PRESS 272

T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

is, an overall case study of the county and sub-cases for each partner organization (Yin, 1984, 1992, 2003). The case studies used mixed methods, both qualitative and quantitative, and followed a developmental approach that adapted to the needs of the implementers as the initiative evolved over time (Patton, 1997; Yin, 1992). The evaluation team interacted directly with stakeholders throughout the initiative in order to enhance the responsiveness to information needs and ensure that evaluation methods were conducted in a culturally competent manner. Case studies were designed to test key components of the theory of change, assess attainment of intermediate outcomes, and test relationships between processes and outcomes (Yin, 2003). The overall case study included multiple sub-cases (Yin, 2003) but the main unit of analysis was Latino students within Hillsborough County. Embedded case studies were conducted with the school district (7 middle schools and 3 high schools), the community college (3 campuses), the university (1 campus), and the community-based organization. Similar protocols were used for each sub-case study, and data collected were combined to create the overall case study. The ultimate goal of this process was to find common and unique trends across cases as well as test the overall theory of change. The case study design sought to include data that addressed process, impact, and sustainability. To assess process, instruments were developed to obtain program descriptions and attendance counts from each partner on a quarterly basis. In addition, participant observation at meetings of the ENLACE-HC management and implementation teams, community advisory groups, and student leadership organizations provided qualitative information about community and family engagement, adherence to guiding principles, accomplishments, and challenges. The evaluation activities were framed by key questions based on three main goals established by the funding agency. The key questions associated with each goal were: (1) Process: Are the interventions being implemented as originally designed (i.e. quantity and quality by type of strategy)? Do the intended participants accept the interventions (i.e. participation levels and satisfaction by category)? (2) Impact: Is the number of Latino students successfully transitioning through the K-12 educational pipeline increasing (i.e. taking college preparatory classes, obtaining grades needed for college acceptance, graduating)? Is the number of Latino students entering and completing higher education increasing? (college acceptance, enrollment, retention) (3) Sustainability: To what extent are students, families, and community members involved in implementing ENLACE (types of involvement, roles played in the initiative)? Are the partners building a relationship by moving cohesively from cooperation to collaboration (communication, decision-making, shared resources)?

What outside resources are the partners bringing to and committing to ENLACE activities (leveraged resources/programs?) To what extent are ENLACE efforts institutionalized within the core partner institutions (new positions created, increased funding, attention to issues raised)? Evaluation instruments were developed with input from implementers and were directly related to strategies and programs/activities articulated in the logic model. A variety of evaluation instruments were used to provide feedback on process, impact, and sustainability of the initiative in each strategy area (see Table 1). For example, focus groups were conducted with students and interviews were conducted with program implementers annually. The impact on students was monitored by collecting aggregated student data from institutional management information systems. Surveys were also developed to assess the impact of key activities on students and a matched pair analysis of student outcomes was developed to assess progress toward the ultimate outcomes of college graduation (See Table 1). Each data source provided information across components of the theory of change and was used to assess progress in the targeted areas of community engagement, systemic change, partnership development and sustainability, and critical transitions of students. Data collection, analysis, and reporting were also organized according to the logic model and addressed process, impact and sustainability research questions. 5.1. Using the theory of change to assess process An ENLACE-HC implementation team included coordinators from each partner institution and a project director. A management team was also formed with high level administrators from each partner institution, individuals who were in a position to communicate directly with the CEO’s of each organization and facilitate needed system changes. The evaluators were included as team members for both the implementation and management teams. Monthly meetings were held by each team in order to address issues, review progress reports and evaluation findings, and make adaptations as needed. These meetings were highly interactive and became an important forum for discussing how the ENLACE theory of change was being played out in the real world of implementation. The original ENLACE-HC strategic plan included 26 different interventions to be conducted throughout a 4-year implementation period. It soon became evident that the plan had been overly ambitious and that it would not be possible to accomplish this large number of interventions with the available resources. Hence the management team, with the permission of the funding agency, decided to narrow the span of interventions and focus on those that had the potential to have the greatest impact. This was a difficult process because it meant that some activities had to be eliminated or reduced in scope. When faced with this

ARTICLE IN PRESS T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

273

Table 1 Measures and instruments were chosen to match each goal of the evaluation Evaluation instruments by goal and strategy area Evaluation goal

Community and family outreach

Leadership development

Policy/advocacy

Critical transition points

Impact

Follow-up phone contact surveys with families served by Hispanic Services Council. Document services received and satisfaction

Mentor/tutor journals document level of involvement, challenges, and lessons learned

Documentation of adoption/application of ENLACE priorities by educational institutions, community organizations and the state legislature

Process

Activity reports: Collected from each program coordinator, includes the major strategy area addressed, the purpose for the activity, the number and type of participants and the outcomes of the activity Participant observation: Documented through structured note-taking during partnership meetings and other events. Described implementation in comparison to strategies and target population in theory of change Benchmarking: Developed annual benchmarks (goals) with each program coordinator and compared actual accomplishments against these goals to inform future planning and adaptation of conceptual framework Focus groups: Held with students at all levels. Provided feedback on adoption of ENLACE goals and values, effectiveness, and level of implementation of strategies

Sustainability

Partner reports: Provided documentation of grants awarded for sustaining existing programs, programs incorporated into educational institutions as modifications or enhancements to existing structure, new partners and/or adoption of programs by partner institutions Benchmarking: Development of annual goals for partnership development/community engagement process with input from implementation and management team Participant observation: Documented through structured note taking the level of partnership development and community involvement at ENLACE meetings

Student surveys document involvement level, satisfaction, course of study, grades, diploma, and college plans Follow-up surveys document initial and continued enrollment in college, financial aid received, and challenges Comparative analysis of ALAS senior outcomes (school district database analysis). Provides matched comparison for statistical significance of ENLACE student level of courses taken, grades, test scores, college application, college admission and financial aid received Annual MIS data on Latino enrollment, retention and degree attainment at USF and HCC documents trends from baseline to final year Participant observation: Documented through structured note-taking during ENLACE meetings and activities. Provided evidence of changes in students/parents/staff/institutions in relation to the ENLACE theory of change (culture of opportunity is established and experienced by students and parents; students are developing leadership, academic and college knowledge skills; faculty/staff/ administrators are paying attention to Latino student issues and progress, etc.)

challenge, the logic model and evaluation data were helpful to the decision-making process. Reviewing data within the framework of the overall theory of change assisted the management team in identifying interventions that could be dropped without compromising the original intent of the initiative and in identifying programs or activities that were critical to sustain. This process helped all of the constituents come to understand that some interventions, while helpful and useful, had limited potential to produce the intended outcomes. As a result, the original 26 interventions were pared to 9 priority interventions, 10 being sustained at a more modest level, and 7 dropped in their entirety. For example, the initial plan called for an intervention to provide GRE preparation classes for college students to facilitate their application to graduate school. The relatively low participation rate and high cost

of the program resulted in the decision to discontinue this program even though the University ENLACE constituencies had found this to be a valuable service for the students who participated. The decision to continue a program was determined based on input obtained from satisfaction surveys, focus groups with students, participant observations, and monitoring program attendance. Increasing numbers of participants were indicators of a growing awareness and acceptance of specific services as well as cultural competence and proficiency in implementation. Programs with low participation rates were examined for implementation challenges and compared with participant satisfaction data. For example, the findings from focus groups provided strong support for the importance of bilingual/ bicultural programs for Latino middle and high school

ARTICLE IN PRESS 274

T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

students and their parents, especially for those students who would be the first in their family to attend college. The financial aid workshops conducted in the Spanish language at participating high schools were among the most frequently cited critical services. These programs had much higher rates of attendance by Latino families than the traditional English-only college information sessions offered by the school district. Participant observers attending the workshops noted that the sessions were conducted in a highly engaging manner by bilingual/bicultural guidance counselors who provided materials and explanations in both Spanish and English. Step by step instructions were given for completing financial aid application materials and interactive discussions emphasized parent and student roles in the process. The process evaluation also included a survey of high school seniors that assessed the level of participation of seniors in each type of activity. Findings confirmed the ENLACE-HC theory of change emphasis on critical transition point interventions and partnerships between educational institutions and the community, and also supported decisions about program continuation. For example, the strategies that seniors participated in the most and found to be most helpful included:

      

Provision of college information to help prepare for college; Field trips to see college campuses; Financial aid/scholarship application assistance; ENLACE personnel at school who were available when needed; Increased awareness of opportunities for Hispanics and exposure to Hispanic community leaders and/or achievers; Step-by-step guidance and encouragement; Mentors/tutors to help with the whole college process and calm fears about the transition to college.

Each of these strategies had been incorporated across multiple programs and services offered throughout the implementation process and were carried out in close collaboration between ENLACE-HC school liaisons and other ENLACE partners. The strong support for this approach led the evaluation team to focus on these strategies in the impact evaluation of the initiative. 5.2. Using the theory of change to assess impact In the logic model, the ultimate impact of ENLACE-HC was to be measured by the number of Latino students who successfully transitioned through the educational pipeline and entered and completed college degrees. From the outset, it was recognized that it would not be possible to attain these goals within the 4-year implementation period of the grant funding. For example, middle school students would not have enough time to complete high school in the 4-year time frame nor would those who transitioned from

high school to college have time to complete their degrees. However, the establishment of these long-term outcomes was an important aspect of the logic model in that the stated outcomes helped the management and implementation teams maintain focus and stay on target. Even though it would be several years before the longterm outcomes could be effectively measured, the evaluators developed a tracking system to monitor changes in student outcomes each year. Data were obtained from the school district’s management information system for students at high schools and middle schools that were targeted for the ENLACE interventions. The results of the data analysis indicated an increase in Latino students in the twelfth grade from 5.7% (n ¼ 290) in the first year to 7.6% (n ¼ 379) at the conclusion of ENLACE-HC. Data from the community college indicated that the total number of Latino students increased but that the percentage of Latino students remained about the same (17%) throughout the initiative because of an overall increase in student enrollment at the community college. At the local university, the Latino student population increased from 9.4% (n ¼ 2567) to 10.6% (n ¼ 3958) of the undergraduate population during the time of the ENLACE-HC program. However, given the multiple factors that could contribute to enrollment profile changes, it was not possible to attribute the increased enrollments at the university solely to the efforts of the ENLACE-HC initiative. For example, enrollment management data indicated that a larger number of Latino students were applying to the university (suggesting a positive impact of ENLACE-HC), but there was not a proportional increase in the number of Latino students being accepted and enrolling at the university. This phenomenon was attributed to the university raising its standards and requiring a higher SAT score for admission. However, information obtained from the qualitative assessment of ENLACE-HC suggested that many Latino students had not taken the SAT exam because they felt college was not a realistic possibility for them. As a result, these students were not eligible for college admission. When the data from the MIS tracking system was compared with the theory of change, the ENLACE management team made several strategy adaptations. For example, the Principal Investigator for ENLACE worked with the University Admissions Office to obtain a change in the admissions to include bilingualism as a special talent to be considered as part of the admission profile. Another change that was made was based on the realization that satisfactory performance on the SAT was a pre-requisite not only to college admissions but also to scholarship eligibility, which was crucial for college attendance of most ENLACE students. As a result, ENLACE school liaisons provided ongoing support and encouragement for Latino students to take the SAT preparation courses that were offered at no charge at their schools, and followed up with them to make sure they completed the course and took the exam. In their focus group and survey responses, students

ARTICLE IN PRESS T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

described this preparation and support as a key factor in their registration and completion of the exam. Assessment of direct impact of ENLACE-HC was challenging due to the difficulty of controlling for all of the extraneous variables. As a result, the decision was made to conduct a quasi-experimental study of a targeted group of students who would complete high school by the time ENLACE funding ended. A logic model was developed to portray the theory of change for this strategy, which targeted Latino students at each participating high school who fit into a ‘‘college-maybe’’ category. College maybe students were defined as college-capable juniors and seniors who needed additional support to navigate the transition to college and met specific criteria for grades, test scores, and commitment to participate in ENLACE-HC activities (Fig. 3). The logic model identified key strategies to be implemented including bilingual/bicultural programs to raise awareness of college and career opportunities, linkages with the community college and university, social supports for Latino families, and instructional/academic supports for students. The theory of change for this strategy was that the availability of these programs would lead to increased participation of Latino students in college preparatory activities, which would result in improved likelihood of acceptance into college. With ongoing collaboration between the school district and ENLACE-HC, the evaluators, the Principal Investi-

275

gator, and a school district evaluator designed a matchedpair study to compare ENLACE and non-ENLACE high school students over a 3-year period. The methodology and results of this study are reported elsewhere (Batsche, Nesman, Hernandez, & Watts, under review). Key findings of this analysis indicated that the interventions were highly successful for high school students transitioning to undergraduate school. Specifically, ENLACE students had significantly higher grade point averages, enrolled in more honors and advanced placement courses, and received more college scholarships than non-ENLACE students. These findings were interpreted as supporting the theory of change for this critical transition point as well as supporting the overall theory of change, which had hypothesized increases in the number of Latino students transitioning through the educational pipeline (Nesman, Callejas, Garcia, & Hernandez, 2005). 5.3. Using the theory of change to assess sustainability A major requirement of the funder was to design interventions that would be sustainable. Sustainability referred to the ability of the program to continue successful practices once the W.K. Kellogg Foundation funding ended. Sustainability was to be demonstrated through institutional commitments, ability to leverage new funds, and/or the continuation of community partnerships. In the ENLACE-HC logic model, sustainability was reflected as a

Fig. 3. A focused logic model developed during year 3 that shows strategies addressing the critical transition from high school to college.

ARTICLE IN PRESS T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

276

highest potential for sustainability and documenting accomplishments and related outcomes. To inform this process, an evaluation instrument was developed to survey ENLACE-HC implementers about components that were thought to have the greatest impact on ENLACE outcomes. Results showed that highest priority was given to activities that provided role models and mentors for Latino K-12 students, academic support services for Latino students at all levels, community advocacy activities, and activities that furthered the ENLACE-HC vision and goals. Written comments about key accomplishments mirrored the original ENLACE-HC theory of change, including the impact of partnerships, accomplishment of institutional systemic change, development of a collegegoing tradition among Latino students, and appreciation of Latino culture within educational institutions. The importance of attention to certain aspects of the theory of change was also noted, including the need to adapt programs to engage more Latino students, maintaining a focus on critical transition points, addressing emergent barriers to pursuing higher education, and continuing outreach to families and communities. Based on this feedback and additional planning discussions, the evaluation used the sustainability theory of

component of community partnerships and systemic change. However, during the last year of the grant the management team decided that a focused logic model was needed to address the sustainability component in more detail. The theory of change portrayed in this logic model was articulated as a ‘‘theory of sustainability’’ (Weiss, Coffman, & Bohan-Baker, 2002), which showed what the implementers believed was needed to sustain the initiative and pointed to desired outcomes that could be evaluated (Fig. 4). The sustainability approach focused not only on continued funding of programs but also the continuation of the ENLACE-HC values and vision, maintenance of collaborative relationships, and continued commitment to key outcomes. The sustainability theory of change linked the continuation of collaborative relationships to the development of needed financial supports, as well as continuation of key programs and strategies within each institution. It was theorized that specific strategies developed to maintain key programs and services would result in continued impact on student educational progress, systemic change, and partnership development. During the final phase of the initiative, the evaluation focused on identifying programs and services that had the

ENLACE: Year 4 and Beyond Strategy Guiding Principles

Outcomes Sustainability Components

•Constructive collaborations

•Perpetuating principles and values

•Complexity accepted •Work within constraints

•Maintaining relationships

•ENLACE endorsed

•Sustaining services & outcomes

•High ethical standards

•Meeting funding needs

•Inclusive/respectful

Strategies and Priority Programs K-12 Services •ALAS

College Level Services: El Centro

Community Outreach

•ENLACE Day •Financial aid, admission, and scholarship workshops •Counseling/Advisement •Hispanic Hometown Heroes

•Bilingual college outreach through media

•MOSI Scientist Program

•Transfer Consortium/Lumina grant

•Website

•Cultural Competence Training

•CPT testing

•Tutoring (DAR) •(P)SAT, (P)ACT Prep •Family Support Specialists •Proyecto Ingles

•College career programs

“Collaboration must continue”

•College information in Spanish, e.g. Boletines, Atrevete •Community Forums

•Café con Leche seminars

Advocacy and Leadership

Professional Development

•Student Concilio

•LIGA

•Parent leadership training/PACT

•ESCALERA

•Statewide policy advocacy

•HCC faculty participation in ENLACE Day

Partnership •Partnership meetings scheduled regularly •Institutions committed to continue collaborative efforts (new MOU) •Tracking plan developed for priority outcomes •Funding plan developed for priority programs Student/Family •Increased enrollment in AP/Honors •Increased grade point averages •Increased FCAT achievement levels •Increased enrollment in college prep courses of study •Improved attendance rates •Decreased dropout •Increased graduation •Increasing enrollment in Dual Enrollment •Increasing number of students applying to college “on-time” •Increasing number of students accepted into college •Increasing number of students enrolling in college •Increasing number of students applying for & receiving scholarships/financial aid and/or attending special programs •Increasing number of h.s. students graduating with regular diploma •Increasing number of Latino parents involved in schools Institutions •Increasing Latino enrollment & retention at HCC/USF •Incorporation of ENLACE services into existing infrastructure •Enhancement of institutional infrastructure for Latino communities

Fig. 4. A sustainability logic model developed during year 4 includes strategies for sustaining the initiative.

ARTICLE IN PRESS T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

change to focus evaluation activities on assessing the sustainability of priority programs, partnership relationships, and institutionalization of programs. Evaluators tracked and documented both the implementation of the sustainability theory of change and the attainment of outcomes for the final evaluation. Key outcomes reported for the area of partnership sustainability included ongoing funding for a collaborative project between the university and a local museum partner to continue an National Hispanic Scientist of the Year Award. The project, which was developed through ENLACE-HC, gained much community support for involvement of Latino middle school students in an annual field trip to the museum and the opportunity to meet and talk with an award-winning Latino scientist. Additional outcomes linked to partnership sustainability included the receipt of grants from the Lumina Foundation, the Children’s Board of Hillsborough County, the US Department of Education, the National Endowment for the Humanities, and the National Science Foundation. These grants were used to support priority areas identified in the sustainability plan for K-12 services, college level services, and professional development. Sustainability outcomes for students and families included the demonstration of positive impacts on student progress from high school to college through the matched pair analysis for ENLACE high school students. At the institutional level, outcomes included continuation of priority programs such as the school district’s implementation of the SAT prep course on a larger scale through a state-funded initiative. At the college level, sustainability outcomes included:

 



Creation of a position at both the university and community college that incorporated the functions of the ENLACE-HC coordinator position Incorporation of the ENLACE mentoring program into the Federal Work Study Program which provided funds for Latino college students to continue to participate in peer mentoring and tutoring with Latino high school students Continuation of funding to ENLACE-HC from the W.K. Kellogg Foundation to design and expand similar services to other community-educational partnerships throughout the state of Florida.

6. Interactive evaluation While ENLACE-HC was notably successful in implementing this comprehensive initiative, there were many challenges for both program implementation and evaluation that can inform future efforts. These challenges paralleled those reported for other comprehensive community initiatives (Hollister & Hill, 1995), suggesting their commonality across community contexts and target populations. When challenges arose, the theory-based

277

approach necessitated interaction between the implementers and evaluators, which impacted both programmatic and evaluation aspects. In some cases, the evaluation highlighted problems with the implementation strategies that needed to be addressed, which led to planning activities that resulted in adaptations. In other cases, the implementation strategies challenged the evaluators to identify new ways to assess program effectiveness. Because the implementers and evaluators had worked closely together on the theory of change from the beginning of the initiative, a natural interaction evolved that had an overall positive impact on ENLACE-HC even as challenges arose. For example, challenges were created by the horizontal and vertical complexity across and within organizations included in this initiative. ENLACE was working horizontally across four complex organizations, each serving different populations and each having different organizational structures, missions, legal responsibilities, and accountability mandates. ENLACE-HC responded to this challenge by developing a theory of change that was broad enough to be relevant to each partner and specific enough to implement and evaluate within the limits of the initiative. The embedded case studies used in the evaluation were helpful in linking the broad theory of change to efforts across institutions. More specific logic models were used to link intermediate outcomes to the ultimate outcomes in the overall logic model. In addition, the evaluation sought to provide information that would make sense to and be useful for all partners and would match the developmental level of ENLACE-HC throughout the 5 years of the initiative. To this end, an interactive approach emerged that included multiple channels for communicating evaluation findings such as discussions in monthly planning meetings, quarterly evaluation reports, and ongoing e-mail and phone communication with program implementers within each partner institution. Vertical complexity also created challenges within this initiative. Based on recommendations of project consultants (Intercultural Development Research Association, 2002), feedback from program implementers, and research on partnership development (Taylor-Powell, 1999; TaylorPowell, Rossing, & Geran, 1998), the evaluation team modified the process evaluation to assess implementation at the individual, group, institutional, systemic and community levels during the third year of the initiative. One result of this adaptation was the ability of the evaluation team to point out the need for increased interactions between management and implementation levels, suggesting the creation of an organizational mechanism to facilitate communication, decision-making and implementation for mid-course corrections. In response to this recommendation, several joint meetings were held with management and implementation teams during the last 2 years of the initiative. Contextual issues were another common challenge faced by the evaluation of ENLACE-HC. The political

ARTICLE IN PRESS 278

T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

environments of the three educational institutions as well as the cultural and economic barriers faced by the targeted population presented challenges to the implementation and sustainability of programs as well as the degree of impact of the initiative. Similar challenges were previously experienced when conducting the Latino Dropout Study (Latino Coalition, 1999) and the ENLACE-HC planning year needs assessment (ENLACE, 2000). However, new barriers arose as ENLACE sought to address systemic change and sustainability of efforts. For example, profound changes in Florida’s educational governance structure (House Bill 2263, State of Florida, 2000) occurred at the inception of ENLACE, affecting both the structure and priority activities of the major partners. In addition, programs such as the One Florida Initiative and Talented 20 program, which were aimed at ending race-conscious college admissions while increasing opportunity and diversity in the state initially, complicated the ability of universities to maintain past levels of minority enrollment (Marin & Lee, 2003). At the K-12 level, the state’s implementation of the No Child Left Behind Act (US Department of Education, 2005) included high-stakes testing through the Florida Comprehensive Assessment Test (FCAT) (Florida Department of Education, 2005). These requirements influenced curricular and budgetary decisions of local districts and called for several mid-program adjustments to accommodate the changes. As these program changes were made, the evaluation team worked with implementers to adapt evaluation instruments and ensure that evaluation findings were considered in conjunction with contextual characteristics. The flexible and evolving nature of the initiative also challenged the evaluation team in its efforts to gather and report meaningful data. From its inception, ENLACE-HC was expected to evolve over time in response to the community needs and institutional climates. This changing dynamic challenged the evaluation team to find ways to modify data collection to meet new approaches created by implementers while maintaining consistency in order to be able to compare data over time. Through ongoing interactions with implementers at service and management levels, the evaluation team was able to point out the need to maintain consistency in assessing the agreed upon theory of change and provide evaluation data in a manner consistent with the original proposal. However, the initial response of the implementers was to question whether the evaluation should drive the implementation or the reverse. This issue was resolved by reviewing the funder’s requirements for evaluation data and modifying some strategies to ensure needed data could be provided and making changes to the evaluation design whenever possible. This challenge spurred solicitation of periodic input on information needs from implementers in order to balance data collection with their changing needs. For example, one result of feedback was the modification of data collection for quarterly reports to include information about challenges and

evolving strategies while also gathering information that answered the research questions. The broad range of outcomes and differing levels of outcomes were also a challenge for the evaluation to document. The initiative was aimed at promoting change in several domains across several institutions and the community, complicating both definition and measurement. In addition, while the long-term outcomes remained constant and straightforward, the short-term outcomes that were sought changed as program emphases changed. As was previously described, logic models were helpful in creating specific implementation strategies, focusing on partnership development, sustainability, and the high school to college critical transition point. Each logic model showed the expected outcomes for that strategy, which could be linked to outcomes shown in the overall logic model. Alignment of outcomes between levels helped manage the complexity by showing outcomes that were expected to build toward the ultimate outcome of increased Latino student graduation from college. 7. Impact of the theory-based approach The impact of the theory-based approach on implementation was assessed through participant observation data throughout the initiative as well as through interviews and surveys with program coordinators. Throughout ENLACE-HC, program implementers provided positive feedback on the importance and usefulness of evaluation data that could be easily matched to components of the theory of change. In a survey conducted in the final year, the majority of respondents emphasized the usefulness of the logic model and theory of change in maintaining focus, guiding implementation decisions, identifying gaps, addressing conflicting goals and strategies, putting theory into practice, and showing the impact of their efforts. Sample responses describing the overall impact of the evaluation approach included:



  

‘‘I have learned there are people who share the same values and goals with respect to the Hispanic community. In addition, I have been able to pass that enthusiasm along to my coworkers and administrators where it has become a trickle down effect to help our Hispanic students.’’ ‘‘It kept me on track and it was good to see theory become practical via charts, graphs and summaries.’’ ‘‘[It] gave me overall perspective and motivation for coming year.’’ ‘‘It kept me on target, focused and encouraged that what we were doing was on target and accomplishing the goals.’’

ENLACE implementers described the theory of change approach not only as facilitating the process of implementation and evaluation, but also serving as a motivator and a tool for advocacy in pursuit of the goals of ENLACE.

ARTICLE IN PRESS T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

There was evidence in implementers’ responses that the theory of change remained clear and served as a guide across partners, levels of implementation, and time, and had an impact on the community that was in line with its vision and goals. However, a different level of confidence was expressed for the sustainability of strategies that worked directly with students at educational institutions in comparison to strategies that were collaborative and crossinstitutional in nature. It was felt that strategies within each institution were more likely to be sustained (e.g. mentoring, educational programs/services), whereas strategies relying on collaboration and cross-institutional coordination would not be sustained unless there was on-going external support. This served as additional motivation for the initiative to develop and implement a strategy for ensuring sustainability. 8. Lessons learned The ENLACE-HC case study reinforces the value of using theory-based evaluation for comprehensive community initiatives. Advantages included the usefulness of the approach in addressing complex issues and structures, promoting partner participation and communication, keeping all parties focused on key strategies, planning for implementation and evaluation, and identifying areas in need of improvement or redirection. The process was particularly well suited to planning and evaluation of an initiative to serve Latino communities because it provided clear linkages between issues faced by Latino students, and the culturally relevant values, principles, and strategies that could lead to desired outcomes. One of the key lessons learned was that in order to maximize the advantages of the theory-based approach, the ENLACE-HC evaluation team needed to take on the responsibility for training participants on logic model development and theory of change approaches early in the process of implementation. It was also found that followup training or technical assistance was needed as adaptations were made in the evaluation and the logic model. Although time consuming, training ultimately facilitated data collection and reporting and increased the buy-in of implementers. It was also a major contributor to the development of interactive and productive relationships between evaluators and implementers. Another key lesson learned was the need to recognize the amount of time and effort required to establish the infrastructure and bring programs to scale within such a complex and wide-reaching initiative. With support from the cross-site evaluation funded by the W.K. Kellogg Foundation, the ENLACE-HC evaluation team adopted a developmental approach that focused initially on start-up processes and partnership development, gradually incorporated impact and outcome evaluations as programs were established, and finally transitioned into a focus on sustainability. In addition, some aspects of the overall theory of change were never implemented in the manner

279

originally intended. For example, the original idea of community and family outreach, leadership development and advocacy was based on an equal partnership of students, parents and community members in decisionmaking and implementation of ENLACE-HC programs. However, it was found that these goals would require extensive investment of time and training for capacity building among some of the constituents and this was outside the scope of the project. As the evaluation provided feedback on this aspect of the theory of change, it was determined that leadership development and advocacy strategies should be incorporated into other activities that were being implemented by institutional partners. For example, leadership development was incorporated into the training for college students who served as mentors and tutors for high school students, and parent leadership development and advocacy was addressed by workshops offered by the Hispanic Services Council and through the school district’s bilingual college and scholarship information programs. For this community-based educational initiative, a theory-based evaluation approach was found to be wellsuited to developing and documenting the impact of strategies targeting Latino students. Other communities attempting to address Latino access to education are encouraged to consider a theory-based approach as a way to ensure attention to key cultural and community contexts and identifying appropriate and meaningful strategies and outcomes. This study provides a case example of advantages and challenges of such an approach for a specific community, as well as important outcomes of this ENLACE-HC initiative. Publication of additional case examples of comprehensive initiatives with Latino communities are recommended so that comparisons can be made across different local contexts and populations. Acknowledgments Funding for the ENLACE of Hillsborough project and its evaluation were provided by the W.K. Kellogg Foundation and the Children’s Board of Hillsborough County. We wish to acknowledge the hard work and collaborative spirit of the implementers of ENLACE-HC, including Maria Pinzo´n and Maggie Sanchez from the Hispanic Services Council; Donna Parrino, Norma CanoAlvarez, and John McKiernan-Gonzalez from USF; Carlos Soto, Salvador Torres, and Glorianna Felix from HCC-Brandon; Dan Valdez, Herman Fernandez, Rose Diaz, Jennifer Gonzales-Young, and Vivien Oliva from the School District of Hillsborough County; Martha Guzma´n from the Children’s Board of Hillsborough County; Family Support Specialists Sasha Classe, Ange´lica Ibarra, Adriana Kierecki, Lisandra Martı´ nez, Karla Polanco, and Yamil Vega; and School Liaisons Brigita Barobs-Gahr, Gregorio Burgos, Patricia Carranza, Sonia Castellana, Alice Eisenberg, Alma Gonzalez-Haskins, Maria Herna´ndez, Marcia Marrero, Doris Ripoll, Nilsa Ruiz, Veronica

ARTICLE IN PRESS 280

T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281

Schmidt, Marie Sternberg, Rose Tozzi, Gisela Triana, and Ansberto Vallejo. We would also like to acknowledge the invaluable contributions of evaluation team members Linda Callejas, Ivy Garcia, and Angela Gomez.

References Batsche, C., Hernandez, M., & Montenegro, M. C. (1999). Community needs assessment with Hispanic, Spanish-monolingual residents. Evaluation and Program Planning, 22, 13–20. Calderon, M. (1998). Adolescent sons and daughters of immigrants: How schools can respond. In K. M. Borman, & B. Schneider (Eds.), Adolescent years: Social influences and educational challenges. Ninety-seventh yearbook of the National Society for the Study of Education. Part 1 (pp. 65–87). Chicago: University of Chicago Press. Carvalho, S., & White, H. (2004). Theory-based evaluation: The case of social funds. American Journal of Evaluation, 25(2), 141–160. Chen, H. T. (1990). Theory-driven evaluations. Newbury Park, CA: Sage. Connell, J. P., Aber, J. L., & Walker, G. (1995). How do urban communities affect youth? Using social science research to inform the design and evaluation of comprehensive community initiatives. In J. P. Connell, A. C. Kubisch, L. B. Schorr, & C. H. Weiss (Eds.), New approaches to evaluating community initiatives: Concepts, methods and contexts (pp. 93–215). Washington, DC: The Aspen Institute, Roundtable on Comprehensive Community Initiatives for Children and Families. Connell, J. P., Kubisch, A. C., Schorr, L. B., & Weiss, C. H. (1995). New approaches to evaluating community initiatives: Concepts, methods and contexts. Washington, DC: The Aspen Institute, Roundtable on Comprehensive Community Initiatives for Children and Families. Cross, T. L., Bazron, B. J., Dennis, K. W., & Isaacs, M. R. (1989). Towards a culturally competent system of care, Vol. I. Washington, DC: National Technical Assistance Center for Children’s Mental Health, Georgetown University Child Development Center. Donaldson, S. I., & Gooler, L. E. (2003). Theory-driven evaluation in action: Lessons from a $20 million statewide work and health initiative. Evaluation and Program Planning, 26, 355–366. ENLACE. (2000). Proposal to the W.K. Kellogg Foundation. Tampa, FL: Office of the Provost, University of South Florida. Florida Department of Education. (2005). Florida comprehensive assessment test. Florida Department of Education. Available: /http:// www.firn.edu/doe/sas/fcat.htmS. Garbarino, J. (1992). Children and families in the social environment (2nd ed.). New York: Aldine de Gruyter. Gutierrez-Mayka, M., & Contreras-Neira, R. (1998). A culturally receptive approach to community participation in system reform. In M. Hernandez, & M. R. Isaacs (Eds.), Promoting cultural competence in children’s mental health services (pp. 133–148). Baltimore, MD: Paul H. Brookes. Hernandez, M. (2000). Using logic models and program theory to build outcome accountability. Education and Treatment of Children, 23(1), 24–40. Hernandez, M., & Hodges, S. (2001). Theory-based accountability. In M. Hernandez, & S. Hodges (Eds.), Developing outcome strategies in children’s mental health (pp. 21–40). Baltimore, MD: Paul H. Brookes. Hernandez, M., & Hodges, S. (2003). Crafting logic models for systems of care: Ideas into action. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute. Hernandez, M., Hodges, S., & Worthington, J. (2000). Turning ideas into action using theory-based frameworks. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, Department of Child and Family Studies. Hernandez, M., & Isaacs, M. R. (Eds.). (1998). Promoting cultural competence in children’s mental health services. Baltimore, MD: Paul H. Brookes Publishing.

Hernandez, M., & Nesman, T. (2004). Issues and strategies for studying Latino student dropout at the local level. Journal of Child and Family Studies, 13(4), 453–468. Hollister, R. G., & Hill, J. (1995). Problems in the evaluation of community-wide initiatives. In J. P. Connell, A. C. Kubisch, L. B. Schorr, & C. H. Weiss (Eds.), New approaches to evaluating community initiatives: Concepts, methods and contexts (pp. 127–172). Washington, DC: The Aspen Institute, Roundtable on Comprehensive Community Initiatives for Children and Families. Intercultural Development Research Association. (2002). Community engagement review and planning guide, Field Test Version-2002. San Antonio, TX: Intercultural Development Research Association. Julian, D. A. (1997). The utilization of the logic model as a system level planning and evaluation device. Evaluation and Program Planning, 20(3), 251–257. Julian, D. A., Jones, A., & Deyo, D. (1995). Open systems evaluation and the logic model: Program planning and evaluation tools. Evaluation and Program Planning, 18(4), 333–341. Kretzman, J. P., & McKnight, J. L. (1993). Building communities from the inside out: A path toward finding and mobilizing a community’s assets. Evanston, IL: Center for Urban Affairs and Policy Research, Neighborhood Innovations Network, Northwestern University. Latino Coalition. (1999). They are our kids: Findings of the Latino dropout study. Tampa, FL: Children’s Board of Hillsborough County. Linney, J., & Wandersman, A. (1991). Prevention plus III: Assessing alcohol and other drug prevention programs at the school and community level. Washington, DC: US Department of Health and Human Services. Lipsey, M. W., & Pollard, J. A. (1989). Driving toward theory in program evaluation: More models to choose from. Evaluation and Program Planning, 12(4), 317–328. Lucas, T. (1997). Into, through, and beyond secondary school: Critical transitions for immigrant youths. In J. Peyton, & D. Christian (Eds.), Topics in immigrant education series. McHenry, IL: Center for Applied Linguistics and Delta Systems, Inc. Marin, P., & Lee, E. K. (2003). Appearance and reality in the sunshine state: The talented 20 program in Florida. Cambridge, MA: The Civil Rights Project, Harvard University. Martinez, Y. G., & Cranston-Gingras, A. (1996). Migrant farm worker students and the educational process: Barriers to high school completion. The High School Journal, 80(1), 28–38. Nesman, T., Callejas, L., Garcia, I., & Hernandez, M. (2005). ENLACE of Hillsborough 2004 annual evaluation report. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, Department of Child and Family Studies. Available: /http:// cfs.fmhi.usf.edu/tread/enlace/Files/EvalReport04.pdfS. Nieto, S. (Ed.). (2000). Puerto Rican students in US schools. Mahwah, NJ: Lawrence Erlbaum Associates. Nieto, S. (2002). Language, culture and teaching: Critical perspectives for a new century. Mahwah, NJ: Lawrence Erlbaum Associates. Patton, M. Q. (1980). Qualitative evaluation methods. Beverly Hills, CA: Sage Publications. Patton, M. Q. (1997). Utilization-focused evaluation: The new century text. Thousand Oaks, CA: Sage Publications. President’s Advisory Commission on Educational Excellence for Hispanic Americans. (1996). Our nation on the fault line: Hispanic American education. Washington, DC: White House Initiative on Educational Excellence for Hispanic Americans. President’s Advisory Commission on Educational Excellence for Hispanic Americans. (2003). From risk to opportunity: Fulfilling the educational needs of Hispanic Americans. Washington, DC: White House Initiative on Educational Excellence for Hispanic Americans. Roderick, M. R. (1993). The path to dropping out: Evidence for intervention. Westport, CT: Auburn House. School District of Hillsborough County. (1999). Secondary school leavers: School year 1997–98. Tampa, FL: Division of Technical, Career and Adult Education.

ARTICLE IN PRESS T.M. Nesman et al. / Evaluation and Program Planning 30 (2007) 267–281 Shadish, W. R. (1987). Program micro- and macro-theories: A guide for social change. In L. Bickman (Ed.), Using program theory in evaluation. New directions for program evaluation, Vol. 33 (pp. 93–110). San Francisco, CA: Jossey-Bass. State of Florida. (2000). House Bill 2263 Education Governance Reorganization. Available: /http://www.myflorida.com/myflorida/ education/learn/egrt_taskforce/index.htmlS. Szapocznik, J., & Kurtines, W. M. (1993). Family psychology and cultural diversity: Opportunities for theory, research, and application. American Psychologist, 48, 400–407. Taylor-Powell, E. (1999). Evaluating collaboratives: Challenges and practice. The Evaluation Exchange, 5(2/3), 6–7. Taylor-Powell, E., Rossing, B., & Geran, J. (1998). Evaluating collaboratives: Reaching the potential. Madison, WI: Program Development and Evaluation, University of Wisconsin-Extension, Cooperative Extension. The Toma´s Rivera Policy Institute & National Hispanic Caucus of State Legislators. (2003). Closing achievement gaps: Improving educational outcomes for Hispanic children. Los Angeles, CA: TRPI. National Hispanic Caucus of State Legislators, Washington, DC. Torres-Guzman, M. E., & Martinez Thorne, Y. (2000). Puerto Rican/ Latino student voices: Stand and deliver. In S. Nieto (Ed.), Puerto Rican students in US schools (pp. 269–291). Mahwah, NJ: Lawrence Erlbaum Associates. US Census Bureau. (1999). State and county quick facts. /http:// quickfacts.census.gov/qfd/states/S. US Department of Education. (2005). NCLB Legislation, regulations and guidance. Office of Elementary and Secondary Education. Available: /http://www.ed.gov/about/offices/list/oese/legislation.html#legS.

281

Weiss, C. H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J. P. Connell, A. C. Kubisch, L. B. Schorr, & C. H. Weiss (Eds.), New approaches to evaluating community initiatives: Concepts, methods and contexts (pp. 65–92). Washington, DC: The Aspen Institute, Roundtable on Comprehensive Community Initiatives for Children and Families. Weiss, H., Coffman, J., & Bohan-Baker, M. (2002). Evaluation’s role in supporting initiative sustainability. Cambridge, MA: Harvard Family Research Project, Harvard University Graduate School of Education. W.K. Kellogg Foundation. (1998). Phase II: Project implementation request for proposals and proposal development guide. Battle Creek, MI: Author. W.K. Kellogg Foundation. (2000). Logic model development guide: Using logic models to bring together planning, evaluation and action. Battle Creek, MI: Author. Yang, H., Shen, J., Cao, H., & Warfield, C. (2004). Multilevel evaluation alignment: An explication of a four-step model. American Journal of Evaluation, 25(4), 493–507. Yin, R. K. (1984). Case study research: Design and methods. Applied social research methods series, Vol. 5. Beverly Hills, CA: Sage. Yin, R. K. (1992). The case study method as a tool for doing evaluation. Current Sociology, 40(1), 121–137. Yin, R. K. (2003). Case study research: Design and methods. Applied social research methods series (3rd ed., Vol. 5). Beverly Hills, CA: Sage.