The evaluation of a Complex Social Program: Lessons learned from the experience of the European Social Fund

Evaluation and Program Planning 29 (2006) 1–9 www.elsevier.com/locate/evalprogplan

The evaluation of a Complex Social Program: Lessons learned from the experience of the European Social Fund Cristina Lion *, Paola Martini Isfol, National European Social Fund Evaluation Unit, Via Lancisi 29, Rome 00161, Italy Received 11 October 2004; received in revised form 8 February 2005; accepted 27 June 2005

Abstract In 1999, the European Commission and the Italian Ministry of Welfare asked ISFOL’s National European Social Fund Evaluation Unit to undertake the evaluation of the Italian European Social Fund Objective 3 Program for the period 2000–2006. This program is financed by European resources in all Member States. In Italy it supports training and labor market policies in 14 Centre-North Regions. The mid-term evaluation, carried out at the end of 2003, faced some important challenges related mainly to: (1) the complexity of the program, implemented through continuing processes of decision and negotiation; (2) an institutional context of multi-level governance, linked to the decentralization process, which has increased the number of actors involved in the management of public policies and in their evaluation. This article describes the Evaluation Unit’s approach to the mid-term evaluation within the new multi-level governance context, describing the main methodological choices. The lessons learned will be discussed. q 2005 Elsevier Ltd. All rights reserved. Keywords: Meta-evaluation; Multi-level governance; European Social Fund; Education; Vocational training; Labor market

1. Introduction Within the framework of Structural Funds, the European Social Fund (ESF) is the main financial tool for implementing the European Union’s strategic employment policy. The ESF provides funding to all European Member States on a major scale for programs which develop or regenerate people’s ‘employability’. The focus is on providing citizens with appropriate work skills as well as developing their social interaction capacity, thereby improving their self-confidence and adaptability in the job marketplace. In the Italian centre-north regions, the European Social Fund amounts to approximately 11 billions US dollars for the 2000–2006 program. Within the framework of Objective 3 ESF is implemented through 14 Regional Operational Programs and one National Operational Program managed by the Ministry of Welfare. In 1999, the European Commission and the Italian Ministry of Welfare asked ISFOL’s1 National European Social Fund * Corresponding author. Tel.: C39 06 44 590 697; fax: C39 06 445 907 06. E-mail address: [email protected] (C. Lion). 1 Isfol is a public research institute in the field of education, vocational training and labor market policies, counseling the Ministry of Welfare. The Evaluation Unit was set-up in 1995.

0149-7189/$ - see front matter q 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.evalprogplan.2005.06.001

Evaluation Unit (referred to below as the Evaluation Unit) to undertake the 2000–2006 ESF mid-term evaluation (referred to below as the mid-term evaluation). In Italy, the first evaluation experiences in the field of training and labor market policies have been developed towards the end of the 1980s under the pressure of the European Structural Funds that made evaluation practice compulsory for programs financed by EU resources. The EU regulations stress the importance of evaluation during the policy life cycle as a mean to support the decision making process. Evaluations have thus been carried out on the basis of the indications and constraints defined in the European Commission methodological framework. The theoretical debate on evaluating social policies in Italy has been speeding up in the last decade (Bezzi, 2001; Palumbo, 2001; Stame, 1998). The evaluation culture is rather fresh and is fed by much more mature academic studies and research started in the United States in the 1960s (Patton, 1978; Scriven, 1967; Weiss, 1972). The mid-term evaluation started in 2000, when the methodological tools were prepared and the main evaluative research lines identified, and concluded in September 2003. The Italian experience of the ESF evaluation is described, retracing the most significant stages in the definition and drafting of the evaluation report, illustrating the main ‘lessons’ learned and the dissemination of results.

2

C. Lion, P. Martini / Evaluation and Program Planning 29 (2006) 1–9

This contribution is thus a meta-evaluative study, that is an evaluation of the evaluation, through an a posteriori rethinking of the work carried out. Since this kind of evaluation studies (Bustelo, 2003) is not often carried out, this self-evaluation exercise seems particularly significant in a perspective of learning and of improving the evaluation. The aim of the article is twofold. First, to describe a concrete experience such as the Italian one to evaluators and other actors interested in improving and rationalizing program and publicpolicy evaluation. Second, to study the relationship between political decision-makers and evaluators, and thus between programming and evaluation, from the evaluation demand stage to the political decision-makers’ utilization of the evaluation results. The article is divided into four parts. The first part describes the reference context of the mid-term evaluation, where new and broader areas of ESF intervention and new actors involved in program planning and evaluation are emerging. In the second part, the evaluative approach is reconstructed. In the third part, the main lessons learned are proposed and analyzed. The last part examines the relationship between policy-makers and evaluators, focusing on the utilization and dissemination of results.

system; the Constitutional law has instead changed the division of legislative jurisdiction between the State and local authorities. The new institutional scenario now has a horizontal instead of vertical organization, in which State, regions, provinces and municipalities are all on the same level. These subjects and other local authorities, to whom functions and tasks have been gradually transferred according to ‘subsidiarity’, differentiation and suitability criteria, have become increasingly involved in the definition of political strategies, shouldering leading roles in the implementation of the ESF interventions. The institutional context in which the ESF policies are programmed and implemented is thus no longer hierarchical, but increasingly spelt out in autonomous local areas with equal powers to represent their own needs. Since the State is no longer the sole arbiter for a policy and the tools to use, it is in the subnational area that the policy-makers involved in the decentralization of that policy are pinpointed and required to act. The programs financed by the ESF are thus implemented through continuous processes of decision, negotiation and mediation, where numerous actors with specific, and often diverging, values and priorities take the stage.

2. The mid-term evaluation reference context

Concurrently with the institutional changes in the national domain, in the Community area the reform of the Structural Funds for the 2000–2006 period has considerably extended the ESF’s aims and thus its spheres of intervention, in particular with reference to the previous 1994–1999 programming. The ESF objectives are laid down in the European Commission (EC) Regulation 1260/99 concerning the general provisions on the Structural Funds. It provides five key policy areas for the ESF:

Some specific features of the reference scenario of the midterm evaluation have influenced the approach to evaluation in terms of both issues and methodologies. These aspects pertain to the plurality of actors involved in the programming and implementation of policies, the variety of objectives pursued, the multiplicity and heterogeneity of actions (new intervention areas with regards to the 1994–1999 program), and stronger links between the ESF and other concurrent or complementary national policies. 2.1. The decentralization process in Italy The recent decentralization process in Italy has increased the number of actors designing and managing public policies. The regional organization of the Italian State, that is the recognition of the regions as local and autonomous authorities with their own powers and functions, was introduced by the Constitution in 1948. Over the last years there has been a gradual but continuous transfer of responsibilities from the centre to the periphery, designing a new model of governance with new responsibilities and new relationships (Lion, Martini, & Volpi, 2004). This model results from a series of provisions adopted in Italy between 1997 and 2001, in particular:

2.2. The new ESF program

† development of active labor market policies to combat and prevent unemployment, to avoid long-term unemployment, to facilitate the reintegration of the long-term unemployed and to support integration into the labor market of young people and persons returning to work after a period of absence; † promotion of equal opportunities for all in terms of access to the labor market, with particular attention to persons at risk of social exclusion; † promotion and improvement of vocational training, education and counseling in the context of a lifelong learning policy; † promotion of a skilled, well-trained and flexible workforce, innovative and adaptable forms of work organization, and entrepreneurship; † specific measures to improve access and active participation of women in the labor market (career prospects, access to new job opportunities, setting up businesses, etc.).

1. Law 59/97 called ‘Bassanini’ and legislative decree 112/98 on administrative decentralization; 2. The reform of Title V of the Constitution introduced with constitutional law 3/2001.

In general, three forms of assistance are eligible for ESF funding:

The first two legislative provisions regard the division of administrative powers among the various actors present in the

† assistance for individuals, which should represent the main form of aid, covering areas such as vocational training or

C. Lion, P. Martini / Evaluation and Program Planning 29 (2006) 1–9

education and careers guidance, etc.; † assistance for structures and systems to make support activities for individuals more effective (e.g. improving personal effectiveness); † accompanying measures (provision of services and equipment for the care of dependent persons, promotion of social skills training, and public awareness and information campaigns). Thus, in this programming period, the ESF is financing not only training policies but also other active labor, educational and social inclusion policies, becoming the financial and operative arm of the European Employment Strategy (EES). The broadening of the ESF’s sphere of intervention is also reflected in the financing of assistance for structures and systems, ranging from employment services to educational and training systems and their integration. Alongside this, the growing awareness on a EC level of the need to define a new model for participating in the labor market, accompanied by a redefinition of the social organization, is reflected on the ESF program. This latter has innovative elements regarding accompanying measures for strengthening the effectiveness of the financed activities. This is done, firstly, by removing social and structural conditions hindering labor market access and permanency and the achievement of higher career levels; and, secondly, by promoting assistance actions for recipients and their families and by reconciling working and private life. These actions are significantly new in the current ESF and cut across the program. 2.3. A new framework for evaluation The Structural Funds have given an important contribution to the development of an evaluation culture and practice in Italy. Over the last decade, there has been a growing demand to evaluate socio-economic programs financed by the European Union’s Structural Funds, supported by the MEANS (Methods for Evaluating Structural Policies) Program launched by the European Commission in 1995. The main aim of this program was to build evaluation capacity, to improve the quality of structural policy evaluations, to develop the usefulness of these evaluations and to enhance their credibility. The nature of European structural policy interventions— which are largely geographically bound and implemented within a partnership framework, often made up of numerous different projects—puts constraints in the way evaluators are able to use the evaluation tools and techniques available to them. The MEANS Program stresses the importance to elaborate an evaluation method on the basis of a mix-andmatch of various tools. In the new 2000–2006 program, the choices made on a EC and national level strengthened the role of the evaluation, considering it an integral part of the ESF policy-cycle management. The evaluation is aimed at reconstructing outputs, outcomes, impacts and intervention implementation processes and also for reprogramming the actions in itinere.

3

In particular, EC regulations decree that the mid-term evaluation must cover various aspects of the implementation of ESF programs. It must also analyze the first results of the interventions, their pertinence and the extent to which targets have been met, as well as the utilization of financial resources. On an organizational level, Italy has chosen to develop an evaluation system that reflects the decentralized character of the institutions, with as many evaluators as there are Regional Operational Programs and a National ESF Evaluation Unit coordinating the evaluation activities on a scientific and methodological basis. Furthermore, to streamline the work and for a better interaction between the various stakeholders, a Technical Evaluation Group was created, consisting of the main actors involved in evaluation issues on a European and national level. These include representatives of the EC, the Ministry of Welfare and other central administrations, regional institutions and social partners, each with their respective cognitive needs and specific priorities. This Technical Evaluation Group tackles numerous matters, including the definition of the standard minimum contents of evaluators’ activities; the analysis of the state of progress of the evaluation activities; the preparation of proposals for the presentation and dissemination of evaluation reports; the analysis of the relationship between monitoring and evaluation. 3. The approach to the evaluation As said earlier, the ESF program is characterized by a multiplicity and heterogeneity of contents, objectives pursued, actors involved and implementation modes. All this implies a particularly complicated evaluative activity, precisely because referred “to a complex and multi-level public programming, placed in a multi-actor context with a high institutional exposure” (Vergani, 2004). The Evaluation Unit had already worked out an approach to the evaluation (Isfol—Struttura nazionale di valutazione Fse, 1999) during the previous ESF program. With the mandate received for the 2000–2006 period, this approach was then systematized by elaborating guidelines and methodological standards for the mid-term evaluation of the ESF (Isfolstruttura nazionale di valutazione, Fse, 2002a,b, 2003a,b,c). The main components of the approach used are described below, namely an evaluation geared to an open system, multilevel, multi-focus and multi-actor. 3.1. An evaluation geared to an ‘open system’ The ESF program constitutes a public-policy intervention which is necessarily inserted in a wider context than the general and specific objectives it pursues. This means that the evaluation has not only to analyze the results, and to check that the objectives indicated during the programming have been achieved, but it has also to take into account the evolution of these objectives in the European, national and regional scenarios. The adoption of a goal-free approach (Scriven, 1991) to the evaluation means reconstructing all the effects

4

C. Lion, P. Martini / Evaluation and Program Planning 29 (2006) 1–9

produced by the program or linked to it, starting with its initial objectives. This choice was based on a series of considerations. First of all, the program can produce different or unforeseen effects which, if not suitably grasped, can lead to biased evaluations. Secondly, a multiplicity of actors, responsible for the implementation of the policies, and particularly the ESF ones, can change the original objectives in a complex process of negotiation and transfer. Finally, the evolution of the reference context, that is the Italian socio-economic, demographic and political-regulatory context, causes new objectives and priorities to emerge during the work, obviously not foreseeable when programming (the 7-year duration of the current program necessarily implies changes in the context). In this sense, the ESF program can be considered an ‘open system’ (Mark, Henry, & Julnes, 2000), that is created with the aim of acting on the external context, but which in turn is inevitably affected by this same context. It is permeable to impulses from outside because it is implemented within specific economic and social structures and has multiple relationships with the environment in which it is placed (Bulgarelli, 2000). The idea of programs as ‘open systems’ means a ‘realistic’ type of approach to the evaluation, where the key element lies in the fact that the programs are inserted in a context, or several contexts, which determine its outputs and results (Pawson & Tilley, 1997). According to this in the mid-term evaluation particular attention has been given to the analysis of the evolution of the political/regulatory and economic/demographic context of the labor market in which the ESF interventions have been implemented. This has helped to define the reference framework and has also enabled reasoning in terms of the relevance of the program strategy. The evaluation has further analyzed the links between the program and the EES and the relationship between the ESF and national policies. Moreover, in the light of the political and institutional context described above, the Evaluation Unit has also promoted and implemented a project for analyzing calls for tender (public-service tenders and public announcements) to compare the local supply with the administrations’ political strategies. In the new ESF 2000–2006 programming period the use of public-service tender procedures for implementing ESF activities has been extended nationwide. The decision to analyze these documents was based on the awareness that the drafting of the calls for tender is a crucial stage to study policies implementation since it provides key indications on the administrations’ strategic choices. Thus, the survey of calls for tender issued by administrations has allowed to define the connections between the original programming and that developed on a decentralized level. A computerized system was created for collecting and filing calls for tenders adopted during the 2000–2002 period by the managing authorities and delegated bodies. The aim was to provide constant updating on strategies and an in itinere check of underlying trends. The data bank enables examining, in real time, if and how the innovative aspects and the specificity of the new ESF program are finding concrete expression in the

managing authorities’ calls, verifying what is occurring nationwide and following its evolution during the programming period. Specifically, information has been collected on the following items: calls issued by each administration, measures and project typologies activated, recipients of the measures, proposing agents, and references to cross-cutting issues such as equal opportunities, local development, the information society and the social economy, financial data, evaluation grids for project selection, call dates of issue and deadlines. 3.2. A multi-level evaluation The evaluation has been carried out considering the various levels in which the interventions are spelt out within the ESF program. In this sense the approach has a multi-level character. The adoption of a multi-level logic means analyzing the ESF and its implementation in an evaluative key, considering the essential levels of implementation as set out in the logical programming structure presented in Table 1. However, a priority axis does not necessarily coincide with a policy objective and this implies a further level of analysis, reading across the priority axes by examining action and project typologies, as well as the targets reached. One of the key innovations of the 2000–2006 program is the different approach to the definition of the specific objectives, no longer target-oriented but policy-oriented. Shifting from a useroriented perspective to a transversal one has meant isolating significant levels of evaluation capable of providing useful and meaningful knowledge for policy-makers. For this reason, projects have been classified on the basis of homogeneous typologies, so that the different types of ESF financed actions implemented on a decentralized level can be compared. This is a classification structured on four levels, intermediate and in cascade, between the measure and the individual project as shown in Table 2. It reconciles the cognitive needs of the actors involved in the programming, that is it proposes a structure of the program implementation that provides information useful for all the different aggregation levels. The classification by project typology was based on the contents of the EC regulations and programming documents and shared with all the managing authorities. It enables an analysis unit to be defined on which to build the common and homogeneous area for the evaluation, thus giving a national reading of the experience. This classification has been incorporated in the monitoring information system and provides information on the implementation of the EES according to the European Statistical Office (EUROSTAT) ‘Labor Market Policy’ classification. 3.3. A multi-focus evaluation The multi-focus logic has mainly involved the analysis of outputs and an initial evaluation of the outcomes based on physical and financial indicators and on the results of the ad hoc job placement surveys for assessing the employment outcomes of the ESF interventions. The analysis of outputs and

C. Lion, P. Martini / Evaluation and Program Planning 29 (2006) 1–9

5

Table 1 Example of ESF program structure

outcomes is based on the indicators system that has been created according to the EC guidelines and in partnership with the other actors involved in the Technical Evaluation Group. Table 3 gives some examples. The national monitoring system enables these indicators to be collected and quantified. Since there are various actors responsible for program implementation in Italy, it was considered necessary to adopt a common language and to standardize the data collected. A centralized monitoring system has been set-up, in partnership with the Ministry for the Economy and the National Institute of Agrarian Economy (INEA) to gather data on the Structural Funds (ESF, the European Regional Development Fund and the European Agricultural Guidance and Guarantee Fund) which enables financial, physical and procedural data to be collected and filed. This is fed by local monitoring systems which use information provided by the executing agents. As far as outcomes are concerned, the Evaluation Unit has quantified the rates of coverage of the potential recipients, measured by comparing the annual average number of recipients involved in each measure with the target population, and the rate of the job placement of those trained by ESF vocational courses 1 year after the end of the training. For the job placement surveys, structured questionnaires were prepared, mainly administered by the Computer assisted telephonic interviewing (CATI) method. Examining interventions effects on recipients in terms of causality, that is to say results which would have occurred

without the intervention, involves problems of estimation. The simple comparison of job placement rates between recipients of the intervention and individuals who did not participate in it (counterfactual situation) produces biased results because of the existence of unobservable differences between trained and non-trained, with a different probability of job placement regardless of participation in training activities (selection bias). To take into account the non-observable differences between trained and non-trained, a stochastic selection model with endogenous switching was used. This enabled calculating the probability of job placement in the case of participation and non-participation in the intervention, together with the probability of belonging to the group of effective recipients. Bearing in mind possible self-selection and sample selection effects, the occupational outcomes of training on recipients were estimated, as well as the potential effect of training on individuals who had not participated in it (Berliri, 2003). In addition to outputs and outcomes quantification, the midterm evaluation focused also on other aspects. In particular, the Evaluation Unit has set-up an innovative model for evaluating the implementation process. This has helped to widen the focus of the evaluation to what has been defined as the ‘black box’ in the lifecycle of a policy (Isfol-Struttura nazionale di valutazione Fse, 2002a,b), in order to understand any differences between the expected and actual results and to study how a program/policy is put into practice (Rossi, Freeman, & Lipsey, 1999). The model is based on the hypothesis that the implementation of the ESF program, and in particular its outputs, are

Table 2 Examples of categorization levels Action macro-typology

Action typology

Project typology

Project sub-typology

Actions addressed to persons Actions addressed to persons

Training Incentives

Training inside compulsory training Incentives for persons’ geographical mobility

Apprenticeship Incentives for job seeking geographical mobility

Table 3 Examples of outputs, outcomes and impact indicators Indicators of outputs

Indicators of outcomes

Indicators of impact

Number of beneficiaries by gender, age, educational level, labor-market condition Coverage rate Activity rate

Number of firms per sector, economic activity and size Gross job placement rate Jobseekers turnover rate

Average duration of interventions; average cost of interventions Net job placement rate School dropout rate

6

C. Lion, P. Martini / Evaluation and Program Planning 29 (2006) 1–9

mainly influenced by: (i) factors within the system of actors involved in the ESF implementation; (ii) the context in which the programming is carried out; and (iii) the system of rules, procedures and technical supports that steer and support the implementation. As for actors, they are inserted in a network of balances of power and reciprocal relationships. Thus they exercise an influence on ESF implementation because they transform/reinterpret the ways in which the program goals are achieved. Moreover, the ESF program is shaped and developed by interacting with a context with which it has an open relationship of mutual conditioning/adaptation. The evolution of the external context is all important for program success or failure, since the implementation of the initially planned goals can be enhanced/changed/hindered by it. The context is also in turn influenced by the program, to the extent in which the outputs have an impact and thus change it. Finally, a program is implemented within the framework of specific regulations and uses an organization, procedures and technical supports that play a crucial role in regulating the outputs and outcomes. In defining the evaluation focus we have taken into account the ‘demand for evaluation’ expressed by different stakeholders. In particular, the Ministry of Welfare asked the Evaluation Unit to set-up a model for evaluating the quality of training systems. The aim was to define a model for evaluating the quality of training systems on a regional level through the analysis of their main components, namely the strategic programming, the executive planning and the monitoring and evaluation. This implied defining a set of indicators for describing and rendering comparable the different regional training systems. The model uses an analysis scheme which pinpoints, with progressive levels of precision, the elements determining the quality of a training system. At the same time, it also enables the performances of the different regional situations to be compared. The analysis is organized, in successive levels of detail, in terms of macro areas of quality dimensions and, in cascade, in factors/indicators (Isfol—Struttura nazionale di valutazione Fse, 2003a,b,c). As for methods and tools, the multiplicity and heterogeneity of the ESF actions has made it necessary to set-up specific tools and methodologies for the evaluation areas considered, in a multi-disciplinary perspective. In general, this option covers two different aspects: the implementation of the evaluation with the support of methods, techniques and tools belonging to different but complementary disciplinary fields, and the coordinated use of quantitative and qualitative methods (De Sanctis, Lion, & Vergani, 2000). Besides this, it seems important to stress that, in the midterm evaluation, the multi-disciplinary dimension mainly implies team work with professionals and experts from different disciplinary fields. This has given an input of perspectives and points of view which has enriched the significance and interpretation of the processes being evaluated.

3.4. A participatory/multi-actor evaluation Participatory evaluation is considered a priority dimension in the EC regulations. In the Italian context a partnership was set-up between the Evaluation Unit and other actors such as institutional bodies, i.e. the already mentioned Technical Evaluation Group as well as specific working groups set-up at the request of the various stakeholders. In particular, the working groups coordinated by the Evaluation Unit discussed scientific and methodological aspects of topics considered particularly important for the mid-term evaluation such as: the indicators system, implementation processes and job placement. This has entailed a process of accompaniment and response to the evolution in the demand for evaluation. This option has represented the most important dimension of the participatory evaluation the Evaluation Unit had decided to undertake, in terms of both contents and working procedures. 4. Some lessons learned from the mid-term evaluation The ESF mid-term evaluation has been carried out on the basis of the approach described above. An a posteriori reflection on the work undertaken has highlighted new elements to bear in mind. Some lessons can be learned when updating the mid-term evaluation, namely the need for more focused evaluation, stakeholders’ commitment, constraints of monitoring system, the necessity of quali-quantitative evaluation. 4.1. The need for a more focused evaluation The mid-term evaluation report covered all the ESF measures, analyzing for each one the administrations’ calls for tender, the implementation process, the links with other measures, the outputs and the financial progress, the first outcomes and impacts, and the relationship between the ESF policies and national policies. On levels above the measure (priority axis and objectives), the analysis followed the same path. Specific attention was devoted to the main changes in the relevant socio-economic as well as regulatory and institutional scenarios. This evaluative pathway is reflected in the final report which has a fractal structure, that is to say an architecture where the same analysis is repeated on different hierarchical levels. These go in a vertical mode from the program to the measure and in a horizontal way which include the analysis of the project typologies, the cross-cutting topics and the EES goals and priorities. This structure means that the individual parts of the report are independent, constituting self-sufficient units in content and in the information they provide. This undoubtedly offered very detailed data on each policy, offering points for reflection and extensive information for decision-makers. However, such a high level of details has sometimes made it difficult to obtain an overall vision of the results achieved. Political decision-makers were presented conclusions and recommendations only on an individual measure level.

C. Lion, P. Martini / Evaluation and Program Planning 29 (2006) 1–9

Furthermore, even though the fractal structure of the final report provides a more user-friendly document, where the relevant information is easier to pinpoint, this choice undoubtedly implies a long and not easily and manageable report. These are the limits of comprehensive evaluation which tries to answer to different questions due to the complexity of the program. One has to consider the opportunity to identify better the evaluation objectives and contents in order to address the main needs of policy-makers. 4.2. A participatory approach as a way to generate stakeholders’ commitment The mid-term evaluation partly changed the concept of ‘participatory evaluation’ as it had been formulated and implemented in the 1994–1999 experience. In the previous program, the participatory evaluation mainly figured as an ‘empowerment evaluation’ (Fetterman, 1996) in which the role of the external evaluator was also that of a facilitator inside a process of learning and improvement of programs by the different stakeholders. In line with Guba and Lincoln’s ‘fourth generation evaluation’ (1995), the concept of participatory evaluation implied guidance and knowledge sharing with the aim of improving programs (De Lellis & Vergani, 1998). In the mid-term evaluation this empowerment aspect has partly disappeared. A different way of interacting with stakeholders (Wholey, 2003), whose evaluation culture has grown over the years, was developed. This is imputable not only to (and thus consistent with) the devolution underway in Italy pertaining to human-resource development policies, but also to a learning process regarding evaluation. This has helped the various institutional actors implementing the ESF interventions to insert the evaluation within the life cycle of the programs and projects. The dimension of the participatory evaluation in the mid-term evaluation has therefore implied an intense preliminary work in defining methodologies and tools as well as priorities in partnership with those responsible for the ESF implementation and with those actually implementing it. This has generated a shared knowledge on the entire evaluation process from the identification of the demand for evaluation to the presentation of the findings. In such a context characterized by a decentralized program/policy implementation and by a growing evaluation culture among stakeholders, participatory evaluation has above all meant to define a common language and practice for evaluation activities in order to allow the comparability of results and an across-the-board analysis on a national level. 4.3. The importance to bear in mind possible constraints of the monitoring system As already observed, the focus of the mid-term evaluation was the analysis of the program outputs and outcomes. These were quantified through the indicators calculated on the basis of the data present in the national monitoring system and of the results of the regional/provincial job placement surveys.

7

Although representing a rich storehouse of information on the ESF policies implemented, the monitoring system presented some critical elements that prevented the Evaluation Unit from fully tapping this source for the mid-term evaluation. The physical and procedural progress data was often lacking, and thus not entirely reliable for the various analysis levels covered in the report. The financial monitoring is now standardized and the information provided has represented a solid base on the program progress. The drawbacks found in the monitoring system are mostly linked to technical problems in transferring the data from the managing authorities’ information systems to the central one and from the final beneficiaries to the managing authorities. In the mid-term evaluation, the Evaluation Unit tackled these difficulties by tapping other information sources to supplement the lack of data in the central monitoring system. To obtain a complete picture for the ESF, it was thus necessary to examine, for example, the annual implementation and evaluation reports and the regional data on single projects and recipients. Many efforts have to be made towards the implementation of efficient monitoring systems which can produce useful information on program outputs and outcomes in real time. However, one should concurrently consider the opportunity of tapping other information sources. 4.4. The importance to combine a quantitative evaluation with a qualitative evaluation The quantification of outputs and results has undoubtedly represented the core of the evaluative analysis. However, when drafting the report, the ‘quantitative dimension’ of the evaluation developed alongside further analyses which addressed qualitative aspects. These refer not only to the contextualization of the program, as mentioned earlier, but also to the analysis of regional evaluation reports, according to an ‘evaluative research’ approach (Blalock, 1999). Therefore, the program physical implementation and financial progress was not assessed solely on the basis of quantitative indicators. The indicators used were not considered explicative in themselves but represented a ‘point of departure’ (Vergani, 2004) for further analyses aimed at combining neo-positivist and constructivist types of approaches to evaluation (Cronbach, 1982). This latter aspect suggests that information sources for the evaluation should not be limited to the system of indicators, anyway mandatory, but extended to a wide and diversified range of sources which, as explained, can also help to fill possible gaps in the monitoring systems themselves. 5. The relationship between stakeholders and evaluators: the utilization and dissemination of the results There are many studies which have dealt with the possible uses of the evaluation and its utilization (Patton, 1978; Weiss, 1998). How can it be ensured that an evaluation is utilized? What are the features an evaluation must possess to be of

8

C. Lion, P. Martini / Evaluation and Program Planning 29 (2006) 1–9

effective help for those who have to program activities and make decisions? How is it that the results of evaluations are often ignored, despite the fact that often considerable financial resources are committed? The relationship between stakeholders and evaluators is often difficult since the two players do not always have the same interests and the same needs. The main critical points boil down, on the one hand, to the timing of the evaluation and, on the other hand, to the results which can sometimes appear as a threat for stakeholders or which might not meet their cognitive needs. As for the timing, a complex program needs lengthy evaluation times and involves a large number of people with different qualifications and specializations. The stakeholders often need rapid information to make decisions and press the evaluator to provide them with it. As for the results of evaluation, in order to achieve a positive contamination by politicizing the scientific field while ‘scientifizing’ the political one (Brousselle, 2004), it was decided to adopt a participatory type of approach. This means involving stakeholders right from the design stage of the evaluation, to increase the validity and credibility of the research results. The Evaluation Unit’s intention has been to accompany the programming stage with the aim of producing an in itinere evaluation that would not only increase the various actors’ knowledge of the functioning of a policy but, above all, that would provide useful elements for making decisions and steering policies. Thus, the choices made when organizing the activities and defining the report were prompted by the fact that the utilization of the evaluation is closely connected to its usefulness for political decision-makers. The evaluation was seen as an analysis of the state of progress of a policy/program for supporting the decision-making process, that is to say, as an activity increasingly inserted in the policy-cycle and strictly linked with the programming of the interventions. In order to implement an evaluation which is not a mere bureaucratic exercise but a useful and utilized activity, it is important to involve the various stakeholders right from the launch of the program through a common decision on the methodological tools to adopt and on the policies to analyze. The Evaluation Unit’s twofold role as ESF evaluators and as scientific and methodological coordinators for the evaluation of the regional programs facilitated this task. The EC regulations attribute an important role to the midterm evaluation since it links the ESF reprogramming process to it. Moreover in this 7-year period, it also links the evaluation to the new mechanism for allocating the performance reserve that awards additional financial resources to the most efficient administrations. Precisely because the evaluation is useful for reprogramming, the Evaluation Unit had been asked to finish it before the deadline fixed by the EC regulations (December 2003), so that the results would be available for supporting policy-makers in their decisions. In fact, the evaluation report was transmitted to the Ministry of Welfare and the European Commission in September 2003. This meant that the various actors involved were provided with information useful for reprogramming the ESF activities at a national and a regional

level. The respect of the deadline was very important in fostering the policy-makers’ utilization of the evaluation. The communication strategy to inform the various stakeholders about the outcomes of the evaluation was multidirectional. The evaluation report was distributed in two versions: a more analytical version which presents the main methodological choices adopted and describes the outcomes in details, and a briefer one summing up the main results of the evaluation. The first version, besides being delivered to the Ministry of Welfare and to the European Commission, was divided into thematic sections to help consultation and inserted in the websites of both the Evaluation Unit and the Ministry of Welfare. The summary evaluation report was first presented to the Technical Evaluation Group. It was then distributed during a national conference organized by the Ministry of Welfare on the role of the ESF in the European employment strategy. During the conference held in 2004, the Ministry of Welfare dwelt on some key topics covered in the report, such as: continuing training within the logic of lifelong learning, equal opportunities and job placement policies, commenting on the main findings emerging from the report. To disseminate the results of the report to the widest possible public, it is currently being published in its complete version. Particular attention has been devoted to the language, rendered less technical to make the publication more approachable for anyone interested in finding out about the features and the results of the ESF policies. The information contained in the evaluation report has also been used by the Ministry of Welfare for the reprogramming of interventions. The recent document for reprogramming the 2000–2006 ESF interventions contains a chapter summarizing the main outcomes of the first 3 years of implementation and the main indications emerging from the mid-term evaluation. The Evaluation Unit’s objective of providing a significant knowledge on human-resource development policies (training, education, social and labor policies), in a systematic and recurrent manner, was achieved thanks to a political context involved right from the evaluation design stage, aware of the importance of the evaluation activity and ready to acknowledge its outcomes. 6. Conclusions This article presents the Evaluation Unit’s experience in evaluating the 2000–2006 ESF program in itinere by describing the pathway taken and the lessons learned. Owing to the complexity of the program and on account of the new institutional scenario which has gradually appeared in recent years, the approach to the evaluation was specifically aimed at: † sharing evaluation methods and tools with the various stakeholders; † analyzing significant policy levels, not necessarily coinciding with the program priority axes and measures; † quantifying observable facts sustained by a reading of the main outputs in a qualitative key;

C. Lion, P. Martini / Evaluation and Program Planning 29 (2006) 1–9

† providing a support for policy-makers for reprogramming; † disseminating outcomes taking into account the various stakeholders. The evaluation experience has highlighted the importance of allowing for the constraints linked to the availability of information data, for the times available for producing information useful for political decision-makers and for the—not always coinciding—cognitive needs of the multiple stakeholders. It was also recognized that a greater dissemination of the evaluation outcomes to a wider public should be encouraged with a more user-friendly format. In the light of the above and in view of the final evaluation it seems appropriate to focus attention on some aspects. The first aspect concerns the complexity and heterogeneity of the ESF programs which means deciding whether to carry out an extensive evaluation or to focus on some key issues. The EC regulations provide an initial indication stating that the evaluation of outcomes should be the main aim of the final evaluation report. In addition, the evaluation demand has already been clearly spelt out both by the European Commission and by the Ministry of Welfare, who have shown interest in various issues that will help to focus the analysis. Also in this case the final evaluation report will have to be completed at least a year before the end of the program to help defining the new 2007–2013 ESF program in terms of strategies and priorities. Another important aspect regards the need for a further monitoring effort on the part of the different actors to render the data transmission more efficient. Finally, the challenge for the future is that of extending the range of potential users of evaluation results and recommendations. For example, the communication strategy might include organizing specific workshops and thematic seminars open to a wider public than that strictly involved in the ESF policies in order to outline the potentialities and limits of certain interventions. Acknowledgements We are grateful to the three anonymous referees and to Patrice Poupon for their useful comments and suggestions. References Berliri, C. (2003). Employment impacts of ESF training interventions on particular target groups. Paper presented at the 5th Conference on Structural Funds Evaluation, Budapest. Bezzi, C. (2001). Il disegno della ricerca valutativa. Milano: Franco Angeli. Blalock, A. B. (1999). Evaluation research and the performance management movement. Evaluation, 5(2), 117–149. Brousselle, A. (2004). What counts is not falling... but landing. strategic analysis: An adapted model for implementation evaluation.. Evaluation, 10(2), 155–173. Bulgarelli, A. (2000). Valutazione di programma: Riflessioni dalla ricerca. In M. Palumbo (Ed.), Esperienze e riflessioni (pp. 106–116). Milano: Franco Angeli.

9

Bustelo, M. (2003). Evaluation of gender mainstreaming: Ideas from a metaevaluation. Evaluation, 9(4), 383–403. Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey Bass. De Lellis, A., & Vergani, A. (1998). Participatory evaluation and transformation of institutional systems. Paper presented at the fourth Conference of the European Evaluation Society, Rome. De Sanctis, A., Lion, C., & Vergani, A. (2000). La valutazione del Docup ob. 4 Fse 1994-99: tra multidisciplinarita` ed eclettismo, tra integrazione e giustapposizione metodologica. Paper presented at the third Congress of the Italian Evaluation Association, Turin. Fetterman, D. M. (1996). Empowerment evaluation: An introduction to theory and practice. In D. M. Fetterman, S. J. Kaftarian, & A. Wandersman (Eds.), Empowerment evaluation: Knowledge and tools for self-assessment and accountability. Thousand Oaks, CA: Sage Publications. Guba, E. G., & Lincoln, Y. S. (1995). Fourth generation evaluation. Newberry Park, CA: Sage Publications. Isfol—Struttura nazionale di valutazione Fse (1999). Valutazione finale dell’Obiettivo 3 in Italia. Milano: Franco Angeli. Isfol—Struttura nazionale di valutazione Fse. (2002a). Linee guida per il monitoraggio e la valutazione del Fondo sociale europeo 2000–2006. Metodologie per la valutazione di programma, Vol. 1. ISSN 1594-9516. Isfol—Struttura nazionale di valutazione Fse. (2002b). Orientamenti metodologici per la valutazione del processo di implementazione di programma. Metodologie per la valutazione di programma, Vol. 2. ISSN 1594-9516. Isfol—Struttura nazionale di valutazione Fse. (2003a). Linee guida per la valutazione degli effetti occupazionali del Fondo sociale europeo 2000– 2006. Metodologie per la valutazione di programma, Vol. 3. ISSN 15949516. Isfol—Struttura nazionale di valutazione Fse. (2003b). Approccio alla valutazione della qualita` dei sistemi formativi. Metodologie per la valutazione di programma, Vol. 4. ISSN 1594-9516. Isfol—Struttura nazionale di valutazione Fse (2003c). La strategia europea per l’occupazione. Il contributo del Fondo sociale europeo in Italia nel 2000– 2002. Milano: Franco Angeli. Lion, C., Martini, P., & Volpi, S. (2004). The evaluation of European social fund programmes in a new framework of multilevel governance: The Italian experience. Regional Studies, 38(2), 207–212. Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation. An integrated framework for understanding, guiding, and improving public and nonprofit policies and programs. San Francisco, CA: Jossey Bass. Palumbo, M. (2001). Il processo di valutazione. Decidere, programmare, valutare. Milano: Franco Angeli. Patton, M. Q. (1978). Utilization-focused evaluation. Beverly Hills, CA: Sage Publications. Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage Publications. Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (1999). Evaluation. A systematic approach (6th ed.). London: Sage Publications. Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, R. M. Gagne, & M. Scriven (Vol. Eds.), Perspectives of curriculum evaluation: Vol. 1, 39–83. Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newberry Park, CA: Sage Publications. Stame, N. (1998). L’esperienza della valutazione. Roma: Edizioni Seam. Vergani, A. (2004). Casi di valutazione. Processi valutativi e azioni formative. Bologna: Il Mulino. Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Englewood Cliffs, NJ: Prentice-Hall. Weiss, C. H. (1998). Evaluation (2nd ed.). Upper Saddle River, NJ: Prentice Hall. Wholey, J. S. (2003). Improving performance and accountability: Responding to emerging management challenges. In S. I. Donaldson, & M. Scriven (Eds.), Evaluating social programs and problems. Vision for the new millennium. Mahwah, NJ: Lawrence Erlbaum Associates Publishers.