Building a worldwide evaluation community: past, present, and future

Evaluation and Program Planning 23 (2000) 449±459

www.elsevier.com/locate/evalprogplan

Building a worldwide evaluation community: past, present, and future A.J. Love a,*, C. Russon b a

Independent Consultant, 40 St. Clair Avenue East, Suite 309, Toronto, ON, Canada M4T 1M9 b The Evaluation Center, Western Michigan University, Kalamazoo, MI 49008-5178, USA Accepted 1 June 2000

Abstract This article describes recent efforts to build a worldwide evaluation community. It notes events that led to an almost 200 percent growth in the number of regional and national evaluation organizations in slightly more than one year. Pro®les are provided on the present status of many national and regional evaluation organizations that participated in the W.K. Kellogg Foundation-sponsored international evaluation meeting in Barbados, West Indies. Afterwards, the authors engage in a scenario planning exercise to picture alternatives of what the future of international evaluation might look like. q 2000 Elsevier Science Ltd. All rights reserved. Keywords: Worldwide evaluation community; National evaluation organizations; Scenario evaluation; Barbados residency

During February 2000, the presidents and of®cial representatives from 15 regional and national evaluation organizations attended a residency meeting in Barbados, West Indies, to explore the possibilities for greater international cooperation among evaluation organizations. Observers from the W.K. Kellogg Foundation, the University of the West Indies, the Caribbean Development Bank, and the United Nations Capital Development Fund also attended. The W.K. Kellogg Foundation provided a grant to support this landmark event, and UNICEF-Barbados assisted with the local arrangements. In many ways, the meeting in Barbados was a step in a process that began at the ®rst International Evaluation Conference in Vancouver in 1995, a conference that brought together more than 1600 evaluators from 65 countries. The success of the 1995 conference identi®ed the need for more comprehensive exchanges among evaluators worldwide and increased opportunities to meet with each other. Following the Vancouver conference, international exchanges began increasing rapidly. Several evaluation organizations facilitated these exchanges through reciprocal memberships and by including speakers and workshop presenters from other evaluation organizations in their national conferences. Powered by the new technology, evaluation began growing at the speed of thought. The number of listservs focused on international and crosscultural evaluation issues increased (e.g. XC-eval). Global * Corresponding author. Tel.: 1416-921-2109; fax: 1416-921-7600. E-mail address: [email protected] (A.J. Love).

evaluation networks expanded and assumed new signi®cance as more evaluators from different countries began to use the Internet to communicate effectively and collaborate on evaluation projects. Evaluation has been growing worldwide more quickly than anyone anticipated, swept along by change as dramatic as the industrial revolution, presenting us with tremendous opportunities and equally signi®cant challenges. One marker of the growth of evaluation is the exponential increase in evaluation associations and evaluation networks in recent years. A decade ago, German political scientist Hans-Ullrich Derlien (1990) traced the growth of public policy evaluation in western industrialized countries. In his comparative study, Derlien used a number of factors to characterize certain countries as the ª®rst waveº of early adopters of public policy evaluation (US, Canada, Sweden, Germany, and the United Kingdom) and others as members of the ªsecond waveº (Denmark, Netherlands, Norway, Switzerland). What we are witnessing now is not only the continued growth in evaluation at many levels (e.g. policy, program, project, product), but the unprecedented growth in evaluation associations. To apply Derlien's concepts to the development of evaluation associations, there were very few ®rst wave evaluation associations (US, Canada, Australasia) that formed in the 1970s and 1980s, followed by several second wave associations in the mid-1990s (United Kingdom, Europe, Switzerland and, more recently, Italy), and then explosive growth of ªtidal waveº associations in the last few years. In 1998 there were 9 national and regional

0149-7189/00/$ - see front matter q 2000 Elsevier Science Ltd. All rights reserved. PII: S 0149-718 9(00)00035-5

450

A.J. Love, C. Russon / Evaluation and Program Planning 23 (2000) 449±459

evaluation associations, and by the fall of 1999 there were 26. In recent months, several new associations in western Europe have incorporated; Central Eastern Europe is rapidly forming its own national evaluation societies; Russia has organized an evaluation network; the very successful 1999 African evaluation conference has spawned not only the African Evaluation Association, but also numerous national associations; several associations have been chartered in South and Southeast Asia; the new Japanese Evaluation Association has been formed; and plans are under way for new evaluation associations in the Caribbean and Latin America. Although these may be exciting developments in themselves, what do they mean to the ®eld of evaluation? In our opinion, the development of national and regional evaluation associations is critical to the ®eld and profession of evaluation because evaluation associations bring together evaluators from multiple disciplines to share knowledge and experiences, bridge disciplinary divides, debate issues of fundamental importance, set standards and ethical guidelines, build skills, and chart the future as a group with a strong and shared identity. Although evaluation may be stimulated by ªdemandº from politicians and government institutions, evaluation associations have an equally important role in stimulating the ªsupplyº of the highest quality evaluation expertise. When assessing the supply role of evaluation associations, it is important to recall the wise observations of Wagner and Wollmann (1986) that evaluation methods have nowhere been invented by the bureaucracy and that a nation's evaluation culture is decisively dependent on the propensity of academics and others to produce evaluation methods, engage in evaluations, and shape the professional roles of evaluators. Likewise, if we view institutionalizing evaluation as desirable, the probability that skilled evaluators are recruited into staff positions also re¯ects evaluator supply and a strong evaluation tradition. These may be good arguments for developing national or even regional evaluation associations, but why should evaluators in established evaluation associations collaborate with less established associations in developing countries or work to foster greater international exchange and cooperation among evaluation associations? Building a worldwide evaluation community provides a tremendous opportunity for all of us to learn from each other and to further the evaluation profession. One indication of this enormous opportunity came during the collaborative work on The Annotated Bibliography of International Programme Evaluation (Russon & Russon, 2000). As the research teams spoke with evaluators worldwide, it was quickly evident that there was a tremendous amount of previously untapped evaluation knowledge, including published works, and that valuable information had not yet been shared with evaluators in North America or in other parts of world, including people in their own regions.

It has been said that evaluation is one of the best kept secrets in the world. If one travels country by country and region by region, it is evident that there is not a strong and uni®ed voice to promote evaluation, standards of evaluation, or to speak on issues relevant to humanity that involve evaluation data. Evaluation needs to have a greater presence, and it seems more likely that this can be done better as an international community than as individual associations. Right now, organizations such as the World Bank and the Organization for Economic Cooperation and Development work with governments at the senior level to create the recognition and demand for evaluation. These efforts are essential. The idea of a worldwide community of evaluators approaches the same issue from the other direction. This is a grass-roots effort by networks and associations. When one examines the growth of evaluation worldwide, it is evident that this interest at the grass-roots level is not just a ¯ash-inthe-pan phenomenon. When meetings are organized, dozens, if not hundreds of people travel great distances to attend. Not just a few people. Not just an elite group. Not just dignitaries and of®cials. There is a tremendous demand for evaluators and users of evaluation to come together, and there is enormous power in these grass-roots networks to move the whole process forward. History is showing that if it is built, they will come. Several of the established and less established evaluation organizations have tried to further this process of international cooperation among evaluation associations, not with the intention of forcing any one point of view or limiting national prerogatives, but to provide mutual support, expertise, and energy in fostering the evaluation profession worldwide. This article has three purposes: (1) to brie¯y trace the history of the effort to build a global evaluation community; (2) to give evaluators a better sense of the present status of evaluation associations and networks; and (3) to sketch a few scenarios about possible future developments. 1. The past: building a worldwide evaluation community The idea of creating a closer relationship among national evaluation organizations began in the fall of 1997 with a discussion on the EVALTALK listserv about the international nature of the evaluation profession. Shortly after the discussion, the International and Cross-Cultural Evaluation Topical Interest Group (I&CCE) of the American Evaluation Association (AEA) invited the presidents of nine national evaluation organizations to sit on a Presidents Panel during the 1998 AEA conference to discuss ways of enhancing collaboration and interchange. In addition to the presidents, the I&CCE TIG designated one of its cochairs to sit on the panel to represent the interests of developing countries and regions of the world that did not have a formal evaluation association. As part of a

A.J. Love, C. Russon / Evaluation and Program Planning 23 (2000) 449±459

communications strategy, a special listserv was created for the exclusive use of the Presidents Panel participants to discuss preliminary issues and agree on the panel format. During a plenary session at the 1998 AEA conference, the presidents of several national evaluation organizations debated the pros and cons of entering into a collaborative relationship. The organizations represented on the panel were the African Evaluation Association, American Evaluation Association, Associazione Italiana di Valutazione, Australasian Evaluation Society, Canadian Evaluation Society, Kenyan Evaluation Society, and the United Kingdom Evaluation Society. Position papers had been submitted in advance by the organizations represented on the panel as well as by the European Evaluation Society and the Malaysian Evaluation Society. Arnold Love, former president of the Canadian Evaluation Society and a principal organizer of the International Evaluation Congress that was held in Vancouver, Canada, in 1995, moderated the panel. The position papers may be found in Russon and Love (1999). The presidents or their representatives met again at the 1999 AEA conference in Orlando, Florida, to discuss developments during the time since they last met. The 1999 Presidents Panel included representatives from the African Evaluation Association, American Evaluation Association, Associazione Italiana di Valutazione, Canadian Evaluation Society, Israeli Association of Program Evaluation, Kenyan Evaluation Association, PREVAL (Latin America), and the Rwandan Evaluation Network. Once again, Arnold Love moderated the Panel, and Michael Bamberger of the World Bank offered his comments and observations. A grant awarded by the W.K. Kellogg Foundation enabled the AEA Presidents Panel to evolve into the residency meeting in Barbados. The idea of holding the meeting in the developing country of Barbados was a way to extend support for evaluation into the Caribbean region (one of the focus areas for the Kellogg Foundation), while providing a venue for many of the leaders and representatives of evaluation societies and networks to examine some pressing questions: What is our vision for developing an international evaluation community? What are the various obstacles that we need overcome? What form of organization, if any, would work well? How can we foster communication and common agendas? Consistent with the primary goal of holding a residency meeting in 2000, the W.K. Kellogg Foundation provided scholarships to partially defray the travel costs of new evaluation organization presidents from developing countries to attend one of the preresidency meetings at the conferences of Australasian Evaluation Society, Canadian Evaluation Society, African Evaluation Association or the American Evaluation Association. During these conferences the presidents had the opportunity to meet other members of the international evaluation community, participate in the preliminary work leading to the residency, and

451

make the residency more effective at moving toward real collaboration among equals. The Barbados residency meeting had several objectives: (1) building trust among the presidents through face-to-face interactions; (2) creating a common vision for collaborative relationships; (3) developing a strategic plan to set a course for the collaborative relationships; and (4) developing a concrete operational plan to ensure that the strategic plan would be accomplished. 2. The present: building evaluation associations and networks at Barbados The organizations that were represented at the Barbados meeting were the African Evaluation Association, American Evaluation Association, Asociacion Centeroamericana de Evaluacion, Associazione Italiana de Valutazione, Australasian Evaluation Society, Canadian Evaluation Society, European Evaluation Society, Israeli Association for Program Evaluation, Kenyan Evaluation Association, La Societe francËaise de l'EÂvaluation, Malaysian Evaluation Society, Programme for Strengthening the Regional Capacity for Evaluation of Rural Poverty Alleviation Projects in Latin America and the Caribbean (PREVAL), Reseau Ruandais de Suivi et Evaluation (Rwanda), Sri Lankan Evaluation Association, and the United Kingdom Evaluation Society. In addition representatives from the W. K. Kellogg Foundation, The University of the West Indies, the Caribbean Development Bank, and the United Nations Capital Development Fund were present. The meeting was facilitated by Arnold Love and Craig Russon. The following summaries about the participating evaluation associations and networks are based upon the published position papers; information from each organization's newsletters, journals, and promotional materials; and presentations made by the presidents or of®cial representatives during the ®rst day of the residency meeting. The following snapshots of each organization that participated in the Barbados meeting gives a good sense of the tremendous potential and growing capacities in evaluation worldwide. Since space limits these descriptions, interested readers may access more information about position papers and the national evaluation organizations by referring to the website list at the end of this article. The snapshots are presented in alphabetical order. African Evaluation Association Ð One of the newest regional evaluation organizations, the African Evaluation Association (AfES) had its ®rst meeting in 1999 with more than 300 people attending from more than 35 countries. The conference operated in two languages with simultaneous translation between French and English. Mahesh Patel, who also represented the Kenyan Evaluation Association at the Barbados meeting, said that one of the main reasons for initiating the African Evaluation Association was to build indigenous capacity, and it was ®tting that

452

A.J. Love, C. Russon / Evaluation and Program Planning 23 (2000) 449±459

this was the overall theme of the ®rst conference. There were 7 parallel strands, more than 80 paper presentations, and a number of training sessions, including keynote training sessions by Michael Quinn Patton. Although most of the agencies had been offering introductory training in evaluation in Africa, there was never anything offered beyond the introductory level training. The conference provided an opportunity to bring in an external trainer who could deliver topnotch training that was not usually available. The African Evaluation Association formed work groups to ensure the progress of the evaluation profession on the African continent. The work groups developed plans of action for adapting The Program Evaluation Standards (Joint Committee on Standards for Educational Evaluation, 1994) to include the DAC Evaluation Guidelines and make the standards culturally and politically appropriate for Africa. For example, the original standards have a heavy emphasis on documentation and no recognition of village oral tradition, and some of the disclosure standards would be risky to apply in countries with military dictatorships. The work groups also discussed plans for forming national evaluation associations in Africa, creating a database of evaluators in Africa that include speci®c skill pro®les, developing a bibliography, and publishing the conference proceedings and individual articles. The African Evaluation Association sees itself as a defender of developing country interests. It views its role as one of bringing developing country issues to the international evaluation community. The association has several concerns about participating in international initiatives. One of these concerns is a loss of diversity. There is a fear that Western evaluation theory and practice may supplant indigenous theory and practice of evaluation. Another concern is that participation in international initiatives would divert energy away from developing capacity on the African continent. American Evaluation Association Ð Incorporated more than 20 years ago, the American Evaluation Association (AEA) is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness. The American Evaluation Association's mission is to improve evaluation practices and methods, increase evaluation use, promote evaluation as a profession, and support the contribution of evaluation to the generation of theory and knowledge about effective human action. The association sponsored the panel that brought the presidents together in 1998 to discuss the idea of forming a partnership (Russon & Love, 1999). In addition, it was AEA's International and Cross-Cultural Evaluation Topical Interest Group (I&CCE) that submitted the proposal on behalf of the organization presidents to the W.K. Kellogg Foundation

for the Barbados meeting. The purpose of the I&CCE TIG is to provide evaluators who are interested in cross-cultural issues with opportunities for professional development. I&CCE sponsors paper sessions and panels during the annual meeting of the association. Donna Mertens spoke on behalf of AEA. She emphasized that AEA welcomes further opportunities to help strengthen the international community of evaluators. The association would prefer a partnership with a ¯at organizational structure that is based on reciprocal relationships. In order to ensure that the partnership is sustainable, AEA would favor a formal agreement among the partners. It is important that the partnership not be dominated by a single voicerather, there should be pluralistic views represented. AsociacõÂon Centroamericana de EvaluacõÂon Ð Xinia Picado spoke both as president of the Central American Evaluation Society (ACE) and the representative of the Costa Rican Professional Evaluation Society. She described the growth of the evaluation ®eld in Costa Rica that led to founding the Masters Program in Evaluation of the University of Costa Rica in 1995. The program has a staff of 15 persons who teach courses on how to evaluate the economic, social, and environmental impact of development programs and projects. Many members of ACE are af®liated with the master's degree program in evaluation of development programs and projects. To date, the program has had 25 graduates and has trained more than 350 government of®cials. One goal of the program is to build evaluation capacity within the government in order to improve political, economic, social, and environmental decisions. Picado said there is a strong interest in Central America and Costa Rica in knowing more about evaluation, training people in evaluation, and developing an evaluation culture. There are ongoing relationships with evaluators in other countries, and ACE strongly supports greater international cooperation in evaluation and sharing evaluation knowledge and experience. Associazione Italiana di Valutazione Ð Established in February 1997, president Nicoletta Stame remarked that the Associazione Italiana di Valutazione (AIV) was one of the ®rst evaluation associations to be established in Europe. The association grew out of an exchange of ideas among an informal network of professionals and academics through seminars and workshops. The network members thought it was important to upgrade the activities that were taking place informally and to provide a forum to discuss professional, theoretical, and political matters in evaluation among people with different persuasions, institutional roles, and social backgrounds. The AIV also was a response to the demand for evaluation caused by the European Union Structural Program evaluation and the public sector reforms initiated by Department of Public Administration. AIV's mission and goals are to promote the culture of evaluation in public administration as well as among professionals, provide training and education on evaluation theories and methods, improve the quality of evaluations,

A.J. Love, C. Russon / Evaluation and Program Planning 23 (2000) 449±459

and promote the utilization of evaluations. The AIV has developed ªGuiding Principles for Evaluatorsº that address ethical issues and evaluation standards. AIV is interested in the development of a worldwide evaluation community. Rather than an formal international organization, however, AIV preferred a less formal task force of representatives to promote ªlearning through diversityº by a real exchange of ideas and experiences among evaluators. From the AIV viewpoint, the most important issue is how to include evaluation organizations or informal networks that were not in the initial core group. AIV was willing to delegate some of its members to a task force. Australasian Evaluation Society Ð Initiated in the late 1980s, the Australasian Evaluation Society (AES) was incorporated in 1991 as the ®rst regional evaluation association that expressly served members in two countries: Australia and New Zealand. With more than 400 members at incorporation, by 1999 AES membership had grown to more than 750. The aim of the society is to improve the theory, practice, and use of evaluation through establishing and promoting ethics and standards in evaluation practice; providing forums for the discussion of ideas including society publications, seminars, workshops, and conferences; linking members who have similar evaluation interests; providing education and training in matters related to evaluation; recognizing outstanding contributions to the theory and/or practice of evaluation; acting as an advocate for evaluation; and other activities consistent with this aim. Penny Hawkins spoke on behalf of AES, building on Sue Funnell's 1998 position paper, and she stressed that AES's support for international initiatives must be consistent with its organizational values, especially diversity and learning through diversity. AES does not endorse the creation of a new society. AES would prefer a strategic alliance that focused on exchange of views on important issues. Hawkins listed three goals that would be appropriate for an alliance to undertake: (1) improve the pro®le of evaluation, (2) improve evaluation's capacity to in¯uence decision making, and (3) support developing nations within developed nations (e.g. aboriginal areas). She also noted the contribution of evaluation to the radical change in the public sector that has taken place in New Zealand. Canadian Evaluation Society/La SocieÂte canadienne d'eÂvaluation Ð The Canadian Evaluation Society (La SocieÂte canadienne d'eÂvaluation) (CES/SCE) was incorporated over 20 years ago and is dedicated to the advancement of evaluation for its members and the public. With more than 1300 individuals and organizations as members, the CES has chapters in every province and territory in Canada that offer services such as resource networks, professional development activities, and forums for CES members to explore, discuss, and get involved with evaluation interests. CES holds the view that the evaluation profession throughout the world faces two major challenges: (1) to convince governments and the voluntary sector that evaluation can make an important contribution to informed

453

decision making (advocacy); and (2) to strengthen the competence of evaluation practitioners (professional development). There is also the need to strengthen the evaluation profession in developing countries. Since it is uncertain at this point whether an international organization is a feasible and cost-effective way to address these challenges, CES supports the idea of international cooperation, rather than an international organization. CES members said there was enough promise in the idea of international cooperation that the national organizations should begin with the most obvious international roles, learn from the process, and not invest heavily in organizational infrastructure. The CES representative suggested starting with a task force with the aim of fostering information exchange and collaboration among national evaluation organizations. European Evaluation Society Ð The European Evaluation Society (EES), was founded in The Hague in 1994. The ®rst of®cial board was elected in autumn 1995 and started its work in January 1996. The society's primary goal is to promote theory, practice, and utilization of high quality evaluation especially, but not exclusively, within the European countries. This goal is obtained by bringing together academics and practitioners from all over Europe and from any professional sector, thus creating a forum where all participants can bene®t from the cooperation and bridge building. Hellmut Wollmann, president of EES, reported in his position paper that the EES board unanimously agreed it was in favor of a general increase in international exchange, would ®nd more formal arrangements for liaison between different evaluation societies useful, and did not support a federation or separate organization. He explained that EES is international by its very nature and that its priority must be to support the many new and emerging evaluation societies in southern and central Eastern Europe. Initiatives such as holding the EES Conference in Italy, the founding of the Italian Evaluation Society, strengthening evaluation in Spain and Greece, and forging links with central eastern Europe with the sponsorship of SIGMA/OECD are examples of this international scope within greater Europe. EES did support this initiative insofar as it aimed at deepening and strengthening the international discourse and exchange without creating a separate organization with its own budget. Elliot Stern attended the Barbados residency to represent EES. Israeli Association for Program Evaluation Ð David Nevo, president of the Israeli Association for Program Evaluation (IAPE), described it as a nonpro®t professional organization comprised of academics, practitioners, and users of program and project evaluation in a variety of ®elds such as education, social services, health, law enforcement, and business. The association's main goals are to increase the use of program evaluation and its ®ndings for decision making, encourage the development of evaluation theory and practice in Israel, achieve the recognition of program evaluation as a means of improving the effectiveness of

454

A.J. Love, C. Russon / Evaluation and Program Planning 23 (2000) 449±459

interventions and strengthening communities, promote program evaluation as a profession, and develop professional ties among evaluators and users of evaluation in Israel and around the world. Currently, there are more than 70 paid members and more unpaid members of IAPE. In February 2000, IAPE held its ®rst conference with more than 70 evaluation professionals in attendance. The theme of the conference was ªWhose Reality Countsº (Chambers, 1997). Although Israeli evaluators are fortunate to have a signi®cant capacity for evaluation, the bad news is that there is a lot of evaluation but nobody uses it. There are fascinating examples of projects that were phased out, but their evaluation was continued. A major concern of IAPE is to improve the use and utilization of evaluation. IAPE is in favor of a partnership of national evaluation organizations. A partnership that would provide international support for national efforts, increase cooperation among evaluation associations, and improve evaluation utilization would be congruent with the goals of the Israeli association. Kenyan Evaluation Association Ð The Kenyan Evaluation Association (KenEA) met every two or three weeks for about two years as a loose network, according to Mahesh Patel, who represented KenEA in Barbados. During that time the Kenyan network had more than 30 meetings with members or visitors who presented papers. In 1999, the network held elections and started to organize into a registered organization with a constitution. The plan for the coming year is to undertake evaluation work as a professional association. The KenEA plans to start taking evaluation commission work in Kenya and to offer its services as quality control or quality assistance for evaluations that organizations contract with individual evaluators in the country. The Kenya Evaluation Association might charge about 10 percent of the costs of the evaluation for its services. It might then keep 10 percent of those fees for the Kenya Evaluation Association and pay the remainder to individual members who are overseeing things like review of the protocol, review of progress reports, comments on the ®rst draft, and similar activities. The KenEA would probably discuss the evaluation within the association as well. Patel said that a substantial number of organizations in Nairobi have already expressed an interest in collaborating with the Kenya Evaluation Association in this manner. The KenEA sees this as a feasible developing model, at least in Kenya. It is likely that the KenEA would export the model to a number of other countries as well over a period of time. La SocieÂte francËaise de l'EÂvaluation Ð Christina Nirup represented La SocieÂte francËaise de l'EÂvaluation (SFE) in Barbados. Founded in 1999, the general aim of SFE is to contribute to developing the evaluation process and promoting its use in public and private organizations. SFE views the evaluation process as helping to promote the decisionmaking process, modernize the running of public adminis-

trations and services, and improve the effectiveness of public expenditure, as well as extend accountability and practices of internal and external reporting. Among SFE's objectives are the improvement of evaluation techniques and methods, the promotion of compliance with ethical and procedural rules to ensure the quality of evaluations, and the appropriate utilization of results. The SFE intends to be a meeting place and a place for debate and training for all parties involved in evaluation. Malaysian Evaluation Society Ð Aru Rasappan spoke on behalf of the Malaysian Evaluation Society (MES). The idea for the MES was ®rst put forward in 1997, and it was of®cially launched in late 1999. The society will hold its ®rst conference in March 2000. Key objectives of MES are to facilitate and promote the exchange of ideas, experiences, and resources regarding evaluation; represent the evaluation profession at regional and international events and forums; plan and analyze the status and future directions for evaluation research, training, and development in Malaysia and internationally; form strategic alliances with other public and private sector organizations involved in evaluation activities; facilitate the formation and effective operation of a national and international network for information exchange pertaining to evaluation; promote and facilitate research and development efforts and publications related with evaluation; provide assistance and advice on evaluation policy matters at all levels; and represent members on evaluation policy matters affecting them. The planned MES activities include regular meetings and networking between MES members on a support group basis; newsletters; an annual conference; evaluation training programs and joint seminars; promoting evaluation; and publication of research materials and joint activities with both national, regional, and international bodies for the promotion and development of evaluation and related activities. Evaluation in Malaysia is problem-driven, and one of the most pressing problems is budget reform. Seventy percent of the funds dedicated to the government's operational budget are not evaluated. MES hopes to legitimize program evaluation and to in¯uence government policy. An international body would bring together policymakers, bureaucrats, agencies, NGOs, funders, and service providers to build capacity, leverage, legitimization, and in¯uence government policy. A loose coalition does not work for the MES. A strong coalition is needed to help push the agenda and produce tangible change, such as having the government link evaluation to policy and leveraging funding with donors. Programme for Strengthening the Regional Capacity for Evaluation of Rural Poverty Alleviation Projects in Latin America and the Caribbean Ð Known better by its acroynm, PREVAL, the Programme for Strengthening the Regional Capacity for Evaluation of Rural Poverty Alleviation Projects in Latin America and the Caribbean is a program funded by a UN organization (International

A.J. Love, C. Russon / Evaluation and Program Planning 23 (2000) 449±459

Fund for Agricultural Development) and implemented by an international institution based in Costa Rica. The main components of PREVAL are training, technical assistance, applied research, and dissemination. One of PREVAL's main strategies is networking, and it has succeeded in building a network of more than 300 evaluators and institutions from all over Latin America and the Caribbean. The network facilitates the exchange of information, provides reference material, and offers opportunities for collaboration. Through the network PREVAL has developed many activities related to the exchange of information and experiences, including an annual electronic conference. The 1998 electronic conference focused on project evaluation and involved more than 300 participants around the world. Ada Ocampo spoke on behalf of PREVAL. She said an international framework would broaden links. Brazil and Peru want to establish networks, which is dif®cult because of very limited resources, but it would be possible with international help. PREVAL would like to see the international group collaborate with initiatives and partnerships. One of the main problems faced by evaluators in Latin America and the Caribbean is the lack of formal or informal evaluation associations. PREVAL members think they have many things to learn from the already constituted international evaluation associations. PREVAL also has much to offer the established associations, especially its experience, many years of work in project evaluation and monitoring, and knowledge of evaluation practices. PREVAL members think that only through a joint effort will the organization be able to attain its purposes and objectives. Reseau Ruandais de suivi et Evaluation Ð The Rwanda National Monitoring and Evaluation Association (RNMEA) was created in 1999. Its ®rst president, James Mugaju, noted that the association came at the right time to respond to the need to build evaluation capacity, thereby improving the effectiveness and ef®ciency of developing a program or project in Rwanda. At the beginning, RNMEA's main challenge was to ®nd an appropriate framework through which the association could operate. It also had a problem identifying the appropriate strategy to attract members. It was dif®cult to contact people and bring them together to form the association. There were several issues of concern. One is the lack of standardized monitoring and evaluation (M & E) tools accepted by all institutions in Rwanda. A second concern is the lack of M & E specialists in most of the national institutions. A third concern is the limited availability of ®nancial resources. The legal framework for making a national decision of this type takes a very long time in Rwanda. To begin addressing these concerns, some initial objectives set by the association are to de®ne and adopt the evaluation standards for program and project evaluations within the government and other institutions, organize sessions on monitoring evaluation systems in the country,

455

and consolidate the data for the socioeconomic planning and decision making in the country. Sri Lanka Evaluation Association Ð Speaking for the Sri Lanka Evaluation Association (SLEvA), Soma de Silva noted that there is an urgent need for strengthening evaluation skills, setting evaluation standards, and promoting an evaluation culture in Sri Lanka. The founders of the association believed that evaluations have the potential to determine whether planned outcomes and impact are achieved, processes are cost-effective, and interventions have been effective and ef®cient. They also have observed that evaluations in themselves will not serve any purpose unless there is commitment on the part of the political leadership, the bureaucracy, project and program management, and other stakeholders to put the results of such evaluations to practical use. Through advocacy, national evaluation associations can create a space to incorporate such results in projects and programs. In particular, SLEvA members believe that a national evaluation association can make signi®cant contributions in strengthening the role of evaluation in the development processes. The Sri Lanka Evaluation Association was formally inaugurated in 1999 at the Sri Lanka Foundation Institute. Professor M. Karunanayake presented the aims and objectives of the association. The association's overall aim is to improve the theory, practice, and use of evaluation by establishing and promoting ethics and standards in evaluation practice, providing education and training related to evaluation, acting as an advocate for promoting a culture of evaluation, providing forums for discussing ideas, linking members who have similar evaluation interests, and recognizing outstanding contributions to the theory and practice of evaluation. The Sri Lankan Evaluation Association also plans to reach out to Bangladesh and India. United Kingdom Evaluation Society Ð Murray Saunders, the current president of the United Kingdom Evaluation Society (UKES), represented the society in Barbados. The UKES exists to promote and improve the theory, practice, understanding, and utilization of evaluation and its contribution to public knowledge. It was founded in 1994 and has more than 200 members. Membership currently includes evaluation professionals, practitioners, and evaluation commissioners from national and local government, the research community, independent consultancies, and the voluntary sector, representing a range of inquiry ®elds including social services, economic development, education, science and technology, health care management, and policy. The Society builds bridges between the various groups and among the different evaluation communities, providing a forum to consider differences and similarities in the problems they face, for example, intellectual, ethical, and professional standards; principles and practices; the theory, practice, utilization, and methods of evaluation; education, training, and professional development; and the need to

456

A.J. Love, C. Russon / Evaluation and Program Planning 23 (2000) 449±459

represent the interests of those concerned with evaluation to governments and other bodies in the UK and elsewhere. The Society also promotes training events, publishes newsletters, and organizes conferences. At the 1998 Presidents Panel, Nick Tilley, president of UKES, reported that the UKES Council saw many bene®ts in fostering dialogue and collaboration between evaluators in different countries. The Council believed there was virtue in diverse approaches to evaluation and had some concerns about convergence toward a particular model at this stage. The UKES vision is a community of mutually respecting practitioners and theorists who represent a wide range of positions. The mission should be to facilitate exchanges of ideas and the fostering of dialogue across differing perspectives. In terms of organization, UKES would favor a league of national associations to indicate the independence of the member societies. UKES also would favor international meetings, including a regular, fully international conference. To make it more accessible, the conference site would move such that no two consecutive meetings took place on the same continent. 3. The future: three scenarios In this section of the paper, the authors engage in scenario planning about the future of international evaluation. Scenarios are de®ned by Schwartz (1991) as a tool for ordering one's perceptions about alternative future environments in which one's decision might be played out. Alternatively: a set of organized ways for us to dream effectively about our own future. Concretely they resemble a set of stories, either written out or often spoken. However, these stories are built around carefully constructed ªplotsº that make the signi®cant elements of the world scene stand out boldly (p. 4). The authors present three possible scenarios for the future of international evaluation. The driving forces that were identi®ed were level of cooperation, funding, and use of technology. It is variations in these forces that account for differences in the scenarios. Scenario one: back to the future Ð The ®rst scenario is a somewhat bleak picture of an evaluation world that is dominated by one or a few evaluation ªsuperpowersº that project their philosophies and approaches on other regions of the globe. Unlike the days of the Cold War, this domination is less likely to result from a deliberate intent, but rather more from a perception of dominance that ¯ows from de facto monopolies over ®scal, human, and other resources. The evaluation superpowers have large numbers of members representing many spheres of practice, signi®cant expertise in sophisticated evaluation methodologies, graduate and professional training programs, multiple publication and communication avenues, and access to sources of funding

and the latest technology. Despite these real strengths, in this scenario, the large evaluation organizations exhibit a type of superpower myopia with a limited world-view that restricts diversity in evaluation theory and practice. Fueled by low trust and increasing rivalries, these large evaluation organizations tend to fragment the worldwide evaluation community. International cooperation is restricted primarily to conference speaking engagements. Evaluation does not develop a strong voice internationally because resources are focused on domestic concerns. Evaluation in developing countries continues to be undertaken largely by expatriates or by contract evaluators brought in by large donor organizations. By and large, evaluations in developing countries remain at the project level and have limited impact at the program or policy levels. Regional disparities in evaluation capacity increase, and the evaluation associations that form in developing countries attain limited impact. None of the fundamental issues of trust, international standards, or access are addressed in a meaningful way. Scenario two: only in my back yard Ð In this scenario, national evaluation organizations continue to emerge rapidly, particularly in developing countries. Initially formed as local support networks, perhaps focused on a single sector, these organizations become proactive repositories of evaluation expertise and capacity building. Through a combination of informal and formal activities, national evaluation organizations champion the adoption of evaluation standards and high-quality evaluation practices. They become very proactive in establishing professional training and methods of information exchange. They work in partnerships with their members to manage and conduct evaluations under contract. In this second scenario, the bonds of af®liation remain strongest at the national level, but members recognize and support the value of regional evaluation associations. Positive motivations for regional associations include the desire to promote evaluation to politicians and bureaucrats by collective efforts, build evaluation capacity through academic education and professional training, ensure the bene®ts of evaluation to the region's inhabitants, and further the development of evaluation as a multidisciplinary profession through standards and ethical codes. Less positive motivations include the desire to protect territory and limit the in¯uence of evaluation organizations outside the region. Emerging evaluation associations from both more and less developed countries ®rst establish formal national bodies and then link quickly to regional evaluation organizations, such as the African Evaluation Association, the European Evaluation Society, and the Australasian Evaluation Society. New regional organizations emerge in Latin America and the Caribbean, in the former Soviet Union, and in several parts of Asia. A new North American Evaluation Association forms that parallels the North American Free Trade zone and embraces the American, Canadian, and Mexican evaluation interests.

A.J. Love, C. Russon / Evaluation and Program Planning 23 (2000) 449±459

International cooperation takes place primarily at the regional level. Some low resource regional organizations form alliances to strengthen their positions. Established evaluation organizations with more resources assist the less established and emerging organizations in their regions. Evaluators from the more developed countries work more collaboratively with those from the less developed countries. They spend more time in-country and develop better knowledge of social, political, and economic issues than do the ªtwo-weekº contract evaluators. Training and capacity building in each region is less ad hoc and more systematic. The relatively close geographical proximity of the member associations make regional conferences more accessible and of greater interest to members of the national societies. The mounting pressures of globalization and fears of a potential loss of diversity, however, carry the greater risk that each region will protect its evaluation interests, including its theory and practice, from in¯uence by other regions. Some of the consequences are the setting of regional evaluation standards and certi®cation requirements. Driven by these narrow requirements, new evaluators in each region primarily learn the dominant theories of that region, and evaluations will be performed only by evaluators from that region. Links among evaluation societies are likely to include periodic meetings of the representatives of the regional associations and informal contacts by individual members at regional and national conferences. A greater regional focus also brings concerns about equity and access. How will regional evaluation organizations ensure equitable representation in the face of disparities in resources, number of members, organizational development, and regional in¯uence? Who will set the ground rules for the regional structure and determine the agenda? Which language(s) will be spoken and used for communication? Scenario three: the long and winding road Ð In the third scenario, the leaders and members of the national and regional evaluation associations decide to move beyond national and regional boundaries and to walk down the long and winding road of building a worldwide evaluation community. In this scenario, although national and regional organizations still exist as independent entities, they establish formal linkages that permit a high degree of international cooperation. They choose international cooperation because they believe that the best way to cope with globalization is by coming together to guide its future direction, rather than by resisting it in isolation. They decide that even if they are not walking an identical path together, they are venturing in the same direction. Right now, simple formal agreements regarding reciprocal membership and cooperation exist between the American Evaluation Association, the Canadian Evaluation Society, and the Australasian Evaluation Society. These agreements represent the ®rst step by which individual

457

members of each association can participate more broadly in evaluation around the world. In this scenario, the wider group of national and regional evaluation organizations af®rm that international cooperation is the preferred way to secure collective bene®ts. These bene®ts are best obtained through multilateral cooperation, such as raising professional standards and professional identity, technical assistance, international monitoring, moral support to evaluation units for ªpolitically courageous measures,º speaking authoritatively about the experience of evaluation in other countries, increasing the visibility and stature of evaluation, improving the provision of evaluative data, discussion of policy issues and making representations, and responding to the global challenges of the day. International cooperation supplies a new collective resource that largely does not exist now by providing supply-side shared experiences, pooling resources, and capacity building. It also generates demand-side pressure, especially in developing countries, where evaluation capacity is limited and politicians and policymakers do not give much attention to evaluation, and where the existing evaluation resources are largely monopolized by the donor agencies and are not available to the national programs. If this collective in¯uence is successful, although expatriates from developed countries continue to conduct evaluations in developing countries, they would work in collaboration with evaluators indigenous to the country/ region. Moreover, indigenous evaluators would be responsible for more evaluations of their own, either as external or internal evaluators. Another important shift is that evaluations in developed countries begin to be conducted by expatriates from developing countries or in collaboration with evaluators from developing countries. Increased international cooperation and exchange also offers the potential for developing a much broader, worldwide evaluation paradigm instead of a paradigm that comes exclusively from Europe or North America. This opens the opportunity for much wider contributions to evaluation theory and practice, such as oral history Ð something not discussed much in evaluation textbooks. These new developments permit new synergies of evaluation theory and practice. The best of Asia will be combined with the best of Africa, with the best of North America, with the best of Europe. Many new and exciting hybrids will emerge. In this scenario, although the work of the majority of evaluation association members in developed countries continues to have a domestic focus, nonetheless the members vote for increased international exchange. They recognize that many of their own domestic programs now address issues long associated with developing countries: poverty, hunger, illiteracy, homelessness, crime, drugs, communicable disease epidemics, environmental degradation, and migration. In a rapidly shrinking world, they wish to bene®t mutually from exchanges with evaluators in developing countries and learn more about their approaches.

458

A.J. Love, C. Russon / Evaluation and Program Planning 23 (2000) 449±459

Because of the geographic dispersion of the members and the high cost of travel, participation in real time conferences will be limited. To enable the building of the worldwide evaluation community, evaluators make extensive use of leading-edge technology and multiple communication channels to shrink distances and enable close collaboration. In addition to local and national conferences, regional conferences will be scheduled periodically and shared worldwide via videoconferencing and daily proceedings will be posted to the Internet. Language translation technology will facilitate transfer of views in real time. Eventually, teleimmersion will give participants the visual experience of being in the same room with each other, even though they are physically located on different continents. E-newletters will keep evaluators around the world in touch and e-journals will facilitate rapid access to peer-reviewed articles. International evaluation archives will serve as a repository of evaluation knowledge. In the third scenario, evaluation also begins to contribute substantively to solving some of the major problems facing humanity. Evaluators from multiple disciplines and from various evaluation associations around the world form ªcommunities of practiceº that contribute actively to improvements in ®elds such as health care, food systems, international development, sustainable development, human rights, rural development, and education. They work to identify effective interventions and prevent the waste of increasingly scarce resources. They share the ®ndings of their international comparative research. The communities of practice cut across national and regional boundaries and build capacity through academic courses, professional training, collaboration with other evaluators, and work on ¯agship projects of international signi®cance. In turn, partnerships of public, nonpro®t, and private sector organizations serve as incubators to fund these cross-cutting evaluations or to add evaluation expertise to important international projects. In short, in the third scenario, a cooperative relationship among the more established and the less established and emerging national/regional evaluation organizations would facilitate the building of evaluation leadership and capacity in developing countries, foster the cross-fertilization of evaluation theory and practice around the world, and assist the evaluation profession to take a more global approach to contributing to the identi®cation and solution of world problems. 4. Conclusions We hope that these scenarios and our brief account of the recent efforts to build a worldwide evaluation community will generate discussions among the individual members of national evaluation associations about the big questions we face as evaluators in a frantically changing world.

Our experience to date has shown that we must ®nd ways to broaden the discourse about greater international cooperation, communicate the issues to members, involve the broad membership in decisions, and have reasonable assurance that such a major effort, if approved, be justi®ed in terms of the bene®ts. There are major issues that need legitimate and necessary debate: Does greater international cooperation imply a new organizational structure? If so, what would the structure look like? Will representation be equitable or strongly in¯uenced by the established evaluation organizations with more resources? Will individual members be able to participate? Will someone be excluded if his/her country does not have a formal evaluation organization? We think it is possible to embrace international cooperation with denigrating national organizations. In fact we would argue the opposite: it is vitally important for national organizations to be strong yet able to address cross-national and cross-regional problems that affect the entire evaluation profession. But, more importantly, despite the efforts of our national evaluation associations, evaluation will remain one of the world's best kept secrets unless we build strong coalitions that go beyond our own backyards. There is no single blueprint. No one has a monopoly on the truth. Our work is at an early stage, and we are learning as we advance. With courage and determination, we can build a dynamic worldwide evaluation profession for the 21st century. Appendix A American Evaluation Association, http://www.eval.org/ Associazione Italiana de Valuatazione, http://www.valutazione.it/ Australasian Evaluation Society, http://www.aes.asn.au/ Canadian EvaluationSociety, http://www.evaluationcanada.ca/ CES International webpage, http://www.evaluationcanada.ca/devprof.html Deutsche Gesellschaft fur Evaluation, http://www.bal.fal.de/staff/tissen//geproval.htm European Evaluation Society, http://www.europeanevaluation.org/ Malaysian Evaluation Society, http://www.angel®re.com/ab/mes PREVAL, http://www.®damerica.cl/preval.htm Swiss Evaluation Society, http://www.seval.ch/ Sri Lanka Evaluation Association, http://www.nsf.ac.lk/slea/ United Kingdom Evaluation Society, http://www.evaluation.org.uk/ I&CCE TIG, http://www.wmich.edu/evalctr/ICCE/index.html List of evaluation organization presidents, http://www.wmich.edu/evalctr/ICCE/Eorg.htm

A.J. Love, C. Russon / Evaluation and Program Planning 23 (2000) 449±459

Photojournal of Barbados meeting, http://www.wmich.edu/evalctr/ICCE/barbados.htm Presidents Panel 2, http://www.wmich.edu/evalctr/ICCE/pres2.htm Presidents Panel 1, http://www.wmich.edu/evalctr/ICCE/ppp.htm AEA Position Paper, http://www.eval.org/Conferences/1998/Papers/ aeainternatialpres2.html AIV Position Paper, http://www.wmich.edu/evalctr/ICCE/aivpp.htm AES Position Paper, http://www.parklane.com.au/aes/international.htm CES Position Paper, http://www.unites.uqam.ca/ces/ceice.html#IEO MES Position Paper, http://www.wmich.edu/evalctr/ICCE/mespp.htm I&CCE TIG Position Paper, http://www.wmich.edu/evalctr/ICCE/iccepp.htm

459

References Chambers, R. (1997). Whose reality counts? Putting the ®rst last. London: Intermediate Technology Publishers. Derlien, H.-U. (1990). Genesis and structure of evaluation efforts in comparative perspective. In R. C. Rist, Program evaluation and the management of government, London: Transaction Publishers. Joint Committee on Standards for Educational Evaluation (1994). The program evaluation standards, Thousand Oaks, CA: Sage. Russon, C., Love, A. J. (1999). Creating a worldwide evaluation community. Paper #15 in Occasional Paper Series. Kalamazoo, MI: The Evaluation Center, Western Michigan University. Russon, C., & Russon, R. (2000). The annotated bibliography of international programme evaluation, New York: Kluwer Academic. Schwartz, P. (1991). The art of the long view, New York: Currency Doubleday. Wagner, P., & Wollmann, H. (1986). Fluctuations in the development of evaluation research: do regime shifts matter?. International Social Science Journal, 205±218.