- Email: [email protected]
Monitoring and evaluation of forestry projects for local community development
Agricultural Administration 19 (1985) 139-160
Monitoring
and Evaluation of Forestry Projects for Local Community Development Eric L. Hyman Appropriate Technology 1331 H St, NW, Washington,
International, DC, 20005, USA
ABSTRACT Monitoring and evaluation are particularly important in forestry projects for local community development because of the diversity and innovative nature of these projects. These projects, taking place on lands that have not conventionally been usedfor forestry, require new skills oj foresters and involve different participants. Monitoring focuses on a project’s inputs, activities, management and outputs to improve the operation of the project as designed. Evaluations re-examine the design of the project and assess its impacts. Ongoing evaluations support midcourse correc.tions while ex post evaluations address mitigation and follow-up activities in the project area and offer lessonsfor the design of other projects. The early experience with monitoring and evaluation of rural development projects indicates that the,findings have not been well utilized. The main reasons for this include resistance by project management and field staff, problems in identifying data requirements, mismatched frequency or timing of reports, use of inappropriate data collection and analysis methods, the location and organization of monitoring and evaluation units, budgeting problems and cost over-runs and inadequate attention to the presentation and communication oj findings and the appraisal of the monitoring and evaluation system itself.
INTRODUCTION This paper monitoring
provides an overview of the nature and purposes of and evaluation (M/E) in forestry projects for local
139 Agricultural Administration 0309-586X/85/$03.30 Ltd, England,
1985. Printed
in Great Britain
0 Elsevier Applied
Science Publishers
Households of the landless poor or people displaced by government projects Co-operatives or farmer associationsa
Village Forest Department
Resettlement
Village woodlots
units
Settled households Co-operatives or farmer associations”
Decision-making
Farm forestry (Smallholder treefarming)
Types of prqjects
Five Common
Village or communal lands
Leased or newly titled government lands or land reform areas Degraded public lands Unused private land Corporate land
Labour
sources
Projects
Hired casual labour Contractual labour
Household labour Co-operatives or farmer associations Shared or exchanged labour from neighbours
Household labour Casual, hired labour Contractual labour Co-operatives or farmer associations Shared or exchanged labour from neighbours
Treefarming
Blocks of land owned or rented by farmers Around homesteads, farm bunds, and land boundaries Unused, private land
Land sources
TABLE 1 Types of Social Forestry of the output
Villages Participating
workers
Households Co-operatives or farmer associationsb
Householdsb
Distribution
Forest Department Non-governmental service organizations
Reforestation
Government lands Public forests Degraded public land Watersheds and tank foreshores Unused, private land
Roadsides Railway rights-of-way Canal and river banks
Degraded public lands Schools, hospitals, and other local institutions Unused, private land
Forest Department Participating workers The landless poor’
Forest Department Hired, casual labour Contractual labour Voluntary, unpaid labour
z
or corporations.
Forest Department Public Works Department Railroad companies Participating workers Villages Schools and other public facilities The landless poor’
Hired, casual labour Contractual labour Forest Department Public Works Department
banks;
Schools, hospitals and other local institutions The poor Non-governmental service organizations
Village co-operatives Voluntary labour Forest Department Schoolchildren
(1Subject to constraints imposed by Forest Department and other government agencies; development b May be partial or full recovery of costs of free inputs, land, or loan capital. ’ Harvesting may be deferred to preserve watersheds, preserve shade, or retain ornamental value.
Forest Department Public Works Department (including Highway and Irrigation Departments) Railroad companies Non-governmental service organizations
Strip plantings
Schools, hospitals, and other local institutions Non-governmental service organizations
142
Eric L. Hyman
community development (FLCD). It also draws from the experienceof using M/E in other types of rural development projects to identify opportunities and constraints in monitoring and evaluating FLCD projects. The purpose of monitoring is to improve the operation of the project as designed. Monitoring focuseson the inputs, activities, management and outputs of a project. In contrast, evaluationscan help decision makers reexamine the design of the project and assessits impacts. In the past, many large development projects lacked systematic monitoring. Forest Departments kept recordson the number of seedlings produced and the locations of planting or distribution, but rarely on survival rates or yields. As a result, some projects that could have been salvaged failed and successful projects could have done even better. Evaluations often have been carried out at project completion before many impacts were ready for investigation. Sinceevaluations were geared to the needs of donors, many countries viewed them as a burdensome requirement. Now, however, there is an increasing awarenessby donors and governments of the importance of making M/E more systematic and integrated within the planning and decision-making process. Monitoring and evaluation are especiallyimportant for FLCD projects which are innovative and diverse, taking place on lands that have not conventionally beenused in forestry (seeTable l), requiring new skills of foresters and involving participants with differing interests,resourcesand abilities. Of course, uncertainty is unavoidable in all planning for the future. Things outside the control of the project with natural or human causescan affect its chances of success.Some examples are droughts, typhoons, pests and diseases,grazing animals, market prices, political instability and the availability of chemicals, machinery, or spare parts. Although monitoring cannot change some of these, it can suggest measuresto reduce their harmful effects. For example, plantings can be delayed if monsoons are late and thorny hedgescan be added around village woodlots if the local people do not control grazing animals. Furthermore, FLCD projects place a high priority on socio-economic objectives. Rural development activities can have major direct impacts that are difficult to foresee; for example, changesin the distribution of income, employment, the social well-being of women and the landless poor, the ability of individuals and institutions to participate in development, and environmental quality. Even within a country, two similar projects may have different impacts. ’ 7
Monitoring
OBJECTIVES
and ecaluation
OF MONITORING
of forestry
AND
projects
143
EVALUATION
Monitoring is a continuous process of collecting, analysing and communicating information on project operation and performance. It takes place during project implementation and is intended to assist project managers in making decisions on whether changes are necessary in the way the project, as designed, is being implemented. Since a ‘welldesigned project’ with the necessary inputs, technical resources and support services can fail as a result of problems in operation and management, monitoring is important to indicate whether the scale, location and timing of project activities are on schedule and are being efficiently carried out. A focus on planned targets for inputs, activities and outputs encourages a rigorous approach to monitoring. However, targets established in advance of project implementation can be arbitrary. If the targets were too high or too low, a project might appear more or less successful than it really is. In fact, the appropriate targets may evolve over time in response to changes in conditions or as a result of the project itself. For example, adhering to targets for supplying pine seedlings for pulpwood may be counterproductive if there is a greater demand for fuelwood species or if changes in market conditions favour hardwoods for veneer production. Thus, non-attainment of a target does not necessarily indicate a failure. What is more important is understanding the reasons for accomplishments and shortfalls. Evaluations examine the ejjkctiveness of the project’s design, estimate the production and distribution of outputs and assess the impacts. Evaluations are concerned with some of the same things as monitoring, but the timing and purpose of the analyses are different. There are two types of evaluation: ongoing and cx post. Ongoing evaluations take place during project implementation either routinely or conditionally when difficulties have arisen or more aid is required. An ongoing evaluation can be useful in making decisions on changing the strategies, techniques, institutional arrangements and resource allocations for the project or government policies affecting forestry. A secondary purpose of ongoing evaluations is to stimulate communication among project staff, project management, the local people and outside organizations. Ex post evaluations occur after completion of the project. They rely on the findings of monitoring and ongoing evaluation, as well as additional
144
Eric L. Hyman
information on subsequent impacts. Since timber, pulpwood and tree fruits can take many years to grow, it often is premature to measure outputs and impacts during the course of the project. Because FLCD projects are relatively new, only a few have been around long enough for their impacts to be assessed.An ex post evaluation can be useful in deciding whether to compensatepeople adversely affected by the project or to mitigate unintended environmental impacts. It can indicate whether the technologies adopted and the investments made under the project are likely to continue-or even spread to other places-without external assistanceor whether follow-up projects in the same or other areas are needed.Expost evaluations also create an information basefor assessing development in the future and can help appraise the prospects for complementary projects, such as pulpmills or wood-based cottage industries, that increasethe value added to projects’ outputs. Finally, ex post evaluations inform decision makers in other provinces, countries, or international agenciesabout the benefitsand costs of the project and the lessons learned in its experience. This information may be useful in designing similar projects elsewhere. An overemphasis on the achievement of targets in an evaluation can divert attention from the measurement of impacts. There are four main difficulties in analysing impacts. First, there are time lags between the production of outputs and the generationof impacts. Income distribution effectsare hard to trace.18,21Furthermore, impacts on health, education, social well-being, and the capacity to participate in development may not be obvious until many years after the increase in incomes. Secondly, without baseline studies of the initial social, economic and environmental conditions in an area before implementation of the project, it is difficult to tell what changeshave occurred. Baselinestudies have been conducted for only a few FLCD projects. Without baseline data, it still may be possible to make some inferences about impacts by studying places that are similar to the project area in all other respects except the absenceof the project. Yet, conditions change over time in developing areasfor many reasons,so it may not be possible to attribute particular changes to the effects of any one project. Thirdly, development projects may have unintended effects on some groups. One of the main reasons for the failure of forestry projects on government or village lands is neglect of the prior users of the land.” When the interests of theseaffected personsare not taken into account, they may graze animals on the planted seedlings;burn stands in order to
Monitoring
and evaluation
of forestry
projects
145
plant crops or re-establish grasses; cut trees prematurely for fuelwood; or take part in violence. Sometimes, land tenure problems are not apparent in advance because the actual tenure differs from the legal status of the land. For example, the amount of land available for a village woodlot programme in Pakistan turned out to be limited because adjoining large landholders had appropriated communal land, even though their ownership is not recognized in land title records or tax ledgers.4 Fourthly, since environmental impacts may be dispersed over a large area and are influenced by many other natural factors and human activities, it can be difficult to isolate their causes.24 After the physical, chemical and biological impacts have been estimated, the next step is to place values on them. Considerable progress has been made in developing these methods, but they are still imperfect.‘2~‘5~‘g Most methods tend to underestimate environmental values and do not handle cumulative or indirect effects well.
I
I
Ongoing
.
I
evaluation
I
4 Generation of early impacts
i i i
-A
I Completion of project implementation
i i i
I Generation of subsequent impacts Fig. 1.
Typical
elements of a project
Ex post evaluation -.-. addressed in monitoring
and evaluation.
i i
Eric L. Hyman
146
Figure 1 summarizes the main elements of M/E. The following example illustrates their differences in emphasis. Suppose that a farm forestry project has been designed to counter deforestation. Monitoring may show that the government nurseries are meeting the targets for seedling production. An ongoing evaluation may identify lack of extension services for landowners under the project as the actual constraint to the participation of smallholders. An ex post evaluation would estimate the costs and benefits of the project compared with alternatives such as providing public lands to slash-and-burn cultivators for agroforestry, or promoting more efficient woodstoves. The Appendix lists some examples of questions that might be relevant in the M/E of FLCD projects.
LESSONS
FROM
THE
EARLY
EXPERIENCE
WITH
M/E
The early experience with M/E of agricultural development projects shows that the findings have not been well utilized. This section discusses the constraints and opportunities surrounding the use of M/E findings. Resistance to M/E Sometimes project management or staff resist the idea of monitoring and evaluation because M/E can expose weaknesses in decision making or implementation.6 People dislike scrutiny of the efficiency and effectiveness of their work and possible inquiries about their judgment, sincerity, or honesty. Evaluation findings often appear critical, particularly if claims for the project have been overstated.’ These fears can be reduced if project managers are reassured about the objectivity and fairness of the individuals carrying out the analyses. Project managers may view M/E as burdensome or trivial requirements if they are confused about what M/E are supposed to do. Thus, project managers need to be convinced that the findings will provide useful and timely information that can be integrated into the planning and decision-making process. Project staff may resent M/E if they perceive that it will increase their workload or burden them with paperwork. New M/E tasks imposed on field staff can detract from day-to-day implementation duties or place more time pressures on them. Even when M/E reports only require a small amount of time, field staff might not see the importance of the activity or may be reluctant to submit uncertain numbers that someone might
Monitoring
and
evaluation
of forestry
projects
147
question later. Furthermore, project staff may be unwilling to undertake critical self-reflection. It takes time to change staff attitudes whenever there is a new emphasis in project implementation. Staff can be motivated if the rationale for the M/E procedures and the practical importance of the information to be collected are explained to them and the chain of responsibilities and system of incentives and disincentives functions well. Identification
of data requirements
and indicators
A common problem with M/E systems is that they often collect too much, too little, or the wrong type of data. As a result, monitoring and ongoing evaluation reports often are produced too late to be useful to project management. If project management is not involved in the design of the information requirements, some of the data may be irrelevant to the needs of decision makers. Data collection and analysis are expensive and time-consuming. Further, since the capacity of users to absorb information is limited, too much information can be confusing. Some evaluations have relied on surveys without carefully considering how the sample size affects the amount of data generated. When the sample size is too large, an evaluation may cost too much for the amount of usable information obtained. If the sample size is too small, it might not be possible to carry out statistical tests or stratify the findings by socio-economic groups or site qualities and sizes. The determination of an efficient sample size and selection of respondents for the sample are discussed elsewhere.3s22 One way to avoid the pitfalls of collecting too much or too little data is for M/E staff and project management to co-operate in developing a flexible plan for data collection and analysis. This plan can help the M/E staff avoid omitting important data or collecting data that are unnecessary, or at the wrong level of aggregation, and can facilitate the interpretability of the data. Frequency
and timing of reports
The appropriate frequency and timing of monitoring and ongoing evaluation reports depend on the timetable of decisions that project managers have to make. Field staff are more likely to feel burdened if monitoring and ongoing evaluation reports are required at too frequent intervals. Due to donor requirements and the schedule for planning
148
Eric L. Hyman
follow-on projects, ex post evaluations often are carried out shortly after project implementation is completed, eventhough this may be too soon to observe the full range of outputs and impacts. Data collection
methods
A mixture of data collection approachesmay bechosento suit data needs, the budget, and staff skills and to provide cross-checksfor key indicators. Monitoring can be accomplished through the use of administrative records, field staff reports and ‘rapid reconnaissance’ approaches. Administrative records document financial expenditures, staff allotments, the purchase of supplies, the use of equipment and hiring of services. Standardized field staff reports may cover such things as the recruitment of participants, the provision of seedlings,the number and location of treesplanted or woodstovesdistributed and seedlingmortality rates. Rapid reconnaissancemethods include quick and inexpensive site visits, interviews, or group meetings emphasizing the use of direct exploratory observation and guided techniques for asking questions.’ In general,lengthy special studies are not necessaryfor routine monitoring. Ongoing and ex post evaluations build on the information gatheredin monitoring to keep costs down and minimize duplication of effort. Since an evaluation may be concernedwith growth of trees or yield of fruits or fodder, some physical measurementsmight be necessary. In some cases, surveys, interviews and workshops may be needed. Surveys are most useful in finding out about socio-economic impacts and the opinions of local people.*,I6 However, surveys can be expensiveand time-consuming, especially if the sample sizeis large. Interviews are more free-ranging than surveysand are aimed at knowledgeable or influential people, rather than the general public. Sometimes, the values and interests of these individuals may differ from those of the rural poor. Workshops are best suited for drawing out information on facts, values and the perceptions of experts.7 An exyost evaluation sometimesincludes more detailed, specialstudies of longer duration-case studies, experiments and cost-benefit analyses to gain some understanding of a project’s cause-effect relationships and impacts. Case studies are in-depth examinations of relatively small representative groups of people, limited geographic areas, or particular aspectsof a problem. Case studies can yield valuable insights, although they require intensive efforts by skilled analysts, and it can be difficult to
Monitoring
and ecaluation
of jbrcstry
projects
149
make generalizations from one case to another. Experiments test hypotheses about technical questions such as the relationship between yields and factors like speciesselection, tree spacings, tree establishment techniques,fertilizer use, or irrigation. Cost--benefit analysesestimate the net economic gains from the project to society and to representative participants.‘,’ 3 The choice of data collection methods and their applicability under various conditions are discussed in detail elsewhere.3,‘0 Problems
in data collection
Data collection can be difficult as a result of poorly developed transportation and communications infrastructure in remote, rural areas and the hard conditions facing M/E staff in the field. There is a danger of ‘rural development tourism’ where M/E staff are steeredto areas where the most successful activities are underway, accessibility is easy, the people are relatively better off and local officials have support.5 Conditions may be very different in the uplands, away from the roadside, acrossthe river, or when the weather is lesscomfortable. Short field visits might not provide enough information about seasonal or periodic variations or local cultural differences. A participatory approach which fosters a dialogue between the local people and M/E staff is necessary to obtain information on social impacts. However, this can be difficult to achieve because of: (a) the unwillingness of the rural poor to speakopenly for fear of retaliation; (b) embarrassment over weak articulation skills; (c) communication failures or errors when there are distinct, local languages; (d) the limited experience of project staff in establishing a close dialogue on an equal basis with the rural poor; (e) the domination of discussion by a few individuals and (f) drifting into tangential issues.14In addition, the sociopolitical situation in some countries can make participatory M/E infeasible. In many cases,too many data are collected from influential peoplewith relatively high incomes and educational levelsor powerful social contacts. Certain ethnic, sex,or agegroups frequently receivemore than their share of attention. As a result, tribal and nomadic people, women, residentsof remote areas and the disadvantaged social groups that are lessvocal are under-represented.23
1.50
Eric L. Hyman
The ways in which questionsare askedcan affect the findings. A leading question, such as, ‘Don’t you agreethat the Forest Department has done an excellentjob in promoting social forestry? is an obvious example, but many casesare more subtle, resulting from the relationship betweenthe M/E staff member and the respondent. One common problem is that people may suspect that the M/E staff may be tax collectors, forest guards, or land registry officials. Or, people may feel that they cannot speak freely without jeopardizing goods or servicesthat they receive from the Forest Department or their personal relationships with extension workers and village-level technicians. Cooperation can be improved if the reasonfor monitoring or evaluation are explained to the people. Hypothetical questions, such as, ‘Would you be willing to take up tree farming if you owned some land? are difficult for respondents. Respondentsalso may have difficulty remembering numbers or activities that took place more than a few months before. Since local measurement units may be imprecise or subject to variation, M/E staff may have to make some physical measurements of wood yields and use rates. Sometimes, respondents have incentives to provide incorrect answers. These incentives may be obvious for questions about income or when people have settled in public forests illegally. However, the causemay be more subtle, such as when people overstate their fuelwood consumption becausethey expectthat the information will be usedto setrestrictions on fuelwood collection. Good survey and interview procedurescan minimize many of these problems. Data processing and analysis
Bottlenecks in data processing and analysis have prevented the timely releaseof M/E findings in many cases.This problem is often causedby the collection of too much data or a premature reliance on computerized data processing. Computerization has caused delays in the M/E of some projects due to shortages of skilled staff, limited accessto mainframe computers, mechanical breakdowns and cost overruns. Minicomputers are less subject to these disadvantages because they are relatively inexpensive and easy to use and repair. Computerization has advantageswhen there are a large number of casesor variables for eachcaseand the input data formats areprecededto
Monitoring
and evaluation
of forestry
projects
151
streamline data entry. Computerization allows the data to be rearranged at different levels of aggregation to meet the requirements of various users. For instance, local project managers may need statistics at the village level while state or national officials may want the data combined at a higher level. In addition, certain statistical tests are feasible only on a computer. Nevertheless, if an FLCD project is small, personnel with computer experience are unavailable, or only simple arithmetic is required, hand calculators may be all that is necessary. Location
and organization
of M/E units
Since monitoring differs from evaluation in its purposes, scope and location and organizational re-, potential users, the appropriate sponsibilities for these two activities may differ. The early experience indicates that it is preferable for monitoring to be carried out internally by the project-implementing agency with external assistance where necessary. In contrast, evaluations tend to be better if they are externalcarried out by outside organizations such as provincial or national planning or finance ministries, government departments of evaluation, universities, research institutes, or consulting firms. The main reasons for this include: (i) the availability of expertise, (ii) the accountability and objectively of the M/E staff and (iii) the likelihood of timely results. Since organizations and the conditions under which they operate vary a great deal across and within countries, the appropriate location and organization of M/E units may differ for specific cases. Most Forest Departments have the expertise to monitor the achievement of silvicultural targets and the reasons for accomplishments or shortfalls. However, few foresters have expertise in social science theories and methods or in research design for surveys, interviews and case studies needed to waluatc impact. Unless it is feasible to hire, retain and promote good social scientists within the Forest Department or provide extensive social science training to foresters currently on the staff, it may be necessary to turn to an external organization for evaluations. Since monitoring is designed to meet the needs of project decision makers, an internal unit has a stronger motivation than an external agency for making monitoring a tool to increase accountability. Working in-house, monitoring staff have a better opportunity to demonstrate their competence and integrity to project management.’ It also may be easier
152
Eric L. Hyman
for an internal monitoring unit to gain the ear of decision makers informally. On the other hand, evaluations require a broader view of the structural factors in an economy and a society that explain people’s behaviour. Internal evaluations tend to focus too narrowly on the competence or integrity of particular individuals on the project staff.’ In addition, an external viewpoint may be necessary in order to question the basic premises of a project, Nevertheless, the project-implementing agency must have confidencein external organizations conducting an evaluation. The objectivity of M/E units also is a concern. In some countries, achievement reports from field staff tend to conform to targets all the time, regardless of actual conditions.20 This is less of a problem for monitoring when measurable targets can be verified by field staff supervisors or the staff of central monitoring units. In contrast, evaluations involve more subjective issues. An internal evaluation is lesslikely to be successfulbecausefew staff memberswill rate their own performance-or that of their colleagues-impartially. Lack of objectivity is especiallya problem when staff perceivethat salary increases or promotions depend on the evaluations. The desire to avoid being criticized can be an evenstronger motivation in many cultures. However, outsiders also may look out for their own interests. For example, consultants or research institutes might produce overly favourable reports to curry favour with clients, or they might cast their criticisms in such a way as to attract follow-up work. An internal monitoring system is more likely to be aware of the time pressuresfacing decision makers. If a good plan for the collection and analysis of data is prepared, monitoring should not create a time burden for project field staff. Ongoing evaluations are more intensive efforts that may require larger blocks of time during implementation than project field staff can spare and this may necessitatethe creation of a separate internal evaluation unit. Another alternative is to subcontract out certain tasks, but this tactic can backfire if one slow organization holds up the sequential tasks of other organizations. Some consulting firms have a good record for producing documents on time, but some academics in universities and researchinstitutes are unaccustomed to working under time pressure.There is often less time pressurefor the results of ex post evaluations.
Continuity of staff is important for both monitoring and ongoing evaluation; otherwise, efforts eventually may slow down or grind to a
Monitoring
and evaluation
of forestry
projects
153
halt, or the techniques and assumptions behind the collection and analysis of data may change over time, causing problems in comparability. Continuity also may help to build an atmosphere of confidenceand trust among M/E staff, field staff and participants. Continuity is likely to be a problem for internal monitoring units where jobs are filled by temporary employees or carry low pay and prestige. In some projects, junior staff have been transferred out of monitoring units into management after receiving training or demonstrating their competence. Lack of continuity can be a problem if too many external organizations are involved in an evaluation. Budgeting
and cost over-runs
Cost is a major factor in determining the amount of information that can be obtained in monitoring and evaluation. Cost also affects the selection of methods for data collection and analysis, as well as the location and organization of M/E units. A budget constraint can force M/E to focus on just the basic essentialinformation. However, the findings of monitoring and ongoing evaluations have not been timely in some caseswhen the budget was too small or there were delays in disbursements of funds. Another possible result of an insufficient budget for M/E can becost overruns which may reducethe confidenceof decision makers in the M/E staff. A M/E system should be designed so that the potential benefits outweigh the costs. The potential benefits stem from improvements in the efficiency or effectivenessof the project and other projects. The benefits are most likely to be high when a project is large, complex, or innovative or when a lot of uncertainty surrounds the project. However, the benefits will be low if M/E findings are not timely or useful to decision makers or if the donor or project management is inflexible about changing the design or implementation of the project. The costs of monitoring can be reduced by relying on project field staff and managers. Sometimes, subcontracting out specific tasks in an evaluation can be cost-effective. However, a heavy reliance on consulting firms or expatriatescan be expensive.In someplaces,local universitiesand research institutes have proved less costly since they are non-profit making and low wages are paid to student assistants who may be enthusiastic and hard-working. However, it is difficult to generalize because the costs, level of skills and motivation within different organizations vary.
154
Presentation
Eric L. Hyman
and use of findings
Some M/E reports have not influenced decision makers because little attention was paid to the presentation and communication of M/E findings. Decision makers require concise and understandable reports that highlight the pros and cons of each option. Since averages can obscure the variation in the data, decision makers needto be given some senseof the distribution of the findings. Informal communication is important, too, becausedelivering adverse findings to decision makers in public or without advance warning can causethem to react defensively. Small, private meetings that include top decision makers, key subordinates and M/E staff may facilitate good communication. In presenting the findings, sensitivity analysis can be used to test the effects that value judgments or key assumptions have on the conclusions of the analysis. Supposethat two decision makers disagreeon whether the income distribution objective is twice or three times as important as the economic growth objective. A sensitivity analysis might show that this differenceof opinion might not affect the decisionsthat are on the agenda. Presenting the findings to the local people will publicize the project’s achievements and may contribute to the search for short-run and longrun solutions to the problems encountered. Action is more likely to be taken by project management, local groups, or other institutions if a public commitment is made verbally in an open forum and then followed up in writing. Periodical
appraisal
of the performance
of the M/E system
Since monitoring and evaluation have specific functions and are not just meant to contribute to basic research, periodic appraisal of the performance of the M/E system is desirable. Project management, other government agencies and international assistanceagenciesall have an interest in ensuring that M/E are on track. Mid-course corrections may prove necessaryin the design or implementation of the M/E system. CONCLUSION Monitoring and evaluation can be useful tools for improving the implementation of a project and designing follow-up schemes in the
Monitoring and evaluation of forestry projects
155
project area or elsewhere. The full potential of M/E has sometimes not been achieved because of (a) confusion on the part of decision makers about what M/E are supposed to do; (b) vagueness or conflict among planned project objectives or targets; (c) undesirable rigidities in adherence to targets for inputs, activities and outputs; (d) the difficulty of measuring impacts; (e) an insufficient or overblown scope of the analysis; (f) problems with the organizational set up for monitoring or evaluation and (g) inadequate attention to the presentation and communication of findings.
ACKNOWLEDGEMENT Support for this paper was provided by FAO’s Forestry Policy and Planning Service. The opinions expressed are those of the author and are not necessarily those of FAO. The author would like to acknowledge the helpful comments of J. E. M. Arnold, Arnold0 Contreras and Philip Wardle (FAO); Tom Catterson (US AID), Roger Slade and Raymond Noronha (IBRD); J. G. Campbell (UNDP) and Robert Chambers (Ford Foundation).
REFERENCES 1. Caro, F. (Ed.) Evaluation research: An overview. In: Readings in evaluation research, New York, Russell Sage Foundation, 197 I. 2. Carruthers, I. and Chambers, R., Rapid appraisal for rural development, Agricultural Administration, 8 (198 l), 407-22. 3. Casley, D. and Lury, D., A handbook on monitoring and evaluation of agriculture and rural development projects, Washington, DC, World Bank, 1981. 4. Cernea, M., Land tenure systems and social implications of forestry development programs, Staff Working Paper No. 452, Washington, DC, World Bank, 1981. 5. Chambers, R., Shortcut methods in information gathering for rural development projects, Sussex, Institute of Development Studies, University of Sussex, 1979. 6. Clayton, E., Role characteristics and operational features of agricultural monitoring systems. In: Monitoring systems for agricultural and rural development projects. (Clayton, E. and Petry, F. (Eds)), Rome, FAO, 198 1.
Eric L. Hyman
156
7. Delbecq, E., Van de Ven, A. and Gustafson, D., Group techniques for program planning, Glenview, Illinois, Scott Foresman and Co., 1975. 8. FAO, Wood fuel surveys. Report GCP/INT/365/SWE, Rome, FAO, 1983. 9. Gregersen, H. and Contreras, A., Economic analysis of forestry projects. Forestry Paper No. 17. Rome, FAO, 1979. 10. Honadle, G., Rapid reconnaissance for development administration: Mapping and moulding organizational landscapes, World Development, 10 (1982) 633-50. 11. Hoskins, M., Benefits foregone as a major issue for FLCD success. In: Proceedings
Washington, 12. Hufschmidt,
of
the
U.S. AID
Workshop
DC, US AID, 1982. M., James, D., Meister,
on Community
Forestry,
A., Bower, B. and Dixon,
J.,
Environment, natural systems and development: An economic valuation guide, Baltimore, Maryland, Johns Hopkins University Press, 1983. 13. Hufschmidt, M. and Hyman, E., Economic approaches to natural resource and environmental quality analysis, Dublin, Tycooly Press, 1982. 14. Huizer, G., Preliminary guidelinesforparticipation monitoring and ongoing evaluation of people’s participation projects, Rome, FAO, 1982.
15. Hyman, E., The valuation of extramarket benefits and costs in environmental impact assessment, Environmental Impact Assessment Review, 2 (1981), 227-58. 16. Hyman, E., How to conduct a rural energy survey in a developing country, Renewable Sources of Energy, 1 (1983) 137-49. 17. Hyman, E., Smallholder treefarming in the Philippines: A comparison of two credit programs, Unasylva, 35 (139) (1983) 25-3 1. 18. McArthur, J., Monitoring the impact of agricultural development projects on low-income groups: Principles and problems. In: Monitoring systemsfor agricultural and rural developmentprojects (Clayton, E. and Petry, F. (Eds)), Rome, FAO, 1981. 19. Nichols, R. and Hyman, E., An evaluation of environmental assessment methods, Journal of the Water Resources Planning and Management Division, American Society of Civil Engineers, 108, No. WRl (March, 1982) 87-105. 20. Palin, D., Management of development forestry, Report No. GCP/RAS 46 (SWE), Rome, FAO. 21. Schuster, E., Economic impact analysis of forestry projects: A guide to evaluation of distributional consequences. In: Economic analysis offorestry projects: Readings (Gregersen, H. (Ed.)), Forestry Paper No. 17, Supplement 2, Rome FAO, 1980. 22. Slade, R. and Feder, G., The monitoring and evaluation of training and visit extension in India: A manual of instruction, Washington, DC, World Bank, 1981. 23. Van Heck, B., Research guidelines for field action projects to promote the participation of the poor in rural organizations, Rome, FAO, 1979. 24. Zimmerman, R., Environmental impacts of forestry: Guidelines for its assessment in developing countries, Rome, FAO, 1983.
Monitoring
APPENDIX:
and evaluation
offorestry
prqjc>cts
157
EXAMPLES OF QUESTIONS FOR MONITORING AND EVALUATING FLCD PROJECTS
Inputs How much land is being devoted to various components of the project? Are the type and intensity of recommended forestry practices matched to the quality of the sites? Are the sites for tree planting being abandoned or converted to agricultural uses? Have large landholders or the landless poor appropriated communal lands intended for the project? Is labour for the project scarce in certain seasons or locations? Are co-operatives and shared, or voluntary labour arrangements working out after the initial period of enthusiasm? Is the right amount of labour time being devoted to tree establishment, mortality replacement, maintenance and harvesting? Is labour given sufficient incentives and supervision? Are plastic bags or other supplies in short supply? If the Forest Department is providing the labour for establishing village woodlots, do the villagers take care of these stands as their own or do they perceive them to be the property and responsibility of the Forest Department? Are the intended beneficiaries participating in the project? Are there unofficial charges for subsidized inputs? Is enough credit at satisfactory terms available to cover the initial costs of investments such as starting a cottage industry or to meet the subsistence needs of treefarmers during the production period? What is the effect of inflation on the adequacy of the loan size over time? Are loan funds being devoted to the agreed purposes or diverted to other uses? Are the right kinds and amounts of fertilizer being applied through the proper techniques? Is the system able to detect the need for chemical or biological pest control and provide it in a timely, safe and effective manner? Do participants have access to appropriate equipment and tools? Is irrigation water efficiently used? Are the project sites sufficiently accessible? Intcrmediatc Outputs Does the supply of seedlings meet the demand and is planting available where needed and at the right time of the year?
stock
158
Eric L. Hyman
Is the age, size and vigour of the seedlings satisfactory? Have the right species of trees been selected for the kinds of sites available and the purposes desired by the intended beneficiaries? Are seedlings being transported successfully? Are seedlings being wasted through improper planting techniques or lack of follow-up care? What are the seedling mortality rates after planting? Are improved woodstoves (or charcoal kilns) properly constructed, maintained, and used? Are the designs of improved woodstoves compatible with cultural preferences (e.g. cost, safety, convenience in desired cooking methods, amount of smoke, speed of use, size and height, cleanliness, kinds of fuels and pots and pans that can be used, aesthetics and symbolic or ceremonial values)? Are the designs of improved charcoal kilns compatible with the preferences of charcoal makers (e.g. cost, safety, quality of output, speed, convenience, capacity and durability)?
Operation and Management Activities Is the project helping to build local institutions or is this constrained by (1) lack of local leadership, (2) domination by elites or (3) factionalism? Have new organizations (co-operatives, associations, village groups, and private voluntary groups) developed under the project continued to thrive after the initial period of establishment? To what extent are the local people allowed to participate in decision making and share the benefits of membership in these organizations? Is there sufficient co-ordination between the various government agencies directly or indirectly involved in the project? Is the project adversely affected by personnel problems or budget constraints and delays in disbursements? Are project staff technically up to date in forestry and related fields such as agriculture, livestock management and soil and water conservation? Do project staff have the communications and inter-personal skills necessary for establishing and maintaining a two-way dialogue with participants? Are staff training programmes timely, successful and sufficiently broad based? Are the skills learned in trainingprogrammes applied on the job or does training lead to promotions to other types of jobs? Are the outreach and mobilization activities informing the intended beneficiaries about the existence and terms of the project?
Monitoring
and evaluation
offorestry
prqjects
159
Is participation in the project limited by cultural factors such as attitudes toward debt or administrative problems such as red tape, rigid eligibility requirements, or conflicts between FLCD activities and law enforcement responsibilities of the Forest Department? Do forestry laws and policies have unintended adverse effects on people’s decisions to participate in the project or the potential profits from participation (e.g. a log export ban, price controls on forest products, or the ability of the government to expropriate land)? How high is the drop-out rate among participants? Do extension workers spend a disproportionate share of their time with relatively well-off treefarmers, rather than poor people in more remote areas? Do the extension services enable people to solve their own problems and help their neighbours? Are the participants who received extension services following the recommended practices? Are there incentives for retaining the most capable staff members in field positions? outputs
How much timber, pulpwood, fuelwood, polewood, and construction wood has been produced? What kind, quality and quantity of non-wood outputs have been produced ? How much wood has been saved through the adoption of improved woodstoves, charcoal kilns, or wood preservation? Are there bottlenecks in harvesting or transporting outputs? Are exploitative arrangements occurring in harvesting or marketing contracts, forward selling, mortgaging, or selective buying of only the best portion of the output? Are output prices fair? Can domestic markets absorb the increased output of raw materials or cottage industry products without a decline in the real prices of the outputs? Have export markets been developed? Impacts Have the prior users of the land been brought into the project or compensated for their losses? Are speculators or local elites gaining control of project sites through unfair practices?
160
Eric L. Hyman
How many person-days of employment are being generated by the project? How much income (net profits) is the project producing? Who is bearing the costs and who is receiving the benefits, of the project? What are the impacts of the project on women and disadvantagedethnic groups? What are the environmental impacts of the project on- and off-site?
Monitoring
and Evaluation of Forestry Projects for Local Community Development Eric L. Hyman Appropriate Technology 1331 H St, NW, Washington,
International, DC, 20005, USA
ABSTRACT Monitoring and evaluation are particularly important in forestry projects for local community development because of the diversity and innovative nature of these projects. These projects, taking place on lands that have not conventionally been usedfor forestry, require new skills oj foresters and involve different participants. Monitoring focuses on a project’s inputs, activities, management and outputs to improve the operation of the project as designed. Evaluations re-examine the design of the project and assess its impacts. Ongoing evaluations support midcourse correc.tions while ex post evaluations address mitigation and follow-up activities in the project area and offer lessonsfor the design of other projects. The early experience with monitoring and evaluation of rural development projects indicates that the,findings have not been well utilized. The main reasons for this include resistance by project management and field staff, problems in identifying data requirements, mismatched frequency or timing of reports, use of inappropriate data collection and analysis methods, the location and organization of monitoring and evaluation units, budgeting problems and cost over-runs and inadequate attention to the presentation and communication oj findings and the appraisal of the monitoring and evaluation system itself.
INTRODUCTION This paper monitoring
provides an overview of the nature and purposes of and evaluation (M/E) in forestry projects for local
139 Agricultural Administration 0309-586X/85/$03.30 Ltd, England,
1985. Printed
in Great Britain
0 Elsevier Applied
Science Publishers
Households of the landless poor or people displaced by government projects Co-operatives or farmer associationsa
Village Forest Department
Resettlement
Village woodlots
units
Settled households Co-operatives or farmer associations”
Decision-making
Farm forestry (Smallholder treefarming)
Types of prqjects
Five Common
Village or communal lands
Leased or newly titled government lands or land reform areas Degraded public lands Unused private land Corporate land
Labour
sources
Projects
Hired casual labour Contractual labour
Household labour Co-operatives or farmer associations Shared or exchanged labour from neighbours
Household labour Casual, hired labour Contractual labour Co-operatives or farmer associations Shared or exchanged labour from neighbours
Treefarming
Blocks of land owned or rented by farmers Around homesteads, farm bunds, and land boundaries Unused, private land
Land sources
TABLE 1 Types of Social Forestry of the output
Villages Participating
workers
Households Co-operatives or farmer associationsb
Householdsb
Distribution
Forest Department Non-governmental service organizations
Reforestation
Government lands Public forests Degraded public land Watersheds and tank foreshores Unused, private land
Roadsides Railway rights-of-way Canal and river banks
Degraded public lands Schools, hospitals, and other local institutions Unused, private land
Forest Department Participating workers The landless poor’
Forest Department Hired, casual labour Contractual labour Voluntary, unpaid labour
z
or corporations.
Forest Department Public Works Department Railroad companies Participating workers Villages Schools and other public facilities The landless poor’
Hired, casual labour Contractual labour Forest Department Public Works Department
banks;
Schools, hospitals and other local institutions The poor Non-governmental service organizations
Village co-operatives Voluntary labour Forest Department Schoolchildren
(1Subject to constraints imposed by Forest Department and other government agencies; development b May be partial or full recovery of costs of free inputs, land, or loan capital. ’ Harvesting may be deferred to preserve watersheds, preserve shade, or retain ornamental value.
Forest Department Public Works Department (including Highway and Irrigation Departments) Railroad companies Non-governmental service organizations
Strip plantings
Schools, hospitals, and other local institutions Non-governmental service organizations
142
Eric L. Hyman
community development (FLCD). It also draws from the experienceof using M/E in other types of rural development projects to identify opportunities and constraints in monitoring and evaluating FLCD projects. The purpose of monitoring is to improve the operation of the project as designed. Monitoring focuseson the inputs, activities, management and outputs of a project. In contrast, evaluationscan help decision makers reexamine the design of the project and assessits impacts. In the past, many large development projects lacked systematic monitoring. Forest Departments kept recordson the number of seedlings produced and the locations of planting or distribution, but rarely on survival rates or yields. As a result, some projects that could have been salvaged failed and successful projects could have done even better. Evaluations often have been carried out at project completion before many impacts were ready for investigation. Sinceevaluations were geared to the needs of donors, many countries viewed them as a burdensome requirement. Now, however, there is an increasing awarenessby donors and governments of the importance of making M/E more systematic and integrated within the planning and decision-making process. Monitoring and evaluation are especiallyimportant for FLCD projects which are innovative and diverse, taking place on lands that have not conventionally beenused in forestry (seeTable l), requiring new skills of foresters and involving participants with differing interests,resourcesand abilities. Of course, uncertainty is unavoidable in all planning for the future. Things outside the control of the project with natural or human causescan affect its chances of success.Some examples are droughts, typhoons, pests and diseases,grazing animals, market prices, political instability and the availability of chemicals, machinery, or spare parts. Although monitoring cannot change some of these, it can suggest measuresto reduce their harmful effects. For example, plantings can be delayed if monsoons are late and thorny hedgescan be added around village woodlots if the local people do not control grazing animals. Furthermore, FLCD projects place a high priority on socio-economic objectives. Rural development activities can have major direct impacts that are difficult to foresee; for example, changesin the distribution of income, employment, the social well-being of women and the landless poor, the ability of individuals and institutions to participate in development, and environmental quality. Even within a country, two similar projects may have different impacts. ’ 7
Monitoring
OBJECTIVES
and ecaluation
OF MONITORING
of forestry
AND
projects
143
EVALUATION
Monitoring is a continuous process of collecting, analysing and communicating information on project operation and performance. It takes place during project implementation and is intended to assist project managers in making decisions on whether changes are necessary in the way the project, as designed, is being implemented. Since a ‘welldesigned project’ with the necessary inputs, technical resources and support services can fail as a result of problems in operation and management, monitoring is important to indicate whether the scale, location and timing of project activities are on schedule and are being efficiently carried out. A focus on planned targets for inputs, activities and outputs encourages a rigorous approach to monitoring. However, targets established in advance of project implementation can be arbitrary. If the targets were too high or too low, a project might appear more or less successful than it really is. In fact, the appropriate targets may evolve over time in response to changes in conditions or as a result of the project itself. For example, adhering to targets for supplying pine seedlings for pulpwood may be counterproductive if there is a greater demand for fuelwood species or if changes in market conditions favour hardwoods for veneer production. Thus, non-attainment of a target does not necessarily indicate a failure. What is more important is understanding the reasons for accomplishments and shortfalls. Evaluations examine the ejjkctiveness of the project’s design, estimate the production and distribution of outputs and assess the impacts. Evaluations are concerned with some of the same things as monitoring, but the timing and purpose of the analyses are different. There are two types of evaluation: ongoing and cx post. Ongoing evaluations take place during project implementation either routinely or conditionally when difficulties have arisen or more aid is required. An ongoing evaluation can be useful in making decisions on changing the strategies, techniques, institutional arrangements and resource allocations for the project or government policies affecting forestry. A secondary purpose of ongoing evaluations is to stimulate communication among project staff, project management, the local people and outside organizations. Ex post evaluations occur after completion of the project. They rely on the findings of monitoring and ongoing evaluation, as well as additional
144
Eric L. Hyman
information on subsequent impacts. Since timber, pulpwood and tree fruits can take many years to grow, it often is premature to measure outputs and impacts during the course of the project. Because FLCD projects are relatively new, only a few have been around long enough for their impacts to be assessed.An ex post evaluation can be useful in deciding whether to compensatepeople adversely affected by the project or to mitigate unintended environmental impacts. It can indicate whether the technologies adopted and the investments made under the project are likely to continue-or even spread to other places-without external assistanceor whether follow-up projects in the same or other areas are needed.Expost evaluations also create an information basefor assessing development in the future and can help appraise the prospects for complementary projects, such as pulpmills or wood-based cottage industries, that increasethe value added to projects’ outputs. Finally, ex post evaluations inform decision makers in other provinces, countries, or international agenciesabout the benefitsand costs of the project and the lessons learned in its experience. This information may be useful in designing similar projects elsewhere. An overemphasis on the achievement of targets in an evaluation can divert attention from the measurement of impacts. There are four main difficulties in analysing impacts. First, there are time lags between the production of outputs and the generationof impacts. Income distribution effectsare hard to trace.18,21Furthermore, impacts on health, education, social well-being, and the capacity to participate in development may not be obvious until many years after the increase in incomes. Secondly, without baseline studies of the initial social, economic and environmental conditions in an area before implementation of the project, it is difficult to tell what changeshave occurred. Baselinestudies have been conducted for only a few FLCD projects. Without baseline data, it still may be possible to make some inferences about impacts by studying places that are similar to the project area in all other respects except the absenceof the project. Yet, conditions change over time in developing areasfor many reasons,so it may not be possible to attribute particular changes to the effects of any one project. Thirdly, development projects may have unintended effects on some groups. One of the main reasons for the failure of forestry projects on government or village lands is neglect of the prior users of the land.” When the interests of theseaffected personsare not taken into account, they may graze animals on the planted seedlings;burn stands in order to
Monitoring
and evaluation
of forestry
projects
145
plant crops or re-establish grasses; cut trees prematurely for fuelwood; or take part in violence. Sometimes, land tenure problems are not apparent in advance because the actual tenure differs from the legal status of the land. For example, the amount of land available for a village woodlot programme in Pakistan turned out to be limited because adjoining large landholders had appropriated communal land, even though their ownership is not recognized in land title records or tax ledgers.4 Fourthly, since environmental impacts may be dispersed over a large area and are influenced by many other natural factors and human activities, it can be difficult to isolate their causes.24 After the physical, chemical and biological impacts have been estimated, the next step is to place values on them. Considerable progress has been made in developing these methods, but they are still imperfect.‘2~‘5~‘g Most methods tend to underestimate environmental values and do not handle cumulative or indirect effects well.
I
I
Ongoing
.
I
evaluation
I
4 Generation of early impacts
i i i
-A
I Completion of project implementation
i i i
I Generation of subsequent impacts Fig. 1.
Typical
elements of a project
Ex post evaluation -.-. addressed in monitoring
and evaluation.
i i
Eric L. Hyman
146
Figure 1 summarizes the main elements of M/E. The following example illustrates their differences in emphasis. Suppose that a farm forestry project has been designed to counter deforestation. Monitoring may show that the government nurseries are meeting the targets for seedling production. An ongoing evaluation may identify lack of extension services for landowners under the project as the actual constraint to the participation of smallholders. An ex post evaluation would estimate the costs and benefits of the project compared with alternatives such as providing public lands to slash-and-burn cultivators for agroforestry, or promoting more efficient woodstoves. The Appendix lists some examples of questions that might be relevant in the M/E of FLCD projects.
LESSONS
FROM
THE
EARLY
EXPERIENCE
WITH
M/E
The early experience with M/E of agricultural development projects shows that the findings have not been well utilized. This section discusses the constraints and opportunities surrounding the use of M/E findings. Resistance to M/E Sometimes project management or staff resist the idea of monitoring and evaluation because M/E can expose weaknesses in decision making or implementation.6 People dislike scrutiny of the efficiency and effectiveness of their work and possible inquiries about their judgment, sincerity, or honesty. Evaluation findings often appear critical, particularly if claims for the project have been overstated.’ These fears can be reduced if project managers are reassured about the objectivity and fairness of the individuals carrying out the analyses. Project managers may view M/E as burdensome or trivial requirements if they are confused about what M/E are supposed to do. Thus, project managers need to be convinced that the findings will provide useful and timely information that can be integrated into the planning and decision-making process. Project staff may resent M/E if they perceive that it will increase their workload or burden them with paperwork. New M/E tasks imposed on field staff can detract from day-to-day implementation duties or place more time pressures on them. Even when M/E reports only require a small amount of time, field staff might not see the importance of the activity or may be reluctant to submit uncertain numbers that someone might
Monitoring
and
evaluation
of forestry
projects
147
question later. Furthermore, project staff may be unwilling to undertake critical self-reflection. It takes time to change staff attitudes whenever there is a new emphasis in project implementation. Staff can be motivated if the rationale for the M/E procedures and the practical importance of the information to be collected are explained to them and the chain of responsibilities and system of incentives and disincentives functions well. Identification
of data requirements
and indicators
A common problem with M/E systems is that they often collect too much, too little, or the wrong type of data. As a result, monitoring and ongoing evaluation reports often are produced too late to be useful to project management. If project management is not involved in the design of the information requirements, some of the data may be irrelevant to the needs of decision makers. Data collection and analysis are expensive and time-consuming. Further, since the capacity of users to absorb information is limited, too much information can be confusing. Some evaluations have relied on surveys without carefully considering how the sample size affects the amount of data generated. When the sample size is too large, an evaluation may cost too much for the amount of usable information obtained. If the sample size is too small, it might not be possible to carry out statistical tests or stratify the findings by socio-economic groups or site qualities and sizes. The determination of an efficient sample size and selection of respondents for the sample are discussed elsewhere.3s22 One way to avoid the pitfalls of collecting too much or too little data is for M/E staff and project management to co-operate in developing a flexible plan for data collection and analysis. This plan can help the M/E staff avoid omitting important data or collecting data that are unnecessary, or at the wrong level of aggregation, and can facilitate the interpretability of the data. Frequency
and timing of reports
The appropriate frequency and timing of monitoring and ongoing evaluation reports depend on the timetable of decisions that project managers have to make. Field staff are more likely to feel burdened if monitoring and ongoing evaluation reports are required at too frequent intervals. Due to donor requirements and the schedule for planning
148
Eric L. Hyman
follow-on projects, ex post evaluations often are carried out shortly after project implementation is completed, eventhough this may be too soon to observe the full range of outputs and impacts. Data collection
methods
A mixture of data collection approachesmay bechosento suit data needs, the budget, and staff skills and to provide cross-checksfor key indicators. Monitoring can be accomplished through the use of administrative records, field staff reports and ‘rapid reconnaissance’ approaches. Administrative records document financial expenditures, staff allotments, the purchase of supplies, the use of equipment and hiring of services. Standardized field staff reports may cover such things as the recruitment of participants, the provision of seedlings,the number and location of treesplanted or woodstovesdistributed and seedlingmortality rates. Rapid reconnaissancemethods include quick and inexpensive site visits, interviews, or group meetings emphasizing the use of direct exploratory observation and guided techniques for asking questions.’ In general,lengthy special studies are not necessaryfor routine monitoring. Ongoing and ex post evaluations build on the information gatheredin monitoring to keep costs down and minimize duplication of effort. Since an evaluation may be concernedwith growth of trees or yield of fruits or fodder, some physical measurementsmight be necessary. In some cases, surveys, interviews and workshops may be needed. Surveys are most useful in finding out about socio-economic impacts and the opinions of local people.*,I6 However, surveys can be expensiveand time-consuming, especially if the sample sizeis large. Interviews are more free-ranging than surveysand are aimed at knowledgeable or influential people, rather than the general public. Sometimes, the values and interests of these individuals may differ from those of the rural poor. Workshops are best suited for drawing out information on facts, values and the perceptions of experts.7 An exyost evaluation sometimesincludes more detailed, specialstudies of longer duration-case studies, experiments and cost-benefit analyses to gain some understanding of a project’s cause-effect relationships and impacts. Case studies are in-depth examinations of relatively small representative groups of people, limited geographic areas, or particular aspectsof a problem. Case studies can yield valuable insights, although they require intensive efforts by skilled analysts, and it can be difficult to
Monitoring
and ecaluation
of jbrcstry
projects
149
make generalizations from one case to another. Experiments test hypotheses about technical questions such as the relationship between yields and factors like speciesselection, tree spacings, tree establishment techniques,fertilizer use, or irrigation. Cost--benefit analysesestimate the net economic gains from the project to society and to representative participants.‘,’ 3 The choice of data collection methods and their applicability under various conditions are discussed in detail elsewhere.3,‘0 Problems
in data collection
Data collection can be difficult as a result of poorly developed transportation and communications infrastructure in remote, rural areas and the hard conditions facing M/E staff in the field. There is a danger of ‘rural development tourism’ where M/E staff are steeredto areas where the most successful activities are underway, accessibility is easy, the people are relatively better off and local officials have support.5 Conditions may be very different in the uplands, away from the roadside, acrossthe river, or when the weather is lesscomfortable. Short field visits might not provide enough information about seasonal or periodic variations or local cultural differences. A participatory approach which fosters a dialogue between the local people and M/E staff is necessary to obtain information on social impacts. However, this can be difficult to achieve because of: (a) the unwillingness of the rural poor to speakopenly for fear of retaliation; (b) embarrassment over weak articulation skills; (c) communication failures or errors when there are distinct, local languages; (d) the limited experience of project staff in establishing a close dialogue on an equal basis with the rural poor; (e) the domination of discussion by a few individuals and (f) drifting into tangential issues.14In addition, the sociopolitical situation in some countries can make participatory M/E infeasible. In many cases,too many data are collected from influential peoplewith relatively high incomes and educational levelsor powerful social contacts. Certain ethnic, sex,or agegroups frequently receivemore than their share of attention. As a result, tribal and nomadic people, women, residentsof remote areas and the disadvantaged social groups that are lessvocal are under-represented.23
1.50
Eric L. Hyman
The ways in which questionsare askedcan affect the findings. A leading question, such as, ‘Don’t you agreethat the Forest Department has done an excellentjob in promoting social forestry? is an obvious example, but many casesare more subtle, resulting from the relationship betweenthe M/E staff member and the respondent. One common problem is that people may suspect that the M/E staff may be tax collectors, forest guards, or land registry officials. Or, people may feel that they cannot speak freely without jeopardizing goods or servicesthat they receive from the Forest Department or their personal relationships with extension workers and village-level technicians. Cooperation can be improved if the reasonfor monitoring or evaluation are explained to the people. Hypothetical questions, such as, ‘Would you be willing to take up tree farming if you owned some land? are difficult for respondents. Respondentsalso may have difficulty remembering numbers or activities that took place more than a few months before. Since local measurement units may be imprecise or subject to variation, M/E staff may have to make some physical measurements of wood yields and use rates. Sometimes, respondents have incentives to provide incorrect answers. These incentives may be obvious for questions about income or when people have settled in public forests illegally. However, the causemay be more subtle, such as when people overstate their fuelwood consumption becausethey expectthat the information will be usedto setrestrictions on fuelwood collection. Good survey and interview procedurescan minimize many of these problems. Data processing and analysis
Bottlenecks in data processing and analysis have prevented the timely releaseof M/E findings in many cases.This problem is often causedby the collection of too much data or a premature reliance on computerized data processing. Computerization has caused delays in the M/E of some projects due to shortages of skilled staff, limited accessto mainframe computers, mechanical breakdowns and cost overruns. Minicomputers are less subject to these disadvantages because they are relatively inexpensive and easy to use and repair. Computerization has advantageswhen there are a large number of casesor variables for eachcaseand the input data formats areprecededto
Monitoring
and evaluation
of forestry
projects
151
streamline data entry. Computerization allows the data to be rearranged at different levels of aggregation to meet the requirements of various users. For instance, local project managers may need statistics at the village level while state or national officials may want the data combined at a higher level. In addition, certain statistical tests are feasible only on a computer. Nevertheless, if an FLCD project is small, personnel with computer experience are unavailable, or only simple arithmetic is required, hand calculators may be all that is necessary. Location
and organization
of M/E units
Since monitoring differs from evaluation in its purposes, scope and location and organizational re-, potential users, the appropriate sponsibilities for these two activities may differ. The early experience indicates that it is preferable for monitoring to be carried out internally by the project-implementing agency with external assistance where necessary. In contrast, evaluations tend to be better if they are externalcarried out by outside organizations such as provincial or national planning or finance ministries, government departments of evaluation, universities, research institutes, or consulting firms. The main reasons for this include: (i) the availability of expertise, (ii) the accountability and objectively of the M/E staff and (iii) the likelihood of timely results. Since organizations and the conditions under which they operate vary a great deal across and within countries, the appropriate location and organization of M/E units may differ for specific cases. Most Forest Departments have the expertise to monitor the achievement of silvicultural targets and the reasons for accomplishments or shortfalls. However, few foresters have expertise in social science theories and methods or in research design for surveys, interviews and case studies needed to waluatc impact. Unless it is feasible to hire, retain and promote good social scientists within the Forest Department or provide extensive social science training to foresters currently on the staff, it may be necessary to turn to an external organization for evaluations. Since monitoring is designed to meet the needs of project decision makers, an internal unit has a stronger motivation than an external agency for making monitoring a tool to increase accountability. Working in-house, monitoring staff have a better opportunity to demonstrate their competence and integrity to project management.’ It also may be easier
152
Eric L. Hyman
for an internal monitoring unit to gain the ear of decision makers informally. On the other hand, evaluations require a broader view of the structural factors in an economy and a society that explain people’s behaviour. Internal evaluations tend to focus too narrowly on the competence or integrity of particular individuals on the project staff.’ In addition, an external viewpoint may be necessary in order to question the basic premises of a project, Nevertheless, the project-implementing agency must have confidencein external organizations conducting an evaluation. The objectivity of M/E units also is a concern. In some countries, achievement reports from field staff tend to conform to targets all the time, regardless of actual conditions.20 This is less of a problem for monitoring when measurable targets can be verified by field staff supervisors or the staff of central monitoring units. In contrast, evaluations involve more subjective issues. An internal evaluation is lesslikely to be successfulbecausefew staff memberswill rate their own performance-or that of their colleagues-impartially. Lack of objectivity is especiallya problem when staff perceivethat salary increases or promotions depend on the evaluations. The desire to avoid being criticized can be an evenstronger motivation in many cultures. However, outsiders also may look out for their own interests. For example, consultants or research institutes might produce overly favourable reports to curry favour with clients, or they might cast their criticisms in such a way as to attract follow-up work. An internal monitoring system is more likely to be aware of the time pressuresfacing decision makers. If a good plan for the collection and analysis of data is prepared, monitoring should not create a time burden for project field staff. Ongoing evaluations are more intensive efforts that may require larger blocks of time during implementation than project field staff can spare and this may necessitatethe creation of a separate internal evaluation unit. Another alternative is to subcontract out certain tasks, but this tactic can backfire if one slow organization holds up the sequential tasks of other organizations. Some consulting firms have a good record for producing documents on time, but some academics in universities and researchinstitutes are unaccustomed to working under time pressure.There is often less time pressurefor the results of ex post evaluations.
Continuity of staff is important for both monitoring and ongoing evaluation; otherwise, efforts eventually may slow down or grind to a
Monitoring
and evaluation
of forestry
projects
153
halt, or the techniques and assumptions behind the collection and analysis of data may change over time, causing problems in comparability. Continuity also may help to build an atmosphere of confidenceand trust among M/E staff, field staff and participants. Continuity is likely to be a problem for internal monitoring units where jobs are filled by temporary employees or carry low pay and prestige. In some projects, junior staff have been transferred out of monitoring units into management after receiving training or demonstrating their competence. Lack of continuity can be a problem if too many external organizations are involved in an evaluation. Budgeting
and cost over-runs
Cost is a major factor in determining the amount of information that can be obtained in monitoring and evaluation. Cost also affects the selection of methods for data collection and analysis, as well as the location and organization of M/E units. A budget constraint can force M/E to focus on just the basic essentialinformation. However, the findings of monitoring and ongoing evaluations have not been timely in some caseswhen the budget was too small or there were delays in disbursements of funds. Another possible result of an insufficient budget for M/E can becost overruns which may reducethe confidenceof decision makers in the M/E staff. A M/E system should be designed so that the potential benefits outweigh the costs. The potential benefits stem from improvements in the efficiency or effectivenessof the project and other projects. The benefits are most likely to be high when a project is large, complex, or innovative or when a lot of uncertainty surrounds the project. However, the benefits will be low if M/E findings are not timely or useful to decision makers or if the donor or project management is inflexible about changing the design or implementation of the project. The costs of monitoring can be reduced by relying on project field staff and managers. Sometimes, subcontracting out specific tasks in an evaluation can be cost-effective. However, a heavy reliance on consulting firms or expatriatescan be expensive.In someplaces,local universitiesand research institutes have proved less costly since they are non-profit making and low wages are paid to student assistants who may be enthusiastic and hard-working. However, it is difficult to generalize because the costs, level of skills and motivation within different organizations vary.
154
Presentation
Eric L. Hyman
and use of findings
Some M/E reports have not influenced decision makers because little attention was paid to the presentation and communication of M/E findings. Decision makers require concise and understandable reports that highlight the pros and cons of each option. Since averages can obscure the variation in the data, decision makers needto be given some senseof the distribution of the findings. Informal communication is important, too, becausedelivering adverse findings to decision makers in public or without advance warning can causethem to react defensively. Small, private meetings that include top decision makers, key subordinates and M/E staff may facilitate good communication. In presenting the findings, sensitivity analysis can be used to test the effects that value judgments or key assumptions have on the conclusions of the analysis. Supposethat two decision makers disagreeon whether the income distribution objective is twice or three times as important as the economic growth objective. A sensitivity analysis might show that this differenceof opinion might not affect the decisionsthat are on the agenda. Presenting the findings to the local people will publicize the project’s achievements and may contribute to the search for short-run and longrun solutions to the problems encountered. Action is more likely to be taken by project management, local groups, or other institutions if a public commitment is made verbally in an open forum and then followed up in writing. Periodical
appraisal
of the performance
of the M/E system
Since monitoring and evaluation have specific functions and are not just meant to contribute to basic research, periodic appraisal of the performance of the M/E system is desirable. Project management, other government agencies and international assistanceagenciesall have an interest in ensuring that M/E are on track. Mid-course corrections may prove necessaryin the design or implementation of the M/E system. CONCLUSION Monitoring and evaluation can be useful tools for improving the implementation of a project and designing follow-up schemes in the
Monitoring and evaluation of forestry projects
155
project area or elsewhere. The full potential of M/E has sometimes not been achieved because of (a) confusion on the part of decision makers about what M/E are supposed to do; (b) vagueness or conflict among planned project objectives or targets; (c) undesirable rigidities in adherence to targets for inputs, activities and outputs; (d) the difficulty of measuring impacts; (e) an insufficient or overblown scope of the analysis; (f) problems with the organizational set up for monitoring or evaluation and (g) inadequate attention to the presentation and communication of findings.
ACKNOWLEDGEMENT Support for this paper was provided by FAO’s Forestry Policy and Planning Service. The opinions expressed are those of the author and are not necessarily those of FAO. The author would like to acknowledge the helpful comments of J. E. M. Arnold, Arnold0 Contreras and Philip Wardle (FAO); Tom Catterson (US AID), Roger Slade and Raymond Noronha (IBRD); J. G. Campbell (UNDP) and Robert Chambers (Ford Foundation).
REFERENCES 1. Caro, F. (Ed.) Evaluation research: An overview. In: Readings in evaluation research, New York, Russell Sage Foundation, 197 I. 2. Carruthers, I. and Chambers, R., Rapid appraisal for rural development, Agricultural Administration, 8 (198 l), 407-22. 3. Casley, D. and Lury, D., A handbook on monitoring and evaluation of agriculture and rural development projects, Washington, DC, World Bank, 1981. 4. Cernea, M., Land tenure systems and social implications of forestry development programs, Staff Working Paper No. 452, Washington, DC, World Bank, 1981. 5. Chambers, R., Shortcut methods in information gathering for rural development projects, Sussex, Institute of Development Studies, University of Sussex, 1979. 6. Clayton, E., Role characteristics and operational features of agricultural monitoring systems. In: Monitoring systems for agricultural and rural development projects. (Clayton, E. and Petry, F. (Eds)), Rome, FAO, 198 1.
Eric L. Hyman
156
7. Delbecq, E., Van de Ven, A. and Gustafson, D., Group techniques for program planning, Glenview, Illinois, Scott Foresman and Co., 1975. 8. FAO, Wood fuel surveys. Report GCP/INT/365/SWE, Rome, FAO, 1983. 9. Gregersen, H. and Contreras, A., Economic analysis of forestry projects. Forestry Paper No. 17. Rome, FAO, 1979. 10. Honadle, G., Rapid reconnaissance for development administration: Mapping and moulding organizational landscapes, World Development, 10 (1982) 633-50. 11. Hoskins, M., Benefits foregone as a major issue for FLCD success. In: Proceedings
Washington, 12. Hufschmidt,
of
the
U.S. AID
Workshop
DC, US AID, 1982. M., James, D., Meister,
on Community
Forestry,
A., Bower, B. and Dixon,
J.,
Environment, natural systems and development: An economic valuation guide, Baltimore, Maryland, Johns Hopkins University Press, 1983. 13. Hufschmidt, M. and Hyman, E., Economic approaches to natural resource and environmental quality analysis, Dublin, Tycooly Press, 1982. 14. Huizer, G., Preliminary guidelinesforparticipation monitoring and ongoing evaluation of people’s participation projects, Rome, FAO, 1982.
15. Hyman, E., The valuation of extramarket benefits and costs in environmental impact assessment, Environmental Impact Assessment Review, 2 (1981), 227-58. 16. Hyman, E., How to conduct a rural energy survey in a developing country, Renewable Sources of Energy, 1 (1983) 137-49. 17. Hyman, E., Smallholder treefarming in the Philippines: A comparison of two credit programs, Unasylva, 35 (139) (1983) 25-3 1. 18. McArthur, J., Monitoring the impact of agricultural development projects on low-income groups: Principles and problems. In: Monitoring systemsfor agricultural and rural developmentprojects (Clayton, E. and Petry, F. (Eds)), Rome, FAO, 1981. 19. Nichols, R. and Hyman, E., An evaluation of environmental assessment methods, Journal of the Water Resources Planning and Management Division, American Society of Civil Engineers, 108, No. WRl (March, 1982) 87-105. 20. Palin, D., Management of development forestry, Report No. GCP/RAS 46 (SWE), Rome, FAO. 21. Schuster, E., Economic impact analysis of forestry projects: A guide to evaluation of distributional consequences. In: Economic analysis offorestry projects: Readings (Gregersen, H. (Ed.)), Forestry Paper No. 17, Supplement 2, Rome FAO, 1980. 22. Slade, R. and Feder, G., The monitoring and evaluation of training and visit extension in India: A manual of instruction, Washington, DC, World Bank, 1981. 23. Van Heck, B., Research guidelines for field action projects to promote the participation of the poor in rural organizations, Rome, FAO, 1979. 24. Zimmerman, R., Environmental impacts of forestry: Guidelines for its assessment in developing countries, Rome, FAO, 1983.
Monitoring
APPENDIX:
and evaluation
offorestry
prqjc>cts
157
EXAMPLES OF QUESTIONS FOR MONITORING AND EVALUATING FLCD PROJECTS
Inputs How much land is being devoted to various components of the project? Are the type and intensity of recommended forestry practices matched to the quality of the sites? Are the sites for tree planting being abandoned or converted to agricultural uses? Have large landholders or the landless poor appropriated communal lands intended for the project? Is labour for the project scarce in certain seasons or locations? Are co-operatives and shared, or voluntary labour arrangements working out after the initial period of enthusiasm? Is the right amount of labour time being devoted to tree establishment, mortality replacement, maintenance and harvesting? Is labour given sufficient incentives and supervision? Are plastic bags or other supplies in short supply? If the Forest Department is providing the labour for establishing village woodlots, do the villagers take care of these stands as their own or do they perceive them to be the property and responsibility of the Forest Department? Are the intended beneficiaries participating in the project? Are there unofficial charges for subsidized inputs? Is enough credit at satisfactory terms available to cover the initial costs of investments such as starting a cottage industry or to meet the subsistence needs of treefarmers during the production period? What is the effect of inflation on the adequacy of the loan size over time? Are loan funds being devoted to the agreed purposes or diverted to other uses? Are the right kinds and amounts of fertilizer being applied through the proper techniques? Is the system able to detect the need for chemical or biological pest control and provide it in a timely, safe and effective manner? Do participants have access to appropriate equipment and tools? Is irrigation water efficiently used? Are the project sites sufficiently accessible? Intcrmediatc Outputs Does the supply of seedlings meet the demand and is planting available where needed and at the right time of the year?
stock
158
Eric L. Hyman
Is the age, size and vigour of the seedlings satisfactory? Have the right species of trees been selected for the kinds of sites available and the purposes desired by the intended beneficiaries? Are seedlings being transported successfully? Are seedlings being wasted through improper planting techniques or lack of follow-up care? What are the seedling mortality rates after planting? Are improved woodstoves (or charcoal kilns) properly constructed, maintained, and used? Are the designs of improved woodstoves compatible with cultural preferences (e.g. cost, safety, convenience in desired cooking methods, amount of smoke, speed of use, size and height, cleanliness, kinds of fuels and pots and pans that can be used, aesthetics and symbolic or ceremonial values)? Are the designs of improved charcoal kilns compatible with the preferences of charcoal makers (e.g. cost, safety, quality of output, speed, convenience, capacity and durability)?
Operation and Management Activities Is the project helping to build local institutions or is this constrained by (1) lack of local leadership, (2) domination by elites or (3) factionalism? Have new organizations (co-operatives, associations, village groups, and private voluntary groups) developed under the project continued to thrive after the initial period of establishment? To what extent are the local people allowed to participate in decision making and share the benefits of membership in these organizations? Is there sufficient co-ordination between the various government agencies directly or indirectly involved in the project? Is the project adversely affected by personnel problems or budget constraints and delays in disbursements? Are project staff technically up to date in forestry and related fields such as agriculture, livestock management and soil and water conservation? Do project staff have the communications and inter-personal skills necessary for establishing and maintaining a two-way dialogue with participants? Are staff training programmes timely, successful and sufficiently broad based? Are the skills learned in trainingprogrammes applied on the job or does training lead to promotions to other types of jobs? Are the outreach and mobilization activities informing the intended beneficiaries about the existence and terms of the project?
Monitoring
and evaluation
offorestry
prqjects
159
Is participation in the project limited by cultural factors such as attitudes toward debt or administrative problems such as red tape, rigid eligibility requirements, or conflicts between FLCD activities and law enforcement responsibilities of the Forest Department? Do forestry laws and policies have unintended adverse effects on people’s decisions to participate in the project or the potential profits from participation (e.g. a log export ban, price controls on forest products, or the ability of the government to expropriate land)? How high is the drop-out rate among participants? Do extension workers spend a disproportionate share of their time with relatively well-off treefarmers, rather than poor people in more remote areas? Do the extension services enable people to solve their own problems and help their neighbours? Are the participants who received extension services following the recommended practices? Are there incentives for retaining the most capable staff members in field positions? outputs
How much timber, pulpwood, fuelwood, polewood, and construction wood has been produced? What kind, quality and quantity of non-wood outputs have been produced ? How much wood has been saved through the adoption of improved woodstoves, charcoal kilns, or wood preservation? Are there bottlenecks in harvesting or transporting outputs? Are exploitative arrangements occurring in harvesting or marketing contracts, forward selling, mortgaging, or selective buying of only the best portion of the output? Are output prices fair? Can domestic markets absorb the increased output of raw materials or cottage industry products without a decline in the real prices of the outputs? Have export markets been developed? Impacts Have the prior users of the land been brought into the project or compensated for their losses? Are speculators or local elites gaining control of project sites through unfair practices?
160
Eric L. Hyman
How many person-days of employment are being generated by the project? How much income (net profits) is the project producing? Who is bearing the costs and who is receiving the benefits, of the project? What are the impacts of the project on women and disadvantagedethnic groups? What are the environmental impacts of the project on- and off-site?