- Email: [email protected]
Surveys and decisions in the context of multidisciplinary programmes:
Agriculture, Ecosystems and Environment 87 (2001) 129–140
Surveys and decisions in the context of multidisciplinary programmes: estimators and indicators F. Laloë a,∗ , B. Lauckner b , M. Piron c , K. Rahim d a b
UMR EGER (IRD-UVSQ), IRD, BP5045, 34032 Montpellier Cedex 1, France CARDI, University Campus, P.O. Bag 212, St. Augustine, Trinidad and Tobago c IRD, UR GEODES, 32 Avenue H. Varagnat, 93143 Bondy Cedex, France d Bangladesh Rice Research Institute, Gazipur 1701, Bangladesh
Abstract Scientific programmes whose objectives are to provide pertinent knowledge and information for sustainable use of a natural resource always include data collection operations. Functions of the data collected are obtained from surveys. These functions (estimators) are formulae constructed according to the survey’s model defined by the design of the survey and the sample selection procedures. In complex designs, a ‘superpopulation model’ is always present. This model, accounting for available knowledge on the observed system, is defined by assumptions on the distribution of the collected data. Users consider not only the quality of the estimates (outputs of the survey) but also, and possibly primarily, the quality of these estimates as input (information) to their own decision frameworks, which also constitute a superpopulation model. Hence, a survey combines at least two models; a survey model and a user’s model. These different models are discussed in this paper with the help of three examples that differ in the types of system under study, in the objectives of the surveys, and in the nature of the collected information. Specific functions of the collected survey data may be considered as estimators when considered as output of the survey model and as indicators when considered as input to the user’s model. A user may build indicators from many sources of information and, as several different users may use the estimates, a survey is an element of an information system. Analysis of data leads to improved knowledge of the system under study and to the better identification of the various associated models. Therefore, surveys are elements of dynamic systems in which theory and practice are complementary. Although classical sampling theory often appears not to be sufficient in this global context, it remains necessary. The use of information from different sources to be used from different points of view implies an increased level of rigour with many models, all of which must be explicitly defined. © 2001 Elsevier Science B.V. All rights reserved. Keywords: Models; Sampling theory; Fisheries; Agricultural production; Integrated pest management (IPM); Indicators
1. Introduction
∗
Corresponding author. Tel.: +33-467-636-972; fax: +33-467-638-778. E-mail address: [email protected] (F. Laloë).
Scientific programmes whose objectives are to provide pertinent knowledge and information for sustainable use of a natural resource always include data collection operations. The main objective of these operations may be to provide better scientific primary
0167-8809/01/$ – see front matter © 2001 Elsevier Science B.V. All rights reserved. PII: S 0 1 6 7 - 8 8 0 9 ( 0 1 ) 0 0 2 7 3 - 0
130
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
knowledge on the resource or on its use and depletion, or on the interface between the resource and its use (in terms of structures and functioning). It may also be to provide information to “actors” who take decisions at different levels (farmers, communities, policy makers, etc.). From surveys, functions of the data collected are obtained. These functions or estimators are formulae based upon the design of the survey and the sample selection procedures. The quality of these estimators depends on the design and on the sample(s) taken. Most of the time quality is expressed in terms of bias, variance and/or mean square error (MSE). This refers to (the distribution of) the difference between the estimate and the true value of the parameter that is being estimated. Sampling theory provides a well-established framework to deal with these questions. Available textbooks (Barnett, 1991; Cochran, 1977; Droesbeke et al., 1987; Scherrer, 1983; Yates, 1981) satisfy the fundamental theoretical and/or practical needs for estimation purposes. However, optimisation of surveys to provide estimates of several parameters or, more fundamentally, to answer the general question “how helpful is the whole set of estimators obtained from a survey” are not in the scope of these textbooks. Questions such as this must be addressed to the users of the information provided by the survey. The users will consider not only the quality of the estimates (output of the survey) but also, and possibly primarily, the quality of these estimates as input (information) in their own decision frameworks. Hence, a survey combines at least two models; a survey model and a user’s model. In a very “simple” case the aim may be to provide to one single user the best possible estimate of only one characteristic. But this “simple” case may entail a survey that could be very complex. For example, in a government survey the target parameter may be the financial loss due to pests at a national level. The survey that is developed for this purpose may need, given available knowledge, a multilevel design with observations at field, farm, village, regional and national levels. Each of these levels will have some stratification(s) accounting for diversity of crops, environmental effects, etc. In such a case, a considerable number of estimates will be computed, the aggregation of which will provide the estimate of the target parameter at the na-
tional level. But it is quite evident that a final report, only giving the estimate of this parameter together with estimate(s) relative to the distribution of the corresponding estimator (MSE), would be rejected as not sufficient. This is because analysis of collected data is likely to provide a lot of information at various levels. This information will be useful both for the initial user, and also for many other potential users at later stages. Users at later stages may be identified through the analysis; they may be persons or institutions whose decisions and actions may explain the data collected. If they are interested in the results of the survey, then providing them with this information may contribute to change the nature of the system under study. Hence, the survey is itself an element of the system and takes place in a dynamic process. The aim of this paper is to make clear that in this complex domain any given function of data can be considered from at least two points of view: as an output (estimator) of a survey and as input(s) (indicators) to user(s) model(s). These points of view are presented, with the help of three examples, through a discussion on the survey model and the user’s model(s). Some of the dynamic implications that emerge from these two aspects are considered.
2. Examples Three examples have been chosen. These examples are very different from each other in terms of the systems under study, in terms of their objectives and in terms of the nature of the collected information. They also differ in terms of the relationships between “survey makers” and “survey users”. 2.1. Small-scale fisheries exploitation system in Senegal 2.1.1. General context In Senegal, both industrial and small-scale fisheries harvest marine living resources. More than 5000 small-scale fishing units obtain about two thirds of the total national annual yields. These units employ more than 20,000 fishermen and provide important yields for both export and local consumption. For small-scale fisheries, a perennial survey was progressively designed during the 1970s and conducted
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
under the authority of the scientists of the Centre de Recherches Océanographiques de Dakar Thiaroye (CRODT) of the Institut Sénégalais de Recherches Agricoles (ISRA). The survey was initially designed to provide information required for research devoted to some particular harvested species. As the catches obtained by small-scale fishing units often contain a very high species diversity, and as there are a great variety of fishing gears, the design became devoted to small-scale fisheries “in general”. So whereas initially many surveys were considered, one for each species of interest, it became clear that there should be one general survey (Pechart, 1982). 2.1.2. Goal and objectives of survey Information on fishing activity and catches are needed for modelling the dynamics of harvested fish populations. Fitting such models, using functions of the collected data as input, leads to output of estimates of fish stock productivity related to the level of fishing activity. This may provide helpful information to be input to fish and fisheries management. The scientists who designed this survey are hence the direct users of the data. Since results are to be used for management purposes, decision-makers are also users of results. 2.2. Surveys to estimate demand for fresh agricultural products in Caribbean countries 2.2.1. General context Agricultural researchers in small developing countries are sometimes criticised for promoting technologies for increased or improved agricultural production that have little positive effect on the welfare of the farming community. Sometimes farmers do not adopt the new technologies at all and at other times they may adopt but then abandon technology that does improve production. Many times this is because the farmers claim that there are no markets for the increased production or, even if markets exist, prices do not justify the inputs. Although production and import figures for fresh agricultural produce are easily available, similar figures for amounts actually purchased or consumed are not routinely collected and these amounts purchased give more accurate figures of the demand for a product.
131
Additionally if prices of purchase by the consumer are also considered, accurate pictures of elasticity of demand can be built up. Prices of goods sold to the final consumer (the householder) are often published, but information on prices paid by the intermediate consumer, for example the restaurant, hotel or supermarket, is usually scanty. However, these prices (to the intermediate consumer) are of far more interest to a farmer trying to increase volume than those paid by the supermarket or market customer. 2.2.2. Goal and objectives of surveys The goal is to increase the level of production and marketing of fresh and processed food products in a number of Caribbean countries with the following major yet broad objectives: • to provide information to quantify the present and potential demand for fresh agricultural produce; • to provide information to assist farmers, agribusiness firms, other individuals and firms in making investment decisions and in developing appropriate plans, strategies and programmes to achieve competitiveness; • to provide information to allow for preparation of plans for market development networks; • to provide timely information to agribusiness sector magazines and other relevant media. The users are therefore at different levels, from individual farmers to small and medium enterprises. 2.3. Integrated pest management (IPM) and farmers field schools in Bangladesh 2.3.1. General context As in many other countries, crop pests are one of the major constraints to agricultural production in Bangladesh. In order to prevent excessive use of pesticides that lead to human health problems and to the appearance of resistant pests integrated pest management (IPM) practices are proposed. Two main questions arise about the consequences of IPM practices in term of ecological and economical efficiency and about the adoption of such practices by farmers. These involve some learning processes and appropriation of the practices by the farmers (Kamal, 2000).
132
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
2.3.2. Goal and objectives of surveys In order to answer the two questions above, two different surveys are conducted. The first is a conventional survey to estimate the differences in terms of inputs and outputs between IPM and non-IPM user farms. Using the general principles provided by sampling theory, farms are selected according to a sampling design specified in order to obtain estimates with a given precision. The second is not specifically a survey, but the organisation of Farmer’s Field Schools in which a “survey like” system is put in place. Small groups of about 25–30 male and female farmers maintain a common field area where they carry out season-long experiments. They meet every week throughout the entire rice crop season (15 sessions) with experienced facilitators. At each meeting, a diagnostic of the state of crop growth is made from observations and future action is decided. Here the survey is continually updated from observations on the growth process, in order to provide information useful for immediate decisions and actions. In this participatory research context, users and data collectors are the same persons (the farmer) and the survey model must be defined to account for the evolution of the ecosystem. Here the farmers become the ecological experts of their own fields.
cal properties of probability sampling (for example, Cochran, 1977, p. 9), there is no need to distinguish between these two aspects. In practice, observation and data collection on complex systems are made using complex designs, and random sampling rules are often not respected. In such cases some general assumptions must be made through the definition of superpopulation models (Cochran, 1977, p. 213) “. . . to regard the finite population (under study) as drawn at random from an infinite superpopulation which has certain properties”. These properties define a model, which may be at least partially rebuilt through the analysis of data collected during the survey. With systematic sampling, for example, estimates of the variance of a characteristic of the population must be related to such a superpopulation model. Typical theoretical examples of such models include those with “periodic variation”, “linear trend”, “autocorrelated populations” or simply “no relation between the characteristic and the order”. 3.2. Definition of target parameters and survey design From the survey examples in this paper, information is obtained about:
3.1. The survey model (estimations)
• fish populations and distributions, marine ecosystem assessment, fisheries activity assessment, etc.; • demand for products, opinion of farmers, opinion of traders, etc.; • damage done by agricultural pests, impact of IPM practices, opinions of farmers, etc.
Surveys are conducted to find out information, that is required for some purpose. Information is a very general term and two related aspects may be considered. The first is the identification of target parameters that are to be estimated and the construction of a design that uses available knowledge of sources of variability in the system to be observed. This is done in order to obtain good unbiased estimates of these parameters (or to obtain a given needed precision). Once the data are collected, the second aspect is the computation of the estimates as functions of these data and the general conclusions from the survey such as those that may be presented in a final report. Theoretically, if sample selection procedure(s) are rigorously conducted, according to the mathemati-
The nature of the required information is more or less clear, depending on the overall addressed question. For each of the above elements a long list of parameters may be identified. For each of these parameters somebody may be found who needs the information that will be provided by the estimate of the parameter. Everybody who has participated in a meeting to prepare a questionnaire to be used in a multidisciplinary survey, can recall the great difficulties and the animated discussions. But some agreed target parameters are usually identified, such as the annual catch per species obtained by small-scale fisheries, demand for each considered agricultural product per month and country, or decrease in chemical pesticide use with IPM practice, this latter possibly combined with increase in crop production.
3. Methodology
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
The estimator used to estimate the annual catch of a given species obtained by the small-scale fishery is a complicated function of the collected data. Fishing units may use different gears that may or may not harvest this species. For many of the species, there was evidence of seasonality in the yields per trip. So strata were defined as combinations of the classifying factors; gear and time period (fortnights in this case). For logistic reasons, surveys take place in villages, with one field investigator in each village. Strata were defined as combinations of different villages, different fortnights and different gears (village × fortnight × gear in statistical terminology). In each stratum a multistage design was defined with: • a selection of days in the fortnight (generally 6 days); • a selection of fishing trips in each selected day (for a maximum of 30 units); • a selection of fish from each selected trip. It is possible, from textbook formulae, to estimate the total catch and MSE of the corresponding estimator in a stratum with the assumption that random sampling rules hold at each selection procedure (Laloë, 1985). It is possible to estimate total catches in the set of villages where the survey is done, and with some further assumptions it is possible to estimate the total annual catch made at the country level from data collected during fishing censuses (Soceco-Pechart, 1982). These results of annual catches are obtained using a formula that combines a considerable number of estimators at various levels: • estimation of the catch during a fishing trip (and corresponding variance); • estimation of the catch obtained during a day (and corresponding variance which combines estimates of “intra trips” and “inter-trips–intra-day” variances); • estimation of the catch obtained during a fortnight (and corresponding variance which combines estimates of “intra days” and “inter-days–intra-fortnight” variances). This is done for each species (there are about 30 important species), and the total number of estimates is greater than 10,000. Although many of these estimates are published annually as official statistics, it would be absurd to try to provide each of these in a final
133
report. It would be equally absurd to provide only the total catch estimate for each species. From all the estimates, seasonality and many other phenomena can be described that are potentially interesting for further studies. This is a sufficient reason for further data analysis. Data analysis is also necessary to assess the quality of estimates. The example to estimate demand for fresh agricultural products is also very complex. In this case there are very many different products, of which around 50 are considered important. These were divided into categories of vegetables, fruits, staples, meat and fish, juices and condiments. Different surveys are conducted in 15 Caribbean countries and in five different segments of agribusiness retailing, namely hotels, restaurants, supermarkets, agro-processors and exporters. In each of these segments strata were defined for sampling purposes. For example, hotels were stratified by number of rooms, but the definition of this stratum varies from country to country. This is because a ‘large’ hotel in one Caribbean country may only be a ‘medium’ hotel in a more tourist-oriented country. Similar differences in strata definition between countries applied to the other retailing segments. Attempts were made to collect the information on a monthly basis to estimate seasonal demands, so once again many estimates are computed at many different levels some of which are nested within others but also there are orthogonal relationships (for example, for each month estimates are available for each country). Thus, with about 50 products, 15 countries, 5 segments and 12 months there are well over 40,000 estimates. Again it would be absurd to consider each of these, but the analysis is once again designed to inform future surveys as well as estimating current demand. For the IPM example, any report providing the estimate of difference in input use or in yield must contain a description of IPM and non-IPM practices. From this point of view estimates of differences in input use provide information both on IPM practice context and on IPM practice results. Furthermore, the choice of the quantities that are estimated is influenced by the views of what IPM practice is and of the likely impact of such a practice. Opinion and interest of farmers in IPM practice, “know-how” of farmers of this practice and the potential for improving this know-how appear to be essential information to understand the estimated dif-
134
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
ference and the potential results that could be obtained with a better IPM practice. Hence, the farmers’ field school cannot be considered as an activity that is independent of the survey that provides the data used for estimation of the impact of IPM practice. The farmers’ field schools also provide invaluable information that is essential for better understanding of the estimates of this impact. In the final report on these activities, the large number of observations is used to provide impact estimates that are more or less explicitly defined. 3.3. Data analysis As noted above estimates are obtained from the surveys by formulae that are provided by sampling theory. Given rules on selection procedures, the theory deals also with the quality of estimates, in terms of precision, bias, MSE. But it is never certain that these rules are respected. In the fishery example, the usual estimates of variance “inter-trips–intra-day” are used. These estimates may be used to assess the quality of day catch estimates if unit selection (fishing trips) satisfies the rules of selection for simple random sampling in the population of fishing trips for the day. It can be shown that this is not possible. For example if a given trip is selected, all the trips ending during this observation process have a nil probability of being selected. In some villages the landing beach may be 3 km long and two successive selected trips cannot have landed too far from each other. If the investigator can select a maximum of, say, eight trips in 1 h, the probability of selection of a trip depends on the time if the rhythm of the returns is not constant during the day (Laloë, 1985). If the variable of interest is not independent of the time or the place of return, those violations of simple random sampling selection rules lead to bias. Therefore, analysis of data is needed to examine these problems and to try to propose some superpopulation model under which the quality of the estimates may be efficiently discussed. In the Caribbean example it cannot be assumed that the individual observations all have the same precision. The data are collected by interview with the business owner or agent; some of them keep very accurate reliable written records of purchase and sales; others do not, at least not on an individual commodity basis. It can be assumed (fairly safely) that where reliable records are kept, the precision of the observations is
very high and, providing the survey personnel make no errors, most of these observations are extremely precise. However, where the respondent does not attempt to keep records some of the observations may be very imprecise. There are ways to overcome this. For example, all historical data can be regarded as available and the establishments surveyed are monitored continually (perhaps on a weekly basis). There will be some initial inconvenience of having to wait for results, but the more serious impediment is that costs of data collection increase by 52 times if weekly visits are made for 1 year as opposed to a single interview. Unfortunately the level of funding does not support the monitoring approach. Thus, the collected data have varying precision. There were also problems with building the sampling frames; for example a supermarket was defined as an establishment selling fresh agricultural produce with two or more cash registers. But no data on number of cash registers existed and there were not even reliable, comprehensive lists of supermarkets available. Similar problems arose with restaurants. Thus, selection bias was almost certainly present, but had to be assumed unimportant as the larger establishments, which handle greater volumes, are easier to identify. Thus, the estimates of demand for each stratum were obtained by multiplying the purchase reported by the inverse of the (estimated) sampling fractions. The overall demand was estimated in the usual way when strata estimates had been obtained. These estimates, however, cannot be given a variance because of the nature of the data collected and the sampling methodology. In the first two examples, survey designs have been built from identified needs of users, but once this design is explicit, the observations can be made without the participation of those users. The IPM example is different in this aspect. It combines in “real time” both the definition of target parameters and their estimations from surveys and data analysis. Each weekly meeting may be viewed as the treatment of the observations made during the preceding days. The facilitators provide a method for this treatment, with guidelines to do it, but the observations that will be made the following week largely depend on the nature of the diagnostic and on the decisions that are taken. The new design may be quite different to the previous one, but the latter is a consequence of the former and they
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
are highly interdependent. In this example, the results of surveys are immediately interpreted from the user’s point of view, which leads to continuous redefinition of the survey design. Therefore, it is necessary to examine the user’s point(s) of view, or in other terms, the user’s model(s). 3.4. The user’s model(s) (indicators) Once the quality of a given estimate or set of estimates resulting from a survey is assessed in the context of the survey model (bias, variance, MSE), users will use it according to their different points of view. A particular estimate may be extremely useful for some users, and may be not at all useful for some others. The quality of an estimate cannot be assessed from the concept of estimation theory only. It must also be considered from the user’s point of view, i.e. in the context of a user’s model. In the Caribbean example, an estimate of seasonality pattern of the demand for a given commodity may interest farmers if this estimate provides knowledge not previously available and if it provides a plan of action that will increase the profitability of the farm. This means that the farmers will use this estimate (and other estimates available from the survey and from elsewhere) as part of the input to their work plan. It is quite evident that the more precise the estimate is, the more it will be useful, but the sense of common words, such as bias for example, may be different in the survey model and the user’s model: • In the survey model, the estimate of a characteristic is unbiased if the expected value of the estimator is equal to the characteristic to be estimated. • In the user’s model the question deals with the decision taken by the user, due to the knowledge of the estimate, in order to obtain an expected result. In this context, the estimate is unbiased if the expectation of the distribution of the result from the taken decision is equal to the expected result. Hence, the word “estimate” and the various common terms used to assess related qualities may not cover all concerns. A solution could be to use the words “estimate” and “estimators” in the survey model, as usual, and to refer to words related to the domain of “indicators” (e.g. indicator, threshold, reference point) in the context of the user model. In the same way Mul-
135
lon and Piron (1998) distinguish two different types of validation for a survey: an internal validation deals with the usual qualities as defined in the statistical domain and an external validation defined by the interest for users, for which they propose various possibilities of evaluation. In fisheries, the models of population dynamics that use time series of estimated yearly catch and fishing effort are used to estimate “reference points” with which current estimates may be compared for giving indications of the status of the system (see for example FAO (1999), and Garcia and Staples (2000)). Such an indicator may typically be Ft /FMSY , where Ft is the current mortality rate (time t) and FMSY is the mortality rate corresponding to the maximum sustainable yield (MSY). These definitions are given in the context of models such as Schaefer’s model (Schaefer, 1957), which describes the relationship between biomass evolution (Bt ) and fishing effort ft : dBt Bt = rBt 1 − − qft Bt (1) dt K where r and K are the growth rate and carrying capacity, the usual parameters of the logistic model, and q is a catchability parameter (probability of catching one unit of biomass with one unit of effort). Such a model is usually fitted through integration of dBt /dt giving fitted catches that are functions of the parameters and of the estimates of fishing efforts provided by the survey. Estimates of the parameters are obtained by minimisation of the sum of squares of differences between fitted and estimated catches. Note that if Ft = qft remains constant, equal to some value between 0 and r, biomass and catch tend to equilibrium values. If this constant value of F is equal to F MSY = qfMSY = r/2, equilibrium catch value is maximum, equal to rK/4, corresponding to the so-called maximum sustainable yield (MSY) reference point. It is important to recognise that Ft , FMSY and Ft /FMSY may be estimated from the data collected in surveys, which means that these estimates are products of the same set of data (but as the model is non-linear, these estimates are obtained through iterative procedures and are functions of data that cannot be explicitly written). But in the user’s model they have very different status: Ft is a characteristic that is estimated, FMSY is a reference point defined in the context of a theory and Ft /FMSY is an indicator.
136
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
From the estimated value of this indicator the user may take decisions. For example the user may decide to fix future mortality to the FMSY value in order to aim in the long term at catches at the MSY level. In the context of this user’s model, when choosing estimation procedures for MSY, FMSY and qfMSY based on the maximisation of some criterion, a second or further superpopulation model on the used data is considered, explicitly or not. For example, one of the most popular methods (Pella and Tomlinson, 1969) consists of minimising the sum of squares of differences between fitted and estimated catches. This is a good solution with a superpopulation model, assuming that: • estimation errors on catches are independent with identical zero mean normal distributions; • Schaefer’s model is “true”; • the effort estimates are error free. Therefore, there may be many superpopulation models on the same data, at least one for estimation purposes and at least one for each user of those estimates. For each model, different assumptions are made.
4. Discussion 4.1. The utility of several distinct models The usual assumptions are certainly not valid, but if they are explicitly stated, the corresponding models may be assessed and can be useful. A lengthy quote from McCullagh and Nelder (1989) may be relevant here: “[. . . ] all models are wrong; some though are more useful than others and we should seek those [. . . ] not to fall in love with one model to the exclusion of alternatives [. . . ] some imagination and introspection is required here to determine the aspects of the model that are most important and more suspect”. Hence, some of the assumptions may be more crucial than others. For example, in Schaefer’s model, one of the key assumptions deals with the catchability “q” (see Eq. (1)). With this parameter a very simple relation F = qf is assumed between mortality “F” and fishing activity “f”. With this assumption, the evolution of annual efforts provides the evolution of mortality, which is the information necessary to fit a population dynamics model. The converse is possibly more im-
portant, i.e. a given relative variation in terms of mortality may be achieved by the same relative variation in terms of fishing activity “f”. Therefore, the more efficient the “institution” controlling the fishing activity can be, the more manageable will be the resource. Therefore, such an institution is clearly a main user and the catchability parameter, which could appear as a parameter defined from assumptions made mainly in the biological and technological domains, is also a key parameter for user’s identification and definition. In the Senegal fisheries example, analysis of data showed great intra strata heterogeneity resulting in potential bias (for example due to the relationship between hour of day and place of landing). This could lead to some improvement of the selection protocols, but the analysis also showed that such phenomena were not only due to natural variation in species catchability, but also to decisions of fishermen (Laloë et al., 1998; Pelletier and Ferraris, 2000). Fishermen can use a given gear in different ways, each having a different catchability for each species. Thus, a given “f” may result in various “F” and, according to the sentence of McCullagh and Nelder, the function F = qf is one of “the aspects of the model that are most important and more suspect”. An analysis of the relationship between F and f becomes necessary. As this relationship depends among others things on fishermen’s decisions, this analysis must take account of the fishermen’s points of view, which makes it necessary also to consider them as users, thus having user’s models. The survey in the Caribbean example was also conceived from an analysis of the specific needs of users. Farmers are not observational units in this survey, they do not appear at any place in the design nor in the functions of collected data. The quality of those functions as estimators are hence independent of them, but the utility of those functions as indicators entirely depends on the use they will make of them. In the IPM example the farmers field schools were designed by the users themselves. At each weekly meeting the presentation and analysis of the data were directly organised for immediate decision making and further observations to be made based on this analysis. The distinction between estimators and indicators appears less important in this case, because users themselves define the design and they themselves make the observations to obtain directly the required indicators.
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
137
Therefore, the evaluation of the quality of estimates in the superpopulation model of the survey and of the estimates in the superpopulation models of the users is an essential part of the evaluation of the utility of the survey. Any criticisms that can be made at the end of the analyses will lead to an update of the design of further surveys to include the new knowledge provided by these analyses. These updates may be necessary because of the need to provide a better understanding of the characteristics of the observed real system and, possibly more importantly, to provide a better identification of the actors and of their needs. Hence, surveys take place in a process of knowledge acquisition and modification, even if their objective is purely applied.
• Decision-makers say that the information provided is not so helpful.
4.2. The need for model diversity
4.3. Benefits of the approach
Estimators and indicators are both functions of data, outputs of survey(s), or inputs in user’s models. Therefore, the same function (a statistic) may be both an estimator and an indicator and the utility of such a distinction depends on the utility of the distinction between the different models from or in which the functions of data are considered. When a sampling design is built, from which estimators of some parameter(s) to help decision-maker(s) are defined, one of the following two situations is often considered:
The “frustrations” presented above may be viewed as positive things if a survey is considered a tool for production of both knowledge and information to users. There may be unexpected differences between the results of decisions based on estimates from surveys and the objectives for which these decisions were taken. These differences may be viewed as data. The analysis of these data identifies the connection between the different models. This may lead to modifications in the assessment of the corresponding functions of data as estimators and/or indicators. Hence, the analysis provides knowledge from which better questions for future surveys may be addressed. In the fisheries example, the analysis of data indicated that for a fishing trip, fishermen may choose a particular gear from those available and that they may also choose to use this gear in one of several possible ways. As gear is a criterion used to define strata in the sampling design, this led to reconsideration of the superpopulation model’s assumptions on the independence of units of the selected samples (fishing trips) within the strata. Furthermore, when choosing a gear fishermen “choose” a stratum and this choice is a component of the sizes of the strata. These sizes, which are given numbers in conventional sampling theory, hence become functions of user’s models. Laloë et al. (1998) showed that the estimated values of some biological parameters will differ considerably according to whether this is or is not taken into account.
• the way a given estimate will be used is known before building the design, or it is at least part of the design building process; • the survey is done to assess the state of some (natural) system and provides a diagnosis, which hopefully will be useful, but there is no consideration of the way users will use it. In these two cases there is no distinction between points of view on data collected. They are not supposed to be used a posteriori to answer the question of how and by whom they are used. There is no need for an analysis based on some conflict between survey and user’s model(s) because they are considered either as totally linked (in the same general framework) or as totally independent. This frequently leads to a double frustration. • Scientists consider that the information and advice that they provide are not sufficiently taken into account by the decision-makers.
These frustrations may occur because the decisionmaker combines information that is provided with other information in a way not considered (or not known) by the scientist. In order to understand the situation better, the connection between the survey model and the user’s model(s) must be considered. This means that each of these models must be explicitly defined. This is important because (i) there may be several user’s models, (ii) decisions are often compromise solutions and (iii) some of the users may react to a decision and this will lead to a change in the system under study.
138
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
In the Caribbean example, the design was built from user’s need analysis after some criticisms on the usefulness of the data that are already available. As users are not present in the sampled populations, their behaviour should not affect the quality of estimates (even if, as discussed in this paper, some difficulties remain). But the quality of these estimates as indicators will have to be assessed a posteriori through their impact on farmer’s decisions and through the improvement of the situation resulting from these decisions. For example the first results in the Caribbean surveys gave surprisingly high estimates of hotel and restaurant consumption for certain commodities (CARDI, 2000). This was of great interest to the users of the surveys, but at the same time it posed questions about the accuracy of the surveys. There was increased focus on these commodities in hotels and restaurants and this led to changes in the design of following surveys. In the IPM example the users directly define the surveys. The points discussed at each meeting cannot be precisely known at the beginning of the season. At each meeting there is an analysis of data, including a criticism of the results of previous decisions and a diagnosis on the state of the system. This analysis leads to both decision-making and an update of the survey. In this case, where the users directly participate in the refinement of the survey design, the analysis of the discrepancies between the functions of data according to different points of view takes place naturally. There is a very important link between the users model and the survey model, but this link may not result in the frustrations mentioned above because the models are, at least implicitly, defined by the same persons. This is not the case when scientists design surveys as they think that their design is “exactly” based on the needs of the users (they implicitly think that they are the users). The three examples highlight in different ways the various discussed aspects. Their main characteristics are summarised in the Table 1.
5. Conclusions The conflicts between models have been presented in different ways in the three very different examples presented in this paper. But in each case it leads
139
to an update of the framework that is used to represent the system under study, and to an update of the definition of the system. Hence, this system may be considered as “a construct with arbitrarily defined boundaries for discourse about complex phenomena to emphasise wholeness, inter-relationships and emergent properties” (Röling, 1994). The various models are elements of such a construct, which is a process that cannot, at any given time, be considered as achieved. By considering the need of confrontation between models the following wish may be fulfilled, as identified by O’Connor (1999): “If scientific questions, which relate directly to society, were researched in a ‘dialogical’ manner, ways would be sought to understand the concerned individuals, populations or stakeholders”. More generally the distinction between estimators and indicators is not straightforward because they may be the same thing considered by different persons. This distinction is needed when scientific results are to be used in some decision process as stated by Roqueplo (1997): “Ce qui transforme un énoncé scientifique en expertise scientifique, c’est le fait que son énonciation soit intégrée au dynamisme d’un processus de décision, et qu’elle soit formulée à l’usage de ceux qui décident. [. . . ] cela change beaucoup de choses” (A scientific statement becomes an expert scientific result when the statement is integrated into the dynamics of a decision process and when it is designed for the use of those who decide. [. . . ] this changes many things.) As described in this paper, the processes in which surveys take place are directly related to information systems in general terms. When considering surveys as elements of these systems, what may appear in some instances as criticisms of sampling techniques theory and application, leads directly to identify an increased need for rigour — and an increasing need and use for this sampling theory — resulting from the critical analysis of superpopulation models. This need is all the more important since the presence of several models broadens the information that may be useful for at least one of them. Information is becoming more and more accessible with more and more user-friendly tools for exploratory data analysis and data mining. These tools are sometimes utilised with little knowledge of how the data are obtained, but they can still be used to improve the data collection procedures.
140
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
References Barnett, V., 1991. Sample Survey Principles and Methods. Arnold, London. CARDI, 2000. Market opportunities and entry requirements for selected products in selected Caribbean countries. Caribbean Agricultural Research and Development Institute, Trinidad. Cochran, W.G., 1977. Sampling Techniques, 3rd Edition. Wiley, New York. Droesbeke, J.-J., Fichet, B., Tassi, P. (Eds.), 1987. Les Sondages. Economica, Paris. FAO, 1999. The development and use of indicators for sustainable development of marine capture fisheries. FAO Technical Guidelines for Responsible Fisheries: no. 8, FAO, Rome. Garcia, S.M., Staples, D.J., 2000. Sustainability reference systems and indicators for responsible marine capture fisheries: a review of concepts and elements for a set of guidelines. Marine Freshwater Res. 51, 385–426. Kamal, N.Q., 2000. Integrated pest management (IPM) in practice: I. Impact of IPM practices on rice fauna, II. Impact of farmers field schools (FFS) and sustainability of IPM. Bangladesh Rice Research Institute, Gazipur, Bangladesh. Laloë, F., 1985. Etude de la précision des estimations des captures et prises par unité d’effort obtenues à l’aide du système d’enquètes sur la pêche artisanale au centre de recherches océanographiques de Dakar Thiaroye. Doc. Scient. Cent. Rech. Océano. Dakar Thiaroye no. 100, 36 pp. Laloë, F., Pech, N., Sabatier, R., Samba, A., 1998. Model identification for flexible multifleet-multispecies fisheries: a simulation study. Fish. Res. 37, 193–202. McCullagh, P., Nelder, J.A., 1989. Generalized Linear Models, 2nd Edition. Monographs on Statistics and Applied Probability no. 37. Chapman & Hall, London.
Mullon, C., Piron, M., 1998. Sur la méthodologie de mise en place des observatoires socio-économiques. In: Laloë, F., Perrier, X. (Eds.), De l’Observation à l’Analyse, Implication de la Biométrie dans les Pays en Développement. Société Française de Biométrie 15, pp. 61–77. O’Connor, M., 1999. Dialogue and debate in a post-normal practice of science: a reflexion. Futures 31, 671–687. Pechart, 1982. Les enquêtes sur la pêche artisanale au CRODT. Arch. Cent. Rech. Océano. Dakar Thiaroye no. 112, 28 pp. Pella, J.J., Tomlinson, P.K., 1969. A generalized stock production model. Bull. Int. Am. Trop. Tuna Comm. 13, 419–493. Pelletier, D., Ferraris, J., 2000. A multivariate approach for defining fishing tactics from commercial catch and effort data. Can. J. Fish. Aquat. Sci. 57, 51–65. Röling, N., 1994. Platforms for decision-making about ecosystems. In: Fresco, L.O., Stroosnijder, L., Bouma, J., van Keulen, H. (Eds.), The Future of the Land. Mobilising and Integrating Knowledge for Land Use Options. Wiley, New York, pp. 385–393. Roqueplo, P., 1997. Entre savoir et décision, l’expertise scientifique. Sciences en questions, Inra éditions Paris, 112 pp. Schaefer, M.B., 1957. A study of the dynamics of the fishery for yellow fin tuna in the Eastern Tropical Pacific Ocean. Bull. Int. Am. Trop. Tuna Comm. 2, 247–585. Scherrer, B., 1983. Techniques de sondage en écologie. In: Frontier, S. (Ed.), Stratégies d’Échantillonnage en Écologie. Masson, Paris et PUL, Que., pp. 63–162. Soceco-Pechart, 1982. Recensements de la pêche artisanale maritime au Sénégal, Avril et Septembre 1981. Doc. Scient. Cent. Rech. Océano. Dakar Thiaroye no. 83. Yates, F., 1981. Sampling Methods for Censuses and Surveys, 4th Edition. Griffin, London.
Surveys and decisions in the context of multidisciplinary programmes: estimators and indicators F. Laloë a,∗ , B. Lauckner b , M. Piron c , K. Rahim d a b
UMR EGER (IRD-UVSQ), IRD, BP5045, 34032 Montpellier Cedex 1, France CARDI, University Campus, P.O. Bag 212, St. Augustine, Trinidad and Tobago c IRD, UR GEODES, 32 Avenue H. Varagnat, 93143 Bondy Cedex, France d Bangladesh Rice Research Institute, Gazipur 1701, Bangladesh
Abstract Scientific programmes whose objectives are to provide pertinent knowledge and information for sustainable use of a natural resource always include data collection operations. Functions of the data collected are obtained from surveys. These functions (estimators) are formulae constructed according to the survey’s model defined by the design of the survey and the sample selection procedures. In complex designs, a ‘superpopulation model’ is always present. This model, accounting for available knowledge on the observed system, is defined by assumptions on the distribution of the collected data. Users consider not only the quality of the estimates (outputs of the survey) but also, and possibly primarily, the quality of these estimates as input (information) to their own decision frameworks, which also constitute a superpopulation model. Hence, a survey combines at least two models; a survey model and a user’s model. These different models are discussed in this paper with the help of three examples that differ in the types of system under study, in the objectives of the surveys, and in the nature of the collected information. Specific functions of the collected survey data may be considered as estimators when considered as output of the survey model and as indicators when considered as input to the user’s model. A user may build indicators from many sources of information and, as several different users may use the estimates, a survey is an element of an information system. Analysis of data leads to improved knowledge of the system under study and to the better identification of the various associated models. Therefore, surveys are elements of dynamic systems in which theory and practice are complementary. Although classical sampling theory often appears not to be sufficient in this global context, it remains necessary. The use of information from different sources to be used from different points of view implies an increased level of rigour with many models, all of which must be explicitly defined. © 2001 Elsevier Science B.V. All rights reserved. Keywords: Models; Sampling theory; Fisheries; Agricultural production; Integrated pest management (IPM); Indicators
1. Introduction
∗
Corresponding author. Tel.: +33-467-636-972; fax: +33-467-638-778. E-mail address: [email protected] (F. Laloë).
Scientific programmes whose objectives are to provide pertinent knowledge and information for sustainable use of a natural resource always include data collection operations. The main objective of these operations may be to provide better scientific primary
0167-8809/01/$ – see front matter © 2001 Elsevier Science B.V. All rights reserved. PII: S 0 1 6 7 - 8 8 0 9 ( 0 1 ) 0 0 2 7 3 - 0
130
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
knowledge on the resource or on its use and depletion, or on the interface between the resource and its use (in terms of structures and functioning). It may also be to provide information to “actors” who take decisions at different levels (farmers, communities, policy makers, etc.). From surveys, functions of the data collected are obtained. These functions or estimators are formulae based upon the design of the survey and the sample selection procedures. The quality of these estimators depends on the design and on the sample(s) taken. Most of the time quality is expressed in terms of bias, variance and/or mean square error (MSE). This refers to (the distribution of) the difference between the estimate and the true value of the parameter that is being estimated. Sampling theory provides a well-established framework to deal with these questions. Available textbooks (Barnett, 1991; Cochran, 1977; Droesbeke et al., 1987; Scherrer, 1983; Yates, 1981) satisfy the fundamental theoretical and/or practical needs for estimation purposes. However, optimisation of surveys to provide estimates of several parameters or, more fundamentally, to answer the general question “how helpful is the whole set of estimators obtained from a survey” are not in the scope of these textbooks. Questions such as this must be addressed to the users of the information provided by the survey. The users will consider not only the quality of the estimates (output of the survey) but also, and possibly primarily, the quality of these estimates as input (information) in their own decision frameworks. Hence, a survey combines at least two models; a survey model and a user’s model. In a very “simple” case the aim may be to provide to one single user the best possible estimate of only one characteristic. But this “simple” case may entail a survey that could be very complex. For example, in a government survey the target parameter may be the financial loss due to pests at a national level. The survey that is developed for this purpose may need, given available knowledge, a multilevel design with observations at field, farm, village, regional and national levels. Each of these levels will have some stratification(s) accounting for diversity of crops, environmental effects, etc. In such a case, a considerable number of estimates will be computed, the aggregation of which will provide the estimate of the target parameter at the na-
tional level. But it is quite evident that a final report, only giving the estimate of this parameter together with estimate(s) relative to the distribution of the corresponding estimator (MSE), would be rejected as not sufficient. This is because analysis of collected data is likely to provide a lot of information at various levels. This information will be useful both for the initial user, and also for many other potential users at later stages. Users at later stages may be identified through the analysis; they may be persons or institutions whose decisions and actions may explain the data collected. If they are interested in the results of the survey, then providing them with this information may contribute to change the nature of the system under study. Hence, the survey is itself an element of the system and takes place in a dynamic process. The aim of this paper is to make clear that in this complex domain any given function of data can be considered from at least two points of view: as an output (estimator) of a survey and as input(s) (indicators) to user(s) model(s). These points of view are presented, with the help of three examples, through a discussion on the survey model and the user’s model(s). Some of the dynamic implications that emerge from these two aspects are considered.
2. Examples Three examples have been chosen. These examples are very different from each other in terms of the systems under study, in terms of their objectives and in terms of the nature of the collected information. They also differ in terms of the relationships between “survey makers” and “survey users”. 2.1. Small-scale fisheries exploitation system in Senegal 2.1.1. General context In Senegal, both industrial and small-scale fisheries harvest marine living resources. More than 5000 small-scale fishing units obtain about two thirds of the total national annual yields. These units employ more than 20,000 fishermen and provide important yields for both export and local consumption. For small-scale fisheries, a perennial survey was progressively designed during the 1970s and conducted
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
under the authority of the scientists of the Centre de Recherches Océanographiques de Dakar Thiaroye (CRODT) of the Institut Sénégalais de Recherches Agricoles (ISRA). The survey was initially designed to provide information required for research devoted to some particular harvested species. As the catches obtained by small-scale fishing units often contain a very high species diversity, and as there are a great variety of fishing gears, the design became devoted to small-scale fisheries “in general”. So whereas initially many surveys were considered, one for each species of interest, it became clear that there should be one general survey (Pechart, 1982). 2.1.2. Goal and objectives of survey Information on fishing activity and catches are needed for modelling the dynamics of harvested fish populations. Fitting such models, using functions of the collected data as input, leads to output of estimates of fish stock productivity related to the level of fishing activity. This may provide helpful information to be input to fish and fisheries management. The scientists who designed this survey are hence the direct users of the data. Since results are to be used for management purposes, decision-makers are also users of results. 2.2. Surveys to estimate demand for fresh agricultural products in Caribbean countries 2.2.1. General context Agricultural researchers in small developing countries are sometimes criticised for promoting technologies for increased or improved agricultural production that have little positive effect on the welfare of the farming community. Sometimes farmers do not adopt the new technologies at all and at other times they may adopt but then abandon technology that does improve production. Many times this is because the farmers claim that there are no markets for the increased production or, even if markets exist, prices do not justify the inputs. Although production and import figures for fresh agricultural produce are easily available, similar figures for amounts actually purchased or consumed are not routinely collected and these amounts purchased give more accurate figures of the demand for a product.
131
Additionally if prices of purchase by the consumer are also considered, accurate pictures of elasticity of demand can be built up. Prices of goods sold to the final consumer (the householder) are often published, but information on prices paid by the intermediate consumer, for example the restaurant, hotel or supermarket, is usually scanty. However, these prices (to the intermediate consumer) are of far more interest to a farmer trying to increase volume than those paid by the supermarket or market customer. 2.2.2. Goal and objectives of surveys The goal is to increase the level of production and marketing of fresh and processed food products in a number of Caribbean countries with the following major yet broad objectives: • to provide information to quantify the present and potential demand for fresh agricultural produce; • to provide information to assist farmers, agribusiness firms, other individuals and firms in making investment decisions and in developing appropriate plans, strategies and programmes to achieve competitiveness; • to provide information to allow for preparation of plans for market development networks; • to provide timely information to agribusiness sector magazines and other relevant media. The users are therefore at different levels, from individual farmers to small and medium enterprises. 2.3. Integrated pest management (IPM) and farmers field schools in Bangladesh 2.3.1. General context As in many other countries, crop pests are one of the major constraints to agricultural production in Bangladesh. In order to prevent excessive use of pesticides that lead to human health problems and to the appearance of resistant pests integrated pest management (IPM) practices are proposed. Two main questions arise about the consequences of IPM practices in term of ecological and economical efficiency and about the adoption of such practices by farmers. These involve some learning processes and appropriation of the practices by the farmers (Kamal, 2000).
132
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
2.3.2. Goal and objectives of surveys In order to answer the two questions above, two different surveys are conducted. The first is a conventional survey to estimate the differences in terms of inputs and outputs between IPM and non-IPM user farms. Using the general principles provided by sampling theory, farms are selected according to a sampling design specified in order to obtain estimates with a given precision. The second is not specifically a survey, but the organisation of Farmer’s Field Schools in which a “survey like” system is put in place. Small groups of about 25–30 male and female farmers maintain a common field area where they carry out season-long experiments. They meet every week throughout the entire rice crop season (15 sessions) with experienced facilitators. At each meeting, a diagnostic of the state of crop growth is made from observations and future action is decided. Here the survey is continually updated from observations on the growth process, in order to provide information useful for immediate decisions and actions. In this participatory research context, users and data collectors are the same persons (the farmer) and the survey model must be defined to account for the evolution of the ecosystem. Here the farmers become the ecological experts of their own fields.
cal properties of probability sampling (for example, Cochran, 1977, p. 9), there is no need to distinguish between these two aspects. In practice, observation and data collection on complex systems are made using complex designs, and random sampling rules are often not respected. In such cases some general assumptions must be made through the definition of superpopulation models (Cochran, 1977, p. 213) “. . . to regard the finite population (under study) as drawn at random from an infinite superpopulation which has certain properties”. These properties define a model, which may be at least partially rebuilt through the analysis of data collected during the survey. With systematic sampling, for example, estimates of the variance of a characteristic of the population must be related to such a superpopulation model. Typical theoretical examples of such models include those with “periodic variation”, “linear trend”, “autocorrelated populations” or simply “no relation between the characteristic and the order”. 3.2. Definition of target parameters and survey design From the survey examples in this paper, information is obtained about:
3.1. The survey model (estimations)
• fish populations and distributions, marine ecosystem assessment, fisheries activity assessment, etc.; • demand for products, opinion of farmers, opinion of traders, etc.; • damage done by agricultural pests, impact of IPM practices, opinions of farmers, etc.
Surveys are conducted to find out information, that is required for some purpose. Information is a very general term and two related aspects may be considered. The first is the identification of target parameters that are to be estimated and the construction of a design that uses available knowledge of sources of variability in the system to be observed. This is done in order to obtain good unbiased estimates of these parameters (or to obtain a given needed precision). Once the data are collected, the second aspect is the computation of the estimates as functions of these data and the general conclusions from the survey such as those that may be presented in a final report. Theoretically, if sample selection procedure(s) are rigorously conducted, according to the mathemati-
The nature of the required information is more or less clear, depending on the overall addressed question. For each of the above elements a long list of parameters may be identified. For each of these parameters somebody may be found who needs the information that will be provided by the estimate of the parameter. Everybody who has participated in a meeting to prepare a questionnaire to be used in a multidisciplinary survey, can recall the great difficulties and the animated discussions. But some agreed target parameters are usually identified, such as the annual catch per species obtained by small-scale fisheries, demand for each considered agricultural product per month and country, or decrease in chemical pesticide use with IPM practice, this latter possibly combined with increase in crop production.
3. Methodology
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
The estimator used to estimate the annual catch of a given species obtained by the small-scale fishery is a complicated function of the collected data. Fishing units may use different gears that may or may not harvest this species. For many of the species, there was evidence of seasonality in the yields per trip. So strata were defined as combinations of the classifying factors; gear and time period (fortnights in this case). For logistic reasons, surveys take place in villages, with one field investigator in each village. Strata were defined as combinations of different villages, different fortnights and different gears (village × fortnight × gear in statistical terminology). In each stratum a multistage design was defined with: • a selection of days in the fortnight (generally 6 days); • a selection of fishing trips in each selected day (for a maximum of 30 units); • a selection of fish from each selected trip. It is possible, from textbook formulae, to estimate the total catch and MSE of the corresponding estimator in a stratum with the assumption that random sampling rules hold at each selection procedure (Laloë, 1985). It is possible to estimate total catches in the set of villages where the survey is done, and with some further assumptions it is possible to estimate the total annual catch made at the country level from data collected during fishing censuses (Soceco-Pechart, 1982). These results of annual catches are obtained using a formula that combines a considerable number of estimators at various levels: • estimation of the catch during a fishing trip (and corresponding variance); • estimation of the catch obtained during a day (and corresponding variance which combines estimates of “intra trips” and “inter-trips–intra-day” variances); • estimation of the catch obtained during a fortnight (and corresponding variance which combines estimates of “intra days” and “inter-days–intra-fortnight” variances). This is done for each species (there are about 30 important species), and the total number of estimates is greater than 10,000. Although many of these estimates are published annually as official statistics, it would be absurd to try to provide each of these in a final
133
report. It would be equally absurd to provide only the total catch estimate for each species. From all the estimates, seasonality and many other phenomena can be described that are potentially interesting for further studies. This is a sufficient reason for further data analysis. Data analysis is also necessary to assess the quality of estimates. The example to estimate demand for fresh agricultural products is also very complex. In this case there are very many different products, of which around 50 are considered important. These were divided into categories of vegetables, fruits, staples, meat and fish, juices and condiments. Different surveys are conducted in 15 Caribbean countries and in five different segments of agribusiness retailing, namely hotels, restaurants, supermarkets, agro-processors and exporters. In each of these segments strata were defined for sampling purposes. For example, hotels were stratified by number of rooms, but the definition of this stratum varies from country to country. This is because a ‘large’ hotel in one Caribbean country may only be a ‘medium’ hotel in a more tourist-oriented country. Similar differences in strata definition between countries applied to the other retailing segments. Attempts were made to collect the information on a monthly basis to estimate seasonal demands, so once again many estimates are computed at many different levels some of which are nested within others but also there are orthogonal relationships (for example, for each month estimates are available for each country). Thus, with about 50 products, 15 countries, 5 segments and 12 months there are well over 40,000 estimates. Again it would be absurd to consider each of these, but the analysis is once again designed to inform future surveys as well as estimating current demand. For the IPM example, any report providing the estimate of difference in input use or in yield must contain a description of IPM and non-IPM practices. From this point of view estimates of differences in input use provide information both on IPM practice context and on IPM practice results. Furthermore, the choice of the quantities that are estimated is influenced by the views of what IPM practice is and of the likely impact of such a practice. Opinion and interest of farmers in IPM practice, “know-how” of farmers of this practice and the potential for improving this know-how appear to be essential information to understand the estimated dif-
134
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
ference and the potential results that could be obtained with a better IPM practice. Hence, the farmers’ field school cannot be considered as an activity that is independent of the survey that provides the data used for estimation of the impact of IPM practice. The farmers’ field schools also provide invaluable information that is essential for better understanding of the estimates of this impact. In the final report on these activities, the large number of observations is used to provide impact estimates that are more or less explicitly defined. 3.3. Data analysis As noted above estimates are obtained from the surveys by formulae that are provided by sampling theory. Given rules on selection procedures, the theory deals also with the quality of estimates, in terms of precision, bias, MSE. But it is never certain that these rules are respected. In the fishery example, the usual estimates of variance “inter-trips–intra-day” are used. These estimates may be used to assess the quality of day catch estimates if unit selection (fishing trips) satisfies the rules of selection for simple random sampling in the population of fishing trips for the day. It can be shown that this is not possible. For example if a given trip is selected, all the trips ending during this observation process have a nil probability of being selected. In some villages the landing beach may be 3 km long and two successive selected trips cannot have landed too far from each other. If the investigator can select a maximum of, say, eight trips in 1 h, the probability of selection of a trip depends on the time if the rhythm of the returns is not constant during the day (Laloë, 1985). If the variable of interest is not independent of the time or the place of return, those violations of simple random sampling selection rules lead to bias. Therefore, analysis of data is needed to examine these problems and to try to propose some superpopulation model under which the quality of the estimates may be efficiently discussed. In the Caribbean example it cannot be assumed that the individual observations all have the same precision. The data are collected by interview with the business owner or agent; some of them keep very accurate reliable written records of purchase and sales; others do not, at least not on an individual commodity basis. It can be assumed (fairly safely) that where reliable records are kept, the precision of the observations is
very high and, providing the survey personnel make no errors, most of these observations are extremely precise. However, where the respondent does not attempt to keep records some of the observations may be very imprecise. There are ways to overcome this. For example, all historical data can be regarded as available and the establishments surveyed are monitored continually (perhaps on a weekly basis). There will be some initial inconvenience of having to wait for results, but the more serious impediment is that costs of data collection increase by 52 times if weekly visits are made for 1 year as opposed to a single interview. Unfortunately the level of funding does not support the monitoring approach. Thus, the collected data have varying precision. There were also problems with building the sampling frames; for example a supermarket was defined as an establishment selling fresh agricultural produce with two or more cash registers. But no data on number of cash registers existed and there were not even reliable, comprehensive lists of supermarkets available. Similar problems arose with restaurants. Thus, selection bias was almost certainly present, but had to be assumed unimportant as the larger establishments, which handle greater volumes, are easier to identify. Thus, the estimates of demand for each stratum were obtained by multiplying the purchase reported by the inverse of the (estimated) sampling fractions. The overall demand was estimated in the usual way when strata estimates had been obtained. These estimates, however, cannot be given a variance because of the nature of the data collected and the sampling methodology. In the first two examples, survey designs have been built from identified needs of users, but once this design is explicit, the observations can be made without the participation of those users. The IPM example is different in this aspect. It combines in “real time” both the definition of target parameters and their estimations from surveys and data analysis. Each weekly meeting may be viewed as the treatment of the observations made during the preceding days. The facilitators provide a method for this treatment, with guidelines to do it, but the observations that will be made the following week largely depend on the nature of the diagnostic and on the decisions that are taken. The new design may be quite different to the previous one, but the latter is a consequence of the former and they
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
are highly interdependent. In this example, the results of surveys are immediately interpreted from the user’s point of view, which leads to continuous redefinition of the survey design. Therefore, it is necessary to examine the user’s point(s) of view, or in other terms, the user’s model(s). 3.4. The user’s model(s) (indicators) Once the quality of a given estimate or set of estimates resulting from a survey is assessed in the context of the survey model (bias, variance, MSE), users will use it according to their different points of view. A particular estimate may be extremely useful for some users, and may be not at all useful for some others. The quality of an estimate cannot be assessed from the concept of estimation theory only. It must also be considered from the user’s point of view, i.e. in the context of a user’s model. In the Caribbean example, an estimate of seasonality pattern of the demand for a given commodity may interest farmers if this estimate provides knowledge not previously available and if it provides a plan of action that will increase the profitability of the farm. This means that the farmers will use this estimate (and other estimates available from the survey and from elsewhere) as part of the input to their work plan. It is quite evident that the more precise the estimate is, the more it will be useful, but the sense of common words, such as bias for example, may be different in the survey model and the user’s model: • In the survey model, the estimate of a characteristic is unbiased if the expected value of the estimator is equal to the characteristic to be estimated. • In the user’s model the question deals with the decision taken by the user, due to the knowledge of the estimate, in order to obtain an expected result. In this context, the estimate is unbiased if the expectation of the distribution of the result from the taken decision is equal to the expected result. Hence, the word “estimate” and the various common terms used to assess related qualities may not cover all concerns. A solution could be to use the words “estimate” and “estimators” in the survey model, as usual, and to refer to words related to the domain of “indicators” (e.g. indicator, threshold, reference point) in the context of the user model. In the same way Mul-
135
lon and Piron (1998) distinguish two different types of validation for a survey: an internal validation deals with the usual qualities as defined in the statistical domain and an external validation defined by the interest for users, for which they propose various possibilities of evaluation. In fisheries, the models of population dynamics that use time series of estimated yearly catch and fishing effort are used to estimate “reference points” with which current estimates may be compared for giving indications of the status of the system (see for example FAO (1999), and Garcia and Staples (2000)). Such an indicator may typically be Ft /FMSY , where Ft is the current mortality rate (time t) and FMSY is the mortality rate corresponding to the maximum sustainable yield (MSY). These definitions are given in the context of models such as Schaefer’s model (Schaefer, 1957), which describes the relationship between biomass evolution (Bt ) and fishing effort ft : dBt Bt = rBt 1 − − qft Bt (1) dt K where r and K are the growth rate and carrying capacity, the usual parameters of the logistic model, and q is a catchability parameter (probability of catching one unit of biomass with one unit of effort). Such a model is usually fitted through integration of dBt /dt giving fitted catches that are functions of the parameters and of the estimates of fishing efforts provided by the survey. Estimates of the parameters are obtained by minimisation of the sum of squares of differences between fitted and estimated catches. Note that if Ft = qft remains constant, equal to some value between 0 and r, biomass and catch tend to equilibrium values. If this constant value of F is equal to F MSY = qfMSY = r/2, equilibrium catch value is maximum, equal to rK/4, corresponding to the so-called maximum sustainable yield (MSY) reference point. It is important to recognise that Ft , FMSY and Ft /FMSY may be estimated from the data collected in surveys, which means that these estimates are products of the same set of data (but as the model is non-linear, these estimates are obtained through iterative procedures and are functions of data that cannot be explicitly written). But in the user’s model they have very different status: Ft is a characteristic that is estimated, FMSY is a reference point defined in the context of a theory and Ft /FMSY is an indicator.
136
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
From the estimated value of this indicator the user may take decisions. For example the user may decide to fix future mortality to the FMSY value in order to aim in the long term at catches at the MSY level. In the context of this user’s model, when choosing estimation procedures for MSY, FMSY and qfMSY based on the maximisation of some criterion, a second or further superpopulation model on the used data is considered, explicitly or not. For example, one of the most popular methods (Pella and Tomlinson, 1969) consists of minimising the sum of squares of differences between fitted and estimated catches. This is a good solution with a superpopulation model, assuming that: • estimation errors on catches are independent with identical zero mean normal distributions; • Schaefer’s model is “true”; • the effort estimates are error free. Therefore, there may be many superpopulation models on the same data, at least one for estimation purposes and at least one for each user of those estimates. For each model, different assumptions are made.
4. Discussion 4.1. The utility of several distinct models The usual assumptions are certainly not valid, but if they are explicitly stated, the corresponding models may be assessed and can be useful. A lengthy quote from McCullagh and Nelder (1989) may be relevant here: “[. . . ] all models are wrong; some though are more useful than others and we should seek those [. . . ] not to fall in love with one model to the exclusion of alternatives [. . . ] some imagination and introspection is required here to determine the aspects of the model that are most important and more suspect”. Hence, some of the assumptions may be more crucial than others. For example, in Schaefer’s model, one of the key assumptions deals with the catchability “q” (see Eq. (1)). With this parameter a very simple relation F = qf is assumed between mortality “F” and fishing activity “f”. With this assumption, the evolution of annual efforts provides the evolution of mortality, which is the information necessary to fit a population dynamics model. The converse is possibly more im-
portant, i.e. a given relative variation in terms of mortality may be achieved by the same relative variation in terms of fishing activity “f”. Therefore, the more efficient the “institution” controlling the fishing activity can be, the more manageable will be the resource. Therefore, such an institution is clearly a main user and the catchability parameter, which could appear as a parameter defined from assumptions made mainly in the biological and technological domains, is also a key parameter for user’s identification and definition. In the Senegal fisheries example, analysis of data showed great intra strata heterogeneity resulting in potential bias (for example due to the relationship between hour of day and place of landing). This could lead to some improvement of the selection protocols, but the analysis also showed that such phenomena were not only due to natural variation in species catchability, but also to decisions of fishermen (Laloë et al., 1998; Pelletier and Ferraris, 2000). Fishermen can use a given gear in different ways, each having a different catchability for each species. Thus, a given “f” may result in various “F” and, according to the sentence of McCullagh and Nelder, the function F = qf is one of “the aspects of the model that are most important and more suspect”. An analysis of the relationship between F and f becomes necessary. As this relationship depends among others things on fishermen’s decisions, this analysis must take account of the fishermen’s points of view, which makes it necessary also to consider them as users, thus having user’s models. The survey in the Caribbean example was also conceived from an analysis of the specific needs of users. Farmers are not observational units in this survey, they do not appear at any place in the design nor in the functions of collected data. The quality of those functions as estimators are hence independent of them, but the utility of those functions as indicators entirely depends on the use they will make of them. In the IPM example the farmers field schools were designed by the users themselves. At each weekly meeting the presentation and analysis of the data were directly organised for immediate decision making and further observations to be made based on this analysis. The distinction between estimators and indicators appears less important in this case, because users themselves define the design and they themselves make the observations to obtain directly the required indicators.
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
137
Therefore, the evaluation of the quality of estimates in the superpopulation model of the survey and of the estimates in the superpopulation models of the users is an essential part of the evaluation of the utility of the survey. Any criticisms that can be made at the end of the analyses will lead to an update of the design of further surveys to include the new knowledge provided by these analyses. These updates may be necessary because of the need to provide a better understanding of the characteristics of the observed real system and, possibly more importantly, to provide a better identification of the actors and of their needs. Hence, surveys take place in a process of knowledge acquisition and modification, even if their objective is purely applied.
• Decision-makers say that the information provided is not so helpful.
4.2. The need for model diversity
4.3. Benefits of the approach
Estimators and indicators are both functions of data, outputs of survey(s), or inputs in user’s models. Therefore, the same function (a statistic) may be both an estimator and an indicator and the utility of such a distinction depends on the utility of the distinction between the different models from or in which the functions of data are considered. When a sampling design is built, from which estimators of some parameter(s) to help decision-maker(s) are defined, one of the following two situations is often considered:
The “frustrations” presented above may be viewed as positive things if a survey is considered a tool for production of both knowledge and information to users. There may be unexpected differences between the results of decisions based on estimates from surveys and the objectives for which these decisions were taken. These differences may be viewed as data. The analysis of these data identifies the connection between the different models. This may lead to modifications in the assessment of the corresponding functions of data as estimators and/or indicators. Hence, the analysis provides knowledge from which better questions for future surveys may be addressed. In the fisheries example, the analysis of data indicated that for a fishing trip, fishermen may choose a particular gear from those available and that they may also choose to use this gear in one of several possible ways. As gear is a criterion used to define strata in the sampling design, this led to reconsideration of the superpopulation model’s assumptions on the independence of units of the selected samples (fishing trips) within the strata. Furthermore, when choosing a gear fishermen “choose” a stratum and this choice is a component of the sizes of the strata. These sizes, which are given numbers in conventional sampling theory, hence become functions of user’s models. Laloë et al. (1998) showed that the estimated values of some biological parameters will differ considerably according to whether this is or is not taken into account.
• the way a given estimate will be used is known before building the design, or it is at least part of the design building process; • the survey is done to assess the state of some (natural) system and provides a diagnosis, which hopefully will be useful, but there is no consideration of the way users will use it. In these two cases there is no distinction between points of view on data collected. They are not supposed to be used a posteriori to answer the question of how and by whom they are used. There is no need for an analysis based on some conflict between survey and user’s model(s) because they are considered either as totally linked (in the same general framework) or as totally independent. This frequently leads to a double frustration. • Scientists consider that the information and advice that they provide are not sufficiently taken into account by the decision-makers.
These frustrations may occur because the decisionmaker combines information that is provided with other information in a way not considered (or not known) by the scientist. In order to understand the situation better, the connection between the survey model and the user’s model(s) must be considered. This means that each of these models must be explicitly defined. This is important because (i) there may be several user’s models, (ii) decisions are often compromise solutions and (iii) some of the users may react to a decision and this will lead to a change in the system under study.
138
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
In the Caribbean example, the design was built from user’s need analysis after some criticisms on the usefulness of the data that are already available. As users are not present in the sampled populations, their behaviour should not affect the quality of estimates (even if, as discussed in this paper, some difficulties remain). But the quality of these estimates as indicators will have to be assessed a posteriori through their impact on farmer’s decisions and through the improvement of the situation resulting from these decisions. For example the first results in the Caribbean surveys gave surprisingly high estimates of hotel and restaurant consumption for certain commodities (CARDI, 2000). This was of great interest to the users of the surveys, but at the same time it posed questions about the accuracy of the surveys. There was increased focus on these commodities in hotels and restaurants and this led to changes in the design of following surveys. In the IPM example the users directly define the surveys. The points discussed at each meeting cannot be precisely known at the beginning of the season. At each meeting there is an analysis of data, including a criticism of the results of previous decisions and a diagnosis on the state of the system. This analysis leads to both decision-making and an update of the survey. In this case, where the users directly participate in the refinement of the survey design, the analysis of the discrepancies between the functions of data according to different points of view takes place naturally. There is a very important link between the users model and the survey model, but this link may not result in the frustrations mentioned above because the models are, at least implicitly, defined by the same persons. This is not the case when scientists design surveys as they think that their design is “exactly” based on the needs of the users (they implicitly think that they are the users). The three examples highlight in different ways the various discussed aspects. Their main characteristics are summarised in the Table 1.
5. Conclusions The conflicts between models have been presented in different ways in the three very different examples presented in this paper. But in each case it leads
139
to an update of the framework that is used to represent the system under study, and to an update of the definition of the system. Hence, this system may be considered as “a construct with arbitrarily defined boundaries for discourse about complex phenomena to emphasise wholeness, inter-relationships and emergent properties” (Röling, 1994). The various models are elements of such a construct, which is a process that cannot, at any given time, be considered as achieved. By considering the need of confrontation between models the following wish may be fulfilled, as identified by O’Connor (1999): “If scientific questions, which relate directly to society, were researched in a ‘dialogical’ manner, ways would be sought to understand the concerned individuals, populations or stakeholders”. More generally the distinction between estimators and indicators is not straightforward because they may be the same thing considered by different persons. This distinction is needed when scientific results are to be used in some decision process as stated by Roqueplo (1997): “Ce qui transforme un énoncé scientifique en expertise scientifique, c’est le fait que son énonciation soit intégrée au dynamisme d’un processus de décision, et qu’elle soit formulée à l’usage de ceux qui décident. [. . . ] cela change beaucoup de choses” (A scientific statement becomes an expert scientific result when the statement is integrated into the dynamics of a decision process and when it is designed for the use of those who decide. [. . . ] this changes many things.) As described in this paper, the processes in which surveys take place are directly related to information systems in general terms. When considering surveys as elements of these systems, what may appear in some instances as criticisms of sampling techniques theory and application, leads directly to identify an increased need for rigour — and an increasing need and use for this sampling theory — resulting from the critical analysis of superpopulation models. This need is all the more important since the presence of several models broadens the information that may be useful for at least one of them. Information is becoming more and more accessible with more and more user-friendly tools for exploratory data analysis and data mining. These tools are sometimes utilised with little knowledge of how the data are obtained, but they can still be used to improve the data collection procedures.
140
F. Laloë et al. / Agriculture, Ecosystems and Environment 87 (2001) 129–140
References Barnett, V., 1991. Sample Survey Principles and Methods. Arnold, London. CARDI, 2000. Market opportunities and entry requirements for selected products in selected Caribbean countries. Caribbean Agricultural Research and Development Institute, Trinidad. Cochran, W.G., 1977. Sampling Techniques, 3rd Edition. Wiley, New York. Droesbeke, J.-J., Fichet, B., Tassi, P. (Eds.), 1987. Les Sondages. Economica, Paris. FAO, 1999. The development and use of indicators for sustainable development of marine capture fisheries. FAO Technical Guidelines for Responsible Fisheries: no. 8, FAO, Rome. Garcia, S.M., Staples, D.J., 2000. Sustainability reference systems and indicators for responsible marine capture fisheries: a review of concepts and elements for a set of guidelines. Marine Freshwater Res. 51, 385–426. Kamal, N.Q., 2000. Integrated pest management (IPM) in practice: I. Impact of IPM practices on rice fauna, II. Impact of farmers field schools (FFS) and sustainability of IPM. Bangladesh Rice Research Institute, Gazipur, Bangladesh. Laloë, F., 1985. Etude de la précision des estimations des captures et prises par unité d’effort obtenues à l’aide du système d’enquètes sur la pêche artisanale au centre de recherches océanographiques de Dakar Thiaroye. Doc. Scient. Cent. Rech. Océano. Dakar Thiaroye no. 100, 36 pp. Laloë, F., Pech, N., Sabatier, R., Samba, A., 1998. Model identification for flexible multifleet-multispecies fisheries: a simulation study. Fish. Res. 37, 193–202. McCullagh, P., Nelder, J.A., 1989. Generalized Linear Models, 2nd Edition. Monographs on Statistics and Applied Probability no. 37. Chapman & Hall, London.
Mullon, C., Piron, M., 1998. Sur la méthodologie de mise en place des observatoires socio-économiques. In: Laloë, F., Perrier, X. (Eds.), De l’Observation à l’Analyse, Implication de la Biométrie dans les Pays en Développement. Société Française de Biométrie 15, pp. 61–77. O’Connor, M., 1999. Dialogue and debate in a post-normal practice of science: a reflexion. Futures 31, 671–687. Pechart, 1982. Les enquêtes sur la pêche artisanale au CRODT. Arch. Cent. Rech. Océano. Dakar Thiaroye no. 112, 28 pp. Pella, J.J., Tomlinson, P.K., 1969. A generalized stock production model. Bull. Int. Am. Trop. Tuna Comm. 13, 419–493. Pelletier, D., Ferraris, J., 2000. A multivariate approach for defining fishing tactics from commercial catch and effort data. Can. J. Fish. Aquat. Sci. 57, 51–65. Röling, N., 1994. Platforms for decision-making about ecosystems. In: Fresco, L.O., Stroosnijder, L., Bouma, J., van Keulen, H. (Eds.), The Future of the Land. Mobilising and Integrating Knowledge for Land Use Options. Wiley, New York, pp. 385–393. Roqueplo, P., 1997. Entre savoir et décision, l’expertise scientifique. Sciences en questions, Inra éditions Paris, 112 pp. Schaefer, M.B., 1957. A study of the dynamics of the fishery for yellow fin tuna in the Eastern Tropical Pacific Ocean. Bull. Int. Am. Trop. Tuna Comm. 2, 247–585. Scherrer, B., 1983. Techniques de sondage en écologie. In: Frontier, S. (Ed.), Stratégies d’Échantillonnage en Écologie. Masson, Paris et PUL, Que., pp. 63–162. Soceco-Pechart, 1982. Recensements de la pêche artisanale maritime au Sénégal, Avril et Septembre 1981. Doc. Scient. Cent. Rech. Océano. Dakar Thiaroye no. 83. Yates, F., 1981. Sampling Methods for Censuses and Surveys, 4th Edition. Griffin, London.