The impact of household food consumption data collection methods on poverty and inequality measures in Niger

Food Policy xxx (xxxx) xxx–xxx

Contents lists available at ScienceDirect

Food Policy journal homepage: www.elsevier.com/locate/foodpol

The impact of household food consumption data collection methods on poverty and inequality measures in Niger ⁎

Prospère Backiny-Yetna , Diane Steele, Ismael Yacoubou Djima Living Standards Measurement Study, Development Research Group, The World Bank, 1818 H Street NW, Washington, D.C. 20433, USA

A R T I C L E I N F O

A B S T R A C T

Keywords: Poverty measurement Consumption Expenditure Survey design

Many countries are faced with the problem of monitoring poverty indicators when different food data collection methodologies have been used in national household surveys over the years. This paper provides a comprehensive analysis of this problem in the case of Niger. The paper assesses the impact of three methods of food data collection on the welfare distribution, and poverty and inequality measures in Niger. The study leverages a food consumption experiment to evaluate the three methods of food data collection implemented in the country’s most recent national household surveys. The first method was 7-day recall, the second was usual month, and the third was 7-day diary. The study finds that there was a large difference in measures of consumption and poverty between the first two methods (which yielded similar results) and the 7-day diary method. Annual per capita consumption from the 7-day recall method was, on average, 28 percent higher than that from the 7-day diary method. This gap exists not only at the mean of the distribution, but at every level. The observed differences in measured annual per capita consumption leads to differences in poverty and inequality measures even when alternate poverty lines are used.

1. Introduction Many countries have used different methods of collecting data to measure poverty, and each of these methods can influence computed poverty levels. Thus, when comparing poverty indicators over time, it is sometimes unclear whether poverty measurements differ because of differences in the well-being of the population or because of differences in the survey design methods used. Most countries use either income or consumption expenditure as indicators of monetary poverty; each indicator has advantages and disadvantages as a measure of living standards. Income shows the real flow of resources to a household at a particular point in time, and the ability to assign diverse sources of income to individual members of the household can allow for some analysis of intra-household inequality. However, income is very difficult to measure in developing economies, where most of the active population derives income from agriculture and other non-agricultural, self-employed activities, which are rarely documented. Moreover, income can fluctuate from year to year due to shocks, particularly in rural agricultural societies. Consumption, on the other hand, is smoother and less variable than income and is a more robust method of ranking households (Deaton and Zaidi, 2002). Thus, consumption (food and non-food), has been the standard variable by which to measure monetary poverty in much of the developing world.

⁎

Food consumption is a key component of welfare measure. As such, a great deal of research has been devoted to the analysis of methods of collecting food consumption data and their potential flaws. Methods can differ in terms of approach (diary or recall), reference period used, and food items considered (in the case of recall). Each of these elements affects the perceived distribution of expenditure, and, therefore, computed poverty levels (Beegle et al., 2012; Lanjouw and Lanjouw, 2001; Tarozzi, 2007). Using Niger as a case study, this paper highlights some of the difficulties involved in generating comparable poverty indicators when there are differences in food consumption data collection methods. In 2005 and 2007/08, the National Institute of Statistics (INS) of Niger implemented two national household surveys that have been used to measure and monitor poverty and assess the impact of public policies on the poor. The 2005 survey, the Core Welfare Indicator Questionnaire (QUIBB), collected food consumption information via the “usual month” method for a comprehensive list of food items. The usual month method consists of ascertaining the usual monthly expenditure for each item and the number of months the item was consumed in the past 12 months. The 2007/08 survey, the National Household Income and Expenditure Survey (ENBC), collected food consumption information via a “7-day diary” method. The “7-day diary” method consists in theory of a self-administered instrument in

Corresponding author. E-mail addresses: [email protected] (P. Backiny-Yetna), [email protected] (D. Steele), [email protected] (I. Yacoubou Djima).

http://dx.doi.org/10.1016/j.foodpol.2017.08.008

0306-9192/ © 2017 Published by Elsevier Ltd.

Please cite this article as: Backiny-Yetna, P., Food Policy (2017), http://dx.doi.org/10.1016/j.foodpol.2017.08.008

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

which households are asked to register all the food consumed by the members as the food is being consumed. As explained later in the paper, the context of Niger imposed practical adaptations to this method. In 2011, the INS decided to institute a third survey, to be used as a baseline for future poverty monitoring. The National Survey of Household Living Conditions and Agriculture (ECVMA), based on Living Standards Measurement Study (LSMS) surveys, was coordinated with Niger’s National Strategy for the Development of Statistics to support a new round of poverty estimates. The data collection method used for the ECVMA was the “7-day recall” approach, which consists of ascertaining the monthly quantity and expenditure of food items consumed in the past 7 days. The original idea was to use the same “7-day diary” method used in the ENBC to better compare poverty indicators between the two surveys, but this was not possible due to logistical, cost, and efficiency reasons. Thus, the 7-day recall approach was chosen because the methodology was close to that of the 7-day diary method, and it was assumed that it would produce more comparable results than if the usual month approach was used. In and of themselves, none of the above methods for collecting food consumption data are incorrect. However, the literature makes clear that the design of the questionnaire can influence the data collected (see Section 2). Thus, when comparing poverty indicators over time, it is uncertain whether poverty measurements differ because of differences in the well-being of the population or because of differences in the methods used. A comparison of the three survey methods used in Niger in a randomized control setting seeks to help answer to this conundrum. This paper discusses the results of a food consumption experiment conducted as part of the pilot survey of the 2011 ECVMA. The objective of the experiment was to assess the extent to which the differences in poverty indicators in Niger could be attributed to differences in the three different consumption data collection methods used. Specifically, the experiment replicated the methods of food consumption data collection used in the ECVMA (7-day recall), the QUIBB (usual month), and the ENBC (7-day diary) to see how they impacted poverty measures. We find there was a large difference in measures of consumption and poverty between the first two methods (which yielded similar results) and the 7-day diary method. The annual per capita consumption from the 7day recall method was, on average, 28 percent higher than that from the 7day diary. Obviously, these differences lead to differences in poverty figures, and any analysis of poverty trends using different survey methods that does not correct for changes in method may lead to errors. The rest of the paper is organized as follows: Section 2 presents a literature review; Section 3 provides a description of the data, including the way the experiment was implemented; and Section 4 explores the impact of the data collection method used on the perceived distribution of economic welfare during the period reviewed and discusses the consequences on poverty and inequality. Section 5 concludes.

program in the early 1980s (Saunders and Grootaert, 1980). Expenditure data (and particularly food expenditure data) can be collected using either diary or recall methods. Each of these survey designs presents specific challenges. With the diary approach, a recording period must be established (a week, a month, or longer). With the recall method, a list of items and the recall period must be determined. For both methods, the time of year when the data collection occurs can be an issue, unless data are collected all year long. The diary method, if properly implemented, can yield results closest to actual levels of household food consumption. In theory, diaries are meant to collect data on a daily basis, and are considered most accurate for overall household consumption. In practical terms, however, there are important design decisions that must be made. First, there needs to be a respondent in the household who is literate and can record the entries in the diary. If no one in the household is literate, the interviewers must assist in compiling the diary, spending more time helping household members, which blurs the line between a diary and a recall survey (Beegle et al., 2012; Deaton and Grosh, 2000). Second, diaries must be left with the household and picked up after the recording period is completed. This poses logistical problems for the interview teams, who must ensure that someone collects the diaries and sends them for processing. Third, the use of a diary alters procedures for interviewing. It reduces the amount of time that the interviewer spends interviewing the household, but may increase the amount of time that the interviewer spends traveling since an additional trip must be made to the household to pick up the diaries. In testing the accuracy of data collected from diaries, several studies analyzed changes in recording over time. (McWhinney and Champion, 1974) observed higher first-week expenditures in Canada; first-week expenditures averaged 8.3 percent above second-week expenditures, and that has come to be accepted as a fact of life in record-keeping surveys. (Wiseman et al., 2005) showed that two-week diaries provide satisfactory estimates for food consumed at home, but are deficient in records of food consumed outside of the home. In addition, missing or unclear data may be difficult to resolve. If researchers must go back to clarify entries with respondents, the data soon become retrospective and subject to recall biases. Using the recall method for collecting consumption module in Living Standards Measurement Study (LSMS) surveys is a common practice. There are known difficulties with recall periods. For example, (Gibson, 2002) showed that recall methods have measurement errors that are correlated with household size. As household size increases, it becomes harder for survey respondents to accurately recall expenditures on food. A key parameter when designing a recall module is the period as it affects the perceived distribution of consumption. The choice of the ideal recall period is among the most important and difficult design issues for the consumption module. Longer recall periods are better than shorter ones for measuring the distribution of consumption because averaging consumption over many days eliminates the randomness of some of the household’s day-to-day purchases that have nothing to do with its standard of living (Deaton and Grosh, 2000). However, people find it harder to remember more distant events; longer reporting periods lead to more forgetfulness for common purchases like food (Deaton, 2001). It is a well-accepted assumption that the longer the recall period, the greater the likelihood of recall errors, but the longer the recall period, the more possible it is to cover a larger sample of transactions for a given number of interviews, and therefore for a given field cost (Scott and Amenuvegbe, 1990). Shorter recall periods may help respondents report more accurate information, but there is also the problem of “telescoping,” in which respondents report events that lie outside the reference period. With telescoping, the more frequent the event, the greater the likelihood of confusion about dates (Bradburn, 2010). This means that frequently purchased items may be recorded in a recall interview even if not purchased during the specific recall period. The recall period that yields the greatest accuracy will vary with the nature of the goods (Friedman

2. Survey design and consumption data: A literature review Measurement issues are at the heart of data collection. Regardless of the information being collected—employment, income, expenditures, mortality, etc.—the way that the data are collected matters for the use of those data. Known measurement issues include the level at which the data are collected (individual or household), the period of the year in which data are collected (employment and some other variables are affected by seasonality), and the person providing the information (the individual or a proxy respondent). In the case of welfare measurement, there has been an ongoing debate over the best method of collecting expenditure information since the inception of the Living Standards Measurement Study (LSMS)1 1

For more information on the LSMS program, visit www.worldbank.org/lsms.

2

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

EAs (24 in Niamey, 12 in urban Tillabéri, and 12 in rural Tillabéri). In the second stage, we randomly selected households from updated household listings in the selected EAs: 9 households from each EA in Niamey and 9 households from each EA in urban Tillabéri and rural Tillabéri Three types of household questionnaires were used, each with a different version of the food consumption module designed to replicate previously used survey instruments in Niger. Questionnaire 1 (7-day recall, as in the ECVMA) included the complete household questionnaire that was subsequently used in the full ECVMA. Questionnaire 2 (usual month, as in the QUIBB) and Questionnaire 3 (7-day diary, as in the ENBC) had lighter household questionnaires that excluded the health, non-agricultural household enterprises, and non-wage revenues modules since they did not contribute to the calculation of total household consumption expenditure. Households were randomly assigned to one of the three types of questionnaires, with one-third of households in each EA assigned to each type. Thus, in all three strata (capital city, other urban, rural), onethird of households (72 households) received each type of questionnaire. Data collection was organized into teams. Each interviewer administered the same number of questionnaires of each type. This ensured that enumerators with a certain level of ability were not assigned to a specific method of data collection, which could introduce bias into the results. When administering the 7-day recall and the usual month questionnaires, the interviewer made a single visit to the household to complete the entire questionnaire. For the usual month approach, this meant collecting information on the usual monthly food consumption for each item during the past 12 months and the number of months the item was consumed. In both questionnaires, the consumption module consisted of 107 identical food items. Because the literacy rate is low in Niger (29.5 percent in 2011 according to the ECVMA main survey), the 7-day diary approach consisted of the interviewer conducting daily interviews on seven consecutive days, rather than asking respondents to record daily expenditures in a personal notebook. Therefore, the method used in the experiment was not truly a 7-day diary approach, it was in effect a one-day recall carried out over seven days.2 Nevertheless, we refer to it as a 7-day diary as it is meant to replicate as much as possible a 7-day diary. The experiment was conducted in February and March, when the contre-saison was coming to an end and the planting season had not yet started.3 We expected that some food items might not be available at that time of the year and might not appear in data related to the 7-day recall and the 7-day diary, but would be present in the data related to the usual month method. For the 2011 ECVMA, the survey management team revised the list of food items from the 2005 QUIBB and the classification used in the 2007/08 ENBC (see Appendix Table A2).4 This new list was used during the experiment for the 7-day recall and the usual month, and was included as an annex in the interviewer manual to be used to assign codes for the 7-day diary (there is no pre-printed list of food items for the diary method). The final sample for the pilot survey was well covered, with only a small percentage of replacement households (2.6 percent). In the end, 627 households were included. Among those 627 questionnaires, 207 were in Niamey, 207 were in urban Tillabéri, and 213 were in rural Tillabéri. By type of questionnaires, there were 205 for the 7-day recall

et al., this issue), suggesting that recall periods must be tailored to the different food groups. Different survey methods affect estimates of the composition of expenditures as well as estimates of average expenditures, which play an important role in the calculation of poverty lines. (Tarozzi, 2007) cited studies of poverty comparisons in India drawn from several crosssection surveys with changes in recall periods, showing that when comparing different surveys over time, compensation methods must be used to correct for changes in method. The study showed that in urban areas, the poverty decline was consistent with the official estimates from the Office of Statistics, but in rural areas, one-third of the measured decline in poverty was a statistical artifact. In Tanzania, (Friedman et al., this issue) decomposed household consumption estimates obtained through different survey designs into item consumption incidence and consumption value. The results showed that various survey designs exhibit widely differing error decompositions, which suggests great difficulties in comparing food consumption estimates across surveys with different designs. In El Salvador, (Jolliffe, 2001) compared poverty numbers using different questionnaire designs and found estimated levels of consumption were highly sensitive to questionnaire design. The study indicates there are differences in the calculation of poverty based on the different questionnaire designs. Measures of absolute poverty are significantly affected by differences in measured consumption levels. Measures of relative poverty do not show significant differences based on the questionnaire design, but the geographic distribution of relatively poor people is significantly different across the different questionnaire designs. As the body of research presented in this sections shows, food data collection methods are subject to contrasting types of survey response error and therefore the resulting consumption estimates confound welfare comparisons (Friedman et al., this issue). 3. The data The data used in this paper come from an experiment conducted as part of the 2011 ECVMA pilot survey. In addition to testing the questionnaire and the data-entry program to be used in field work, the ECVMA pilot survey aimed to evaluate the quality of the food consumption data-collection methods used in previous surveys in Niger. The data of the ECVMA pilot was collected as an experiment to numerically assess the implications of the use of different food data collection instruments in national household surveys over the years on measures of food consumption and poverty in Niger. 3.1. Description of the experiment The survey instruments for the experiment consisted of three questionnaires. The household questionnaire collected information on household demographics, education, health, anthropometrics, employment, non-farm enterprises, dwelling, durable goods, access to social services, non-salary income, transfers and remittances, and food security. In addition, food consumption and non-food expenditure information was collected. There was an agriculture and livestock questionnaire, and a community questionnaire to collect information about the areas around the households selected for the survey, on topics such as access to infrastructure and availability of services. The community questionnaire also collected price information that can be used to make regional adjustments to the welfare measure. For a pilot survey, a nationally representative sample would have been too dispersed and would have considerably increased the costs of data collection (especially transportation costs). Thus, the pilot survey was limited to Niamey (urban area) and Tillabéri (urban and rural areas). The final sample size of 648 households was drawn randomly from the 2001 population census, excluding enumeration areas (EAs) already selected for the main survey. A two-stage random sample strategy was used. In the first stage, we randomly drew the

2 This is slightly different from the methodology used in ENBC where, for the purpose of nutrition analysis, the interviewer also had to weigh the quantity of food consumed in the morning and in the evening. 3 The contre-saison, or off-season agriculture, is the period after the harvest when there is still water available for small scale irrigation. 4 There were two food items, peanuts with shell (arachides en coque) and shelled peanuts (arachides décortiquées) in Questionnaire 1 (7-day recall) that did not appear in Questionnaire 2 (usual month). They were removed from the calculation of the consumption aggregate; they made up less than 0.2 percent of food consumption.

3

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

Table 1 Household and head of household characteristics by type of questionnaire. Source: ECVMA Pilot survey 7-day recall

Usual month

7-day diary

Statistics

Mean

Std. Err.

Mean

Std. Err.

Mean

Std. Err.

χ2

Prob

Household Household size Number of childrena Number of adultsb Number of elderlyc

5.77 2.58 2.87 0.22

0.271 0.140 0.154 0.051

6.40 2.78 3.21 0.26

0.331 0.188 0.179 0.039

6.14 2.71 3.08 0.24

0.383 0.194 0.202 0.036

4.87 1.06 3.57 0.50

0.10 0.59 0.18 0.78

Head of household demographic Female (%) Average age (years) Married (%)

0.11 45.33 0.87

0.031 1.471 0.037

0.10 47.71 0.86

0.026 1.519 0.034

0.19 48.93 0.85

0.035 1.123 0.030

5.07 5.34 0.41

0.09 0.08 0.81

Head of household education None Primary Secondary & Post-secondary

0.65 0.18 0.17

0.049 0.036 0.033

0.68 0.17 0.15

0.041 0.033 0.028

0.69 0.16 0.15

0.038 0.033 0.030

0.57 0.17 0.26

0.75 0.92 0.88

Head of household labor market Labor force participation – 30 days Labor force participation – 12 months

0.83 0.89

0.049 0.029

0.79 0.84

0.045 0.039

0.73 0.82

0.043 0.038

4.90 4.23

0.10 0.13

Head of household occupation (12 months) Wage worker, except household personnel. Other dependent, incl. household personnel Self, agricultural Self, non-agricultural Unemployed, not in labor force Number of households

0.22 0.08 0.28 0.31 0.11 205

0.037 0.028 0.060 0.046 0.029

0.16 0.13 0.20 0.36 0.16 211

0.028 0.062 0.047 0.056 0.039

0.17 0.10 0.19 0.37 0.18 211

0.034 0.024 0.052 0.049 0.038

1.96 0.78 6.70 2.80 4.23 627

0.38 0.68 0.04 0.26 0.13

d

Note: The statistics were survey design corrected. The χ 2 test was used to test whether belonging to one questionnaire group has predictive power on household characteristics. a Children are defined as those 0–15 years old. b Adults are those 16–64 years old. c Elderly are those 65 years old and older. d We performed a join Wald test of equality.

method, 211 for the usual month method, and 211 for the 7-day diary.5 (See Appendix Table A3.)

percent (2007/08) lower welfare level than a household whose head was male (Institut National de la Statistique du Niger, 2008). The same study showed a very low negative correlation between the age of the head of the household and welfare, but one that was still significant. Therefore, differences in terms of the gender and age of the head of the household across the different questionnaire types had to be considered when comparing the distributions of welfare in this experiment. Though the sample was designed to avoid those types of disparities, not all errors can be avoided, especially given the relatively small sample size. As Table 1 shows, differences for other variables across the questionnaire types are less pronounced. The distributions of the education and labor market situations of household heads are very similar. The majority of household heads, regardless of the questionnaire type, have no education. Labor market participation is very high; in most cases, more than 80 percent of the heads have been involved in some type of activity during the previous 12 months. Most household heads work first as self-employed in non-agricultural activities, second as self-employed in agricultural activities, and third as wage workers. This employment structure, in which self-employment in non-agricultural activities is more important than self-employment in agricultural activities, is consistent with the fact that the sample was urban oriented. Although the experiment focused on food consumption, it is worthwhile to assess whether any differences we find in the consumption aggregate might come from non-food items. Across questionnaire types, we used an identical list of items, the same recall period, and the same methodology to calculate the non-food component of the consumption aggregates. Table 2 shows that there is no significant statistical difference in the non-food component of per capita expenditure between the three types of questionnaires. Nevertheless, the overall method used for the collection of data could have influenced

3.2. Descriptive statistics In this experimental setting, it was important to determine if any differences found in the consumption aggregate could be attributed to any factor other than the type of questionnaire. Table 1 shows the basic characteristics of the households and of the household heads, by type of questionnaire. The households seem similar in terms of basic characteristics, but there are significant differences in other key characteristics. Household size and household composition (number of children, adults, and elderly)—two key welfare explanatory variables—are consistent across questionnaire type. Regarding household size, the biggest difference is between the 7-day recall and diary approaches, which at 0.33 is not statistically significant. But the gender and age of the head of household exhibit some statistically significant differences across types of questionnaire. Heads of household from 7-day diary data are more often women and are also older than those from the two other questionnaires; 19 percent of the heads of household in the sub-sample are women versus approximately 10 percent from the two other questionnaires. This could present an issue for the validity of the results, since many studies show the impact of the gender of the head of the household, and sometimes of the age of the head, on the distribution of welfare. In fact, in Niger, when controlling for other variables, it was estimated that a household whose head was female had a 27 percent (2005) and 42 5 The distribution of non-response suggests that there was no particular effect by location, which can lead to selection bias. In Niamey, where there was concern over the ability to find people in their homes, only one household was replaced due to absence and refusal.

4

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

Table 2 Household non-food per capita expenditure by questionnaire type (FCFA). Source: ECVMA Pilot survey Distribution

7-day recall Usual month 7-day diary

Test equality of means

Mean

10th percentile

25th percentile

Median

75th percentile

90th percentile

SE

χ2

Prob

143343 144259 131033

36933 41337 40819

50336 53891 56492

85053 79448 80650

144047 125570 154493

199428 273200 293858

149867 19294 11185

0.87

0.649

Questionnaire 2, and lower after this percentile. Per capita consumption is significantly higher for 7-day recall and usual month than for the 7-day diary. The average per capita expenditure for the 7-day recall is 28 percent higher than the 7-day diary, while the average per capita expenditure for the usual month method is 33 percent higher than the 7-day diary. A Wald test confirmed that those differences are statistically significant. As Table 3 shows, differences between 7-day recall and usual month on the one hand, and 7-day diary on the other hand, are not only evident at the mean. There are differences along the distribution of welfare indicators. When comparing Questionnaires 1 and 3, which were assumed to be very close because both were based on 7-day food data collection, we see that the differences are very important at the left tail of the distribution, less important in the middle of the distribution, and somewhat important at the right side of the distribution. The gap of total household consumption per capita for 7-day diary compared to usual month equals 44 percent of average consumption at the 10th percentile, 22 percent at the 25th percentile, and only 10 percent at the median. When comparing 7-day recall and usual month, the gap is moderate and more consistent at every level of the welfare distribution. The gap is higher at the right tail of the distribution, reaching 22 percent at the 90th percentile, but does not exceed 15 percent at any other part of the distribution. Comparing the distribution of welfare indicators from the 7-day recall and the usual month methods, there is no significant difference. When comparing distributions from the 7-day recall and the 7-day diary, we find no difference at the 1 percent level, but the difference is clearly significant at the 5 percent level, confirming that distribution of welfare indicators derived from the 7-day recall is higher than that using the 7-day diary. There is an identical result when comparing data from the usual month method and 7-day diary. The differences on the entire distribution of welfare among the three types of questionnaires is assessed using stochastic dominance. Following (Araar and Duclos, 2013), distribution 1 dominates distribution 2 at order s over the range z −,z+ if and only if:

the data collected for the non-food expenditure section. In particular, the fact that data were collected on a single day for the first two methods (7-day recall and usual month) could have led to a better interview since the interviewer had more time to connect with the household. But this does not seem to have been the case. The small differences seen in Table 2 come from the usual sampling and data collection errors and do not invalidate the assertion that the three questionnaires were comparable on non-food expenditures. 4. Welfare distribution and poverty This section explores the impact of the data collection method on the distribution of economic welfare. We used per capita annual consumption, which was used in the past in Niger, as the welfare indicator. The consumption aggregate contained both food and non-food components. The food component included cash expenditures and the value of auto-consumption and gifts received by the household. The non-food module, which was the same in all three types of questionnaires, collected information on expenditures for non-food items over the course of 7 days, 30 days, 3 months, 6 months, and 12 months depending on the frequency with which items were usually bought. Imputed rent was estimated using a hedonic model for those households living in their own dwellings. A usage value for durable goods was also calculated. Health expenditures, which were part of the Niger consumption aggregate in the past, were excluded because they were collected only for the 7-day recall method. We also assumed that, because Tillabéri is very close to Niamey, the cost of living was identical across regions, so there was no spatial deflator. 4.1. Assessing welfare differences using simple tests and stochastic dominance When looking at mean per capita expenditure (Table 3), Questionnaire 1 (7-day recall) and Questionnaire 2 (usual month) exhibit similar results. On average, the difference in mean per capita consumption is not statistically significant, though it is interesting to note that the difference is not identical throughout the welfare distribution. For households at the bottom of the distribution (poorest) up to the 25th percentile, the average per capita expenditure is higher for

P1 (u;α ) < P2 (u;α ) ∀ u ∈ z −,z+for α = s−1 where P1 and P2 are the cumulative distribution function of distribution 1 and distribution 2, respectively.

Table 3 Household annual per capita expenditure (in FCFA) by questionnaire type. Source: ECVMA Pilot survey Distribution

Test equality of means

Mean

10th percentile

25th percentile

Median

75th percentile

90th percentile

SE

χ2

Prob

All 7-day recall Usual month 7-day diary

273,612 272,288 215,375

77,123 97,876 68,126

111,745 123,637 101,001

187,486 179,689 157,138

294,482 286,507 281,686

422,551 504,801 414,157

18,940 27,138 14,764

10.67

0.008

Food 7-day recall Usual month 7-day diary

130269 128029 84342

30423 38243 14926

48363 55070 28744

87339 93870 68023

169464 154225 121319

267786 233090 170594

8410 103012 6573

44.67

0.000

5

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

0

.2

.4

.6

.8

1

Fig. 1. Per capita consumption cumulative distribution function curves by type of questionnaire. Source: ECVMA Pilot survey

1500

81200

160900

240600

320300

400000

Per-capita Food Consumption (FCFA) 7-day recall

Usual month

7-day diary

Fig. 2. Per capita household expenditure by quintiles of various welfare variables. Source: ECVMA Pilot survey

900000 800000 700000 600000 500000 400000 300000 200000 100000 0 7-day recall

Usual month

7-day diary

Total per capita expenditure

7-day recall

Usual month

7-day diary

7-day recall

Non-food per capita expenditure Q1

Q2

Q3

Q4

Usual month

7-day diary

Assets index

Q5

If there is a value of u for which P1 (u;α ) = P2 (u;α ) and the value of P1 becomes greater than the value of P2, then neither of the distribution dominates the other on the interval z −,z+. The first order stochastic dominance test consists of comparing the plot of the cumulative distribution functions. If distribution 1 dominates distribution 2 using the first order stochastic dominance, then the poverty headcount is lower for distribution 1. On the plot of the food per capita expenditure (Fig. 1), the curve from the 7-day diary is always above the two other curves, meaning that it is dominated. The two other curves cross at different points of the distribution. We use the “dompov” procedure of the DASP software (Araar and Duclos, 2013) to formally assess the difference between the three pairs of distribution: (i) 7-day recall versus usual month, (ii) 7-day recall versus 7-day diary, and (iii) usual month versus 7-day diary (see Appendix Table A4). The results confirmed what appears in Fig. 1. There are at least 10 crossing points between the 7-day recall versus usual month, confirming that none of the two distributions dominates the other. We find two crossing points between the 7-day recall and the 7day diary, at the lower part of the cumulative curves. Beyond these two points the 7-day recall dominates the the 7-day diary. The usual month dominates the 7-day diary, with no crossing points among those two distributions. We have shown so far that the mean and the distribution of per-

capita expenditure is statistically different across the 7-day diary and the 7-day recall. The random assignment of households across questionnaires types makes it unlikely that the experiment was affected by external factors, except for head of household demographics (gender and age), as previously described. Those household characteristics are discussed later. However, it is interesting to further compare the welfare distribution across questionnaires using welfare indicators that are in theory not affected by the design of the food consumption section of the questionnaires. For this exercise, we use two welfare indicators, the first of which is per capita non-food consumption. The second indicator is an asset welfare indicator, which is constructed using 6 housing variables and 16 durable goods variables. All the variables are dummy variables. The variables are weighted by the inverse of the number of households having this good for 7-day recall. The distribution of the asset welfare indicator obtained as a result is close for the three questionnaires. We compare capita per expenditure across quintiles of the two welfare variables— per capita non-food consumption and the asset welfare indicator—alongside quintiles of the per capita expenditure itself (Fig. 2). This figure shows the two welfare indicators are positively correlated with total consumption per capita. More interestingly, at each level of the distribution of each of the two exogenous indicators, the mean of the household per capita consumption for 7-day recall and 6

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

household's disposable income is allocated to other products, and thus, final consumption levels would be close. But that assumption is based on the idea that all consumption comes from purchases, which is not the case. An important part of consumption comes from own-produced products and gifts, and this part is underestimated in the case of the scarcity of the product. In addition, there is the issue of consumer preference. The household may not be willing to spend money for a substitute product. All these factors may partly explain the fact that the level of household consumption indicated in the usual month approach was higher than that of the 7-day diary approach. One of our main findings is that average expenditure from the 7-day recall method is higher than that of the 7-day diary. One would expect the opposite—that the 7-day diary would yield the highest (and presumably closer to actual) levels of consumption despite the lack of overwhelming evidence provided in the literature (Friedman et al., this issue). The logic underlying this expectation is first that recording information using a diary was supposed to be more precise, while recoding information via recall (i.e., based on memory) could have led to the omission of products due to forgotten information. This expectation does not necessarily account for telescoping errors which could lead to higher reported expenditure Another potential source of difference between the 7-day diary and the 7-day recall is that as described in Section 3.1, there was no list of items for the 7-day diary method (it was only in the appendix for item-code recording). As such, in the case of the 7-day diary, enumerators asked open-ended questions (i.e., “What foods items were consumed in the household the day before?”). These questions were potentially more difficult to answer compared to prompting the respondent to specify whether or not each of the food items in the consumption module was consumed. In other words, without enough prompting one could expect that some interviewees had cognitive blocks in terms of recalling of the items eaten the day before for the 7-day diary. In the next paragraphs we examine the sources of the differences between the 7-day diary and the 7-day recall methods by looking at the incidence of reported expenditure. The total numbers of observations recorded for each of the two methodologies are compared in Table 5. There are nearly 14,000 observations for the 7-day diary—but a product can be purchased several times a week or several times a day. If we aggregate only one occurrence for each of the products consumed for each household, then there are 4121 observations for the 7-day diary, which is very close to the 4152 observations obtained for the 7-day recall method. Moreover, the same products are generally reported under the two survey designs. These findings contradict the results from Friedman et al. (this issue), who find that consumption incidence is largely downward biased for the 7-day recall when compared to diary results. The comparable numbers of food items observations across the 7-day diary and the 7-

the usual month method is greater than that for the 7-day diary. Looking at the quintiles of the non-food consumption, it is possible to see that the differences between the first two methods (7-day recall and usual month) and the third method (7-day diary) are very high on the tails of the distribution, and somewhat less in the middle of the distribution, but with the assets index, there is no such clear pattern. If we take the diary as the gold standard, then the results indicate poor households and wealthier households are the ones that overestimate most of their food consumption when using alternative methods. If the distribution of non-food consumption is statistically the same across the three methods, then differences in welfare distribution must come from differences in food consumption. For households reporting food expenditure using the 7-day recall and usual month methods, the results are similar for the total amounts spent at the mean. A comparison between the distributions of food per capita expenditure shows that for low food consumption up to the median, food per capita consumption is greater for the usual month method, and the reverse thereafter. All amounts reported by households using the 7-day diary are significantly lower than amounts reported using the other questionnaires. We now take a closer look at the differences in the composition of items consumed by households across questionnaires. We focus on the percentage of households that reported eating a particular food product, by type of questionnaire. We are particularly interested in the most-consumed products (cereals, meat and fish, cooking oil, eggs, and milk and dairy products), which amount to two-thirds of total food consumption expenditure. An interesting finding is demonstrated in Table 4. While the 7-day recall and 7-day diary methods show similar percentages of households consuming a specific product, the usual month method shows a higher percentages of households saying they consumed the items. This can be explained by the time of year in which the pilot survey was administered (i.e., February and March, when some products were not available) and by the method. Households indicating that they did not consume a specific food product during the last seven days, or during the collection of diary data may not have consumed that product because it was not available at the time of the survey. But if the product was available during the previous 12 months and households consumed it during that year, such data were collected via the usual month method. This is another drawback of the 7-day diary method: it is difficult to capture all products consumed at any given time of the year, even after harvests. With a shorter reference period, surveys cannot capture expenditure allocated to products that are rare or not available in the market at the time of data collection. Assuming households substitute available products for those that are rare, it could be argued that a

Table 4 Household annual expenditure by method of data collection and nature of products. Source: ECVMA Pilot survey

Maize Millet Rice Other cereals Bread/pasta Meat & fish Cooking oil Milk products Eggs & other Other food Total

Percent households consuming

Average expenditure

7-day recall

Usual month

7-day diary

7-day recall

Usual month

41.2 31.1 54.6 4.9 47.6 66.1 78.6 43.7 18.1 99.6 100

78.1 75.1 87.2 21.4 70.3 91.5 93.3 69.6 47.2 100 100

43.8 34.8 59.8 11.0 31.9 54.6 87.6 49.5 11.8 100 100

90,295 71,232 103,997 10,616 38,371 108,691 31,784 22,177 10,545 320,869 808,578

67,690 106,364 88,213 7,533 28,760 121,150 42,721 25,506 12,260 341,781 841,978

Average expenditure if expenditure for product group > 0

Share in total consumption

7-day diary

7-day recall

Usual month

7-day diary

7-day recall

Usual month

7-day diary

55,104 47,793 77,242 8,977 14,238 58,003 38,048 15,275 2,532 210,674 527,885

219,359 228,682 190,492 214,483 80,576 164,404 40,461 50,792 58,261 322,197 808,578

86,694 141,540 101,113 35,144 40,884 132,355 45,811 36,655 25,989 341,781 841,978

125,711 137,248 129,263 81,794 44,586 106,206 43,414 30,880 21,473 210,674 527,885

11.2 8.8 12.9 1.3 4.7 13.4 3.9 2.7 1.3 39.7 100.0

8.0 12.6 10.5 0.9 3.4 14.4 5.1 3.0 1.4 40.6 100.0

10.5 9.0 14.6 1.7 2.7 11.0 7.2 2.9 0.5 39.9 100.0

7

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

taken away from food. Expenditures for meals and non-alcoholic beverages taken away from food is recorded in Table 4 (item “Other foods”). Households in the 7-day recall report in average a statistically significant higher level of expenditures than the households in the 7day diary, even though the percentage of households reporting having consumed meals outside of the home is not significantly different between the two groups. Even with this drawback in the 7-day diary method, the possibility of an overestimation of household expenditure cannot be excluded when using the 7-day recall method as it is subject to telescoping error which were likely minimized by the frequent visits in the case of the 7day diary. To get a sense of whether the 7-day recall overestimate actual food consumption expenditure, we calculate the number of kilocalories per person per day from each of the two methodologies (Table 6). The idea underlying our analysis was that a person should consume around 2,1008 calories per day. If the results deviate too much from this standard, then the data must be unrealistic. The basket used (39 products) represents 80 percent of food consumption for the 7-day recall and 76 percent for the 7-day diary. The table shows there are 1,558 kilocalories for the 7-day diary and more than 2,442 for 7-day recall method. Adding to these figures an average number of calories for the part of the consumption not taken into account in the calculation (i.e., meals consumed outside the home, plus some less-consumed products for which prices were unavailable) the results indicate there are about 1,900 kilocalories for the 7-day diary and more than 2,700 kilocalories for the 7-day recall. The median calories intake obtained from both methods underestimate the 2100 kilocalories norm with the 7-day recall’s estimate slightly higher than the 7-day diary estimate. These figures show that in the specific case of Niger, the 7-day diary underestimates food consumption, while the 7-day recall method tends to overestimate it.

Table 5 Comparison of the number of observations and expenditure per methodology. Frequency

Expenditure Mean

SD

2128 2025 2050 1908 1843 1884 1980 13,818 4121

190.1 170.0 161.7 161.4 147.3 159.8 183.4 168.2 564.0

711.6 449.0 384.4 284.3 245.2 314.1 600.9 462.5 1195.8

7-day recall Total

4152

850.9

1985.1

Usual month Total

7487

2862.7

5190.3

7-day diary Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Total Total (single occurrence of products per household)

Source: ECVMA Pilot survey. Table 6 Comparison of kilocalorie Intake per Methodology of Data Collection.

Mean Median # Obs.

All sample

1% trim

5% trim

Winsoring 5%

7-day recall

7-day diary

7-day Recall

7-day diary

7-day recall

7-day diary

7-day Recall

7-day diary

2442 1408 213

1558 1363 212

2212 1408 207

1504 1363 208

1887 1407 192

1419 1363 192

2134 1408 213

1494 1363 212

Source: ECVMA Pilot survey.

day recall indicate that the underreporting isn’t caused by the cognitive bias of the lack of prompting in the 7-day diary administration. Another possible explanation of the difference in the estimates of consumption might be that some products were purchased, but not recorded. This is often the case for food consumed away from home, the recording of which is often deficient in the case of diaries. To evaluate this possibility, we examine the expenditures reported for the diary and the recall method. We find that the average expenditure per item with the 7-day recall approach was 52 percent higher than the 7-day diary. This result is in parallel with the number of observations for the 7-day diary, which decreased gradually from day 1 to day 5. Although the number of observations increased somewhat during the last two days of data collection, the number of observations in day 1 remained the highest overall.6 This indicates that fatigue of the respondent and/or the enumerator7 affected data collected for the 7-day diary. Another source of the discrepancy might be the respondent’s lack of knowledge of food purchases made for the household. It is likely that for products purchased multiple times a day (i.e., products purchased by different family members on the same day, such as doughnuts or bread) or meals consumed outside the household, the respondent would not have all the information (Wiseman et al., 2005). We cannot formally verify this assertion because the respondent was not identified in the questionnaires. However we are able to look at the expenditure for meals

4.2. More on welfare impact using regression analysis The preceding analysis shows that the choice of method used to collect data on food consumption results in significant differences in the welfare indicator as measured by per capita annual consumption. In this sub-section, we use linear regression techniques to further explore those differences. The first model is the regression of the logarithm of the total per capita food consumption on the type of questionnaire (Model 1). Because the randomized assignment of questionnaires types to household was successful, the only variable that we expect to affect food consumption is the type of questionnaire. Other factors such as the interviewer should have no effect. Since the type of questionnaire is a dummy variable, the model is no different from a simple test of means among the three methodologies. In the previous section, we noted that for two characteristics of the head of the household, namely gender and age, there are statistically significant differences across households assigned to the three methodologies. To control for these differences, we introduce the interaction between the type of questionnaire and each of those variables in a second set of regressions in the second model. So, in the second model, we introduce the interaction of the type of questionnaire with gender (Model 2). In the third model, we use age instead of the gender variable (Model 3). The last model includes both variables interacting with the type of questionnaire (Model 4). Formally, yi is the logarithm of the total per capita food consumption for household i, Qki is the questionnaire type (k = 1, 2, 3; with type 1 being the 7-day recall, type 2 the usual month and type 3 the 7-day diary), Femalei and Agei are respectively the dummy for female head of household and Age of the head of the household, and Ui is the error term. We estimate two models: Model 1:

6 The same trend in the number of observations per day was also observed during the 2007 ENBC in Niger. A decrease of 9 percent in the number of observations between day 1 of data collection and day 7 of data collection was recorded. 7 One type of error that is not investigated here but might explain the decrease of diary consumption from day 1 to day 7 is intentional error from enumerators or even respondents who rush to complete the questionnaire. The error would have been exacerbated by the 7-day daily visit structure.

8 FAO’s estimate for the average dietary energy consumption for the least developed countries for 2006–08 is 2120 kcal.

8

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

Table 7 Regression of per capita expenditure on type of questionnaire. Dependent variable: Log of per capita food expenditure Model 1 Questionnaire 7-day recall Usual month

ref 0.071 (0.09) −0.332** (0.08)

7-day diary Female Questionnaire#Female Usual month*Female

Model 2

Model 3

Model 4

0.131 (0.11) −0.299** (0.10) 0.329 (0.27)

-0.472 (0.41) −0.715 (0.44)

-0.546 (0.40) −0.764* (0.43) 0.437 (0.31)

−0.580* (0.32) −0.264 (0.39)

7-day diary *Female Age Questionnaire#age Usual month*age 7-day diary*age Constant # Observations

11.501** (0.11) 627

11.564** (0.09) 627

−0.010 (0.01)

−0.727** (0.34) -0.353 (0.44) −0.012* (0.01)

0.012 (0.01) 0.009 (0.01) 12.118** (0.33) 627

0.015* (0.01) 0.011 (0.01) 12.142** (0.31) 627

Source: ECVMA Pilot survey. Note: Standard errors in parentheses. *** Indicates significant difference compared with 7-day recall at 1%; ** at 5%; and * at 10% . 3

yi = α +

∑

experiment, the 7-day recall and the usual month method give higher values than the 7-day diary approach. They also allow us to robustly assess the difference due to the survey designs by capturing the effects of food consumption covariates that were not successfully randomized across questionnaire types. The next sub-section explores the impact on poverty and inequality.

βk Q ki + Ui

k= 1

and Model 4: 3

yi = α +

3

∑

βk Q ki + μFemalei + ρAgei +

k= 1

∑

γk Q ki ∗Femalei

k= 1

3

+

∑

4.3. Poverty and inequality

δk Q ki ∗Agei + Ui

k= 1

Changes in poverty rate and profile, not the level, are what often matter for the evaluation of public policies. Policy makers are more interested in the change of the poverty levels over time. Thus, it is important to have consistent poverty numbers over time. But as shown in the previous section, different methods of data collection result in different perceived distributions of welfare, and potentially different levels of poverty. In this study, we used three different poverty lines to compute standard poverty indicators: the poverty headcount, the poverty gap, and the squared poverty gap. The first poverty line (low) corresponds to $1 dollar per capita per day in PPP 1993; this gives 154,539 FCFA per capita per year in local currency. The second one (medium) is the national poverty line computed by the INS and The World Bank (The World Bank, 2012) using the 2005 QUIBB, inflated by the ratio of the consumption price index between 2005 and 2011. This second poverty line is 166,087 FCFA per capita per year (Institut National de la Statistique du Niger, 2008, 2006). The third poverty line (high) corresponds to $1.25 dollar per capita per day in PPP 1993, which gives 193,174 FCFA per capita per year in local currency. Since the results using the three poverty lines converged, we present only the one with the national poverty line. The results in Table 8 show important differences between the three methods for all poverty indicators. The poverty headcount is 8.5 percentage points higher with the 7-day diary compared to the 7-day recall, and 4.5 percentage points higher with the 7-day diary compared to the usual month method. Using a formal test, the difference between the poverty headcounts using the three methods is not statistically significant; but using pairwise tests, the difference between the 7-day

Models 2 and 3 are derived from Model 4 by removing, respectively, either the gender and the interaction between the type of questionnaire and the gender of the head of the household, or the age and the interaction between the type of questionnaire and age of the head of the household. The comparison between the three methodologies is based on tests on the estimates of the parameters β. The results of the regression are recorded in Table 7. When considering food consumption and the first regression (which has questionnaire dummies as the only explanatory variables) and taking the 7-day recall as the reference methodology, the 7-day diary has an estimated gap of -28 percent (exponential of −0.332), and the estimated parameter is statistically significant even at the 1-percent level. Always taking the same methodology as the reference, the difference with the usual month approach is not significant, confirming the results found using simple statistical tests. Introducing the possibility of the gender of the head of the household interacting with the type of questionnaire tells the same story, with on average a lag of 25.8 percent of the per capita food consumption from the 7-day diary methodology compared to the 7-day recall. When we remove gender and introduce the age of the head of the household, the difference between the food per capita from the 7-day recall and the 7-day diary is even higher, suggesting that part of the gap was captured by the head of the household age effect. But the gap from this regression seems large (53.4 percent) compared to the previous ones. The results from the regression confirm the assessments made from the analysis of the descriptive statistics: in the case of this Niger 9

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

three randomly formed groups. The first method was the 7-day recall, the second one was the usual month method and the third was a 7-day diary method. We found that there was a difference in the recorded distribution of welfare, between the 7-day diary and the usual month methods on the one hand and the 7-day diary method on the other hand. The gap in annual per capita consumption between the 7-day recall and the 7-day diary was estimated at the mean as being one-third of per capita annual consumption. The differences found between the 7-day recall method and the 7-day diary were due to overestimation of food consumption on the 7-day recall and underestimation on the 7-day diary. These differences lead to differences in poverty and inequality measurements even when using alternate poverty lines. As is the case for randomized experiments that are limited to a relatively small sample, the question of external validity of the results can be a concern if lessons are to be taken from this work. However, although the experiment was limited to Niamey and the nearby region of Tillabéri, there was a coverage of both rural and urban areas. In addition, the fact that the household composition and other demographic statistics calculated in the experiment do not widely differ from the statistics obtained from the 2011 ECVMA - see (Institut National de la Statistique du Niger, 2013)- provides relative confidence in the validity of the inferences that can be made from the results obtained here. As such, the fact that the paper finds that there are differences in poverty and inequality measures despite the use of alternate poverty lines is important, since trends are important in evaluating the impact of public policies on poverty. Given that various methods of food consumption data collection have different drawbacks; it is important for statistical agencies to be consistent in the choice of methodologies they use if they want to ensure comparability between surveys. In fact, the three methodologies used in this study were implemented in different surveys in Niger, blurring the real trend of poverty indicators, particularly when those trends are difficult to reconcile with economic growth. To obtain a better understanding of poverty trends in Niger and assess how the country has performed over time, it might be useful to revisit poverty trends in Niger. Yet revisiting the poverty trends in Niger is not enough; it is also important to prepare for and ensure the comparability of poverty measurements in the future. The 2011 ECVMA used the 7-day recall approach. It is essential to use the same method in the future, to ensure more robust comparison of poverty levels. Finally, because the magnitude of the differences in poverty measurements seems high, it might be useful to repeat this experiment in other countries facing the same problem (such as Guinea, Mali, Burkina Faso, etc.), with a larger sample. It would also be interesting to make these comparisons in cases where the 7-day recall and 7-day diary methods are implemented with more than one round during the year.

Table 8 Poverty Indicators by Type of Questionnaire Using the National Poverty Line (Niger).

Poverty headcount Poverty Gap Squared poverty gap

7-day recall

Usual month

7-day diary

Test

Mean

SE

Mean

SE

Mean

SE

χ2

Level

0.425

0.052

0.465

0.050

0.510

0.051

1.91

0.392

0.150 0.070

0.024 0.014

0.136 0.053

0.018 0.010

0.199 0.100

0.028 0.019

8.57 11.64

0.020 0.006

Note: The national poverty line is 166,087 FCFA per capita per year. Source: ECVMA Pilot survey. Table 9 Inequality indicators by type of questionnaire. Source: ECVMA Pilot survey 7-day recall

Gini Atkinson (1) Theil (1)

Usual month

7-day diary

Mean

SE

Mean

SE

Mean

SE

0.457 0.297 0.423

0.028 0.030 0.054

0.431 0.264 0.368

0.027 0.029 0.069

0.393 0.231 0.257

0.017 0.019 0.022

recall and the 7-day diary is significant at the 10% level. As for the other two poverty indicators, the gaps among the three methods go in the same direction, and the differences are even significant at the 1% level. One of the reasons some of the differences are not statistically significant even if they appear large in absolute value is that the standard errors are high; the size of our sample is only 200 households for each method. This probably exaggerates the magnitude of the differences. However, this should not undermine the credibility of research results, considering that other studies with higher sample-sizes have obtained similar findings. Not only did we find that poverty measures were not consistent across methods, inequality was also inconsistent. Table 9 presents three different inequality measures: the Gini index, the Atkinson index, and the Theil index for each type of questionnaires. The 7-day recall method, which has the highest level of expenditure and the lowest poverty indicators, exhibited the highest inequality, regardless of the inequality measure used. The differences between 7-day recall and usual month are not important, but the differences between the 7-day recall and the 7-day diary are larger: the gini index calculations results in a difference of 6 percentage point. up to 6 percentage points are merely differences in the method of data collection. Similar trends have been found in inequality measures in other countries when the methodology of data collection has changed between two surveys. In Niger, the Gini index decreased by more than 7 percentage points between 2005 (usual month method) and 2007/08 (7-day diary). It is now assumed that part, if not all, of this decrease can be attributed to difference in methods (Institut National de la Statistique du Niger, 2008).

Acknowledgements The authors would like to thank Alberto Zezza, Dean Jolliffe, two anonymous referees, and seminar participants at the World Bank for comments. The data for this research was collected as part of the Living Standards Measurement Study – Integrated Survey in Agriculture, for which financial support of the Bill and Melinda Gates Foundation is gratefully acknowledged. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.

5. Conclusions In this paper, we assessed the impact of three methods of food data collection (which were either previously used or were about to be used in national surveys in Niger) on welfare distribution readings, and on poverty and inequality measurements using a randomized experiment. We assigned one of the food data collection methods to each of the Appendix A. Experiment details See Tables A1–A4

10

11

recall HH1 recall HH2 month HH3 month HH4 diary HH5

Listing the EA

Day 1

Introduction

Introduction

Interview

Day 2

Daily food expenditures Household composition Daily food expenditures Household composition

Interview

Day 3

Daily food expenditures Education, Employment Daily food expenditures Education, Employment,

Day 4

Daily food expenditures Shocks, Non-food expenditures Daily food expenditures Shocks, Non-food Expenditures

Dwelling, Revenues, Transfers Daily food expenditures Dwelling, Revenues, Transfers

Day 6

Daily food expenditures

Day 5

Daily food expenditures Agriculture

Interview Daily food expenditures Agriculture

Day 7

Daily food expenditures Revision

Daily food expenditures Revision

Interview

Day 8

Daily food expenditures

Daily food expenditures

Day 9

Leave the EA

Day 10

Note: Each interviewer was responsible for interviewing six households per EA. Two households for each type of questionnaire. For 7-day recall and usual month questionnaires, the entire interview was administered in one day per household. For the 7-day diary questionnaire, seven daily visits were made to the household to collect daily food expenditure information and one or two other modules per day. The schedule was designed so that an interviewer could administer multiple households and multiple questionnaire types in a single day.

7-day diary HH6

7-day 7-day Usual Usual 7-day

Questionnaire/ household

Table A1 Organization of the interviewer work load.

P. Backiny-Yetna et al.

Food Policy xxx (xxxx) xxx–xxx

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

Table A2 Questionnaire contents. 7-day recall

Usual month

7-day diary

Socio-demographic Characteristics of household members* Education* Health Employment*

Socio-demographic Characteristics of household members* Education* Employment* Dwelling characteristics* Durable goods* Non-food expenditures* Transfers* Shocks and survival strategies* Food consumption – 12 month average

Socio-demographic Characteristics of household members* Education* Employment* Dwelling characteristics* Durable goods* Non-food expenditures* Transfers* Shocks and survival strategies* Food consumption – 7day daily diary

Non-agricultural enterprises Dwelling characteristics* Durable goods* Non- salary revenues Non-food expenditures* Transfers* Shocks and survival strategies* Food security Food consumption – 7-day recall Complement to food consumption of the last 7 days

All households received an agricultural questionnaire that was identical. A community questionnaire was applied in all enumeration areas. * Note: Modules for Socio-demographic characteristics of household members, Education, Employment, Dwelling characteristics, Durable goods, Non-food expenditures, Transfers, and Shocks and survival strategies were identical in all three questionnaire types.

Table A3 Distribution of surveyed households by type of questionnaire. Total

7-day recall

Usual month

7-day diary

Interviewed Niamey Tillabéri urban Tillabéri rural Total

207 207 213 627

69 66 70 205

70 69 72 211

68 72 71 211

Refusals/incomplete Niamey Tillabéri urban Tillabéri rural Total

4 6 7 17

1 2 2 5

1 2 4 7

2 2 1 5

Dropped as outliers Niamey Tillabéri urban Tillabéri rural Total

5 4 3 12

2 3 3 8

2 1 0 3

1 0 0 1

Note: Outliers are households having a daily per capita calorie intake greater than the mean plus three times the standard deviation.

Table A4 Stochatic dominance tests comparing the three distributions. Number of intersection

Critical poverty line

Case

7-day recall vs usual month 1 2 3 4 5 6 7 8 9 10

7660 12709.821 99708.336 108685.266 109250 112541.664 126950 285645.094 391510 762589.25

B A B A B A B A B A

7-day recall vs 7-day diary 1 2

2607.143 6430.952

A B

7-day-diary vs Usual month No intersection

12

Food Policy xxx (xxxx) xxx–xxx

P. Backiny-Yetna et al.

Institut National de la Statistique du Niger, 2008. Tendances, profil et déterminants de la pauvreté au Niger 2005–2007/08. Niamey. Institut National de la Statistique du Niger, 2006. Profile de Pauvreté: Questionnaire des Indicateurs de Base du Bien-être (QUIBB 2005), Orientations. Niamey. Jolliffe, D., 2001. Measuring absolute and relative poverty: the sensitivity of estimated household consumption to survey design 1. J. Econ. Soc. Meas. 27, 1–23. Lanjouw, J.O., Lanjouw, P., 2001. How to compare apples and oranges: poverty measurement based on different definitions of consumptio. Rev. Income Wealth 47, 25–42. http://dx.doi.org/10.1111/1475-4991.00002. McWhinney, I., Champion, H., 1974. The Canadian experience with recall and diary methods in consumer expenditure surveys. Ann. Econ. Soc. Meas. Ellipsis 3, 411–437. Saunders, C., Grootaert, C., 1980. Reflections on the LSMS Group Meeting (No. 10), LSMS Working Papers. Washington, D.C. Scott, C., Amenuvegbe, B., 1990. SOCIAL DIMENSIONS OF ADJUSTMENT IN SUB-Effect of Recall Duration on Reporting of Household Expenditures An Experimental Study in Ghana (No. 6), Social Dimensions of Adjustment (SDA) in sub - Saharan Africa working pape. Washington, D.C. Tarozzi, A., 2007. Calculating comparable statistics from incomparable surveys, with an application to poverty in India. J. Bus. Econ. Stat. 25, 314–336. http://dx.doi.org/10. 1198/073500106000000233. The World Bank, 2012. Niger: Investing for Prosperity – A Poverty Assessment. Washington, D.C. Wiseman, V., Conteh, L., Matovu, F., 2005. Using diaries to collect data in resource-poor settings: questions on design and implementation. Health Policy Plan. 20, 394–404. http://dx.doi.org/10.1093/heapol/czi042.

References Araar, A., Duclos, J.-Y., 2013. DASP: Distributive Analysis Stata Package. Beegle, K., De Weerdt, J., Friedman, J., Gibson, J., 2012. Methods of household consumption measurement through surveys: experimental results from Tanzania. J. Dev. Econ. 98, 3–18. http://dx.doi.org/10.1016/j.jdeveco.2011.11.001. Bradburn, N.M. (NORC/University of C., 2010. Recall Period in Consumer Expenditure Surveys Program, in: Bureau of Labor Statistics Consumer Expenditure Survey Methods Workshop, December 8–9. Alexandria, Virginia, pp. 1–13. Deaton, A., 2001. Counting the world’s poor: problems and possible solutions. World Bank Res. Obs. 16, 125–147. http://dx.doi.org/10.1093/wbro/16.2.125. Deaton, A. (Woodrow W.S. of P. and I.A. at P.U., Zaidi, S.) The W.B., 2002. Guidelines for Constructing Consumption Aggregates for Welfare Analysis (No. 135), LSMS Working Paper. Washington, D.C. Deaton, A., Grosh, M.E., 2000. Consumption. In: Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 Years of the Living Standards Measurement Study. The World Bank, Washington, D.C., pp. 91–134. Friedman, J., Beegle, K., De Weerdt, J., Gibson, J., this issue. Decomposing Response Errors in Food Consumption Measurement Implications for Survey Design from a Survey Experiment in Tanzania. Gibson, J., 2002. Why does the Engel method work? Food demand, economies of size and household survey methods. Oxf. Bull. Econ. Stat. 64, 341–359. http://dx.doi.org/10. 1111/1468-0084.00023. Institut National de la Statistique du Niger, 2013. Agriculture et Pauvreté au Niger: Etat des lieux. Niamey.

13