- Email: [email protected]
A new approach to large-scale effectiveness evaluation
Correspondence
more-or-less inactive in providing benevolent functions.3 The Maoist insurgents expanded their support base in this context, protecting tribal people from the rapacious state and providing basic services.4 Since 1981, Binayak Sen worked to improve public health in tribal areas, and his activities have encompassed social and political-economic issues.5 In 2005, Sen documented the brutality of a counterinsurgent militia that was supported by the state and had strong links to mining corporations.3 This put him in direct opposition to the neocolonial state, laying the ground for his persecution. These points show that a full understanding of India’s health crisis— and Sen’s incarceration—must appreciate the political and economic forces that shape the social determinants of health. We declare that we have no conflicts of interest.
*Jonathan Kennedy, Lawrence King [email protected] Department of Sociology, Free School Lane, Cambridge CB2 3RQ, UK 1 2
3
4
5
The Lancet. Binayak Sen’s conviction: a mockery of justice. Lancet 2011; 377: 98. International Institute for Population Sciences. National family health survey 3, 2005–06. Mumbai: IIPS, 2007. Sundar N. Subalterns and sovereigns: an anthropological history of Bastar, 1854–2006. Oxford: Oxford University Press, 2007. Planning Commission of India. Development challenges in extremist affected areas: report of an expert group. New Delhi: Government of India, 2008. Doctors in Defence of Dr Binayak Sen. Indian doctor in jail: the story of Binayak Sen. New Delhi: Promilla, 2008.
A new approach to large-scale effectiveness evaluation We endorse the view of Cesar Victora and colleagues (Jan 1, p 85)1 that traditional evaluation designs are limited in today’s complex global health environment. The national evaluation platform approach proposed will require the linking of www.thelancet.com Vol 377 April 16, 2011
multiple data sources at the district level to permit analysis of programme effectiveness. Geographic identifiers are a key to linking data sources from both vertical programme and more integrated information systems. In Kenya, work is underway to develop a two-pronged approach that addresses both the organisational and technical facets of linking data sources by use of geography in the context of programmes for orphans and vulnerable children. Organisationally, the Government’s Department of Children’s Services will work with their primary programme supporters (including donors) to develop a consensus on the sharing of information that will empower child welfare officers and service providers at the district and subdistrict levels. Technically, stakeholders will develop a spatial data model that will allow data sharing across programmes. The data model will not require changes within the vertical reporting structures, but will enable interoperability between systems and allow the Department of Children’s Services to know who is doing what where. Geographic identifiers also provide the opportunity for production of maps and our experience in other countries suggests that maps can be a powerful motivation for strengthening data quality to support such mapping for decision making. We believe that such efforts compliment the proposed national evaluation platform approach and advocate for explicit attention to the spatial data infrastructure as the concept evolves. We declare that we have no conflicts of interest.
*John Spencer, Charles Pill, Siân Curtis, Edward Kunyanga [email protected] MEASURE Evaluation, Carolina Population Center, University of North Carolina, Chapel Hill, NC 27516, USA (JS, SC); MEASURE Evaluation, Washington, DC, USA (CP); and ICF Macro, APHIA II Evaluation, Nairobi, Kenya (EK) 1
Victora CG, Black RE, Boerma JT, Bryce J. Measuring impact in the Millennium Development Goal era and beyond: a new approach to large-scale effectiveness evaluations. Lancet 2011; 377: 85–95.
With parallel programmes and many agents operating in intervention and control areas, assessments of the effectiveness of large-scale programmes has become increasingly complex. Recognising this complexity, Cesar Victora and colleagues1 suggest a new approach. The district-based design proposed is welcome, and its value is multiplied when its use extends beyond comparative effectiveness assessment. With little progress towards the Millennium Development Goals in sub-Saharan Africa, complementary institutional capacity is necessary. Trained district multidisciplinary teams need to be put in place for diagnosis before interventions; relevant crosssectoral interventions need to be designed; and supervision, monitoring, and information need to be shared to ensure accountability.2,3 Additionally, an independent national quality-control mechanism is justified. Such a body should have the authority to approve the relevance of the programme design and the riskreduction process.4 Ex-ante quality assurance mechanisms reduce the risk of starting ineffective programmes that subsequently make poor use of evaluation resources. For instance, in the UNICEF Accelerated Child Survival and Development programme, execution of scheduled interventions such as exclusive breastfeeding for the initial 6 months was incomplete; moreover, implementers reported little or no focus on nutrition with feedback to generate knowledge within communities during programme execution.5 Limited data for diagnosing dominant causes of chronic child undernutrition (eg, infections vs absolute deficits in dietary energy) in intervention and potential control areas, meant uncertainty of differential effects of cross-sectoral preventive interventions after the assessment. Prevention of the persistently high prevalence of stunting and micronutrient deficiencies across sub-Saharan Africa—within the relevant institutional capacity—remains a priority in reducing 1317
more-or-less inactive in providing benevolent functions.3 The Maoist insurgents expanded their support base in this context, protecting tribal people from the rapacious state and providing basic services.4 Since 1981, Binayak Sen worked to improve public health in tribal areas, and his activities have encompassed social and political-economic issues.5 In 2005, Sen documented the brutality of a counterinsurgent militia that was supported by the state and had strong links to mining corporations.3 This put him in direct opposition to the neocolonial state, laying the ground for his persecution. These points show that a full understanding of India’s health crisis— and Sen’s incarceration—must appreciate the political and economic forces that shape the social determinants of health. We declare that we have no conflicts of interest.
*Jonathan Kennedy, Lawrence King [email protected] Department of Sociology, Free School Lane, Cambridge CB2 3RQ, UK 1 2
3
4
5
The Lancet. Binayak Sen’s conviction: a mockery of justice. Lancet 2011; 377: 98. International Institute for Population Sciences. National family health survey 3, 2005–06. Mumbai: IIPS, 2007. Sundar N. Subalterns and sovereigns: an anthropological history of Bastar, 1854–2006. Oxford: Oxford University Press, 2007. Planning Commission of India. Development challenges in extremist affected areas: report of an expert group. New Delhi: Government of India, 2008. Doctors in Defence of Dr Binayak Sen. Indian doctor in jail: the story of Binayak Sen. New Delhi: Promilla, 2008.
A new approach to large-scale effectiveness evaluation We endorse the view of Cesar Victora and colleagues (Jan 1, p 85)1 that traditional evaluation designs are limited in today’s complex global health environment. The national evaluation platform approach proposed will require the linking of www.thelancet.com Vol 377 April 16, 2011
multiple data sources at the district level to permit analysis of programme effectiveness. Geographic identifiers are a key to linking data sources from both vertical programme and more integrated information systems. In Kenya, work is underway to develop a two-pronged approach that addresses both the organisational and technical facets of linking data sources by use of geography in the context of programmes for orphans and vulnerable children. Organisationally, the Government’s Department of Children’s Services will work with their primary programme supporters (including donors) to develop a consensus on the sharing of information that will empower child welfare officers and service providers at the district and subdistrict levels. Technically, stakeholders will develop a spatial data model that will allow data sharing across programmes. The data model will not require changes within the vertical reporting structures, but will enable interoperability between systems and allow the Department of Children’s Services to know who is doing what where. Geographic identifiers also provide the opportunity for production of maps and our experience in other countries suggests that maps can be a powerful motivation for strengthening data quality to support such mapping for decision making. We believe that such efforts compliment the proposed national evaluation platform approach and advocate for explicit attention to the spatial data infrastructure as the concept evolves. We declare that we have no conflicts of interest.
*John Spencer, Charles Pill, Siân Curtis, Edward Kunyanga [email protected] MEASURE Evaluation, Carolina Population Center, University of North Carolina, Chapel Hill, NC 27516, USA (JS, SC); MEASURE Evaluation, Washington, DC, USA (CP); and ICF Macro, APHIA II Evaluation, Nairobi, Kenya (EK) 1
Victora CG, Black RE, Boerma JT, Bryce J. Measuring impact in the Millennium Development Goal era and beyond: a new approach to large-scale effectiveness evaluations. Lancet 2011; 377: 85–95.
With parallel programmes and many agents operating in intervention and control areas, assessments of the effectiveness of large-scale programmes has become increasingly complex. Recognising this complexity, Cesar Victora and colleagues1 suggest a new approach. The district-based design proposed is welcome, and its value is multiplied when its use extends beyond comparative effectiveness assessment. With little progress towards the Millennium Development Goals in sub-Saharan Africa, complementary institutional capacity is necessary. Trained district multidisciplinary teams need to be put in place for diagnosis before interventions; relevant crosssectoral interventions need to be designed; and supervision, monitoring, and information need to be shared to ensure accountability.2,3 Additionally, an independent national quality-control mechanism is justified. Such a body should have the authority to approve the relevance of the programme design and the riskreduction process.4 Ex-ante quality assurance mechanisms reduce the risk of starting ineffective programmes that subsequently make poor use of evaluation resources. For instance, in the UNICEF Accelerated Child Survival and Development programme, execution of scheduled interventions such as exclusive breastfeeding for the initial 6 months was incomplete; moreover, implementers reported little or no focus on nutrition with feedback to generate knowledge within communities during programme execution.5 Limited data for diagnosing dominant causes of chronic child undernutrition (eg, infections vs absolute deficits in dietary energy) in intervention and potential control areas, meant uncertainty of differential effects of cross-sectoral preventive interventions after the assessment. Prevention of the persistently high prevalence of stunting and micronutrient deficiencies across sub-Saharan Africa—within the relevant institutional capacity—remains a priority in reducing 1317