- Email: [email protected]
Evaluation of maternal health programs: approaches, methods and indicators
Evaluation of maternal health programs: approaches, methods and indicators J.L. Bobadilla
Evaluation of maternat health services is an essential function of any health system. Single or multiple aspects of a program can be evaluated but they must be clearly defined. To assess success of a program the dimensions of coverage. equity, quality. women’s satisfaction, ef”ciency and cost-effectiveness must be considered. These can be measured accordine to wane standard criteria, and various methods are &scribed. in&dine randomized clinical trials, randomized commkitv trials. before and
tial in&iries into motern~~deotbs. Indicok to assess effectiveness of o maternal health program may relate to structure, i.e., available resources or organizational arrangement: process. including changes in quantity of services provided: or outcome indicators that measure maternal mortality or morbidity.
Keywords: Evaluating maternity care; Research methodology; Safe motherhood. As part of the momentum aaincd bv the Safe tiotherhood Initiative, m&y programs are beinr! imulcmcated throughout the developing world claiming to bc pan of the crusade. Unfortunately only a few have documented any reductions of maternal morbidity and mortality [.5,6,10]. This may bc because
most of the programs have not been evaluated or their evaluation has been undertaken with inappropriate approaches and methods. Further, maternal health programs arc panicularly difticult to evaluate, since the final outcome, improvements in the health of mothers and newborns, for example, depends on the functioning of the whole system and not on only a single intervention, and there arc other determining factors, most importantly. the physique and health condition of the women and the socioeconomic status of their family [1,3]. Therefore, trying to relate a single provider or intervention to improvements on matciaa! sa+val is impossible, u&s: a!! other relevant elements of the health program and the characteristics of the women arc strictly controlled or held constant. Finally, maternal death is a rare event, even in the worst condition; therefore any attempt to measure mortality is faced with the need to study thousands of births to achieve signilicant comparisons between these rates [6]. This paper presents some of the main ap proaches that have been successful in evaluating health programs, discusses some of the available methods and their limitations, and suggests a classification of indicators that arc useful in assessment of maternal health pmgrams. Approaches to evaluate bcalth progrluns Evaluation of health services is an essential function of all good health systems. Common-
ly, evaluation refers to a stage of the planning cycle when routinely collected data are processed and interpreted to assess whether a set of specified services has achieved some stated objectives in a determined time period 191. More broadly the concept embraces all types of activities that compare the achievements of a given intervention with those of a predetermined standard. These definitions clearlv show that the purpose of evaluating services G to pass judgments as to the goodness of the services. It therefore requires that evaluators specify the characteristics of the program, the dimensions to be evaluated and the stan-
dard(s). The following is a brief review of the alternative ways of specifying these aspects, in the context of maternal health programs,
As obvious kit of health oroerams program I& 6een
may seem the evaluation often
f&s
because
the
poorly defined. The simplest program will consist of a single intervention, typically delivered by one provider (e.g., administration of a vaccine or a drug). But most programs involve several interventions that are applied, with similar purposes, to a specific population group, as is the case of antenatal care. More complex programs include referral of some patients to receive diagnostic and therapeutic services from other providers. For example, a maternal health program could encompass family planning, antenatal and delivery care, with the participation of traditional midwives, community nurses, professional midwives, and doctors with different training (among other providers). Typically such a program would require referral of some women to a hospital setting where essential obstetric functions are available. For the purpose of this workshop, to evaluate an innovative program that delegates obstetric functions to nurses or midwives, it is essential to define the program in terms of provider, setting, activities, interventions and, most importantly, expected outcomes.
There are at least six dimensions that are relevant to assess the snccess of a health program: coverage, equity, quality (technical), women’s satisfaction, efficiency and costeffectiveness.
Coverage. This refers to the proportion of the population that has reasonable access to the health program in question [12]. One of the most important goals of delegating resaonsibilities in obstetrics is certainlv to iocrease the coverap cf maternal he&b services. Maternal health care is strongly influenced by culture and individual or family preferences. The actual utilization of a maternal health program is thus determined by the amenities of the service, the content of care, and most importantly by the way in which providers attend the emotional needs of women who are pregnant and being delivered. Eg&. Although there is no single accepted delinition of equity, its most common eonnotation refers to the distribution of resources according to the health needs of the population Ill]. Delegation of obstetric respcmsibilities in a community can lead to a more equitable distribution if the extension in coverage benefits primarily the most needy socioeconomic groups, typically, the residents of rural areas and urban slums. &a@ ofcar.?. Changes in the health status attributable to health care depend on the technical auatitv of the services received 141. The limits of delegating responsibilities‘ ib obstetrics are largely determined by the tmining and performance of the personnel who will assume the responsibilities, as well as by the amount and appropriateness of the resources and technology at their disposal. Deficiencies in uerformance or resources can lead to standark of care which may be considered unacceptable. Women’s satisfaction. Maternal health ser-
ELtdlmling IMmd
vices are provided. most of the time. for women who are healthy and carry their megnancies and give birth without corn&aiiotk. Some dev&p minor problems and only the minoritv will oresent clinicallv imwrtant pathology. -It is quite possible to imagine a health care program that is of high quality, from the technical point of view, but is inadequate to deal with the emotional needs of women and their families, and therefore produces dissatisfaction. This often is considered as one dimension of quality of care 191,but its importance for maternal care warrants a separate place [ 131. Efficiency. All of the dimensions described above can adequately he addressed by a given health program, but the cost of doing so might be unacceptable. The relationship between the amount of resources to the amount of output units depicts the efficiency of a health program. Cost-e~c:iwness.
This
is the
most
com-
txehensive dimension for evaluating a health ~rograrn 171.It has the advantage gf produc‘mg info&&m on units that a& c¶ble across all the health oroarams 181.It considers inE0rmation on the a;no;Ot a& Comptexity of the inputs, expressed in monetary units, and the outcome, expressed in some composite measure of disease. Exceptfor equity and women’s satisfaction all the other dimensions described before are considered when this dimension is chosen. The cost of the program will be determined largely by the complexity of the interventions within the program, its scale (coverage). and the ef%iency of the rescwccs. Eff&iveness will depend on the covemae of the wnulation. the eff%acy of the . _ intmve&ons and the quality of CC&. This dimension is being criterion for selectina
proposed as the main health care uriorities in
developing countri&[8]. Defining the sran&r&
Each of the aforementioned dimensions can be measured according to certain criteria. But
absolute women’s
tkz,ures
on
knkh
coverage,
s9
pro&.,‘zns
quality
or
s&faction are Insufiicient to make a judgment as to how the program in question is performing. A compari&m has to be made with some standard. There are many ways of defining a standard for these dime&o&The following are the most commonly used.
Theoretical. According to the current understanding of the program and its components, it is possible to detine theoretical standards. For example, a maternal health program can be provided with suf%ient resources to care for all :he pregnant women in the target population; it is therefore reasonable to set a total coverage as the standard. Other dimensions, however, are more difticult lo anticipate and a theoretical standard can be very speculative; this is the case with qutity of care or efficiency. In these dimensions the relationship between inputs and outcome is unknown mainly because the few studies undertaken are not applicable to all contexts. Alternativelv. emuiricallv derived standards have to be ;sed;-the foilowing two are the most important.
Best possible. Largely derived from randcmized clinical and community trials, information on effectiveness and efficacy is available for some specific drugs and interventions. Two problems arise with this type of standard: fiat, most of the innovative maternity health programs have not been evaluated under controlled conditions. Second, even if the information on best possible outcome is available, often it does not apply to the context in which the program is being implemented. In addition, tinding out that a given intervention is, for example, 50% below the
best
possible
standard
might
be futile.
since the technology or resources are below those used to achieve the standard. It will be
to fmd
more meaningM achievable outcome available. provides.
This
out what is the best with the resources
is what
the
next
standard
Best achievable. The main difference of this standard from the previous one is that the best achievable refers to outcomes obtained among the same or similar population and with the same amount and complexity of resources. The limitation of this standard is that it is only useful when there are significant differences between the outcomes of the health program in question and the best achievable standard. Explicit criteria are preferable because they can provide a more transparent account of the evaluation process and permit a wider participation of professionals in evaluation of the program. However, sometimes it is difficult or impossible to obtain reliable explicit criteria. Commonly this occurs when the conditions in which services are provided cannot be anticipated and therefore the performance of the program cannot be objectively evaluated. A good example of the use of exdicit criteria is given by ihe Confidential Enquiries into Maternal Deaths, in which preventability of death (used to indicate quality of care) depends on an endless list of conditions, mainly derived from :he women’s health and the tvoe and stage of the pathology. A group of-&parts review case by case, and decide, based on explicit procedures to reach consensus, whether the death was preventable or not. Similarly, measurement of the quality of care provided by a group of traditional midwives who are assuming new responsibilities in obstetric care could be done by reviewing the preventability of severe maternal morbidity, using implicit criteria. Table I summarizes the possible options in defining the program, the dimensions and the standards. Combining these options it is possible to derive 90 approaches for the evaluation of maternity health programs. Since the dimensions for evaluation are not mutually exclusive, there are actually more than 90 approaches. Those that combine multiple dimensions are preferable. Effectiveness is probably the dimension of greatest interest to obstetricians, but also the one that presents
the greatest methodologic challenges for measurement. The rest of this paper concentrates mainly on the measurement of eflectiveness, although some of the issues may be relevant to other dimensions.
The selection of methods and indicators for evaluating health programs depends, to a large extent, on the choices made on the three aspects discussed before which define the approach to be used. There are not, however, clear and unequivocal definitions as to what method is best for each approach. Rather, each method is best suited to answer some specific types of questions; the following are the most commonly used in evaluating health programs. A randomized clinical trial is, of course, the most rigorous method for evaluating the efficacy of health interventions [!S]. It is best suited to deal with single interventions and it can only be applied when there are genuine doubts bn the-&nefits of the intervet&on. It is the method of choice when a completely new intervention or model of care is designed, and its etlicacy is unknown. The distinction
between efticacvand effectiveness is critical. ElBcacy refers tb effects of the intervention on the population’s health under controlled conditions, effectiveness to those effects under real life conditions. Results from randomized clinical trials undertaken for specific drugs and treatments provide the necessary quantitative infomtaiian to set the best possible standard. When observational methods are used to evaluate a program, the randomized clinical trial model serves as the paradigm to
be imitated, being extremely useful in decisions about control of confounding variables and other related issues. Randomized communiry trials. Although this is feasible on paper, and is possibly the
method of choice to evaluate (for the first time) the absolute eflicacy of a health program that includes several interventions,~this method is not used because of the high costs involved and the difficulties of obtaining comparable communities in sufticient quantities for an adequate sample size. Quasi-experiments. This is probably the most cmnmon method used to evaluate the effectiveness and quality of care of health programs. Three characteristics distinguish them from other methods. First, there are before and after measurements of the outcome indicators, to compare the absolute change during the period of program implementation. Second, twoormore communities or pupulatiun gmups are selected, intervened and measured; in one group the program in question is implemented, whereas in the rest other existing or new alternative programs ax studied or implemented, respectively. Typically in the second group a bealth program of known effectiveness is implemented, to assess additional benefit gained with the program of interest. A demonstration health project will be similar to a quasi-experiment, except that there would be no control cmnmunities. and therefore the effect of other variables on the differences ‘n outcome between the before
and the after cannut be reliably attributed to the program. Observarional studies The onlv accentable observational method to mea& the &fectiveness of a health program is the case-control approach. Measurement of the relationship of the program with changes in the health status is made retrospectively. Typically all individuals with negative outcomes are selected (cases),together with a random (small) sample of persons who do not have the negative outcome (controls). Both groups are investigated to study the exposure to the program in question. The odds ratio of exposure between the two groups provides an excellent proxy measurement of the odds of disease, which in turn is a good indicator of effectiveness [14]. It is the most inexpensive method, but not the simplest. Its complexities derive from the difficulties to draw a repres&tative samme of controls and the need to specify, measure and statistically control for all the confoundine variables ii4i. A case-control -study is the ~tnethucl of choice when a program has already been implemented, so it is not possible to make a before measurement, or the coverage of the program is incomplete, but there is a reasonable amount of information to study the intensity and characteristics of the exposure to the program interventions, among both cases and controls. Confidenrial enquiries into maternal deaths These have been described and are known to obstetricians and midwives. The measure of effectiveness is a proportionate mortality ratio of prcvec:able maternal deaths. The theoretical standard is, of course, that no preventable death should occur, but more commonly hospitals or ccnmmmity health systems are evaluated by comparing the current proportion of preventable deaths with that of the orevious l-3 wars. Three characteristics make them vcr$ attractive. First, they provide information on the specilic aspects of the process that went wrong and con-
tributed to the death(s), and thus produce clear recommendations for change, at different levels of the organization of services. Second, they are comparatively inexpensive. Third, as they require a multidisciplinary committee. wide oarticioation is rcouircd in the process; this &ds td legitimize tde results and improves the acceptance of the rccommendations that emerge from it. Their main oroblem is that they rcuuire a large number of births, and therefbre are not advisable for small progrtuns. This can be corrected, if, instead of deaths, a more common outcane is selected, such as severe maternal morbidity, e.g., patients with severe hemorrhage or eclampsia. A second limitation is that they require competent professionals, including specialists in obstetrics, internal medicine and pathology, not always available in small communities or rural hospitals.
Indicators for evaluating effectiveness of maternal health programs The design and selection of indicators is of critical importance to achieve useful results from evaluations. Three types have been used to assess effectiveness, largely representing the major components of the programs. These are structure, process and outcome. As will be shown, it is the combination of structure and process indicators with outcome indicators which produces the most useful results. Structure indicators arc the crudest. They include quantitative and qualitative review of the resources available, es well of the organizational arrangements that make them work. They may be useful in conditions of scarcity of resources, as long as they refer to essential resources or arc somewhat associated with outcome. indicators. Lack of essential resowces can reasonably lead to lower effectiveness, but availability does not necessarily guarantee good pcrfom&xx or satisfactory butcome. Moreover, the excess of resources and the availability of inadequate technology is often associated with suboptimal quality of care [I]. In summary, structure indicators arc useful
when they explain why a program is not producing the expected outcomes. Outcome indicators are strongly advocated in maternal health programs due to the justified concern with maternal survival in developing countries. Unfortunately, very little has been done to improve the shortcomings of the maternal mortality ratio; namely, the rarity of the event and low sensitivity to depict quality of care [?!I. In general, it is preferable to work with morbidity indicators, but they entail serious problems of measurement and reliability. Soecific interventions oriented to red”& morbidity should be evaluated with indicators of morbiditv or mortality associated with the disease c&dition of interest. Using the crude maternal mortality ratio or surrogate indicators, such as low birthweight, leads to underestimating the real effect of the intervention. Generic interventions, such as improvements in the referral system or the organization of delivery of care can be evaluated with maternal mortality ratios, as long as they arc somewhat standardized for biological and socioeconomic differences of the women.
573
Evaluation of maternat health services is an essential function of any health system. Single or multiple aspects of a program can be evaluated but they must be clearly defined. To assess success of a program the dimensions of coverage. equity, quality. women’s satisfaction, ef”ciency and cost-effectiveness must be considered. These can be measured accordine to wane standard criteria, and various methods are &scribed. in&dine randomized clinical trials, randomized commkitv trials. before and
tial in&iries into motern~~deotbs. Indicok to assess effectiveness of o maternal health program may relate to structure, i.e., available resources or organizational arrangement: process. including changes in quantity of services provided: or outcome indicators that measure maternal mortality or morbidity.
Keywords: Evaluating maternity care; Research methodology; Safe motherhood. As part of the momentum aaincd bv the Safe tiotherhood Initiative, m&y programs are beinr! imulcmcated throughout the developing world claiming to bc pan of the crusade. Unfortunately only a few have documented any reductions of maternal morbidity and mortality [.5,6,10]. This may bc because
most of the programs have not been evaluated or their evaluation has been undertaken with inappropriate approaches and methods. Further, maternal health programs arc panicularly difticult to evaluate, since the final outcome, improvements in the health of mothers and newborns, for example, depends on the functioning of the whole system and not on only a single intervention, and there arc other determining factors, most importantly. the physique and health condition of the women and the socioeconomic status of their family [1,3]. Therefore, trying to relate a single provider or intervention to improvements on matciaa! sa+val is impossible, u&s: a!! other relevant elements of the health program and the characteristics of the women arc strictly controlled or held constant. Finally, maternal death is a rare event, even in the worst condition; therefore any attempt to measure mortality is faced with the need to study thousands of births to achieve signilicant comparisons between these rates [6]. This paper presents some of the main ap proaches that have been successful in evaluating health programs, discusses some of the available methods and their limitations, and suggests a classification of indicators that arc useful in assessment of maternal health pmgrams. Approaches to evaluate bcalth progrluns Evaluation of health services is an essential function of all good health systems. Common-
ly, evaluation refers to a stage of the planning cycle when routinely collected data are processed and interpreted to assess whether a set of specified services has achieved some stated objectives in a determined time period 191. More broadly the concept embraces all types of activities that compare the achievements of a given intervention with those of a predetermined standard. These definitions clearlv show that the purpose of evaluating services G to pass judgments as to the goodness of the services. It therefore requires that evaluators specify the characteristics of the program, the dimensions to be evaluated and the stan-
dard(s). The following is a brief review of the alternative ways of specifying these aspects, in the context of maternal health programs,
As obvious kit of health oroerams program I& 6een
may seem the evaluation often
f&s
because
the
poorly defined. The simplest program will consist of a single intervention, typically delivered by one provider (e.g., administration of a vaccine or a drug). But most programs involve several interventions that are applied, with similar purposes, to a specific population group, as is the case of antenatal care. More complex programs include referral of some patients to receive diagnostic and therapeutic services from other providers. For example, a maternal health program could encompass family planning, antenatal and delivery care, with the participation of traditional midwives, community nurses, professional midwives, and doctors with different training (among other providers). Typically such a program would require referral of some women to a hospital setting where essential obstetric functions are available. For the purpose of this workshop, to evaluate an innovative program that delegates obstetric functions to nurses or midwives, it is essential to define the program in terms of provider, setting, activities, interventions and, most importantly, expected outcomes.
There are at least six dimensions that are relevant to assess the snccess of a health program: coverage, equity, quality (technical), women’s satisfaction, efficiency and costeffectiveness.
Coverage. This refers to the proportion of the population that has reasonable access to the health program in question [12]. One of the most important goals of delegating resaonsibilities in obstetrics is certainlv to iocrease the coverap cf maternal he&b services. Maternal health care is strongly influenced by culture and individual or family preferences. The actual utilization of a maternal health program is thus determined by the amenities of the service, the content of care, and most importantly by the way in which providers attend the emotional needs of women who are pregnant and being delivered. Eg&. Although there is no single accepted delinition of equity, its most common eonnotation refers to the distribution of resources according to the health needs of the population Ill]. Delegation of obstetric respcmsibilities in a community can lead to a more equitable distribution if the extension in coverage benefits primarily the most needy socioeconomic groups, typically, the residents of rural areas and urban slums. &a@ ofcar.?. Changes in the health status attributable to health care depend on the technical auatitv of the services received 141. The limits of delegating responsibilities‘ ib obstetrics are largely determined by the tmining and performance of the personnel who will assume the responsibilities, as well as by the amount and appropriateness of the resources and technology at their disposal. Deficiencies in uerformance or resources can lead to standark of care which may be considered unacceptable. Women’s satisfaction. Maternal health ser-
ELtdlmling IMmd
vices are provided. most of the time. for women who are healthy and carry their megnancies and give birth without corn&aiiotk. Some dev&p minor problems and only the minoritv will oresent clinicallv imwrtant pathology. -It is quite possible to imagine a health care program that is of high quality, from the technical point of view, but is inadequate to deal with the emotional needs of women and their families, and therefore produces dissatisfaction. This often is considered as one dimension of quality of care 191,but its importance for maternal care warrants a separate place [ 131. Efficiency. All of the dimensions described above can adequately he addressed by a given health program, but the cost of doing so might be unacceptable. The relationship between the amount of resources to the amount of output units depicts the efficiency of a health program. Cost-e~c:iwness.
This
is the
most
com-
txehensive dimension for evaluating a health ~rograrn 171.It has the advantage gf produc‘mg info&&m on units that a& c¶ble across all the health oroarams 181.It considers inE0rmation on the a;no;Ot a& Comptexity of the inputs, expressed in monetary units, and the outcome, expressed in some composite measure of disease. Exceptfor equity and women’s satisfaction all the other dimensions described before are considered when this dimension is chosen. The cost of the program will be determined largely by the complexity of the interventions within the program, its scale (coverage). and the ef%iency of the rescwccs. Eff&iveness will depend on the covemae of the wnulation. the eff%acy of the . _ intmve&ons and the quality of CC&. This dimension is being criterion for selectina
proposed as the main health care uriorities in
developing countri&[8]. Defining the sran&r&
Each of the aforementioned dimensions can be measured according to certain criteria. But
absolute women’s
tkz,ures
on
knkh
coverage,
s9
pro&.,‘zns
quality
or
s&faction are Insufiicient to make a judgment as to how the program in question is performing. A compari&m has to be made with some standard. There are many ways of defining a standard for these dime&o&The following are the most commonly used.
Theoretical. According to the current understanding of the program and its components, it is possible to detine theoretical standards. For example, a maternal health program can be provided with suf%ient resources to care for all :he pregnant women in the target population; it is therefore reasonable to set a total coverage as the standard. Other dimensions, however, are more difticult lo anticipate and a theoretical standard can be very speculative; this is the case with qutity of care or efficiency. In these dimensions the relationship between inputs and outcome is unknown mainly because the few studies undertaken are not applicable to all contexts. Alternativelv. emuiricallv derived standards have to be ;sed;-the foilowing two are the most important.
Best possible. Largely derived from randcmized clinical and community trials, information on effectiveness and efficacy is available for some specific drugs and interventions. Two problems arise with this type of standard: fiat, most of the innovative maternity health programs have not been evaluated under controlled conditions. Second, even if the information on best possible outcome is available, often it does not apply to the context in which the program is being implemented. In addition, tinding out that a given intervention is, for example, 50% below the
best
possible
standard
might
be futile.
since the technology or resources are below those used to achieve the standard. It will be
to fmd
more meaningM achievable outcome available. provides.
This
out what is the best with the resources
is what
the
next
standard
Best achievable. The main difference of this standard from the previous one is that the best achievable refers to outcomes obtained among the same or similar population and with the same amount and complexity of resources. The limitation of this standard is that it is only useful when there are significant differences between the outcomes of the health program in question and the best achievable standard. Explicit criteria are preferable because they can provide a more transparent account of the evaluation process and permit a wider participation of professionals in evaluation of the program. However, sometimes it is difficult or impossible to obtain reliable explicit criteria. Commonly this occurs when the conditions in which services are provided cannot be anticipated and therefore the performance of the program cannot be objectively evaluated. A good example of the use of exdicit criteria is given by ihe Confidential Enquiries into Maternal Deaths, in which preventability of death (used to indicate quality of care) depends on an endless list of conditions, mainly derived from :he women’s health and the tvoe and stage of the pathology. A group of-&parts review case by case, and decide, based on explicit procedures to reach consensus, whether the death was preventable or not. Similarly, measurement of the quality of care provided by a group of traditional midwives who are assuming new responsibilities in obstetric care could be done by reviewing the preventability of severe maternal morbidity, using implicit criteria. Table I summarizes the possible options in defining the program, the dimensions and the standards. Combining these options it is possible to derive 90 approaches for the evaluation of maternity health programs. Since the dimensions for evaluation are not mutually exclusive, there are actually more than 90 approaches. Those that combine multiple dimensions are preferable. Effectiveness is probably the dimension of greatest interest to obstetricians, but also the one that presents
the greatest methodologic challenges for measurement. The rest of this paper concentrates mainly on the measurement of eflectiveness, although some of the issues may be relevant to other dimensions.
The selection of methods and indicators for evaluating health programs depends, to a large extent, on the choices made on the three aspects discussed before which define the approach to be used. There are not, however, clear and unequivocal definitions as to what method is best for each approach. Rather, each method is best suited to answer some specific types of questions; the following are the most commonly used in evaluating health programs. A randomized clinical trial is, of course, the most rigorous method for evaluating the efficacy of health interventions [!S]. It is best suited to deal with single interventions and it can only be applied when there are genuine doubts bn the-&nefits of the intervet&on. It is the method of choice when a completely new intervention or model of care is designed, and its etlicacy is unknown. The distinction
between efticacvand effectiveness is critical. ElBcacy refers tb effects of the intervention on the population’s health under controlled conditions, effectiveness to those effects under real life conditions. Results from randomized clinical trials undertaken for specific drugs and treatments provide the necessary quantitative infomtaiian to set the best possible standard. When observational methods are used to evaluate a program, the randomized clinical trial model serves as the paradigm to
be imitated, being extremely useful in decisions about control of confounding variables and other related issues. Randomized communiry trials. Although this is feasible on paper, and is possibly the
method of choice to evaluate (for the first time) the absolute eflicacy of a health program that includes several interventions,~this method is not used because of the high costs involved and the difficulties of obtaining comparable communities in sufticient quantities for an adequate sample size. Quasi-experiments. This is probably the most cmnmon method used to evaluate the effectiveness and quality of care of health programs. Three characteristics distinguish them from other methods. First, there are before and after measurements of the outcome indicators, to compare the absolute change during the period of program implementation. Second, twoormore communities or pupulatiun gmups are selected, intervened and measured; in one group the program in question is implemented, whereas in the rest other existing or new alternative programs ax studied or implemented, respectively. Typically in the second group a bealth program of known effectiveness is implemented, to assess additional benefit gained with the program of interest. A demonstration health project will be similar to a quasi-experiment, except that there would be no control cmnmunities. and therefore the effect of other variables on the differences ‘n outcome between the before
and the after cannut be reliably attributed to the program. Observarional studies The onlv accentable observational method to mea& the &fectiveness of a health program is the case-control approach. Measurement of the relationship of the program with changes in the health status is made retrospectively. Typically all individuals with negative outcomes are selected (cases),together with a random (small) sample of persons who do not have the negative outcome (controls). Both groups are investigated to study the exposure to the program in question. The odds ratio of exposure between the two groups provides an excellent proxy measurement of the odds of disease, which in turn is a good indicator of effectiveness [14]. It is the most inexpensive method, but not the simplest. Its complexities derive from the difficulties to draw a repres&tative samme of controls and the need to specify, measure and statistically control for all the confoundine variables ii4i. A case-control -study is the ~tnethucl of choice when a program has already been implemented, so it is not possible to make a before measurement, or the coverage of the program is incomplete, but there is a reasonable amount of information to study the intensity and characteristics of the exposure to the program interventions, among both cases and controls. Confidenrial enquiries into maternal deaths These have been described and are known to obstetricians and midwives. The measure of effectiveness is a proportionate mortality ratio of prcvec:able maternal deaths. The theoretical standard is, of course, that no preventable death should occur, but more commonly hospitals or ccnmmmity health systems are evaluated by comparing the current proportion of preventable deaths with that of the orevious l-3 wars. Three characteristics make them vcr$ attractive. First, they provide information on the specilic aspects of the process that went wrong and con-
tributed to the death(s), and thus produce clear recommendations for change, at different levels of the organization of services. Second, they are comparatively inexpensive. Third, as they require a multidisciplinary committee. wide oarticioation is rcouircd in the process; this &ds td legitimize tde results and improves the acceptance of the rccommendations that emerge from it. Their main oroblem is that they rcuuire a large number of births, and therefbre are not advisable for small progrtuns. This can be corrected, if, instead of deaths, a more common outcane is selected, such as severe maternal morbidity, e.g., patients with severe hemorrhage or eclampsia. A second limitation is that they require competent professionals, including specialists in obstetrics, internal medicine and pathology, not always available in small communities or rural hospitals.
Indicators for evaluating effectiveness of maternal health programs The design and selection of indicators is of critical importance to achieve useful results from evaluations. Three types have been used to assess effectiveness, largely representing the major components of the programs. These are structure, process and outcome. As will be shown, it is the combination of structure and process indicators with outcome indicators which produces the most useful results. Structure indicators arc the crudest. They include quantitative and qualitative review of the resources available, es well of the organizational arrangements that make them work. They may be useful in conditions of scarcity of resources, as long as they refer to essential resources or arc somewhat associated with outcome. indicators. Lack of essential resowces can reasonably lead to lower effectiveness, but availability does not necessarily guarantee good pcrfom&xx or satisfactory butcome. Moreover, the excess of resources and the availability of inadequate technology is often associated with suboptimal quality of care [I]. In summary, structure indicators arc useful
when they explain why a program is not producing the expected outcomes. Outcome indicators are strongly advocated in maternal health programs due to the justified concern with maternal survival in developing countries. Unfortunately, very little has been done to improve the shortcomings of the maternal mortality ratio; namely, the rarity of the event and low sensitivity to depict quality of care [?!I. In general, it is preferable to work with morbidity indicators, but they entail serious problems of measurement and reliability. Soecific interventions oriented to red”& morbidity should be evaluated with indicators of morbiditv or mortality associated with the disease c&dition of interest. Using the crude maternal mortality ratio or surrogate indicators, such as low birthweight, leads to underestimating the real effect of the intervention. Generic interventions, such as improvements in the referral system or the organization of delivery of care can be evaluated with maternal mortality ratios, as long as they arc somewhat standardized for biological and socioeconomic differences of the women.
573