- Email: [email protected]
Concurrent evaluation: An approach to action research
Sot. Sci. & Med., 1970, Supplement 1, Vol. 4, pp. 25-30
CONCURRENT
Pergamon Press
EVALUATION:
AN APPROACH TO ACTION RESEARCH
Jane 2. Moss
Introduction Today there is a great awareness of major human and social problems which we have not yet solved. Many of these problems are concentrated in the cities where large populations suffer multiple deprivations. New and expanded programs are taking place in our cities to relieve these deprivations of low income, poor housing, inadequate education, poor health, unemployment, and, concomitant with these, the overall despair and feeling of the unimportance of life. With limited financial and human resources, it becomes crucial to know which programs and which parts of programs are working most effectively. To compare the objectives of programs in an effort to find the most effective and reasonable ones for a particular population is a legitimate rationale for evaluation research. To make findings available to program staff in a comparatively short time is another legitimate rationale. These are two major objectives of what we have named concurrent evaluation. Every problem generates its own reformers and the cry of the reformer is almost always, in one form or another, money. But money alone will not solve the deep social and psychological problems found in our underprivileged populations. !Ne must ask in what manner can this money be spent most effectively; how do we reach different segments of the recipient population for full participation; in what manner can we use scarce professional personnel most effectively, i.e., how do we organize available resources to solve a problem and how do we limit our objectives to feasible ones for the resourcesavailable? Let us assume that the federal government would provide the funds for poor people to have private physician’s care equivalent to that available to the more affluent. Indeed this is being attempted in the case of Medicaid and other Title XIX programs. The assumption is that more money alone will now provide the servicesfor which the poor have the greatest need and the fewest resources. Implicit in the system is the assumption of class differences in medical care and that the private physician belongs to the affluent class. The private physician’s fee for service is a form of distributing services which assumes there is more demand than service available. The simple truth of the matter is-if we could pay for and induce everyone requiring service to seek it, the personnel qualified to cope with the problems in existing facilities would be overwhelmed. It follows that if the goal is adequate care for all, the present system of care itself must be modified and the number and the nature of the personnel increased. No single “program” would be able to provide adequate care for all because the total existing resourcesare insufficient. This situation presents any existing program with the necessity for describing self-critically its
25
J.Z. Moss
26
own priorities for expenditures and for use of personnel. The total context of which a program is a part must be considered in the evaluation of a single project within a larger system of services.Each facility must document the kinds of people it cannot serve. Each project must assessthose problems with which it is incapable of coping, because these will usually reflect inadequacies of the total system and suggest modifications of it. Constantly reporting “failures” focuses on continuing objectives for long range planning and for innovations in “program” solutions. Within this larger context, how would we allocate skilled personnel for the problems presented in a single urban community in order to give the most adequate servicesto those most in need and for limited, feasible objectives? To answer such a question we must know what probiems exist in the community and who has defined these problems? Do the description and priorities of problems come from the community designated for service or from the professional staff providing that service?The professional staff may find their concerns rank low in the community. But the origin and definition of problems and the consequent action deriving from program design must be constantly in mind, so that the program is always open and flexible to the adoption of more appropriate goals and their implementation. We consider the effectiveness of achieving goals and the choice of appropriate goals the legitimate concern of evaluation. This discussion will concern itself with some of the aspects of this question as dealt with by traditional procedures of evaluation, and suggestmodifications. The Experimental Design The main problem we face in evaluating a program within a community is control of other events occurring in the community while a program is in process. Most evaluation research follows an experimental design in which all other factors relevant to the interpretation of the outcome are controlled as nearly as possible through such techniques as random sampling and the use of control groups. The purpose in the experimental design is to eliminate all variables except the ones under consideration by the experimenter. The experimental method is not appropriate for most instances of evaluation in urban settings. Let us assume that we could find two neighborhoods similar in composition according to age, sex, ethnic background and occupation. Then assume that a special program is introduced into one neighborhood and not into the other. How do we know that any differences within the experimental neighborhood as compared with the control neighborhood are due to the program? I think the answer is that we do not know. We cannot control for the multiple affiliations taking place in one neighborhood or another, the different informal organizations, friendships, different agencies and qualities of leadership affecting these two neighborhoods. These are all qualities that may be crucial to the effectiveness of any program. If we look at few variables presented by an experimental design, removing them from the context’
of other variables, we sterilize the concepts. Such insistence on the recognition of the
multiplicity of variables and their interaction is making a rather broad assumption about the nature of “response” to programs or “program impact”. TO call the experimental design inappropriate for the a We are aware of the “contextual analysis” that has been very ably done, but this is not generally a control situation in the strict sense, but rather analysis done after data collection has been completed and the various groups classified on the basis of this information.
Concurrent Evaluation: An Approach To Action Research
27
reasons stated assumes that behavior is not explained in terms of an oversimplified “action and reaction” or “stimulus and response” - assumptions appropriate to the experimental design. Instead, we assume responsescan occur in such complicated patterns that its very design will escape our vision. If a new program is provided, let us say for child care, its effectiveness depends among other things, on the attitudes of the parents, their capacities for and beliefs about child care, and implicit assumptions about the needs and future of their children. Where an effective job training program exists for the youth in one area, this might result in parents’ changes in aspirations for their children, which in turn will affect the impact of a child care program by redefining parents’ aspirations for young children. There are three different kinds of difficulties with evaluation research: 1) Very often, the experimental design in evaluation research is impossible because the research is generally tacked on to the “action” programs and is not included in the initial design of the program. 2) However, even with careful planning, it is often impossible to measure individuals along the relevant variables for control until after the data is collected. Even when we know which variables are most likely to be relevant beforehand, data distributing individuals or other units along a given variable are often not available until the data collection is completed. 3) Finally, however, we question the inherent suppositions of the control group design for evaluation. Generally, in action research we feel that the variables, even if they could be selected beforehand for the purposes of control, would still not provide an effective model for the actions and reactions of the programs with a target population. Any single variable reacts with so many others concurrently that it is not feasible, as assumed by the experimental design, to specify the precise way in which a single variable acts or reacts with one or two others. CONCURRENT
EVALUATION
It is the major task of concurrent evaluation to assesscontinuously the progress of a particular program in a particular neighborhood, thereby determining how a program works and with whom it works; and, accordingly, to make necessary corrections. This can be done with existing techniques. A baseline survey provides an initial description and classification of the target population. A history of the community should be kept to document major events that might be a further source of interaction. Target populations should be reinterviewed or revisited in order to know what changes have occurred, i.e., behavioral or attitudinal changes or changes in a living situation. Within a short period after commencing services, and periodically after that, both participants and non-participants should be revisited to assessthe “reasons” for their different responsesto the program. These two kinds of information are quite different but are central features of a concurrent evaluation procedure. One details the sequence of events and choices and the other the motivating factors or varying circumstances which affect change or stability. To illustrate our point, we have selected a single example from a retirement and health services program established for older persons residing in a middle income cooperative housing project. The overall objectives of the program were to provide services and referral to services for the physical and emotional difficulties of retirement and old age. Since health is a major concern for older people, the program director chose two sub-goalsin this area which he hoped would provide a) maximum preventive care, b) assuranceof good medical care in the
J.Z. Moss
28
event of an emergency
and confidence
to the elderly person about adequate provision of care for his
health needs.
To
implement
selection
these objectives,
of a physician
“family
doctor”
patient
during
examination
two
actions
were
encouraged
who was familiar an emergency;
with the patient
and who had some commitment
and 2) the acceptance
by the older
From
the information
collected
the social case worker
had had a recent
physical
in a baseline interview,
check-up.
From
operational
health
or received a health examination invited
selected
issues-selecting
were
to
a meeting
with
had a health
social worker
to
The twelve
discuss these two
to these different
proportions?
anxiety
objective-obtaining
Further follow-up
revealed that all three persons
had been given “a clean bill of health”
This, in the subjects’ own words made them “feel so much better”.
that may have been felt about their own health. But the accomplishment an exam-defeated
the accomplishment
selected in the event of an emergency.
of the second. No physician
Between the two seemingly
the successful completion
priorities
by this
only five had also selected a physician. Was
exam but had not selected a physician
the health examination.
contradicting
since the original interview.
the
of their
from the original list
had also received a health examination
the eight who received a health examination
there any significance
It eliminated
physician nor
by the staff
a physician and obtaining a free check-up.
Five of the six persons who selected a physician time. Among
as part of the research and
those persons were eliminated
persons
thus
conducted
records maintained
who had selected a physician
different
to care for the
of a diagnostic
was able to locate persons who had neither a family
activities and contacts with the eligible population,
following
person
1) the
there would be a
which was available at no cost from the public health station.
evaluation,
who
by the social worker:
by the older person so that in the event of an emergency
independent
of the other. With this knowledge;
for these two objectives:
First-it
encouraged
of one
had been
objectives,pne
was
the program established
people to select a physician; second,
it encouraged a diagnostic check-up.
From program
the questionnaire
objectives.
through volunteer daily
and operational
basis. This proved unfeasible
in the neighborhood
discovered
information
time
within
the project
as a whole.
persons did not want the responsibility
persons whom they did not know.
at the
other
helped obtain
additional aid
services in the event of emergencies or to check on the well being of residents on a
seemed clear that elderly
collected
records,
It had been one of the initial objectives of the program to establish mutual
to venture of
the
a small network
Furthermore,
From the information of providing
on the
of existing mutual
care-for
they were afraid because of muggings and robberies
out at night even in case of emergency. interview
we had, it
emergency
interpersonal
aid through
relations
informal
Examining
among
arrangements,
elderly
information people
we
already operating
within the project. Mutual aid in the event of sickness or accidents occurred within the closest physical proximity.
Neighbors would
became a matter these and tying
look in on neighbors who were next door or on the same floor.
of articulating in isolated
individuals
based on a physical proximity defeat of a meaningful the “appropriate
various forms of assistance already partially
Volunteer
services
from floor to floor proved feasible. What might have originally
been the
objective,
approach”
and newcomers
the use of volunteers
to volunteer services.
to the self-help
It then
established, implementing
for mutual
network.
aid, was salvaged by re-examining
Concurrent Evaluation: An Approach To Action Research
29
A further example of information systematically collected by research interviews and used to redefine a procedure occurred in a health center located in a low-income urban neighborhood of mixed ethnic origins. The staff of the Center was having a problem with broken appointments. The staff had been sending letters to persons who missed their appointments and in the course of the letter made a new appointment for them. Less than two-thirds of the total appointments made were kept by the patients. The staff of the Center wondered if the patients were unused to the appointment system through long years of experience with clinic care or whether there was dissatisfaction with servicesat the Center. Health aides working for the Center were sent out with questionnaires to interview those who had missed their appointments. They discovered that in the majority of cases,outside events had made the return on a specific date too difficult. Common problems were baby-sitting, an unexpectedly sick child who could not be left, and special facilities for individuals who needed help to get to the Center because of physical or psychological disabilities. Furthermore, patients used to a clinic system did not understand that they were expected to call the Center to report their inability to keep an appointment. Frequently patients had moved and their new address was unknown or they lived with another family whose name appeared on the mail-box. We revised our system of making appointments. No more appointments were made by mail. When an appointment was not kept, an aide visited the home to find out why and frequently to help make plans for anticipated visits. Health teams were much more alerted to the extent to which a patient’s disability might inhibit his return and plans were more consistently made to send aides to assist these patients with return arrangements. Finally, at every appropriate occasion patients were encouraged to telephone the Center when they could not make an appointment. Within a short period of. time, the patient “no-show” rate was greatly reduced and at the same time the staff was maintaining a higher level of information about the problems of their patients. Modification of Standard Research Techniques The use of the survey as a research tool has too often been in one senseextended and in another narrowed. It has been inappropriately extended when interpretations are presented without historical and cultural perspective. We find, for instance, that some of our Spanish-speaking populations do not utiliie preventive child health care facilities, as compared with other groups. What the survey does not tell us is the reluctance on the part of the parents “to make a healthy child sick” as the result of an inoculation or vaccination. Nor is the role of this attitude evaluated in a religious context of sin and divine providence. The survey is all too likely to focus on “attitudes toward health care”. A survey is analogous to the examination of a cross-section of an organism. One must first understand the anatomy of the organism and that part of the structure from which the cross-section is taken. It is critically important to understand the context of the respondent in order to interpret the findings from the survey, in order to make them not just predictive but practicable for new approaches. The survey has been too narrowed by the concern for objectivity which has taken on ritualistic form. For instance, requirements for the role of interviewer generally specify a stranger who asks for information in a detached manner, usually justifying it for “research” and guaranteeing confidentiality -from any but the research team. In concurrent evaluation, the research is intended to work as an
J.Z. Moss
30
integral part of the program-and the interviewers have an expanded role. We found it possible, in surveying for a neighborhood health center, to use interviewers who explained the services of the program to the respondent at the time of interviewing. The information collected was confidential within the program but not exclusively to the research team. We found this interviewing could be better performed by well-trained persons from within the target population. If care is taken to place community people as interviewers in the community, but at a sufficient distance from personal acquaintances, we can manage the problems of confidientiality
and honesty of reports from
respondents.* There is no reason to hold that the information collected, with the interviewer playing the role of program contact and conventional interviewer, cannot be useful to the program and the research. Certainly poor people, constantly questioned by authorities, find it easier to give a straight-forward response to a sympathetic listener who is offering a service than to an “objective stranger”. Relating to the respondent with an exchange of information need not jeopardize the validity of the response, although it does not eliminate the need for the usual kinds of controls for interviewer validity and reliability. Another concern of concurrent evaluation is that of determining what the priority of the program objectives is to the population itself. Often, objectives are designed by professionals whose specialist interests overlook the relevance of those objectives to the population concerned. While good health is the concern of any medical program, it may very well be that problems of employment, child care, and other behavior problems within the family take priority over health. If personnel of the health program become aware of this, they can refer individuals or families for other services either within the program or to other agencies.As the problems are resolved according to the priorities of the individuals rather than the professionals, the health problems of the family will be tackled as other urgencies are eased. The number of persons helped with appropriate referrals can be one measure of the impact of a health program and a means of making professional goals of the program and individual goals of the target population compatible. We must examine those persons directed by the program to resources outside itself. It is necessary to point out however that the specific factors affecting the successof failure of a program in one community are not necessarilyappropriate to the next. How then can we transmit what we have learned in one community to another if our results cannot be generalized? We can follow similar procedures of investigation and compare effective and ineffective implementations of action. Each study becomes a unit in a sample of action programs so that with enough studies of action in varying contexts we can begin to find what general program characteristics are necessary for the selection and implementation of appropriate goals and what population characteristics are most amenable to the acceptance of program innovation. Such generalizations will have the added advantage that they will be derived from life rather than the laboratory.
*Poverty studies have, of course, used community interviewers and a few experimental health programs have used information collected by interviewers for programs.
CONCURRENT
Pergamon Press
EVALUATION:
AN APPROACH TO ACTION RESEARCH
Jane 2. Moss
Introduction Today there is a great awareness of major human and social problems which we have not yet solved. Many of these problems are concentrated in the cities where large populations suffer multiple deprivations. New and expanded programs are taking place in our cities to relieve these deprivations of low income, poor housing, inadequate education, poor health, unemployment, and, concomitant with these, the overall despair and feeling of the unimportance of life. With limited financial and human resources, it becomes crucial to know which programs and which parts of programs are working most effectively. To compare the objectives of programs in an effort to find the most effective and reasonable ones for a particular population is a legitimate rationale for evaluation research. To make findings available to program staff in a comparatively short time is another legitimate rationale. These are two major objectives of what we have named concurrent evaluation. Every problem generates its own reformers and the cry of the reformer is almost always, in one form or another, money. But money alone will not solve the deep social and psychological problems found in our underprivileged populations. !Ne must ask in what manner can this money be spent most effectively; how do we reach different segments of the recipient population for full participation; in what manner can we use scarce professional personnel most effectively, i.e., how do we organize available resources to solve a problem and how do we limit our objectives to feasible ones for the resourcesavailable? Let us assume that the federal government would provide the funds for poor people to have private physician’s care equivalent to that available to the more affluent. Indeed this is being attempted in the case of Medicaid and other Title XIX programs. The assumption is that more money alone will now provide the servicesfor which the poor have the greatest need and the fewest resources. Implicit in the system is the assumption of class differences in medical care and that the private physician belongs to the affluent class. The private physician’s fee for service is a form of distributing services which assumes there is more demand than service available. The simple truth of the matter is-if we could pay for and induce everyone requiring service to seek it, the personnel qualified to cope with the problems in existing facilities would be overwhelmed. It follows that if the goal is adequate care for all, the present system of care itself must be modified and the number and the nature of the personnel increased. No single “program” would be able to provide adequate care for all because the total existing resourcesare insufficient. This situation presents any existing program with the necessity for describing self-critically its
25
J.Z. Moss
26
own priorities for expenditures and for use of personnel. The total context of which a program is a part must be considered in the evaluation of a single project within a larger system of services.Each facility must document the kinds of people it cannot serve. Each project must assessthose problems with which it is incapable of coping, because these will usually reflect inadequacies of the total system and suggest modifications of it. Constantly reporting “failures” focuses on continuing objectives for long range planning and for innovations in “program” solutions. Within this larger context, how would we allocate skilled personnel for the problems presented in a single urban community in order to give the most adequate servicesto those most in need and for limited, feasible objectives? To answer such a question we must know what probiems exist in the community and who has defined these problems? Do the description and priorities of problems come from the community designated for service or from the professional staff providing that service?The professional staff may find their concerns rank low in the community. But the origin and definition of problems and the consequent action deriving from program design must be constantly in mind, so that the program is always open and flexible to the adoption of more appropriate goals and their implementation. We consider the effectiveness of achieving goals and the choice of appropriate goals the legitimate concern of evaluation. This discussion will concern itself with some of the aspects of this question as dealt with by traditional procedures of evaluation, and suggestmodifications. The Experimental Design The main problem we face in evaluating a program within a community is control of other events occurring in the community while a program is in process. Most evaluation research follows an experimental design in which all other factors relevant to the interpretation of the outcome are controlled as nearly as possible through such techniques as random sampling and the use of control groups. The purpose in the experimental design is to eliminate all variables except the ones under consideration by the experimenter. The experimental method is not appropriate for most instances of evaluation in urban settings. Let us assume that we could find two neighborhoods similar in composition according to age, sex, ethnic background and occupation. Then assume that a special program is introduced into one neighborhood and not into the other. How do we know that any differences within the experimental neighborhood as compared with the control neighborhood are due to the program? I think the answer is that we do not know. We cannot control for the multiple affiliations taking place in one neighborhood or another, the different informal organizations, friendships, different agencies and qualities of leadership affecting these two neighborhoods. These are all qualities that may be crucial to the effectiveness of any program. If we look at few variables presented by an experimental design, removing them from the context’
of other variables, we sterilize the concepts. Such insistence on the recognition of the
multiplicity of variables and their interaction is making a rather broad assumption about the nature of “response” to programs or “program impact”. TO call the experimental design inappropriate for the a We are aware of the “contextual analysis” that has been very ably done, but this is not generally a control situation in the strict sense, but rather analysis done after data collection has been completed and the various groups classified on the basis of this information.
Concurrent Evaluation: An Approach To Action Research
27
reasons stated assumes that behavior is not explained in terms of an oversimplified “action and reaction” or “stimulus and response” - assumptions appropriate to the experimental design. Instead, we assume responsescan occur in such complicated patterns that its very design will escape our vision. If a new program is provided, let us say for child care, its effectiveness depends among other things, on the attitudes of the parents, their capacities for and beliefs about child care, and implicit assumptions about the needs and future of their children. Where an effective job training program exists for the youth in one area, this might result in parents’ changes in aspirations for their children, which in turn will affect the impact of a child care program by redefining parents’ aspirations for young children. There are three different kinds of difficulties with evaluation research: 1) Very often, the experimental design in evaluation research is impossible because the research is generally tacked on to the “action” programs and is not included in the initial design of the program. 2) However, even with careful planning, it is often impossible to measure individuals along the relevant variables for control until after the data is collected. Even when we know which variables are most likely to be relevant beforehand, data distributing individuals or other units along a given variable are often not available until the data collection is completed. 3) Finally, however, we question the inherent suppositions of the control group design for evaluation. Generally, in action research we feel that the variables, even if they could be selected beforehand for the purposes of control, would still not provide an effective model for the actions and reactions of the programs with a target population. Any single variable reacts with so many others concurrently that it is not feasible, as assumed by the experimental design, to specify the precise way in which a single variable acts or reacts with one or two others. CONCURRENT
EVALUATION
It is the major task of concurrent evaluation to assesscontinuously the progress of a particular program in a particular neighborhood, thereby determining how a program works and with whom it works; and, accordingly, to make necessary corrections. This can be done with existing techniques. A baseline survey provides an initial description and classification of the target population. A history of the community should be kept to document major events that might be a further source of interaction. Target populations should be reinterviewed or revisited in order to know what changes have occurred, i.e., behavioral or attitudinal changes or changes in a living situation. Within a short period after commencing services, and periodically after that, both participants and non-participants should be revisited to assessthe “reasons” for their different responsesto the program. These two kinds of information are quite different but are central features of a concurrent evaluation procedure. One details the sequence of events and choices and the other the motivating factors or varying circumstances which affect change or stability. To illustrate our point, we have selected a single example from a retirement and health services program established for older persons residing in a middle income cooperative housing project. The overall objectives of the program were to provide services and referral to services for the physical and emotional difficulties of retirement and old age. Since health is a major concern for older people, the program director chose two sub-goalsin this area which he hoped would provide a) maximum preventive care, b) assuranceof good medical care in the
J.Z. Moss
28
event of an emergency
and confidence
to the elderly person about adequate provision of care for his
health needs.
To
implement
selection
these objectives,
of a physician
“family
doctor”
patient
during
examination
two
actions
were
encouraged
who was familiar an emergency;
with the patient
and who had some commitment
and 2) the acceptance
by the older
From
the information
collected
the social case worker
had had a recent
physical
in a baseline interview,
check-up.
From
operational
health
or received a health examination invited
selected
issues-selecting
were
to
a meeting
with
had a health
social worker
to
The twelve
discuss these two
to these different
proportions?
anxiety
objective-obtaining
Further follow-up
revealed that all three persons
had been given “a clean bill of health”
This, in the subjects’ own words made them “feel so much better”.
that may have been felt about their own health. But the accomplishment an exam-defeated
the accomplishment
selected in the event of an emergency.
of the second. No physician
Between the two seemingly
the successful completion
priorities
by this
only five had also selected a physician. Was
exam but had not selected a physician
the health examination.
contradicting
since the original interview.
the
of their
from the original list
had also received a health examination
the eight who received a health examination
there any significance
It eliminated
physician nor
by the staff
a physician and obtaining a free check-up.
Five of the six persons who selected a physician time. Among
as part of the research and
those persons were eliminated
persons
thus
conducted
records maintained
who had selected a physician
different
to care for the
of a diagnostic
was able to locate persons who had neither a family
activities and contacts with the eligible population,
following
person
1) the
there would be a
which was available at no cost from the public health station.
evaluation,
who
by the social worker:
by the older person so that in the event of an emergency
independent
of the other. With this knowledge;
for these two objectives:
First-it
encouraged
of one
had been
objectives,pne
was
the program established
people to select a physician; second,
it encouraged a diagnostic check-up.
From program
the questionnaire
objectives.
through volunteer daily
and operational
basis. This proved unfeasible
in the neighborhood
discovered
information
time
within
the project
as a whole.
persons did not want the responsibility
persons whom they did not know.
at the
other
helped obtain
additional aid
services in the event of emergencies or to check on the well being of residents on a
seemed clear that elderly
collected
records,
It had been one of the initial objectives of the program to establish mutual
to venture of
the
a small network
Furthermore,
From the information of providing
on the
of existing mutual
care-for
they were afraid because of muggings and robberies
out at night even in case of emergency. interview
we had, it
emergency
interpersonal
aid through
relations
informal
Examining
among
arrangements,
elderly
information people
we
already operating
within the project. Mutual aid in the event of sickness or accidents occurred within the closest physical proximity.
Neighbors would
became a matter these and tying
look in on neighbors who were next door or on the same floor.
of articulating in isolated
individuals
based on a physical proximity defeat of a meaningful the “appropriate
various forms of assistance already partially
Volunteer
services
from floor to floor proved feasible. What might have originally
been the
objective,
approach”
and newcomers
the use of volunteers
to volunteer services.
to the self-help
It then
established, implementing
for mutual
network.
aid, was salvaged by re-examining
Concurrent Evaluation: An Approach To Action Research
29
A further example of information systematically collected by research interviews and used to redefine a procedure occurred in a health center located in a low-income urban neighborhood of mixed ethnic origins. The staff of the Center was having a problem with broken appointments. The staff had been sending letters to persons who missed their appointments and in the course of the letter made a new appointment for them. Less than two-thirds of the total appointments made were kept by the patients. The staff of the Center wondered if the patients were unused to the appointment system through long years of experience with clinic care or whether there was dissatisfaction with servicesat the Center. Health aides working for the Center were sent out with questionnaires to interview those who had missed their appointments. They discovered that in the majority of cases,outside events had made the return on a specific date too difficult. Common problems were baby-sitting, an unexpectedly sick child who could not be left, and special facilities for individuals who needed help to get to the Center because of physical or psychological disabilities. Furthermore, patients used to a clinic system did not understand that they were expected to call the Center to report their inability to keep an appointment. Frequently patients had moved and their new address was unknown or they lived with another family whose name appeared on the mail-box. We revised our system of making appointments. No more appointments were made by mail. When an appointment was not kept, an aide visited the home to find out why and frequently to help make plans for anticipated visits. Health teams were much more alerted to the extent to which a patient’s disability might inhibit his return and plans were more consistently made to send aides to assist these patients with return arrangements. Finally, at every appropriate occasion patients were encouraged to telephone the Center when they could not make an appointment. Within a short period of. time, the patient “no-show” rate was greatly reduced and at the same time the staff was maintaining a higher level of information about the problems of their patients. Modification of Standard Research Techniques The use of the survey as a research tool has too often been in one senseextended and in another narrowed. It has been inappropriately extended when interpretations are presented without historical and cultural perspective. We find, for instance, that some of our Spanish-speaking populations do not utiliie preventive child health care facilities, as compared with other groups. What the survey does not tell us is the reluctance on the part of the parents “to make a healthy child sick” as the result of an inoculation or vaccination. Nor is the role of this attitude evaluated in a religious context of sin and divine providence. The survey is all too likely to focus on “attitudes toward health care”. A survey is analogous to the examination of a cross-section of an organism. One must first understand the anatomy of the organism and that part of the structure from which the cross-section is taken. It is critically important to understand the context of the respondent in order to interpret the findings from the survey, in order to make them not just predictive but practicable for new approaches. The survey has been too narrowed by the concern for objectivity which has taken on ritualistic form. For instance, requirements for the role of interviewer generally specify a stranger who asks for information in a detached manner, usually justifying it for “research” and guaranteeing confidentiality -from any but the research team. In concurrent evaluation, the research is intended to work as an
J.Z. Moss
30
integral part of the program-and the interviewers have an expanded role. We found it possible, in surveying for a neighborhood health center, to use interviewers who explained the services of the program to the respondent at the time of interviewing. The information collected was confidential within the program but not exclusively to the research team. We found this interviewing could be better performed by well-trained persons from within the target population. If care is taken to place community people as interviewers in the community, but at a sufficient distance from personal acquaintances, we can manage the problems of confidientiality
and honesty of reports from
respondents.* There is no reason to hold that the information collected, with the interviewer playing the role of program contact and conventional interviewer, cannot be useful to the program and the research. Certainly poor people, constantly questioned by authorities, find it easier to give a straight-forward response to a sympathetic listener who is offering a service than to an “objective stranger”. Relating to the respondent with an exchange of information need not jeopardize the validity of the response, although it does not eliminate the need for the usual kinds of controls for interviewer validity and reliability. Another concern of concurrent evaluation is that of determining what the priority of the program objectives is to the population itself. Often, objectives are designed by professionals whose specialist interests overlook the relevance of those objectives to the population concerned. While good health is the concern of any medical program, it may very well be that problems of employment, child care, and other behavior problems within the family take priority over health. If personnel of the health program become aware of this, they can refer individuals or families for other services either within the program or to other agencies.As the problems are resolved according to the priorities of the individuals rather than the professionals, the health problems of the family will be tackled as other urgencies are eased. The number of persons helped with appropriate referrals can be one measure of the impact of a health program and a means of making professional goals of the program and individual goals of the target population compatible. We must examine those persons directed by the program to resources outside itself. It is necessary to point out however that the specific factors affecting the successof failure of a program in one community are not necessarilyappropriate to the next. How then can we transmit what we have learned in one community to another if our results cannot be generalized? We can follow similar procedures of investigation and compare effective and ineffective implementations of action. Each study becomes a unit in a sample of action programs so that with enough studies of action in varying contexts we can begin to find what general program characteristics are necessary for the selection and implementation of appropriate goals and what population characteristics are most amenable to the acceptance of program innovation. Such generalizations will have the added advantage that they will be derived from life rather than the laboratory.
*Poverty studies have, of course, used community interviewers and a few experimental health programs have used information collected by interviewers for programs.