- Email: [email protected]
Increasing cultural sensitivity in evaluation practice: A South Korean illustration
Studies in Educational Evaluation PERGAMON
Studies
in Educational
Evaluation
25 (2002) 61-69 www.elsevier.com/stueduc
INCREASING CULTURAL SENSITIVITY IN EVALUATION PRACTICE: A SOUTH KOREAN ILLUSTRATION
Nick L. Smith’ and Soojung Jang**
School of Education, Syracuse University Syracuse, USA **Korea Education and Research Information Service, Seoul, South Korea
The Expanding World Scene in Evaluation Professional evaluation activities have rapidly expanded throughout the world in the past few years. Prior to 1995, there were only five national or multi-national evaluation organizations but by 2000 there were over 25 such societies and associations (Patel, 2000; Russon & Russon, 2000). Hundreds of evaluators from scores of countries have participated in international evaluation meetings and conferences held in Vancouver (1995), Stockholm (1997), Chicago (1998), Rome (1998), and Nairobi (1999). In February of 2000, representatives f?om fifteen national and multi-national organizations met in Barbados to discuss possible collaborative activities and relationships, discussions that continue at present (see Russon & Russon, 2000, for a summary of this international activity). The evaluation of international development programs is also growing, requiring that evaluators tiom diverse cultural contexts learn to work better together, and forcing U.S.’ evaluators to deepen their own understanding of cultural influences in evaluation practice. As Bamberger notes: International program evaluation is a booming business, with important and challenging evaluations of development programs being conducted in almost every 0191-491X/02/$ - see front matter 0 2002 Published PII: SOl91-491X(02)00012-3
by Elsevier
Science
Ltd
62
N.L. Smith,
S. Jung
/Studies
in Educational
Evaluation
28 (2002)
61-69
country in the developing world. However, many U.S. domestic evaluation practitioners are not yet familiar with this field. Evaluators of international development programs normally must operate in a very different environment than one would expect to f!nd when evaluating U.S. programs. (2000, p. 95)
Of course, evaluation within the U.S. is not exactly a mono-cultural enterprise itself. The U.S. evaluation profession increasingly recognizes the need for cultural diversity among evaluation practitioners and the necessity of preparing all students of evaluation to work in culturally diverse settings. Clearly, we would no longer thoughtlessly assign a middle-aged, white, male mid-westerner to evaluate a family planning clinic for Hispanic women in a Los Angeles barrio without knowing something about his prior experience, training and sensitivities. Our understanding of cultural differences continues to improve as more U.S. evaluators work in international and domestic cross cultural settings, and as international students increasingly come to the U.S. to study evaluation. Teachers of evaluation, themselves unfamiliar with the home cultures of their own students, are confronted with the challenges of preparing these students to conduct evaluations in both the U.S. and in the students’ own settings. As an illustration of the difftculty of understanding and working with crosscultural differences in evaluation practice, we here review the comparative nature of evaluation in just one non-U.S. setting, South Korea. We examine the nature of South Korean culture and its implications for what is considered useful and valid evaluation in that setting as contrasted with the U.S. setting. We follow Bamberger’s (2000) suggestion that a closer exchange of experiences between U.S. evaluation practitioners and their international colleagues can be mutually beneficial. Understanding Cultural Variations in Evaluation Practice Anyone who has worked outside his or her own cultural setting knows what a considerable impact cultural differences can have on evaluation work. Some cultural differences such as climate, language and dress are dramatic and immediately apparent; a U.S. evaluator getting off the plane in a new setting may be reminded of Dorothy in the Wizard of Oz, “We’re not in Kansas anymore, Toto.” But cultural differences are also more pervasive and subtle than first recognized, such as a different sense of time or different rules of courtesy and protocol. As Chapman (1992) suggests, power relationships, patterns of information flow, and access to resources often follow f&Gal, tribal or political lines that may be invisible to an outsider - even an outsider with substantial in-country experiences. Even more hidden are the tacit beliefs of a culture which not only influence what behavior is appropriate, but why it is so; such as the belief that the individual is inherently more important than the group, or the belief that elders should be respected
N.L. Smith, S. Jang /Studies
in Educational
Evaluation
28 (2002)
61-69
63
above all others, These latter, implicit, aspects of culture can have a profound effect on an evaluation study and are exceedingly difficult for an outsider to discern. Much of our early knowledge of cultural differences in evaluation came from case while working in unfamiliar reports of difficulties U.S. evaluators encountered international settings; for example, as reported in Patton’s (1985) important volume: Culture and Evaluation. Many writers have argued that evaluators who work in international settings must understand the cultures within which they work, whether it be Egypt (Seefeldt, 1985) or the Caribbean (Cuthbert, 1985). Others, however, have argued that many U.S. evaluators fail to appreciate fully the complexity of other cultures, that they treat fundamental cultural differences like simple variations in U.S. stakeholder preferences (Ginsberg, 1988). In their study of the possibility of culturally-sensitive evaluation standards, Smith, Chircop, and Mukherjee (1993) provide an illustration of the differences between the “insider” versus the “outsider” analysis of culture in evaluation. They suggest that while a U.S. analysis tends to define the problems of using U.S. evaluation standards in say, Malta or India, as an issue of technology transfer, as viewed by Maltese and Indian evaluators, the concept of standards itself plays a different cultural role. The concept of standards as employed in the U.S. is much less relevant within the Maltese and Indian traditional cultures. Although standards may be imposed (or “transferred”) from the outside, indigenous standards are unlikely to emerge. (Smith, Chircop, & Mukherjee, 1993, p. 12) In the development of new standards, or in the study of differences in practice, then, it seems important that any analysis of cultural influences be conducted both by members and non-members of the cultures being examined. Reciprocal understanding is essential for effective evaluation practice. We provide several brief examples below of how differences in culture between South Korea and the U.S. affect evaluation practice. The examples are framed in terms of what a U.S. evaluator would need to know in order to work effectively in the South Korean cultural setting. The aspects of evaluation setting selected are based on Smith (1991) and include a general overview of social structure and communication, with a consideration of how these differences in culture subsequently influence the choice of appropriate evaluation methods and relationships with clients, local professionals and stakeholders in general. Evaluation Implications of South Korean Social Structure and Communication Social Structure Like most East Asian countries, South Korea has been influenced by both Confucianism and Buddhism, but more especially Conticianism. Seniority, one of the most important Korean traditions stemming from Confucianism, is still cherished and there
64
N.L. Smith, S. Jung /Studies
in Educational
Evaluation
28 (2002)
61-69
is a clear structure of authority in any kind of social group such as a family, business corporation or school. Greater respect is accorded to elders and authorities, like a father in the family, a top manager in a business corporation, or a principal in a school. People pay considerable attention to family values and prioritize group interests over individual rights. Many marriages are still arranged, although with each partner’s consent. Issues of seniority cannot be missed in explaining Korean behavioral norms. Koreans bow to elders and speak in a more polite tone. Many Koreans feel bad about disagreeing with older persons. It is awkward and uncomfortable to evaluate a person who is older than the evaluator, whatever the evaluation purpose. Salaries and benefits are based on how long one has worked for an organization, and not solely on the quality of achievement. Koreans pay much attention to one’s class membership, order in the family structure, and position in the job hierarchy. Implications: Program evaluations that focus on the performance adequacy of senior staff will generally be problematic. Managerial evaluation approaches are likely to be effective. Evaluations must demonstrate respect for leaders. Since Koreans live in a world fundamentally defined by social relationships, they are expected to know a great deal about the people around them, including their beliefs and attitudes. Not only is public argumentation considered unnecessary but it is actively shunned because engaging in it means to stand out, to risk public disagreement and to lose favor with fellow members of society (Becker, 1986; Galtung, 1981). Implications: Public discussion of strong stakeholder differences inappropriate. Adversarial approaches are not likely to prove useful.
would
be
Conflicts of interest happen in any society and the South Korean way of conflict resolution is, in most cases, based on what is best for the group the individuals belong to. The nation’s interest is especially placed above everything. This phenomenon, of the priority of group interest, is common in most organizations of Korean society. Koreans live in a culture where an authority figure, for example a father in a &tmily, an immediate supervisor in a business team, or a teacher in a class, can exercise considerable control over individuals. Formation of public opinion by a stakeholder group is not as prevalent in South Korea compared to western societies in which individuals have opinions about most issues and have the right to speak out. Many foreigners notice that South Koreans do not seem to demand their rights as individuals; Koreans unquestioningly rely upon, and accept, the right of the hierarchy to make the best decision on their behalf. This is because Koreans think that small concerns (e.g., personal interest) sometimes should be ignored for larger concerns (e.g., group interest). An example will illustrate the point. Several years ago, the Korean government evaluated the city traffic system in Seoul to determine the best way to maxim& public service. Outside evaluators, specialists from human resources, environmental law and transportation, did surveys and interviewed stakeholders throughout Seoul and its suburban area. As a result of the
N.L. Smith, S. Jang /Studies
in Educational
Evaluation
28 (2002)
61-69
65
evaluation’s conclusions, everyone was prohibited from operating his or her automobile once every ten days. This meant that each person could drive a car nine out of ten days, and that these days rotated so that there would be about a ten percent reduction in traffic every day. Most Koreans willingly obeyed the regulation and the traffic situation improved, so the evaluation project was judged successful. It is a South Korean sensibility that an individuals’ sacrifice will benefit the nation’s interest. Implications: Evaluation approaches based on group rights, like consumer models focused on the collective good, would be compatible. Evaluations designed to promote individual gain at the cost of society would not be appropriate. Communication South Korea is composed almost entirely of a one hundred percent homogeneous ethnic group. Koreans highly value education and have great respect for well educated people; kinship and college alumni ties are very strong and comprise a critical part of Korean social relationships. The South Korean literacy rate is near one hundred percent. The Korean language is a unique blend of a distinct phonetic alphabet called Hangul, traditional Chinese characters, a visual ideogram in a written mode, and English, which is taught in junior and senior high schools. South Koreans will usually think that they should communicate with foreigners in English, not in Korean. Needless to say, it is considered a favor to try to learn Korean, and this will make it easier to establish rapport. While language is significant, U.S. evaluators do not have to worry about language deficiency, Koreans are generous to foreigners who can not speak Korean very well. Also, U.S. evaluators can usually find English translators inside the Korean organization. It will be difficult, however, for U.S. evaluators to obtain and interpret authentic and rich information if they do not understand the Korean ethnic communication characteristics of indirection and complementary ways of thinking. U.S. evaluators may observe that there are few South Koreans who answer any question clearly and openly. This does not mean Koreans are shy or uncooperative, rather, it is because they are not as experienced as U.S. citizens in expressing their personal opinions. Many Koreans are reluctant to say anything to strangers, even though the strangers may guarantee confidentiality. Therefore, it is better to start a conversation in a roundabout way, instead of directing the conversation towards data collection right from the start. The Korean method of communication is indirect. This cultural norm of indirection contrasts, in particular, with the U.S. preference for direct communication. Most Koreans do not like to reveal their feelings to others, much less to those they do not know. They use a small number of facial expressions to convey their feelings. This partly stems from the historical influence of Confucianism. Korean ancestors held the philosophy that a serious person; stern or strict, is better than a smiling one (“Koreans Practice Smiling,” 1998). Foreigners say that most Koreans look like they have trouble smiling, have flat
66
N.L. Smith, S. Jang /Studies
in Educational
Evaluation
28 (2002)
61-69
facial expressions, and are poor at expressing opinions verbally. For example, a U.S. evaluator might feel offended in working with Korean staff members because they rarely say “Thank you,” “Excuse me,” or “I’m sorry”, even when they bump into someone else. Sometimes, paradoxically, Koreans cannot express their feelings because they feel so sorry. This lack of expression is not because most Koreans are just shy or rude, but because they value indirection rather than expressing themselves openly. Implications: U.S. evaluators may find interpersonal relationships with Korean clients, program staff, and stakeholders difftcult. Communications may often be unclear. Evaluation approaches requiring extensive stakeholder interaction, such as participative or collaborative models, will be especially problematic. U.S. evaluators will need to enlist the aid of a Korean staff member as a key collaborator throughout the evaluation process. This kind of cultural norm may dictate that respondents answer questions indirectly, thereby complicating the interpretation of responses. U.S. evaluators need to keep in mind that Koreans may be constrained by their inability to articulate their own expressions, feelings, or actions. More often than not, evaluators may meet ambiguous responses which are difficult to interpret, such as “It’s hard to say”, “I can’t tell right this moment”, or “Please, ask my friend.” This is especially true if evaluators want to get valid and reliable information from Koreans about bad personal experiences. In such cases, the U.S. evaluator must turn to Korean staff members for advice. For example, in asking about a high school drop-out experience, it is better to give open-ended questions with enough time for respondents to write their experiences down, than to ask direct specific questions. Evaluators must determine what will be hard to interpret, and look for alternative ways to get more frank responses. Patton (1990, p. 339) cautions that researchers cannot simply presume that they have the right to ask intrusive questions, and many topics may be taboo. Koreans do not share western norms of the acceptability of discussing oneself and one’s experiences in public, even though it is hard for western evaluators to consider it taboo. The reluctance to share strong personal opinions also comes from the. Confucian tradition of complementary thinking. Briefly, Confucianism can be expressed by the terms “moderation and balance”. Confucian thinking never goes to extremes. When a complementary way of response exists, lie “I don’t know”, Koreans will usually prefer it over direct agreement or disagreement. They are reluctant to respond with “strongly yes” or “strongly no” to an interviewer, even though that is what they think. Koreans believe that a balanced response which minimizes the loss of each alternative is best. Implications: Direct questions about personal feelings are likely to violate cultural taboos. Strong attitudes for or against any given position will be difficult to obtain. Most data collection from stakeholders will require high inference measures and the support of Korean professional staff.
N.L. Smith, S. Jang /Studies
in Educorionol Evaluation
28 (2002)
61
61-69
Utility and Validity - Cultural Variations These few examples are illustrative of the many cultural factors to be considered in moving between U.S. and South Korean evaluation contexts. These simple examples have rather obvious implications concerning utility and validity in evaluation practice. It is clear that a number of approaches designed for, and used in, the U.S. context would be both unfeasible and inappropriate within the South Korean cultural context. Such approaches would not be used by Korean evaluators, nor would the methods have the desired effects if they were imposed by some outside agency. It is also clear that the criteria for judging an evaluation’s utility are different in South Korea than in the U.S. In South Korea, for example, a major criterion would be the extent to which the evaluation assisted senior leaders with their vested responsibility to promote the collective good of the group commissioning the evaluation. What about validity? In the U.S., evaluators often seek direct, low inference data in order to maximally discriminate between extreme positions among stakeholders with regard to personal judgments about a program’s worth. This would be difficult in the South Korean context, since the data would need to be high inference, collected in an indirect manner from individuals unaccustomed and perhaps unwilling to take strong positions, let alone to share them with strangers. Different procedures would be required to establish the validity of the data collected and the validity of the inferences the evaluators made about stakeholder opinions. The criteria for establishing the validity of an evaluation would have to include the extent to which the evaluation promoted group rights, provided a balance between extremes, fostered harmony among stakeholders, and preserved respect. Conclusion We offer these examples and implications to emphasize the need to examine carefully the cultural boundedness of evaluation practice. Our simple illustrations by no means capture the complexity confronting a U.S. evaluator working in South Korea, but perhaps they begin to help clarif) the task of understanding cultural differences in evaluation practice. These examples suggest the conceptual tasks required to develop more culturally-sensitive evaluation procedures for U.S. evaluators working in South Korea. Similar work would be required for outside evaluators working in, say, Botswana, or Venezuela, or Jamaica, or.. . , but, of course, with very different results in eachsetting. References Bamberger, M. (2000). The evaluation of international the front. American Journal of Evaluation, 21 (l), 95-102.
development
Becker, C.B. (1986). Reasons for the lack of argumentation International Journal of Intercultural Relations, IO, 75-92.
programs:
A view from
and debate in the
far East.
68
N.L. Smith, S. Jang /Studies
Chapman, D. (1992). Investigative N.L. Smith (Ed.), Varieties of investigative
in Educational
Evaluation
28 (2002)
61-69
strategy in international technical assistance projects. In evaluation. New Directions for Program Evaluation, 5.3,
63-74. Cuthbert, M. (1985). Evaluation encounters in third world settings: A Caribbean perspective. M.Q. Patton (Ed.), Culture and evaluation. New Directions for Program Evaluation, 25, 29-35. Galtung, J. (198 1). Structure, culture and intellectual style: An essay comparing Teutonic, Gallic & Nipponic approaches. Social Science Information, 6, 817-856. Ginsberg, P.E. (1988).
Evaluation
In
Saxonic,
in cross-cultural perspective. Evaluation and Program
Planning, II (2), 189-195. Koreans practice smiling. (1998, December 22). New York Times. Patel, M. (2000). The inaugural conference of the African Evaluation from a report by Mahesh Patel. Evaluation, 6 (2), 231-232. Patton,
M.Q. (Ed.) (1985). Culture and evaluation.
Association:
Extracts
New Directions for Program Evaluation,
25. Patton, M.Q. (1990). Qualitative evaluation and research methods. Newbury Park, CA: Sage. Russon, C., & Russon, K. (Eds.) (2000). The annotated bibliography of international program Kluwer.
evaluation. Dordrecht, The Netherlands:
context.
Seefeldt, F.M. (1985). Cultural considerations In M.Q. Patton (Ed.), Culture and evaluation.
for evaluation consultation in the Egyptian New Directions for Program Evaluation, 25,
69-78. Smith, N.L. (1991).
The context
of investigations
in cross-cultural
evaluations.
Studies in
Educational Evaluation, I7 (I), 3-2 1. Smith, N.L., Chircop, S., & Mukherjee, P. (1993). Considerations on the development culturally relevant evaluation standards. Studies in Educational Evaluation, 19, 3-13.
of
Note 1.
We use the term “U.S.” evaluators since the broader term, “American”, also properly refers to evaluators from other countries such as Canada, Jamaica, and Venezuela; the term “Western” similarly obscures important national and continental differences across various forms of evaluation practice.
The Authors L. SMITH is Professor in the School of Education at Syracuse University. His research interests include the design of investigative research and evaluation studies.
NICK
N.L. Smith, S. Jang /Sfudies
in Educational
Evaluation
28 (2002)
61-69
69
Teaching in the Instructional Design, Development, and Evaluation program, he daily works with students from all the continents of the world. SOOJUNG JANG is Senior Researcher in the Research and Evaluation Division of the Korea Education and Research Information Service, Seoul, South Korea. She earned her Ph.D. from Syracuse University, and is currently working on the effects of diverse educational technologies in K-12 settings.
Studies
in Educational
Evaluation
25 (2002) 61-69 www.elsevier.com/stueduc
INCREASING CULTURAL SENSITIVITY IN EVALUATION PRACTICE: A SOUTH KOREAN ILLUSTRATION
Nick L. Smith’ and Soojung Jang**
School of Education, Syracuse University Syracuse, USA **Korea Education and Research Information Service, Seoul, South Korea
The Expanding World Scene in Evaluation Professional evaluation activities have rapidly expanded throughout the world in the past few years. Prior to 1995, there were only five national or multi-national evaluation organizations but by 2000 there were over 25 such societies and associations (Patel, 2000; Russon & Russon, 2000). Hundreds of evaluators from scores of countries have participated in international evaluation meetings and conferences held in Vancouver (1995), Stockholm (1997), Chicago (1998), Rome (1998), and Nairobi (1999). In February of 2000, representatives f?om fifteen national and multi-national organizations met in Barbados to discuss possible collaborative activities and relationships, discussions that continue at present (see Russon & Russon, 2000, for a summary of this international activity). The evaluation of international development programs is also growing, requiring that evaluators tiom diverse cultural contexts learn to work better together, and forcing U.S.’ evaluators to deepen their own understanding of cultural influences in evaluation practice. As Bamberger notes: International program evaluation is a booming business, with important and challenging evaluations of development programs being conducted in almost every 0191-491X/02/$ - see front matter 0 2002 Published PII: SOl91-491X(02)00012-3
by Elsevier
Science
Ltd
62
N.L. Smith,
S. Jung
/Studies
in Educational
Evaluation
28 (2002)
61-69
country in the developing world. However, many U.S. domestic evaluation practitioners are not yet familiar with this field. Evaluators of international development programs normally must operate in a very different environment than one would expect to f!nd when evaluating U.S. programs. (2000, p. 95)
Of course, evaluation within the U.S. is not exactly a mono-cultural enterprise itself. The U.S. evaluation profession increasingly recognizes the need for cultural diversity among evaluation practitioners and the necessity of preparing all students of evaluation to work in culturally diverse settings. Clearly, we would no longer thoughtlessly assign a middle-aged, white, male mid-westerner to evaluate a family planning clinic for Hispanic women in a Los Angeles barrio without knowing something about his prior experience, training and sensitivities. Our understanding of cultural differences continues to improve as more U.S. evaluators work in international and domestic cross cultural settings, and as international students increasingly come to the U.S. to study evaluation. Teachers of evaluation, themselves unfamiliar with the home cultures of their own students, are confronted with the challenges of preparing these students to conduct evaluations in both the U.S. and in the students’ own settings. As an illustration of the difftculty of understanding and working with crosscultural differences in evaluation practice, we here review the comparative nature of evaluation in just one non-U.S. setting, South Korea. We examine the nature of South Korean culture and its implications for what is considered useful and valid evaluation in that setting as contrasted with the U.S. setting. We follow Bamberger’s (2000) suggestion that a closer exchange of experiences between U.S. evaluation practitioners and their international colleagues can be mutually beneficial. Understanding Cultural Variations in Evaluation Practice Anyone who has worked outside his or her own cultural setting knows what a considerable impact cultural differences can have on evaluation work. Some cultural differences such as climate, language and dress are dramatic and immediately apparent; a U.S. evaluator getting off the plane in a new setting may be reminded of Dorothy in the Wizard of Oz, “We’re not in Kansas anymore, Toto.” But cultural differences are also more pervasive and subtle than first recognized, such as a different sense of time or different rules of courtesy and protocol. As Chapman (1992) suggests, power relationships, patterns of information flow, and access to resources often follow f&Gal, tribal or political lines that may be invisible to an outsider - even an outsider with substantial in-country experiences. Even more hidden are the tacit beliefs of a culture which not only influence what behavior is appropriate, but why it is so; such as the belief that the individual is inherently more important than the group, or the belief that elders should be respected
N.L. Smith, S. Jang /Studies
in Educational
Evaluation
28 (2002)
61-69
63
above all others, These latter, implicit, aspects of culture can have a profound effect on an evaluation study and are exceedingly difficult for an outsider to discern. Much of our early knowledge of cultural differences in evaluation came from case while working in unfamiliar reports of difficulties U.S. evaluators encountered international settings; for example, as reported in Patton’s (1985) important volume: Culture and Evaluation. Many writers have argued that evaluators who work in international settings must understand the cultures within which they work, whether it be Egypt (Seefeldt, 1985) or the Caribbean (Cuthbert, 1985). Others, however, have argued that many U.S. evaluators fail to appreciate fully the complexity of other cultures, that they treat fundamental cultural differences like simple variations in U.S. stakeholder preferences (Ginsberg, 1988). In their study of the possibility of culturally-sensitive evaluation standards, Smith, Chircop, and Mukherjee (1993) provide an illustration of the differences between the “insider” versus the “outsider” analysis of culture in evaluation. They suggest that while a U.S. analysis tends to define the problems of using U.S. evaluation standards in say, Malta or India, as an issue of technology transfer, as viewed by Maltese and Indian evaluators, the concept of standards itself plays a different cultural role. The concept of standards as employed in the U.S. is much less relevant within the Maltese and Indian traditional cultures. Although standards may be imposed (or “transferred”) from the outside, indigenous standards are unlikely to emerge. (Smith, Chircop, & Mukherjee, 1993, p. 12) In the development of new standards, or in the study of differences in practice, then, it seems important that any analysis of cultural influences be conducted both by members and non-members of the cultures being examined. Reciprocal understanding is essential for effective evaluation practice. We provide several brief examples below of how differences in culture between South Korea and the U.S. affect evaluation practice. The examples are framed in terms of what a U.S. evaluator would need to know in order to work effectively in the South Korean cultural setting. The aspects of evaluation setting selected are based on Smith (1991) and include a general overview of social structure and communication, with a consideration of how these differences in culture subsequently influence the choice of appropriate evaluation methods and relationships with clients, local professionals and stakeholders in general. Evaluation Implications of South Korean Social Structure and Communication Social Structure Like most East Asian countries, South Korea has been influenced by both Confucianism and Buddhism, but more especially Conticianism. Seniority, one of the most important Korean traditions stemming from Confucianism, is still cherished and there
64
N.L. Smith, S. Jung /Studies
in Educational
Evaluation
28 (2002)
61-69
is a clear structure of authority in any kind of social group such as a family, business corporation or school. Greater respect is accorded to elders and authorities, like a father in the family, a top manager in a business corporation, or a principal in a school. People pay considerable attention to family values and prioritize group interests over individual rights. Many marriages are still arranged, although with each partner’s consent. Issues of seniority cannot be missed in explaining Korean behavioral norms. Koreans bow to elders and speak in a more polite tone. Many Koreans feel bad about disagreeing with older persons. It is awkward and uncomfortable to evaluate a person who is older than the evaluator, whatever the evaluation purpose. Salaries and benefits are based on how long one has worked for an organization, and not solely on the quality of achievement. Koreans pay much attention to one’s class membership, order in the family structure, and position in the job hierarchy. Implications: Program evaluations that focus on the performance adequacy of senior staff will generally be problematic. Managerial evaluation approaches are likely to be effective. Evaluations must demonstrate respect for leaders. Since Koreans live in a world fundamentally defined by social relationships, they are expected to know a great deal about the people around them, including their beliefs and attitudes. Not only is public argumentation considered unnecessary but it is actively shunned because engaging in it means to stand out, to risk public disagreement and to lose favor with fellow members of society (Becker, 1986; Galtung, 1981). Implications: Public discussion of strong stakeholder differences inappropriate. Adversarial approaches are not likely to prove useful.
would
be
Conflicts of interest happen in any society and the South Korean way of conflict resolution is, in most cases, based on what is best for the group the individuals belong to. The nation’s interest is especially placed above everything. This phenomenon, of the priority of group interest, is common in most organizations of Korean society. Koreans live in a culture where an authority figure, for example a father in a &tmily, an immediate supervisor in a business team, or a teacher in a class, can exercise considerable control over individuals. Formation of public opinion by a stakeholder group is not as prevalent in South Korea compared to western societies in which individuals have opinions about most issues and have the right to speak out. Many foreigners notice that South Koreans do not seem to demand their rights as individuals; Koreans unquestioningly rely upon, and accept, the right of the hierarchy to make the best decision on their behalf. This is because Koreans think that small concerns (e.g., personal interest) sometimes should be ignored for larger concerns (e.g., group interest). An example will illustrate the point. Several years ago, the Korean government evaluated the city traffic system in Seoul to determine the best way to maxim& public service. Outside evaluators, specialists from human resources, environmental law and transportation, did surveys and interviewed stakeholders throughout Seoul and its suburban area. As a result of the
N.L. Smith, S. Jang /Studies
in Educational
Evaluation
28 (2002)
61-69
65
evaluation’s conclusions, everyone was prohibited from operating his or her automobile once every ten days. This meant that each person could drive a car nine out of ten days, and that these days rotated so that there would be about a ten percent reduction in traffic every day. Most Koreans willingly obeyed the regulation and the traffic situation improved, so the evaluation project was judged successful. It is a South Korean sensibility that an individuals’ sacrifice will benefit the nation’s interest. Implications: Evaluation approaches based on group rights, like consumer models focused on the collective good, would be compatible. Evaluations designed to promote individual gain at the cost of society would not be appropriate. Communication South Korea is composed almost entirely of a one hundred percent homogeneous ethnic group. Koreans highly value education and have great respect for well educated people; kinship and college alumni ties are very strong and comprise a critical part of Korean social relationships. The South Korean literacy rate is near one hundred percent. The Korean language is a unique blend of a distinct phonetic alphabet called Hangul, traditional Chinese characters, a visual ideogram in a written mode, and English, which is taught in junior and senior high schools. South Koreans will usually think that they should communicate with foreigners in English, not in Korean. Needless to say, it is considered a favor to try to learn Korean, and this will make it easier to establish rapport. While language is significant, U.S. evaluators do not have to worry about language deficiency, Koreans are generous to foreigners who can not speak Korean very well. Also, U.S. evaluators can usually find English translators inside the Korean organization. It will be difficult, however, for U.S. evaluators to obtain and interpret authentic and rich information if they do not understand the Korean ethnic communication characteristics of indirection and complementary ways of thinking. U.S. evaluators may observe that there are few South Koreans who answer any question clearly and openly. This does not mean Koreans are shy or uncooperative, rather, it is because they are not as experienced as U.S. citizens in expressing their personal opinions. Many Koreans are reluctant to say anything to strangers, even though the strangers may guarantee confidentiality. Therefore, it is better to start a conversation in a roundabout way, instead of directing the conversation towards data collection right from the start. The Korean method of communication is indirect. This cultural norm of indirection contrasts, in particular, with the U.S. preference for direct communication. Most Koreans do not like to reveal their feelings to others, much less to those they do not know. They use a small number of facial expressions to convey their feelings. This partly stems from the historical influence of Confucianism. Korean ancestors held the philosophy that a serious person; stern or strict, is better than a smiling one (“Koreans Practice Smiling,” 1998). Foreigners say that most Koreans look like they have trouble smiling, have flat
66
N.L. Smith, S. Jang /Studies
in Educational
Evaluation
28 (2002)
61-69
facial expressions, and are poor at expressing opinions verbally. For example, a U.S. evaluator might feel offended in working with Korean staff members because they rarely say “Thank you,” “Excuse me,” or “I’m sorry”, even when they bump into someone else. Sometimes, paradoxically, Koreans cannot express their feelings because they feel so sorry. This lack of expression is not because most Koreans are just shy or rude, but because they value indirection rather than expressing themselves openly. Implications: U.S. evaluators may find interpersonal relationships with Korean clients, program staff, and stakeholders difftcult. Communications may often be unclear. Evaluation approaches requiring extensive stakeholder interaction, such as participative or collaborative models, will be especially problematic. U.S. evaluators will need to enlist the aid of a Korean staff member as a key collaborator throughout the evaluation process. This kind of cultural norm may dictate that respondents answer questions indirectly, thereby complicating the interpretation of responses. U.S. evaluators need to keep in mind that Koreans may be constrained by their inability to articulate their own expressions, feelings, or actions. More often than not, evaluators may meet ambiguous responses which are difficult to interpret, such as “It’s hard to say”, “I can’t tell right this moment”, or “Please, ask my friend.” This is especially true if evaluators want to get valid and reliable information from Koreans about bad personal experiences. In such cases, the U.S. evaluator must turn to Korean staff members for advice. For example, in asking about a high school drop-out experience, it is better to give open-ended questions with enough time for respondents to write their experiences down, than to ask direct specific questions. Evaluators must determine what will be hard to interpret, and look for alternative ways to get more frank responses. Patton (1990, p. 339) cautions that researchers cannot simply presume that they have the right to ask intrusive questions, and many topics may be taboo. Koreans do not share western norms of the acceptability of discussing oneself and one’s experiences in public, even though it is hard for western evaluators to consider it taboo. The reluctance to share strong personal opinions also comes from the. Confucian tradition of complementary thinking. Briefly, Confucianism can be expressed by the terms “moderation and balance”. Confucian thinking never goes to extremes. When a complementary way of response exists, lie “I don’t know”, Koreans will usually prefer it over direct agreement or disagreement. They are reluctant to respond with “strongly yes” or “strongly no” to an interviewer, even though that is what they think. Koreans believe that a balanced response which minimizes the loss of each alternative is best. Implications: Direct questions about personal feelings are likely to violate cultural taboos. Strong attitudes for or against any given position will be difficult to obtain. Most data collection from stakeholders will require high inference measures and the support of Korean professional staff.
N.L. Smith, S. Jang /Studies
in Educorionol Evaluation
28 (2002)
61
61-69
Utility and Validity - Cultural Variations These few examples are illustrative of the many cultural factors to be considered in moving between U.S. and South Korean evaluation contexts. These simple examples have rather obvious implications concerning utility and validity in evaluation practice. It is clear that a number of approaches designed for, and used in, the U.S. context would be both unfeasible and inappropriate within the South Korean cultural context. Such approaches would not be used by Korean evaluators, nor would the methods have the desired effects if they were imposed by some outside agency. It is also clear that the criteria for judging an evaluation’s utility are different in South Korea than in the U.S. In South Korea, for example, a major criterion would be the extent to which the evaluation assisted senior leaders with their vested responsibility to promote the collective good of the group commissioning the evaluation. What about validity? In the U.S., evaluators often seek direct, low inference data in order to maximally discriminate between extreme positions among stakeholders with regard to personal judgments about a program’s worth. This would be difficult in the South Korean context, since the data would need to be high inference, collected in an indirect manner from individuals unaccustomed and perhaps unwilling to take strong positions, let alone to share them with strangers. Different procedures would be required to establish the validity of the data collected and the validity of the inferences the evaluators made about stakeholder opinions. The criteria for establishing the validity of an evaluation would have to include the extent to which the evaluation promoted group rights, provided a balance between extremes, fostered harmony among stakeholders, and preserved respect. Conclusion We offer these examples and implications to emphasize the need to examine carefully the cultural boundedness of evaluation practice. Our simple illustrations by no means capture the complexity confronting a U.S. evaluator working in South Korea, but perhaps they begin to help clarif) the task of understanding cultural differences in evaluation practice. These examples suggest the conceptual tasks required to develop more culturally-sensitive evaluation procedures for U.S. evaluators working in South Korea. Similar work would be required for outside evaluators working in, say, Botswana, or Venezuela, or Jamaica, or.. . , but, of course, with very different results in eachsetting. References Bamberger, M. (2000). The evaluation of international the front. American Journal of Evaluation, 21 (l), 95-102.
development
Becker, C.B. (1986). Reasons for the lack of argumentation International Journal of Intercultural Relations, IO, 75-92.
programs:
A view from
and debate in the
far East.
68
N.L. Smith, S. Jang /Studies
Chapman, D. (1992). Investigative N.L. Smith (Ed.), Varieties of investigative
in Educational
Evaluation
28 (2002)
61-69
strategy in international technical assistance projects. In evaluation. New Directions for Program Evaluation, 5.3,
63-74. Cuthbert, M. (1985). Evaluation encounters in third world settings: A Caribbean perspective. M.Q. Patton (Ed.), Culture and evaluation. New Directions for Program Evaluation, 25, 29-35. Galtung, J. (198 1). Structure, culture and intellectual style: An essay comparing Teutonic, Gallic & Nipponic approaches. Social Science Information, 6, 817-856. Ginsberg, P.E. (1988).
Evaluation
In
Saxonic,
in cross-cultural perspective. Evaluation and Program
Planning, II (2), 189-195. Koreans practice smiling. (1998, December 22). New York Times. Patel, M. (2000). The inaugural conference of the African Evaluation from a report by Mahesh Patel. Evaluation, 6 (2), 231-232. Patton,
M.Q. (Ed.) (1985). Culture and evaluation.
Association:
Extracts
New Directions for Program Evaluation,
25. Patton, M.Q. (1990). Qualitative evaluation and research methods. Newbury Park, CA: Sage. Russon, C., & Russon, K. (Eds.) (2000). The annotated bibliography of international program Kluwer.
evaluation. Dordrecht, The Netherlands:
context.
Seefeldt, F.M. (1985). Cultural considerations In M.Q. Patton (Ed.), Culture and evaluation.
for evaluation consultation in the Egyptian New Directions for Program Evaluation, 25,
69-78. Smith, N.L. (1991).
The context
of investigations
in cross-cultural
evaluations.
Studies in
Educational Evaluation, I7 (I), 3-2 1. Smith, N.L., Chircop, S., & Mukherjee, P. (1993). Considerations on the development culturally relevant evaluation standards. Studies in Educational Evaluation, 19, 3-13.
of
Note 1.
We use the term “U.S.” evaluators since the broader term, “American”, also properly refers to evaluators from other countries such as Canada, Jamaica, and Venezuela; the term “Western” similarly obscures important national and continental differences across various forms of evaluation practice.
The Authors L. SMITH is Professor in the School of Education at Syracuse University. His research interests include the design of investigative research and evaluation studies.
NICK
N.L. Smith, S. Jang /Sfudies
in Educational
Evaluation
28 (2002)
61-69
69
Teaching in the Instructional Design, Development, and Evaluation program, he daily works with students from all the continents of the world. SOOJUNG JANG is Senior Researcher in the Research and Evaluation Division of the Korea Education and Research Information Service, Seoul, South Korea. She earned her Ph.D. from Syracuse University, and is currently working on the effects of diverse educational technologies in K-12 settings.