- Email: [email protected]
Cross-National Comparisons in Education: Findings from PISA
Cross-National Comparisons in Education: Findings from PISA Ming M Chiu, Purdue University, West Lafayette, IN, USA Sung W Joh, Seoul National University, Seoul, Korea Ó 2015 Elsevier Ltd. All rights reserved.
Abstract Multilevel analyses of large-scale international studies’ data have shown how characteristics at multiple levels (country, family, school, student) influence student achievement. Students in countries that are richer academically outperform other students, in part because they have more resources at home (family socioeconomic status, books, etc.) and school (teachers, schoolmates, educational materials, etc.). Students in countries with greater household inequality, inequality across schools, or economic segregation of richer students from poorer students show lower academic achievement than other students. Cultural values and family characteristics showed interaction effects. Together, these results show the importance of an ecological approach to understanding student achievement.
Introduction Student achievement on international tests differs sharply across countries. Students in Finland, Korea, Japan, and Canada consistently score over 2 standard deviations higher than students in Indonesia, Panama, Peru, and Kyrgyzstan in mathematics, reading, and science across several international comparisons. These differences in academic achievement might stem from the countries’ economic and cultural differences. First, the countries with higher average test scores are generally much wealthier (e.g., as measured by gross domestic product (GDP) per capita) and more equal economically (e.g., income Gini) than the countries with lower average test scores. These differences suggest that both greater educational resources and more equal distribution contribute to greater student learning and academic achievement. Second, the countries within each set are culturally diverse, suggesting that cultural values do not account for differences in test scores. We examine the antecedents of these differences more systematically by adopting Bronfenbrenner’s (2005) ecological framework and discussing the results of previous international studies. Bronfenbrenner (2005) modeled children’s interactions with surrounding environments including persons and entities within immediate and direct contexts (such as family and school), the relationships across these contexts, and the
broader country context (such as economic and cultural differences). Economic and cultural differences across and within countries can fuel differences in resources across families, schoolmates, schools, and students that influence students’ learning opportunities and subsequent achievement. Lastly, we discuss the methodologies involved in collecting and analyzing large-scale international data.
Ecology of Student Achievement Student interactions with their environment can influence their academic achievement (see Figure 1; Bronfenbrenner, 2005). Students have frequent and direct contacts with family (or any immediate context to create a microsystem), which offers learning opportunities on which they can capitalize to learn and achieve in school. For instance, students living in families with higher socioeconomic status (SES) tend to outperform other students on achievement tests (Baker et al., 2002). Connections between family and school contexts (or any two microsystems that form a mesosystem) can also influence academic achievement. For example, when parents are more involved in children’s school activities, their children show higher academic performance (Chiu, 2010). Thus, both the immediate contexts and the interactions among them influence
Country GDP per capita (+) Family inequality School inequality Economic segregation
Family Family SES Educational resources at home …
Student Academic achievement
School Schoolmates Teachers …
Figure 1 Ecological model of relationships among country, family, school, and student academic achievement. GDP, gross domestic product; SES, socioeconomic status.
342
International Encyclopedia of the Social & Behavioral Sciences, 2nd edition, Volume 5
http://dx.doi.org/10.1016/B978-0-08-097086-8.92144-5
Cross-National Comparisons in Education: Findings from PISA
students’ academic achievement. A more distal context, such as a parent’s stress at work, might result in less school involvement and their children’s poorer academic performance (links between distal and immediate contexts are called exosystems, Bronfenbrenner, 2005). These microsystems, mesosystems, and exosystems are all under the influences of the larger cultural contexts, such as the economy and cultural values of the society (macrosystem). For example, students in richer countries showed higher academic achievement, and had stronger links between family cultural communication and academic performance (Chiu, 2007). Hence, ecologies of microsystems, mesosystems, exosystems, and macrosystems can help us understand the factors that influence student achievement within and across countries. The results discussed in this article are largely drawn from the Organization for Economic Cooperation and Development’s Program for International Student Assessment (OECD-PISA; OECD, 2010). OECD-PISA assessed 193 076 15-year-old students from 41 countries in 2000–02; 276 165 15-year-old students from 41 countries in 2003; and 475 760 15-year-old students from 65 countries in 2009 (see Table 1). They also asked students and principals to fill out questionnaires. International experts from participating OECD and non-OECD countries defined mathematics, reading, and science achievement; built assessment frameworks; created test items; forward translated them; backward translated them; and pilot tested them to check their validity and reliability (for details and sample items, see OECD 2002, and www.pisa.oecd. org). Participating students completed a 2-h assessment booklet and then, a 30- to 40-min questionnaire. Studies using OECD-PISA data have shown that differences in student achievement at the country macrosystem, school microsystem, and student levels are all substantial (Chiu and Khoo, 2005). Country macrosystem differences account for 31% of the differences in students’ mathematics achievement (24% for reading achievement and 22% for science achievement). Meanwhile, school differences account for 25, 30, and 26% of the differences in mathematics, reading, and science achievements, respectively. Lastly, student-level differences account for 44, 46, and 52% of the differences in mathematics, reading, and science achievements, respectively. Together, these results show the importance of examining factors that influence student academic achievement within a country’s ecology.
Economic Differences Students in countries with greater economic productivity (real GDP per capita) show higher academic achievement (Chiu and McBride-Chang, 2010). Richer countries can raise student achievement through extra resources at the country, family, and school levels (Chiu, 2009). These resources provide students with more learning opportunities. In richer countries, governments tend to invest more in education by building more educational infrastructure (schools, libraries, and museums) and providing better educational resources and teacher training. The availability of these educational resources and opportunities can directly enhance students’ learning. Students in richer countries can also benefit indirectly through higher nutritional standards or better health care. These help ensure
Table 1 The years in which the following countries participated in OECD-PISA Country
2000–02
Argentina Albania Argentina Australia Austria Azerbaijan Belgium Brazil Bulgaria Canada Chile Colombia Croatia Czech Republic Denmark Dubai (UAE) Estonia Finland France Germany Greece Hong Kong–China Hungary Iceland Indonesia Ireland Israel Italy Japan Jordan Kazakhstan Korea Kyrgyz Republic Latvia Liechtenstein Lithuania Luxembourg Macao–China FYR Macedonia Mexico Republic of Montenegro The Netherlands New Zealand Norway Panama Peru Poland Portugal Qatar Romania Russian Federation Republic of Serbia Serbia and Montenegro Shanghai (China) Singapore Slovak Republic Slovenia Spain Sweden
X X
2003
X X
X X
X X X
X X X
X
X X
X X
X X X X X X X X X X X X
X X X X X X X X X
X
X
X X
X X
X
X X
X X
X X
X
X X X
X X X
X X X
X X
X X
X
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
X
X X X
X X
X X X X X X (Continued)
343
344
Cross-National Comparisons in Education: Findings from PISA
Table 1 The years in which the following countries participated in OECD-PISAdcont'd Country
2000–02
2003
2009
Switzerland Chinese Taipei Thailand Trinidad and Tobago Tunisia Turkey United Kingdom United States Uruguay
X
X
X
X
X X X X X X X X X
X X
X X X X X
students are healthy and have adequate energy when they study, which in turn contributes to better academic performance in richer countries.
Family SES Families in richer countries tend to have more material, human, and cultural capitals that help their children learn more (Chiu and Khoo, 2005). In richer countries, families often have more educational resources at home (such as books, computers, and tutors), which provide students with more learning opportunities. By capitalizing on these learning opportunities at home, students in richer countries can academically outperform students from poorer countries. Apart from material support, children also benefit from family members’ human capital (Chiu and McBride-Chang, 2010). As the overall education level is higher in richer countries, parents in these countries often completed more years of schooling and have higher aspirations for their children’s educational attainment. As a result, these parents encourage their children to study, excel academically, and complete higher levels of education. Furthermore, families with both parents are more likely to have more and better educational resources at home and are more involved in their children’s schooling. Parents in richer countries also often give their children more opportunities to learn cultural knowledge, skills, and values, which contribute to better academic performance (Chiu and Chow, 2010). Cultural possessions at home, such as paintings and poetry books, can highlight the importance of one’s culture and facilitate communication about cultural values and norms. This family cultural capital can influence students’ academic performance in two ways. First, families with more cultural capital often understand their children’s schools better and are more connected with these schools, thereby are more likely to be involved in activities and works in these schools. Second, this cultural capital also helps children better understand school norms and their teachers and classmates’ expectations in school. So, these children often behave appropriately in school and have better relationships with teachers and classmates. With greater support from their parents, teachers, and fellow students, these children feel a stronger sense of belonging at school, are more motivated to learn, and have greater academic success than other students (Chiu et al., 2012).
School Resources At the school level, students in richer countries often have more school resources, teacher resources, and schoolmates who share educational resources with one another, compared to students in poorer countries (Chiu, 2010). Schools in richer countries tend to offer more hours of classroom instruction and have better facilities and teaching materials. By utilizing these school facilities and materials, students in richer countries can learn more than those in poorer countries. Schools in richer countries often have teachers with higher qualifications and better training (e.g., teacher certification), so their students typically show higher academic achievement (Chiu and Khoo, 2005). These teachers are more capable of using effective teaching methods and raising students’ learning interest, all of which enhance student learning. Also, these teachers are more likely to create a supportive classroom atmosphere of mutual respect and caring, and foster students’ feelings of being valued for academic efforts, reinforcing their students’ school engagement. As discussed earlier, students who have better relationships with their teachers feel a greater sense of belonging at their school and higher academic achievement (Chiu et al., 2012). Furthermore, teachers in wealthier nations are more likely to choose a broader degree of curriculum coverage, a more advanced level of learning activities, and more advanced materials, compared to those in poorer countries. These teachers often have higher expectations of their students and help them realize those expectations through higher academic achievement. Students enrolled in privileged schools are perceived as smarter. Their teachers are likely to have higher expectations of them and are willing to invest more in their learning compared to teachers of other students. Also, teachers can directly or indirectly communicate these expectations to their students, such as adopting a more advanced level of learning activities, thereby affecting their expectations and behaviors. These advantages help students in privileged schools learn more than their counterparts in poorer schools. Students in schools with good disciplinary climates also show higher academic achievement (Chiu and Chow, 2011). A good disciplinary climate at school provides students with a physically and psychologically safe school environment. These feelings of safety throughout a school contribute to stronger school engagement, and students can concentrate on their learning in this environment. As schools in richer countries often have better disciplinary climates, students in these schools tend to have greater school engagement and higher academic achievement.
Schoolmates Schoolmates can affect one another’s learning through their educational resources and group culture (Chiu and Khoo, 2005). Compared to their counterparts in poorer countries, families in wealthier nations are more educated, are more capable in creating an enriched learning environment, and have more positive attitudes toward formal education. When students share their resources and experiences with their
Cross-National Comparisons in Education: Findings from PISA
schoolmates, they learn more and show higher academic achievement. Students with proportionately more high-SES and higher-achieving schoolmates showed better academic performance than those with more low-SES and lowerachieving schoolmates. Higher-achieving students are more likely than other students to value learning and academic success, and thereby support such a school culture at school, which supports other students’ learning.
Microsystem Effects in Richer vs Poorer Countries Country wealth also moderates family or school characteristics’ links with student learning (Heyneman and Loxley, 1983). Distinguishing between different types of family resources can explain the apparently contradictory results regarding whether the link between family resources and student achievement is stronger or weaker in richer countries (Chiu, 2007). In richer countries, physical resources at home, such as books and computers, have weaker links to student achievement, but intangible family resources, such as family involvement and communication, have stronger links to student achievement. As richer countries often have more public physical resources (e.g., library books, museum exhibits) that can substitute for a family’s physical resources, these public resources dilute the importance of family learning opportunities, thereby weakening their links to student achievement (modernization). Family’s physical resources (e.g., books) are strongly linked to student achievement in poorer countries, but much less so in richer countries whose GDP per capita exceeds $16 000 per year (in 1990 US dollars; Gamoran and Long, 2006). While basic physical educational resources (textbooks, maps) are widely available in public arenas (schools, libraries, etc.) in richer countries, these physical resources are not as publicly available in poorer countries (especially schools in Latin American countries), allowing family physical resources to have larger effects on student achievement. On the other hand, the widespread availability of public physical resources (such as public libraries and museums) in richer countries also raises the value of intangible processes such as family communication (complementary intangible processes, Chiu, 2007). Unlike physical resources (e.g., textbooks), processes are intangible and dynamic (e.g., family communication, teacher–student). For example, a student benefits from reading an extra book, but that benefit can be magnified substantially by discussing the book with a parent or a teacher (like a left shoe is more useful with a right shoe). Indeed, children in richer countries often spend more time with their parents due to fewer competing siblings, less parent time on housework, and multitasking parents. Thus, family intangible processes’ links to student achievement in richer countries are stronger, while those of family physical resources are weaker.
Inequalities and Their Mechanisms In addition to a country’s total educational resources, the distribution of these resources influences student achievement (Chiu and Khoo, 2005). Specifically, inequality of family
345
income (family inequality), inequality of school resources (school inequalities), and degree of separation of rich students from poor students into different sets of schools (economic segregation) show effects on student achievement.
Family Inequality National and school policies can influence students’ family inequality, school inequalities, and economic segregation (Chiu, 2009). Regressive taxes, corruption, and a weak legal system can increase inequality of family incomes (family inequality), while progressive taxes, social welfare, transparency, and health care and safety standards (e.g., clean water) can reduce it. Influenced by national policies, family inequality reduces overall student achievement through fewer overall educational resources, diminishing marginal returns, and privileged student bias. Greater family inequality results in less overall investment in educational resources, which reduces academic achievement among both poor and rich students (fewer educational resources, Chiu, 2009). The elite of very unequal countries can send their children to private schools, so they might advocate for fewer public school resources and greater family payment for each child’s education. Even if the elite send their children to public schools, they have many more educational resources (e.g., books or tutors) at home than others, so they may not advocate for more public spending. When students’ families have to pay for much of their education, poorer parents cannot afford the optimal amount of education for their children, resulting in suboptimal educational resources in the society. As a result, poorer students have fewer educational resources and learn less, thereby reducing overall achievement. Family inequality lowers overall student achievement due to diminishing marginal returns (Chiu and Khoo, 2005). Consider a thirsty woman and two glasses of water. She greatly values the first glass of water and drinks it all. Her thirst quenched, she does not finish the extra glass of water, revealing its lower value (diminishing marginal returns). Likewise, rich students have more educational resources than poor students do, and a richer student typically benefits less from an extra book than a poorer student does. With a less equal distribution of a fixed set of resources, richer students have extra resources and obtain less utility from them compared to poorer students who have fewer resources, resulting in less efficient utility of resources and lower education outcomes overall. In school systems with larger distribution inequality, privileged parents have more incentive to use their capital to get more educational resources for their children (Chiu and Khoo, 2005). Schools that allocate more resources to richer students exacerbate the effect of disparities in income. Because of diminishing marginal returns, greater privileged student bias results in lower overall student achievement. Some methods of resource allocation (open markets, cronyism, and social–cultural affinity) allow privileged parents to move more resources to their children compared to others (uniform or truly random allocations; Chiu, 2009). In open markets, privileged parents can use their financial capital by moving to a rich neighborhood to give their children richer schoolmates for example. They can use their social capital through cronyism, for example, by asking the vice principal to
346
Cross-National Comparisons in Education: Findings from PISA
assign their child to a course with a better teacher. Through their human capital, privileged parents can teach their children social norms and cultural capital to create greater affinity with teachers. Teachers have higher expectations of these students and often give them more attention and assistance (cultural gatekeeping).
School Inequality Countries with greater family inequality often have greater school inequalities (Chiu, 2010). A system of strictly free market, private schools without national standards (e.g., no teacher certification) can magnify the impact of family inequalities as richer families pay higher tuition for their children to attend richer schools with more educational resources (teacher quality, computers, richer schoolmates, and so on). In contrast, a universal public school system with required teacher certification standards can reduce inequalities among students. Schools with greater inequalities showed lower academic achievement among mostly poor students (Chiu, 2009). As with family inequality, school inequalities are accompanied by diminishing marginal returns, which result in lower overall achievement among poorer students. Furthermore, school resources (books, teachers, classmates) and family resources (books, siblings, parents) are mostly equivalent substitutes for one another; richer students often have a book at home that is largely equivalent to a book at school, so the latter’s absence does not affect the richer student’s academic achievement (substitution effect). Hence, fewer school resources did not reduce richer students’ academic achievement.
Economic Segregation Economic segregation is also related to academic achievement (Chiu and Khoo, 2005). The degree of economic segregation differs across school systems. In the US, schools are often funded through local taxes, so richer students often attend one set of schools in richer neighborhoods, while poorer students attend a separate set of schools in poorer neighborhoods. In contrast, rich and poor students often mix together in the same schools in other countries such as Japan and Finland. Economic segregation reduces interactions between richer and poorer students, resulting in less diverse interactions for all students. Less exposure to experiences of students in other economic strata results in less shared knowledge, less learning, and lower academic achievement among all students (diversity effect; Chiu, 2009).
A hierarchical nation emphasizes authority, obedience and hierarchical status (e.g., Russia), while an egalitarian country emphasizes equal status and treatment of one another (e.g., Sweden). Also, a collectivist society favors group interests (e.g., Hong Kong), while an individualistic nation favors individual interests (e.g., New Zealand). Cultural values did not directly affect students’ test scores, but they interact with the effects of family characteristics on academic achievement (Chiu, 2007). As egalitarian societies have less rigid societal status, individuals have greater economic and social mobility through their individual skills and talents. Thus, students in these countries often view academic achievement as linked to future success, so they are more likely to study harder and achieve greater academic success. Furthermore, students are more motivated to make better use of their resources in egalitarian societies, yielding stronger links between family resources and student achievement (egalitarian motivation). Specifically, family SES and numbers of books at home had stronger links to academic achievement in more egalitarian cultures. Meanwhile, students in collectivist societies are more inclined to cooperate, support, and learn from their schoolmates and family members (Inglehart et al., 2004). In more collective societies, people tend to rely more on their extended family members who often live nearby, and so students are more likely to benefit from them. As extended family resources dilute the effects of immediate family resources in collective societies, the link between immediate family resources and academic achievement is weaker than in individualistic societies (collectivist dilution; Chiu, 2007). Specifically, single parents, family SES, resident grandparents, and birth order were all less negatively linked to test scores in more collectivist cultures.
Methodological Issues International studies require appropriate data samples, precise test and questionnaire designs, and suitable analytic tools (see Table 2; Chiu, 2010). Ideally, data samples are representative of the populations, with few sampling errors and no missing data. Meanwhile, tests and questionnaires aim to maximize data validity by minimizing measurement errors and maximizing equivalence across countries. Tests seek to optimize coverage of target concepts and skills. Analyses can model multiple levels of data, cross-level moderation and mediation, and multiple outcomes.
Cultural Differences
Data Samples
Countries have different cultural values and the ways they address basic societal issues (Hofstede, 2003). People learn these values through both formal socialization, including direct teachings of parents and teachers, and informal socialization, such as daily exposure to social norms. Among different cultural values, two basic societal issues are particularly important to students’ academic achievement, and they are the degree of hierarchy and obedience to authority vs equality (hierarchical vs egalitarian), and favoring group interests vs individual interests (collectivism vs individualism).
To create a representative sample of a country, researchers typically categorize schools (e.g., by neighborhood SES), randomly select schools from each category, and then randomly select students from each selected school (stratified samples) with corresponding weights for each student. To reduce sampling error, researchers select several subsamples to create plausible values, which must be modeled in the subsequent analyses. In such a huge data collection enterprise, missing data are typically unavoidable. Missing data can reduce estimation
Cross-National Comparisons in Education: Findings from PISA
Table 2
Strategies to address each analytical difficulty
Analytical difficulty
Strategy
l
l
l
l
Representative sample Sampling error l Missing data l
Measurement equivalence of test and questionnaire items
l
Wide coverage of skills while minimizing student fatigue and learning effects l Competence estimates from test answers l Measurement errors on questionnaires l Nested data of students within schools within countries l Cross-level moderation l Indirect, mediation effects l
False positives
l
Simultaneous outcomes
Stratified sample Plausible values l Markov Chain Monte Carlo multiple imputation l Forward and backward translation of each item; l Multigroup Rasch model l Balanced incomplete block test
l
Factor Analysis Item Response models l Factor Analysis l Item Response models l Multilevel analysis (aka Hierarchical linear modeling) l Random effects model l Multilevel mediation tests l Structural equation models l Two-stage linear step-up procedure l Multivariate outcome models l
efficiency, complicate data analyses, and bias results. Markov chain Monte Carlo multiple imputation estimates the values of the missing data, which addresses these missing data issues more effectively than deletion, mean substitution, or simple imputation.
Tests and Questionnaires Tests and questionnaires have special designs. Test items are divided into overlapping subsets (subtests) in a balanced incomplete block test design to (1) maximize coverage of target concepts and skills, (2) reduce student fatigue, and (3) reduce test-learning effects. To reduce measurement error on questionnaires, multiple items were used for each theoretical construct (e.g., SES) to create an index that is more precise than any single variable. To test for data validity, translations and a multigroup Rasch model are used. Each item on a test or a questionnaire is forward translated into the new language of the test by one person and then backward translated by another to check the quality of the translation. Multigroup Rasch models for each item in each country test whether the measures are equivalent across countries. Unlike factor analysis, a multigroup Rasch model has the advantages of requiring only one invariant anchor item across countries and modeling heterogeneous use of the ordinal rating scale. To capitalize on the special designs of the tests and questionnaires to reduce measurement error, researchers use factor analyses and item response models. A factor analysis tests whether a set of test items reflects one or more underlying achievement dimensions (e.g., mathematics achievement vs reading achievement). For each construct, its test item’s characteristics (difficulty, discrimination, chance) are estimated
347
with an item response model. Some test items capture higher levels of a construct more precisely, while others capture lower levels of a construct more precisely (item difficulty). Likewise, each test item’s precision for distinguishing each level of a construct can differ (discrimination). The chance parameter models the likelihood of successfully guessing a correct answer for a test item. For questionnaire items, factor analysis and item response models are also used. A factor analysis tests whether a set of questions that appears to test similar constructs (e.g., mother’s and father’s years of schooling) reflects one or more underlying constructs (e.g., SES). For each construct, its questionnaire item’s characteristics (difficulty, discrimination) are estimated with an item response model. Some questionnaire items capture higher levels of a construct more precisely, while others capture lower levels of a construct more precisely (item difficulty). Likewise, each question’s precision for distinguishing each level of a construct can differ (discrimination).
Analysis As the data are nested (students within schools within countries), a multilevel analysis of weighted, plausible values is needed (also known as hierarchical linear modeling). Students in the same school (or country) likely resemble one another more than students in different schools (or countries, group heterogeneity). Applying an ordinary least squares regression to nested data yields biased standard errors. In contrast, a multilevel analysis yields appropriate standard errors. For multilevel analyses, at least partial metric and partial scalar invariance of the measurement instruments are needed to compare latent means and regression coefficients over countries (Davidov et al., 2011). Testing many hypotheses increases the likelihood of a false positive. To control for the false discovery rate, the two-stage linear step-up procedure is used, which outperformed 13 other methods in computer simulations. As the data are nested, cross-level moderation and mediation tests are needed. A random effects model precisely tests whether the relationship between an explanatory variable and an outcome variable differs across contexts or interacts with another explanatory variable via random effects models. The empirical M-test corrects for potential nonnormal distributions in nested data by using multiple data simulations. Multivariate outcome models can be used to analyze multiple outcome variables simultaneously (e.g., mathematics, reading, and science tests). Ignoring the relationships among outcome variables can result in correlated residuals that underestimate standard errors. Multilevel, multivariate outcome models address this issue to yield proper standard errors. In short, international studies require high-quality data, precise tests and questionnaires, and appropriate analytic tools. Stratified samples, representative weights, plausible values, and Markov chain Monte Carlo multiple imputation can improve their data quality. A balanced incomplete block test design, construct indices based on sets of questionnaire items, forward and backward translations, multigroup Rasch models, factor
348
Cross-National Comparisons in Education: Findings from PISA
analyses, and item response models can help increase data validity. Lastly, multilevel, multivariate outcome analysis of weighted, plausible values; two-stage linear step-up procedure; random effects model; and empirical M-tests can be used to analyze these data.
Future Research The availability of international data sets, their data beyond student achievement, and the potential expansion of databases portend tremendous growth in international research in the coming decade. Research on international comparisons of education is accelerating through large, international teams of scholars collecting enormous, high-quality data sets and making them freely available and easily accessible by Website downloads (e.g., www.pisa.oecd.org). Thus, scholars can use them readily both for research and teaching. The data can benefit young scholars, especially students, by providing data with which they can immediately test their hypotheses. While most research has focused on the student test data, the extensive student questionnaire data present invaluable opportunities for analyses of other aspects of schooling aside from achievement, such as social relationships, student discipline, emotional sense of belonging at school, and so on. Furthermore, the questionnaire responses of parents, teachers, and principals can be examined as outcome variables in addition to their relationships to student outcomes. Future research will link large-scale, cross-country data sets and create longitudinal data sets. Linked data sets combine a student achievement database with school databases or country databases to model the effects on student outcomes of school or country variables that were not available in the original student database. Longitudinal data sets follow individual students, which both control for individual student differences for more precise results and allow testing of hypotheses involving time. Altogether, these are promising developments for large-scale international research.
See also: Comparative Research in Education: Iea Studies; Cross-Cultural Study of Education; Education, Economics of; Family and Schooling; Teacher Behaviours and Student Outcomes.
Bibliography Baker, D.P., Goesling, B., Letendre, G.K., 2002. Socioeconomic status, school quality, and national economic development: a cross-national analysis of the “HeynemanLoxley effect” on mathematics and science achievement. Comparative Education Review 46, 291–312. Bronfenbrenner, U., 2005. Making Human Beings Human: Bioecological Perspectives on Human Development. Sage, Thousand Oaks, CA.
Chiu, M.M., 2007. Families, economies, cultures and science achievement in 41 countries: country, school, and student level analyses. Journal of Family Psychology 21, 510–519. Chiu, M.M., 2009. Inequalities’ harmful effects on both disadvantaged and privileged students: sources, mechanisms, and strategies. Journal of Education Research 3, 109–128. Chiu, M.M., 2010. Inequality, family, school, and mathematics achievement. Social Forces 88 (4), 1645–1676. Chiu, M.M., Chow, B.W.-Y., 2011. Classroom discipline across 41 countries: school, economic, and cultural differences. Journal of Cross-Cultural Psychology 42, 516–533. Chiu, M.M., Chow, B.W.-Y., 2010. Culture, motivation, and reading achievement: high school students in 41 countries. Learning and Individual Differences 20, 579–592. Chiu, M.M., Khoo, L., 2005. Effects of resources, inequality, and privilege bias on achievement: country, school, and student level analyses. American Educational Research Journal 42, 575–603. Chiu, M.M., McBride-Chang, C., 2010. Family and reading in 41 countries: differences across cultures and students. Scientific Studies of Reading 14, 514–543. Chiu, M.M., Pong, S.L., Mori, I., Chow, B.W.Y., 2012. Immigrant students’ cognitive and emotional engagement at school: a multilevel analysis of students in 41 countries. Journal of Youth and Adolescence 41, 1409–1425. Davidov, E., Datler, G., Schmidt, P., Schwartz, S.H., 2011. Testing the invariance of values in the Benelux countries with the European Social Survey: accounting for ordinality. In: Davidov, E., Schmidt, P., Billie’s, J. (Eds.), Cross-Cultural Analysis: Methods and Applications. Routledge, London, pp. 149–172. Gamoran, A., Long, D.A., 2006. Equality of educational opportunity: a 40 year perspective. In: Teese, R., Lamb, S., Duru-Bellat, M. (Eds.), Education and Equity: International Perspectives on Theory and Policy. Springer, New York. Heyneman, S.P., Loxley, W.A., 1983. The effect of primary-school quality on academic performance across twenty-nine high- and low-income countries. American Journal of Sociology 88, 1162–1194. Hofstede, G., 2003. Culture’s Consequences: Comparing Values, Behaviors, Institutions and Organizations across Nations. Sage, Thousand Oaks, CA. Inglehart, R., Basanez, M., Diez-Medrano, J., Halman, L., Luijkx, R., 2004. Human Beliefs and Values: A Cross-Cultural Sourcebook. Siglo XXI, Mexico City. Organization for Economic Cooperation and Development (OECD), 2010. PISA 2009 Technical Report. OECD, Paris.
Relevant Websites International Studies of Students www.pisa.oecd.org – OECD Program for International Student Assessment (PISA). TIMSS & PIRLS studies of students. timss.bc.edu/. Trends in international math and science study & Progress in international reading literacy study.
Country Data geert-hofstede.com/dimensions.html – Hofstede Cultural values. http://www.oecd.org/pisa/pisaproducts/ – OECD statistics. data.worldbank.org/ – World Bank Data.
Abstract Multilevel analyses of large-scale international studies’ data have shown how characteristics at multiple levels (country, family, school, student) influence student achievement. Students in countries that are richer academically outperform other students, in part because they have more resources at home (family socioeconomic status, books, etc.) and school (teachers, schoolmates, educational materials, etc.). Students in countries with greater household inequality, inequality across schools, or economic segregation of richer students from poorer students show lower academic achievement than other students. Cultural values and family characteristics showed interaction effects. Together, these results show the importance of an ecological approach to understanding student achievement.
Introduction Student achievement on international tests differs sharply across countries. Students in Finland, Korea, Japan, and Canada consistently score over 2 standard deviations higher than students in Indonesia, Panama, Peru, and Kyrgyzstan in mathematics, reading, and science across several international comparisons. These differences in academic achievement might stem from the countries’ economic and cultural differences. First, the countries with higher average test scores are generally much wealthier (e.g., as measured by gross domestic product (GDP) per capita) and more equal economically (e.g., income Gini) than the countries with lower average test scores. These differences suggest that both greater educational resources and more equal distribution contribute to greater student learning and academic achievement. Second, the countries within each set are culturally diverse, suggesting that cultural values do not account for differences in test scores. We examine the antecedents of these differences more systematically by adopting Bronfenbrenner’s (2005) ecological framework and discussing the results of previous international studies. Bronfenbrenner (2005) modeled children’s interactions with surrounding environments including persons and entities within immediate and direct contexts (such as family and school), the relationships across these contexts, and the
broader country context (such as economic and cultural differences). Economic and cultural differences across and within countries can fuel differences in resources across families, schoolmates, schools, and students that influence students’ learning opportunities and subsequent achievement. Lastly, we discuss the methodologies involved in collecting and analyzing large-scale international data.
Ecology of Student Achievement Student interactions with their environment can influence their academic achievement (see Figure 1; Bronfenbrenner, 2005). Students have frequent and direct contacts with family (or any immediate context to create a microsystem), which offers learning opportunities on which they can capitalize to learn and achieve in school. For instance, students living in families with higher socioeconomic status (SES) tend to outperform other students on achievement tests (Baker et al., 2002). Connections between family and school contexts (or any two microsystems that form a mesosystem) can also influence academic achievement. For example, when parents are more involved in children’s school activities, their children show higher academic performance (Chiu, 2010). Thus, both the immediate contexts and the interactions among them influence
Country GDP per capita (+) Family inequality School inequality Economic segregation
Family Family SES Educational resources at home …
Student Academic achievement
School Schoolmates Teachers …
Figure 1 Ecological model of relationships among country, family, school, and student academic achievement. GDP, gross domestic product; SES, socioeconomic status.
342
International Encyclopedia of the Social & Behavioral Sciences, 2nd edition, Volume 5
http://dx.doi.org/10.1016/B978-0-08-097086-8.92144-5
Cross-National Comparisons in Education: Findings from PISA
students’ academic achievement. A more distal context, such as a parent’s stress at work, might result in less school involvement and their children’s poorer academic performance (links between distal and immediate contexts are called exosystems, Bronfenbrenner, 2005). These microsystems, mesosystems, and exosystems are all under the influences of the larger cultural contexts, such as the economy and cultural values of the society (macrosystem). For example, students in richer countries showed higher academic achievement, and had stronger links between family cultural communication and academic performance (Chiu, 2007). Hence, ecologies of microsystems, mesosystems, exosystems, and macrosystems can help us understand the factors that influence student achievement within and across countries. The results discussed in this article are largely drawn from the Organization for Economic Cooperation and Development’s Program for International Student Assessment (OECD-PISA; OECD, 2010). OECD-PISA assessed 193 076 15-year-old students from 41 countries in 2000–02; 276 165 15-year-old students from 41 countries in 2003; and 475 760 15-year-old students from 65 countries in 2009 (see Table 1). They also asked students and principals to fill out questionnaires. International experts from participating OECD and non-OECD countries defined mathematics, reading, and science achievement; built assessment frameworks; created test items; forward translated them; backward translated them; and pilot tested them to check their validity and reliability (for details and sample items, see OECD 2002, and www.pisa.oecd. org). Participating students completed a 2-h assessment booklet and then, a 30- to 40-min questionnaire. Studies using OECD-PISA data have shown that differences in student achievement at the country macrosystem, school microsystem, and student levels are all substantial (Chiu and Khoo, 2005). Country macrosystem differences account for 31% of the differences in students’ mathematics achievement (24% for reading achievement and 22% for science achievement). Meanwhile, school differences account for 25, 30, and 26% of the differences in mathematics, reading, and science achievements, respectively. Lastly, student-level differences account for 44, 46, and 52% of the differences in mathematics, reading, and science achievements, respectively. Together, these results show the importance of examining factors that influence student academic achievement within a country’s ecology.
Economic Differences Students in countries with greater economic productivity (real GDP per capita) show higher academic achievement (Chiu and McBride-Chang, 2010). Richer countries can raise student achievement through extra resources at the country, family, and school levels (Chiu, 2009). These resources provide students with more learning opportunities. In richer countries, governments tend to invest more in education by building more educational infrastructure (schools, libraries, and museums) and providing better educational resources and teacher training. The availability of these educational resources and opportunities can directly enhance students’ learning. Students in richer countries can also benefit indirectly through higher nutritional standards or better health care. These help ensure
Table 1 The years in which the following countries participated in OECD-PISA Country
2000–02
Argentina Albania Argentina Australia Austria Azerbaijan Belgium Brazil Bulgaria Canada Chile Colombia Croatia Czech Republic Denmark Dubai (UAE) Estonia Finland France Germany Greece Hong Kong–China Hungary Iceland Indonesia Ireland Israel Italy Japan Jordan Kazakhstan Korea Kyrgyz Republic Latvia Liechtenstein Lithuania Luxembourg Macao–China FYR Macedonia Mexico Republic of Montenegro The Netherlands New Zealand Norway Panama Peru Poland Portugal Qatar Romania Russian Federation Republic of Serbia Serbia and Montenegro Shanghai (China) Singapore Slovak Republic Slovenia Spain Sweden
X X
2003
X X
X X
X X X
X X X
X
X X
X X
X X X X X X X X X X X X
X X X X X X X X X
X
X
X X
X X
X
X X
X X
X X
X
X X X
X X X
X X X
X X
X X
X
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
X
X X X
X X
X X X X X X (Continued)
343
344
Cross-National Comparisons in Education: Findings from PISA
Table 1 The years in which the following countries participated in OECD-PISAdcont'd Country
2000–02
2003
2009
Switzerland Chinese Taipei Thailand Trinidad and Tobago Tunisia Turkey United Kingdom United States Uruguay
X
X
X
X
X X X X X X X X X
X X
X X X X X
students are healthy and have adequate energy when they study, which in turn contributes to better academic performance in richer countries.
Family SES Families in richer countries tend to have more material, human, and cultural capitals that help their children learn more (Chiu and Khoo, 2005). In richer countries, families often have more educational resources at home (such as books, computers, and tutors), which provide students with more learning opportunities. By capitalizing on these learning opportunities at home, students in richer countries can academically outperform students from poorer countries. Apart from material support, children also benefit from family members’ human capital (Chiu and McBride-Chang, 2010). As the overall education level is higher in richer countries, parents in these countries often completed more years of schooling and have higher aspirations for their children’s educational attainment. As a result, these parents encourage their children to study, excel academically, and complete higher levels of education. Furthermore, families with both parents are more likely to have more and better educational resources at home and are more involved in their children’s schooling. Parents in richer countries also often give their children more opportunities to learn cultural knowledge, skills, and values, which contribute to better academic performance (Chiu and Chow, 2010). Cultural possessions at home, such as paintings and poetry books, can highlight the importance of one’s culture and facilitate communication about cultural values and norms. This family cultural capital can influence students’ academic performance in two ways. First, families with more cultural capital often understand their children’s schools better and are more connected with these schools, thereby are more likely to be involved in activities and works in these schools. Second, this cultural capital also helps children better understand school norms and their teachers and classmates’ expectations in school. So, these children often behave appropriately in school and have better relationships with teachers and classmates. With greater support from their parents, teachers, and fellow students, these children feel a stronger sense of belonging at school, are more motivated to learn, and have greater academic success than other students (Chiu et al., 2012).
School Resources At the school level, students in richer countries often have more school resources, teacher resources, and schoolmates who share educational resources with one another, compared to students in poorer countries (Chiu, 2010). Schools in richer countries tend to offer more hours of classroom instruction and have better facilities and teaching materials. By utilizing these school facilities and materials, students in richer countries can learn more than those in poorer countries. Schools in richer countries often have teachers with higher qualifications and better training (e.g., teacher certification), so their students typically show higher academic achievement (Chiu and Khoo, 2005). These teachers are more capable of using effective teaching methods and raising students’ learning interest, all of which enhance student learning. Also, these teachers are more likely to create a supportive classroom atmosphere of mutual respect and caring, and foster students’ feelings of being valued for academic efforts, reinforcing their students’ school engagement. As discussed earlier, students who have better relationships with their teachers feel a greater sense of belonging at their school and higher academic achievement (Chiu et al., 2012). Furthermore, teachers in wealthier nations are more likely to choose a broader degree of curriculum coverage, a more advanced level of learning activities, and more advanced materials, compared to those in poorer countries. These teachers often have higher expectations of their students and help them realize those expectations through higher academic achievement. Students enrolled in privileged schools are perceived as smarter. Their teachers are likely to have higher expectations of them and are willing to invest more in their learning compared to teachers of other students. Also, teachers can directly or indirectly communicate these expectations to their students, such as adopting a more advanced level of learning activities, thereby affecting their expectations and behaviors. These advantages help students in privileged schools learn more than their counterparts in poorer schools. Students in schools with good disciplinary climates also show higher academic achievement (Chiu and Chow, 2011). A good disciplinary climate at school provides students with a physically and psychologically safe school environment. These feelings of safety throughout a school contribute to stronger school engagement, and students can concentrate on their learning in this environment. As schools in richer countries often have better disciplinary climates, students in these schools tend to have greater school engagement and higher academic achievement.
Schoolmates Schoolmates can affect one another’s learning through their educational resources and group culture (Chiu and Khoo, 2005). Compared to their counterparts in poorer countries, families in wealthier nations are more educated, are more capable in creating an enriched learning environment, and have more positive attitudes toward formal education. When students share their resources and experiences with their
Cross-National Comparisons in Education: Findings from PISA
schoolmates, they learn more and show higher academic achievement. Students with proportionately more high-SES and higher-achieving schoolmates showed better academic performance than those with more low-SES and lowerachieving schoolmates. Higher-achieving students are more likely than other students to value learning and academic success, and thereby support such a school culture at school, which supports other students’ learning.
Microsystem Effects in Richer vs Poorer Countries Country wealth also moderates family or school characteristics’ links with student learning (Heyneman and Loxley, 1983). Distinguishing between different types of family resources can explain the apparently contradictory results regarding whether the link between family resources and student achievement is stronger or weaker in richer countries (Chiu, 2007). In richer countries, physical resources at home, such as books and computers, have weaker links to student achievement, but intangible family resources, such as family involvement and communication, have stronger links to student achievement. As richer countries often have more public physical resources (e.g., library books, museum exhibits) that can substitute for a family’s physical resources, these public resources dilute the importance of family learning opportunities, thereby weakening their links to student achievement (modernization). Family’s physical resources (e.g., books) are strongly linked to student achievement in poorer countries, but much less so in richer countries whose GDP per capita exceeds $16 000 per year (in 1990 US dollars; Gamoran and Long, 2006). While basic physical educational resources (textbooks, maps) are widely available in public arenas (schools, libraries, etc.) in richer countries, these physical resources are not as publicly available in poorer countries (especially schools in Latin American countries), allowing family physical resources to have larger effects on student achievement. On the other hand, the widespread availability of public physical resources (such as public libraries and museums) in richer countries also raises the value of intangible processes such as family communication (complementary intangible processes, Chiu, 2007). Unlike physical resources (e.g., textbooks), processes are intangible and dynamic (e.g., family communication, teacher–student). For example, a student benefits from reading an extra book, but that benefit can be magnified substantially by discussing the book with a parent or a teacher (like a left shoe is more useful with a right shoe). Indeed, children in richer countries often spend more time with their parents due to fewer competing siblings, less parent time on housework, and multitasking parents. Thus, family intangible processes’ links to student achievement in richer countries are stronger, while those of family physical resources are weaker.
Inequalities and Their Mechanisms In addition to a country’s total educational resources, the distribution of these resources influences student achievement (Chiu and Khoo, 2005). Specifically, inequality of family
345
income (family inequality), inequality of school resources (school inequalities), and degree of separation of rich students from poor students into different sets of schools (economic segregation) show effects on student achievement.
Family Inequality National and school policies can influence students’ family inequality, school inequalities, and economic segregation (Chiu, 2009). Regressive taxes, corruption, and a weak legal system can increase inequality of family incomes (family inequality), while progressive taxes, social welfare, transparency, and health care and safety standards (e.g., clean water) can reduce it. Influenced by national policies, family inequality reduces overall student achievement through fewer overall educational resources, diminishing marginal returns, and privileged student bias. Greater family inequality results in less overall investment in educational resources, which reduces academic achievement among both poor and rich students (fewer educational resources, Chiu, 2009). The elite of very unequal countries can send their children to private schools, so they might advocate for fewer public school resources and greater family payment for each child’s education. Even if the elite send their children to public schools, they have many more educational resources (e.g., books or tutors) at home than others, so they may not advocate for more public spending. When students’ families have to pay for much of their education, poorer parents cannot afford the optimal amount of education for their children, resulting in suboptimal educational resources in the society. As a result, poorer students have fewer educational resources and learn less, thereby reducing overall achievement. Family inequality lowers overall student achievement due to diminishing marginal returns (Chiu and Khoo, 2005). Consider a thirsty woman and two glasses of water. She greatly values the first glass of water and drinks it all. Her thirst quenched, she does not finish the extra glass of water, revealing its lower value (diminishing marginal returns). Likewise, rich students have more educational resources than poor students do, and a richer student typically benefits less from an extra book than a poorer student does. With a less equal distribution of a fixed set of resources, richer students have extra resources and obtain less utility from them compared to poorer students who have fewer resources, resulting in less efficient utility of resources and lower education outcomes overall. In school systems with larger distribution inequality, privileged parents have more incentive to use their capital to get more educational resources for their children (Chiu and Khoo, 2005). Schools that allocate more resources to richer students exacerbate the effect of disparities in income. Because of diminishing marginal returns, greater privileged student bias results in lower overall student achievement. Some methods of resource allocation (open markets, cronyism, and social–cultural affinity) allow privileged parents to move more resources to their children compared to others (uniform or truly random allocations; Chiu, 2009). In open markets, privileged parents can use their financial capital by moving to a rich neighborhood to give their children richer schoolmates for example. They can use their social capital through cronyism, for example, by asking the vice principal to
346
Cross-National Comparisons in Education: Findings from PISA
assign their child to a course with a better teacher. Through their human capital, privileged parents can teach their children social norms and cultural capital to create greater affinity with teachers. Teachers have higher expectations of these students and often give them more attention and assistance (cultural gatekeeping).
School Inequality Countries with greater family inequality often have greater school inequalities (Chiu, 2010). A system of strictly free market, private schools without national standards (e.g., no teacher certification) can magnify the impact of family inequalities as richer families pay higher tuition for their children to attend richer schools with more educational resources (teacher quality, computers, richer schoolmates, and so on). In contrast, a universal public school system with required teacher certification standards can reduce inequalities among students. Schools with greater inequalities showed lower academic achievement among mostly poor students (Chiu, 2009). As with family inequality, school inequalities are accompanied by diminishing marginal returns, which result in lower overall achievement among poorer students. Furthermore, school resources (books, teachers, classmates) and family resources (books, siblings, parents) are mostly equivalent substitutes for one another; richer students often have a book at home that is largely equivalent to a book at school, so the latter’s absence does not affect the richer student’s academic achievement (substitution effect). Hence, fewer school resources did not reduce richer students’ academic achievement.
Economic Segregation Economic segregation is also related to academic achievement (Chiu and Khoo, 2005). The degree of economic segregation differs across school systems. In the US, schools are often funded through local taxes, so richer students often attend one set of schools in richer neighborhoods, while poorer students attend a separate set of schools in poorer neighborhoods. In contrast, rich and poor students often mix together in the same schools in other countries such as Japan and Finland. Economic segregation reduces interactions between richer and poorer students, resulting in less diverse interactions for all students. Less exposure to experiences of students in other economic strata results in less shared knowledge, less learning, and lower academic achievement among all students (diversity effect; Chiu, 2009).
A hierarchical nation emphasizes authority, obedience and hierarchical status (e.g., Russia), while an egalitarian country emphasizes equal status and treatment of one another (e.g., Sweden). Also, a collectivist society favors group interests (e.g., Hong Kong), while an individualistic nation favors individual interests (e.g., New Zealand). Cultural values did not directly affect students’ test scores, but they interact with the effects of family characteristics on academic achievement (Chiu, 2007). As egalitarian societies have less rigid societal status, individuals have greater economic and social mobility through their individual skills and talents. Thus, students in these countries often view academic achievement as linked to future success, so they are more likely to study harder and achieve greater academic success. Furthermore, students are more motivated to make better use of their resources in egalitarian societies, yielding stronger links between family resources and student achievement (egalitarian motivation). Specifically, family SES and numbers of books at home had stronger links to academic achievement in more egalitarian cultures. Meanwhile, students in collectivist societies are more inclined to cooperate, support, and learn from their schoolmates and family members (Inglehart et al., 2004). In more collective societies, people tend to rely more on their extended family members who often live nearby, and so students are more likely to benefit from them. As extended family resources dilute the effects of immediate family resources in collective societies, the link between immediate family resources and academic achievement is weaker than in individualistic societies (collectivist dilution; Chiu, 2007). Specifically, single parents, family SES, resident grandparents, and birth order were all less negatively linked to test scores in more collectivist cultures.
Methodological Issues International studies require appropriate data samples, precise test and questionnaire designs, and suitable analytic tools (see Table 2; Chiu, 2010). Ideally, data samples are representative of the populations, with few sampling errors and no missing data. Meanwhile, tests and questionnaires aim to maximize data validity by minimizing measurement errors and maximizing equivalence across countries. Tests seek to optimize coverage of target concepts and skills. Analyses can model multiple levels of data, cross-level moderation and mediation, and multiple outcomes.
Cultural Differences
Data Samples
Countries have different cultural values and the ways they address basic societal issues (Hofstede, 2003). People learn these values through both formal socialization, including direct teachings of parents and teachers, and informal socialization, such as daily exposure to social norms. Among different cultural values, two basic societal issues are particularly important to students’ academic achievement, and they are the degree of hierarchy and obedience to authority vs equality (hierarchical vs egalitarian), and favoring group interests vs individual interests (collectivism vs individualism).
To create a representative sample of a country, researchers typically categorize schools (e.g., by neighborhood SES), randomly select schools from each category, and then randomly select students from each selected school (stratified samples) with corresponding weights for each student. To reduce sampling error, researchers select several subsamples to create plausible values, which must be modeled in the subsequent analyses. In such a huge data collection enterprise, missing data are typically unavoidable. Missing data can reduce estimation
Cross-National Comparisons in Education: Findings from PISA
Table 2
Strategies to address each analytical difficulty
Analytical difficulty
Strategy
l
l
l
l
Representative sample Sampling error l Missing data l
Measurement equivalence of test and questionnaire items
l
Wide coverage of skills while minimizing student fatigue and learning effects l Competence estimates from test answers l Measurement errors on questionnaires l Nested data of students within schools within countries l Cross-level moderation l Indirect, mediation effects l
False positives
l
Simultaneous outcomes
Stratified sample Plausible values l Markov Chain Monte Carlo multiple imputation l Forward and backward translation of each item; l Multigroup Rasch model l Balanced incomplete block test
l
Factor Analysis Item Response models l Factor Analysis l Item Response models l Multilevel analysis (aka Hierarchical linear modeling) l Random effects model l Multilevel mediation tests l Structural equation models l Two-stage linear step-up procedure l Multivariate outcome models l
efficiency, complicate data analyses, and bias results. Markov chain Monte Carlo multiple imputation estimates the values of the missing data, which addresses these missing data issues more effectively than deletion, mean substitution, or simple imputation.
Tests and Questionnaires Tests and questionnaires have special designs. Test items are divided into overlapping subsets (subtests) in a balanced incomplete block test design to (1) maximize coverage of target concepts and skills, (2) reduce student fatigue, and (3) reduce test-learning effects. To reduce measurement error on questionnaires, multiple items were used for each theoretical construct (e.g., SES) to create an index that is more precise than any single variable. To test for data validity, translations and a multigroup Rasch model are used. Each item on a test or a questionnaire is forward translated into the new language of the test by one person and then backward translated by another to check the quality of the translation. Multigroup Rasch models for each item in each country test whether the measures are equivalent across countries. Unlike factor analysis, a multigroup Rasch model has the advantages of requiring only one invariant anchor item across countries and modeling heterogeneous use of the ordinal rating scale. To capitalize on the special designs of the tests and questionnaires to reduce measurement error, researchers use factor analyses and item response models. A factor analysis tests whether a set of test items reflects one or more underlying achievement dimensions (e.g., mathematics achievement vs reading achievement). For each construct, its test item’s characteristics (difficulty, discrimination, chance) are estimated
347
with an item response model. Some test items capture higher levels of a construct more precisely, while others capture lower levels of a construct more precisely (item difficulty). Likewise, each test item’s precision for distinguishing each level of a construct can differ (discrimination). The chance parameter models the likelihood of successfully guessing a correct answer for a test item. For questionnaire items, factor analysis and item response models are also used. A factor analysis tests whether a set of questions that appears to test similar constructs (e.g., mother’s and father’s years of schooling) reflects one or more underlying constructs (e.g., SES). For each construct, its questionnaire item’s characteristics (difficulty, discrimination) are estimated with an item response model. Some questionnaire items capture higher levels of a construct more precisely, while others capture lower levels of a construct more precisely (item difficulty). Likewise, each question’s precision for distinguishing each level of a construct can differ (discrimination).
Analysis As the data are nested (students within schools within countries), a multilevel analysis of weighted, plausible values is needed (also known as hierarchical linear modeling). Students in the same school (or country) likely resemble one another more than students in different schools (or countries, group heterogeneity). Applying an ordinary least squares regression to nested data yields biased standard errors. In contrast, a multilevel analysis yields appropriate standard errors. For multilevel analyses, at least partial metric and partial scalar invariance of the measurement instruments are needed to compare latent means and regression coefficients over countries (Davidov et al., 2011). Testing many hypotheses increases the likelihood of a false positive. To control for the false discovery rate, the two-stage linear step-up procedure is used, which outperformed 13 other methods in computer simulations. As the data are nested, cross-level moderation and mediation tests are needed. A random effects model precisely tests whether the relationship between an explanatory variable and an outcome variable differs across contexts or interacts with another explanatory variable via random effects models. The empirical M-test corrects for potential nonnormal distributions in nested data by using multiple data simulations. Multivariate outcome models can be used to analyze multiple outcome variables simultaneously (e.g., mathematics, reading, and science tests). Ignoring the relationships among outcome variables can result in correlated residuals that underestimate standard errors. Multilevel, multivariate outcome models address this issue to yield proper standard errors. In short, international studies require high-quality data, precise tests and questionnaires, and appropriate analytic tools. Stratified samples, representative weights, plausible values, and Markov chain Monte Carlo multiple imputation can improve their data quality. A balanced incomplete block test design, construct indices based on sets of questionnaire items, forward and backward translations, multigroup Rasch models, factor
348
Cross-National Comparisons in Education: Findings from PISA
analyses, and item response models can help increase data validity. Lastly, multilevel, multivariate outcome analysis of weighted, plausible values; two-stage linear step-up procedure; random effects model; and empirical M-tests can be used to analyze these data.
Future Research The availability of international data sets, their data beyond student achievement, and the potential expansion of databases portend tremendous growth in international research in the coming decade. Research on international comparisons of education is accelerating through large, international teams of scholars collecting enormous, high-quality data sets and making them freely available and easily accessible by Website downloads (e.g., www.pisa.oecd.org). Thus, scholars can use them readily both for research and teaching. The data can benefit young scholars, especially students, by providing data with which they can immediately test their hypotheses. While most research has focused on the student test data, the extensive student questionnaire data present invaluable opportunities for analyses of other aspects of schooling aside from achievement, such as social relationships, student discipline, emotional sense of belonging at school, and so on. Furthermore, the questionnaire responses of parents, teachers, and principals can be examined as outcome variables in addition to their relationships to student outcomes. Future research will link large-scale, cross-country data sets and create longitudinal data sets. Linked data sets combine a student achievement database with school databases or country databases to model the effects on student outcomes of school or country variables that were not available in the original student database. Longitudinal data sets follow individual students, which both control for individual student differences for more precise results and allow testing of hypotheses involving time. Altogether, these are promising developments for large-scale international research.
See also: Comparative Research in Education: Iea Studies; Cross-Cultural Study of Education; Education, Economics of; Family and Schooling; Teacher Behaviours and Student Outcomes.
Bibliography Baker, D.P., Goesling, B., Letendre, G.K., 2002. Socioeconomic status, school quality, and national economic development: a cross-national analysis of the “HeynemanLoxley effect” on mathematics and science achievement. Comparative Education Review 46, 291–312. Bronfenbrenner, U., 2005. Making Human Beings Human: Bioecological Perspectives on Human Development. Sage, Thousand Oaks, CA.
Chiu, M.M., 2007. Families, economies, cultures and science achievement in 41 countries: country, school, and student level analyses. Journal of Family Psychology 21, 510–519. Chiu, M.M., 2009. Inequalities’ harmful effects on both disadvantaged and privileged students: sources, mechanisms, and strategies. Journal of Education Research 3, 109–128. Chiu, M.M., 2010. Inequality, family, school, and mathematics achievement. Social Forces 88 (4), 1645–1676. Chiu, M.M., Chow, B.W.-Y., 2011. Classroom discipline across 41 countries: school, economic, and cultural differences. Journal of Cross-Cultural Psychology 42, 516–533. Chiu, M.M., Chow, B.W.-Y., 2010. Culture, motivation, and reading achievement: high school students in 41 countries. Learning and Individual Differences 20, 579–592. Chiu, M.M., Khoo, L., 2005. Effects of resources, inequality, and privilege bias on achievement: country, school, and student level analyses. American Educational Research Journal 42, 575–603. Chiu, M.M., McBride-Chang, C., 2010. Family and reading in 41 countries: differences across cultures and students. Scientific Studies of Reading 14, 514–543. Chiu, M.M., Pong, S.L., Mori, I., Chow, B.W.Y., 2012. Immigrant students’ cognitive and emotional engagement at school: a multilevel analysis of students in 41 countries. Journal of Youth and Adolescence 41, 1409–1425. Davidov, E., Datler, G., Schmidt, P., Schwartz, S.H., 2011. Testing the invariance of values in the Benelux countries with the European Social Survey: accounting for ordinality. In: Davidov, E., Schmidt, P., Billie’s, J. (Eds.), Cross-Cultural Analysis: Methods and Applications. Routledge, London, pp. 149–172. Gamoran, A., Long, D.A., 2006. Equality of educational opportunity: a 40 year perspective. In: Teese, R., Lamb, S., Duru-Bellat, M. (Eds.), Education and Equity: International Perspectives on Theory and Policy. Springer, New York. Heyneman, S.P., Loxley, W.A., 1983. The effect of primary-school quality on academic performance across twenty-nine high- and low-income countries. American Journal of Sociology 88, 1162–1194. Hofstede, G., 2003. Culture’s Consequences: Comparing Values, Behaviors, Institutions and Organizations across Nations. Sage, Thousand Oaks, CA. Inglehart, R., Basanez, M., Diez-Medrano, J., Halman, L., Luijkx, R., 2004. Human Beliefs and Values: A Cross-Cultural Sourcebook. Siglo XXI, Mexico City. Organization for Economic Cooperation and Development (OECD), 2010. PISA 2009 Technical Report. OECD, Paris.
Relevant Websites International Studies of Students www.pisa.oecd.org – OECD Program for International Student Assessment (PISA). TIMSS & PIRLS studies of students. timss.bc.edu/. Trends in international math and science study & Progress in international reading literacy study.
Country Data geert-hofstede.com/dimensions.html – Hofstede Cultural values. http://www.oecd.org/pisa/pisaproducts/ – OECD statistics. data.worldbank.org/ – World Bank Data.