Modesty Bias and the Attitude-Achievement Paradox Across Nations: A Reanalysis of TIMSS

LEAIND-01388; No of Pages 8 Learning and Individual Differences xxx (2016) xxx–xxx

Contents lists available at ScienceDirect

Learning and Individual Differences journal homepage: www.elsevier.com/locate/lindif

Modesty Bias and the Attitude-Achievement Paradox Across Nations: A Reanalysis of TIMSS Inah Min a,⁎, Kai S. Cortina b, Kevin F. Miller c a b c

Combined Program in Education & Psychology, University of Michigan, 530 Church St. Ann Arbor, MI, 48109-1043 Department of Psychology, Combined Program in Education & Psychology, University of Michigan, 530 Church St., Ann Arbor, MI, 48109-1043 School of Education, Department of Psychology, Combined Program in Education & Psychology, University of Michigan, 610 E. University, Ann Arbor, MI, 48109-1043

a r t i c l e

i n f o

Article history: Received 23 June 2015 Received in revised form 30 July 2016 Accepted 15 September 2016 Available online xxxx Keywords: academic self-concept achievement international comparison modesty bias

a b s t r a c t Academic self-concept (ASC) and achievement are generally seen as mutually reinforcing and hence positively correlated. However, international comparisons have found negative associations between academic self-concept and achievement when country averages are compared. One possible explanation for this paradox is the ‘modesty bias’, which refers to cultural preferences of people from different cultural backgrounds when responding to scales pertaining to their own abilities. In the present article, we analyzed three TIMSS data sets from 2003 to 2011 to test the importance of modesty bias by introducing average self-concept on the country level to a hierarchical linear model (HLM) with student ASC as the dependent variable. Modesty bias substantially reduced the negative country level effect, accounting for about a third of the effect size. On an exploratory basis, Enjoyment of Math on the country-level was added to the model, which further reduced the negative associations between academic self-concept and achievement on the country level. High achieving countries are characterized by high salience of modesty norms but lower level of enjoyment of math. Findings support the idea that the paradox is mainly due to a combination of cultural response bias and cultural differences in students' liking of school. © 2016 Elsevier Inc. All rights reserved.

International comparative studies such as Trends in International Mathematics and Science Study (TIMSS) and Programme for International Student Assessment (PISA) revealed large cross-national differences in adolescent achievement in reading, mathematics, and sciences. Most of the developed East Asian countries including Korea, Japan, Hong Kong, and Singapore consistently score among the top, capturing the attention of researchers to closely examine what factors contribute to such high achievement. Of the psychological factors, the relationship between students’ achievement and their self-perceptions of competence has garnered scientific interests (e.g., Maaz, Trautwein, Lüdtke, & Baumert, 2008; Marsh & Hau, 2004; Seaton, Marsh, & Craven, 2009). Psychologists have long believed that students’ positive attitudes and self-perceptions of their competence positively affect their learning motivation, which leads to improved achievement (Bandura, 1994; Marsh, Trautwein, Lüdkte, Köller, & Baumert, 2005). However, largescale international studies have consistently displayed a peculiar

⁎ Corresponding author. E-mail addresses: [email protected] (I. Min), [email protected] (K.S. Cortina), [email protected] (K.F. Miller).

paradox that challenges existing psychological models regarding this relationship. When considering students’ average self-reports of attitudes such as academic self-concept (ASC), most countries at the top in mathematics achievement in TIMSS scored among the lowest on ASC (Lee, 2009; Loveless, 2006). This phenomenon has also been observed in other international comparative studies, such as PISA and Progress in international Reading and Literacy Study (PIRLS) (Maaz et al., 2008; Marsh & Hau, 2004). Van de gaer, Grisay, Schulz, and Gebhardt (2012) coined the term ‘attitude-achievement paradox’ (AAP) for this phenomenon. A majority of research on AAP has focused on comparing the United States with high-achieving East Asian countries. In a series of such analyses, Japanese and Chinese students were found to have lower confidence levels than their American counterparts (Stevenson, Chen, & Lee, 1993a,b) and the results remained consistent for all grade samples, further supporting the idea that higher performing East Asian countries tend to breed lower satisfaction and confidence. Wilkins (2004) used multilevel modeling to demonstrate a similar relationship between self-concept and mathematics using TIMSS 1995 data. The negative country-level correlation between academic self-concept and achievement has been documented consistently in several cross-cultural comparison studies with varying instruments over more than two

http://dx.doi.org/10.1016/j.lindif.2016.09.008 1041-6080/© 2016 Elsevier Inc. All rights reserved.

Please cite this article as: Min, I., et al., Modesty Bias and the Attitude-Achievement Paradox Across Nations: A Reanalysis of TIMSS, Learning and Individual Differences (2016), http://dx.doi.org/10.1016/j.lindif.2016.09.008

2

I. Min et al. / Learning and Individual Differences xxx (2016) xxx–xxx

decades. Moreover, the paradox seems to exist across school subject and academic domains (e.g., math, reading, sciences, etc.), grades, and cohorts (e.g., Kennedy & Trong, 2006; Lie & Turmo, 2005; Shen & Tam, 2008). Current literature suggests different explanations for this phenomenon. Heine, Takata, and Lehman (2000) argue that cultural biases exist in the way people respond to scales pertaining to their own abilities. Students from East Asian cultures may be more likely to downplay their competence because of the salient modesty norm which discourages people from drawing attention to personal strength in order to avoid making others feel inferior. This creates a tendency in Asian students to avoid the positive end of the Likert response scale and to choose categories closer to the center of the scale – a trend not as common among students from Western cultures. This “modesty bias” (Chen, Lee, & Stevenson, 1995) or “self-presentation bias” (Heine et al., 2000) is related to the fact that achievement feedback is processed differently in East Asian populations compared to Western cultures (Heine, Kitayama, Lehman, Takata, Ide, et al, 2001), an effect arguably rooted historically in Confucian philosophy (Nisbett, 2004). Other explanations of the phenomenon focus on potential translation imprecisions, which can produce measures conveying slightly different meanings of items across cultures or anchor statements for the response categories for Likert scales that are slightly stronger or weaker than in English (Brislin, 1970). Studies also suggest that members of some cultures may have more familiarity with completing questionnaires than do others, further contributing to the distortion of results in cross-cultural comparisons (e.g., Greenfield, 1997). Another hypothesized reason for a more critical self assessment of students in Asian countries points to the structure of their educational systems and its inherent performance pressure (Van de gaer et al., 2012). Higher achieving countries tend to have more demanding curricula, higher academic standards and more pressure to achieve, which may lead to less enjoyable learning experience in school, lowering the levels of absolute self-concept or perceived competence (Eccles et al., 1993). 1. Big Fish Little Pond Effect Self-concept research has consistently demonstrated that ASC cannot be adequately understood if the frame of reference is ignored (Marsh & Hau, 2003); the same accomplishments can lead to divergent self-concepts depending on the frame of reference that individuals use to evaluate themselves. Holding a student’s achievement constant, the academic self-concept of a student is lower when surrounded by high achieving students and higher when surrounded by low achieving students (“Big Fish Little Pond” Effect, BFLP, Marsh,1984; Marsh & Parker, 1984; Marsh et al., 2008). Conceptually, the BFLP effect is based on Festinger’s (1954) social comparison theory. Accordingly, reference group effects occur when responses to self-report items are based not on respondents’ absolute level of a construct but rather on the level relative to a salient comparison or reference group (Credé, Bashshur, & Niehorster, 2010). This occurs because people tend to think about themselves in comparison to similar others and the existence of such a reference group affects their perceptions (Heine, Lehman, Peng, & Greenholtz, 2002). The consistent presence of BFLP effects shown in virtually all largescale international data (e.g., Marsh & Hau, 2003; Seaton et al., 2009) suggests that the BFLP effect is a rather universal phenomenon in all school settings. Although Marsh and his colleagues have studied academic selfconcept and school-average achievement at an international level, BFLP studies have not included how country-average achievement may be related to academic self-concept. Such a relationship is not the focus of BFLP research; yet international research suggests certain structural similarities to the school-level effect, such that both

showcase a puzzling negative relationship between ASC and achievement in the presence of a strong positive association on the individual level. Van de gaer et al. (2012) and other cross-cultural researchers (e.g., Peng, Nisbett, & Wong, 1997) argue that culture can be a frame of reference that affects self-concept in the same way the school context does. However, while the underlying psychological mechanisms of the BFLP effect in the context of classrooms and schools are well understood, they do not generalize to differences across nations. It simply seems implausible that students from one country adjust their ASC based on performance comparison with students from other countries when this cannot be demonstrated even between schools within the same country. Also, there are, to our knowledge, no studies that systematically investigate the AAP based on an integrative frame of reference model. With the current study, we sought to better understand the causes of the AAP by specifying an integrated three level model, estimating simultaneously BFLP effect, modesty effects, and performance pressure. 2. Enjoyment/Liking of Mathematics Current research on academic self-concept focuses on its relationship with student academic achievement. However, there are other important factors that may also contribute to the formation of academic self-concepts. For instance, children differ significantly with regard to what academic subjects they like and dislike and such feelings affect the development of domain-specific ability self-concepts (Denissen, Zarret, & Eccles, 2007). Doing well in a subject and liking it are related but distinguishable predictors of academic self-concept. In fact, the core element of Eccles’ expectancy-value model (Eccles et al., 1993) lies in the relative independence of expectancy (here: ability perception) and value (here: liking/intrinsic motivation) in the prediction of academic behavior and career choices (Eccles, Wigfield, & Shiefele, 1998; Wigfield & Eccles, 1992, 2002). Eccles (1994) further maintains that values can influence children’s ability perceptions and predict their career choices. Although a myriad of social factors may also influence occupational aspirations, Eccles argues that psychological factors of an individual, particularly their value system, are of critical importance when it comes to the way an individual views his or her own ability in a domain. Such claims surrounding subjective value (liking/motivation) of an academic domain further corroborate the need to investigate this variable in greater detail, particularly in relation to academic self-concept. 3. Purpose and Hypotheses The present study had three goals. First, it extended on prior international research on the BFLPE and international achievement paradox to examine the relationship between mathematics self-concept and mathematics achievement across different countries. This three-level model accounts for the relationships at the individual, school, as well as country levels simultaneously, providing a comprehensive analysis of the phenomena involved. We wanted to first confirm that the attitudeachievement paradox indeed exists consistently in large-scale international data. Second, this study tested the extent to which the modesty effect explains the attitude-achievement paradox. We hypothesized that modesty bias is a substantial factor in explaining the complex relationship between achievement and academic self-concept. Finally, we sought to explore the role of cultural differences in enjoyment/liking in the explanation of the attitude-achievement paradox. The enjoyment variable is particularly relevant due to its link to academic pressure. The higher level of academic pressure in high achieving countries is linked to less enjoyable learning experiences in school (Van de gaer et al., 2012). We hypothesized that less enjoyable learning experiences in school lower students’ perceived competence. We hypothesized that since East Asian students have been consistently reported as

Please cite this article as: Min, I., et al., Modesty Bias and the Attitude-Achievement Paradox Across Nations: A Reanalysis of TIMSS, Learning and Individual Differences (2016), http://dx.doi.org/10.1016/j.lindif.2016.09.008

I. Min et al. / Learning and Individual Differences xxx (2016) xxx–xxx

having low levels of “liking mathematics” compared to their Western counterparts, the country-level average of liking the subject will also be able to (partly) explain the Attitude-Achievement-Paradox. 4. Methods 4.1. Datasets and Sample Three TIMSS datasets from 2003 to 2011 were used to estimate the effects of interest and to investigate how stable they are as a trend over eight years, in particular since the TIMSS achievement tests in mathematics and sciences have undergone some changes over time (Olson, Martin, & Mullis, 2008). Only the eighth-grade samples were used in this study since younger children, such as fourth graders included in TIMSS, tend to overestimate their competence and often lack the ability to critically evaluate their abilities (Harter, 1999). In order to be able to replicate the findings across three waves of TIMSS, we restricted our analyses to those thirteen countries that participated in TIMSS 2003, 2007 and 2011 and for which sampling procedures within countries remained stable over the observational period, including: Australia, Chinese Taipei, England, Hong Kong, Italy, Japan, Korea, Norway, Russia, Singapore, Sweden, Tunisia, and the United States. While not strictly a random sample of countries, the sample includes five cultures where modesty bias presumably exists (Chinese Taipei, Hong Kong, Japan, Korea, Singapore) and three Anglo-American cultures for which modesty is not assumed to bias responses (Australia, England, United States). In total, the TIMSS 2003 data set used contained 65,332 students and 2,092 schools, the TIMSS 2007 data set contained 58,940 students and 2,162 schools, and TIMSS 2011 data set contained 69,874 students and 2,517 schools. The specific breakdown of students and classes/schools from each country in the two datasets can be found in Table 1. 4.1.1. Mathematics achievement At grade 8, TIMSS focuses on four domains of mathematical ability: numbers, algebra, geometry, data and chance. Additionally, TIMSS studies estimate mathematics ability total scores of students by generating five plausible values (IRT scores) for each student (Yamamoto & Kulick, 2000). We used all five plausible values to compute an unbiased estimation of students’ achievement scores. All five plausible values were also entered into the HLM software to produce appropriate parameter estimates. 4.1.2. Academic Self-Concept (ASC) in mathematics The dependent variable was based on four items, ‘I usually do well in mathematics’; ‘Mathematics is more difficult for me than for many of my classmates’; ‘Mathematics is not one of my strengths’; ‘I learn things Table 1 Sample sizes for countries participating in TIMSS 2003~2011. TIMSS 2003 Country Australia England Hong Kong Italy Japan Korea Norway Russia Singapore Sweden Taipei Tunisia United States Total

TIMSS 2007

3

quickly in mathematics’. The same four items were used in the three TIMSS studies. Responses were given on a Likert scale with 1 = Strongly Agree, 2 = Agree a little, 3 = Disagree a little, and 4 = Strongly Disagree. Positively worded items were reverse-coded and the average was used to create the variables for this study (higher score = higher ASC). 4.1.2.1. Enjoyment. This variable is based on two items from the 2003 dataset, ‘I would like more math’; ‘I enjoy learning mathematics’ and three items from the 2007 and 2011 dataset, ‘I would like to take more math’; ‘I enjoy learning mathematics’; ‘I like mathematics’. The item, ‘I like math’ was a new addition in the 2007 and 2011 questionnaire but was included in this study because it is a direct measure of the construct. In the 2011 questionnaire set, the variable ‘I would like more math’ was altered to indicate a reverse statement, ‘I wish I did not have to study math’. This item was recoded in the direction of all other items. Responses were given on a Likert scale with 1 = Strongly Agree, 2 = Agree a little, 3 = Disagree a little, and 4 = Strongly Disagree. The average was used to create the variables for this study (higher score = liking math more). 4.1.3. Control variables Sex (student self report) and parental education (both mother and father) were included since prior studies have identified gender and socio-economic effects on ASC. 4.1.4. Modesty Bias We operationalized this construct indirectly as the average ASC on the country level. If cultures differ in their tendency to feel uncomfortable with open social comparison (and thus respond in a more modest manner), this will be reflected in mean differences in country level selfconcept while holding achievement constant. This operationalization implies that the averages of the true ASC do not differ across countries. This theorem is derived from Marsh’s (1984) “encapsulate frame of reference” theory (Marsh et al., 2015, p. 259) that is arguably the most empirically confirmed effect in educational psychology. This theory maintains that ASC is a result of students’ achievement comparisons with their classmates in the same class and/or the same school, but not based on comparisons of students across schools, let alone across nations. Differences on the level beyond the reach of the bound comparison group context therefore must reflect other cultural response biases or imprecision in the translation of the questionnaires. By controlling for this “bias”, we examined the extent to which the attitude-achievement paradox is due to modesty bias. Because country level differences in reported levels of ASC could overlap with language differences (e.g., Brislin, 1970; Greenfield, 1997), we propose that controlling for modesty bias by introducing country-level self-concept into the proposed HLM model will also account for any potential translation imprecisions in large-scale international datasets. 4.2. Data Analysis

TIMSS 2011

Students

Schools

Students

Schools

Students

Schools

4791 2830 4972 4278 4856 5309 4133 4667 6018 4256 5379 4931 8912 65,332

207 87 125 171 146 149 138 214 164 159 150 150 232 2,092

4069 4025 3470 4408 4312 4240 4627 4472 4599 5215 4046 4080 7377 58,940

228 137 120 170 146 150 139 210 164 159 150 150 239 2,162

7556 3842 4015 3979 4414 5166 3862 4893 5927 5573 5042 5128 10477 69,874

277 118 117 197 138 150 134 210 165 153 150 207 501 2,517

4.2.1. Weights Using weights is recommended for the analysis of large-scale datasets such as TIMSS in order to account for selection biases. The TIMSS database contains weight factors and weight adjustments at the student, classroom, and school levels that can be used to generate the inverse of the probability that a student, classroom or school was selected. For the purpose of this multilevel analysis, weights for students were created by multiplying the student weight factor and weight adjustment following the recommendation by Rutkowski, Gonzalez, Joncas, and von Davier (2010). 4.2.2. Missing Values For all three waves of data, the missing values in the dependent variable (mathematics self-concept) make up approximately 1% of the

Please cite this article as: Min, I., et al., Modesty Bias and the Attitude-Achievement Paradox Across Nations: A Reanalysis of TIMSS, Learning and Individual Differences (2016), http://dx.doi.org/10.1016/j.lindif.2016.09.008

4

I. Min et al. / Learning and Individual Differences xxx (2016) xxx–xxx

Table 2 Missing cases across all variables included in the analysis (TIMSS 2003-2011). TIMSS 2003

Mathematics Self-Concept Mathematics Achievement Enjoyment/Liking of Mathematics Gender Mother Education Father Education

TIMSS 2007

TIMSS2011

Valid

Missing

%Missing

Valid

Missing

%Missing

Valid

Missing

%Missing

64609 65332 64374 65331 64018 63916

723 0 958 1 1314 1416

1.10 .00 1.50 .00 2.00 2.20

58346 58940 58261 58940 53706 53660

594 0 679 0 5234 5280

1.00 .00 1.20 .00 8.90 9.00

69221 69874 69213 69869 67955 68037

653 0 661 5 1919 1837

.90 .00 .90 .00 2.70 2.60

dataset. For the independent variables, the highest percentage of missing cases is 9% for father’s education in TIMSS 2007 (see Table 2). Logistic regression with missing as dependent variable (missing in mathematics self concept yes/no) revealed three significant demographic variables associated with missing status: math achievement, sex, and mother’s education. Since those three variables were included in all model estimations, we are confident that the missingness process can be described as missing at random within our statistical model (Little & Rubin, 2002) 4.2.3. HLM Hierarchical Linear Modeling (HLM 7.0, Raudenbush & Bryk, 2002) was used to distinguish relationships at the individual, the school, and the country level. We used a three-level model to accurately present the relationships at student, school, and country level of analysis. Missing on the individual level were assumed at random (following our missing case analysis) and handled by HLM accordingly. The first model (Model 1) replicated prior international BFLPE research and extends it by including the country level. The second model (Model 2) included control variables, sex of student, and parental education (both mother and father) on level 1 to account for any potential variables that could be related to achievement and self-concept. The third model (Model 3) included country average self-concept in addition to math achievement on all 3 levels. The final model (Model 4) added the enjoyment variable (on individual and country level) into Model 3. HLM specification (mixed model notation): Model 1 (Unconditional Model)

Math self‐concept ¼ γ 000 þ γ 100 Individual math achievement þ γ 010 School‐average math achievement þ γ 001 Country‐average math achievement þ r 0 þ u00 þ e

Model 4 (Conditional Model)

Math self‐concept ¼ γ 000 þ γ100 Individual math achievement þ γ 200 Individual enjoyment þ γ 300 gender þ γ 400 mother education þ γ 500 father education þ γ010 School‐average math achievement þ γ001 Country‐average math achievement þ γ 002 Country‐average self‐concept þ γ 003 Country‐average enjoyment þ r 0 þ u00 þ e 5. Results 5.1. Attitude-Achievement Paradox Table 3 contains the student-level correlations between mathematics achievement and mathematics self-concept for those 13 countries that participated in TIMSS 2003, 2007, and 2011. Self-concept was significantly (p b .001) correlated with mathematics achievement at the student level in all countries across the three waves of TIMSS data, ranging from r = .35 in Tunisia and r = .66 in Norway. Note that across the entire sample, the correlation between individual achievement and selfconcept is lower than every within-country correlation. On the country level, the average self-concept was negatively correlated with average mathematics achievement in TIMSS 2003 data, r = -.720, TIMSS 2007 data, r = -.734, and TIMSS 2011 data, r = -.727. 5.2. Hierarchical Linear Modeling Results Table 4 presents the results of Model 1 for the three samples with ASC in mathematics as dependent variable and, as predictors, individual

Model 2 (Conditional Model)

Math self‐concept ¼ γ 000 þ γ 100 Individual math achievement þ γ 200 gender þ γ300 mother education þ γ 400 father education þ γ 010 School‐average math achievement þ γ 001 Country‐average math achievement þ r 0 þ u00 þ e Model 3 (Conditional Model)

Math self‐concept ¼ γ 000 þ γ 100 Individual math achievement þ γ 200 gender þ γ 300 mother education þ γ 400 father education þ γ 010 School‐average math achievement þ γ001 Country‐average math achievement þ γ002 Country‐average self‐concept þ r 0 þ u00 þ e

Table 3 Student-level correlations between mathematics achievement and self-concept. Country Australia England Hong Kong Italy Japan Korea Norway Russia Singapore Sweden Taipei Tunisia United States Total

Student-Level (2003)

Student-Level (2007)

Student-Level (2011)

.500 .369 .401 .490 .455 .595 .609 .495 .372 .554 .546 .397 .427 .265

.547 .457 .369 .475 .453 .627 .608 .528 .428 .556 .537 .456 .445 .289

.547 .473 .407 .528 .502 .629 .657 .500 .413 .599 .576 .350 .448 .297

Note. All Correlations are significant at p b .01.

Please cite this article as: Min, I., et al., Modesty Bias and the Attitude-Achievement Paradox Across Nations: A Reanalysis of TIMSS, Learning and Individual Differences (2016), http://dx.doi.org/10.1016/j.lindif.2016.09.008

I. Min et al. / Learning and Individual Differences xxx (2016) xxx–xxx

5

Table 4 Model 1, effects of individual, school, and country-average math achievement on math self-concept in TIMSS 2003-2011. TIMSS 2003

Intercept, γ000 Level1 Individual math, γ100 Level2 School -average Achievement, γ010 Level3 Country-average Achievement, γ001

γ

SE

.021

.050

.648

TIMSS 2007 γ

SE

.678

.017

.053

.027

b.001

.629

-.126

.010

b.001

-.508

.058

b.001

p

TIMSS 2011 γ

SE

.756

-.019

.049

.705

.034

b.001

.687

.038

b.001

-.109

.018

b.001

-.102

.022

b.001

-.486

.053

b.001

-.503

.059

b.001

p

p

Note. Individual math achievement and individual self-concept were standardized (M = 0. SD = 1). **p b .001.

mathematics achievement at Level 1, school average mathematics achievement at Level 2, and country average mathematics achievement at Level 3. Individual mathematics achievement was positively related to individual mathematics self-concept, γ = .648, p b .001, reflecting the within-school correlation between achievement and ASC. At the school level, school average achievement was negatively related to individual self-concept, γ = -.126, p b .001, with a standard deviation increase in school’s average achievement associated with over onetenths (about 13%) of a standard deviation decrease in the self-concept of a student within that school. This reflects the “classic” BFLP effect. At the country level, country average achievement had, as hypothesized, a negative regression weight on self-concept, γ = -.508, p b .001, with a standard deviation increase in a country’s average achievement associated with approximately one half of a standard deviation decrease (51%) in self-concept for an individual student in that country. This represented the net effect known as the attitude-achievement paradox. The results from TIMSS 2007 and 2011 data show similar patterns on all three levels, where increases in mathematics achievement is associated with increases in mathematics self-concept on the individual level, but on the school and country level, the same increases in mathematics achievement is associated with decreases in self-concept. Model 2 included three student background variables (sex, mother’s education, father’s education). Table 5 demonstrates that the relationships between mathematics achievement (at the individual, school, and country levels) and individual mathematics self-concept remained almost identical even after controlling for these student background variables. But it should be noted that sex of the student did have a statistically significant effect on mathematics self-concept, γ = .224, p b .001, suggesting that boys tend to have slightly higher levels of self-concept in mathematics, approximately .2 standard deviations higher than girls. Model 3 includes modesty bias operationalized as the country average of ASC, holding individual achievement constant. If the modesty bias is an explanatory factor for the attitude-achievement paradox, inserting country average self-concept in the model should offset (or at least

substantially reduce) this negative relationship between country average math achievement and individual self-concept. Since country level differences overlap with language differences, we also controlled for potential translation imprecisions when we introduced the country-level self-concept into the model (Table 6) Using TIMSS 2003 data, the relationships between mathematics achievement (at the individual and school levels) and individual mathematics self-concept remained almost identical to those found in the unconditional model, with γ =.644, p b .001 at Level 1 and γ = -.117, p b .001 at Level 2. As predicted, country average self-concept did significantly predict individual student self-concept after controlling for individual school and country math achievement. That is, at Level 3, country average self-concept was significantly related to individual math selfconcept, γ = .244, p b .001. The attitude-achievement paradox coefficient (i.e. the effect of country average achievement) was substantially reduced from -.508 to -.349, p b .001. Thus, on the country level, the strength of the negative association between achievement and self-concept decreased while still substantial, after controlling for modesty bias. The corresponding analyses for the TIMSS 2007 and 2011 datasets show similar findings: Introducing average self-concept on the country level did not affect the regression weight of individual and school math on self-concept but substantially reduced the effect of country achievement level on self-concept. For all three TIMSS datasets, one standard deviation increase in a country’s average achievement was associated with about .3 SD decrease in the self-concept of a student within that country. In the final Model 4, we included enjoyment/liking of mathematics as a statistical predictor on individual and country level. As Table 7 shows, liking of math and self concept were moderately correlated in all countries included in the study. On the country level, self concept and enjoyment were also positively correlated, 2003: r = .513, p b .000; 2007: r = .548, p b .000; 2011: r = .629, p b .000. Table 8 shows the results for Model 4: On the individual level, a positive relationship exists between enjoyment of mathematics and ASC. In

Table 5 Model 2. Conditional models of effects of individual, school average, and country-average math achievement on math self-concept in TIMSS 2003-2011, controlling for student gender, mother and father’s education level. TIMSS 2003

Intercept, γ000 Level 1 Individual math, γ100 Gender, γ101 Mother Education, γ102 Father Education, γ103 Level2 School-average Achievement, γ010 Level3 Country-average Math, γ001

γ

se

.020

.050

.645 .224 .004 .002

TIMSS 2007 γ

se

.693

-.002

.053

.027 .029 .002 .001

b.001 b.001 .101 .172

.634 .204 .004 -.001

-.122

.011

b.001

-.508

.057

b.001

p

TIMSS 2011 γ

se

.971

-.017

.050

.734

.035 .028 .003 .002

b.001 b.001 .128 .808

.685 .194 -.004 -.001

.038 .031 .004 .002

b.001 b.001 .320 .772

-.112

.019

b.001

-.097

.022

b.001

-.483

.057

b.001

-.503

.059

b.001

p

p

Note. Individual math achievement and individual self-concept were standardized (M = 0. SD = 1). **p b .001.

Please cite this article as: Min, I., et al., Modesty Bias and the Attitude-Achievement Paradox Across Nations: A Reanalysis of TIMSS, Learning and Individual Differences (2016), http://dx.doi.org/10.1016/j.lindif.2016.09.008

6

I. Min et al. / Learning and Individual Differences xxx (2016) xxx–xxx

Table 6 Model 3. Conditional models of effects of individual, school average, and country-average math achievement on math self-concept in TIMSS 2003-2011, controlling for student gender, mother and father’s education level, and country average self-concept. TIMSS 2003

Intercept, γ000 Level 1 Individual math, γ100 Gender, γ101 Mother Education, γ102 Father Education, γ103 Level 2 School-average Achievement, γ010 Level 3 Country-average math, γ001 Country-average ASC, γ002

γ

se

.013

.013

.644 .224 .004 .002

TIMSS 2007 γ

se

.334

.016

.014

.027 .030 .002 .002

b.001 b.001 .105 .285

.633 .204 .004 -.000

-.117

.011

b.001

-.349 .244

.029 .019

b.001 b.001

p

TIMSS 2011 γ

se

.267

.018

.012

.156

.035 .028 .003 .002

b.001 b.001 .121 .848

.684 .194 .003 -.000

.005 .007 .002 .002

b.001 b.001 .070 .739

-.106

.017

b.001

-.089

.007

b.001

-.305 .272

.029 .023

b.001 b.001

-.369 .226

.017 .017

b.001 b.001

p

p

Note. Individual math achievement and individual self-concept were standardized (M = 0. SD = 1). **p b .001.

TIMSS 2003, individual enjoyment of mathematics is positively related to individual mathematics self-concept, γ = .415, p b .001. A similar pattern can be seen in both TIMSS 2007 and TIMSS 2011 data. Note that the coefficient for achievement is only slightly reduced compared to Model 3, suggesting that the liking is associated with self-concept over and above the variance attributable to achievement. More specifically, in all three waves of TIMSS data, one SD increase in individual enjoyment/liking of mathematics was associated with approximately .4 SD increase in individual self-concept, while controlling for the relationship between math achievement and self-concept. On the country level, it was of particular interest whether the introduction of the country aggregate of enjoyment/interest would further reduce the coefficient of the attitude achievement paradox. The results show that the existing negative relationship between ASC and country-average math achievement decreased further compared to Model 3, with γ = -.246, p b .001 for TIMSS 2003, γ = -.217, p b .001 for TIMSS 2007, and γ = -.259, p b .001 for TIMSS 2011, indicating an additional reduction of approximately .1 SD. Although the country level effect of achievement on individual ASC is still negative and significant, its nominal value in Model 4 is almost half the size of the attitude achievement paradox effect in Model 1. 6. Discussion The main purpose of this study was to test the hypothesis that modesty bias explains the attitude achievement paradox (AAP) in international large-scale studies like TIMSS and PISA. Modesty bias and AAP are phenomena described in the literature but were, to our knowledge, never analyzed simultaneously.

Table 7 Correlation coefficients of Enjoyment/Liking of mathematics and mathematic self-concept – separate for 13 nations. Country Australia England Hong Kong Italy Japan Korea Norway Russia Singapore Sweden Taipei Tunisia United States Total

Student-Level (2003)

Student-Level (2007)

Student-Level (2011)

.473 .444 .553 .638 .454 .570 .433 .470 .639 .382 .636 .531 .532 .513

.505 .492 .595 .684 .573 .664 .474 .528 .691 .490 .702 .486 .580 .548

.604 .612 .687 .724 .654 .650 .581 .594 .685 .600 .720 .588 .617 .629

Our study demonstrates that about one third of the AAP is attributable to modesty bias when comparing the effect of country-average mathematic achievement in Model 1 and Model 2. These results are in line with the argument put forth by Heine, Kitayama, & Lehman (2001) in their study with Canadian and Japanese students suggesting that East Asians use achievement feedback differently and do not feel the need to maintain a positive self-view when they receive achievement-critical feedback. Although such cultural biases explain a substantial part of the attitude-achievement paradox, the remaining effect of the AAP is strong enough to make it very unlikely that it is only due to modesty bias. One reasonable candidate to explain an additional piece of the AAP is achievement pressure that is consistently reported to differ between Western and Eastern countries. Although not measured directly, TIMSS does include measures of enjoyment/liking, which we used as a proxy measure of experienced achievement pressure in Model 4. The results confirm that an additional share of the variance of academic self-concept is explained on the individual level, independent of the effect of student achievement. The coefficient for enjoyment/liking on academic self-concept is remarkably high and almost rivals the effect of achievement. Holding this psychological positive effect constant, there is a small but significant negative effect of country average enjoyment/liking on self-concept that reduces the AAP effect even further. Taken together, country differences in modesty bias and enjoyment/liking account for about half of the attitude-achievement paradox (comparing the AAP effect size in Model 1 and Model 4). We take the effect of enjoyment/liking on ASC as indication that cultural differences in achievement pressure – particularly in the comparison of East Asian and Western countries – takes its toll on the overall attitude students have towards school as a positive environment for general development (Van de gaer et al., 2012). It is plausible that having higher academic standards or a more competitive educational atmosphere is associated with lower levels of self-concept or perceived competence because students are constantly pressured not only by the achievement of their peers, but also by their parents and teachers, based on their academic results. However, this is speculative at this point since this hypothesis would need stronger corroboration with direct measures of academic pressure and perceived support by teachers and school regarding academic and social development during adolescence. This is an area that could be explored in future research by incorporating teacher and school climate survey items to further unpack the relationships among culture, pressure, and attitudes toward school. With respect to the tradition in research to the Big-Fish-Little-Pond (BFLP) effect (Marsh et al., 2008), our results underscore that the AAP is a qualitatively different effect despite its similarity at first glance (negative effect on ASC from a higher level, such as classroom and school). Comparing the BFLP effect (negative effect of school mean on ASC) across the three models demonstrate that the BFLP is robust irrespective

Please cite this article as: Min, I., et al., Modesty Bias and the Attitude-Achievement Paradox Across Nations: A Reanalysis of TIMSS, Learning and Individual Differences (2016), http://dx.doi.org/10.1016/j.lindif.2016.09.008

I. Min et al. / Learning and Individual Differences xxx (2016) xxx–xxx

7

Table 8 Model 4. Conditional models of effects of individual mathematics enjoy/like on mathematics self-concept, controlling for individual and school-average math achievement, gender, and mother/father education. TIMSS 2003

Intercept, γ000 Level 1 Individual math, γ100 Gender, γ101 Mother Education, γ102 Father Education, γ103 Individual enjoy/like, γ200 Level2 School -average Math achievement, γ010 Level3 Country-average Self-concept, γ002 Country-average math, γ001 Country-average enjoy, γ003

γ

se

.026

.009

.512 .183 .006 .001 .415

TIMSS 2007 γ

se

.021

.012

.009

.029 .028 .002 .002 .021

b.001 b.001 .007 .504 b.001

.477 .175 .005 .000 .454

-.090

.016

b.001

.201

.015

-.246 -.106

.018 .009

TIMSS 2011 γ

se

.193

.010

.010

.364

.005 .007 .002 .002 .004

b.001 b.001 .016 .891 b.001

.502 .123 .002 .001 .457

.041 .002 .002 .002 .014

b.001 b.001 .459 .676 b.001

-.077

.007

b.001

-.074

.016

b.001

b.001

.261

.015

b.001

.220

.006

b.001

b.001 b.001

-.217 -.146

.016 .010

b.001 b.001

-.259 -.109

.026 .016

b.001 b.001

p

p

p

Note. Individual math achievement and individual self-concept were standardized (M = 0. SD = 1). **p b .001.

of the inclusion of modesty bias indicators or enjoyment/liking variables. BLFP is a phenomenon related to social comparison processes based on direct social interaction within smaller social aggregates like classroom and schools while the AAP is a genuine cultural effect in part due to modesty bias. In line with our hypothesis that modesty bias is a rather stable cultural phenomenon, we did not find variation of the finding across the span of 8 years from 2003 to 2011. This is not trivial since absolute mean scores and rank order of the countries in achievement have changed substantially during that period (with the United States as case in point). 7. Limitations Our study is based on datasets that are very strong with respect to comparability across nations and measurement quality regarding its key variables, particularly self-concept and achievement. However, there are several limitations that should be addressed. First, a trend study over eight years is not a longitudinal study and the epistemological question of causality is of particular importance in the context of cultural differences for which it is inherently impossible to manipulate the key explanatory variable (cultural exposure). Attitudes regarding learning and self assessment are established in the socialization process early on and eighth grade students have a long experience with their specific school system and cultural traditions. How cultural differences are transmitted over generations is not the focus of this study and the structure of the data can only contribute little to this question. But the hierarchical structure of the data does lend our models some internal validity with regard to the direction of influence: The regression effect of country-average achievement on individual self-concept is logically irreversible since the individual self concept will not influence the average score in achievement or self-concept of a country apart of a negligible amount (contribution to the mean score). The direction of causality (in the broader sense of the term) is therefore less of an issue than the unobserved mediation processes that we were unable to study. For example, if the learning climate is systematically different in Asian schools compared to their Western counterparts, a complete model would need to measure this school-level variable and demonstrate that cultural differences in ASC disappear once this factor is controlled. For the regression model on the individual level, it was established in longitudinal studies (for an overview, see Marsh et al., 2008) that academic self concept and achievement have a bidirectional relationship in the sense that higher self-concept might improve students' learning habit which indirectly leads to higher achievement. The focus of the current paper was not to clarify the causal direction between the two

variables but to establish the attitude-achievement paradox and to test whether modesty bias explains this effect in addition to the established BFLP effect. The rationale of the BFLP effect necessitates specifying individual achievement as an independent variable and we controlled for gender and parent education to reduce potential for endogeneity bias. Another important limitation of the current study is that its findings are limited to the academic self-concept and achievement in mathematics. It seems reasonable to assume that our findings will generalize to other school subjects that are highly valued across cultures and for which established achievement standards exists. This is usually assumed to be true for mathematics and natural sciences which are considered core subjects in all developed countries (Meyer & Ramirez, 2000). It is logically impossible to establish an attitude achievement paradox for school subject that cannot be measured on a common metric across countries, like achievement in native language or the arts. 8. Conclusions The attitude achievement paradox is a vexing phenomenon in intercultural research on student achievement and self-concept. In this study, we demonstrated that modesty bias is one piece of the puzzle to explain this effect but there are other mechanisms to consider as well – for example, experienced academic pressure. Cultural differences in self-concept, particularly between western and eastern cultures, are complex and not easy to study with convenience sample commonly used in comparative research. Nonetheless, international large-scale achievement data have the potential to further our understanding of underlying psychological processes. References Bandura, A. (1994). Self-efficacy. Encyclopedia of Human Behavior, 4, 71–81. Brislin, R. W. (1970). Back-translation for cross-cultural research. Journal of Cross-Cultural Psychology, 1, 185–216. Chen, C., Lee, S. -Y., & Stevenson, H. W. (1995). Response style and cross-cultural comparisons of rating scales among East Asian and North American students. Psychological Science, 6, 170–175. Credé, M., Bashshur, M., & Niehorster, S. (2010). Reference group effects in the measurement of personality and attitudes. Journal of Personality Assessment, 92(5), 390–399. Denissen, J. A., Zarret, N. R., & Eccles, J. S. (2007). I like to do it, I’m able, and I know I am: Longitudinal couplings between domain-specific achievement, self-concept, and interest. Child Development, 78(2), 430–447. Eccles, J. S. (1994). Understanding women's educational and occupational choices. Psychology of Women Quarterly, 18(4), 585–609. Eccles, J. S., Midgley, C., Wigfield, A., Miller, C., Reuman, D., Flanagan, C., & MacIver, D. (1993). Development during adolescence: The impact of stage-environment fit on young adolescents’ experiences in schools and in families. American Psychologist, 48(2), 90–101.

Please cite this article as: Min, I., et al., Modesty Bias and the Attitude-Achievement Paradox Across Nations: A Reanalysis of TIMSS, Learning and Individual Differences (2016), http://dx.doi.org/10.1016/j.lindif.2016.09.008

8

I. Min et al. / Learning and Individual Differences xxx (2016) xxx–xxx

Eccles, J. S., Wigfield, A., & Shiefele, U. (1998). Motivation to Succeed. Handbook of child psychology, 5(3), 1017–1095. Festinger, L. (1954). A theory of social comparison processes. Human Relations, 7(2), 117–140. Greenfield, P. M. (1997). Culture as process: Empirical methods for cultural psychology. In J. W. Berry, Y. H. Poortinga, & J. Pandey (Eds.), Handbook of cross-cultural psychology. 1. (pp. 301–346). Boston: Allyn & Bacon. Harter, S. (1999). The construction of the self: A developmental perspective. New York: Guildford Press. Heine, S. J., Kitayama, S., & Lehman, D. R. (2001). Cultural Differences in Self-Evaluation. Japanese Readily Accept Negative Self-Relevant Information. Journal of CrossCultural Psychology, 32, 434–443. Heine, S. J., Kitayama, S., Lehman, D. R., Takata, T., Ide, E., Leung, C., & Matsumoto, H. (2001). Divergent consequences of success and failure in Japan and North America: An investigation of self-improving motivations and malleable selves. Journal of Personality and Social Psychology, 81, 599–615. Heine, S. J., Lehman, D. R., Peng, K., & Greenholtz, J. (2002). What’s wrong with cross-cultural comparisons of subjective Likert scales: The reference-group problem. Journal of Personality and Social Psychology, 82, 903–918. Heine, S. J., Takata, T., & Lehman, D. R. (2000). Beyond self-presentation: Evidence for selfcriticism among Japanese. Personality and Social Psychology Bulletin, 26(1), 71–78. Kennedy, A., & Trong, K. (2006). A comparison of fourth-graders’ academic self-concept and attitudes toward reading, mathematics, and science in PIRLS and TIMSS countries. Proceedings of the IEA International Research Conference. 2. (pp. 49–60). Lee, J. (2009). Universals and specifics of math self-concept, math self-efficacy, and math anxiety across 41 PISA 2003 participating countries. Learning and Individual Differences, 19(3), 355–365. Lie, S., & Turmo, A. (2005). Cross-country comparability of students’ self-reports. Evidence from PISA 2003 (OECD/PISA, Document: TAG(0505)11). Paris: OECD. Little, R. J. A., & Rubin, D. B. (2002). Statistical Analysis with Missing Data. New York: Wiley. Loveless, T. (2006). The 2006 Brown Center report on American education: How well are American students learning? Volume II, Number 1. Washingoton, D.C.: Brookings Institution Press (Web site: http://www.brookings.edu/index/publications.htm). Maaz, K., Trautwein, U., Lüdtke, O., & Baumert, J. (2008). Educational Transitions and Differential Learning Environments: How Explicit Between-School Tracking Contributes to Social Inequality in Educational Outcomes. Child Development Perspectives, 2, 99–106. Marsh, H. W., & Parker, J. W. (1984). Determinants of student self‐concept: Is it better to be a relatively large fish in a small pond even if you don't learn to swim as well? Journal of Personality and Social Psychology, 47(1), 213–231. Marsh, H. W., & Hau, K. (2003). Big-fish–little-pond effect on academic self-concept: A cross-cultural (26-country) test of the negative effects of academically selective schools. American Psychologist, 58(5), 364–376. Marsh, H. W., & Hau, K. T. (2004). Explaining paradoxical relations between academic self-concepts and achievements: Cross-cultural generalizability of the internal/external frame of reference predictions across 26 countries. Journal of Educational Psychology, 96(1), 56. Marsh, H. W., Abduljabbar, A. S., Morin, A. J., Parker, P., Abdelfattah, F., Nagengast, B., & Abu-Hilal, M. M. (2015). The big-fish-little-pond effect: Generalizability of social

comparison processes over two age cohorts from Western, Asian, and Middle Eastern Islamic countries. Journal of Educational Psychology, 107(1), 258. Marsh, H. W., Seaton, M., Trautwein, U., Lüdkte, O., Hau, K. T., & O’Mara, A. J. (2008). The big-fish-little-pond-effect stands up to critical scrutiny: Implications for theory, methodology, and future research. Educational Psychology Review, 20(3), 319–350. Marsh, H. W., Trautwein, U., Lüdkte, O., Köller, O., & Baumert, J. (2005). Academic selfconcept, interest, and standardized test scores: Reciprocal effects models of causal ordering. Child Development, 76(2), 397–416. Meyer, J. W., & Ramirez, F. O. (2000). The world institutionalization of education. In J. Schriever (Ed.), Discourse formation in comparative eduation (pp. 111–132). Frankfurt/M.: Lang. Nisbett, R. E. (2004). The Geography of thought: How Asians and Westerners think differently… and why. New York: Free Press. Olson, J. F., Martin, M. O., & Mullis, I. V. (Eds.). (2008). TIMSS 2007 technical report. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. Peng, K., Nisbett, R. E., & Wong, N. Y. C. (1997). Validity problems comparing values across cultures and possible solutions. Psychological Methods, 2, 329–344. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage. Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Research, 39(2), 142–151. Seaton, M., Marsh, H. W., & Craven, R. G. (2009). Earning its place as a pan-human theory: Universality of the big-fish-little-pond effect across 41 culturally and economically diverse countries. Journal of Educational Psychology, 101(2), 403–419. Shen, C., & Tam, H. K. (2008). The paradoxical relationship between student achievement and self-perception: A corss-national analysis based on three waves of TIMSS data. Educational Research and Evaluation, 14(1), 87–100. Stevenson, H. W., Chen, C., & Lee, S. (1993a). Motivation and achievement of gifted children in East Asia and the United States. Journal for the Education of the Gifted, 16(3), 223–250. Stevenson, H. W., Chen, C., & Lee, S. (1993b). Mathematics achievement of Chinese, Japanese, and American children: Ten years later. Science, 259(5091), 53–58. Van de gaer, E., Grisay, A., Schulz, W., & Gebhardt, E. (2012). The reference group effect: An explanation of the paradoxical relationship between academic achievement and self-confidence across countries. Journal of Cross-Cultural Psychology, 43(8), 1205–1228. Wigfield, A., & Eccles, J. S. (1992). The development of achievement task values: A theoretical analysis. Developmental Review, 12, 265–310. Wigfield, A., & Eccles, J. S. (2002). The development of competence beliefs, expectancies for success, and achievement values from childhood through adolescence. In A. Wigfield, & J. S. Eccle (Eds.), The development of achievement motivation. (pp. 91–120) San Diego, CA: Academic Press. Wilkins, J. L. M. (2004). Mathematics and science self-concept: An international investigation. The Journal of Experimental Education, 72(4), 331–346. Yamamoto, K., & Kulick, E. (2000). Scaling Methodology and Procedures for the TIMSS Mathematics and Science Scales. In M. O. Martin, K. D. Gregory, & S. E. Stemler (Eds.), TIMSS 1999 technical report. Chestnut Hill, MA: Boston College.

Please cite this article as: Min, I., et al., Modesty Bias and the Attitude-Achievement Paradox Across Nations: A Reanalysis of TIMSS, Learning and Individual Differences (2016), http://dx.doi.org/10.1016/j.lindif.2016.09.008