Moving beyond academic achievement goal measures: A study of social achievement goals

Contemporary Educational Psychology 32 (2007) 667–698 www.elsevier.com/locate/cedpsych

Moving beyond academic achievement goal measures: A study of social achievement goals 夽 S. Jeanne Horst a,¤, Sara J. Finney a, Kenneth E. Barron b a

Center for Assessment and Research Studies—MSC 6806, Department of Graduate Psychology, James Madison University, Harrisonburg, VA 22807, USA b Motivation Research Institute, Department of Psychology—MSC 7401, James Madison University, Harrisonburg, VA 22807, USA Available online 19 December 2006

Abstract The current research explored the theory of social goal orientation. More speciWcally, we conducted three studies utilizing six-independent university student samples to evaluate the construct validity of the Social Achievement Goal Orientation Scale (SAGOS; Ryan & Hopkins, 2003), a measure representing the construct of social goal orientation. The purpose of Study 1 was to: (1) compare the three-dimensional (mastery, performance-approach, and performance-avoidance) model of social goal orientation to three theoretically based competing models, (2) examine item functioning, and (3) assess generalizability of the factor structure. The Wt of the proposed three-factor model was promising; however, areas of misWt and problematic items were identiWed. Stronger support for the threefactor structure of goal orientation was found using scores from an abbreviated 13-item SAGOS. In Study 2, item wording was altered slightly to evaluate a revised Social Achievement Goal Scale (SAGS), yet resulted in similar Wndings. Study 3 examined external validity evidence for the SAGS, garnering some support for the meaning of the scores. Although continued reWnement of the SAGOS and SAGS is recommended, the Wndings help contribute to our general understanding and conceptualization of social goal theory and the role that social goals may play in academic contexts. © 2006 Elsevier Inc. All rights reserved. Keywords: Social goals; Achievement goals; Competence; Goal orientation; Motivation; Construct validity; ConWrmatory factor analysis

夽 *

The authors thank Dr. Dena A. Pastor and Dr. John Hattie for their gracious and insightful comments. Corresponding author. Fax: +1 540 568 7878. E-mail address: [email protected] (S. Jeanne Horst).

0361-476X/$ - see front matter © 2006 Elsevier Inc. All rights reserved. doi:10.1016/j.cedpsych.2006.10.011

668

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

1. Introduction Student motivation is critical for success in school, and it is important to understand its origins. One prominent approach has been to examine students’ underlying achievement goals for their coursework (Pintrich & Schunk, 2002). Recently, a growing number of researchers have recognized the importance of studying social goals along with academic goals to better understand motivational dynamics (Anderman & Anderman, 1999; Covington, 2000; Deci & Ryan, 2000; Dowson & McInerney, 2001; Patrick, Anderman, & Ryan, 2002; Urdan, 1997; Wentzel, 2000). For example, there may be many reasons why a student tries hard (or does not try hard) in an academic setting, some of which are social in nature. Consider a student who is working on a group project. The student may work hard and contribute to the group, not only out of an interest in the subject, but also for social reasons, such as to make friends, impress others, or out of social responsibility. Covington (2000) noted that, although we have a fairly well developed understanding of achievement goals, we understand much less about students’ social goals and their role in academic settings. Moreover, he asserted that students often place as much or more importance on pursuing social goals. Urdan and Maehr (1995, p. 232) suggested that “there is a critical need to untangle the many constructs represented by the term social goals.” Thus, a call for further investigation and continued construct development of social goals has been made. Goals, in general, have been described as the purposes for behavior (Pintrich & Schunk, 2002). Wentzel (2002) oVered a deWnition of goals as “cognitive representations of future events that are powerful motivators of behaviors” (p. 222). Building upon that deWnition, social goals, she asserted, are goals that people set for themselves to achieve particular social outcomes or interactions. Others have oVered a deWnition of social goals that is more speciWc to academic achievement, saying that social goals are the social purposes for behavior in an academic setting (Dowson & McInerney, 2001; Patrick et al., 2002; Urdan, 1997). 1.1. Current approaches to studying social goals Currently, the study of social goals reXects a variety of perspectives. Researchers have attempted to name and quantify speciWc social goals by asking students how frequently they desire or carry out speciWc social behaviors (such as getting together with friends or following classroom rules; Wentzel, 2000, 2002). This focus is directed at deWning and quantifying the content of the goal (e.g., goals that are prosocially oriented, relationship oriented, or status oriented) and then exploring how certain goal content relates to other variables, such as achievement. Utilizing this approach, a researcher may examine whether high achieving students are more likely to espouse a certain goal, such as being responsible (Wentzel, 1989). Another emphasis on social goals intertwines social and academic goals by examining social aspects and their relationship to achievement motivation. For example, Urdan (1997) examined the relationship between peers’ attitudes toward school and a student’s own achievement goal orientation. Rather than approaching the study of social goals from the perspective of trying to Wgure out how much of a certain personal goal a student aspires to, this line of research focuses on describing certain social features (such as peers’ attitudes) and relating those features to achievement goals.

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

669

Gable (2006) conceptualized social goals yet another way by using a dispositional, approach-avoidance perspective. SpeciWcally, she investigated hope for aYliation, in which people have a strong approach motive for aYliation, and fear of rejection, in which people have a strong avoidance motive. Rather than naming speciWc content of social goals, this conceptualization emphasizes the more general approach and avoidance nature of social goals. Taking a diVerent approach, Dweck and Leggett (1988) applied their work on goal orientation in the achievement domain to the social domain. They labeled behavior directed toward the outcome of fulWlling the need for aYliation as social learning goals, and behavior directed toward the outcome of social approval as social performance goals. SpeciWcally, a social learning goal orientation was deWned as behavior directed toward increasing social competence and developing relationships. A social performance goal orientation was deWned as behavior directed toward gaining “positive judgments/avoid(ing) negative judgments of social attributes” (p. 265). Current theories of achievement goal orientation are grounded in Dweck and Leggett’s, 1988 approach. Moreover, a number of goal theorists have recently argued that achievement goals fall under a broader category of competence goals that are “ubiquitous in life” and just as easily applied to “mundane activities” as they are to “grand pursuits” (see Elliot & Dweck, 2005, p. 7). That is, the current conceptualization of goal orientation by leaders in the Weld describes academic achievement goal orientation as just one domain of a broader form of motivation: “competence-relevant motivation.” Elliot and Dweck (2005) argued for “Establishing competence as the central focusƒ,” which then implies that in addition to standard academic competence, competence is applicable to other domains, such as “social competence, emotional competence, cognitive competence, health competence, cultural competence, and moral competence” (p. 8). Hence, competence goals can be just as present in social domains as they can be in academic domains. Building on Dweck and Leggett’s approach, Ryan and Hopkins (2003) formally deWned social goal orientations that parallel the recent three-dimensional framework of achievement goal orientation (Elliot & Church, 1997). For example, students with an academic mastery goal orientation focus on the development of competence and the use of intrapersonal standards to evaluate their success in school. Likewise, a social mastery goal orientation reXects a focus on the development of competence in relationships (e.g., the development of deep friendships) and centers on enjoyment, interest, understanding, as well as caring and respect in relationships (Hopkins & Ryan, 2000). In contrast, students with an academic performance goal orientation focus on the demonstration of competence and the use of interpersonal standards to evaluate success. Similarly, a social performance goal orientation reXects a focus on the demonstration of competence in social relationships (e.g., being socially accepted), using an interpersonal or normative standard of comparison (Ryan & Hopkins, 2003). Social performance goals can also be divided into approach and avoidance components. Students with a social performance-approach goal orientation focus on demonstrating behavior that will be received positively by others, and have a desire to be important and widely liked. Students with a social performance-avoidance goal orientation focus on avoiding behaviors that would result in negative social consequences. Ryan and Hopkins (2003) further proposed, analogous to academic goal orientations (Elliot & McGregor, 2001), that social mastery would be particularly related to positive behaviors and that social performance-avoidance would be related to maladaptive behaviors.

670

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

More recently, work in the academic domain supports a four-dimensional (2 £ 2) framework of goal orientation (mastery-approach, mastery-avoidance, performance-approach, and performance-avoidance; Elliot, 2005; Elliot & McGregor, 2001; Finney, Pieper, & Barron, 2004). The four-dimensional framework adds a fourth goal orientation, masteryavoidance, which is deWned as a focus on the avoidance of incompetence based upon one’s own standards (e.g., attempting to avoid loss of ability; Elliot, 2005). Ryan and Hopkins (2003) conceptualized their original model of social goal orientation as three-dimensional (mastery-approach, performance-approach, and performance-avoidance); however, they discussed the possibility of exploring social mastery-avoidance goal orientation in future work. Social mastery-avoidance was described as “a focus on not acting with a lack of skills or sensitivity towards others, preventing the quality of their relationships from diminishing, and avoiding negative relationships” (Ryan & Hopkins, 2003, p. 32). However, to-date, it has not been included in their measurement of social goal orientations and, more generally, is still being reWned and debated by academic achievement goal researchers. In sum, there are numerous approaches to the study of social goals, each addressing diVerent facets of students’ social motivation. Wentzel’s (1998, 1999, 2000, 2002, 2005) goal content approach explores speciWc goals of students (e.g., prosocial goals) and their relationship to their academic achievement and adjustment. Urdan’s (1997) approach blends the study of academic achievement goal orientations and peer relationships. Gable’s (2006) work utilizes a broader approach-avoidance perspective. Dweck and Leggett’s (1988) approach applies the concept of achievement goal orientation and competence pursuits to the social domain. Each of the various approaches considers students’ social goals at various levels of speciWcity, and vary in the conceptualization of social goals in relation to academic goals. Wentzel (1999) called for further work in this area, saying “Although it makes intuitive as well as theoretical sense to consider social and academic domains as distinct, empirical evidence to support this conclusion has not yet been generated by the Weld” (p. 84). She later added “Although these methodological problems require careful and systematic work to resolve, such work will be necessary to understand the entire range of processes involved in social goal setting” (Wentzel, 2002, p. 233). Recognizing that there may be a number of viable approaches to the study of social goals, we chose to further explore the theory of achievement goal orientation in the social domain. In particular, we focused on the recent scale developed by Allison Ryan (Hopkins & Ryan, 2000; Ryan & Hopkins, 2003), which was written to represent social achievement goal orientation. This approach could provide valuable information that could feed back into the further understanding of social goals and the distinctness (or lack of distinctness) of the social and academic domains. There are many similarities between social and academic goal orientations that suggest that the two would be related. In both domains, the distinction between mastery and performance is based upon competence (development versus demonstration) and the referent to judge success (self versus task versus normative; Elliot, 2005; Hopkins & Ryan, 2000; Ryan & Hopkins, 2003). As in the earlier theories of motivation (Ford & Nichols, 1987; McClelland, 1985), each goal orientation may be further described as having a valence (i.e., approach versus avoidance). Orientations with an approach valence have been related to positive behaviors and outcomes, whereas those with an avoidance valence have been related to maladaptive outcomes (Barron & Harackiewicz, 2000; Elliot & McGregor, 2001; Pintrich & Schunk, 2002; Ryan & Hopkins, 2003). However, the focus of the two domains

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

671

diVers, which suggests that social and academic goals, while related, will be distinct. Both are competence-relevant motivation, yet academic achievement goal orientation is focused on academic competence; whereas, social achievement goal orientation is focused on social competence. 1.2. Measurement of social goal orientation and the need for additional validity evidence In order to study social goal orientation, the Social Achievement Goal Orientation Survey (SAGOS; Hopkins & Ryan, 2000; Ryan & Hopkins, 2003) was created. As highlighted by Elliot and Dweck (2005), exploring the applicability of goal orientation in other domains may shed additional light on competence-related motivation. Ryan and Hopkins (2003) made an initial attempt at this by developing a social goal orientation measure that closely paralleled academic achievement goal measures. SpeciWcally, an initial pool of items was formed by modifying items from various academic achievement goal orientation measures. Items were written by Ryan and Hopkins to represent social mastery orientation, social performance-approach orientation, and social performance-avoidance orientation. In the early stages of scale development, Ryan and Hopkins (2003) conducted focus groups of college students to aid in reWnement of their measure. Students in the focus groups completed the SAGOS at the beginning of the session. They then discussed the face validity of the items. Students discussed the applicability of social goal orientation theory to their own experiences in middle school, high school, and college. Ryan et al. applied the Wndings from the focus groups to reWne the scale. Exploratory factor analyses (EFA) were next conducted on scores from the resultant scale and the hypothesized three-factor structure emerged for both middle school and college-aged students (Hopkins & Ryan, 2000; Ryan & Hopkins, 2003). In addition, when the social goal items were simultaneously evaluated with items from a three-factor academic achievement goal questionnaire, the hypothesized six-factor structure emerged. Ryan and Hopkins (2003) have provided an important Wrst step in developing a social goal orientation measure to represent one approach to the study of social goals. Although this provides initial validity evidence, further attention is necessary in order to more clearly understand the properties of the measure. SpeciWcally, the study of the SAGOS is limited, with only preliminary exploration of the factor structure underlying the scores and preliminary exploration of relationships with other constructs (Ryan & Hopkins, 2003). Thus, more stringent examination of both the instrument and theory are warranted. Put simply, in order to make appropriate inferences about students’ social goals and whether this approach of focusing on developing or demonstrating social competence is warranted, we must have evidence that scores collected actually represent the construct of interest—social goal orientation. 1.3. The current research The purpose of the current research was to provide valuable information that could feed back into the further understanding of social goals and the distinctness (or lack of distinctness) of the social and academic domains. In order to inform social goal theory, the validity of the SAGOS was explored. Benson (1998) described a strong program of construct validation that emphasized three stages: substantive, structural, and external. In the Wrst stage, the substantive stage, the theoretical and empirical domains of the construct are

672

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

deWned. In the second stage, the structural stage, internal relationships among the observed variables representing the construct are examined in order to assess whether or not they relate according to theory (e.g., assessing dimensionality). In the third stage, the external stage, the construct is compared to external constructs in order to evaluate expected theoretical relationships. This is the stage where a network of relationships with other constructs is built (i.e., developing the nomological network) and is tested in order to gather convergent and discriminant validity evidence. In the current research, three studies were conducted in order to gather additional construct validity evidence. Study 1 and 2 focused on the structural stages, and Study 3 focused on the external stage. SpeciWc purposes of each study are detailed below. Although the focus in the current research is directed toward the structural and external stages, information gained through the testing of theoretically based rival hypotheses concerning the factor structure and external relationships may also feed back into the substantive stage and aid in further theory development and testing. This approach builds upon Ryan’s work in the substantive stage, which operationally deWned the theory of social goal orientation as competence-based motivation in the social domain. Thus, we will be able to conclude the paper with recommendations for use and continued development of social achievement goal measures. 2. Study 1 Study 1 had two main purposes. First, conWrmatory factor analysis (CFA) was used to examine whether the three-dimensional model reXecting social goal orientation theory (i.e., mastery, performance-approach, and performance-avoidance) would Wt the pattern of relationships among the scores from the SAGOS. In addition, a two-factor mastery/performance model was evaluated to test the possibility that social goal orientation could be more parsimoniously described in terms of mastery and performance, without the performance-approach and performance-avoidance distinction. Likewise, a two-factor approach/ avoidance model was evaluated to test the possibility that students are motivated to either approach success in their social interactions or to avoid failure. Finally, a one-factor model was evaluated to test whether social goal orientation could be adequately described as one overarching, unidimensional construct. Thus, CFA allowed us to test rival hypotheses, which have not been evaluated in past research on the SAGOS. Our second purpose was to examine the stability of model Wt (or misWt) by replicating analyses using several-independent samples and the generalizability of the results by sampling from a diVerent population of students who were further along in their college career than the initial population (upperclass college students vs. freshman). Replications, such as these, continue to build construct validity evidence for social achievement goal measures. SpeciWcally, utilizing scores from additional large independent samples provides important evidence of the stability and generalizabilty of the dimensionality of the scores (Gerbing & Hamilton, 1996; Hurley et al., 1997; MacCallum, Roznowski, & Necowitz, 1992; Raykov & Widaman, 1995). Finally, internal consistency (reliability) of the scores was evaluated. 2.1. Method 2.1.1. Samples SAGOS data were collected from 2720 freshmen and 610 upperclassmen students during two university-wide Assessment Days. Trained proctors read standardized instructions

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

673

and administered the instruments to the students. Students completed the Social Achievement Goal Orientation Survey (SAGOS; Hopkins & Ryan, 2000; Ryan et al., 2004) along with a battery of other academic and developmental questionnaires. 2.1.1.1. Freshman student sample. In order to address the purposes of Study 1, the initial sample of freshmen students was randomly divided into three sub-samples denoted “freshman sample 1,” “freshman sample 2,” and “freshman sample 3.” This was done to provide an important test of the stability of the factor structure and to prepare for possible model misWt, which, in turn, could lead to possible modiWcations to the scale. MacCallum et al. (1992) suggested that when model modiWcations are suggested based upon the Wndings from one sample, it is essential to replicate the study using independent samples in order to examine whether initial Wndings are simply due to idiosyncracies in the Wrst sample. This recommendation was followed in the current study to address the possibility of capitalization on chance associated with any post hoc modiWcations (see also Gerbing & Hamilton, 1996; Hurley et al., 1997; Raykov & Widaman, 1995). Of the total of 2720 freshmen students with complete data on the SAGOS, a sub-sample of 1368 students was randomly selected as freshman sample 1. The remaining 1352 students were randomly subdivided into freshman sample 2 (n D 698) and freshman sample 3 (n D 654). The three samples were of similar demographic composition (see Table 1). 2.1.1.2. Upperclassman student sample. Scores from the SAGOS were also collected from 610 college sophomores and juniors. This sample was denoted “upperclassman sample.” The median age was 20, compared to 18 for the freshman sample, and there were slightly fewer females proportionately than with the freshman sample (See Table 1). 2.1.2. Measures 2.1.2.1. Social Achievement Goal Orientation Survey (SAGOS; Hopkins & Ryan, 2000; Ryan & Hopkins, 2003). The SAGOS is a 22-item measure with items written to represent three social goal orientations: social mastery, social performance-approach, and social performance-avoidance (see Appendix A). The items are rated on a response scale of 1 (not at all true of me) to 5 (very true of me). The social mastery subscale consists of eight items with possible scores ranging from 8 to 40. Both of the social performance subscales consist Table 1 Demographic information for studies one, two, and three Subsample

N Student type Age range Median age % Female % Anglo-American

Study 1

Study 2

Study 3

Freshman sample 1

Freshman sample 2

Freshman sample 3

Upperclass sample

—

—

1368 1st year 17–23 18 64 84

698 1st year 17–20 18 62 85

654 1st year 17–20 18 66 86

610 2nd or 3rd year 18–55 20 58 83

1880 1st year 17–21 18 64 82

610 1st–4th year 18–37 19 60 86

Note. Freshman samples 1–3 were random sub-samples created from a sample of 1st-year students administered the SAGOS in August 2003. The upperclassman sample consisted of sophomores and juniors who were administered the SAGOS in February 2004. Data for Study 2 were collected in August 2004. Data for Study 3 were collected throughout fall 2004. All samples are independent of each other.

674

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

of seven items with possible subscale scores ranging from 7 to 35. For each subscale, higher scores indicate stronger endorsement of that goal. 2.2. Results 2.2.1. Data screening Data from each of the four samples (freshman samples 1, 2, and 3 and the upperclass sample) were examined for multicollinearity, outliers, and normality (both univariately and multivariately). There was no evidence of bivariate or multivariate multicollinearity. No cases were deemed outliers. For each of the four samples, the mastery items were negatively skewed and leptokurtic, and the multivariate kurtosis statistic (Mardia’s Normalized Kurtosis CoeYcient) was greater than the suggested value of three (Bentler & Wu, 2003), indicating that the data were multivariately kurtotic. Means and standard deviations for each of the items may be found in Appendix B. Means were 3.95 and above for mastery items, 2.10–3.46 for performance-approach items, and 2.22–3.01 for performance-avoidance items.1 2.2.2. ConWrmatory factor analyses All CFA’s were performed using LISREL 8.54 (Jöreskog & Sörbom, 1993). Upon consideration of univariate and multivariate kurtosis, maximum likelihood estimation (ML) with adjustments for non-normality was employed. SpeciWcally, the Satorra–Bentler scaled 2, robust Wt indices, and robust standard errors were estimated (e.g., Satorra & Bentler, 1994; Yu & Muthén, 2002). Three Wt indices were used to assess model Wt: the robust root mean square error of approximation (RMSEA), the standardized root mean square residual (SRMR), and the robust comparative Wt index (CFI). Yu and Muthén, 2002 suggested that cutoVs at or below approximately .05 for the robust RMSEA, at or above approximately .96 for the robust CFI, and at or below .07 for the SRMR indicate adequate Wt. 2.2.3. Freshman sample 1 2.2.3.1. Assessing Model Fit. Table 2 presents the Wt indices for each of the hypothesized models for freshman sample 1. Although the Wt statistics did not suggest close Wt, the threefactor model appeared the most promising of the four models tested.2 2.2.3.2. Diagnosing misWt: standardized covariance residuals and modiWcation indices. Although the Wt of the three-factor model appeared promising, it was not adequate. Consequently, in order to diagnose speciWc areas of misWt, standardized covariance residuals, modiWcation indices (MIs), and theoretical considerations of item wording were examined. The standardized covariance residuals represent how well the three-factor model reproduced the relationships among the individual pairs of items. Large positive values indicate that the relationship between the pair of items is not wellreproduced by the model (i.e., underestimated), implying that the items share variability after controlling for the constructs of interest (construct irrelevant variance). An examination of the standardized covariance residuals from the three-factor model indicated Covariance matrices for all samples used in the current study are available from the Wrst author. Due to the poor Wt of all the models, chi-square diVerence tests were not performed and parameter estimates were not interpreted. 1 2

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

675

that items 1, 2, 14, 19, and 22 from mastery; items 10, 11, 15, and 18 from performanceapproach; and items 4, 5, 9, 16, and 21 from performance-avoidance all exceeded a cutoV of greater than three (Raykov & Marcoulides, 2000). Examination of MIs identiWed the same problematic items. However, before recommending modiWcations based on the results of the initial sample, we needed to investigate if the three-factor model would continue to show misWt, and if we would discover the same problematic items across independent samples (MacCallum et al., 1992). In other words, we would only feel comfortable modifying the scale if we found stable patterns of misWt across multiple samples. 2.2.4. Freshman sample 2 The purpose of analyzing freshman sample 2 was to, Wrst and foremost, replicate the analyses performed using freshman sample 1. This included comparing the Wt of the theoretically based three-factor, two-factor mastery/performance, two-factor approach/avoidance, and one-factor models. This also included evaluating whether speciWc areas of model misWt replicated across the two samples (MacCallum et al., 1992). If so, the SAGOS items would be examined in order to diagnose possible reasons for misWt (e.g., item wording), and we would be in a much better position to recommend if problematic items should be removed from the scale. The Wt indices associated with each of the four models for freshmen sample 2 are presented in Table 2. Results were similar to those found using freshman sample 1, suggesting less than adequate Wt of the models to the data. Standardized covariance residuals and modiWcation indices were evaluated, and patterns similar to, but not identical to, those of freshman sample 1 were found. Based upon the standardized residuals and modiWcation indices across the two studies, the following items seemed to be problematic for each subscale: items 1, 2, 14, 19, and 22 (mastery subscale); items 10, 11, 15, and 18 (performance-approach subscale); and items 4, 5, 9, 16, and 21 (performance-avoidance subscale). Table 2 Fit statistics for the various hypothesized models from freshman sample 1 (N D 1368) and freshman sample 2 (N D 698) Model

ML 2

S–B scaled df 2

Robust Robust RMSEA SRMR CFI RMSEA 90% conWdence interval

Freshman sample 1 (a) Three-factor (b) Two-factor mastery/perform (c) Two-factor approach/avoid (d) One-factor

2162.83 3319.34 4796.64 Did not converge

1926.23 3477.11 5901.17 —

206 208 208 —

0.91 0.83 0.70 —

0.08 0.11 0.14 —

0.08–0.08 0.10–0.11 0.14–0.14 —

0.08 0.08 0.14 —

Freshman sample 2 (a) Three-factor (b) Two-factor mastery/perform (c) Two-factor approach/avoid (d) One-factor

1143.45 1945.11 3269.26 3351.08

1025.73 2025.41 4024.56 3516.59

206 208 208 209

0.91 0.79 0.56 0.62

0.08 0.11 0.16 0.15

0.07–0.08 0.11–0.12 0.16–0.17 0.15–0.16

0.07 0.09 0.19 0.15

Note. ML 2 D maximum likelihood 2; S–B2 D Satorra–Bentler scaled 2; Robust CFI D robust comparative Wt index; Robust RMSEA D robust root mean square error of approximation; SRMR D standardized root mean square residual.

676

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

2.2.5. Suggestions for alterations to scale Given the fairly consistent patterns of large standardized covariance residuals and MI’s found across both samples, we examined the instrument to identify possible item wording or theoretical issues associated with problematic items. From this examination, we recommend deleting items 14, 19, and 22 from the mastery subscale. Items 14 and 19 had large standardized residuals associated with both items, as well as redundant wording. Item 14 “I like friendships that challenge me to learn something new about myself” was also chosen for deletion for theoretical reasons. It could be argued that learning new things about oneself is inherent in the development of competence in relationships; however, we were concerned that it may also be addressing other constructs related to personal mastery as opposed to social mastery. Further, item 19 “I feel successful when I learn something new about myself and how I relate to other people” was double-barreled with the Wrst phrase also addressing general self-development. Although item 22 appeared to function well with the other mastery items, its relationships with all of the performance-avoidance items were underestimated, suggesting it shared something in common with these items. When examining the content of item 22 (“I would be successful if I had friends who accepted me for who I am”), we felt that it may be representing constructs related to performance-avoidance, such as insecurity or feeling unaccepted. In addition, any item (including 22) that included the word “successful” tended to be related across the subscales, as evidenced by large standardized residuals. Further, it was questioned whether wording of mastery items, such as “I feel successful when,” “I like friendships,” or use of the word “important” may better assess valuation or attributions, rather than goals. We recommend deleting items 11, 15, and 18 from the performance-approach subscale. Standardized covariance residuals for the relationships between items 15 and 18 and all of the mastery subscale items were large. Item 15 included the word “successful,” which as noted above appeared to be problematic across the subscales for both freshman samples 1 and 2. Further, the item was double-barreled. We were concerned that the content of Item 18, “I want to be seen as important by other people,” may be driven by more than one latent construct in the sense that one can be “important” to someone else in terms of status (performance-approach) or “important” to someone else in terms of closeness in a relationship (mastery). Item 11, “It is important to me to have ‘cool’ friends” was deleted due to ambiguity of the word “cool.” Friends may be considered “cool” because they are popular (performance-approach), or friends may be considered “cool” because they are a close friend and one enjoys their company (mastery). Item 11 shared a large standardized covariance residual with item 10 across both samples. It was selected for deletion over item 10 on the basis of the use of the word “important,” and because we felt that the word “popular” in item 10 may be less ambiguous than the word “cool” in representing the performanceapproach construct. Finally, we recommend deleting items 4, 9, and 12 from the performance-avoidance subscale. The relationship between items 4 and 5 on the performance-avoidance subscale was not reproduced well in either sample. Item 4 was deleted over item 5, because item 5 was worded as “my goal is to avoidƒ” oVering more face validity as a goal item. Further, relationships between item 4 and all of the remaining performance-avoidance items were not represented well by the model. Items 9 and 12 both included the wording “successful,” and both used double negatives. Finally, Item 12 included the word “dork,” which has additional inappropriate connotations.

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

677

In sum, nine items were recommended for removal from the original 22-item scale, resulting in a reduced 13-item version of the SAGOS; Wve mastery items, four performance-approach items, and four performance-avoidance items. However, it was still unclear whether or not the three-factor model would Wt using this abbreviated version. 2.2.6. Freshman sample 3 2.2.6.1. Purpose. The purpose of using freshmen sample 3 was to examine the Wt of each of the four theoretically based models using scores from the 13-item abbreviated scale.3 2.2.6.2. Assessing Model Fit: Abbreviated 13-item SAGOS. Fit indices for the three-factor model indicated support for the Wt of the model to the data (see Table 3). All but six of the standardized covariance residuals were below an absolute value of three with none exceeding the value of 5.13. Therefore, both the Wt indices and standardized covariance residuals lent support to the three-factor model of social goal orientation as measured by scores from the abbreviated 13-item scale. 2.2.6.3. Parameter estimates. Given the adequate Wt of the three-factor model to the 13item SAGOS, the unstandardized and standardized coeYcients, error terms, and variance explained (R2) were examined (see Table 3). All unstandardized paths were signiWcant (p < .05). However, for more than half of the items, less than 50% of their variance was explained by the factor for which they were written. In turn, some of the standardized error terms (1 ¡ R2) were high, suggesting random variation (i.e., unreliability) or unique systematic variance associated with something other than the construct of interest (e.g., variance associated with another construct, wording issues, or method variance; DeShon, 1998; Jöreskog, 1993). In other words, although the three-factor model Wt responses from the 13item SAGOS, some of the items had a large amount of unexplained variance. 2.2.6.4. Relationships among factors and reliabilities. Table 3 also includes the intercorrelations among the three factors, variance explained for each subscale, the mean subscale scores, and the reliabilities. The correlation between the two performance scales was moderate and positive (r D .59), as would be expected, given that they are both performance goals, and that this relationship has been found previously (Hopkins & Ryan, 2000; Ryan & Hopkins, 2003). The mastery subscale was uncorrelated with the performance-approach and performance-avoidance subscales (r D .08 and r D .07, respectively). This was not fully consistent with previous studies, as a weak positive relationship has been found between mastery and performance-approach (Hopkins & Ryan, 2000; Ryan & Hopkins, 2003); however, this is not unexpected, as we were modeling a revised 13-item scale rather than the original 22-item scale. The reliabilities for scores from the mastery, performance-approach, and performanceavoidance subscales were adequate (.79 and higher). Interestingly, the mean for the mastery subscale was high (22.81 out of a possible 25). This, in combination with the negative skew that was found at the item level, indicated the presence of a ceiling eVect for this subscale.

3 It must be noted that scores from the full 22-item scale were also examined. Importantly, Wt indices, standardized residuals and modiWcation indices were similar to those found for samples 1 and 2. This provided further evidence that the misWt was consistently associated with the problematic items that were identiWed.

678

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

Table 3 Fit indices, unstandardized (standardized) parameter estimates, and subscale characteristics for freshman sample 3 Abbreviated 13-item scale Model

ML 2

S–B scaled 2

df Robust CFI

(a) Three-factor (b) Two-factor mastery/perform (c) Two-factor approach/avoidance (d) One-factor

263.17 611.72 1307.03 Did not converge

225.81 612.31 1398.79 —

62 64 64 —

0.95 0.84 0.61 —

Robust RMSEA RMSEA 90% conWdence interval .064 0.06–0.07 0.11 0.11–0.12 0.18 0.17–0.19 — —

SRMR

0.05 0.07 0.19 —

Path coeYcients

Error variance

R2 value

.57 (.76) .46 (.81) .54 (.62) .34 (.55) .40 (.54)

.24 (.42) .11 (.34) .46 (.62) .27 (.70) .40 (.71)

.58 .66 .38 .30 .29

Performance-approach 3 6 10 20

.48 (.52) .77 (.72) .77 (.77) .88 (.85)

.63 (.73) .55 (.48) .42 (.41) .29 (.27)

.27 .52 .59 .73

Performance-avoidance 5 16 17 21

.68 (.60) .90 (.83) .76 (.68) .86 (.68)

.82 (.64) .37 (.31) .67 (.53) .84 (.53)

.36 .69 .47 .47

Mastery 1.00a 0.08 0.07

Performance-approach

Performance-avoidance

Mastery Performance-approach Performance-avoidance

1.00 0.59

1.00

Means Standard deviations Reliabilities Variance explainedb

22.81 2.64 0.79 0.44

8.95 3.21 0.81 0.53

10.08 3.59 0.80 0.50

Items Mastery 1 2 7 8 13

Note. ML 2 D maximum likelihood 2; S–B2 D Satorra–Bentler scaled 2; Robust CFI D robust comparative Wt index; Robust RMSEA D robust root mean square error of approximation; SRMR D standardized root mean square residual. N D 654. a Correlations were disattenuated for measurement error (Jöreskog, 1993). b Variance explained indicates how much variance the factor can explain in the items that represent it.

2.2.7. Upperclassman sample 2.2.7.1. Purpose. The purpose of analyzing the upperclassman sample was to investigate the generalizability of the Wndings from the previous three samples to scores gathered from a slightly diVerent population. SpeciWcally, the Wrst purpose was to repeat the steps performed on the freshman samples using a sample of upperclass students’ responses to the original 22-

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

679

item scale. This was done in order to investigate if the lack of Wt found with the freshman samples was simply due to studying that level of student. The second purpose was to investigate whether speciWc areas of misWt found with the freshmen samples replicated using an upperclassmen sample. The third purpose, pending the Wndings from purposes one and two, was to examine the Wt of the three-factor, two-factor mastery/performance, two-factor approach/avoidance, and the one-factor models to scores from the 13-item abbreviated scale. 2.2.7.2. Assessing Model Fit: Full 22-item SAGOS. The Wt of the four models was evaluated, and once again, although misspeciWed, the three-factor model was most promising (Robust CFI D 0.91; Robust RMSEA D 0.08; SRMR D 0.08). The two-factor mastery/performance (Robust CFI D 0.85; Robust RMSEA D 0.10; SRMR D 0.09) and the two-factor approach/avoidance (Robust CFI D 0.64; Robust RMSEA D 0.16; SRMR D 0.20) models clearly did not Wt; and the one-factor model did not converge (most likely because it was grossly misspeciWed). 2.2.7.3. Diagnosing misWt: standardized covariance residuals and modiWcation indices. Examination of standardized covariance residuals and MI’s associated with the threefactor model revealed strikingly similar patterns to those found with the three freshmen samples. These results provided further evidence that these items were not functioning well and, importantly, that the same patterns of misWt replicated across four-independent samples representing two diVerent populations of students (freshmen and upperclassmen). 2.2.7.4. Assessing Model Fit: 13-item Abbreviated SAGOS. Fit indices for the three-factor model using scores from the abbreviated 13-item scale indicated that the Wt of the model to the data was supported (see Table 4). All but four of the standardized covariance residuals were below an absolute value of three with none exceeding the value of 4.45. Taken together, the Wt statistics and standardized covariance residuals suggested that the threefactor model represented the relationships among the 13 items well. As found with the previous samples, the competing models clearly did not Wt the data. 2.2.7.5. Parameter estimates and relationships among factors. Table 4 includes the unstandardized and standardized path coeYcients, error terms, and variance explained (R2) in the individual items by the factors for the 13-item abbreviated SAGOS. All the unstandardized path coeYcients were signiWcant. As with freshman sample 3, some of the items had a large amount of unexplained variance. The pattern of intercorrelations among the three factors was similar to those found with freshman sample 3. Reliabilities were adequate (.79 and higher), and the mean for the mastery factor (subscale mean D 22.30 out of a possible 25) was again high, suggesting a ceiling eVect. 2.3. Discussion In summary, some support was found for the three-factor model of scores from the SAGOS. SpeciWcally, the Wt of the three-factor model using scores from the full 22-item version of the scale was promising, but not adequate. In order to make informed decisions regarding alterations to the SAGOS, careful study was undertaken, revealing a consistent pattern of model misWt across four-independent samples of college students. Based upon empirical Wndings, item wording, and theoretical issues, nine items were removed from the

680

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

Table 4 Fit indices, unstandardized (standardized) parameter estimates, and subscale characteristics for the upperclass sample Abbreviated 13-item scale Model

ML 2

(a) Three-factor (b) Two-factor mastery/perform (c) Two-factor approach/avoidance (d) One-factor

229.09 524.61 1396.97 Did not converge

S–B scaled 2 167.95 428.34 1296.79 —

df Robust CFI 62 0.97 64 0.88 64 0.61 — —

Robust RMSEA 0.05 0.10 0.18 —

RMSEA 90% conWdence interval 0.04–0.06 0.09–0.11 0.17–0.19 —

SRMR 0.05 0.07 0.21 —

Path coeYcients

Error variance

R2 value

.65 (.76) .62 (.80) .61 (.68) .58 (.77) .61 (.76)

.31 (.43) .22 (.36) .43 (.54) .23 (.41) .28 (.43)

.57 .64 .46 .59 .57

Performance-approach 3 6 10 20

.57 (.58) .81 (.75) .83 (.81) .91 (.85)

.65 (.66) .52 (.44) .36 (.34) .32 (.28)

.34 .56 .66 .72

Performance-avoidance 5 16 17 21

.68 (.62) .81 (.82) .74 (.69) .74 (.65)

.73 (.61) .32 (.33) .61 (.53) .75 (.58)

.39 .67 .47 .42

Mastery 1.00a ¡0.02 ¡0.07

Performance-approach

Performance-avoidance

Mastery Performance-approach Performance-avoidance

1.00 0.63

1.00

Means Standard deviations Reliabilities Variance explainedb

22.30 3.45 .87 .57

9.01 3.44 .84 .57

9.42 3.37 .79 .49

Items Mastery 1 2 7 8 13

Note. ML 2 D maximum Likelihood 2; S–B 2 D Satorra–Bentler scaled 2; Robust CFI D robust comparative Wt index; Robust RMSEA D robust root mean square error of approximation; SRMR D standardized root mean square residual. N D 610. a Correlations were disattenuated for measurement error (Jöreskog, 1993). b Variance explained indicates how much variance the factor can explain in the items that represent it.

scale, resulting in a 13-item abbreviated version of the SAGOS. Strong support for the 13item abbreviated version of the SAGOS was found in the form of a well-Wtting three-factor model using two-independent samples representing two diVerent populations of students (freshmen and upperclassmen). However, two issues emerged that caused us concern. First, all of the mastery items had non-normal distributions that were leptokurtic and negatively skewed, reXecting a ceiling

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

681

eVect. SpeciWcally, students were choosing the highest two response options at an alarming rate. Recognizing that there may be multiple reasons for the ceiling eVect, we thought that extending the response scale might be one possible solution. Therefore, we decided to conduct another study in which the response scale was increased to seven options (the original scale used Wve). Second, some items had low R2 values across each sample. This suggested that, although the overall Wt of the model was adequate, some items had a great deal of unexplained variability. After examining item wording, we wondered whether some of the items were tapping into constructs other than achievement goals. SpeciWcally, items that begin with the phrasing “It is important to meƒ” or “I feel successful whenƒ” may be representing constructs that are related to achievement goals (e.g., valuation or eYcacy), rather than a direct measure of achievement goals. For example, Pintrich (2003) stressed that we “still do not understand how this ‘binding’ of goals and values occurs or why students may pursue goals that they do not necessarily value or why they don’t pursue goals that reXect their values” (p. 675). We recognize that inclusion of words such as “successful” and “important” were important to some of the varying conceptualizations of academic goal orientation (hence the reasoning behind the speciWc wording of the items). Nonetheless we decided to investigate the eVect of beginning each item with the phrase “My goal isƒ” in order to remove any ambiguity as to what the item was representing. To address the concerns that emerged across the multiple samples used in Study 1, we conducted Study 2 to evaluate alterations to both the response scale format as well as item phrasing of the original SAGOS. All 22 of the original SAGOS items were retained in order to honor the SAGOS’ attempt to cover the breadth of the social goal orientation construct. Therefore, keeping as much of the original wording as possible, we re-wrote the 22-items, beginning each item with the phrase “My goal isƒ” (see Appendix C). Starting each item with “my goal is” allows greater precision and less ambiguity associated with the item. Furthermore, Elliot (2005) outlined three approaches to understanding and measuring the “goal” in “achievement goals” (p. 65): aim, a combination of reason and aim, or an overarching orientation. Our approach to the rewording of the items was to more closely examine students’ “aim,” given its more “conceptual precision” (Elliot, 2005) without, as Pintrich (2003) stated, the “binding of goals and values” (p. 675). Furthermore, Elliot (2005) stated “With regard to the conceptualization of goals as overarching orientations, I think it is best to keep aims conceptually separate from the many diVerent dispositions, tendencies, processes, and outcomes to which aims are associated, and to empirically examine the links between the antecedents of aims and their aVective, cognitive, and behavioral consequences.” (p. 65) Following this suggestion we slightly reworded items in order to more clearly represent the speciWc goal, or aim. This change aligns with current calls to represent goals more precisely and, thus, disentangling values and goals (Elliot, 2005; Elliot and Thrash, 2001). Because we focused on an achievement goal measure conceptualized as aims (a.k.a. “my goal is”), we also opted to call this revised scale the Social Achievement Goal Scale (SAGS) versus the SAGOS to capture and highlight this conceptual precision. Thus, we dropped orientation from the title of the scale. 3. Study 2 The purpose of Study 2 was threefold. The Wrst purpose was to repeat the steps performed in Study 1 using the rewritten SAGS items. This included comparing the Wt of the four theoretically based models. All 22 items were used in order to investigate if

682

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

this wording change would diminish model misWt, and thus eliminate the need to remove the nine items that had been deleted from the SAGOS in Study 1. In addition, given the wording and response option changes, we wanted to examine our concerns of potential ceiling eVects associated with certain items. The second purpose was to investigate whether the speciWc areas of misWt found in Study 1 would replicate when examining scores from the 22-item SAGS. The Wnal purpose, pending the Wndings from purposes one and two, was to examine (a) the Wt of the three-factor model, as well as the three competing models, to scores from the 13-item abbreviated scale, (b) the internal consistency of scores, (c) the correlations among the resultant factors, and (d) whether or not the variance explained increased from the previous study given the wording changes. 3.1. Method 3.1.1. Sample Data were again collected during a university-wide assessment day. A total of 1880 freshmen had complete SAGS data (see Table 1). Administration of the instruments was the same as in Study 1. 3.1.2. Measures 3.1.2.1. Social Achievement Goal Survey (SAGS). The wording of the original 22-items on the SAGOS was altered. SpeciWcally, the phrase “My goal is toƒ” was added to the beginning of each item, while retaining the portion of the item that was written to capture the theoretical content of social achievement goals. With the exception of the beginning phrase of each item, an attempt was made to keep the wording as close to the original as possible. For instance, item 1, “It is important to me to have friends who really understand me” was rewritten as “My goal is to have friends who really understand me”. (see Appendix C). Additionally, the response scale was increased to seven response options (the original scale used Wve); therefore, students responded on a scale of 1 (not at all true of me) to 7 (very true of me). 3.2. Results 3.2.1. Data screening and analyses As in Study 1, data were examined for multicollinearity, the presence of outliers, and normality, both univariately and multivariately. The mastery items were negatively skewed and leptokurtic, and Mardia’s Normalized Kurtosis CoeYcient suggested that the data were multivariately kurtotic (Bentler & Wu, 2003). 3.2.1.1. Assessing Model Fit: Full 22-Item SAGS. When modeling the full 22-item SAGS scores, the three-factor model appeared promising. Interestingly, the model-data Wt was better after the wording change than with the original phrasing; however, Wt was not ideal (Robust CFI D 0.95; Robust RMSEA D 0.08; SRMR D 0.06). Thus, changing the item wording did not alleviate all of the model misWt found when modeling the original SAGOS scores. The two-factor mastery/performance (Robust CFI D 0.90; Robust RMSEA D 0.10; SRMR D 0.07), two-factor approach/avoidance (Robust CFI D 0.77; Robust RMSEA D 0.16; SRMR D 0.16), and one-factor (Robust CFI D 0.73; Robust RMSEA D 0.17; SRMR D 0.16) models clearly did not Wt.

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

683

3.2.1.2. Diagnosing misWt: standardized residuals and modiWcation indices. The same procedures used in Study 1 to diagnose misWt were used in Study 2. In general, although some diVerences were noted, the pattern of residuals and MI’s from the three-factor model in Study 2 resembled the pattern found in Study 1. The similar pattern of residuals and MI’s emphasized that the areas of misWt were stable across multiple samples. The Wndings also suggested that altering item wording did not remedy all problems associated with model misWt when modeling all 22 original SAGOS items. Given that the standardized residuals and MI’s replicated, the three-factor model was Wt to scores from the 13-item SAGS. 3.2.1.3. Assessing Model Fit: Abbreviated 13-item SAGS. Model Wt for the 13-item SAGS was examined. The Wt of the three-factor model to the data was supported. As before the alternative models clearly did not Wt (see Table 5). 3.2.1.4. Parameter estimates and relationships among factors. Table 5 includes the unstandardized and standardized path coeYcients, error terms, and variance explained (R2) in the individual items by the factors, as well as the intercorrelations among the factors, mean subscale scores, reliabilities, and variance explained associated with each factor for the 13item SAGS. The Wndings were similar to Study 1 and suggested that, although the Wt of the model was supported, there were a number of items for which half of their variance was unexplained. The pattern of intercorrelations among the three factors was similar to that found in the other phases of the study, with the exception of the relationship between the mastery and performance-avoidance factors (r D .11) being higher than what was found in Study 1 (nil). The reliabilities for scores from the three subscales were again adequate (.83 and higher). Even with the broader range of response options, the average mastery score remained high (mean was 31.56 out of a possible 35). 3.3. Discussion The newly worded items and broader range of response options (SAGS) did not remedy the issues that were identiWed in Study 1. Although the change in wording aided in theoretical precision to address the call from Elliot and Thrash (2001), it did not result in empirical precision. In fact, the results were similar to Study 1 in that there were similar patterns of misWt when using scores from the full 22-item revised scale, and the three-factor model was supported when using scores from the 13-item abbreviated version. In both Study 1 and Study 2, the 13-item version of the scale functioned better than the 22-item scale. Additionally, despite a broader range of response options that was provided for the SAGS, a ceiling eVect for the mastery items remained. As noted in the introduction, Study 1 and Study 2 were conducted to gather needed evidence concerning the structure of the SAGOS scores. Although, CFA can help us to understand the structure or the dimensionality of the scores, it does not inform about what is being studied (Benson & Nassar, 1998). Therefore, the next necessary step was to address the third stage of a strong program of construct validity: the external stage (Benson, 1998). In order to provide further validity evidence for the scores from the SAGS, a network of relationships must be established through the testing of rival hypotheses and building the nomological net (Cronbach & Meehl, 1955). SpeciWcally, this requires investigating hypothesized relationships with theoretically related constructs. Benson (1998) noted that this is the “most crucial stage” in the validity process as it allows us to apply meaning to the

684

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

Table 5 Fit Indices, unstandardized (standardized) parameter estimates, and subscale characteristics for Study 2 Abbreviated 13-item SAGS Model

ML 2

(a) Three-factor (b) Two-factor mastery/perform (c) Two-factor approach-avoidance (d) One-factor

616.47 1512.44 5093.86 Did not converage

S–B scaled 2 544.21 1355.00 5840.00 —

df Robust CFI 62 0.97 64 0.90 64 0.77 — —

Robust RMSEA 0.06 0.11 0.22 —

RMSEA 90% conWdence interval 0.06–0.07 0.099–0.10 0.21–0.22 —

SRMR 0.050 0.061 0.24 —

Items Mastery 1 2 7 8 13

Path coeYcients

Error variance

R2 value

.88 (.83) .77 (.87) .68 (.57) .42 (.59) .66 (.64)

.36 (.32) .19 (.25) .94 (.67) .27 (.65) .63 (.59)

.68 .75 .33 .35 .41

Performance-approach 3 6 10 20

.95 (.64) 1.22 (.72) 1.29 (.85) 1.44 (.87)

1.30 (.59) 1.38 (.48) .63 (.27) .68 (.25)

.41 .52 .73 .75

Performance-avoidance 5 16 17 21

1.20 (.70) 1.50 (.87) 1.38 (.84) 1.05 (.59)

1.48 (.51) .69 (.23) .82 (.30) 2.10 (.66)

.49 .77 .70 .34

Mastery 1.00a 0.06 0.11 31.56 3.45 .83 .50

Performance-approach

Performance-avoidance

Mastery Performance-approach Performance-avoid Means Standard deviations Reliabilities Variance explainedb

1.00 0.77 12.42 3.44 .86 .60

1.00 14.76 3.37 .84 .58

Note. ML 2 Dmaximum likelihood 2; S–B 2 DSatorra–Bentler scaled 2; Robust CFIDrobust comparative Wt index; Robust RMSEADrobust root mean square error of approximation; SRMR Dstandardized root mean square residual. N D 1880. a Correlations were disattenuated for measurement error (Jöreskog, 1993). b Variance explained indicates how much variance the factor can explain in the items that represent it.

scores that we collect (p. 14). In all, information gathered can feed back into the theory of competence-relevant motivation. Thus, we conducted one additional study. 4. Study 3 The Wrst purpose of Study 3 was to examine the distinctness of social and academic achievement goals by collecting participant responses on both social and academic achievement goal measures. We could then test a six-factor CFA model (social mastery,

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

685

social performance-approach, social performance-avoidance, academic mastery, academic performance-approach, and academic performance-avoidance). Alternatively, a three-factor CFA model (social and academic mastery, social and academic performance-approach, and social and academic performance-avoidance), which speciWed no distinction between social and academic goals was Wt to the data. As social and academic goals represent competence goals in diVerent domains, we hypothesized superior Wt for the six-factor model. The second purpose was to further explore the relative importance of social and academic achievement goal domains by asking students to rank order deWnitions of each of the six achievement goal orientations (social mastery, social performance-approach, social performance-avoidance, academic mastery, academic performance-approach, and academic performance-avoidance). This would provide further validity evidence for scores from the 13-item SAGS by examining whether or not students’ rank orders of the deWnitions corresponded with responses to the SAGS items. Moreover, if students’ rank orders emphasized the importance of social achievement goals, it would provide additional evidence that students’ social goals may be as important to them as their academic goals (Covington, 2000; Dowson & McInerney, 2001). The third and Wnal purpose of Study 3 was to explore hypothesized relationships with other theoretically-related constructs. SpeciWcally, in addition to academic achievement goals, we hypothesized relationships between the 13-item SAGS and measures of implicit personality theory, well-being, and fear of negative evaluation. 4.1. Hypothesized relationships with academic achievement goals Ryan and Hopkins (2003) examined the correlations between corresponding social and academic goals and found that they were moderate and positive: .41, .38, and .51 for mastery, performance-approach, and performance-avoidance, respectively. Consequently, we hypothesized that the correlations between corresponding social and academic achievement goals would be positive and moderate, showing that they are distinct, yet related, constructs. 4.2. Hypothesized relationships with theories of implicit personality Analogous to the theory of implicit intelligence in the academic domain (Dweck, 1999; Dweck & Leggett, 1988), the theory of implicit personality suggests that children either adopt an incremental theory about their own personality (i.e., personality is malleable and can be changed through eVort; Dweck, 1999; Dweck & Leggett, 1988; Erdley, Cain, Loomis, Dumas-Hines, & Dweck, 1997) or an entity theory (i.e., personality is Wxed and what you were born with). It has been suggested that those with an incremental theory about their personality tend to adopt mastery goals; in contrast, those with an entity theory tend to adopt performance goals (Dweck & Leggett, 1988). In a study in which students were presented with hypothetical social situations, those with an entity theory of personality selected performance goals (e.g., trying to be popular) in response to a hypothetical social situation (Erdley et al., 1997). Consequently, in the current study, we hypothesized that the social performance achievement goals would be positively related to an entity theory of personality. This would be expressed as a negative correlation between theory of personality (i.e., low score suggests entity theory) and both social performanceapproach and social performance-avoidance achievement goals. Conversely, we hypothe-

686

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

sized that social mastery achievement goal would be positively related to an incremental theory of personality. 4.3. Hypothesized relationships with well-being Given that social mastery is deWned as a focus on the development of social competency, we hypothesized that social mastery scores would be positively correlated with a positive relations subscale from a measure of well-being (RyV, 1989). Similarly, given that social performance-avoidance is deWned as the avoidance of demonstrating a lack of social competency, we hypothesized that social performance-avoidance scores would be negatively related to positive relations subscale scores. Others have found positive correlations between relatedness (deWned as feeling “close and connected”) and well-being (e.g., Reis, Sheldon, Gable, Roscoe, & Ryan, 2000). 4.4. Hypothesized relationships with fear of negative evaluation Social performance-avoidance goals are described as a focus on the avoidance of negative judgments about one’s social competency. Consequently, we hypothesized that social performance-avoidance scores would be positively correlated with scores from a measure of fear of negative evaluation (Leary, 1983), which is deWned as concern over negative evaluations from others. Previous research has reported moderate positive correlations between fear of negative evaluation and social anxiety (r D .49), public self-consciousness (r D .41), and private self-consciousness (r D .29; Monfries & Kafer, 1994). In the academic goal domain, Elliot and McGregor (2001) found that fear of failure positively predicted performance-avoidance goal orientation. 4.5. Method 4.5.1. Sample Data were collected from 637 university students who volunteered to complete the instruments either for subject pool credit (n D 408) or extra course credit (n D 229). Of the total sample, 610 students had complete data on the SAGS and each of the external criterion measures. Demographic information may be found in Table 1. This sample was comprised of 189 (31%) freshmen, 142 (23%) sophomores, 232 (38%) juniors, 41 (7%) seniors, and 6 (1%) students did not indicate grade level. 4.5.2. Procedure Students met in a classroom setting. At the beginning of each session the students signed informed consent forms and were handed a packet containing the set of measures. Prior to completing each measure, students were instructed to remove a particular measure from the envelope, and a proctor read standardized instructions. Students were instructed to turn the measure over when Wnished and to wait quietly until everyone was Wnished. When all students had completed the measure, the students were instructed to return the measure to the envelope and told which measure to remove next. This procedure was completed in an attempt to minimize students rushing quickly through the set of measures. In each session, the SAGS was administered Wrst, followed by the academic goal questionnaire. The ranking of goal orientation deWnitions and demographic form was administered last in

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

687

each session. The order of presentation of the remaining measures was counterbalanced across each of the sessions. 4.5.3. Measures 4.5.3.1. Attitudes toward Learning (ATL; Finney et al., 2004). Three subscales from the Attitudes Toward Learning Scale (ATL; Finney et al., 2004), a simple adaptation of the achievement goal questionnaire (AGQ; Elliot & McGregor, 2001), were employed in the current study: mastery-approach (MAP), performance-approach (PAP), and performanceavoidance (PAV). Originally, Elliot and McGregor (2001) wrote the AGQ to assess students’ achievement goals in a speciWc course. The ATL (Finney et al., 2004) is more general and assesses students’ goals across all of their coursework for a given semester.4 The three subscales employed in the current study each consisted of three items. Students rated the items on a scale of 1 (not at all true of me) to 7 (very true of me), with scores ranging from 3 to 21. Endorsement of the MAP items suggests a focus on the development of academic competence (e.g., “I want to learn as much as possible this semester”). Endorsement of the PAP items suggests a focus on the demonstration of academic competence in relation to others (e.g., “My goal this semester is to get better grades than most of the other students”). Endorsement of the PAV items suggests a focus on avoiding others’ negative judgments of academic competence, especially when compared to others (e.g., “My goal this semester is to avoid performing poorly compared to other students”). Cronbach’s coeYcient alphas computed using the current sample were .85, .92, and .73 for MAP, PAP, and PAV, respectively. 4.5.3.2. Theories of Implicit Personality (TOP; Erdley et al., 1997). A set of four items that were written to measure students’ theories of their own personality (Erdley et al., 1997) were utilized in the current study. The items are based on the theory that students hold either an incremental or entity theory about their own personality (Dweck, 1999). A sample item is “You have a certain personality, and it is something that you can’t do much about.” Students responded to the items on a scale of 1 (really agree) to 6 (really disagree), with possible scores ranging from 4 to 24. Low scores suggest an entity theory, whereas high scores suggest an incremental theory of personality. Cronbach’s coeYcient alpha for the current sample was .91. 4.5.3.3. Scales of psychological well-being: positive relationships with others (PRO; RyV, 1989). The “positive relations with others” (PRO) subscale (9-item per subscale version) from the scales of psychological well-being was used. A sample item is “I enjoy personal and mutual conversations with family members or friends.” Students responded to items using a six-point scale of 1 (strongly disagree) to 6 (strongly agree). After Wve of the items are reversed-scored, the possible composite scores can range from 9 to 54. Those who score high on the scale are said to have “ƒwarm, trusting relationships with others; capable of strong empathy, aVection, and intimacy; understands give and take of human relationships” (p. 1072); those who score low are reported to have few close relationships and report frustration in their relationships (RyV, 1989). Cronbach’s coeYcient alpha for the current study was .79. 4.5.3.4. Brief Fear of Negative Evaluation Scale (FNE; Leary, 1983). The FNE was developed as a measure of a person’s uneasiness about being negatively evaluated by others. 4

The fourth subscale is a mastery-avoidance subscale, which was not used in this study.

688

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

Table 6 Rank order of goal deWnitions and subscale means from SAGS and ATL in study 3 Goal 1. SMAP 2. MAP 3. PAP 4. SPAP 5. PAV 6. SPAV

1st b

330 176 44 7 12 7

2nd

3rd

4th

5th

6th

Subscale meana

146 251 88 49 24 15

53 76 170 133 89 56

31 41 111 176 129 90

8 19 73 108 180 186

7 12 89 102 141 221

6.24 5.24 4.89 3.18 4.60 3.63

Note. SMAP D social mastery; MAP D mastery; PAP D performance-approach; SPAP D social performanceapproach; PAV D performance-avoidance; and SPAV D social performance-avoidance. N D 575. a Subscale means ranged from 1 to 7 for both the SAGS and the ATL subscales. b Value represents number of students selecting the particular rank for the particular achievement goal.

Originally developed as a 30-item true–false measure (Watson & Friend, 1969), it was later shortened into a 12-item measure (Leary, 1983). Participants rated the items on a scale of 1 (not at all characteristic of me) to 5 (extremely characteristic of me), with possible total scores ranging from 12 to 60. Four out of the nine items are reverse-scored. A sample item is “I am afraid that people will Wnd fault with me.” Cronbach’s coeYcient alpha for the current study was .90. 4.5.3.5. Rank-order of social and academic goals. DeWnitions of the six social and academic goals were listed in random order. Students were instructed to read through each of the deWnitions and then rank them in terms of their importance to the student for the current semester, ranging from 1 (most important) to 6 (least important). Appendix D contains the deWnitions that were used in the rank-order task. 4.6. Results 4.6.1. Dimensionality of social and academic goal domains Scores from the 13 SAGS items and the 9 ATL items were simultaneously submitted to CFA in order to test their distinctiveness.5 Fit statistics for the six-factor model suggested good Wt (Robust CFI D 0.97; Robust RMSEA D 0.05; SRMR D 0.05). In addition, Wt indices from the three-factor model specifying no distinction between social and academic goals showed model misWt (Robust CFI D 0.69; Robust RMSEA D 0.15; SRMR D 0.13). These results suggest that social and academic goals are distinct and therefore can not be combined into general mastery, performance-approach, and performance-avoidance goals. 4.6.2. Rank-order of social and academic goals Table 6 lists the rank-ordering of the goals, frequencies of ratings, as well as the subscale means for the SAGS and ATL. Ranked from most to least important were social mastery, academic mastery, academic performance-approach, social performance-approach, academic performance-avoidance, and social performance-avoidance. Notice that the order of The Wt of the three-factor model to just the 13-item SAGS was assessed using this sample in order to gather further evidence of the stability of the three-factor structure. The Wt statistics and parameter estimates were similar to Study 2 and suggest that although Wt of the three-factor model was supported, the unexplained variance for several items was larger than desired. 5

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

689

the averages of the SAGS subscales and ATL subscales (both used a 1–7 response scale) were in the same order as the rank-order of the goal deWnitions, with the exception of social performance-approach, which were the lowest endorsed items, on average. This provides validity evidence for the SAGS, in the sense that students endorsed the items in approximately the same order as their rank order of the deWnitions of the goal orientations. Additionally, it lends credence to the idea that social and academic achievement goals are intertwined, in terms of importance to students (Covington, 2000). 4.6.3. Correlations with external criteria Table 7 includes means, standard deviations, minimum and maximum scores, and reliabilities for each of the scales, as well as correlations among the various criteria. It was hypothesized that scores on the corresponding social and academic goal scales would be positively correlated. Previous research reported correlations of .41 for mastery, .38 for performance-approach, and .51 for performance-avoidance (Ryan & Hopkins, 2003). Although our correlations were also positive, they were weaker than previous research, showing greater distinction between social and academic goals. It must be noted that diVerent measures of academic goal orientation, as well as the revised social goal measure, were utilized in this study than were utilized in previous studies. Additionally, we hypothesized that the theories of personality scores would be positively related to the social mastery scores and negatively related to social performance scores, and that the positive relationships with others scores would be positively related to social mastery scores and negatively related to social performance-avoidance scores. Although the relationships between theories of personality and social goals were in the hypothesized directions, the magnitudes of the correlations were low. The correlations between social goals and both positive relationships with others and fear of negative evaluTable 7 Correlations, means, standard deviations, minimum, maximum and Cronbach’s coeYcient alpha associated with the SAGS subscales, ATL subscales, TOP, PRO, and FNE SMAP

SPAP

SPAV

MAP

PAP

PAV

TOP

PRO

FNE

1. SMAP 2. SPAP 3. SPAV 4. MAP 5. PAP 6. PAV 7. TOP 8. PRO 9. FNE

(.83)a .048 .179 .250 .050 .157 .056 .231 .148

(.86) .584 ¡.065 .273 .297 ¡.133 ¡.102 .326

(.83) .009 .245 .394 ¡.082 ¡.181 .551

(.85) .178 .071 .108 .143 .020

(.92) .438 ¡.039 ¡.063 .168

(.73) ¡.094 ¡.065 .245

(.91) .096 ¡.066

(.79) ¡.230

(.90)

Mean SD Minimum Maximum Possible range of scores

31.19 3.50 12 35 5–35

12.71 4.82 4 27 4–28

14.53 4.92 4 28 4–28

15.71 3.40 4 21 3–21

14.66 4.40 3 21 3–21

13.82 4.06 3 21 3–21

16.14 5.02 4 24 4–24

44.83 6.55 20 54 9–54

34.45 8.93 14 60 12–60

Note. SMAP D social mastery; MAP D mastery; PAP D performance-approach; SPAP D social performanceapproach; PAV D performance-avoidance; and SPAV D social performance-avoidance; TOP D theories of personality; PRO D positive relations with others scale; FNE D fear of negative evaluation. N D 610. a Cronbach’s coeYcient alpha values are on the diagonal of correlation matrix.

690

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

ation were more encouraging. SpeciWcally, positive relationships with others showed a moderate positive relationship with social mastery and a moderate negative relationship with social performance-avoidance. In addition, fear of negative evaluation and social performance-avoidance were strongly related (r D .55), as hypothesized. 4.7. Discussion The primary purpose of Study 3 was to collect external validity evidence for scores from the SAGS using a sixth independent sample of students. When scores from the SAGS and the ATL were simultaneously analyzed via CFA, support was found for six distinct factors, whereas a three-factor model suggesting no distinction between the social and academic domains clearly did not Wt. Support for the six-factor model lends credence to the theory that social achievement goals are related to, yet distinct from, academic achievement goals (see Elliot & Dweck, 2005). When students were asked to rank order deWnitions of social and academic goals, social mastery was clearly ranked the most important, followed by academic mastery. These results are consistent with Covington’s (2000) notion that students may place as much or more importance on pursuing social goals as they do on academic goals. Interestingly, the order of the averages of the SAGS subscales and ATL subscales were in the same order as the rank-order of the goal deWnitions, with the exception of social performance-approach, which were the lowest endorsed items, on average. This provides further validity evidence for the SAGS scores. In general, the correlations with the external criterion measures were in the hypothesized direction. The strongest correlation was between social performance-avoidance and the fear of negative evaluation, clearly supporting their hypothesized relationship. Unfortunately, the hypothesized relationships between social goals and theories of personality were not supported. This could be a function of using TOP as a measure of theories of personality; there has been limited study of the scale. In addition, the failure to Wnd stronger relationships between social mastery and the external criteria may be related to the mastery scale’s ceiling eVect, which may be dampening the relationships found between social mastery and other measures. 5. General discussion Using a framework for goals based upon competence-related pursuits (Elliot, 2005), Ryan and colleagues developed a promising measure for use in studying students’ social goals (Hopkins & Ryan, 2000; Ryan & Hopkins, 2003). Across three studies, we have tried to replicate and further extend the construct validity evidence for scores from the SAGOS, as well as an alternative measure of social achievement goals (SAGS). This provided further study and reWnement of the conceptualization of social goals. Support for the three-factor model of social goal orientation, particularly when using a 13-item abbreviated version of the measure, was found in the current study utilizing responses from six-independent samples of university students involving nearly 6000 respondents. Additionally, the three-factor model consistently outperformed three theoretically-based alternative models. Moreover, the idea that social achievement goals appear to be distinct, yet related to academic achievement goals was supported through Wndings from CFA, students’ rank-ordering of social and academic goal deWni-

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

691

tions, and the magnitude of the average responses to items comprising each of the subscales. Finally, correlations with other measures were, for the most part, in the hypothesized directions. Upon reXecting on the results of the three studies, we have a number of recommendations regarding the SAGOS. The Wrst recommendation for the scale concerns content coverage. Although the items on the original SAGOS underwent careful scrutiny, additional item review from a theoretical standpoint is always beneWcial. In addition, focus groups consisting of people from diverse backgrounds would permit more detailed exploration of relevance and representativeness of the items to actual students’ lives. This can be accomplished by students examining the already existing items, or by conducting open-ended discussions regarding social goals. No matter the methodology, construct coverage and representation must be addressed further. The second recommendation for the scale would be to further explore the ceiling eVect that was consistently found across all three studies for the social mastery items. With respect to the current study, the ceiling eVect may have attenuated the relationships with other measures. A number of possible explanations for the ceiling eVect warrant consideration. One is that this may be a function of the sample that participated in the study. The students in each of the three studies were predominately Anglo-American females and were students who had already “made it” into college. Studies examining responses from various known groups (e.g., the same aged participants who are versus are not attending college, or participants who have versus have not been diagnosed with social anxiety disorder) may aid in understanding whether or not the ceiling eVect is a function of the current sample. An alternative explanation for the ceiling eVect is that this may be an instance of construct under-representation. That is, perhaps the items are not fully addressing the construct of social mastery goals, speciWcally at the higher end of the social mastery continuum. This would suggest that further work needs to be done in deWning the construct and writing items. Another explanation is that the social mastery construct is adequately deWned and, given that humans have a need for aYliation, items addressing social mastery are naturally highly endorsed. For example, in the current study, we attempted to evaluate this by increasing the number of response options. Rather than Wnding an increased range of variability of responses using the more sensitive 1–7 scale, students continued to respond at the highest end of the continuum. That is, perhaps what we are measuring is simply a basic human need for aYliation that a healthy, non-clinical population would highly endorse. A Wnal explanation for the ceiling eVect is that it may be socially desirable to highly endorse the social mastery items, which is an issue that needs to be addressed with any self-report measure. Thus, we feel that the consistent ceiling eVect warrants a return to Benson’s (1998) substantive stage, which would involve a re-examination of the conceptualization and measurement of social mastery. In building a strong program of construct validity, movement through the stages (i.e., substantive stage, structural stage, and external stage) may not be a linear progression, but may weave back and forth through the stages, with each stage providing information that prompts the directions that are taken. The third recommendation for the scale would be the consideration of a masteryavoidance social goal, given emerging support for a 2 £ 2 framework of goal orientation

692

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

(Elliot, 2005; Elliot & McGregor, 2001). However, prior to developing a scale to address social mastery-avoidance, further deWnition of mastery-avoidance, in general, is needed (Brophy, 2004). If students with social mastery-avoidance are those who focus on the development of competence in relationships from the standpoint of avoiding failure in their ability to make and keep friends, one may ask whether this overlaps with other constructs, such as a psychological/social disorder. Talking with students about their own experiences would be an important Wrst step. The fourth recommendation for the scale concerns the conceptualization of goals. With respect to framing goal items as “aim”, “aim and reason”, or “overarching orientation” (Elliot, 2005), much more attention on the link between theory and measurement is needed. If theoretically one conceptualizes goals as “overarching orientations”, then empirically one should construct items that represent overarching orientations. If theoretically one conceptualizes goals as “aims”, then empirically one should construct items that represent aims. Elliot (2005) has recently made an explicit call for the Weld to assess achievement goals as “aims” for conceptual clarity. We agreed and, therefore, altered the original SAGOS scale to reXect aim. However, in the current studies this rewording did not result in substantial improvement of the scale’s properties. This is only, of course, one study, and additional research that continues to evaluate measures using the diVerent framings (aim, aim and reason, and orientation) would be advantageous. There are a few limitations of the study we would like to note, which would help to put the current Wndings in context and help to inform future investigations. First, as previously noted, the samples were predominately female Anglo-American college students who are already successful, in terms of being enrolled in a university. Hence, the Wndings may be limited in their generalizability to other populations. Therefore, measurement invariance studies that examine if social goals are conceptualized equivalently (i.e., same factor structure) across various participant characteristics (e.g., age, gender, ethnicity, and culture) need to be conducted. If measurement invariance can be established, then diVerences across participant characteristics can be studied. For example, developmentally, young adolescents may diVer from older adolescents in terms of the relevance and importance of the various social goals. Anderman, Austin, and Johnson (2002) described the development of goal orientation and suggested that while infants have an intrinsic curiosity toward everything in the environment, interest narrows in later childhood and adolescence, and becomes more speciWc. It may be that the diVering contexts across developmental stages are what aVect diVerences in goal orientation. For example, as students progress through grade levels they tend to become more concerned with their performance (including grades; Anderman et al., 2002). In fact, Anderman et al. (2002) suggested that developmental diVerences in goal orientation may be due to the academic environment being more competitive, thereby promoting performance achievement goals. The same could be true for social goals; performance goals may become more important during diVerent stages of a person’s life. Developmental studies that investigate the change (or lack of change) in individuals’ social goal proWles are needed in order to better understand this phenomenon. Currently there is some evidence that social goals are endorsed diVerently across time. For example, Wentzel (1999) reported that middle school students emphasize social goals of “appropriate” behavior over socializing with peers, whereas high school students emphasize having fun with friends along with being responsible. In sum, competence motivation is salient across a person’s lifetime and across cultures (Elliot & Dweck, 2005). Therefore,

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

693

systematic studies that explore competence-related motivation developmentally warrant further exploration. A second limitation of the current study is the sole reliance on the use of self-report measures as criteria used to gather validity evidence (self-report measures have a number of inherent problems, such as measurement error, restriction of range, and participant bias). SpeciWc to this study, validity evidence for the theories of personality scores has not been well established. This may have directly inXuenced the Wndings by attenuating the correlations between it and the SAGOS subscales. However, this may be a “real” Wnding; additional work is needed to further investigate the relationship between social goals and theories of personality. In general, additional investigations that compare and contrast known groups would provide additional, and possibly stronger, validity evidence. For example, one could compare the social goal proWle for a group of student leaders on campus with the social goal proWle for a group of students diagnosed with social anxiety. In addition, studies that move beyond self-report measures by observing student behavior in real-life settings (e.g., quality and number of interpersonal interactions) would clearly aid in construct validation. Put simply, we understand that this is one attempt in a long and diverse process of gathering validity evidence, and we strongly recommend additional external validity studies using diverse criteria. In addition to the usefulness of scales such as the SAGOS or SAGS, further qualitative work or studies that employ methodologies such as interviews, diaries or thinkalouds would aid in our understanding of students’ social strivings in an academic setting. Similarly, continued research studying the relationship between social and academic goals in academic settings is also needed. That is, when faced with an academic task, such as working on a group project, how do social and academic goals come into play? 6. Conclusions In closing, it is important to highlight that social interactions are quite complex, as reXected by the numerous conceptualizations of students’ social goals: Wentzel’s (1998, 1999, 2000, 2002, 2005) study of social goal content theory, Urdan’s (1997) blend of the study of academic achievement goal orientations and peer relationships, Gable’s (2006) broader approach-avoidance perspective, and Dweck and Leggett’s (1988) goal orientation approach. The current paper evaluated this last approach and the viability of competence-related motivation as one important conceptualization of social motivation by studying the recently developed SAGOS scale (Elliot & Dweck, 2005; Ryan & Hopkins, 2003). This paper focused on further uncovering important theoretical and practical issues associated with social goal theory (i.e., factor structure, speciWcity of item wording, interplay of academic and social goals, and relationships between social goals and important academic constructs). In order to acquire a more precise understanding of how social goals function and interact with other variables, adequate measurement is necessary. We have strived to follow recommendations for best practice in construct validation in order to take one step closer to creating an adequate measure of social goals (Benson, 1998; Gerbing & Hamilton, 1996; Hurley et al., 1997; MacCallum et al., 1992; Raykov & Widaman, 1995). We look forward to the continued development and use of the SAGOS or SAGS as one approach to better our understanding of student motivation and achievement, and we

694

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

would like to point reader toward recent work by Ryan and Shim (2006) and the further development of her scales. In the words of Urdan and Maehr (1995), we hope that this and further studies will contribute to the process of “untangle (ing) the many constructs represented by the term social goals”. Appendix A. Social Achievement Goal Orientation Scale (SAGOS) The following statements concern your general attitudes about relationships. Please indicate how true each statement is of you. If you think the statement is very true of you, mark a 5. If a statement is somewhat true of you, mark a 3. If a statement is not at all true of you, mark a 1. If the statement is more or less true of you, Wnd the number between 1 and 5 that best describes you. There are no right or wrong answers. Just answer as accurately as possible. PLEASE NOTE: THE CHOICES ARE ONE TO FIVE

1 Not at all true of me

2

3 Somewhat true of me

4

5 Very true of me

//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22.

It is important to me to have friends who really understand me. It is important to me to have friends who truly care about me. My goal in most social situations is to impress others. It is important to me that I avoid looking foolish. My goal is to avoid doing things that would cause others to make fun of me. It is important to me to be seen as having a lot of friends. It is important to me to work on improving the quality of my relationships with my friends. It is important to me that I feel that I have friends I enjoy spending time with. I would be successful if I could avoid being socially awkward. I want to be friends with “popular” people. It is important to me to have “cool” friends. In social situations, I feel successful if I manage to avoid having others think I am a dork. I want to have friends who are interested in me. I like friendships that challenge me to learn new things about myself. I feel successful when I impress others with my personality or social skills. In social situations I am often concerned about the possibility that others will think I am a loser. I try not to goof up when I am out with people. I want to be seen as important by other people. I feel successful when I learn something new about myself and how I relate to other people. It is important to me that others think of me as popular. I am often concerned that others won’t like me. I would be successful if I had friends who accepted me for who I am.

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

695

Appendix B. Table of means and standard deviations for individual items on the SAGOS and SAGS

Study 1

Study 2

Freshman sample 1 (n D 1368)

Freshman sample 2 (n D 698)

Freshman sample 3 (n D 654)

Upper classman (n D 1880) sample (n D 610)

SAGOS (responses on a scale of 1–5)

Study 3 (n D 610)

SAGS (responses on a scale of 1–7)

Item Mean

SD

Mean

SD

Mean

SD

Mean

SD

Mean SD Mean SD

Mastery subscale 4.54 1a 2a 4.74 4.37 7a 4.71 8a 13a 4.47 14 4.06 19 3.95 22 4.00

0.75 0.61 0.81 0.63 0.79 0.99 1.04 1.30

4.53 4.75 4.39 4.71 4.52 4.08 3.99 4.06

0.81 0.67 0.82 0.62 0.76 0.96 1.03 1.26

4.51 4.75 4.33 4.71 4.52 4.09 3.92 4.00

0.75 0.57 0.87 0.62 0.75 1.01 1.06 1.30

4.46 4.62 4.25 4.56 4.42 4.19 3.98 3.74

.88 .81 .91 .78 .83 .94 1.09 1.39

6.27 6.54 5.93 6.73 6.20 5.82 5.52 6.69

1.06 .848 1.17 .646 1.01 1.30 1.44 .772

6.08 6.50 5.87 6.64 6.10 5.60 5.38 6.57

1.03 .835 1.04 .674 .959 1.33 1.32 .768

Performance-approach subscale 2.35 0.97 2.33 3a 2.36 1.10 2.36 6a 10a 2.13 1.03 2.15 11 2.24 1.09 2.31 15 3.46 1.17 3.48 18 3.34 1.13 3.34 1.01 2.10 20a 2.10

0.97 1.10 1.07 1.10 1.15 1.14 1.05

2.43 2.33 2.08 2.24 3.54 3.48 2.12

0.92 1.07 1.01 1.06 1.19 1.14 1.03

2.49 2.33 2.04 2.23 3.46 3.49 2.15

.99 1.09 1.04 1.11 1.13 1.12 1.08

3.34 3.56 2.57 3.10 4.06 3.68 2.96

1.48 1.66 1.49 1.73 1.66 1.82 1.65

3.61 3.42 2.61 3.06 4.31 4.14 3.07

1.29 1.51 1.40 1.58 1.49 1.57 1.54

Performance-avoidance subscale 4 3.01 1.12 3.00 5a 2.80 1.17 2.73 9 2.46 1.23 2.44 12 2.23 1.08 2.22 1.19 2.20 16a 2.22 1.12 2.49 17a 2.50 1.27 2.68 21a 2.65

1.12 1.15 1.21 1.13 1.12 1.10 1.27

3.00 2.74 2.40 2.23 2.20 2.51 2.64

1.15 1.13 1.23 1.09 1.08 1.12 1.26

2.91 2.58 2.35 2.06 2.02 2.39 2.44

1.15 1.10 1.13 1.00 .993 1.09 1.14

4.31 4.09 4.32 3.15 3.18 3.28 4.41

1.65 1.69 1.74 1.67 1.66 1.67 1.76

4.27 4.01 4.12 3.04 3.15 3.19 4.18

1.47 1.47 1.57 1.48 1.47 1.43 1.67

a

Items included on both the full and reduced scales.

Appendix C. Social Achievement Goal Scale (SAGS) The following statements concern your goals for your social relationships this semester. Please indicate how true each statement is of you. If you think the statement is very true of you, mark a “7”. If the statement is not at all true of you, mark a “1.” If the statement is more or less true of you, Wnd a number between “7” and “1” that best describes you. There are no right or wrong answers, just answer as honestly as possible. 1. 2. 3. 4.

My goal is to have friends who really understand me. My goal is to have friends who truly care about me. My goal in most social situations is to impress others. My goal is to avoid looking foolish in social situations.

696

5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22.

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

My goal is to avoid doing things that would cause others to make fun of me. My goal is to be seen as having a lot of friends. My goal is to work on improving the quality of my relationships with my friends. My goal is to have friends I enjoy spending time with. My goal in most social situations is to try to avoid appearing socially awkward. My goal is to be friends with “popular” people. My goal is to have “cool” friends. My goal is to avoid having others think I am a dork. My goal is to have friends who are interested in me. My goal is to have friendships that challenge me to learn new things about myself. My goal is to impress others with my personality or social skills. My goal is to try to prevent others from thinking I am a loser. My goal is to try not to goof up when I am out with people. My goal is to be seen as “important” by other people. My goal is to understand how to better relate to others. My goal is to be thought of as popular by others. My goal is to avoid doing things that would cause others not to like me. My goal is to have friends who accept me for who I am.

Appendix D. Rank-ordering task The following six paragraphs describe three diVerent approaches to social and academic achievement goals. Think about this current semester and then rank order the descriptions in terms of their importance to you, with 1 being most important and 6 being least important. • My main focus is on avoiding negative judgments from others. My goal is to avoid being perceived by others as socially awkward. • My main focus is on developing positive, supportive relationships. My goal is to form, maintain, and develop friendships for the insights and inherent positive qualities that friendships provide. • My main focus is demonstrating my skills and competence relative to others. In other words, I feel most successful when I am able to demonstrate that I am more competent than others. • My main focus is on gaining positive judgments from others. My goal is to receive positive feedback from others, gain social prestige, and have a good reputation (i.e., being seen as important or “popular”). • My main focus is developing my skills and competence. In other words, I feel most successful when I am able to acquire new skills and improve my competence. • My main focus is to avoid demonstrating that I lack skills and competence relative to others. That is, I feel most successful when I am able to avoid negative judgments from others about my skills and competence. References Anderman, E. M., & Anderman, L. (1999). Social predictors of changes in students’ achievement goal orientations. Contemporary Educational Psychology, 25, 21–37.

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

697

Anderman, E. M., Austin, C. C., & Johnson, D. M. (2002). The development of goal orientation. In A. WigWeld & J. S. Eccles (Eds.), Development of achievement motivation (pp. 197–220). New York: Academic Press. Barron, K. E., & Harackiewicz, J. M. (2000). Achievement goals and optimal motivation: A multiple goals approach. In C. Sansone & J. M. Harackiewicz (Eds.), Intrinsic and extrinsic motivation: The search for optimal motivation and performance (pp. 229–254). San Diego, CA: Academic Press. Benson, J. (1998). Developing a strong program of construct validation: a test anxiety example. Educational Measurement: Issues and Practice, 17(1), 10–17. Benson, J., & Nassar, F. (1998). On the use of factor analysis as a research tool. Journal of Vocational Education Research, 23, 13–33. Bentler, P. M., & Wu, E. J. C. (2003). EQS for Windows User’s Guide. Encino, CA: Multivariate Software, Inc. Brophy, J. (2004). Motivating students to learn (2nd ed.). Mahwah, NJ: Lawrence Erlbaum. Covington, M. (2000). Goal theory, motivation, and school achievement: an integrative review. Annual Review of Psychology, 51, 171–200. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281– 302. Deci, E., & Ryan, R. (2000). The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11, 227–268. DeShon, R. P. (1998). A cautionary note on measurement error corrections in structural equation models. Psychological Methods, 3, 412–423. Dowson, M., & McInerney, D. M. (2001). Psychological parameters of students’ social and work avoidance goals: A qualitative investigation. Journal of Educational Psychology, 93, 35–42. Dweck, C. S. (1999). Self theories: Their role in motivation, personality and development. Philadelphia: Psychology Press. Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95, 256–273. Elliot, A. J. (2005). A conceptual history of the achievement goal construct. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation. New York: Guilford. Elliot, A. J., & Church, M. (1997). A hierarchical model of approach and avoidance achievement motivation. Journal of Personality and Social Psychology, 72, 218–232. Elliot, A. J., & Dweck, C. S. (2005). Competence as the core of achievement motivation. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation. New York: Guilford. Elliot, A. J., & McGregor, H. A. (2001). A 2 £ 2 achievement goal framework. Journal of Personality and Social Psychology, 80, 501–519. Elliot, A. J., & Thrash, T. M. (2001). Achievement goals and the hierarchical model of achievement motivation. Educational Psychology Review, 13, 139–156. Erdley, C. A., Cain, K. M., Loomis, C. C., Dumas-Hines, F., & Dweck, C. S. (1997). Relations among children’s social goals, implicit personality theories, and responses to social failure. Developmental Psychology, 33, 263– 272. Finney, S. J., Pieper, S. L., & Barron, K. E. (2004). Examining the psychometric properties of the achievement goal questionnaire in a general academic context. Educational and Psychological Measurement, 64, 365–382. Ford, M. E., & Nichols, C. W. (1987). A taxonomy of human goals and some possible applications. In M. E. Ford & D. H. Ford (Eds.), Humans as self-constructing living systems: Putting the framework to work. Hillsdale, NJ: Lawrence Erlbaum Associates. Gable, S. L. (2006). Approach and avoidance social motives and goals. Journal of Personality and Social Psychology, 74, 175–222. Gerbing, D. W., & Hamilton, J. G. (1996). Viability of exploratory factor analysis as a precursor to conWrmatory factor analysis. Structural Equation Modeling, 3, 62–72. Hopkins, N., & Ryan, A.M. (2000). Expanding achievement goal theory into the social domain. Poster presented at the annual meeting of the American Educational Research Association, New Orleans, LA. Hurley, A. E., Scandura, T. A., Schriesheim, C. A., Brannick, M. T., Seers, A., Vandenberg, R. J., et al. (1997). Exploratory and conWrmatory factor analysis: guidelines, issues, and alternatives. Journal of Organizational Behavior, 18, 667–683. Jöreskog, K. G. (1993). Testing structural equation models. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 294–316). Newbury Park: Sage. Jöreskog, K., & Sörbom, D. (1993). LISREL 8: Structural equation modeling with the SIMPLIS command language. Chicago: ScientiWc Software International.

698

S. Jeanne Horst et al. / Contemporary Educational Psychology 32 (2007) 667–698

Leary, M. R. (1983). A brief version of the fear of negative evaluation scale. Personality and Social Psychology Bulletin, 9, 371–375. MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modiWcations in covariance structure analysis: the problem of capitalization on chance. Psychological Bulletin, 111, 490–504. McClelland, D. C. (1985). How motives, skills, and values determine what people do. American Psychologist, 40, 812–825. Monfries, M. M., & Kafer, N. F. (1994). Private self-consciousness and fear of negative evaluation. The Journal of Psychology, 128, 447–454. Patrick, H., Anderman, L., & Ryan, A. M. (2002). Social motivation and the classroom social environment. In C. Midgley (Ed.), Goals, goal structures, and patterns of adaptive learning (pp. 85–108). Mahwah, NJ: Lawrence Erlbaum. Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95, 667–686. Pintrich, P., & Schunk, D. (2002).Chapter Wve: The role of goals and goal orientation. Motivation in education: Theory, research and applications (2nd ed.) Upper Saddle River, NJ: Merrill. Raykov, T., & Marcoulides, G. A. (2000). A Wrst course in structural equation modeling. Mahwah, NJ: Lawrence Erlbaum. Raykov, T., & Widaman, K. F. (1995). Issues in applied structural equation modeling research. Structural Equation Modeling, 24, 289–318. Reis, H. T., Sheldon, K. N., Gable, S. L., Roscoe, J., & Ryan, R. M. (2000). Daily well-being: the role of autonomy, competence, and relatedness. Personality and Social Psychology Bulletin, 26, 419–435. Ryan, A. M., & Hopkins, N. B. (2003). Achievement goals in the social domain. University of Illinois, UrbanaChampaign, unpublished manuscript. Ryan, A. M., & Shim, S. O. (2006). Social achievement goals: The nature and consequences of diVerent orientations toward social competence. Personality and social Psychology Bulletin, 32, 1246–1263. RyV, C. D. (1989). Happiness is everything, or is it? Explorations on the meaning of psychological well-being. Journal of Personality and Social Psychology, 57, 1069–1081. Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors on covariance structure analysis. In A. von Eye & C. C. Clogg (Eds.), Latent variables analysis (pp. 399–419). Thousand Oaks, CA: Sage. Urdan, T. C. (1997). Examining the relations among early adolescent students’ goals and friends’ orientation toward eVort and achievement in school. Contemporary Educational Psychology, 22, 165–191. Urdan, T. C., & Maehr, M. L. (1995). Beyond a two-goal theory of motivation and achievement: a case for social goals. Review of Educational Research, 65, 213–243. Watson, D., & Friend, R. (1969). Measurement of social-evaluative anxiety. Journal of Consulting and Clinical Psychology, 33, 448–457. Wentzel, K. R. (1989). Adolescent classroom goals, standards for performance and academic achievement: An interactionist perspective. Journal of Educational Psychology, 81, 131–142. Wentzel, K. R. (1999). Social-motivational processes and interpersonal relationships: Implications for understanding motivation at school. Journal of Educational Psychology, 91, 76–97. Wentzel, K. R. (2000). What is it that I’m trying to achieve? Classroom goals from a content perspective. Contemporary Educational Psychology, 25, 105–115. Wentzel, K. R. (2002). The contribution of social goal setting to children’s school adjustment. In A. WigWeld & J. S. Eccles (Eds.), Development of achievement motivation (pp. 221–246). New York: Academic Press. Wentzel, K. R. (2005). Peer relationships, motivation, and academic performance at school. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation. New York: Guilford. Yu, C., & Muthén, B. (2002). Evaluation of model Wt indices for latent variable models with categorical and continuous outcomes. Paper presented at the annual meeting of the American Educational Research Association, New Orleans.