Testing a model of school learning: Direct and indirect effects on academic achievement

CONTEMPORARY

Testing

EDUCATIONAL

16, 2844 (1991)

PSYCHOLOGY

a Model of School Learning: Direct and Indirect Effects on Academic Achievement VALERIE A.COOL of Imu

The University

AND

TIMOTHY Virginiu

Polytechnic

Z. KEITH

Institute

and State

University

Continued concern about the quality of American education highlights the need for a better understanding of the influences on school learning. Theories of school learning are consistent in pointing to the importance of variables such as ability, time (in particular, homework), quality of instruction, motivation, and academic coursework in their influence on learning. However, most investigators have not included all these variables simultaneously in investigations of school learning, and few have focused on indirect as well as direct effects. The purpose of the present paper was to determine the extent of the direct and indirect influence of each of these variables on academic achievement. Path analysis was conducted with data from the High School and Beyond study to examine the effects of these variables on the academic achievement of high school seniors. Results suggest that both ability and academic coursework have strong direct effects on achievement; motivation and quality of instruction were found to have meaningful indirect or total effects on achievement, but negligible direct effects. Surprisingly, homework had inconsistent direct effects. The results offer support for these variables as important influences on school learning, a finding which further supports their inclusion in prominent theories of school learning. 0 1991 Academic Pres. Inc.

Continued concern about the quality of American education highlights the need to understand better the influences on school learning. Theories of such learning are fairly consistent in pointing to a number of important variables, many of which are potentially manipulable. Walberg’s theory of educational productivity (1981), for example, focuses on the imporPortions of this work were completed while the second author was Senior Research Fellow, Office of Educational Research and Improvement (OERI), U.S. Department of Education. We are grateful to OERI, the University of Iowa, and Virginia Tech for their support. We are responsible for any errors and for the opinions expressed. Correspondence should be sent to Timothy Z. Keith, 206 UCOB, Virginia Tech, Blacksburg, VA 24061-0302 (Bitnet ID: TZKEITH at VTVMI). 28 0361-476X193 Copyright A,, rider

$3.00

0 1991 by Academic Press, Inc. nf rPnmAl,rti,m I” il”ll fr.r.” ve.‘.-u-A

SCHOOL

LEARNING

29

tance of student aptitude (ability, age, and motivation) and environmental variables (home, classroom, peers, and television viewing). Student achievement is increased when each of these factors is at an optimal level. In Carroll’s model of school learning (1963, 1989), student aptitude, ability to understand, and quality of instruction are variables that affect time needed to learn. Also considered is the student’s opportunity to learn, which combines with perseverance to influence time spent learning. These variables interact with each other so that, for instance, the higher the quality of instruction, the lower the amount of understanding required. The degree of school learning is a function, then, of the time actually spent learning and the time needed to learn. Similarly, Wiley and Harnischfeger (1974), in their examination of instructional time and learning, propose that increases in school achievement arise from increases in total time allocated to learning, the portion of allocated time actually allowed for learning, the portion of the allowed time actively devoted to learning, and a reduction in time needed to learn. The theoretical support for quantity of coursework, quality of instruction, motivation, time, and aptitude as important influences on learning thus appears consistent (for a comparison of such theories, see Walberg, 1986). Research has also supported the importance of these and related variables in their impact on learning (cf. U.S. Department of Education, 1986; Walberg, 1986). The influence of quality of instruction on achievement, for example, has been studied using large representative samples of high school students. Some research has shown only minimal effects, but small positive effects are the norm (Walberg, 1986). Motivation, or willingness to persevere on learning tasks, has been identified as a variable affecting school achievement for elementary, junior high (Gottfried, 1985), and high school students (Walberg, 1986). Specific content areas have been studied, including science achievement of high school students and math achievement of elementary and junior high students (Gottfried, 198.5; Walberg, 1986). In addition, motivation has been demonstrated to have an important influence in investigations of minority (Gottfried, 1985) and foreign students (Uguroglu & Walberg, 1986). Quantity of academic coursework has been demonstrated to influence achievement in studies with large representative samples of high school students (Walberg & Shanahan, 1983) and with minority students in public and Catholic schools (Keith & Page, 1985a) using nonexperimental methods. Others, using different data and different conceptualizations of the same constructs, have reported similar results (Alexander & Pallas, 1984). Time variables have also been suggested as affecting school achieve-

30

COOL AND KEITH

ment. In particular, homework time has been shown to have a positive impact on achievement. The influence of homework on achievement has been shown for elementary (Paschal, Weinstein, & Walberg, 1984), high school (Keith & Page, 1985b; Keith, Reimers, Fehrmann, Pottebaum, & Aubey, 1986), and college students (Polachek, Kneisner, & Harwood, 1978). Additionally, time spent doing homework seems to improve minority students’ (Keith & Page, 1985b), and gifted students’ (Stanley, 1980) achievement. Thus, the variables of quality of instruction, motivation, academic coursework, and homework time are supported as important influences on school learning by research evidence, and they, or closely related variables, appear consistently in theories of school learning (Walberg, 1986). But few researchers (with the notable exception of Walberg and his colleagues) have examined the effects of all variables simultaneously where each variable can compete with each other. Yet this simultaneous testing of variables is important, especially in nonexperimental research, because if important common causes of the variables of interest are neglected, their effects may be spuriously overestimated (Keith, 1988). For instance, in more complex analyses the effect of homework on achievement appears to diminish (e.g., Walberg & Shanahan, 1983), perhaps because the influence of variables such as motivation, quality of instruction, and quantity of academic coursework are statistically controlled. Even fewer researchers have investigated the indirecf effects of these variables on achievement, although indirect effects are important in the understanding of influences on achievement. For example, motivation may not affect achievement directly, but rather may affect achievement indirectly by leading students to take more courses and spend more time studying. The purpose of the present research was to determine the extent of the direct and indirect influence of quality of instruction, motivation, quantity of academic coursework, and homework on academic achievement while controlling for the important background variables of ethnicity, family background, gender, and intellectual ability or aptitude. A large, contemporary data set and techniques appropriate for analysis of nonexperimental data were used. METHOD Subjects Subjects for the present study were 28,051 high school seniors from the first wave (1980) of the National Center for Education Statistics’ (NCES) High School and Beyond Longitudinal Study (HSB). The HSB data set contains extensive information on 58,728 seniors and sophomores in high school. The HSB sample was a two-stage stratified probability

SCHOOL

LEARNING

31

sample of 1016 high schools, with schools selected with a probability proportional to their estimated enrollment. Data for up to 36 seniors and 36 sophomores, selected at random, were obtained from each school. The present sample consisted of all seniors in the HSB sample (N = 28,051).

Variables The following variables were either selected or developed using other HSB variables. 1. Erhnicity. Ethnicity was a dichotomous variable, coded 1 for white, nonhispanic students and 0 for students from all other ethnic groups (see Pedhazur, 1982, chap. 9, for a discussion of the use of such dichotomous or dummy variables in regression and related methods). Although other methods of categorization of ethnicity are certainly possible, we chose this categorization because it is common, and it contrasts minority with majority students. 2. Family background. Family background was a composite socioeconomic status @ES) variable computed for HSB for each student. Responses to questions concerning father’s occupational status, mother’s and father’s educational attainments, family income, and a composite of possessions in the home were converted to z scores and averaged. The range of family background was - 3.068 to 2.202, and the variable has an estimated validity of at least 80 (estimated by correlating student responses with parent responses; NCES, 1984). 3. Gender. Gender was a dummy variable; males were coded 0 and females coded 1. 4. Ability. Ability, a single-factor score, was derived from factor analysis of six HSB verbal and nonverbal ability (relatively school-free) tests: Vocabulary 1 and II, Mosaic Comparisons 1 and II, Picture Number, and Visualization in Three Dimensions (cf. Keith & Page, 198Sa). Factor score coefficients from the unrotated first principal factor were used to create an ability factor score for each subject. The range of the ability variable was 28.32 to 97.28, and its estimated reliability was .92 using the formula provided by Guilford (1954, p. 393) and the reliability estimates for the individual tests (Heyns & Hilton, 1982). 5. Qualify ofinstruction. Quality of instruction and schooling was an average of student’s ratings of their school on the following topics: (a) quality of academic instruction; (b) reputation in the community; and (c) teacher interest in students. Responses to each item ranged from poor (coded 1) to excellent (coded 4). Each item was converted to a z score and averaged. It should be recognized that the quality variable reflects srudents’ ratings of quality, and that it reflects their perceptions of overall school quality in addition to quality of instruction. The range of the quality variable was - 2.12 to 1.63, and its estimated internal consistency reliability was .84 (corrected by the Spearman-Brown formula). 6. Academic motivation. The motivation variable focused on educational motivation and aspiration. It was a composite of student’s responses to three questions: (a) “I am interested in school” (true or false); (b) “I like to work hard in school” (true or false); and(c) “Do you plan to go to college at some time in the future?” For the last question, responses were recoded 1 for “No,” 2 for “Don’t know,” 3 for “After a longer’period out of school,” 4 for “After staying out 1 year,” and 5 for “Next year.” Each item was converted to a z score and the scores were averaged to create the motivation variable (range = - 1.67 to .85); its estimated corrected reliability was .72. 7. Quantity of academic coursework was a composite of a number of courses in the following subjects: algebra I and II, geometry, trigonometry, the calculus. physics, chemistry, and advanced English. Each course variable (coded 0 for “Not taken” and 1 for “Have taken”) was standardized (z scores) and averaged (range = - 1.49 to 2.03) to create an equally weighted composite. The estimated corrected reliability of the coursework vari-

32

COOL AND KEITH

able was .89. The HSB data include other items that also reflect high school coursework. The items used here were chosen for inclusion because they are clearly academic in focus. 8. Homework. The homework variable consisted of student response to a question concerning the average amount of time spent on homework a week. Responses ranged from “No homework is ever assigned” or “I have homework but I don’t do it” (both coded 0) to “More than IO hours a week” (coded 5). Intermediate responses were less than I, l-3,3-5, and 5-10 h. Each interval of this variable represents an increased amount of time. 9. Achievement. Achievement was an average of the HSB Reading, Mathematics I and Mathematics II standardized tests (T scores), with equal weighting for reading and math (range: 26.96 to 70.60; reliability = .88). The achievement and ability tests. designed for HSB by the Educational Testing Service, are short (ranging from 3 to I5 min each), but also appear to be reliable and valid (cf. Heyns & Hilton, 1982; Keith & Page, 1985a). Although the coursework, quality, and motivation variables were each simple composites, the composition of the composites was based on factor analysis of these and similar items. Briefly, items thought to measure each of the constructs of interest were factor analyzed, and those items that loaded substantially on factors most clearly reflecting the constructs were then used to create the composites. Similarly, the family background variable was created based on factor analyses of the earlier national longitudinal study. In all cases, the equal weighting of the items in the composites appeared warranted based on the factor analytic results.

Analysis The direct and indirect influences of quality of instruction, motivation, academic coursework, and homework on seniors’ achievement were estimated using path analysis, while also controlling for other relevant background effects. Research, formal and informal theory, logic, and time precedence lead the researcher in determining which variables to include in the path model and in determining the presumed direction of causation. Figure I presents the model used in this research: a simple, recursive path model in which the arrows represent a “weak causal ordering.” As such, a direct causal relation is not asserted, but rather the arrows imply that ifthe two variables are causally related, the influence is in the direction of the arrow rather than the reverse. Paths in such models may be estimated by the p weights from standard multiple regression analysis (Kenny, 1979, chap. 4; see also Pedhazur, 1982). The curves in the model represent unanalyzed correlations among exogenous variables (presumed causes). We used pairwise deletion of missing data in all analyses. The justification for the inclusion of quality of instruction, motivation, and academic coursework has already been discussed. Like them, intellectual ability or aptitude is consistently featured in theories of school learning. Our primary interest, however, is with more easily manipulated variables such as quality or quantity of coursework. The variables ethnicity, family background, and gender are included in the model because they are commonly controlled background variables and because previous research has shown them to be important to consider when studying the effects of many of the key variables on achievement (cf. Keith et ul., 1986). One reason for using path analysis as the method of analysis was that it. unlike many other regression approaches, focuses on both direct and indirect effects; each variable of interest was expected to have both direct and indirect effects on achievement. For example, quality of instruction was expected to have both direct and indirect (through motivation, academic coursework. and homework) influences on achievement. Accordingly, paths were drawn from quality of instruction to achievement, motivation, academic coursework, and home-

SCHOOL

LEARNING

33

:NT

FIG. 1. Effects of ability, quality of instruction, motivation. quantity of coursework, and homework on seniors’ academic achievement.

work. Similarly, the effects of motivation could be direct to achievement or indirect through academic coursework or homework. Academic coursework was hypothesized to have both a direct effect and an indirect effect through homework.’

RESULTS AND DISCUSSION

Table 1 contains the intercorrelations among the variables, their means, and standard deviations (in the units used in this analysis). The path analytic model which explains achievement as a function of homework, ’ The terms direct and indirect effects may need clarification. One could easily insert additional intervening variables in the model shown in Fig. I (e.g., parental involvement or peer influences) and some of the influence we categorize as direct effects might then be transmitted through these additional variables (indirect effects). Thus, the terms “direct” and “indirect” effects are model dependent and really mean direct (or indirect) effects within the confines of the model presented. An advantage of path analysis over many other regression approaches is that with it one can easily study these indirect effects. Most regression approaches concentrate on what we here call direct effects. The addition of other intervening variables to the regression equation change those estimates and thus our interpretation of the research. The addition of intervening variables would change our estimates of direct and indirect effects in a path analysis as well, but the estimates of total effects (direct + indirect) would remain the same, and thus would not affect the conclusions of the study.

34

COOL

VARIABLE

1. Ethnicity 2. Family background 3. Gender 4. Ability 5. Quality 6. Motivation 7. Coursework 8. Homework 9. Achievement M SD Note.

N = 28,051;

AND

KEITH

TABLE

1

MEANS,

STANDARD

DEVIATIONS,

AND

INTERCORRELATIONS

1

2

3

4

5

6

I

8

9

1.000 - ,090 ,373 ,180 ,131 ,381 ,184 .420 - ,094 ,761

1.000 .026 - ,024 .145 - .089 .158 - ,080 1.522 ,500

1.000 ,208 .I83 ,500 ,246 .72 1 65.696 9.429

l.ooO ,264 ,228 .248 .250 - ,013 .810

1.000 .336 ,434 ,232 ,026 ,737

1.000 .372 .639 ,017 ,670

1.000 ,295 2.457 1.340

1.000 49.531 8.677

1.000 .33 1 - ,033 ,342 .075 -.132 ,117 ,000 ,369 .685 ,464 minimum

pairwise

N = 21,008.

academic coursework, motivation, quality of instruction, aptitude (ability), and other background influences, is displayed in Fig. 1. Similar to previous studies of this type, the strongest direct influence on achievement was from intellectual ability (path = .463; cf. Keith et al., 1986). The path from coursework to achievement was the second strongest (.333), suggesting that as the rigor of academic coursework is increased, so is the level of achievement. Consistent with other studies (Keith et af., 1986), the path from gender to achievement (- .063), with its negative sign, suggests that males achieve at a higher level than females once other influences are controlled (males were coded 0 and females coded 1 on the gender variable). The path from ethnicity to achievement (.156) suggests that white students achieve at a higher level than do minority students even when the other variables in the model are statistically controlled. Although all of remaining paths to achievement were statistically significant (because of our huge sample size), they were below the level chosen as representing a meaningful effect (a path of .05 or greater, a common criterion in large-sample path analyses [Pedhazur, 1982, chap. 151). In our discussion we focus on whether paths reach this criterion of meaningfulness, and also compare and make judgments about the size of meaningful paths. The most surprising of the nonmeaningful paths was that from homework (.035). Other authors have found paths from homework to achievement for the HSB seniors as high as .141 (Keith et al., 1986) using test scores and .192 (Keith & Page, 1985b) using grades as the achievement measure, but without controls for coursework, motivation, and quality of instruction. Our results suggest that when these variables (primarily coursework) are included, little direct effect on achievement is seen. Similarly, the direct effects of motivation (.034) and quality of in-

35

SCHOOL LEARNING

struction (.039) on achievement no meaningful direct effects. Effects on Homework,

were below the .05 criterion,

Coursework,

Motivation,

suggesting

and Quality

The strongest effects on homework were from motivation (.293), academic coursework (.225), gender (. 140), and quality of instruction (. 107), suggesting that those reporting the most homework are students who are also the most motivated, who take the most rigorous courses, and who rate the quality of instruction in their school at higher levels. The path from gender to homework was positive (. 140), indicating females report more homework than males. The direct effect from ability to homework was not meaningful, but ability does seem to affect homework indirectly through coursework and motivation. For academic coursework, intellectual ability was the strongest influence (.395). Motivation (.232) also had a large impact on coursework, suggesting those who are most motivated take higher level academic courses. Another meaningful path was from family background to coursework (.205); more rigorous coursework was reported by those from more advantaged backgrounds. The strong path to motivation from quality of instruction (.229) suggests that higher quality of instruction leads to higher motivation in these seniors. Interestingly, the negative path from ethnicity to motivation (- .242) suggests that, within ability and background levels, minority students report higher academic motivation than do white seniors. Other meaningful paths were from ability (. 170) and gender (. 150). Ability and family background both had substantial influences on quality of instruction (.172 and .123, respectively), suggesting that higher ability and higher SES students are exposed to higher quality of instruction and schooling than are other students (or that they rate their instruction/schooling more highly).* ’ Although it is not the focus of this manuscript, several reviewers requested comment concerning the path from ethnicity to ability (246). This path suggests that controlling for family background (SES) explains part, but not all, of the correlation between ethnic status and ability (if SES were not included in the model, the path would have been close to .342, the correlation between the two variables). This finding of a residual correlation between ethnic status and ability after controlling for SES is common (e.g., Jensen & Reynolds, 1982), but is often misinterpreted. The correlation and path simply show a difference between white and other ethnic groups. It says nothing about heredity or “innate” ability; the model controls for only one of many possible environmental influences. It also says nothing about bias in the tests, which would require more extensive analysis (cf. Anastasi, 1988, chap. 7). In sum, the model is not designed to explore the existence or nature of group differences on either of the dichotomous exogenous variables (ethnicity or gender).

36

COOL AND KEITH

Indirect

Effects

These results suggest that there is little direct effect for homework, motivation, and quality of instruction on achievement but that there are strong direct effects from academic coursework and ability. However, motivation does seem to affect coursework, which in turn influences achievement, thus suggesting that there may well be indirect effects of motivation on achievement. And while these indirect effects are lost in standard multiple regression analysis, they are calculable in path analysis by multiplying and summing paths. The indirect effect of motivation on achievement through coursework is .232 x .333 = .077, for example, and other indirect effects (e.g., through homework) could be added to this value. Table 2 contains additional information about direct and indirect effects for the variables of greatest interest: academic coursework, motivation, quality of instruction, and ability. By adding the direct and indirect effects, the total effect of one variable on another can be estimated. The total effect of coursework on achievement is primarily a direct effect, as is the effect of homework on achievement (since there are no intervening variables between homework and achievement in our model). Of greater interest are the indirect and total effects of quality and motivation. Although neither variable was an important direct influence on achievement (and thus would not show up as important in an ordinary multiple regression), motivation had a meaningful indirect effect on achievement (.089), and both motivation (. 123) and quality (.088) had meaningful total effects. Interactions The possibility of interactions among the variables in their effects on achievement has also been suggested by previous research. Specifically,

TABLE INDIRECT

2

AND TOTAL EFFECTS OF ABILITY,QUALITY, COURSEWORK ON SENIORS' ACADEMIC

MOTIVATION, ACHIEVEMENT

AND ACADEMIC

Variable

Direct effect

Indirect effect

Total effect

Ability Quality Motivation Academic courses Homework

,463 ,039 ,034 ,333 .035

,172 .049 ,089 .008 -

.635 .088 .123 .341 ,035

SCHOOL

LEARNING

37

a variable was created to test the possibility of an interaction of ability with coursework (cf. Alexander & Pallas, 1984). The interaction variable was created by multiplying the two interacting variables (ability and coursework). All the variables shown in the path model were regressed, in a block, on achievement, and then the interaction variable was added to the equation, with the significance of the change in R2 providing a test of the existence of an interaction (Pedhazur, 1982). Given the huge sample size, virtually any increase in R2 would be significant. Therefore, we calculated F tests and probabilities as if the sample size was 1000. Other researchers have found that high-ability students are more affected by the rigor of coursework (Alexander & Pallas, 1984), and our finding of a small, but significant, interaction supports this contention (change in R2 = .0033, F = 9.59, p < .Ol). Further analysis suggested that achievement increased systematically with added courses for all three groups, but that slightly larger gains were shown for the high-ability students. Additional

Interaction

Analyses

The results of these analyses suggest that academic coursework is a powerful influence on seniors’ academic achievement, and that it may have differential effects on achievement depending on those seniors’ ability. For this reason, it seemed important to examine the possibility of other interactions with coursework. To do so, we split our sample into three subsamples: those taking the most academic coursework (high coursework, minimum pairwise n = 7150), those reporting the least academic coursework (low, n = 6971), and those taking a medium amount (medium, n = 6722). The model shown in Fig. 1 was analyzed separately for each of these subsamples to provide more information concerning interactions with coursework. The direct and total effects of each of the variables of interest are shown for the low, medium, and high coursework groups in Table 3. The table includes both the standardized coeflicients and the metric coefficients. Metric or unstandardized coefficients are generally used to make comparisons across groups because changes in the standardized coefficients may simply be a function of differences in variance across groups (Kenny, 1979, chap. 13). Nevertheless, we continued to use the standardized coefficients in our discussion below for two reasons. First, all of the interpretations below hold for both the standardized and the unstandardized coefficients. Second, the metric of most of the variables in our analyses is arbitrary (e.g., averages of z scores) and so the metric coefficients hold little meaning.

38 DIRECTANDTOTAL SENIORS

COOL

AND

KEITH

TABLE 3 EFFECTSOFVARIABLESOF INTERESTON TAKING DIFFERENT AMOUNTS OF ACADEMIC Amount

of academic

Ability Quality Motivation Coursework Homework

Note. Standardized listed in parentheses

coursework

Medium

Low Variable

THEACHIEVEMENTOF COURSEWORK

High

Direct

Total

Direct

Total

Direct

Total

.482 (.352) .024 (.185) .032 (.252) .I42 (4.378) .027 (.139)

.514 (.376) .038 (.295) ,056 (.443) ,144 (4.427) ,027 (.139)

,505 (.443) ,050 (.457) ,027 (.277) ,152 (5.428) ,033 (.190)

,540 (.475) ,067 (.613) ,055 (.561) .I55 (5.592) ,033 (.lW

,540 (.484) .048 (.462) .055 (.701) ,145 (3.117) ,058 (.332)

,607 (.544) .083 (.796) ,096 (1.215) ,152 (3.271) ,058 (.332)

coefficients are listed below the standardized

first; unstandardized coefficients.

(metric)

coefficients

are

The influence of coursework in each of these models is smaller because of the reduced variance in coursework for each of the subsamples; however, academic coursework was still a meaningful influence for each subgroup. One interesting finding was that as the rigor of coursework increased from low to high, there was a corresponding increase in the number of meaningful direct influences on achievement from 4 to 7. The number of meaningful total effects also increased; for low-coursework students, motivation and coursework were the only manipulable variables with meaningful effects. For high-coursework seniors, all four manipulable variables influenced achievement either directly or indirectly. Finally, for many of the paths in the models, there was a substantial increment in effect between low and high coursework, suggesting, for example, that quality of instruction has a small influence on students taking the least demanding courses (total effect = .038) but that there is an increase in its impact for students enrolled in the most demanding courses (effect = .083). Similarly, motivation had a larger effect for students with high coursework than for students with low (or medium) coursework. Another interesting finding was the change in effect of homework across the models. When only students who took the lowest amount of coursework were included, the path from homework to achievement dropped to .027, whereas with students who took the most rigorous coursework, the path size increased to meet the criterion for meaningful-

SCHOOL

LEARNING

39

ness (.O%). These results, then, suggest that homework has more influence on achievement for students taking more difftcult courses. Missing Data One other possible concern with these analyses is our decision to use pairwise instead of listwise deletion of missing data in our path analyses. Our primary reason for using pairwise deletion was that we wanted to use all available data. Nevertheless, to check the robustness of our results, the model shown in Fig. 1 was rerun using a matrix generated using listwise deletion of missing data. The results of that reanalysis were only trivially different from those shown. Furthermore, the two matrices (listwise and pairwise) were statistically indistinguishable.3 CONCLUSIONS

Theories of school learning are consistent in pointing to the importance of variables such as ability, time (in particular, homework), quality of instruction, motivation, and academic coursework in their influence on learning. However, most investigators have not included all these variables simultaneously in investigations of school learning. Furthermore, few have focused on indirect as well as direct effects. Our results suggest the importance of both of these considerations. If, for example, academic coursework were not included in our analyses, the effect of homework would have been apparently, but spuriously, much larger.4 The results of this research also illustrate the importance of focusing on both direct and indirect effects; two key variables, quality and motivation, had trivial direct effects, but their total effects were meaningful. By including a number of important variables, the present study was able to delineate more precisely both the direct and indirect influence of these important variables on academic achievement, while controlling for important background influences. 3 The two matrices were compared using a multi-group LISREL analysis (Joreskog & Sorbom, 1984, chap. 5). This procedure violates the assumption that the compared matrices are independent, but provides a reasonable comparison of their similarity. 4 This example illustrates the biggest danger of path analysis or nny orher rype ofnonexperimenral analysis (including multiple regression analysis and latent variable structural equation modeling). If an important third variable (a common cause of a presumed cause and effect of interest) is excluded from the model, the effects of the presumed cause will be spuriously increased (or, under some circumstances, decreased). For further discussion of these and other dangers of nonexperimental research see Keith (1988).

40

COOL

AND

KEITH

These results suggest that aptitude has the strongest direct effect on achievement and that coursework has the second strongest effect. Not only does academic coursework have a direct effect on achievement, but it also interacts with ability and a number of other variables analyzed so that students with higher aptitude seem more affected by the rigor of academic coursework. Motivation and quality of instruction were found to have substantial indirect effects, although their direct effects were not meaningful. For instance, motivation influenced academic coursework, which, in turn, influenced achievement. In the present study, the inconsistent effect of homework on achievement was both puzzling and surprising. One possible explanation is that the effect of homework on achievement is smaller than previously estimated; that is, the substantial positive effect found in previous research investigations-some using the same data base-was due to the exclusion of important uncontrolled variables. This diminished effect of homework in more complex models has been shown in other research (Walberg & Shanahan, 1983). On the other hand, our analyses suggest that this explanation does not hold for those seniors involved in a highly academic curriculum. Another possibility may be that the variables used to measure homework and achievement were inadequate. This criticism seems especially telling for homework. The homework variable, based on a single general question about normal homework practice, is probably an unreliable measure of true homework practice (if so, this would reduce the correlation, and therefore the path, from homework to achievement). The achievement variable, which was limited to reading and math only, may also be an inappropriate measure of achievement. It is possible, for example, that the inconsistency between the achievement criterion (focusing on basic achievement) and the academic coursework variable (focusing on advanced coursework) masked some of the influences on achievement. Additionally, the interaction between homework and coursework may further cloud the effects of homework on academic achievement. Obviously, further research is needed in which the achievement variable includes test scores from subject areas in addition to reading and math and in which there are multiple or more powerful homework measures. Further research might also focus on school-level influences on achievement. Although the quality of instruction variable focuses, to a certain extent, on school and classroom influences, the model is designed primarily to test the influence of individual level variables on achievement in a manner consistent with relevant theory. Future research might also profitably focus on school effects. Further research is also needed using different measures of other constructs in the model (quality, motivation, etc.). Most of the variables used

SCHOOL

LEARNING

41

in this research were based on student self-report, and therefore questions about reliability and validity are warranted. Nevertheless, there is evidence to support the overall reliability and validity of HSB items, especially factual and contemporaneous items (NCES, 1984). In addition, when possible, we have used factor-analytic-based composites rather than single items, and those composites appear to have adequate reliability for research purposes. Therefore, we believe that cautious interpretation of these variables as measuring the constructs intended is warranted. Still, the variable quality of instruction deserves special attention because we do not know the extent to which students’ responses to questions about instructional and school quality provide good measures of school quality. Thus, while the quality variable appears reliable (corrected (Y = .84), its validity is unknown. On the other hand, the magnitude of the effect found for quality is similar to that found in other analyses of national survey data (cf. Walberg, 1986). Furthermore, the current findings that quality seems to have meaningful total, but not direct, effects may help explain its inconsistent effects in previous research and are also consistent with relevant theory (e.g., Carroll, 1989). If we had examined only direct effects-a common approach in nonexperimental analyseswe would have concluded that quality had no effect on achievement. On the other hand, if we had not controlled for other important influences and background effects, quality would have appeared to have an effect on achievement, no matter what analysis approach were used. Readers may also question the overall size of the effects shown here. Although the effects of the variables of quality and motivation met the requirements of statistical significance and meaningfulness, they appear tiny in comparison to the effects of variables such as ability, and may seem especially small to those who are more familiar with smaller scale nonexperimental or experimental research. These results suggest, for example, that a standard deviation change in quality should result in only .088 SD change in achievement, other things being equal. Yet recent reviews of research on teaching suggest that this finding is fairly consistent with previous research; Brophy and Good (1986), for example, note that quality of teaching-whether measured by observations of teachers’ behavior or by surveys-has only low to moderate correlations with learning, correlations that “are not always strong enough to reach statistical significance” (p. 360). Furthermore, the effects found here are consistent with previous research that controlled adequately for other influences (Walberg, 1986). Apparent differences may result from several differences in method: (1) different metrics are used; experimental research, for example, generally focuses on significance rather than the magnitude

42

COOL

AND

KEITH

of change (by way of comparison, even the nonmeaningful direct effect for quality (.039) was significant at p < .OOOl); (2) small-scale nonexperimental research generally does not control for important background characteristics; and (3) different definitions of learning are used, some of which (e.g., teacher ratings or tests more closely tied to the curriculum) may be more easily influenced than are general achievement test scores. This final point is particularly noteworthy. Broad, general measures of achievement were used here; measures more closely attuned to the curricular content of the schools would likely show increased effect for the school-related variables in the model. Given such uncertainties, it may be prudent to consider the small, significant effects of variables, such as quality, as minimum estimates of their true effects. In any case, these results are consistent with previous research, and the effects are not trivial, although they seem so in comparison to powerful background influences. Still, additional research comparing these operationalizations of constructs such as quality and motivation with other such measures is needed. Future analysis might also profitably use covariance structures analysis (e.g., LISREL, Joreskog & Sorbom, 1984), which uses latent variables in place of the simple composites used here. Such analyses would reduce concerns about the reliability of the measures used and would allow increased confidence that those measures provide valid indicators of the constructs of interest.’ Despite such cautions, and despite the puzzling homework findings, this research offers support for these variables as important influences on school learning (as measured by academic achievement), a finding that further supports their inclusion in prominent theories of school learning. Such findings should be of interest to educators because many of the variables studied are potentially manipulable (e.g., increasing quality of instruction/schooling may lead to increases in motivation, which may in turn lead to more rigorous coursework. and greater achievement), and therefore may suggest worthwhile systems-level and individual interventions. 5 In fact, LISREL was used to estimate the model shown in Fig. 1 (primarily because the program outputs a convenient table of indirect and total effects), but simple composites were used in only the structural equation model portion of LISREL; there was no measurement model. With a just-identified model such as shown in Fig. 1, LISREL and ordinary regression results are the same. We did not use the full LISREL model because for most of the variables in our model we would have had to use items as indicators of the latent constructs. In our experience, the use of items rather than scales can be problematic with the measurement model portion of LISREL. We should note, however, that similar analyses using the HSB sophomore data have produced very similar results using both composites and latent variables as indices of the constructs.

SCHOOL LEARNING

43

REFERENCES K. L., & PALLAS, A. M. (1984). Curriculum reform and school performance: An evaluation of the “New basics.” American Journal of Education, 92, 391420. ANASTASI, A. (1988). Psychological testing (6th ed.). New York: Macmillan & Co. BROPHY, J., & GOOD, T. L. (1986). Teacher behavior and student achievement. In M. C. Wittrock (Ed.), Handbook of research on reaching (3rd ed., pp. 328-375). New York: Macmillan & Co. CARROLL, J. B. (1963). A model for school learning. Teachers College Record, 64, 723-733. CARROLL, J. B. (1989). The Carroll Model: A 25-year retrospective and prospective view. Educational Researcher, 18( 1), 26-3 1. GOTTFRIED, A. E. (1985). Academic intrinsic motivation in elementary and junior high school students. Journal of Educafional Psychology, 77, 631-645. GLJILFORD, J. P. (1954). Psychometric merhods (2nd ed.). New York: McGraw-Hill. HEYNS, B., & HILTON, T. L. (1982). The cognitive tests for High School and Beyond: An assessment. Sociology of Education, 55, 89-102. JENSEN, A. R., & REYNOLDS, C. R. (1982). Race, social class and ability patterns on the WISC-R. Personality and Individual Differences, 3, 423-438. J~RESKOG, K. G., & S~RBOM, D. (1984). LISREL VI: Analysis of linear structural relationships by the method of maximum likelihood: User’s guide. Mooresville, IN: Scientific Software. KEITH, T. Z. (1988). Path analysis: An introduction for school psychologists. School Psychology Review, 17, 343-362. KEITH, T. Z., & PAGE, E. B. (1985a). Do Catholic high schools improve minority student achievement? American Educational Research Journal, 22, 337-349. KEITH, T. Z., & PAGE, E. B. (1985b). Homework works at school: National evidence for policy changes. School Psychology Review, 14, 351-359. KEITH, T. Z., REIMERS, T. M., FEHRMANN, P. G.. POTTEBAUM, S. M., & AUBEY, L. W. (1986). Parental involvement, homework, and TV time: Direct and indirect effects on high school achievement. Journal of Educational Psychology, 78, 373-380. KENNY, D. A. (1979). Correlation and causaliry. New York: Wiley. National Center for Education Statistics (NCES) (1984). Quality ofresponses ofhigh school students to questionnaire items. Washington, DC: U.S. Govt. Printing Otfice. PASCHAL, R. A., WEINSTEIN, T., & WALBERG, H. J. (1984). The effects of homework on learning: A quantitative synthesis. Journal of Educational Research, 78, 97-104. PEDHAZUR, E. J. (1982). Multiple regression in behavioral research: Prediction and explanation (2nd ed.). New York: Holt, Rinehart & Winston. POLACHEK, S. W., KNEISNER, T. J., & HARWOOD, H. J. (1978). Educational production functions. Journal of Educational Statistics, 3, 209-231. STANLEY, J. C. (1980). Manipulate important educational variables. Educational Psychologist, 19, 164-171. UGUROGLU, M. E., & WALBERG, H. J. (1986). Predicting achievement and motivation, Journal of Research and Development in Education, 19(3), I-12. U.S. Department of Education (1986). What works: Research about teaching and Iearning. Washington, DC: U.S. Department of Education. ALEXANDER,

44

COOL

AND

KEITH

WALBERG, H. J. (1981). A psychological theory of educational productivity. In F. H. Farley & N. Gordon (Eds.), Psychology and education. Berkeley, CA: McCutchan. WALBERG, H. J. (1986). Synthesis of research on teaching. In M. C. Wittrock (Ed.), Handbook ofresearch on teaching (3rd ed., pp. 214-229). New York: Macmillan & Co. WALBERG, H. J., & SHANAHAN, T. (1983). High school effects on individual students. Educational

Researcher,

12(7),

4-9.

WILEY, D. E., & HARNISCHFEGER, A. (1974). Explosion of a myth: Quantity of schooling and exposure to instruction, major educational vehicles. Educational Researcher, 3(4), 7-12.