The problems with current analytical procedures

CHAPTER

THE PROBLEMS

3

WITH CURRENT PROCEDURES

ANALYTICAL

K. C. CHEUNG and J. P. KEEVES

Introduction By the early 1980s three main analytical procedures had been developed for the examination of path models or networks of influence in educational research such as the one presented in Figure 2.1 in the previous chapter. These procedures are: 1. Ordinary least squares path analysis (OLS) (Peaker, 1971; Keeves, 1972). 2. Partial least squares path analysis (PLS) (Wold, 1982; Noonan & Weld, 1985; Sellin. 1986). 3. Linear structural relations analyses (LISREL) (Munck, 1979; Joreskog & Sorbom, 1989). It should be noted that all three analytical procedures had been developed in such a way that they could be employed in the analysis of data collected in large cross-sectional surveys of educational achievement. Nevertheless, while each of these procedures has certain advantages in the analysis of data collected in educational research, their use has helped to expose certain problems which arise in the analysis of cross-sectional survey data in the field of education. The basic problem which confronts analysts of large bodies of data is that of selecting or combining together data on many variables which are considered to be relevant within a simple and coherent model that is capable of being tested. Partial least squares analysis and linear structural relations analysis employ constructs, regarded as latent variables, for which observed or manifest variables act as multiple indicators. Observed variables can be combined together by principal components analysis, by canonical analysis, or by rosettes as carriers of regression (an idea suggested by Tukey), but such an approach requires the preselection of variables for inclusion in a model. Analyses that compare the use of these methods (see Keeves, 1986) would appear to indicate that partial least squares path analysis has more flexibility, although linear structural relations analysis provides a more rigorous testing of the model. In this sense partial least squares analysis can be regarded as exploratory and linear structural relations analyses as confirmatory. One apparent advantage that both ordinary least squares and partial least squares analysis have over linear structural relations analysis is that the two least squares 233

3-t

K.C.CHEUNGcrai.

procedures do not require that the shape of the distributions of the variables included in an analysis should be specified for the estimation of the path coefficients, although most tests of significance assume that the parameters so estimated are normally distributed, However, linear structural relations analysis demands that the shape of the distributions of the variables should be specified, and in general, normahty is assumed both for the operation of the maximum Iikelihood procedures and for the use of significance tests. Nevertheless, a weighted least squares procedure is now provided in LISREL 7 as one of the estimation strategies in order to override these constraints (Joreskog & S&born, 1989). The analyses that are presented in this chapter are directed towards the exploration of several issues. I. ~~~~o~~f~~zgof vnriables. In the development and refinement of a path model it is commonly advantageous to combine together into a composite variable, the separate variables that would logically appear to be finked. Consequently consideration must be given to effective ways of doing this. 2. Jrzreraction between constructs. Where measures are made at approximately the same time it is sometimes necessary to consider the mutual interaction between them. Linear structural relations analysis permits the estimation of paths in which mutual interaction occurs, while ordinary least squares regression analysis and partial least squares analysis normally do not. Only through the use of indirect regression or two stage least squares regression analysis is it possible to estimate such interactive paths within the framework of least squares regression procedures. 3. Levels of analysis. The analyses reported in this monograph seek to examine the effects of both student level variables and classroom level variables on achievement outcomes. As explained in Chapter 2, a stratified random sample of classrooms was drawn and all students within each classroom selected were tested. The data can be analyzed at three levels of analysis, between students across classrooms, between classrooms, and pooled between students within classrooms. Theoretical considerations raised in Chapter 1 and discussed in greater detail in Chapter 6 indicate that differences are to be expected between the estimates ofcorresponding path coefficients in the analyses at the three levels. Likewise, because of the reduction in the variance associated with the aggregation of data the estimates of the path coefficients are also likely to be different between the three levels of analysis. The nature and extent of such differences in the estimates of the path coefficients in complex path models warrant thorough examination. However, this chapter does not attempt to resolve the many issues which arise, and seeks only to explore them. Subsequent chapters examine these issues in greater detail.

Analysis 1 -

Ordinary Least Squares Regression Analysis

In the first analysis carried out at the level of between students across ctassrooms, the variable, Sex of student (xi), was included in the analysis. As a consequence of the analysis reported in Chapter 2, the eight home variables were combined into two variables, namely Home sociai status (SES) and Parental attitudes (PATT), and correspond to Block 1 and Block 3 respectively in Figure 2.1. The weights used were consistent with the results of the confirmatory factor analysis recorded in Table 2.2. The two student attitude variables were not combined in order to examine their separate contributions to achievement

Analysis of Multilevel Data

235

outcomes. The results of the path analysis, using ordinary least squares regression analysis are presented in Figure 3.1. Paths that do not satisfy the crude criterion of statistical significance as discussed in Chapter 2, being not greater than two standard errors (i.e., 2 x 0.033 = 0.066) in magnitude, are recorded in parenthesis. Each variable in the figure is identified by its Code name and the block it forms. and each is given its variable number. The Sex of the student entered the path model in only one relationship as a contributing factor to attitudes towards Mathematics and society. with the girls holding more favourable attitudes. The strongest relationship was between Mean school aptitude (B2 (x6)) although both student attitudes (x,~ and xzo) also contributed significantly to Mathematics achievement (x~,). Home status (B,) was found to influence Mean school aptitude (B2(.~6))r which in turn influenced Parental attitudes (Bs). In addition, both student attitudes (xl9 andx& were influenced by Mean school aptitude (Bz (n6)) and Parental attitudes (B;). in Table 3.1 the results of the decomposition into direct and indirect effects of variables in this analysis, using ordinary least squares regression procedures with Mathematics achievement (,Q,) as the criterion measure at the between student level, are recorded. The last two variables in the table, Parental attitudes (Bs) and Sex of student (x,), do not have significant total effects, although Parental attitudes has a marginally significant indirect effect, acting through student attitudes. It should also be noted in Table 3.1 that while Home social status (B,) does not have a significant direct effect on Mathematics achievement (B6), it has relatively strong indirect effects operating through both Mean school aptitude (&) and Parental attitudes (B,). The results also support the hypothesized relationships in the model that the influences of the students’ background characteristics act through the standing of the secondary schools that the students attend and through the effects of parental attitudes on the students’ attitudes.

Decomposition

of

Tabte 3. I Effects on mathematics Achievement in the Path Model Using Ordinary Least Squares Regression Analysis (N=5429)

Source

Total

Direct

Indirect

Correlation

Mean school aptitude (x6) Mathematicsandself (.x,y) Home social status Mathematics and society (.q,J

0.66 0.28 0.17 0.12

0.57 0.23 -0.01 0.12

0.08 0.05 0.17

0.67 0.42 0.17 0.37

ParentaL attitudes (B3) Sex of student (x,)

0.04 0.02

-0.03 0.01

0.07 0.01

0.15 0.02

From this analysis the weak effects of Sex of student (xi) and the not unexpected strong effects of Mean school aptitude (B2) are beyond question. However, the significant direct effects of the two attitude measures Mathematics and self (A-,~)and Mathematics and society (x& raise the issue as to whether attitudes influence achievement or whether achievement influences attitudes. The assumption that attitudes affect achievement and not vice versa is a severe limitation on this model. In order to test adequately the interaction between attitudes and achievement the two attitude measures were combined.

236

er al.

K. C. CHEUNG

PARENTAL AT TITUOES

I

PATT

/

MATHEMATICS

STUOEN T BACKGROUND CHARACTERISTICS

STUOEN

T MATHEMATICS AND

SCHOOL CHARACTERISTICS

SOCIETY

~

X19 MATHEMATICS AND SELF

Figure 3.

I

Path model for ordinary

Analysis

2-

least squares regression

LISREL

analysis (N = 54%)

Analysis

An alternative analytical strategy is to use the findings presented in Analysis 1 and to estimate the two-way interaction between mathematics achievement and student attitudes towards mathematics by employing the factor analytic approach provided by LISREL. Without consideration of the structural equation model and the measurement model employed in this analysis, we present the results of the use of LISREL in the analysis of these data in the path diagram in Figure 3.2. In order to obtain a satisfactory solution for this model, which was not identified, it was necessary to assume that there was no measurement error associated with School characteristics (LAPT (B?)). This was a reasonable assumption considering the large number of cases on which this measure was based. Sex of student was deleted from the model simply because Analysis 1 showed that it made no recognizable contribution. The estimates of the path coefficients obtained from the use of LISREL are similar in magnitude to those obtained by ordinary least squares regression analysis. However, Parental attitudes (I!,) not only have a significant positive influence on Student attitudes (Bj), but also have a significant negative influence on Mathematics achievement (Bh). Such a negative direct relationship is somewhat surprising, but is offset by a strong positive

Analysis of Multilevel

Data

237

PARENTAL STUOENT SACKOROUNO CHARACTERISTICS

1.00

/ LAPTY

SCHOOL CHARACTERISTICS

Fipurc 3.2 Path model for LISREL

MATHEMATICS ACHIEVEMENT

/LSATT~

x-0

i

STUDENT ATTITUDES

analysis at hctwccn student lcvcl (N = 5429).

indirect effect (0.20) in which Parental attitudes (B,) act on Mathematics achievement (&) through Student attitudes (B,). The meaning of the negative path suggests that after allowance is made for the positive effects of student attitudes on achievement and for the indirect effects of parental attitudes acting through student attitudes in a positive manner, the students who report that their parents hold stronger attitudes are those who perform less well in mathematics. It would appear that in some circumstances strong parental attitudes are counterproductive. It should be noted that the influence of Mathematics achievement directly on Student attitudes is of borderline significance, and is slight (0.06) in comparison with the more powerful influence of Student attitudes on Mathematics achievement (0.54). In the analysis of these data there was some evidence that the estimation of these reciprocal paths was unstable. However, the several criteria for the assessment of fit recommended in the LISREL VI Manual (Jiireskog & Slirbom, 1986) were considered and the evidence available indicated that the estimation of the reciprocal paths was not unsatisfactory. Consequently, it was concluded that the effect of attitude on achievement was substantially greater than that of achievement on attitude and the former relationship was specified in subsequent analyses. In the data that we have submitted to analysis, in which a prior measure of student achievement was not available but a surrogate measure involving school characteristics, namely Mean school Mathematics aptitude (B,), could be employed, the attitudes of the student towards mathematics would appear to be a powerful determinant of achievement outcomes, after alllowance was made in the nonrecursive model for the interaction of achievement on attitudes. The goodness of fit index for this model in the LISREL analysis was 0.99, and the adjusted index was 0.97, with the mean square residual equal to 0.008, indicating an excellent fit of the model to the data.

238

K. C. CHEUNG

STUDENT BACKGROUNO CHARACTERISTICS

R’.

0.04

et ul

r..

/LPATT\

~l.l”__

MATUEMATICS

_d-++K ” . .

ATTITUDES

CHARA-CTERISTICS

Figure 3.3 Path model for PLS for between

Analysis 3 -

ACHIEVEMENT

students

across classrooms

level (N = 5429)

PLS at Between Students across Classrooms Level

Partial least square path analysis permits the many variables on which data were obtained for the examination of determinants of mathematics achievement to be incorporated into the one model. These 21 variables, referred to as manifest variables, were grouped to form the six latent variables shown in Figure 3.3 and the analyses carried out allowed not onty the estimation of the relationships between the six latent variables, but also of the links between the manifest variables and the latent variables which they form (inward mode) or reflect (outward mode). In Figure 3.3 all estimated path coefficients have been recorded in the path model presented, and non-significant coefficients have been shown in parenthesis. For further details of the partial least squares approach, Wold (1952), Noonan and Wold (1985) and Sellin (1986) should be consulted. The latent variable, Student background characteristics (B,), was formed with approximately equal weights for the measures of Father’s and Mother’s occupation and education and with a very small weight for Sex of student. The school characteristic latent variable (&) was determined almost completeIy by the mean level of mathematical aptitude of the students in the school. The Parental attitudes and aspirations latent variable (BJ) was formed using approximately equal weights from the four attitude

Analysis

of Multilevel

Data

239

measures available. Likewise, the latent variable concerned with the Learning conditions in the schools and classrooms (B,) was formed from the seven manifest variables, but with a negative weight for Pupil-teacher ratio as might be expected. The latent variable associated with Student attitudes (B,), however, had a slightly stronger weight associated with the attitude of Mathematics and self (xi9) than with the attitude, Mathematics and society (xzO). It should be noted that the latent variables of B2 and B4 were disaggregated from the school and classroom levels respectively to the student level for the analysis. The major determinant of Mathematics achievement was B2 associated with school characteristics. However, Student attitudes also had a strong influence on achievement. The variables affecting Student attitudes were Parental attitudes and aspirations (B,) (ps3 = 0.25) and School characteristics (B:) (pjz = 0.19). Both of these variables were influenced by Student background characteristics (B,). In addition, School characteristics (B,) influenced Parental attitudes (B3) as well as Learning conditions (B,) and subsequently Student attitudes (B,), although path pu was of borderline significance (0.06). In this model, in excess of half of the variance in the outcome of Mathematics achievement was accounted for by the variables included in the model. However, the direct effect of the Learning conditions in the classroom and school (B,) was not significant &, = O.OOS),and the indirect effect acting, through student attitudes (B,) was also slight (0.018). This evidence suggests that in the Hong Kong setting, the type of school a student enters at the secondary level has a profound influence on achievement outcomes through a direct effect, presumably of prior achievement, and through parental and student attitudes rather than through the learning conditions provided in the schools and classrooms.

Analysis 4 - PLS at Between Classrooms Level In Figure 3.4 the results are recorded of the examination of the same model when analyzed at the between classroom level. In order to analyze the data at this level, the measures obtained at the student level were aggregated to the classroom level. Apart from the general increase in the magnitudes of the path coefficients recorded at the between classrooms level in Figure 3.4, compared with those found at the level of between students across classrooms, the relationships reported are similar in direction and size to those recorded at the between student level shown in Figure 3.3. The effects of aggregation bias are clearly seen from these two analyses. It should be noted that in Figure 3.4 there is a negative relationship of borderline significance (pJ3 = -0.17) between mean Parental attitudes and aspirations (B,) and Learning conditions in classrooms and schools (B,). This is associated with a suppressor effect and with a change in sign from that reported for the simple correlation between these two variables. As a concomitant of the suppressor effect, the relationship between School characteristics (BJ and Learning conditions (B,) is very strong kJz = 0.84). It should be recalled that a similar suppressor effect occurred in the between students across classrooms level analysis, and that a weak negative link (pu = -0.03) was recorded, which was, however, clearly below the level of statistical significance set for the analyses reported in Figure 3.3. Furthermore, it should be noted that while the Learning conditions in schools and classrooms (B,) do not have a significant direct effect on Mean mathematics achievement

‘10

K. C. CHEUNG

era1

MEAN SrUDENr BACKQROUNO

MEAN MArHEMArICS ACHIEVEMENT

outcomes (JIM = 0.04), the indirect effect through student attitudes involves significant paths, but are still slight (PjJpbj = 0.06). While the findings presented as a result of analyzing the data at the between classrooms level appear sound, it is nevertheless necessary to caution the reader that the inflation of the coefficients reported can be viewed as aggregation bias. In Chapters 5 and 6 of this monograph the issue of aggregation bias is considered in greater detail.

Analysis

5-

PLS at Between

Students

within

Classrooms

Level

In undertaking analyses at the between students within classrooms level, the measures for each student must be subtracted from the classroom mean value and the analysis carried out using the deviation scores. Such an operation only applies to the student level variables and the jchool and classroom measures do not enter into the analysis. Consequently the blocks of variables associated with School characteristics (B,) and the Learning conditions in schools and classrooms (B,) are of necessity omitted from the PLS causal path model. The results of the analysis conducted at the between students within classrooms level are recorded in Figure 3.5. There is clear evidence of a significant chain or sequence of

Analysis of Multilevel

Data

ATTITUDES MATHEMATICS ACHIEVEMENT

EACKQROUNO CHARACTERISTICS

STUOEN T ATTITUDES

Figure 3.5 Path model for PLS for between students within classrooms level.

relationships from the latent variable for Student background characteristics (B,) to Parental attitudes and aspirations (B3) to Student attitudes (B,) to Mathematics achievement (&) with all paths in this sequence being statistically significant and with no other paths in the model reaching levels that were considered to be significant, namely, greater than 0.066. The importance of attitudes for achievement in mathematics relative to other members of the classroom group is apparent, although only 14% of the variance of the deviation scores of achievement outcomes was found to be explained by the model. The absence of an actual measure of prior achievement in mathematics at the student level must be considered to account for this low proportion of the variance in the criterion variable being explained in this cross-sectional study.

Summary and Conclusion In order to summarize the findings of this set of analyses we have presented in Table 3.2 the measures of the total effects, the direct effects and the indirect effects for the different blocks of variables on mathematics achievement which were obtained from the examination of the data at the three different levels of analysis using PLS. It is important to recognize that the initial analyses which were undertaken provided justification for three important aspects of the path models that were tested in these analyses. First, the use of confirmatory factor analysis reported in Chapter 2 provided strong support for the separation of Student background characteristics from Home or Parental attitudes and aspirations. Secondly, the use of ordinary least squares regression analysis provided

K. C. CHEUNG

er al.

Analysis of Multilevel Data

233

evidence, since both attitude variables have significant effects and one mediates the other, for the combining of the two different attitudinal measures into a single latent attitudinal variable. Thirdly, the use of linear structural relations analysis provided strong support for the postulation of a path from attitudes to achievement rather than for a path from achievement to attitudes or for the inciusion of non-recursive paths in the modei. Using computer programs available at the time for partial least squares we could not examine such a complex interactive relationship. In Table 3.2 in order to simplify the discussion of the findings, we have assumed the general levels of significance for total, direct and indirect effects that were applied in the analyses of the path models, and have set in parentheses those causal effects that are regarded as non-significant. The key results may be summarized as follows: 1. There was no significant direct effect of student background characteristics acting on mathematics achievement. However, there were significant indirect effects at both the between students and between classroom levels of analysis. 2: School characteristics had very strong direct and significant indirect effects on mathematics achievement. In particular, the procedures for the allocation of students into schools streamed students in a highly selective manner. 3. Home attitudes and aspirations had a significant indirect effect on mathematics achievement at the two student levels but not at the between classrooms level. Although this indirect effect at the between classrooms level, while not statistically significant, exceeded a commonly accepted level of practical significance of 0.10. 4. The learning conditions in the schools and classrooms did not have significant direct or indirect effects on mathematics achievement, although there was a small and nonsignificant path operating through student attitudes to influence mathematics achievement at the between classrooms level of analysis. 5. Student attitudes had substantial direct effects at all three levels of analysis on mathematics achievement. The students’ attitudes and the parents’ attitudes were the major factors influencing achievement at the students within classrooms level. This study, although cross-sectional in nature, was carried out in a quest to identify learning conditions in schools and classrooms that influence mathematics achievement. The latent variable that was formed to indicate the effects of learning conditions in schools and classrooms (B,) was strongly correlated with achievement outcomes, but these learning conditions of schools and classrooms were also strongly related to the school characteristics of Block 2. While the latent variable, Learning conditions in schools and classrooms (B,) could be said to have an effect on mathematics achievement at the between classrooms level of analysis, it was slight and operated indirectly through student attitudes. Thus the key finding of this set of analyses is the importance of student attitudes to the learning of mathematics in Hong Kong schools, and that the formation of favourable attitudes is a consequence of the student background characteristics, the school characteristics, the parental attitudes and aspirations and the learning conditions in schools and classrooms. These relationships are supported by the analyses carried out at the student, classroom and students within classrooms levels. However, from the point of view of identifying appropriate procedures for the analysis of data, the results presented are less clear. The effects of aggregation bias are substantial at the between class level. Secondly, relatively crude, although conservative, estimates of sampling errors were necessary before the findings could be tested for statistical significance. The use of such procedures must be regarded as unsatisfactory. Finally, the

2-u

K. C. CHEUNG

et al.

strategy of providing three analyses of the one path model and inviting the reader to make a choice in the interpretation of the evidence can hardly be regarded as sound. This issue is considered in detail in Chapter 10, where a solution to the problem is proposed.