- Email: [email protected]
The examination of alternative models enhances confirmatory factor analyses
Person. individ. 01% Vol. 15, No. 5, pp. 593-594, 1993 Printed in Great Britain. All rights reserved
0191-8869/93
$6.00 + 0.00
Copyright 0 1993 Pergamon Press Ltd
The examination of alternative models enhances confirmatory factor analyses MITCHELL EABLEYWINE Departmen!
of Psychology,
University of Southern CaliJomia, (Received
10 February
CA 90089- 1061, U.S.A.
1993)
Summary-A recent study using confirmatory factor analysis on the Tridimensional Personality Questionnaire (TPQ) concludes that the proposed 3-factor structure was supported. Nevertheless, the 3-factor model failed to fit the data and was not compared to any alternative models, such as a single-factor or null model. Given the failure of the model to fit the data and the absence of comparisons to any alternative models, this support for the 3-factor model of the TPQ must be. viewed as only preliminary. Testing alternative models when confirming the factor structure of any instrument is strongly recommended.
Cloninger has developed an intriguing 3dimensional model of personality with interesting implications for psychology in general and psychopathology in particular (Cloninger, 1986, 1987a, b). Appropriate tests of the model require an assessment instrument with good psychometric properties. The proposed instrument, the Tridimensional Personality Questionnaire (IPQ), appears to assess three distinguishable dimensions when subjected to exploratory factor analyses (Kozeny, Kubicka & Prochazkova, 1989; Svrakic, Przybeck & Cloninger, 1991). Bagby, Parker and Joffe (1992) point out that exploratory factor analysis has only a limited ability to examine the factor structure of scales. Exploratory analyses can reveal the minimum number of factors required to account for variation in scores. Nevertheless, exploratory factor analysis often does not specify the number of factors apriori, nor does it designate specific indicators for each factor. Bagby et al. (1992) argue quite cogently that confirmatory factor analysis can provide more compelling evidence for the factor structure of the TPQ. ConlIrmatory factor analysis can test whether or not a 3-factor model accounts for a significant amount of variance in scores, and does so by specifically choosing the suitable subscales as indicators of their respective underlying factors. In an attempt to conlirm the factor structure of the TPQ, Bagby et al. (1992) performed a confirmatory factor analysis of scores on the IOO-item scale obtained from 216 undergraduates. The 100 items were converted into 12 subscale scores, 4 for each of the 3 hypothesized factors: novelty seeking, harm avoidance, and reward dependence. The confirmatory factor analysis produced a significant cm-squared of 2180.39, P < 0.001, suggesting that the data are significantly different from the model. In fact, data with this magnitude of difference from the specified model should occur only once in 1000 samples if the model were true in the population. Despite this failure for the data to fit the model, three indices of goodness-of-fit, the GFI (0.903), AGFI (0.849), and RMS (0.086), were interpreted as suggesting that the model was acceptable based on standards suggested by other authors (Anderson & Gerbing, 1984; Cole, 1987; Marsh, Balla & McDonald, 1988). Nevertheless, another measure of fit suggests that these data are not consistent with the model. More importantly, because the 3-factor model was not compared to alternatives, it is difficult to tell if this model accounts for more variance than other tenable models. The alternative measure of fit and the use of alternative models is discussed below. Although the GFI, AGFI, and RMS measures of goodness-of-fit can suggest how well the data coincide with the model, an alternative measure has also been suggested, the ratio of chi-squared to degrees of freedom. The ratio of chi-squared to degrees of freedom has been recommended by several authors (Joreskog & Sorbom, 1979; Wheaton, Mughen, Alwin & Summers, 1977; Carmines & McIver, 1981). The expected value of chi-squared is its degrees of freedom. Ratios of &i-squared to degrees of freedom are generally considered acceptable at no higher than 5 (Wheaton er ol., 1977) with some authors suggesting 2 or 3 as more appropriate (Carmines LkMcIver, 1981). Smaller ratios reflect better fits of the data to the model. Although the degrees of freedom are not listed in the confirmatory factor analysis of the TPQ (Bagby er ul., 1992), they can be gleaned thanks to the detailed description of the model. Twelve indicators were employed, yielding a covariance matrix of 78 covariances. Each indicator’s weighting on its respective factor was estimated, yielding 12 estimates. The error for each indicator was estimated, yielding 12 more estimates. The relation between each of the 3 underlying factors also was estimated, yielding 3 more estimates, for 27 total estimated values. In addition, 1 of the novelty seeking subscales was permitted to weigh on the harm avoidance factor, requiring 1 additional estimated value. These 28 estimated values are subtracted from the 78 covariances, yielding 50 df: The &i-square to degrees of freedom ratio, (2180.39/50) equals 43.61, markedly higher than values considered appropriate for interpreting the model as a good fit to the data. Although the ratio of &i-squared to degrees of freedom suggests these data differ significantly from the 3-factor model, this index and other measures of goodness-of-fit are extremely difficult to interpret in a vacuum. An improved way to evaluate the fit of this model would require comparing it to alternative models. Indices have been developed for evaluating the goodness-of-fit of a given model relative to an alternative null model, such as the normed fit index (Bentler & Bonett, 1980), and the parsimonious fit index (James, Mulaik & Brett, 1982). Comparisons between models can also be performed by evaluating the differences in their respective &i-squares (see Hayduk, 1987). If the 3-factor model produced a chi-squared that was significantly smaller (that is, less different from the data), than an alternative model, we can conclude that the 3-factor model has superior fit. The 3-factor model of the TPQ has been compared to alternative models in one previous study of 298 undergraduates (Rarleywine, Finn, Peterson & Pihl, 1992). A single-factor model, where each of the 12 subscales was considered an indicator of an underlying ‘self-report’ factor, was evaluated. A null model, where each of the 12 subscales was considered an indicator of its own distinguishable latent variable, was also tested. These two models were then compared to the 3-factor model. Chi-squares revealed that none of the models provided a good fit to the data. The chi-squared for the 3-factor model in this study was less than a tenth of the chi-squared obtained by Bagby et al. (1992), 167.91, and yet was still significant 593
594
NOTESANDSHOflTERCOMMUNlCA~ONS
with 51 d’, P < 0.001. [Bagby et al. (1992) had one fewer degree of freedom because they permitted one of the novelty seeking subscales to load on the harm avoidance factor.] Normed fit indices were employed to compare the 3 models. No 1 model provided a significantly better fit than the others. Thus, the proposed 3-factor model could not account for variation in scores any better than a l- or 1Zfactor model. These results suggest that perhaps the TPQ requires some modifications before it is used to test Cloninger’s hypotheses, or that undergraduates may not be an appropriate sample for confirming its factor structure. Additional studies of the TPQ may come to similar conclusions if they compare the 3-factor model to alternatives. The use of alternative models when employing contkmatory factor analysis may also aid the study of other assessment instruments.
REFERENCES
Anderson, J. C. & Gerbing, goodness-of-fit indices for Bagby, R. M., Parker, J. D. Questionnaire. Personality Bender, P. M. & Bonett, D:
D. W. (1984). The effect of sampling error on convergence, improper solutions, and maximum likelihood confirmatory factor analysis. Psychometrika, 49, 155-173. A. & Joffe, R. T. (1992). Cont%natory factor analysis of the Tridimensional Personality and Individual Differences, 13, 1245-1246.
G. (1980). Signif%ance tests and goodness of fit in the analysis of covariance structures.
Psychological Bulletin, 88, 588-606.
Carmines, E. & McIver, J. (1981). Analyzing models with unobserved variables: Analysis of covariance structures. In Bohrnstedt, G. & Borgatta, E. (Eds), Social measurement: Current issues (pp. 65-115). Beverly Hills, CA: Sage. Cloninger, C. R. (1986). A unified biosocial theory of personality and its role in the development of anxiety states. Psychiatric Developments, 3, 167-226.
Cloninger, C. R. (1987a). A systematic method for clinical description and classification of personality variants. Archives of General Psychiatry, 44, 573-588.
Cloninger, C. R. (1987b). Neurogenetic adaptive mechanisms in alcoholism. Science, 236, 410-416. Cole, D. A. (1987). Utility of con6rmatory factor analysis in test validation research. Journal of Consulting and Clinical Psychology, 55, 584-594.
Earleywine, M., Finn, P. R., Peterson, J. B. & Pihl, R. 0. (1992). Factor structure and correlates of the Tridimensional Personality Questionnaire. Journal of Studies on Alcohol, 53, 233-238. Hayduk, L. R. (1987). Structural equation moaWng with LISREL: Essentials and advances. Baltimore, MD: The Johns Hopkins University Press. James, L. R., Mulaik, S. A. 8r Brett, J. M. (1982). Causal analysis: Assumptions, models, and data. Beverly Hills, CA: Sage. Joreskog, K. G. & Sorbom, D. (1979). Advances in factor analysis and structural equation models. Cambridge, MA: Abt Books. Koxeny, J., Kubicka, L. & Prochaxkova, Z. (1989). Psychometric properties of the Czech version of Cloninger’s three dimensional nersonalitv auestionnaire. Personality and Individual Differences, 10. 1253-1259. Marsh, H. W., Balla, J. R: & McDonald, R. P. (1988). Goodness-of-fit Hdexes in confirmatory factor analysis: The effect of sample size. Psychological Rulletin, 103, 391410. Svrakic, D., Prxybeck, T. R. & Cloninger, C. R. (1991). Further contribution to the conceptual validity of the unified biosocial model of personality: U.S. and Yugoslav data. Comprehensive Psychiatry, 32, 195-209. Wheaton, B., Mughen, B., Alwin, D. & Summers, G. (1977). Assessing reliability and stability in panel models. In Heise, D. (Ed.), Sociological methodology 1977 (pp. 84-136). San Francisco: Jossey-Bass.
0191-8869/93
$6.00 + 0.00
Copyright 0 1993 Pergamon Press Ltd
The examination of alternative models enhances confirmatory factor analyses MITCHELL EABLEYWINE Departmen!
of Psychology,
University of Southern CaliJomia, (Received
10 February
CA 90089- 1061, U.S.A.
1993)
Summary-A recent study using confirmatory factor analysis on the Tridimensional Personality Questionnaire (TPQ) concludes that the proposed 3-factor structure was supported. Nevertheless, the 3-factor model failed to fit the data and was not compared to any alternative models, such as a single-factor or null model. Given the failure of the model to fit the data and the absence of comparisons to any alternative models, this support for the 3-factor model of the TPQ must be. viewed as only preliminary. Testing alternative models when confirming the factor structure of any instrument is strongly recommended.
Cloninger has developed an intriguing 3dimensional model of personality with interesting implications for psychology in general and psychopathology in particular (Cloninger, 1986, 1987a, b). Appropriate tests of the model require an assessment instrument with good psychometric properties. The proposed instrument, the Tridimensional Personality Questionnaire (IPQ), appears to assess three distinguishable dimensions when subjected to exploratory factor analyses (Kozeny, Kubicka & Prochazkova, 1989; Svrakic, Przybeck & Cloninger, 1991). Bagby, Parker and Joffe (1992) point out that exploratory factor analysis has only a limited ability to examine the factor structure of scales. Exploratory analyses can reveal the minimum number of factors required to account for variation in scores. Nevertheless, exploratory factor analysis often does not specify the number of factors apriori, nor does it designate specific indicators for each factor. Bagby et al. (1992) argue quite cogently that confirmatory factor analysis can provide more compelling evidence for the factor structure of the TPQ. ConlIrmatory factor analysis can test whether or not a 3-factor model accounts for a significant amount of variance in scores, and does so by specifically choosing the suitable subscales as indicators of their respective underlying factors. In an attempt to conlirm the factor structure of the TPQ, Bagby et al. (1992) performed a confirmatory factor analysis of scores on the IOO-item scale obtained from 216 undergraduates. The 100 items were converted into 12 subscale scores, 4 for each of the 3 hypothesized factors: novelty seeking, harm avoidance, and reward dependence. The confirmatory factor analysis produced a significant cm-squared of 2180.39, P < 0.001, suggesting that the data are significantly different from the model. In fact, data with this magnitude of difference from the specified model should occur only once in 1000 samples if the model were true in the population. Despite this failure for the data to fit the model, three indices of goodness-of-fit, the GFI (0.903), AGFI (0.849), and RMS (0.086), were interpreted as suggesting that the model was acceptable based on standards suggested by other authors (Anderson & Gerbing, 1984; Cole, 1987; Marsh, Balla & McDonald, 1988). Nevertheless, another measure of fit suggests that these data are not consistent with the model. More importantly, because the 3-factor model was not compared to alternatives, it is difficult to tell if this model accounts for more variance than other tenable models. The alternative measure of fit and the use of alternative models is discussed below. Although the GFI, AGFI, and RMS measures of goodness-of-fit can suggest how well the data coincide with the model, an alternative measure has also been suggested, the ratio of chi-squared to degrees of freedom. The ratio of chi-squared to degrees of freedom has been recommended by several authors (Joreskog & Sorbom, 1979; Wheaton, Mughen, Alwin & Summers, 1977; Carmines & McIver, 1981). The expected value of chi-squared is its degrees of freedom. Ratios of &i-squared to degrees of freedom are generally considered acceptable at no higher than 5 (Wheaton er ol., 1977) with some authors suggesting 2 or 3 as more appropriate (Carmines LkMcIver, 1981). Smaller ratios reflect better fits of the data to the model. Although the degrees of freedom are not listed in the confirmatory factor analysis of the TPQ (Bagby er ul., 1992), they can be gleaned thanks to the detailed description of the model. Twelve indicators were employed, yielding a covariance matrix of 78 covariances. Each indicator’s weighting on its respective factor was estimated, yielding 12 estimates. The error for each indicator was estimated, yielding 12 more estimates. The relation between each of the 3 underlying factors also was estimated, yielding 3 more estimates, for 27 total estimated values. In addition, 1 of the novelty seeking subscales was permitted to weigh on the harm avoidance factor, requiring 1 additional estimated value. These 28 estimated values are subtracted from the 78 covariances, yielding 50 df: The &i-square to degrees of freedom ratio, (2180.39/50) equals 43.61, markedly higher than values considered appropriate for interpreting the model as a good fit to the data. Although the ratio of &i-squared to degrees of freedom suggests these data differ significantly from the 3-factor model, this index and other measures of goodness-of-fit are extremely difficult to interpret in a vacuum. An improved way to evaluate the fit of this model would require comparing it to alternative models. Indices have been developed for evaluating the goodness-of-fit of a given model relative to an alternative null model, such as the normed fit index (Bentler & Bonett, 1980), and the parsimonious fit index (James, Mulaik & Brett, 1982). Comparisons between models can also be performed by evaluating the differences in their respective &i-squares (see Hayduk, 1987). If the 3-factor model produced a chi-squared that was significantly smaller (that is, less different from the data), than an alternative model, we can conclude that the 3-factor model has superior fit. The 3-factor model of the TPQ has been compared to alternative models in one previous study of 298 undergraduates (Rarleywine, Finn, Peterson & Pihl, 1992). A single-factor model, where each of the 12 subscales was considered an indicator of an underlying ‘self-report’ factor, was evaluated. A null model, where each of the 12 subscales was considered an indicator of its own distinguishable latent variable, was also tested. These two models were then compared to the 3-factor model. Chi-squares revealed that none of the models provided a good fit to the data. The chi-squared for the 3-factor model in this study was less than a tenth of the chi-squared obtained by Bagby et al. (1992), 167.91, and yet was still significant 593
594
NOTESANDSHOflTERCOMMUNlCA~ONS
with 51 d’, P < 0.001. [Bagby et al. (1992) had one fewer degree of freedom because they permitted one of the novelty seeking subscales to load on the harm avoidance factor.] Normed fit indices were employed to compare the 3 models. No 1 model provided a significantly better fit than the others. Thus, the proposed 3-factor model could not account for variation in scores any better than a l- or 1Zfactor model. These results suggest that perhaps the TPQ requires some modifications before it is used to test Cloninger’s hypotheses, or that undergraduates may not be an appropriate sample for confirming its factor structure. Additional studies of the TPQ may come to similar conclusions if they compare the 3-factor model to alternatives. The use of alternative models when employing contkmatory factor analysis may also aid the study of other assessment instruments.
REFERENCES
Anderson, J. C. & Gerbing, goodness-of-fit indices for Bagby, R. M., Parker, J. D. Questionnaire. Personality Bender, P. M. & Bonett, D:
D. W. (1984). The effect of sampling error on convergence, improper solutions, and maximum likelihood confirmatory factor analysis. Psychometrika, 49, 155-173. A. & Joffe, R. T. (1992). Cont%natory factor analysis of the Tridimensional Personality and Individual Differences, 13, 1245-1246.
G. (1980). Signif%ance tests and goodness of fit in the analysis of covariance structures.
Psychological Bulletin, 88, 588-606.
Carmines, E. & McIver, J. (1981). Analyzing models with unobserved variables: Analysis of covariance structures. In Bohrnstedt, G. & Borgatta, E. (Eds), Social measurement: Current issues (pp. 65-115). Beverly Hills, CA: Sage. Cloninger, C. R. (1986). A unified biosocial theory of personality and its role in the development of anxiety states. Psychiatric Developments, 3, 167-226.
Cloninger, C. R. (1987a). A systematic method for clinical description and classification of personality variants. Archives of General Psychiatry, 44, 573-588.
Cloninger, C. R. (1987b). Neurogenetic adaptive mechanisms in alcoholism. Science, 236, 410-416. Cole, D. A. (1987). Utility of con6rmatory factor analysis in test validation research. Journal of Consulting and Clinical Psychology, 55, 584-594.
Earleywine, M., Finn, P. R., Peterson, J. B. & Pihl, R. 0. (1992). Factor structure and correlates of the Tridimensional Personality Questionnaire. Journal of Studies on Alcohol, 53, 233-238. Hayduk, L. R. (1987). Structural equation moaWng with LISREL: Essentials and advances. Baltimore, MD: The Johns Hopkins University Press. James, L. R., Mulaik, S. A. 8r Brett, J. M. (1982). Causal analysis: Assumptions, models, and data. Beverly Hills, CA: Sage. Joreskog, K. G. & Sorbom, D. (1979). Advances in factor analysis and structural equation models. Cambridge, MA: Abt Books. Koxeny, J., Kubicka, L. & Prochaxkova, Z. (1989). Psychometric properties of the Czech version of Cloninger’s three dimensional nersonalitv auestionnaire. Personality and Individual Differences, 10. 1253-1259. Marsh, H. W., Balla, J. R: & McDonald, R. P. (1988). Goodness-of-fit Hdexes in confirmatory factor analysis: The effect of sample size. Psychological Rulletin, 103, 391410. Svrakic, D., Prxybeck, T. R. & Cloninger, C. R. (1991). Further contribution to the conceptual validity of the unified biosocial model of personality: U.S. and Yugoslav data. Comprehensive Psychiatry, 32, 195-209. Wheaton, B., Mughen, B., Alwin, D. & Summers, G. (1977). Assessing reliability and stability in panel models. In Heise, D. (Ed.), Sociological methodology 1977 (pp. 84-136). San Francisco: Jossey-Bass.