- Email: [email protected]
Comparing methods: What is unfair?
ORGANIZATIONAL BEHAVIOR AND HUMAN PERFORMANCE
23, 117-119 (1979)
Comparing Methods: What Is Unfair? DOUGLAS R.
EMERY
The University of Calgary
Cattin (1979) questions the " f a i r n e s s " to regression in a study reported b y E m e r y (1978) that c o m p a r e s M O N A N O V A (Kruskal, 1965) and regression. Cattin puts forth three explanations as to why the results were obtained. This reply examines Cattin's explanations in the light of the purpose of E m e r y ' s study. The conditions and reasons for E m e r y ' s study can be stated as: H o w do the methods c o m p a r e w h e n certain assumptions (e.g., interval-scaled, linear data) are violated? To explain p o o r p e r f o r m a n c e in t e r m s of these violations is to reiterate the point that a method m a y be inappropriate w h e n its underlying assumptions are violated. To examine Cattin's c o m m e n t , it must be asked, What constitutes fairness in a c o m p a r i s o n of methods? I f the original study was " u n f a i r " to regression, then the notion of bootstrapping is also " u n f a i r " to regression. People using bootstrapping models do not necessarily believe that h u m a n decision makers use an exact linear c o m p e n s a t o r y model to process information, or that humans will provide data that are without violations of the regression assumptions.
PARAMETER ESTIMATION Cattin's first reason for the study findings is that there is a difference in the n u m b e r of p a r a m e t e r s that are estimated b y the methods.I This difference is caused b y using more than two levels, along a dimension. When only two levels of a dimension are used, b o t h methods are effectively using a " d u m m y v a r i a b l e " to represent that dimension and are therefore estimating the same n u m b e r of p a r a m e t e r s . Cattin and Bliemel's (1978) entire study used only dimensions (attributes) with two levels. T h e y in fact state of future research: Additional experiments could vary the type of model [they used only an additive model] . . . and also the amount and the type of disturbance introduced in the data [the scales used by Emery can certainly be viewed as a type of disturbance], the number of levels per attribute . . . . Requests for reprints should be sent to: Douglas R. Emery, Faculty of Management, The University of Calgary, Calgary, Alberta T2N 1N4, Canada. 1 It is assumed here that Cattin is referring to this difference as causing a difference in the number of degrees of freedom. 117 0030-5073/79/010117-03502.00/0 Copyright © 1979 by Academic Press, Inc. All rights of reproduction in any form reserved.
118
D O U G L A S R: E M E R Y
Emery (1978) did essentially what they are suggesting. Cattin suggests that a "fair" comparison would involve regression using dummy variables. This technique has been described by Searle and Udell (1970, p. B-397) for use on "phenomena for which no metric has yet been d e v e l o p e d . " In other words, Searle and Udell describe the technique as a nonmetric procedure because the independent variables would be treated nonmetrically. Therefore, using dummy variable regression would be treating the independent variables nonmetrically. Emery's (1978) study involved the comparison of a totally metric technique (multiple linear regression) and a totally nonmetric technique (MONANOVA). Cattin and Bliemel's (1978) study, on the other hand, involved the comparison of metric and nonmetric treatment for only the dependent variable, with both methods treating the independent variables nonmetrically. In addition to altering the study, using dummy variable regression would have deviated from the way in which regression was actually applied (e.g., Dawes, 1971; Dawes & Corrigan, 1974) in previous bootstrapping studies. The issue Cattin raises about the difference in the number of parameters estimated by the two methods is more appropriately dealt with by a better method of comparison. Cattin suggests an improved method of comparison in his third explanation, which is discussed below. VIOLATIONS OF ASSUMPTIONS Cattin's second reason for the results obtained by Emery is that regression violates the assumptions of the data, whereas MONANOVA does not violate the assumptions of the data. As Cattin himself (1979, p.1) points out: "Regression might still produce 'good' estimates if the violations of the assumptions are not too serious." Cattin, however, does not specify how serious "too serious" is, other than to allude that, in his opinion, the violations in Emery's study were "too serious." The effects of violations of assumptions on the methods are the underlying r e a s o n for Emery' s study. The study represents an attempt to begin to answer the question: How serious is too serious? Although the question is only answered in part, the study indicates that metric treatment of nonmetrically measured independent variables may compromise multiple linear regression. CROSS VALIDATION Cattin's third reason for the study findings is that the comparison was made only on observations used in the estimation procedure. This is a limitation in the study that was also pointed out by an anonymous referee. The point is well taken and represents a logical direction for future research. If the nonmetric treatment of the variables is unable to provide better predictions than the metric treatment in a cross-validation sample,
COMPARING METHODS
119
the simpler metric treatment would be preferred. It must be noted, however, that this issue represents only a potential explanation, and that additional research is n e e d e d to establish what differences might exist using this better method of comparison. Finally, it should also be noted that with a proper cross-validation sample for comparison, the problem of the difference in the number of parameters being estimated (and hence, difference in degrees of freedom) will essentially vanish. Both methods merely produce predictive models to be tested on new data. That is, if the nonmetric treatment of the variables provides better predictions than the metric treatment, it represents evidence that the dimensions have been correctly 2 rescaled and that there is useful predictive information to be gained in treating some variables nonmetrically.
REFERENCES Cattin, P. Comment on a Monte Carlo investigation of scaling as an alternative to regression. Organizational Behavior and Human Performance, 1979, 23, 113-116. Cattin, P., & Bliemel, F. Metric vs. nonmetric procedures for multiattribute modeling: Some simulation results. Decision Sciences, 1978, 9, 472-480. Dawes, R. M. A case study of graduate admissions: Application of three principles of human decision making. American Psychologist, 1971, 26, 180-188. Dawes, R. M., & Corrigan, B. Linear models in decision making. Psychological Bulletin, 1974, 18, 96-106. Emery, D. R. A Monte Carlo investigation of scaling as an alternative to regression in the bootstrapping model. Organizational Behavior and Human Performance, 1978, 22, 1-16. Kruskal, J. B. Analysis of factorial experiments by estimating monotone transformations of the data. Journal of the Royal Statistical Society, Series B, 1965, 27,251-263. Searle, S. R., & Udell, J. G. The use of regression on dummy variables in management science. Management Science, 1970, 16, B-397-409. RECEIVED: September 25, t978
2 Theoretically, of course, it can never be proved that a representation is correct, only that evidence supports a particular formulation.
23, 117-119 (1979)
Comparing Methods: What Is Unfair? DOUGLAS R.
EMERY
The University of Calgary
Cattin (1979) questions the " f a i r n e s s " to regression in a study reported b y E m e r y (1978) that c o m p a r e s M O N A N O V A (Kruskal, 1965) and regression. Cattin puts forth three explanations as to why the results were obtained. This reply examines Cattin's explanations in the light of the purpose of E m e r y ' s study. The conditions and reasons for E m e r y ' s study can be stated as: H o w do the methods c o m p a r e w h e n certain assumptions (e.g., interval-scaled, linear data) are violated? To explain p o o r p e r f o r m a n c e in t e r m s of these violations is to reiterate the point that a method m a y be inappropriate w h e n its underlying assumptions are violated. To examine Cattin's c o m m e n t , it must be asked, What constitutes fairness in a c o m p a r i s o n of methods? I f the original study was " u n f a i r " to regression, then the notion of bootstrapping is also " u n f a i r " to regression. People using bootstrapping models do not necessarily believe that h u m a n decision makers use an exact linear c o m p e n s a t o r y model to process information, or that humans will provide data that are without violations of the regression assumptions.
PARAMETER ESTIMATION Cattin's first reason for the study findings is that there is a difference in the n u m b e r of p a r a m e t e r s that are estimated b y the methods.I This difference is caused b y using more than two levels, along a dimension. When only two levels of a dimension are used, b o t h methods are effectively using a " d u m m y v a r i a b l e " to represent that dimension and are therefore estimating the same n u m b e r of p a r a m e t e r s . Cattin and Bliemel's (1978) entire study used only dimensions (attributes) with two levels. T h e y in fact state of future research: Additional experiments could vary the type of model [they used only an additive model] . . . and also the amount and the type of disturbance introduced in the data [the scales used by Emery can certainly be viewed as a type of disturbance], the number of levels per attribute . . . . Requests for reprints should be sent to: Douglas R. Emery, Faculty of Management, The University of Calgary, Calgary, Alberta T2N 1N4, Canada. 1 It is assumed here that Cattin is referring to this difference as causing a difference in the number of degrees of freedom. 117 0030-5073/79/010117-03502.00/0 Copyright © 1979 by Academic Press, Inc. All rights of reproduction in any form reserved.
118
D O U G L A S R: E M E R Y
Emery (1978) did essentially what they are suggesting. Cattin suggests that a "fair" comparison would involve regression using dummy variables. This technique has been described by Searle and Udell (1970, p. B-397) for use on "phenomena for which no metric has yet been d e v e l o p e d . " In other words, Searle and Udell describe the technique as a nonmetric procedure because the independent variables would be treated nonmetrically. Therefore, using dummy variable regression would be treating the independent variables nonmetrically. Emery's (1978) study involved the comparison of a totally metric technique (multiple linear regression) and a totally nonmetric technique (MONANOVA). Cattin and Bliemel's (1978) study, on the other hand, involved the comparison of metric and nonmetric treatment for only the dependent variable, with both methods treating the independent variables nonmetrically. In addition to altering the study, using dummy variable regression would have deviated from the way in which regression was actually applied (e.g., Dawes, 1971; Dawes & Corrigan, 1974) in previous bootstrapping studies. The issue Cattin raises about the difference in the number of parameters estimated by the two methods is more appropriately dealt with by a better method of comparison. Cattin suggests an improved method of comparison in his third explanation, which is discussed below. VIOLATIONS OF ASSUMPTIONS Cattin's second reason for the results obtained by Emery is that regression violates the assumptions of the data, whereas MONANOVA does not violate the assumptions of the data. As Cattin himself (1979, p.1) points out: "Regression might still produce 'good' estimates if the violations of the assumptions are not too serious." Cattin, however, does not specify how serious "too serious" is, other than to allude that, in his opinion, the violations in Emery's study were "too serious." The effects of violations of assumptions on the methods are the underlying r e a s o n for Emery' s study. The study represents an attempt to begin to answer the question: How serious is too serious? Although the question is only answered in part, the study indicates that metric treatment of nonmetrically measured independent variables may compromise multiple linear regression. CROSS VALIDATION Cattin's third reason for the study findings is that the comparison was made only on observations used in the estimation procedure. This is a limitation in the study that was also pointed out by an anonymous referee. The point is well taken and represents a logical direction for future research. If the nonmetric treatment of the variables is unable to provide better predictions than the metric treatment in a cross-validation sample,
COMPARING METHODS
119
the simpler metric treatment would be preferred. It must be noted, however, that this issue represents only a potential explanation, and that additional research is n e e d e d to establish what differences might exist using this better method of comparison. Finally, it should also be noted that with a proper cross-validation sample for comparison, the problem of the difference in the number of parameters being estimated (and hence, difference in degrees of freedom) will essentially vanish. Both methods merely produce predictive models to be tested on new data. That is, if the nonmetric treatment of the variables provides better predictions than the metric treatment, it represents evidence that the dimensions have been correctly 2 rescaled and that there is useful predictive information to be gained in treating some variables nonmetrically.
REFERENCES Cattin, P. Comment on a Monte Carlo investigation of scaling as an alternative to regression. Organizational Behavior and Human Performance, 1979, 23, 113-116. Cattin, P., & Bliemel, F. Metric vs. nonmetric procedures for multiattribute modeling: Some simulation results. Decision Sciences, 1978, 9, 472-480. Dawes, R. M. A case study of graduate admissions: Application of three principles of human decision making. American Psychologist, 1971, 26, 180-188. Dawes, R. M., & Corrigan, B. Linear models in decision making. Psychological Bulletin, 1974, 18, 96-106. Emery, D. R. A Monte Carlo investigation of scaling as an alternative to regression in the bootstrapping model. Organizational Behavior and Human Performance, 1978, 22, 1-16. Kruskal, J. B. Analysis of factorial experiments by estimating monotone transformations of the data. Journal of the Royal Statistical Society, Series B, 1965, 27,251-263. Searle, S. R., & Udell, J. G. The use of regression on dummy variables in management science. Management Science, 1970, 16, B-397-409. RECEIVED: September 25, t978
2 Theoretically, of course, it can never be proved that a representation is correct, only that evidence supports a particular formulation.