- Email: [email protected]
More confusion
P~non. v&w/. fJ!fl Vol. 6. No. 4. pp. 5 13-514. Printed I” Great Bntain. All rights reserved
1985 Copyright
0191-8869/85 $3.00 + 0.00 c) 1985 Pergamon Press Ltd
More confusion A. Department of Psychology,
FURNHAM
University College London, 26 Bedford Way, London WC1 OAP, England J. JASPARS
Department of Experimental
Psychology, Universily of Oxford, South Parks Road, Oxford OXI 3UD, England
(Received 23 November 1984)
In a recent study (Furnham and Jaspars, 1983) we reviewed the evidence for interactionism in psychology by presenting a critical analysis of virtually all situation-response inventory studies. We concluded that serious problems were apparent in the methodology and analyses of these studies as well as in the conclusions drawn from them, and we argued that these studies have brought as much confusion as clarity to an area of research bedevilled by both conceptual and methodological problems. In summary, we pointed out that: (1) most studies suffer from inadequate sampling of Ss, situations and responses, which implies that one can produce, as has been pointed out before by others, any result one wants; (2) in many studies actor and observer effects are confounded; (3) variance in response modes is irrelevant in testing the importance of latent invariant dispositions for the explanation of behaviour in a variety of situations; (4) in many studies analyses were performed and/or interpreted incorrectly; in particular, replicable interaction terms are often not distinguished from residual, error or within-cell variance. Despite all these criticisms, which are stated several times in our study, Ayton and Wright (1985, this issue, p. 509) in a reolv to our article. araue that we claim that a reanalysis of nearly all the published studies in the field provided strong support for the interact&St position. In fact we claimed this only for a reanalysis of three-way studies after having pointed out several limitations and reminding the reader afterwards of various pitfalls. They argue that the statistics computed by us do not constitute a proper test for interactions and that Tukey’s test for non-additivity would be more appropriate for this purpose. Focusing only on the P x S x R studies, and in particular in our reanalysis of these studies, they point out that variance accounted for by persons, situations and person x situation interaction is quite low in each case. According to Ayton and Wright, our reanalysis seems to show that the variance accounted for by these terms is substantially higher, and they warn the reader against accepting these higher values as being better estimates of the relative size of person, situation and interaction effects. We do not assume readers to be so naive as to attach any great importance to the ‘highness’ of the percentages mentioned in our reanalysis, since they presumably know that percentages add up to 100. The reason why we conducted the reanalyses was to make the three-way studies more comparable to the two-way studies which were in fact in all cases, except one, also three-way studies which ignored the response mode. In fact the residual variance for the reanalysed three-way studies appears to be substantially smaller than the residual variance of the original two-way studies. One should not forget that the latter were mainly observation studies and the former self-report studies. In those two-way studies where P x S interaction can be separated from the within-cell variance we find that the former is on average about half of the latter. Since questionnaires usually have a higher reliability than observations, it is quite possible that the proportion of genuine interaction of the residual term is in fact higher in these studies. The fact that the amount of interaction variance varies with the nature of the behaviour studied, in an understable way, would tend to support this idea (see Furnham and Jaspers, 1983, p. 638). Perhaps we should not have said that the reanalyses provide sfrong support for the interactionist position, but ‘some’ support, a conclusion which is much more in line with the rest of our study. Ayton and Wright appear to ignore however, the rest of our article. It seems to us rather redundant if not pedantic to point out that not all residual variance consists of interaction variance since we called attention to that problem at least five times in our study. We certainly do not disagree with our critics on this point therefore. We do disagree, however, with their conclusion that nothing can be said about the size or even presence or absence of any component of non-additive variance. Taking into account the evidence from other analyses and considering the nature of the behaviour studied one can at least make a reasonable guess. But even if Ayton and Wright want to confine themselves to our reanalysis, they seem to contradict themselves by proposing, later on, Tukey’s test for non-additivity as a means of detecting interaction. We also disagree with them when they argue that we should not have taken the response mode out in the analysis of variance. The variance contributed by the R factor and its interactions with the P and S factors are surely irrelevant if one is interested in the situational variability of a latent trait or disposition. The fact that certain response modes receive strong endorsement and others very little does not exclude the possibility of an underlying latent continuum. The issue is whether a reliable measure of the latent axis position in one situation will generalize to another. There seems to us little point in trying to predict for example how anxious a person might be in another situation as measured by just one response mode. The variability of response modes should not be neglected but it would seem to make more sense to take this into account by constructing proper scales rather than including them in an analysis of variance aimed at detecting P x S interaction. A three-way analysis does give quite a misleading view therefore if interactionism is the issue. If there are any factors over which we would like to generalize it is not response mode but (replicability over) time within the same situation. The two-way studies reported in our study suggest a substantial amount of rep!icable interaction variance. 513
514
NOTES AND SHORTER COMMUNICATIONS
Finally, it would seem to us a bit like carrying coal to Newcastle to suggest as a method for explaining residual variance a test for non-additivity which is extensively described in the most popular book in psychology* if it were not that Ayton and Wright fail to mention the limitations of this approach. First of all the test was designed by Tukey for detecting non-additivity in factorial experiments, and since P x S( x R) studies are by their very nature uni- or bifactor experiments with repeated measures it is doubtful whether the F-test suggested by Tukey would apply. Secondly, although the test detects systematic variation in the data it does not follow that it is also a predictable or replicable element in the residual variance, because there is only one observation or measurement per condition. Thirdly it should be stressed, as Winer does repeatedly, that it is only one particular type of non-additivity that is tested by Tukey which may make sense in some cases but not in others. In short it does not mean P x S interaction, it means a linear x linear component somewhat related to that component in the analysis of trend “In working with this component there is the implicit assumption that the larger the main effects of the individual levels in a treatment combination the larger the potential interaction effect for the treatment combination if this does exist” (Winer, 1971, 396). Ayton and Wright confuse the issue somewhat by suggesting that the interaction produced by ceiling effects can be detected by Tukey’s test. This is true enough to the extent that such effects are linear. However, in the area of performance behaviour to which they refer it is much more likely that such effects do not occur in isolation but that also the opposite effect will be present. When tasks are very easy everyone can do them, when tasks are very difficult no one can master them. It is only with the moderately easy or difficult tasks that Ss can be differentiated with respect to the underlying disposition. Such a pattern of data which is very familiar in test construction would not be detected by Tukey’s test, although it represents probably one of the commonest forms of P x S interaction. Since such an effect is readily understood in psychometrics as a consequence of the selection of items which vary in degree of difficulty, it has never presented a serious problem in test or trait theory. In fact various latent structure models (both deterministic and stochastic) take such differences into account. We do not know to what extent such a model does apply to other types of behaviour, but recent studies by Mischel (personal communication) seem to show exactly the same pattern for social behaviour. In fact, we have considered whether these findings would not suggest a general situational filter model of the P x S interaction. Whatever the case may be, it would seem important to us not to confuse non-additivity in the sense of Tukey with interaction effects as discussed in our article and in the P x S literature. Indeed we agree with the wise counsel of Golding to which we referred in our publication. It is a pity that our critics fail to take their own advice to heart. REFERENCES
Ayton P. and Wright W. (1985) The evidence for interactionism in psychology: a reply to Furnham and Jaspars. in&id. DX 6. 5099512. Furnham A.**anh Jaspars J. (1983) The evidence for interactionism in psychology. A critical analysis situation-response inventories. Person. in&d. D@ 4, 627-644. Winer B. (1971) Staristical Principles in Experimenlal Design, 2nd edn. McGraw-Hill, New York.
*According Freud’s
to a quotation study by Endler complete works.
et al., Winer’s
Srahlical
Principles in Experimental Design is second
Person. of
the
only to
1985 Copyright
0191-8869/85 $3.00 + 0.00 c) 1985 Pergamon Press Ltd
More confusion A. Department of Psychology,
FURNHAM
University College London, 26 Bedford Way, London WC1 OAP, England J. JASPARS
Department of Experimental
Psychology, Universily of Oxford, South Parks Road, Oxford OXI 3UD, England
(Received 23 November 1984)
In a recent study (Furnham and Jaspars, 1983) we reviewed the evidence for interactionism in psychology by presenting a critical analysis of virtually all situation-response inventory studies. We concluded that serious problems were apparent in the methodology and analyses of these studies as well as in the conclusions drawn from them, and we argued that these studies have brought as much confusion as clarity to an area of research bedevilled by both conceptual and methodological problems. In summary, we pointed out that: (1) most studies suffer from inadequate sampling of Ss, situations and responses, which implies that one can produce, as has been pointed out before by others, any result one wants; (2) in many studies actor and observer effects are confounded; (3) variance in response modes is irrelevant in testing the importance of latent invariant dispositions for the explanation of behaviour in a variety of situations; (4) in many studies analyses were performed and/or interpreted incorrectly; in particular, replicable interaction terms are often not distinguished from residual, error or within-cell variance. Despite all these criticisms, which are stated several times in our study, Ayton and Wright (1985, this issue, p. 509) in a reolv to our article. araue that we claim that a reanalysis of nearly all the published studies in the field provided strong support for the interact&St position. In fact we claimed this only for a reanalysis of three-way studies after having pointed out several limitations and reminding the reader afterwards of various pitfalls. They argue that the statistics computed by us do not constitute a proper test for interactions and that Tukey’s test for non-additivity would be more appropriate for this purpose. Focusing only on the P x S x R studies, and in particular in our reanalysis of these studies, they point out that variance accounted for by persons, situations and person x situation interaction is quite low in each case. According to Ayton and Wright, our reanalysis seems to show that the variance accounted for by these terms is substantially higher, and they warn the reader against accepting these higher values as being better estimates of the relative size of person, situation and interaction effects. We do not assume readers to be so naive as to attach any great importance to the ‘highness’ of the percentages mentioned in our reanalysis, since they presumably know that percentages add up to 100. The reason why we conducted the reanalyses was to make the three-way studies more comparable to the two-way studies which were in fact in all cases, except one, also three-way studies which ignored the response mode. In fact the residual variance for the reanalysed three-way studies appears to be substantially smaller than the residual variance of the original two-way studies. One should not forget that the latter were mainly observation studies and the former self-report studies. In those two-way studies where P x S interaction can be separated from the within-cell variance we find that the former is on average about half of the latter. Since questionnaires usually have a higher reliability than observations, it is quite possible that the proportion of genuine interaction of the residual term is in fact higher in these studies. The fact that the amount of interaction variance varies with the nature of the behaviour studied, in an understable way, would tend to support this idea (see Furnham and Jaspers, 1983, p. 638). Perhaps we should not have said that the reanalyses provide sfrong support for the interactionist position, but ‘some’ support, a conclusion which is much more in line with the rest of our study. Ayton and Wright appear to ignore however, the rest of our article. It seems to us rather redundant if not pedantic to point out that not all residual variance consists of interaction variance since we called attention to that problem at least five times in our study. We certainly do not disagree with our critics on this point therefore. We do disagree, however, with their conclusion that nothing can be said about the size or even presence or absence of any component of non-additive variance. Taking into account the evidence from other analyses and considering the nature of the behaviour studied one can at least make a reasonable guess. But even if Ayton and Wright want to confine themselves to our reanalysis, they seem to contradict themselves by proposing, later on, Tukey’s test for non-additivity as a means of detecting interaction. We also disagree with them when they argue that we should not have taken the response mode out in the analysis of variance. The variance contributed by the R factor and its interactions with the P and S factors are surely irrelevant if one is interested in the situational variability of a latent trait or disposition. The fact that certain response modes receive strong endorsement and others very little does not exclude the possibility of an underlying latent continuum. The issue is whether a reliable measure of the latent axis position in one situation will generalize to another. There seems to us little point in trying to predict for example how anxious a person might be in another situation as measured by just one response mode. The variability of response modes should not be neglected but it would seem to make more sense to take this into account by constructing proper scales rather than including them in an analysis of variance aimed at detecting P x S interaction. A three-way analysis does give quite a misleading view therefore if interactionism is the issue. If there are any factors over which we would like to generalize it is not response mode but (replicability over) time within the same situation. The two-way studies reported in our study suggest a substantial amount of rep!icable interaction variance. 513
514
NOTES AND SHORTER COMMUNICATIONS
Finally, it would seem to us a bit like carrying coal to Newcastle to suggest as a method for explaining residual variance a test for non-additivity which is extensively described in the most popular book in psychology* if it were not that Ayton and Wright fail to mention the limitations of this approach. First of all the test was designed by Tukey for detecting non-additivity in factorial experiments, and since P x S( x R) studies are by their very nature uni- or bifactor experiments with repeated measures it is doubtful whether the F-test suggested by Tukey would apply. Secondly, although the test detects systematic variation in the data it does not follow that it is also a predictable or replicable element in the residual variance, because there is only one observation or measurement per condition. Thirdly it should be stressed, as Winer does repeatedly, that it is only one particular type of non-additivity that is tested by Tukey which may make sense in some cases but not in others. In short it does not mean P x S interaction, it means a linear x linear component somewhat related to that component in the analysis of trend “In working with this component there is the implicit assumption that the larger the main effects of the individual levels in a treatment combination the larger the potential interaction effect for the treatment combination if this does exist” (Winer, 1971, 396). Ayton and Wright confuse the issue somewhat by suggesting that the interaction produced by ceiling effects can be detected by Tukey’s test. This is true enough to the extent that such effects are linear. However, in the area of performance behaviour to which they refer it is much more likely that such effects do not occur in isolation but that also the opposite effect will be present. When tasks are very easy everyone can do them, when tasks are very difficult no one can master them. It is only with the moderately easy or difficult tasks that Ss can be differentiated with respect to the underlying disposition. Such a pattern of data which is very familiar in test construction would not be detected by Tukey’s test, although it represents probably one of the commonest forms of P x S interaction. Since such an effect is readily understood in psychometrics as a consequence of the selection of items which vary in degree of difficulty, it has never presented a serious problem in test or trait theory. In fact various latent structure models (both deterministic and stochastic) take such differences into account. We do not know to what extent such a model does apply to other types of behaviour, but recent studies by Mischel (personal communication) seem to show exactly the same pattern for social behaviour. In fact, we have considered whether these findings would not suggest a general situational filter model of the P x S interaction. Whatever the case may be, it would seem important to us not to confuse non-additivity in the sense of Tukey with interaction effects as discussed in our article and in the P x S literature. Indeed we agree with the wise counsel of Golding to which we referred in our publication. It is a pity that our critics fail to take their own advice to heart. REFERENCES
Ayton P. and Wright W. (1985) The evidence for interactionism in psychology: a reply to Furnham and Jaspars. in&id. DX 6. 5099512. Furnham A.**anh Jaspars J. (1983) The evidence for interactionism in psychology. A critical analysis situation-response inventories. Person. in&d. D@ 4, 627-644. Winer B. (1971) Staristical Principles in Experimenlal Design, 2nd edn. McGraw-Hill, New York.
*According Freud’s
to a quotation study by Endler complete works.
et al., Winer’s
Srahlical
Principles in Experimental Design is second
Person. of
the
only to