- Email: [email protected]
Acquiescence and response skewness in scale construction: a paradox
person. ;m’;urd. 01% Vol. 6, No. 5, pp. 655-656, Printed in Great Britain. All rights reserved
Acquiescence
1985 Copyright
and response skewness in scale construction:
0
0191-8869/85 33.00 + 0.00 1985 Pergamon Pmss Ltd
a paradox
J. J. RAY Department
of Sociology,
University
of New South (Received
Wales, P.O. Box I, Sydney,
26 November
2033 NSW,
Australia
1984)
Stunmary-Kirton’s version of the Wilson C Scale is primarily a version wherein items of the C Scale that attract very skewed responses have been eliminated. Contrary to what seems obvious, it is shown that the Kirton scale shows more, rather than less, influence from acquiescent response bias. Although such influence was not a serious problem with this particular scale, the finding does give grounds for believing that some skewed items may make a valuable contribution to an additive (‘Likert’) scale.
INTRODIJCHON
If we have an attitude or personality scale wherein there are three response options (e.g. ‘Yes’, ‘?’ ‘No’) for each item, it seems logically necessary that a tendency towards acquiescent response bias in the data from that scale will be associated with skewness in the responses to at least some of the items in the scale. The distribution of answers to at least some items will be skewed towards the high end (the ‘Yes’ end) of the 3-point scale. It would seem, then, that the common item-analysis procedure of eliminating from the preliminary version of a scale those items showing excessive skewness (i.e. items wherein most respondents answer ‘Yes’ or ‘No’) should, inter uliu reduce the influence of acquiescent response bias on the final form of the scale. The items most affected by acquiscence bias will, by definition, have been removed. The inter alia above is, however, important. As part of another debate on acquiescence effects (Ray and Pratt, 1979; Feather, 1980) Ray (1982) argued that not all skewness is the outcome of meaningless acquiescence and that some skewed items can make a valuable contribution to a scale. He argued that the skewed responses to some items are highly meaningful and that such items perform a valuable function in sorting out the sheep from the goats as far as the construct measured by the scale is concerned. Thus in an additive scale of conservatism, if there is one item expressing conservative sentiments with which most respondents agree, those people who disagree would have to be ‘real’ radicals. The identification of radicals provided by that item would be an especially certain one and to throw away that item would be throwing away valuable discrimination power. Proof that skewed items can be valuable was offered by showing that some skewed items had high correlations with the total score on the scale of which they formed part. The data used to illustrate this were from administrations of the (balanced) Wilson (1973) Conservatism Scale. It might, however, be contended that such usefulness in skewed items is probably rare and that the routine elimination of skewed items during scale construction should continue. Some test of whether the routine elimination of skewed items is overall beneficial or detrimental would therefore be useful. METHOD AND RESULTS As it happens, there does exist a short form of the Wilson Conservatism (C) Scale which was constructed almost entirely by eliminating from the original form of the scale those items which showed skewed responses-The Kirton (1978) C Scale. A comparison of this scale with the original should therefore be of considerable interest. It was reported at the time of its construction that the Kirton scale showed a similar reliability (a) to the Wilson original, even though it was 20 items shorter than the 50 of the original. This represents a substantial gain in internal consistency (mean inter-item correlation) and appears to justify the item-selection method used. What, however, can be said of the new scale’s openness to acquiscent responding? If (as should ‘obviously’ be the case) the deletion of skewed items has also led to the deletion of items that attracted indiscriminate acquiescence, the shorter scale should have far fewer acquiescence effects visible in it. There are two ways that the influence of indiscriminate acquiescence can be detected in a balanced scale: the correlation between the positively and negatively keyed items (rPN) will be low and the a of the scale scored for acquiescence (i.e. without normal reverse-scoring of negatively-worded items) will be high. See Martin (1964), Ray and Pratt (1979) and Ray (1983). The two statistics mentioned were therefore calculated using the data on which the Kirton (1978) C Scale had originally been constructed.* The rPN for the Kirton C Scale was -0.63 vs -0.72 for the Wilson C Scale. This indicates, if anything, more rather than less influence of acquiescence in the Kirton C Scale. The reliability (u) of the two scales scored for acquiescence was 0.2 1 for the Kirton scale and 0.15 for the Wilson C Scale. Again the evidence indicates more acquiescence in the Kirton C Scale. As these results were rather counter-intuitive, the possibility that some artifact might be involved seemed worth considering An obvious possibility is that Kirton C Scale was not exactly balanced. It had 16 positive and 14 negative items. To overcome this a new 28-item form of the Kirton C Scale was constructed using the same criteria as originally employed, which deleted two positive items and left an exactly balanced 28-item scale. This scale had an rPN of -0.63 and an acquiescence a of 0.23. This scale was, then, even worse than the 30-item scale as far as acquiescence contamination was concerned. Using Hoyt’s (1941) approach to testing the significance of a, the significant level of a for the given sample size (286) is 0.18 (P < 0.01). Thus the original Wilson C Scale had no significant internal consistency as a measure of acquiescence but both the Kirton C Scales did have.
*Data
kindly
supplied
by M. Kirton. 655
NOTES AND SHORTER COMMUNICATIONS
656
DISCUSSION It has been shown that, contrary to almost all expectation, the elimination of skewed items in the course of scale construction leads to a final scale that is more, rather than less, open to influence from acquiescent response bias, This can only be explained by the theory mentioned earlier-that many skewed items are highly meaningful to respondents and make a valuable contribution to the final scale. At least on the present occasion, it seems that the skewed items eliminated were on balance more likely to be skewed for reasons of their meaning than for reasons of acquiesence bias. Skewness is on balance more likely to be meaningful than not. Elimination of skewed items tends to reduce the proportion of highly meaningful items in a scale and thus leads to a higher aggregate influence of meaningless acquiescence in the final scores on the scale. This is not to say that scale constructors should in future ignore the problem of skewed responses to their scale items or that the Kirton C Scale in particular is misconceived. As it happens, the Kirton (1978) data shows remarkably little influence from meaningless acquiescence by the standard of what has been reported for some other administrations of the Wilson C Scale (Ray, 1980). Thus, even though the Kirton C Scale showed more influence from acquiescence bias than did the Wilson C Scale, it still remains true that such influence is minimal and compares very favorably with other scales. Nor is this finding peculiar to one sample. In a replication of the original study, another community sample of 276 people a a in S.E. England received the Kirton C Scale.* The rPN on this occasion was still a high -0.57 and the acquiescence low 0.25. The a when the scale was scored for content was 0.82. There is nothing to cavil at in these statistics. The lesson from the present work then would appear to be that skewed items can sometimes be valuable and should not be routinely eliminated. Instead, skewness of responses in any item should be considered in conjunction with the correlation of that item with the total score on the scale. There may well be a considerable number of occasions when a skewed item shows a relatively high item-total correlation. Such items might well be profitably retained instead of being routinely eliminated.
REFERENCES Feather N. T. (1980) Conservatism, acquiescence and the effects of sample heterogeneity. Aust. J. Psychol. 32, 1 l-16. Hoyt C. J. (1941) Note on a simplified method of computing test reliability. Educ.>sychol. Meusur. 1; 93-95. Kirton M. J. (1978) Wilson and Patterson’s Conservatism Scale: a shortened alternative form. Br J. sot. c&r. Psvchol. 17. 319-323. . ’ Martin J. (1964) Acquiescence: measurement and theory. Br J. sot. clin. Psychol. 3, 216225. Ray J. J. (1980) Acquiescence and the Wilson Conservatism Scale. Person. individ. Di,$j 1, 303-305. Ray J. J. (1982) Sample homogeneity, response skewness and acquiescence: a reply to Feather. Aust. J. Psychol. 33, 41-46. Ray J. J. (1983) Reviving the problem of acquiescent response bias. J. sot. Psychol. 121, 81-96. Ray J. J. and Pratt G. J. (1979) Is the influence of acquiescence on “catchphrase” type attitude scale items not so mythical after all? Amt. J. Psychol. 31, 73-78. Wilson G. D. (1973) The Psychology of Conservatism Academic Press, London.
*Data
kindly
supplied
by M. Kirton.
Acquiescence
1985 Copyright
and response skewness in scale construction:
0
0191-8869/85 33.00 + 0.00 1985 Pergamon Pmss Ltd
a paradox
J. J. RAY Department
of Sociology,
University
of New South (Received
Wales, P.O. Box I, Sydney,
26 November
2033 NSW,
Australia
1984)
Stunmary-Kirton’s version of the Wilson C Scale is primarily a version wherein items of the C Scale that attract very skewed responses have been eliminated. Contrary to what seems obvious, it is shown that the Kirton scale shows more, rather than less, influence from acquiescent response bias. Although such influence was not a serious problem with this particular scale, the finding does give grounds for believing that some skewed items may make a valuable contribution to an additive (‘Likert’) scale.
INTRODIJCHON
If we have an attitude or personality scale wherein there are three response options (e.g. ‘Yes’, ‘?’ ‘No’) for each item, it seems logically necessary that a tendency towards acquiescent response bias in the data from that scale will be associated with skewness in the responses to at least some of the items in the scale. The distribution of answers to at least some items will be skewed towards the high end (the ‘Yes’ end) of the 3-point scale. It would seem, then, that the common item-analysis procedure of eliminating from the preliminary version of a scale those items showing excessive skewness (i.e. items wherein most respondents answer ‘Yes’ or ‘No’) should, inter uliu reduce the influence of acquiescent response bias on the final form of the scale. The items most affected by acquiscence bias will, by definition, have been removed. The inter alia above is, however, important. As part of another debate on acquiescence effects (Ray and Pratt, 1979; Feather, 1980) Ray (1982) argued that not all skewness is the outcome of meaningless acquiescence and that some skewed items can make a valuable contribution to a scale. He argued that the skewed responses to some items are highly meaningful and that such items perform a valuable function in sorting out the sheep from the goats as far as the construct measured by the scale is concerned. Thus in an additive scale of conservatism, if there is one item expressing conservative sentiments with which most respondents agree, those people who disagree would have to be ‘real’ radicals. The identification of radicals provided by that item would be an especially certain one and to throw away that item would be throwing away valuable discrimination power. Proof that skewed items can be valuable was offered by showing that some skewed items had high correlations with the total score on the scale of which they formed part. The data used to illustrate this were from administrations of the (balanced) Wilson (1973) Conservatism Scale. It might, however, be contended that such usefulness in skewed items is probably rare and that the routine elimination of skewed items during scale construction should continue. Some test of whether the routine elimination of skewed items is overall beneficial or detrimental would therefore be useful. METHOD AND RESULTS As it happens, there does exist a short form of the Wilson Conservatism (C) Scale which was constructed almost entirely by eliminating from the original form of the scale those items which showed skewed responses-The Kirton (1978) C Scale. A comparison of this scale with the original should therefore be of considerable interest. It was reported at the time of its construction that the Kirton scale showed a similar reliability (a) to the Wilson original, even though it was 20 items shorter than the 50 of the original. This represents a substantial gain in internal consistency (mean inter-item correlation) and appears to justify the item-selection method used. What, however, can be said of the new scale’s openness to acquiscent responding? If (as should ‘obviously’ be the case) the deletion of skewed items has also led to the deletion of items that attracted indiscriminate acquiescence, the shorter scale should have far fewer acquiescence effects visible in it. There are two ways that the influence of indiscriminate acquiescence can be detected in a balanced scale: the correlation between the positively and negatively keyed items (rPN) will be low and the a of the scale scored for acquiescence (i.e. without normal reverse-scoring of negatively-worded items) will be high. See Martin (1964), Ray and Pratt (1979) and Ray (1983). The two statistics mentioned were therefore calculated using the data on which the Kirton (1978) C Scale had originally been constructed.* The rPN for the Kirton C Scale was -0.63 vs -0.72 for the Wilson C Scale. This indicates, if anything, more rather than less influence of acquiescence in the Kirton C Scale. The reliability (u) of the two scales scored for acquiescence was 0.2 1 for the Kirton scale and 0.15 for the Wilson C Scale. Again the evidence indicates more acquiescence in the Kirton C Scale. As these results were rather counter-intuitive, the possibility that some artifact might be involved seemed worth considering An obvious possibility is that Kirton C Scale was not exactly balanced. It had 16 positive and 14 negative items. To overcome this a new 28-item form of the Kirton C Scale was constructed using the same criteria as originally employed, which deleted two positive items and left an exactly balanced 28-item scale. This scale had an rPN of -0.63 and an acquiescence a of 0.23. This scale was, then, even worse than the 30-item scale as far as acquiescence contamination was concerned. Using Hoyt’s (1941) approach to testing the significance of a, the significant level of a for the given sample size (286) is 0.18 (P < 0.01). Thus the original Wilson C Scale had no significant internal consistency as a measure of acquiescence but both the Kirton C Scales did have.
*Data
kindly
supplied
by M. Kirton. 655
NOTES AND SHORTER COMMUNICATIONS
656
DISCUSSION It has been shown that, contrary to almost all expectation, the elimination of skewed items in the course of scale construction leads to a final scale that is more, rather than less, open to influence from acquiescent response bias, This can only be explained by the theory mentioned earlier-that many skewed items are highly meaningful to respondents and make a valuable contribution to the final scale. At least on the present occasion, it seems that the skewed items eliminated were on balance more likely to be skewed for reasons of their meaning than for reasons of acquiesence bias. Skewness is on balance more likely to be meaningful than not. Elimination of skewed items tends to reduce the proportion of highly meaningful items in a scale and thus leads to a higher aggregate influence of meaningless acquiescence in the final scores on the scale. This is not to say that scale constructors should in future ignore the problem of skewed responses to their scale items or that the Kirton C Scale in particular is misconceived. As it happens, the Kirton (1978) data shows remarkably little influence from meaningless acquiescence by the standard of what has been reported for some other administrations of the Wilson C Scale (Ray, 1980). Thus, even though the Kirton C Scale showed more influence from acquiescence bias than did the Wilson C Scale, it still remains true that such influence is minimal and compares very favorably with other scales. Nor is this finding peculiar to one sample. In a replication of the original study, another community sample of 276 people a a in S.E. England received the Kirton C Scale.* The rPN on this occasion was still a high -0.57 and the acquiescence low 0.25. The a when the scale was scored for content was 0.82. There is nothing to cavil at in these statistics. The lesson from the present work then would appear to be that skewed items can sometimes be valuable and should not be routinely eliminated. Instead, skewness of responses in any item should be considered in conjunction with the correlation of that item with the total score on the scale. There may well be a considerable number of occasions when a skewed item shows a relatively high item-total correlation. Such items might well be profitably retained instead of being routinely eliminated.
REFERENCES Feather N. T. (1980) Conservatism, acquiescence and the effects of sample heterogeneity. Aust. J. Psychol. 32, 1 l-16. Hoyt C. J. (1941) Note on a simplified method of computing test reliability. Educ.>sychol. Meusur. 1; 93-95. Kirton M. J. (1978) Wilson and Patterson’s Conservatism Scale: a shortened alternative form. Br J. sot. c&r. Psvchol. 17. 319-323. . ’ Martin J. (1964) Acquiescence: measurement and theory. Br J. sot. clin. Psychol. 3, 216225. Ray J. J. (1980) Acquiescence and the Wilson Conservatism Scale. Person. individ. Di,$j 1, 303-305. Ray J. J. (1982) Sample homogeneity, response skewness and acquiescence: a reply to Feather. Aust. J. Psychol. 33, 41-46. Ray J. J. (1983) Reviving the problem of acquiescent response bias. J. sot. Psychol. 121, 81-96. Ray J. J. and Pratt G. J. (1979) Is the influence of acquiescence on “catchphrase” type attitude scale items not so mythical after all? Amt. J. Psychol. 31, 73-78. Wilson G. D. (1973) The Psychology of Conservatism Academic Press, London.
*Data
kindly
supplied
by M. Kirton.