Consistency measures revisited

J. FLUENCY

CONSISTENCY

DISORD. 13 (1988) 1-9

MEASURES REVISITED

WALTER L. CULLINAN University

of Oklahoma Health Sciences

Center,

Oklahoma City, Oklahoma

Stuttering consistency measures suggested 25 yr ago are reexamined and new data are provided. Problems with the maximum difference measure as frequency of stuttering increases are discussed. The recently proposed “recurrence ratio” is shown to be a restatement of

the earlier weighted percentage measure. High correlations of consistency scores with level of stuttering severity continue to be of interest. Use of the weighted percentage measure while testing significance of individual subject consistency performance in the manner suggested for the maximum difference or normal deviate measures may be the most acceptable procedure.

Fifty years ago Johnson and Knott (1937) noted the “marked tendency for the loci of stuttering to be constant from reading to reading of the same material,” the phenomenon that has become known as the “consistency effect.” The importance of the consistency effect lies in the evidence it provides that the loci of stuttering are not random events. Despite a half-century of investigation, however, there is still a question regarding how to measure consistency, as recent articles by Wingate (1984, 1986a) demonstrate. Twenty-five years ago Tate and Cullinan (1962) discussed the problems of the percentage measure for quantifying consistency and considered several alternative measures. Quantifying the degree of consistency shown by individual stutterers and testing the levels of consistency for statistical significance were indicated as important considerations. The purpose of the present paper is to reexamine the consistency measures and to provide information regarding some of the points raised by Wingate. This paper is not concerned directly with the question of whether the consistency effect supports any particular explanation of stuttering.

MEASURES OF CONSISTENCY The most widely used measure of consistency has been the percentage measure which, in each of its many forms, is the percentage of words stuttered in any given reading(s) that were also stuttered in a previous Address Oklahoma

correspondence to Walter City, OK 73190.

L. Cullinan,

8 1988 by Elsevier Science Publishing Co., Inc. 52 Vanderbilt Ave., New York, NY 10017

Ph.D.,

JKSH,

OUHSC,

PO Box 26901,

1 0094-730x&3/$03.50

2

W. L. CULLINAN

reading or readings. Tate and Cullinan (1962) noted two concerns with the percentage measure: (1) it does not make allowances for the number of repeated stutterings that can be accounted for by chance alone and (2) it does not allow for a test for the significance of the difference between the observed consistency and the consistency expected due to chance. The consistency ratio measure, suggested by Oxtoby (1955), was reviewed by Tate and Cullinan who noted that, although the measure was designed to take chance repeated measures into account, it did not permit a test of significance nor did it give sufficient weight to repeated stutters. Tate and Cullinan suggested three alternative measures, two of which (the maximum difference and the normal deviate measures) provide tests for significance and one (the weighted percentage measure) which, though it does not provide for a test for significance, does include increased weightings for words having higher numbers of repeated stutterings. The maximum difference measure (C,,,) derives from the comparison of the observed numbers, Oi, of words stuttered 0, 1, . . . , i, . . . , i = k times in k readings with the expected numbers, Ei. The expected numbers of words, Ei, stuttered by the subject are determined by computing the probability of i stutters (i = 0, I, . . . , k) on any given word in reading a passage k times and multiplying the respective probabilities by the total number of words in the passage. The differences between Oi and Ei are summarized by means of a cumulative percentage distribution. The two distributions are compared at the various levels of k and the maximum difference between the two distributions is taken as the C,,, score. The normal deviate measure (C,,) of consistency is obtained by expressing the difference between the observed (00) and the expected (Eo) numbers of nonstuttered words in standard units:

The weighted percentage C wp =

score (C,,)

is defined as follows:

100(2X2 + 3X3 + ... + kXjJ k(X,

+ X, + a** + X,J

where X1, X,, . . . , Xk, are the numbers of words stuttered 1, 2, . . . , k times, respectively, and k is the number of readings. The reader is referred to Tate and Cullinan (1962) for further details regarding the rationale and procedures for obtaining these measures and for a discussion concerning their relative advantages and disadvantages. THE RECURRENT

RECURRENCE

RATIO

Though they have been referred to as superior to the percentage measure (Bloodstein 1981, p. 222; Wingate, 1984, p. 22), the maximum difference, normal deviate, and weighted percentage measures have not, as Wingate

CONSISTENCY

3

MEASURES

points out, been widely adopted. The reason for this may be, as Bloodstein suggests (not so much defending the percentage measure as explaining its continued use), that the percentage measure is “simple and intuitively comprehensible” or, as Wingate suggests, that the alternative measures are “cumbersome” to compute. Being “cumbersome” to compute may have been a valid criticism in 1962 when hand calculators and computers were not so readily available. Such is not the case at the present time. Wingate (1984) states also that the methods discussed by Tate and Culbecause they “are based on sets of linan ( 1962)) are ‘ ‘inappropriate” proportions and involve assumptions about probabilities.” The bases of the criticisms are not well expiained, however. Regarding “proportions,” it is stated that “Computation is based on sets of proportions, rather than in regard to single words. Such treatment distorts the meaning of ‘same words’ repeatedly stuttered.” Presumably, this refers to the failure of the c maxand Gd measures to include a weighting of stuttered words according to the number of readings in which they are stuttered. However, the C,, measure does include such weighting. High correlations (see discussion below) of C,,, and C,d with C,, scores suggest that this criticism might not be important. For the second criticism it is said that the “proportions are computed in terms of probabilities, a procedure not justified for several reasons, not the least of which is contained in research evidence on the loci of stutter occurrences.” It is not clear what the “several reasons” are or which research evidence on the loci of stuttering occurrences is intended. If Wingate is referring to characteristics of stuttered words, such as those reported by Brown (1945), it would be puzzling because Wingate suggests that the “stuttering may not occur simply in regard to words as such” (1986b, p. 37), but according to some suprasegmental variable such as linguistic stress (1986b, p. 44). Whether the loci of stuttering are associated with specific types of words, points of primary linguistic stress, etc., is of theoretical significance but is secondary to the first question, that is, whether stutterers demonstrate consistency in loci of stuttering. To solve the measurement problem, Wingate (1984, 1986a) proposes a measure which he calls R,, the “recurrence ratio,” describing it as “superior to the previously developed measures.” The measure is given as follows: R = (2n2 + 3n3 + *+* + zn,) r

5(nl

+ n2 + ~1. + n,)

where nl, nL, . . . nz, are the numbers of words stuttered 1, 2, . . . , r, . . . ) r = z, times, respectively, and z is the number of readings. The obtained scores, ranging from 0.00 to 1.00, represent the proportion of actual weighted recurrence of stuttering to the maximum recurrence were each stuttered word stuttered on all readings. A comparison of this measure with the weighted percentage measure

W. L.

4

CULLINAN

described above shows that the recurrence ratio is not a new measure but merely a restatement of the weighted percentage measure. Tate and Cullinan multiplied the quantity which Wingate calls R, by 100 to change proportions to percentages, thus eliminating the decimal and providing scores ranging from 0 to 100. The multiplication by 100 does not change the nature or the characteristics of the scores but merely moves the decimal. INDIVIDUAL

CONSISTENCY

PERFORMANCES

It has been long recognized (e.g., Cullinan, 1961; Tate and Cullinan, 1962) that not all stutterers demonstrate the consistency effect. Cullinan (1961), using the maximum difference measure and the Kolmogorov-Smirnov test (Siegel, 1956), tested the significance of the consistency scores for each of 23 subjects for five readings of a 300-word passage on each of three days. The numbers of subjects manifesting significant consistency

Table 1.

Weighted Percentage, Maximum Difference, and Normal Deviate Measures of Consistency and Normal Deviate Measure of Adaptation for Cullinan’s” 23 Stutterers for Day 1 Readings Subject

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

C WP 07 00 00 00 12 13 11 11 38 14 15 29 39 36 39 73 22 66 55 48 29 00 39

C nmx 0.4 0.1 0.1 0.0 1.3 2.6 2.3 2.3 7.0 9.6’ 3.8 8.9’ 10.6’ 16.86 21.sb 34.1b 1o.7b 31.9b 31.6b 30.4b 17.0b 0.1 23.4’

c nd 0.19 -0.35 -0.18 -0.50 0.96 1.44 1.20 1.18 3.52’ 3.31b 1.70 3.78’ 4.58’ 5.986 7.47b 28.90’ 3.83’ 23.88’ 17.00b 10.90b 6.05’ -0.12 8.16b

A “d 0.00 0.00 1.50 0.00 2.48’ 0.00 3.10b 3.54b 1.44 ll.OOb 3.62’ 2.6Sb -0.60 3.59b 0.00 2Hb 4.37b 3Hb 4.06’ 2.37’ I .98’ 2.40’ 1.96’

Abbreviations: C,,, weighted percentage consistency; C,,,_, maximum difference consistency; normal deviate consistency; And, normal deviate adaptation. a Cullinan 1961. b Significant (p < 0.05). two-tailed test.

Cnd,

CONSISTENCY

MEASURES

Table 2. Weighted Percentage, Maximum Difference, and Normal Deviate Consistency Scores and Ranks, and Normal Deviate Adaptation Scores for Wingate’s” Nine Stutterers c max

c WP Subjects

Score

Rank

Score

1. 2. 3. 4. 5. 6. I. 8. 9.

49 25 22 19 14 14 8 5 3

1 2 3 4 5.5 5.5 7 8 9

31.2’ 12.7’ 1o.P 6.6 6.1 3.3 1.8 1.0 0.4

c “d

Rank 1 2 3 4 5 6 7 8 9

Abbreviations: C,, weighted percentage consistency; C,,,, normal deviate consistency; And. normal deviate adaptation. R Wingate, 1986a. b Significant (p < 0.05), two-tailed test.

Score ll.50b 3.60’ 3.16’ 2.28b 1.92 0.75 0.75 0.37 0.06

Rank 1 2 3 4 5 6.5 6.5 8 9

And

Score 4.27’ 2.65’ 2.94’ 2.2Sb 4.88’ 6.12’ 2.94’ 0.87 3.0lb

maximum difference consistency;

Cnd,

were 12, 10 and 9 on Days 1, 2, and 3, respectively. Nine (39%) of the subjects displayed greater than chance consistency on all three days. Eleven (48%) subjects failed to show greater than chance consistency on any of the three days.. Because adaptation and consistency are antagonistic, it is of interest to know how many subjects show both significant adaptation and consistency. The normal deviate adaptation scores (Tate et al., 1961), as well as the consistency scores, for the Day 1 readings for Cullinan’s (1961, 1963a,b) 23 stutterers are contained in Table 1. Of the 23 subjects, 15 showed significant adaptation. Of these 15, 10 also displayed significant consistency. Using the data provided by Wingate (1986a), maximum difference and normal deviate consistency scores and normal deviate adaptation scores were computed for his nine subjects and are presented in Table 2 along with the weighted percentage scores (obtained by dropping the decimal point from Wingate’s R, measures). Inspection of the scores indicates that three of the nine subjects showed significant consistency using the C,,, measure and four using the Cnd measures. Eight of the nine subjects demonstrated significant adaptation. These data support the point of view that not all stutterers display the consistency effect. Any theory of stuttering must explain this fact.

FURTHER CONSIDERATIONS Tate and Cullinan (1962) reported Spearman rank-order correlation coefficients of 0.82,0.76 and 0.78 for samples of 89, 23 and 30 adult stutterers, respectively, for the weighted percentage score and severity of stuttering

6

W. L. CULLINAN

as measured by the frequency of stuttering on the first of the five readings. A corresponding coefficient of 0.82 was obtained by Culiinan (1961, 1963b) for another sample of 23 adult stutterers for the first of three days on which they provided consistency data. Wingate’s (1986a) data on nine stutterers yield a correlation coefficient of 0.76. The correlations for the five samples are strikingly similar despite the differences that probably existed in the various experimental procedures. Presented in Table 3 are Spearman rank-order correlation coefficients for comparisons of the weighted percentage, maximum difference and normal deviate measures within each of the three groups of subjects from the Tate and Culiinan (1962), Cuiiinan (1961, 1963b) and Wingate (1986a) studies. The coefficients are uniformly very high, ranging from 0.96 to 1.OO.This suggests that the maximum difference and normal deviate measures, which do not allow for the weighting provided in the weighted percentage measure, might be used in place of the weighted percentage measure so that the significance of consistency for individual stutterers might be tested. Since the maximum difference and normal deviate measures correlate so highly with the weighted percentage measure, they also correlate highly with severity. The high correlations between the consistency scores and severity present a problem unless the point of view is held that consistency with a high frequency of stuttering should be weighted higher than consistency with a low frequency. Hypothetical data, demonstrating differences and similarities among the maximum difference, normal deviate and weighted percentage scores relative to increases in frequency of stuttering, are presented in Table 4 for a 180-word passage read five times. Example A shows that for perfect consistency (that is, every word that is stuttered is stuttered on ail five readings), the C,,, and Cnd scores increase with increases in the frequency of stuttering whereas the C,, scores remain at 100, the highest possible C,, score. For example B, where there is adaptation and where the numbers of words stuttered in each reading and the numbers of words stuttered i times increase proportionately with in-

Table 3. Spearman Rank-Order Consistency for Three Groups

Correlation Coefficients of Adult Stutterers

for Three

Measures

of

Stuttering group Comparison

Tate & Cullinan (1962) (n = 30)

Cullinan (1963) (n = 23)

Wingate (1984) (n = 9)

C,, versus C,,, C,, versus Cnd C max versus Cnd

0.96 0.98 0.98

0.96 0.97 0.99

1.00 0.99 0.99

Abbreviations: consistency.

C,,,

weighted

percentage

consistency; C,,,,

maximum

difference;

Cnd. normal

deviate

CONSISTENCY MEASURES

7

Table 4. Hypothetical Consistency Performances Number of words stuttered i times

Frequency of stuttering in reading Ri R, Example A: ;: 21

Rz

R3

Perfect 21 21

R.I

R5

x0

Xl

x2

x3

x4

Consistency; No Adaptation 21 21 179 0 178 0 0

Consistency x5

0

21

d.

3 4

3 4

3 4

3 4

3 4

177 176

0 0

0 0

0 0

0 0

3 4

E’

6 5

6 5

6 5

5 6

6 5

174 175

0

0

0

0

6 5

Relative Consistency; Adaptation 4 2 165 5 4 3 2 8 4 150 10 8 6 4 12 6 135 15 12 9 6 16 8 120 20 16 12 8

1 2 3 4

24 20

C.

Example B: Constant a. 15 8 6 b. 30 16 12 C. 45 24 18 d. 60 32 24 B

90 75

Example C: a. 5 b. 6 :: 7 8

2’

10 9

Example D: a. 5 b. 6 C. 7 d. 8 ;:

10 9

40 48

36 30

12 10

15 18

10 12

6 5

Increasing Consistency; Adaptation 4 3 2 1 175 1 1 5 4 3 2 174 1 1 7 6 6 5 4 5 4 3 172 173 1 1

1 1 1

1 1 1

1 2 4 3

9 8

1

1

6 5

Decreasing Consistency; Adaptation 4 3 2 1 175 1 1 1 4 3 2 1 174 2 1 1 4 3 2 1 173 3 1 1 4 3 2 1 172 4 1 1

1 1 1

1

1 1 1 1

4

1

1

7 8

3

7 6

2

6 5

1

105 90 25 30

170 171

170 171

1

6 5

20 24

1

1

Abbreviations: C,, weighted percentage consistency; imum difference consistency.

1

measure

cm,,

C”d

2.2 4.3 6.4 8.41 10.36 12.26

1.57 2.39 3.01 3.54 4.01 4.43

100 100 100 100 100 100

9.80 17.12 22.22 25.31 26.59 26.27

3.31 4.78 5.90 6.82 7.59 8.20

40 40 40 40 40 40

5.29 7.31 9.26 11.17 13.00 14.79

2.47 3.06 3.57 4.02 4.44 4.84

56 63 69 72 76 78

5.29 5.27 5.24 5.21 5.18 5.14

2.47 2.39 2.31 2.24 2.17 2.11

C,,d, normal deviation consistency;

cw,

56 47 40 35 31 28 C,,,

max-

creases in frequency, the C,,, and Cnd scores once again increase with increases in frequency whereas the C,, scores remain stable. Examples C and D show an increase and decrease, respectively, for all three measures as frequency of stuttering increases. The maximum difference measure presents an additional difficulty. The C max scores in example A increase to a maximum when RI, the number of words stuttered in reading 1, is about 33% of the number of words in the passage. The scores then decrease somewhat until an R, of 50% is reached and then increase again to the maximum at R, of 67% followed by a decrease to 0 at R, of 100%. For example B, the Cm,, scores increase

W. L. CULLINAN

8

to a maximum when RI is around 40% and then decrease until R, is around 85-90%, at which point they begin to increase again. Changes in the direction of the C,,, scores occur in other examples as well. The high correlations of the C,, scores with the C,,, and Cnd scores suggest that few stutterers reach the frequency of stuttering levels where the changes in the direction of the C,,, scores become a problem. The possibility of reaching that level does exist, however, and should be considered by users of the measure. The Cnd measure does not share this difficulty. The Cnd measure also appears to provide a more sensitive test for significance than does the C max measure. The C,,, score does not reach significance (P < 0.05) until RI equals 5, > 15, and 8 in examples A, B, and C, respectively, and none of the C,,, scores in example D are significant. For the Cnd measure, on the other hand, all of the scores in examples B and C are significant. In example A, significance is reached when RI equals 2 and in example D the scores are significant unit1 RI equals 13. To use the normal probability scale for testing Cnd, & and N - & should be about 5 or more. In Cullinan’s (1961) sample of 23 stutterers only two failed to satisfy this criterion. Weighted percentage consistency has the disadvantage of not taking chance repeated stutters into account. Tate and Cullinan (1962) give the example of a subject reading a passage five times with the expected numbers of words stuttered 0, 1, 2, 3, 4, and 5 times being 2. I, 16.5, 45.2, 62.2, 42.4, and 11.6, respectively, and the corresponding observed numbers being 3, 16,45,62,42, and 12. The weighted percentage score would be 56 and yet fail to represent significant consistency. Of the measures of consistency which have been proposed to date, the weighted percentage measure, or, as it is called by Wingate, the “recurrence ratio,” and the normal deviate measure appear to be the most acceptable. The weighted percentage measure does not allow a test of significance and does not take chance repeated stutterings into account. The normal deviate measure does provide for a test of significance. If the weighted percentage measure is used, the testing procedures used with the maximum difference and normai deviate measures might be used to test for statistical significance.

REFERENCES Bioodstein, 0. A Handbook Seal Society, 1981.

on Stuttering

Brown, S.F. The loci of stutterings Disorders. 1945, 10, 181-192. Cullinan,

W.L. An Evaluation

Performance

of Stutterers.

(3rd ed.). Chicago: National

in the speech sequence.

of the Stability

Journal

of Certain Aspects

Doctoral dissertation,

University

Easter

of Speech

of the Speaking

of Iowa, 1961.

CONSISTENCY

Cullinan,

W.L. Stability of adaptation

nal of Speech

Cullinan,

MEASURES

and Hearing

Research,

in the oral performance 1963a, 6, 70-83.

W.L. Stability of consistency measures in stuttering. Research, 1963b, 6, 134-138.

of stutterers. Journal

Jour-

of Speech

and Hearing

Johnson, W., and Knott, J.R. Studies in the psychology of stuttering: I. the distribution of moments of stuttering in successive readings of the same material. Journal

of Speech

Disorders,

1937, 2, 17-19.

Oxtoby, E.T. Frequency of stuttering in relation to induced modifications following expectancy of stuttering. In: Stuttering in Children and Adults, Johnson, W. (ed.). Minneapolis: University of Minnesota Press, 1955. Siegel, S. Nonparametric McGraw-Hill, 1956.

Statistics

for

the Behavioral

Sciences.

New

York:

Tate, M.E., Cullinan, W.L., and Ahlstrand, A. Measurement of adaptation stuttering. Journal of Speech and Hearing Research, 1961, 4, 321-339. Tate, M.E., and Cullinan, nal of Speech

Wingate,

W.L. Measurement

and Hearing

M.E. The recurrence

Research,

of consistency

of stuttering.

in

Jour-

1962, 5, 272-283.

ratio. Journal

of Fluency

Disorders,

1984, 9, 21-

29.

Wingate, M.E. Adaptation, consistency and beyond: I. limitations dictions. Journal of Fluency Disorders, 1986a, 11, l-36. Wingate, M.E. Adaptation, nal of Fluency

Disorders,

and contra-

consistency and beyond: II. an integral account. Jour1963b, 11, 37-53.