The effects of dynamic assessment procedures on Raven Matrices performance, visual search behavior, test anxiety and test orientation

INTELLIGENCE 6, 8%97 (1982)

The Effects of Dynamic Assessment Procedures on Raven Matrices Performance, Visual Search Behavior, Test Anxiety and Test Orientation* HANS-JORG BETHGE,

JERRY S. CARLSON AND KARL HEINZ WIEDL

Erziehungswissenchaftliche Hochschule Rheinland-Pfalz, Mainz and The University of California, Riverside

The effects of dynamic testing procedures on Raven Coloured Matrices performance, visual scanning procedures, test anxiety, and orientation to the test situation were assessed. The subjects were 72 third-grade children. The results supported the hypotheses of the study showing that (l) dynamic assessment procedures involving either verbalization or elaborated feedback lead to higher levels of Raven Matrices performance; that (2) dynamic assessment modifies visual search behaviors; and that (3) dynamic assessment reduces test anxiety and negative orientation to the testing situation. The results are interpreted as offering construct validation to the assessment approaches used.

Dynamic assessment permits analysis of how individual differences in test performance are affected through modifications in testing conditions. It does not serve as a replacement for traditional psychometric approaches, but is an alternative approach which incorporates human factors considerations in analysis of cognitive performance. A series of investigations (see Carlson & Wiedl, 1980a, 1980b for a summary) was designed to ascertain the effects of specific dynamic testing procedures on performance on cognitive and perceptual tasks. Conditions involving the subject's active verbalization while solving the problem and providing the subject with elaborated feedback concerning the details of why a previous response was correct or not, led to higher levels of performance. It was demonstrated that compensatory effects for non-target variables such as impulsivity can be obtained. Explanation for the effects of the salient testing procedures can be based on general models of human functioning which emphasize information processing. A particularly useful and encompassing model has been described by Rubinstein

*The authors wish to express their thanks to the German Research Council for financial support of the project. Research Grant 484/3 awarded to K. H. Wiedl. Reprint requests should be sent to Jerry S. Carlson, School of Education, University of California, Riverside, CA 92521.

89

90

BETHGE, CARLSON, WIEDL

(1958). It has been made more specific by Lompscher (1968) and applied to dynamic assessment approaches by Carlson and Wiedl (1980a). Although the model is comprehensive, two dimensions are of specific interest and help explain variations in performance that are most likely affected by testing procedures. They are: (1) "procedural characteristics;" and (2) "orientation." Procedural characteristics include the individual's level of planfulness and exactness- in a problemsolving situation. Orientation involves the person's evaluation of the problemsolving tasks and the atmosphere of the testing situation. Self-image, expectancy for success, and situational test anxiety are included here. The purpose of the present study was to investigate the relationship between procedural characteristics, orientation factors, and the performance optimizing conditions, verbalization, and elaborated feedback. Research (Wagner & Cimiotti, 1975) has demonstrated that both planfulness and exactness can be assessed through eye movement patterns. Thus, visual search behaviors were used to operationalize this dimension of procedural characteristics. Orientation was operationalized through the subject's evaluation of the test situation and on achievement anxiety measure. The instrument used for assessing cognitive functioning was the Raven Coloured Progressive Matrices Test (Raven, 1965). The general hypotheses in the study were (1) that d2vr.amic testing procedures lead to higher levels of performance; (2) that dynamic testing procedures affect procedural characteristics, i.e., visual search behavior; and (3) that dynamic testing procedures reduce the effects of orientation factors such as text anxiety and subjects' negative evaluation of the test situation. METHOD

Subjects The sample consisted of 72 third-grade children. The mean age was 8.7, SD = 3 months. Males and females were about equally represented in the sample. Random assignment was made to each of three testing conditions: standard, elaborated feedback, and verbalization. To confirm the equality of the groups, scores on the Cattell Culture-Fair Intelligence test were compared and found not to be significantly different (F2,69 = 0.2, NS). Procedures

Dynamic Testing Conditions C1. Standard Instruction--After the subject is shown the first practice item of the test, neither instruction to verbalize, nor feedback is provided. After solution is reached on one item, the following one is presented. This conforms to the usual manner by which the CPM is given (see Raven, 1965). Figure 1 shows an item from the CPM.

91

THE EFFECTS OF DYNAMIC ASSESSMENT PROCEDURES

AB 8

I

2

$

S

FIG. 1. response.

Sample Item from the Raven Coloured Progressive Matrices Test. Number 4 is the correct

C2. Elaborated Feedback--After the subject chose a particular alternative, he/she was told whether the selection was correct or, not. Then, it was indicated in detail why or why not the chosen response was correct. Reference was made to the demands of the task as outlined by Raven (1965) and Carlson and Jensen (1980). Each item was scored after solution and prior to any feedback. C 3. Problem Verbalization The subject was asked first to describe the main stimulus pattern before making a response. After solution was made, the subject

92

BETHGE, CARLSON, WlEDL

was asked to explain why that particular solution was chosen. Each task was scored as in the other conditions, i.e., immediately after a solution was made. Procedural Characteristics (Eye Movements) Fixation times and frequency of comparisons were measured using all 35 CPM tasks. The tasks were divided into three elements: the main pattern to the left and the two rows of answer alternatives to the right. This modification of the test allowed for greatly increased distance between the main stimulus pattern and the response alternatives, thereby facilitating the localization and measurement of the specific eye movements. The apparatus used for registering the eye movements was a Digital-Analog System designed by Zack. Six measurements were made: two for fixation-time, two for fixation-frequency, one for omission, and one for distractor comparisons. Fixation-time per task area was defined as (1) the time the subject observed the main stimulus pattern of the CPM; and (2) time spent on the answer alternatives. Fixation-frequency per task area was operationalized by (1) the number of fixations per main stimulus pattern; and (2) number of fixations per row of answer alternatives. Omissions was the frequency of movement from the main stimulus pattern to one of the row of answer alternatives without fixation on the other row of alternatives. Solution comparisons was the frequency of comparisons made between the two rows of answer alternatives. All measurements were made between the time when the task was first presented until solution was made. The values for each of the six measurements are summed across the 35 items of the CPM test. Orientation (Evaluation of the Test Situation and Achievement Anxiety This aspect of the study was approached by the development of two scales. Traditional psychometric analysis was applied to the items. Evaluation of the test situation was assessed with an instrument with five items. Each could be answered " y e s " or " n o . " A high score (greater number of yes answers) reflected a more positive evaluation of the helpfulness of the testing procedures. The individual items were as follows: 1. Were the instructions clear enough to you? 2. Did the instructions help you solve the problems? 3. Could you have solved more of the problems correctly if the instrtictions were clearer? 4. Could you understand the instructions easily? 5. Do you think that the instructions helped you solve the problems? The Spearman-Brown reliability was .76.

THE EFFECTS OF DYNAMIC ASSESSMENTPROCEDURES

93

The achievement anxiety scale consisted of six items and is based on the work of Mandler and Sarason (1952). The items were: 1. While solving the problems were you afraid that you would make an error? 2. Was it fun for you to solve the problems? 3. Did you feel that you knew how to solve the problems? 4. While solving the problems did you feel less afraid of making a mistake than at the beginning of the test? 5. As you solved the problems did you start to feel less afraid? 6. Do you feel less afraid now than at the beginning of the test? The Spearman-Brown reliability was .85. The Pearson product moment correlation between scorer on the measure of the test situation and the measure of test anxiety was significant (r = - . 36, p. < .05). All subjects were tested individually, first with the CPM according to condition and then with the orientation tasks. Eye movement indices were obtained while solutions for the CPM items were made. RESULTS The means and standard deviations of the converted CPM IQ scores are reported in Table 1. The effects of the testing conditions on performance were assessed through analysis of variance. A significant main effect difference was detected (F = 4.11,df = 2,69,p = .02). Individual comparisons between conditions were made using the NewmanKeuls test. Differences between C1 and C2, and between C1 and C3 were significant ( p < . 0 5 ) . The results of the eye movement measures, summed across all 35 CPM items for the three testing conditions, are presented in Figure 2. Inspection o f the figure

TABLE 1 Means and Standard Deviations of Raven Matrices IQ Scores Evaluations of Test Situation and Test Anxiety. Testing Condition Mean Raven SD Mean Test Evaluation SD Mean Anxiety SD

Ct

C2

C3

95.3 11.82 3.62 .82 1.9 1.15

105.0 10.45 4.21 ,58 1.0 ,90

102.7 14.18 4.15 .40 1.0 1.17

700

220

~

600

"•200

~

500

uJ 180

--~400

_~ 160

x

_x

E z

u_

3OO

140

ILl

2

O. 2 0 0 Z

~

/

12C

C3 IOC

100

I

I

I

l

220

170 Z z o

200

~ 150 u.

to

~•

--~180

x_

130

x

E

z ILl II0

I.-

160

I

I

//

O_

~

9¢

u') 140

I

I

I

I

l

I

co 7O z Qc

6

60

O

o L~

.34

~, 5o

o

(:3

I

I

I

I

t

I

Cl

C2

C3

Ci

C2

C3

TESTING

CONDITIONS

TESTING CONDITIONS

FIG. 2. Results of eye movement measures summed across all CPM items for the three testing conditions.

94

95

THE EFFECTS OF DYNAMIC ASSESSMENT PROCEDURES TABLE 2 Summary of Significant Differences in Eye Movement Patterns between the Testing Conditions Testing Conditions Eye Movement Measure

Ct-C2

C~-C3

C2-C3

Fixation-time/main pattern Fixation-time distractors Fixation-frequency main pattern Fixation-frequency distractors Omissions Distractor comparisons

<.01 <.01 <.05 <.01 < .05 <.01

<.01 <.01 <.05 <.01 NS <.05

<.01 NS <.05 <.05 NS NS

will reveal the general trend of the results. Individual analyses of variance were carried out in order to test for significant differences in the measures. The results, summarized in Table 2, show that significant differences (p<.05) were detected between C1 and C2 on all eye movement measures and between Ci and C3 on all measures, except omissions with this difference approaching significance (. 10
96

BETHGE, CARLSON, WIEDL

the modifiability of the procedural characteristics used by the subject as he/she solves the tasks and the individual's orientation toward the testing situation were analyzed. Both of these variables have been shown to be related to test performance.

Procedural characteristics The eye movement analyses indicate that subject verbalization and elaborated feedback significantly affect the procedural characteristics which children use in solving the tasks presented. The results show that the time spent on and the number of fixations made to the main stimulus patterns, and the answer alternatives of the CPM were increased by the dynamic assessment conditions. Additionally, the number of omissions of comparing the main pattern to both rows of answer alternatives was reduced while frequency of comparisons between the two rows of distractors was increased. These findings were general, pertaining to both the verbalization and elaborated feedback conditions. Elsewhere it has been documented (Dillon & Wisher, 1980) that eye movement scan patterns correlate significantly with performance on the Raven Advanced Matrices where analogical reasoning is required. Heretofore, however, there has been no empirical evidence showing the effect of differing dynamic assessment approaches on search strategies. Taken together, the eye movement data suggest that more systematic and planful strategies are used under the salient testing conditions. It would appear that the gains in performance noted under C2 and C3 can be at least partially explained by modifications in problem solving procedures brought about by the testing condition. Orientation factors The results obtained from evaluation of the test situation and reduction of achievement anxiety confirm the hypothesis that test situational variables do affect the subject's orientation to the test situation. It has been documented (Mandler & Sarason, 1952) that test anxiety can negatively affect performance. The finding that acheivement anxiety in the testing situation was significantly reduced through both the verbalization and elaborated feedback conditions offers another indication of how salient dynamic assessment procedures lead to increased levels of performance. Similarly, positive evaluation of the testing situation can lead to higher levels of motivation and interest to succeed. It was shown that the dynamic assessment approaches involved in the present study do lead to more positive attitudes on the part of the examinee. Although a test-retest design was not used in the present investigation, solid evidence does exist that subjects who show gains under salient dynamic assessment conditions (verbalization and elaborated feedback) maintain these gains when retested several weeks later under standard conditions (Wiedl & Carlson, 1981). Nonetheless, future research should be directed to establish the stability of the changes not only on the same or similar performance measures, but also on visual strategies and the orientation variables studied.

THE EFFECTS OF DYNAMIC ASSESSMENT PROCEDURES

97

In s u m m a r y , this study was shown that selected dynamic testing procedures can result in modifications in procedural characteristics and lead to reduction o f anxiety and m o r e positive evaluation of the testing situation. This offers construct validation to the d y n a m i c assessment approach as the nature of the effect of testing conditions on test performance is clarified.

REFERENCES Carlson, J. & Jetlsen, C. The factorial structure of th Raven Coloured Progressive Matrices test: A reanalysis. Educational and Psychological Measurement, 1980, 40, l I I 1-1116. Carlson, J., & Wiedl, K. Towards a differential testing approach: Testing-the limits employing the Raven Matrices. Intelligence, 1979, 4, 323-344. Carlson, J., & Wiedl, K. Applications of a dynamic testing approach in intelligence assessment: Empirical results and theoretical considerations. Zeitschrift far lndividuelle und Differentielle Psychologie, 1980a, 1, 303-318. Carlson, J., & Wiedl, K. Application of dynamic assessment to the measurement of cognition abilities. Paper presented at the Annual Meeting of the American Psychological Association, Montreal, Quebec, September 1980b. Dillon, R., & Wisher, R. The predictive validity of eye movement indices for technical school qualifying test performance. Applied Psychological Measurement, 1980, 4, in press. Guthke, J. 1st Intelligenz Messbar? Berlin: Deutscher Verlag der Wissenschaften, 1980. Lompscher, J. (Ed.), ZurEntwicklung geistigerFiihigkeiten. Berlin: Volk and Wissen Verlag, 1968. Mandler, G., & Sarason, S. A study of anxiety and learning. Journal of Abnormal and Social Psychology, 1952, 47, 166-173, Raven, J. C. Guide to Using the Progressive Matrices. Dumfries, Scotland, 1965. Rubinstein, S. L. Grundlagen der allgemeinen Psychologie. Berlin: Deutscher Verlag der Wissenschaften, 1958. Wagner, I., & Cimiotti, E. Impulsive und reflexive Kinder priifen Hypothesen: Strategien beim Problemlosen, aufgezeigt an Blickbewegungen. Zeitschrift far Entwicklungspsychologie und P~idagogische Psychologie, 1975, 7, 1-15. Wiedl, K., & Carlson, J. Dynamisches Testen bei lembehinderten Sonder-schiilem mit dem Farbigen Matrizentest von Raven. Heilpiidagogische Forschung, 1981, 9, 19--37.