Chapter 5 Analyzing Diagnostic Expertise of Competitive Swimming Coaches

COGNITIVE ISSUES IN MOTOR EXPERTISE 1.L. Stakes and F. AUard (Editors) 0 1993 Elsevier Science Publishers B.V. All rights reserved.

75

CHAPTER 5 ANALYZING DIAGNOSTIC EXPERTISE OF COMPETITIVE SWIMMING COACHES REBECCA RU?T LEAS AND MICHELENE T.H. CHI 821 Learning Research and Development Centre 3939 O’Hara Street. Pittsburgh, PA 15260 Introduction: Expertise and Diagnosis in Competitive Swimming This study is an attempt to understand the diagnostic knowledge and skills of expert competitive swimming coaches. The ability of a competitive swimming coach to effectively diagnose the weaknesses and strengths of a swimmer’s stroke and to prescribe a remedy is recognized as one of the very most important skills in developing high levels of coaching expertise. This diagnostic task, particularly from an underwater view, entails the consideration of three dimensions of movement analysis, in a medium which is loo0 times more dense than air. Whether instructing young athletes on basic fundamentals or fine tuning the elite athlete’s skills, knowing and understanding how to effectively analyze the competitive strokes, how to articulate the desired movements, and to increase the swimmer’s efficiency, is the hallmark of a successful and effective swimming coacWdiagnostician. Sport studies on experthovice differences in coaches have primarily focused on quantitative and descriptive characteristics of coaches and their skills. To clarify this point, most studies have focused on the outcome of the results of coaching actions and have not investigated the knowledge base and structures which enabled the coach to obtain those results. An example would be the study of the number, types, and intervals of feedback which a coach gives an athlete or team. The sport literature abounds with these sorts of analyses (Barrette, Feingold, Rees, & Pieron, 1987; Darst, Zakrajsek, & Mancini, 1989) which may be of great utility for pedagogical purposes but which do not investigate the state of knowledge that created the outcomes. Currently, little is known about the coach’s knowledge base, its organization and how that interfaces with diagnosing, even though several experthovices studies have been carried out on coaches’ diagnostic abilities in a variety of sport domains such as tennis (Armstrong & Hoffman, 1979; DiCicco, 1990). golf (Skrinar & Hoffman, 1979), gymnastics (Imwold, 1980, Imwold & Hoffman), and track and field (Pinheiro, 1989). The present study used verbal reports obtained through interviews and clinical diagnosis of the competitive freestyle swimming stroke to elicit and identify the coach’s diagnostic

76

RR.Leas and M.T.H. Chi

knowledge. Such a methodology has been pioneered by scientists in cognitive psychology, artificial intelligence (Al), decision science, medicine, and sport. Research on the diagnostic knowledge base of expert coaches in the sport domain, however, is rather sparse and in the area of competitive swimming it is virtually nonexistent. The pragmatic need to understand competent coaching arises from two sources: First the efficiency for swimming is lower than any other sport (Costill, Maglischo, & Richardson, 1992) and second, relatively few coaches attain expert diagnostic skills while the majority remain rather average, even after 10 or more years of coaching. What are the underlying mechanisms which really distinguish the expert swimming coach from less expert peers? Interest in the determination of expertise in sport coaches and in the general teaching profession has increased greatly in the last 10 years (Berliner, 1986; Siedentop & Eldar, 1989). This desire to objectivize and quantify expertise is not a trivial task, particularly in classrooms, on sport fields, and in pools which are notorious for providing many challenging variables that need to be accounted for or controlled in order for scientific study to proceed. Expertise in Sport: Competitive Swimming How do we define and operationalize expertise in a practical domain such as a sport setting? Unequivocally, the answer is, "it depends". The upshot of most references and definitions of experts and expertise in sports settings is that they are most often vague and tenuous at best. The common folk view of expertise (Lesgold, 1984). which assigns the status of expert as strongly related to experience, seems to prevail. Many experts who have invested ten- to twenty thousand hours of training in a specific skill to develop such expertise are simply written off as being intuitive, gifted, and imaginative owners of special knowledge or innate abilities. Some suffer a less complimentary fate and are thought to excel because of mere hours of practice and persistence. Relatively little attention has been directed toward understanding the expert coach's complex development of specially honed cognitive skills in a specialized domain and requiring the investment of a considerable amount of time. That some individuals apparently never learn much from their years of experience (Berliner, 1987) is a commonly observed phenomenon whose explanation continues to elude researchers. The sport of competitive swimming, like most other sports, has been hard pressed to clearly define expertise. Typically, expertise is attributed to a coach according to a number of result oriented criteria. Years of experience and accomplishment (including the number of championship teams, number of national championship qualifiers, national champions, AllAmericans, Olympic Trials qualifiers, and number of Olympians) are typical descriptors. Researchers in many fields have concurred that the development of expertise is a process that occurs over a fairly lengthy period of time. Hayes (1981) argued that at least 20,000 hours of experience are needed to achieve expert status. That is quite revealing when one realizes there are only about 2,000 working hours per year. A recent study by Could, Giannini, Krane, and Hodge (1990) surveyed backgrounds and histories of 130 elite national level coaches in 30 different Olympic sports in the United States. The average number of years of coaching for this elite group was 15 and the majority of coaches had competed in the sport they coached. Even

Diagnostic expertise

I1

more striking is that 48% of the coaches had competed on an international team and 21% had competed on an Olympic team. In the 1991 NCAA women's national championship basketball tournament, both the first and second place team head coaches had played elite basketball at Division I institutions. Personal experience in a sport seems to be in some way associated with achieving coaching expertise and with providing the coaching with additional insight into technical elements of the performance. Perhaps the link is the "perspective" (Thorndyke, 1984) one brings to the arena as a result of experience. The previous list of experiences is qualified according to the level of competition in which the swimming coach participates (e.g., age g o u p versus senior level swimming; high school or summer teams versus year around club swimming). Regardless of level, some form of the aforementioned criteria are the closest definition of expertise that one can generally find in the sport of competitive swimming. These criteria, with the exception of experience, only reflect the of expertise in action and do not reveal the special skills or mental processes and knowledge structures that the expert possesses and utilizes to obtain the results. Additionally, the exact contribution of experience to the development of expertise has continued to elude researchers. What has been established is that long periods of study and practice are a necessary but not a sufficient condition for becoming an expert (Chi, Bassok, Lewis, Reiman, & Glaser, 1989; L. S. Shulrnan, [personal communication, April 25, 19911).

a

Rather than focusing on experience, Klein, Caldenvood, and MacGregor (1989) examined the role of knowledge in expert performance. They concluded that explicit knowledge, sometimes referred to as declarative knowledge, is not sufficient for proficient performance. What remains to be determined is exactly what role experience and various types of knowledge do play in the development of expert human performance. Clinical Diagnosis in Sport Clinical diagnosis in sport is defined as the act of directly observing an athlete's performance for the purpose of analyzing the technique to identify the possible errors or weaknesses and to recommend remedial action. In swimming, and in sport in general, clinical diagnosis is the first step in a chain of action through which performance skills are improved. For the swimmer, developing efficient swimming technique is one of the most important factors in achieving fast swimming times. Thus, understanding what the expert smoke diagnostician knows and can do should be very helpful in providing insight and guidance for the development of novice swimming coaches. Gaining entrance to the intricacies and internal saucture of expert swimming diagnosticians' skills should paint a clearer picture of what it really is that expert coaches know and do and should provide an additional dimension to what is currently known about coaching expertise in competitive swimming in particular, and sport in general. Stroke Prescription

On the heels of diagnosis follows prescription, the next most important sequential step in the coach's effort to improve a swimmer's skill. Determining that a stroke is incorrect is not enough. Simply knowing it is wrong will not abet performance. Athletes often complain to

78

RR. Leas and M.T.H. Chi

coaches that their technique feels off or wrong but are completely bewildered as to how to correct it. Novice observers can often tell that the stroke does not look quite right but cannot carry the analysis to its next extension. Knowing what it is that is wrong is one thing, knowing how to fix it is quite another. It takes an experienced and knowledgeable coach to concretely and correctly interpret what is wrong. Further, the coach must be able to figure out how it can be fixed and to successfully communicate this knowledge to the athlete. The skills of interpreting and recommending are the essence of diagnosis/prescription.

-

The Knowledge Base The knowledge base responsible for the psychological processes must be explored to account for the underlying diagnostic and prescriptive skills of stroke analysis by the coach. Although relatively little is known about the nature of coaches' thought processes and their knowledge of the subject matter in adjusting instruction to the level of the athlete (Putnam, 1987). a model of sport skill diagnosis and prescription was developed by Hoffman (1983) to clarify the sequential nature of the diagnostic/prescriptive process. This model defines the difference between the athlete's response and the desired response as a discrepancy. The recognition of the nature and extent of the discrepancy as well as the identification of the cause of the discrepancy are the basic skills of diagnosis. The recommendation and application of a remedy comprise the prescriptive skill. Upon completion of this process, the initial discrepancy should be reduced or eliminated.

This skill of recognizing and understanding the cause of these discrepancies, which in swimming revolve around stroke cues, is analogous to what Chase and Simon (1973b) referred to as having a "critical eye". Experts are able to "see" (Chase & Simon, 1973b) more than less expert individuals. By "see", it is not meant that the novice cannot see the explicit cues, but that only the experts realize the importance or meaningfulness of certain patterns and configurations (Chi & Bjork, 1991). Thus the experts are able to "see" beyond the explicit cues and take the analysis to a deeper level and determine precisely what the cause of the error is and how it can be fixed. Because recognition and identification of causes of stroke cues in a dynamic sports skill is a complex, knowledge-rich task, it demands that the problem solver has command of a large body of domain-specific knowledge in order for the problem solver to find a solution to the problem. (Knowledge-lean tasks, on the other hand, are problems which can be solved without domain specific knowledge.) Studies of knowledge-rich tasks have occurred primarily in non-sport domains such as algebra, physics, thermodynamics, chess, bridge, geometry, medical diagnosis, public-policy formation, and computer programming (VanLehn, 1989).

Methods Description of Study This study was designed to investigate the differences in the diagnostic knowledge base of expert and novice competitive swimming coaches. Think aloud protocols were elicited for two tasks. In the interview task, the coaches reported their vision of the ideal freestyle stroke. A coach's conception of a good competitive swimming stroke may be reflected in the elaboration

Diagnostic expertise

19

of a stroke prototype and the particular qualities of the stroke that may be more specifically referred to as a reference of correctness. A sample from one of the expert protocols reflects these kind of parameters.

...And to counter torque, what you have to do is, you have two choices, you can counter torque with a physical exertion from the legs, that is you can kick yourself to a horizontal position, or you can counter torque by constructing the strokes such that there is more weight, connected weight, in front of your floatation point, that is the center of air, generally around the lungs... It is these images of the prototype that provide the coach with a basis of comparison by which to judge the observed performance and to offer prescriptive recommendations. The importance of the coach having a mental prototype corresponding to a schema against which to compare and contrast the observed performance has been established by numerous researchers in the field of sport skill analysis (Armstrong, 1986; Mosston, 1986; Christina & Corcos, 1988; Kreighbaum & Barthels, 1990; Schmidt, 1991). Each coach was permitted to describe the stroke for as long as they needed and the responses were audiotaped.

In the diagnosing task, the coaches analyzed the competitive freestyle stroke. Diagnosing stroke technique is a skill and so the development of expertise in that skill would seem to be dependent on the evolution of a coach's procedural knowledge. An example is "the feet and hips are swinging laterally causing excessive form drag but it is all caused by the overreaching of the hands across the body's center line on the entry". The coach saw the &of the problematic movement surface in the hip and leg motion and then correctly nasoned that the cause was in the hand entry, quite a distance away from the hips and legs. A coach's ability to elaborate on dynamic features requires adequate knowledge about the principles which guide movement in a water medium (hydrodynamics) as well as an understanding of stroke mechanics (biomechanics). This latter method seems ecologically valid inasmuch as the clinical diagnosis of strokes is normally based on a verbal interchange, either on tape or directly between coach and swimmer. Thus one would not expect verbalization to alter the problem solving process and its contents. The think aloud technique has also been successfully used by knowledge engineers (Fischoff, 1989; Klein et al., 1989; Klein, 1990) and many other researchers in tasks such as examining chess players (Chase & Simon, 1973a; de Groot, 1966). physicians (Elstein, Shulman, & Sprafka, 1978; Feltovich, Johnson, Moller, & Swanson, 1984; Gale & Marsden, 1983; Kuipers & Kassirer, 1984; Pate1 & Groen, 1986; Wortman, 1972), psychologists (Ericsson, Chase, & Faloon. 1980; Ericsson & Simon, 1980; 1984; Flanagan, 1954; Johnson, 1988; Lawrence, 1988; Leinhardt, 1983; 1986). and teachers (Cummings,Murray, & Martin, 1989; Jones, 1989; Taheri, 1982; DiCicco, 1990). See Chi (in press) for a practical guide on this type of data analysis. Subjects The nature of this research was not to estimate some population value but rather to select subjects with whom an in depth case study approach could be used and from whom the most

80

RR.Leas and M.T.H. Chi

could be learned. Criterion based sampling was utilized to obtain six subjects for an expert group and six subjects for a novice group of competitive swimming coaches. Subjects were chosen from a national population of swimming coaches. To be eligible for selection, novice coaches had a maximum of two years full time head coaching experience or three years part time or assistant coaching experience. Multiple criteria were utilized to select the expert group. First, a minimum of twelve years as full time head coach was required. Second, each of the experts was recognized by their colleagues in United States Swimming ( U S S ) and American Swim Coaches Association (ASCA) as an outstanding coacWdiagnostician. Third, each of the experts had produced anywhere from 20 to 100 of top national caliber swimmers. For the purpose of this chapter, data from the verbal protocols of two expert and two novice subjects will be reported and discussed. One exception is the "measures of coherence" data in table 3 where the complete compliment of subjects was used (six experts and six novices). It should be noted that in the diagnostic task section of this study Expert #2 had very little experience in diagnosing underwater strokes from videotape while Expert #1 had extensive experience. Expert #2 does not have access to an underwater window and thus is restricted to "above water" analysis in the daily coaching job. Both novices had some, but very limited, underwater analysis experience. Materials A videotape was developed as the stimulus condition for the diagnostic task. The tape

was filmed from an underwater window and featured four women collegiate swimmers performing the competitive freestyle stroke. The tape featured swimmers swimming toward and away from the camera for four widths of the pool lasting about 35-45 seconds or nine to eleven seconds per width. This time frame was chosen to replicate what is typically used by coaches in underwater video analysis. The four swimmers differed in their swimming skill levels. Two were national caliber (NCAA Division II) with best times of 51.7 and 53.1 for the 100 yard freestyle. The other two swimmers were much slower, 60.2 and 61.0 and were not national caliber. The 10 second difference in speed for the two groups of swimmers is a significant amount of time in the sport of swimming. Although this difference in speed cannot be detected in the film, the stroke technique which enabled the faster swimmers to excel and which resmcted the slower swimmers, was readily observable. Procedures This study specifically investigated two types of knowledge; conceptual and procedural. The interview task was designed to capture the coach's conceptual knowledge by asking for a description of hisher ideal vision of the competitive freestyle stroke. It was hypothesized that the coach's accessing of specific smke knowledge via hisher schematic representations of a prototypical freestyle stroke and the accompanying parameters for the execution of the stroke should reveal the amount, depth, and nature of the knowledge base. Each coach was permitted to describe the prototype for as long as they needed and the responses were audiotaped.

81

Diagnostic expertise

The diagnostic task was designed to capture the coaches' procedural knowledge and required each coach to watch and diagnose the underwater videotape of four swimmers swimming the freestyle stroke. After each swimmer, the coaches were asked three questions. First, they were asked to rate the swimmer's stroke technique on a scale of 1-10. Second, they were asked to provide a general assessment of the swimmer's stroke. T h i i they were asked to give a qualitative holistic diagnosis of the stroke. The same questions were asked in the same order for each coach and responses were audiotaped.

Results The Interview Task The interview task was designed to probe the coach's knowledge of the ideal freestyle stroke. Four stroke components are recognized as the major categories of movement in the freestyle stroke (Colwin, 1992; Costill et al., 1992; Maglischo, 1982). The first analysis therefore counted the number of protocol citations that fell into each of the four major freestyle stroke components of body position, armstroke, kick, and breathing (see Figure 5.1). What is interesting is that both experts cited each of the four major stroke components while both novices only used two components,both of whom failed to utilitc the "body position" component. Most of the novices' stroke features (mean of 78% for the two novices, versus 55% for the experts) centered on the armstroke component group, suggesting that the novices had limited knowledge of the other components of body position, kick, and breathing. Although it is not surprising that the experts had a greater total number of citations (58) than the novices (22), this would seem to point out the fragmented schematic representation that the novices had as compared to the more complete mental picture which the experts gave.

Novice.

Export. ~

Componontm

Body P o a i t i o n

NO.

t

~

No.

t

NO.

_

_

t

_

~

~

NO.

t

5

13

5

25

0

0

0

0

Annmtroko

20

54

11

55

7

78

10

77

Kick

12

31

1

5

0

0

3

23

Bro~thing

1

3

3

15

2

22

0

0

20

100

9

100

13

100

TOtAl.

38 100

Figure 5.1. Frequency of Stroke Components Cited for Freestyle Stroke Prototype. While Figure 5.1 clearly shows that the experts attend to a more complete view of the

82

RR.Leas and M.TH. Chi

stroke, it does not provide us with an insight into the experts’ knowledge and use of specific stroke features within each of the major stroke components. By utilizing the 15 stroke features most commonly associated with each component of the freestyle (Colwin, 1992; Costill et al., 1992; Maglischo, 1982), a second template was developed from which to compare coaches’ responses. Figure 5.2 decomposes the statements about each “broad” stroke component into one vague and 15 specific stroke feature categories. A sample of vague statements from one of the novice’s prototypes discussing armstroke and kick follows: The hand should be under the body and not outside the hips, the swimmer’s should be moving at the same time but in opposite directions, the swimmer’s arms should have high elbows and the & will be in flutter kick motion.

-

From this sample, it is clear which components are being addressed. The first three comments describe the armstroke component and the last phrase describes the kick component. There are, however, a number of reasons why this excerpt reflects a vague novice description. First, this explanation lacks the kind of sequential and qualitative descriptors common to the elaboration of stroke features by the experts. In other words, one phrase does not biomechanically relate to the next. Second, the above statements focus on specific body parts (see underlined words) whereas the experts focused on process-oriented descriptors (see below underlined words). Third, the descriptions rendered are so general and fragmented that they could apply to other competitive strokes. They do not make it clear which phase of the stroke is being described within each stroke cycle. Lastly, this novice expressed a commonly held misconception which would be considered to be a serious error in the prototype (the swimmer’s arms should be moving at the same time but in different directions) and which would prevent proper execution of the stroke. As compared to the novices, the experts reported the stroke in a much more processoriented manner. An analogous section from one of the experts’ prototypes discussing the armstroke and the kick follows: In the freestyle I look for a nice outsweep, fairly wide at the top and then a downsweeD and an insweee, and then a finish with a full extension of the (elbow) .... the best freestylers utilize a 6-beat kick and that is usually a fairly nmow kick, not very wide, within a radius of about 8 or 9 inches, 10-10 and 1/2 inches at the most .... In this excerpt, the expert cited specific biomechanical movements in the armstroke in a manner which indicates a knowledge of their sequential relationship. Additionally, the description of the kick includes specific parameters of acceptance. This expert also cited a commonly held misconception (full extension of the elbow), however, this error is of much lesser magnitude than the previously cited novice’s error. Across all four broad stroke categories (body position, armstroke, kick, breathing)

83

Diagnostic expertise

experts used 10 (El) and 11 (E2) of the 15 available feature categories (a mean of 70%) whereas novices used only four (Nl) and six (N2) of the available feature categories (a mean of 34%). This is another indication that novices’ knowledge about smke features was more deficient.

Expert.

Novice.

2

1 NO.

S

No.

2

1 S

No.

%

No.

S

C 1 Body P o a i t i o n

ceneral

0

0

2

10

0

0

0

0

Soecific F1

Lateral

5

13

1

5

0

0

0

0

F2

Vertical

0

0

2

10

0

0

0

0

5

13

5

25

0

0

0

0

2

22

4

31

Subtotal

C2 Armatroke

General

1

3

4

20

swcific F3

Entry

8

21

1

5

1

11

2

15

F4

Catch

3

8

1

5

1

11

0

0

P5

Downawoop

1

3

1

5

0

0

0

0

P6

Inaweep

1

3

1

5

1

11

0

0

Fl

UpSWOOQ

3

8

1

5

1

11

0

0

P8

Exit

0

0

1

5

0

0

1

8

P9

Recovery

3

8

1

5

0

0

1

8

P10

Timing

0

0

0

0

1

11

2

15

Subtotal

20

54

11

55

7

78

10

77

(table continue.)

RR. Leas and M.T.H. Chi

a4

Expert8

NOViC.8

1 NO.

2 b

NO.

1 b

2

NO.

b

NO.

b

C3 l i c k

CenerlL

2

5

0

0

0

0

3

23

azadLL€ F11

Depth

0

0

0

0

0

0

0

0

I12

Width

5

13

0

0

0

0

0

0

P13

Timing

5

13

1

5

0

0

0

0

Subtotal

12

31

1

s

0

0

3

23

C4 Breathing

G!umzA

0

0

2

10

0

0

0

0

swcitic F14

Po8ition

0

0

0

0

0

0

0

0

F15

Timinq

1

3

1

5

2

22

0

0

Subtotal

1

3

3

15

2

22

0

0

38

loo

20

loo

9

100

13

100

10

67

11

73

6

40

4

27

Total8 Poaturem C i t e d

Figure 5.2. Features Cited in Major Stroke Components for Protorype.

Not only was the novices’ knowledge more deficient in terms of amount, but what they It is commonly accepted that the foundation for speed in a water medium revolves around the swimmer’s body position in the water. Both experts specifically addressed the importance of lateral rotation as it relates to body position. Neither of the novices even mentioned body position anywhere in their prototype. Instead, most of the novices’ comments centered on the amstroke category, again demonstrating that the novices had a limited knowledge of the other stroke components of body position, kick, and breathing.

did know was of lesser importance.

Diagnostic expertise

85

Besides looking at the amount and salience of knowledge experts and novices possess, it was important to assess the extent to which coaches' knowledge is coherently organized. One method to assess coherence is to measure "connectedness" of the coaches' elaborations of the stroke prototype. This method was developed (Lesgold et al., 1988) to analyze the diagnostic X-Ray expertise of physicians, a somewhat analogous task to analyzing videotaped underwater strokes of swimmers. First, "stroke findings" were located in the protocols. Stroke fmdings were defined as the attribution of special stroke properties or characteristics to the stroke or swimmer. Unlike the previous analysis which placed responses either into the vague or specific feature categories, a stroke finding was identified whenever positional or movement attributes in the stroke were indicated. An example of a stroke finding would be "keeping elbows high", "streamlined body", or "full extension of elbow". Second, a "reasoning chain" was then defined by a relationship connecting one or more stroke findings. For example, a coach might first notice excessive eddies and turbulence around the swimmer's feet and then reason that these were created by excessive lateral hip and leg swing. The coach may then take the analysis a step further and point out that lateral hip and leg swing are results caused by overreaching on the entry part of the stroke. Each of these statements would be considered a relationship and would be scored as a reasoning chain of length two: Turbulence (is caused by) -> hip & leg swing (is caused by) -> overreach on entry = reasoning chain of 2. A chain was terminated one of two ways; when the coach ended the sentence or when they began discussing another component. The mean length of chains for experts was 2.6 as compared to 1.6 for novices. Third, a set of stroke findings or reasoning chains that shared a common stroke component, regardless of sequencing, were scored as a "cluster". Consequently, in this analysis, clusters were comprised of and delimited by the four broad stroke categories of body position, armstroke, kick, and breathing. A finding which did not fit into the four aforementioned categories was scored as an independent cluster. Thus, the method of first identifying stroke findings, then reasoning chains, and finally clusters, helps to illuminate the coherence of the coach's ideal prototype of the freestyle stroke and, most imponantly, quantifies the interconnectedness of the protocols. Figure 5.3 presents the mean frequencies for six experts and six novices for the analysis just described. Figure 5.3 presents the means for experts as significantly different from novices (approximately .05)in the categories of number of stroke findings, number of chains, longest chain, mean length of chains, number of clusters, biggest cluster, and mean cluster size. Although experts clearly generated a significantly greater number of stroke findings @=.013), this is not surprising given that they have a more elaborate knowledge base. However, the five subsequent measures of number of chains @=.026), longest chain @=.026), mean length of chains @=.039), number of clusters @=.006), and mean cluster size @=.013) all reflect the coherence and connectedness of the knowledge base. In particular, they measure the coherence

RR. Leas and M.T.H. Chi

86

independent of the amount of protocols (reflecting amount of knowledge) that the coaches articulated.

ExDerts Mean Med.

Novices Mean Med.

Total number of stroke findings

100* 42

17.6

16

Number of chains

28*

14

6.33

6

Longest chain

5.3*

6

2.33

2

Number of different clusters Biggest cluster size

6

4

39.66 22

2.5

2

13

12

Note: The means with an asterisk (*) were significantly different (Pc.05).

Figure 5.3. Quantitative Protocol Measures for Freestyle Prototype.

The least significant difference between the two groups was in the biggest cluster category @=.056). This suggests that it is not the case that novices are unable to produce large clusters. Novices’ large clusters came from the component that they were most knowledgeable about, the armstroke. However, the combined results suggest there is a relationship between the amount of knowledge and the connectedness of that knowledge. It should be noted, however, that the correctness of the coaches’ responses is not treated in Figures 5.1, 2, and 3, so that what the subjects cited about the stroke may have, in fact, been false or erroneous. For example, the novices expressed some features about the freestyle stroke which reflected incorrect biomechanics and faulty reasoning. Both novices expressed the idea that the arms should always be moving in opposite directions from each other which is a very naive and common error in diagnosing the freestyle stroke and which would lead to serious stroke difficulties. One of the experts also made an error in recommending the elbow be fully extended at the finish of the stroke, another commonly held misconception about the freestyle stroke. As compared to the novices, however, experts made fewer and less serious errors. In summary of the interview task for the freestyle stroke, the experts explicitly elaborated each of the major stroke components in a manner which flowed and was connected while the novices gave a very narrowly focused, fragmented prototype which dealt almost exclusively with only one component, the armstroke. The coherence of the experts’ knowledge base was further

Diagnostic expertise

87

evidenced by their greater number and size of chains and clusters. These findings are completely consistent with those reported in Chi, Hutchinson and Robin (1989) in which 5- to 7-year old children who were more or less knowledgeable about dinosaurs were asked to describe what they might know about some novel dinosaurs when pictures of them were shown. The novice children typically recited a list of visibly depicted features such as "He has sharp teeth, he has three fingers; he has sharp fingers, sharp toes, a big tail"; whereas an expert child recited a sequence of causally-connected features such as "And so he had webbed feet, so that he could swim, and his nose was shaped like a duck's bill, which gave him his name." The use of connecting words such as "so that" and "which" suggest that the expert child's knowledge of dinosaur features were interconnected so that citing one explicit feature led to the activation of another implicit feature. The Diagnostic Task The diagnostic task consisted of three subtasks. First, both the expert and novice coaches were asked to rate the technique of each of the four videotaped swimmers performing the. freestyle stroke. They rated each swimmer on a scale of 1-10 similar to that used in diving and gymnastics. To facilitate the rating comparison, the four swimmers were ranked according to their best time for the 100 yard freestyle, along with the coaches' ratings assigned to each swimmer (see Figure 5.4). The coaches were not informed of this comparison. It is evident that the experts' rankings correlated perfectly with the swimmers' actual times, whereas the novices were not able to detect the difference between the very good freestyle strokes of swimmer 1 (51 seconds) and 2 (53 seconds) and the very poor strokes of swimmers 3 (60seconds) and 4 (61 seconds). Again, it should be noted that the coaches' ratings were based on technique and not observed speed.

Experts 2 Mean

Time'

1

Swimmer 1

51.7

8

Swimmer 2

53.1

7

Swimmer 3

60.2

5

Swimmer 4

61.0

5.5

4

1

Novices 2 Mean

8.00

9

8

8.50

6

6.5

9.5

6

7.5

4.5

4.75

7

6

6.50

4.75

8

7

7.5

8

'Time for 100 yards freestyle.

Figure 5.4. Ratings of Four Swimmers.

In the second task, the coaches were asked to give an assessment of each swimmer's

RR. Leas and M.T.H.Chi

88

stroke. This was asked to see exactly what it was about the stroke that each of the coaches noticed and attended to first. After examining the protocols, one had to wonder whether these coaches watched the same swimmers! Figure 5.5 presents a collective representation of the general diagnosis rendered by the coaches for swimmer 1.

Itomm Identified

Idontitior Novicom

N L c o body roll

N1 C N 2

Elbow bont on oxtonmion

N1

Should lock olbow out front

N1

Right arm not undornoath body

N2

Loft arm not extending fully

N1 C NZ

NZ(0)

CaUS@ And offoct otatmontm

Nl(0) C

Prorcription atatomonta

N l ( 0 ) L N2(0)

Expert. Hamitation in loft foot of k i c k Littlo

C

unoqual body roll

Rotatom only to tho right Wid.

pull

Broathom to on.

El El C E2

El

C

E2

E2 mido

Stroko unbalancd Caumo and offoct mtatomontm

Proocription mtatamntm

I2 I2

El(1) C Z ( 0 ) Cl(2) C C l ( 2 )

Figure 5 5 . General Diagnoses for Swimmer No. 1

A number of items are immediately evident from this table. First, there is little overlap between the experts and the novices in the features identified either as problematic or nonproblematic for the same swimmer. Second, novices' analyses focused on the flaws of specific body parts (underlined) in a static context (for instance, "elbow bent extension"), whereas the experts focused on process aspects of the stroke such as "wide pull" or "stroke unbalanced". Thus the stroke features identified by the experts and used for diagnosis are more

Diagnostic expertise

89

"second-ordered" and "dynamic", in that they combine and relate several components often maintaining the dynamic nature of the movement. This is completely analogous to the "secondorder" features identified by Chi, Feltovich, and Glaser (1981) in their study of physics expertise and evaluating what makes a problem difficult to solve. In Figure 5.5. all but one of the novices' comments dealt with the specific features of the arms. In contrast, all but one of the experts' comments focused on a dynamic, more holistic assessment including all four components of the stroke; the kick, breathing, pull, and body position. It is of course not surprising that the novices' diagnoses tend to be less accurate. Interestingly, in the only stroke function that both the experts and novices focused upon, they had exactly the opposite diagnosis; the novices thought the swimmer had a nice body roll while both experts noted she had virtually none! Neither novice detected the lack of roll to the swimmer's left side, a serious problem resulting in stroke imbalance which often leads to shoulder injury. Lastly, the locked elbow on the entry and extension recommended by novice #1 is biomechanically incorrect and would likely lead to a shoulder injury. Additionally, one expert provided two cause and effect statements and both experts provided two prescriptive remedies in their general diagnosis of swimmer 1. The novices, in contrast, provided neither cause and effect nor prescriptive statements in their diagnosis. After the coaches completed their general diagnosis of the stroke, the third diagnostic task required them to give a holistic impression of the stroke. Expert 1 made a very astute observation in recognizing that swimmer 1 is actually a breaststroker. Expert 1 further explained why this swimmer attends to only the front pomon of the stroke, why she seemingly ignores the back portion of the stroke, and how her breaststroke movements relate to her deficiencies in freestyle. Expert 2 jumped right to the task of trying to solve the problem by recommending remedial sculling work. Both experts responded with a diagnosis within three to five seconds. A sample of the expert protocols follows: First expert: She is a breaststroker trying to swim freestyle. But I would say she's not feeling the stroke from the midway point of the stroke through the back. She is only feeling the front half of the stroke which is a real natural thing for a breaststroker to do because they don't really feel what it is like at the back of a stroke, only up front as in breaststroke; so she just throws away about half of her stroke, which in freestyle is the most powerful portion of the smke, is the last half and not the first half. .. Second expert: I would ask her to just work on the sculling motion of her underwater pull. Compared to the experts' quick and easy continued diagnosis, the novices struggled. Novice 1 was startled and flustered by having to provide a more holistic diagnosis. He asked if the tape recorder could be turned off to have additional time to think. He ended up merely

90

RR.Leas and M.T.H. Chi

reiterating his comments from the general (first) diagnosis as well as incomctly describing the movement of the hand through the armstroke pull. He also incorrectly attributed an increase in power to a phase of the stroke that is not even a power phase. Novice 2 was also stumped as to what further could be added to the general diagnosis. He required three minutes before reporting he had nothing more to add to the diagnosis. A sample of the novice protocols follows: First novice: I think it was a nice stroke. Really, probably, what we are looking for is that full extension out front--that elbow and it looked like she was bringing it up right, you know,to her throat, her thumb was coming up on her up and insweep to her throat and probably could get a lot more power if she reached out front and was able to bring it in right underneath the rib cage. Second novice: I don't have anything else to say. One interpretation of these responses is that experts have a coherent mental model of the swimmer whereas novices can only focus on the different body parts and analyze the efficiency of each part's movement. Thus, experts are able to give a qualitative holistic analysis of swimming much like the "basic approach" analysis expert physicists can give (Chi et al., 1981). Discussion The experts clearly demonstrated a superior knowledge base as reflected by three main indices: 1) amount of knowledge 2) connectedness of knowledge 3) the depth of representation of knowledge. The results in Figures 5.1 and 2 illustrate the experts' larger knowledge base as manifested by their use of all four of the major freestyle stroke components in their prototype, as compared with the novices. Additionally, within each major freestyle stroke component, the experts utilized more of the specific stroke features which incorporate hydrodynamic and biomechanic parameters. In contrast, the novices' prototype highlighted their use of mostly vague descriptions. The coherence of the coaches' knowledge base was examined via the analyses of the prototypical freestyle as reflected by the results in Figure 5.3. Experts not only had more stroke findings, but these findings were embedded in more reasoning chains, longer reasoning chains, more clusters, and larger clusters thus supporting the experts' ownership of a more coherent knowledge base. That experts represent their knowledge at a deeper level was substantiated by the results of the diagnostic task as shown in Figure 5.5. Their general diagnosis of the swimmer clearly showed they use a dynamic process analysis which reflects a holistic and unified understanding of the stroke and movement. Comparatively, novices' diagnoses were static-based, focused on specific body parts, and dwelled mainly on the pull phase of the armstroke. Additionally, they were not able to give a more in-depth holistic impression of the swimmer, nor did they cite causes, effects, or prescriptions. Further, by comparing the coaches' verbal responses from the

Diagnostic expertise

91

interview task (Figures 5.1 and 2) to the diagnoses rendered in the diagnostic task (Figure 5.5). it is clear that the experts consistently approached the description of freestyle in a processoriented manner. Their analysis was based on analyzing the swimmer's movements as they related to movement in water both for the prototype and the real-world diagnostic task. This is decidedly different from the novices' approach which was to consistently describe limb movements as they occurred relative to the swimmer's body. The experts' qualitative, holistic, and process-based diagnosis is clearly more correct since they accurately rate the stroke skill of the swimmers while the novices seemed to lack accurate discrimination abilities. Figure 5.4 showed that what the experts were saying was a good stroke was, in fact, a good stroke. The holistic process diagnoses rendered by the expert coaches are analogous to those given by expert physicists as compared to novices when they were asked to state the features of a physics problem that led to their assessment of the general "basic approach" to solving the problem (Chi, Feltovich & Glaser, 1981). As in the case of the coaches, the novices cited explicit objects in the problem statements (such as "inclined plane") as the features responsible for eliciting their basic approach, whereas experts mentioned process-like "second-order'' features, meaning that features that are not explicitly mentioned in the problem statements. These abstracted features consisted of comments such as "no initial or final conditions." or "interaction objects." The transition of explicit objected-oriented features to process-based features corresponds to a kind of "ontological"conceptual shift that Chi (1992; Chi, Slotta & de Leeuw. in press) has proposed for radical conceptual change that comes with deep understanding and the acquisition of expertise.

References Armstrong, C.W. (1986). Research on movement analysis: Implications for the development of pedagogical Competence. In M. Pieron & G.Graham (Eds.) Sport pedagogy (pp. 2739). Champaign, IL: Human Kinetics. Armstrong, C.W., & Hoffman, S.J. (1979). Effects of teaching experience, knowledge of performer competence, and knowledge of performance outcome on performance error identification. Research Quarterly, 50(3), 3 18-327. Barrette, G.T., Feingold, R.S., Rees, C.R.. & Pieron, M. (1987). Myths, models, andmethods in sporr pedagogy. Champaign, IL: Human Kinetics. Berliner, D.C. (1986). In pursuit of the expert pedagogue. Educational Researcher, AugusVSeptember, 5-13. Berliner, D.C. (1987). Ways of thinking about students and classrooms by more and less experienced teachers. In J. Calderhead (Ed.), Exploring teachers' thinking (pp.60-82). Philadelphia, PA: Cassell. Chase, W.G. & Simon, H.A. (1973a). Perception in chess. Cognitive Psychology, 5, 55-81. Chase, W. G.& Simon, H. A. (1973b). The mind's eye in chess. In W. G. Chase (Ed.), Visual information processing (pp. 215-227). New York: Academic Press. Chi, M.T.H. (In press). A practical guide to content analysis of verbal data: A reflection of

92

RR. Leas and M.T.H. Chi

knowledge representation. Journal of the Learning Sciences, [special issue] on "Methodological Issues in the Study of Learning". Chi, M.T.H. (1992). Conceptual chmge within and across ontological categories: Examples from learning and discovery in science. In R. Giere (Ed.), Cognitive models of science: Minnesota srudies in the philosophy of science. (pp. 129-186) University of Minnesota Press. Chi, M.T.H., Bassok, M., Lewis, M.W., Reiman, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182. Chi, M.T.H. & Bjork, R. (1991). Modeling expertise. In D.R. Druckman & R. Bjork (Eds.),In rhe mind's eye: Undersranding human performance @p. 57-79). Washington, DC: Academy Press. Chi, M.T.H., Feltovich, P.J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognirive Science, 5, 121-152. Chi, M.T.H., Hutchinson, J.E., & Robin, A.F. (1989). How inferences about novel domainrelated concepts can be constrained by suuctured knowledge. Merrill-Palmer Quarrerly, 35(1), 27-62. Chi, M.T.H., Slotta. J.D. & de Leeuw, N. (in press). From things to processes: A theory of conceptual change for learning science concepts. Learning and Instruction: The Journal of the European Association for Research on Learning and Instrucrion, Special issue on 'Conceptual Change." Colwin, C.M. (1992). Swimming inro the 21sr century. Champaign, IL: Leisure Press. Costill, D.L., Maglischo, E.W., & Richardson, A.B. (1992). Swimming. Boston, MA: Blackwell Scientific Publications. Cummings, A.L., Murray, H.G., & Martin, J. (1989). Protocol analysis of the social problem solving of teachers. American Educational Research Journal, 26( 1). 25-43. Darst, P.W., Zakrajsek, D.B., & Mancini, V.H. (1989). Analyzing physical education and sporr insnuction. Champaign, k Human Kinetics. de Groot, A. (1966). Perception and memory versus thought: Some old ideas and recent findings. In B. Kleinmuntz (Ed.), Problem solving. New York Wiley. DiCicco, G.L.(1990). The effects of playing and reaching experience on abiliry ro perform a diagnostic rusk. Unpublished doctoral dissertation, University of Pittsburgh, Pittsburgh, PA. Elstein, AS., Shulman, L.S., & Sprafka, S.A. (1978). Medical problem solving: An analysis of clinical reasoning. Cambridge, MA: Harvard University Press. Ericsson, K.A., Chase, W.G., & Faloon, S. (1980). Acquisition of a memory skill. Science, 208, 1 181-1182. Ericsson, K.A., & Simon, H.A. (1980). Verbal reports as data. Psychological Review, 87(3), 2 15-251. Ericsson, K.A., & Simon, H.A. (1984). Prorocol analysis: Verbal reports as dara. Cambridge, MA: MIT Press. Feltovich, P.J., Johnson, P.E., Moller, J.H., & Swanson, D.B. (1984). LCS: The role and development of medical knowledge in diagnostic expertise. In W. J. Clancey & E. H.

Diagnostic expertise

93

Shortliffe (Eds.), Readings in medical artificial intelligence (pp. 275-319). Reading, MA: Addison-Wesley. Fischoff, B. (1989). Eliciting knowledge for analytical representation. IEEE Transactions on Systems, Man. and Cybernetics, 19(3), 448-461. Flanagan, J.C. (1954). The critical incident technique. Psychological Bulletin, 51(4), 327-358. Gale, J., & Marsden, P. (1983). Medical dingnosisfrom student to clinician. New Yo* Oxford University Press. Gould, D., Giannini, J., Krane. V.. & Hodge, K. (1990). Educational needs of elite U.S. National Team, Pan American, and Olympic coaches. Journal ofTeaching in Physical Education, 9, 332-344. Hayes, J.R. (1981). The complete problem solver. Philadelphia, PA: Franklin Institute Press. Hoffman, S. J. (1983). Clinical diagnosis as a pedagogical skill. In T. J. Templin & J. K. Olson (Eds.), Teaching in Physical Education @p. 35-45), Big Ten Body of Knowledge Symposium. Champaign, 11: Human Kinetics. Imwold, C.H. (1980). The relationship between teaching experience and performance diagnosis of a gymnastics skill. Unpublished doctoral dissertation, University of Pittsburgh, Pittsburgh, PA. Imwold, C.H., & Hoffman, S.J. (1983). Visual recognition of a gymnastics skill by experienced and inexperienced insmctors. Research Quarterly, 54(2), 149-155. Johnson, E.J. (1988). Expertise and decision under uncertainty: Performance and Process. In M.T.H. Chi, R. Glaser, & M.J. Farr (Eds.), The name of expem'se (pp. 209-228. Hillsdale, NJ: Lawrence Erlbaum. Jones, D.F. (1989). A comparative anaIysis of expert and novice basketball coaches during planning and practice sessions. Unpublished doctoral dissertation, University of Pittsburgh. Klein, G. (1990). Knowledge engineering: Beyond expert systems. Informution and Decision Technologies, 16(1), 27-41. Klein, G.A., Calderwood, R. & Macgregor, D. (1989). Critical decision method for eliciting knowledge. IEEE Tramactions on System, Man, and Cybernetics, 19(3), 462-472. Kuipers, B., & Kassirer, J.P. (1984). Causal reasoning in medicine: Analysis of a protocol. Cognitive Science, 8, 363-385. Lawrence, J.A. (1988). Expertise on the bench: Modeling magistrates' judicial decisionmaking. In M. T. H. Chi, R. Glaser, & M. J. Farr (Eds.), The name of expertise (pp.229-259). Hillsdale, NJ: Lawrence Erlbaum. Leinhardt, G. (1983). Novice and expert knowledge of individual student's achievement. Educational Psychologist, 18(3), 165-179. Leinhardt, G. (1983). Expertise in mathematc is teaching. Educational Leadership, April, 28-33. Lesgold, A.M. (1984). Acquiring expertise. In J. R. Anderson & S. M. Kosslyn (Eds.), Tutorials in learning and memory @p. 31-60). San Francisco, CA: W. H. Freeman. Lesgold, A.M., Glaser, R., Rubinson, H., Klopfer, D., Feltovich, P., & Wang, Y. (1988). Expertise in a complex skil1:Diagnosing X-ray pictures. In M. T. H. Chi, R. Glaser, &

94

RR. Leas and M.T.H. Chi

M.J. Farr (Eds.), The nature of expertise (pp.311-342). Hillsdale, NJ: Lawrence Erlbaum. Maglischo, E.W. (1982). Swimming faster: A comprehensive guide to the science of swimming. Chico. CA: Mayfield Publishing. Patel, V.L., & Groen, G.J. (1986). Knowledge based solution strategies in medical reasoning. Cognitive Science, 10, 91-116. Pinheiro, V.E.D. (1989). Motor skill analysis: Diagnostic processes of expet? and novice coaches. Unpublished doctoral dissertation, University of Pittsburgh, Pittsburgh, PA. Putnam, R.T. (1987). Structuring and adjusting content for students: A study of live and simulated tutoring addition. American Educational Research Journal, 24( l), 13-48. Schmidt, R.A. (1991). Motor learning & performance: From principles to practice. Champaign, IL: Human Kinetics. Siedentop, D., & Eldar, E. (1989). Expertise, experience, and effectiveness. Journal of Teaching in Physical Education, 8(3), 254-260. Skrinar, G.S., & Hoffman, S.J. (1979). Effect of outcome information on analytic ability of golf teachers. Perceptual and Motor Skills, 48, 703-708. Taheri, M.A. (1982). Analysis of expertise in planning and interactive decision-making among heolth-relatedphysicalfitness teachers. Unpublished doctoral dissertation, University of Pittsburgh, Pittsburgh, PA. Thomdyke, P.W.(1984). Applications of schema theory in cognitive research. In J. R. Anderson & S , M.Kosslyn (Eds.). Tutorials in learning andmemory (pp.167-191). San Francisco, CA: W.H. Freeman. VanLehn, K. (1989). Problem solving and cognitive skill acquisition. In M. Posner (Ed.), Foundations of cognitive science (pp. 527-579). Cambridge, MA: MIT Press. Womnan, P.M. (1972). Medical diagnosis: An information processing approach. Computers and Biomedical Research, 5, 315-328.

Aknowledgement Preparation of this chapter was supported in part by the Office of Naval Research, Contract #N00014-91-J-1532 to the second author.