- Email: [email protected]
A factor-analytic study of criteria for evaluating student clinicians in speech pathology
JOURNAL
OF COMMUNICATION
DISORDERS
9 (1976), 199-210
A FACTOR-ANALYTIC STUDY OF CRITERIA FOR EVALUATING STUDENT CLINICIANS IN SPEECH PATHOLOGY ALBERT
R. ORATIO,
MA
Division of Speech Pathology-Audiology, Bowling Green State University, Bowling Green, Ohio 43403
This study was designed to determine the most significant criteria used in university training programs to evaluate the therapeutic effectiveness of student clinicians enrolled in clinical practicum. A total of 152 supervisors in 53 ASHA accredited training programs used 40 criteria to evaluate 207 clinicians engaged in speech and language therapy. Factor analysis yielded two distinct dimensions, Technical Skills and Interpersonal Relationship Factors, and indicated that 18 variables contributed most significantly to the evaluative process. Student clinicians were consistently rated higher on variables related to the interpersonal dimension than on those related to technical skills. The identification and structure of the 18 variables suggests that both technical and interpersonal skills are widely regarded as essential to therapeutic effectiveness. The results provide insight into the nature of clinical evaluation and behavior and suggest a restructuring of criteria for evaluating the process of therapeutic treatment .
Introduction Developing clinical competencies in students is a major function of university training programs in speech pathology. Competencies begin to develop during enrollment in a formal academic program of study and continue following completion of that program. At least three orderly processes contribute to the development of clinical competence: (1) academic course work, (2) supervised clinical practice, and (3) independent work experience (Clinical Fellowship Year). Finally, after successful completion of the National Examination in Speech Pathology and Audiology, professionals become recognized by the American Speech and Hearing Association (ASHA) for national certification. Although each of the above areas presents its unique problems, supervised clinical practice has been considered a neglected, but vital area for fostering the development of clinical skills (Perkins, 1962; Halfond, 1964; Van Riper, 1965; Ward and Webster, 1965a,b; Darley, 1969). Since 1964, ASHA seminars have taken place in order to upgrade clinical supervision (Villarreal, 1964; Kleffner, 1964; Miner, 1967). During a conference on guidelines for supervision of clinical practicum (Vallarreal, 1964) it was reported that “Objective evidence of the validity of criteria for the evaluation of clinical competence is severely limited” (p. 15). This lack of criteria is still evident today. Anderson (1973, 1974a,b) suggested that the lack of objective criteria for evaluating student clinicians
o American
Elsevier Publishing
Company,
Inc.,
1976
199
ALBERT R. ORATIO
200
Haller, 1967; and Volz, Turton, 1973; et al., represent major supervisory tools for describing and evaluating clinical processes in speech pathology. Shriberg et al. (1975) developed a 3%item clinician appraisal instrument that yielded high reliability coefficients among supervisors who used the instrument. This evaluation scale represents one of the only tools of its kind to be subjected to statistical scrutiny. Subjective procedures used to construct descriptive and evaluative instruments in speech pathology have resulted in highly speculative and equivocal criteria. These criteria are primarily based upon each author’s a priori assumptions about the nature of clinical competence prior to clinical fact. Evaluative criteria derived by extracting data from clinical performance are presently nonexistent in the speech pathology literature. Objective identification and comprehension of the underlying constructs or competencies in the therapeutic process may enhance prediction and control of the therapeutic outcome. The division of competency-based criteria into discrete areas, such as diagnosis, treatment, interpersonal and professional relations, has been suggested by Diedrich (1969) and Perkins et al. (1970) and received statistical support by Shriberg et al. (1975). These content areas may serve as tentative parameters for defining the construct of “clinical competence. ” The present study explores evaluative criteria related to one aspect of clinical competence: the parameter of therapeutic treatment. Therapeutic treatment is considered the most important parameter of competence in clinical training, and therefore was selected as the focus of the present investigation. The purpose of this study was to determine the variables that contribute most significantly to supervisors’ evaluations of student clinicians during the direct provision of therapeutic treatment. This study represents an initial attempt in speech pathology to objectively validate and define evaluative criteria throughaposteriori experimentation. Method Preparation
of Evaluative
Instrument
A prerequisite to this study was the construction of an evaluation form that would be broadly representative of those used by clinical supervisors in university training programs throughout the country. In order to devise such a form, 46 university training programs accredited by the Education and Training Board of
A STUDY
OF CRITERIA
FOR EVALUATING
STUDENT
CLINICIANS
201
the American Boards of Examiners in Speech Pathology and Audiology were randomly chosen from the American Speech and Hearing Association Directory (Johnson, 1975). l This procedure allowed for a broad geographical representation of responses, which included the 33 states plus the District of Columbia listed in the directory under accredited training programs. Letters were sent to all 46 programs soliciting the criteria used by the department of speech pathology for evaluating student clinicians enrolled in clinical practicum. Of the 46 programs selected, 30 replies were received; five were eliminated for the following reasons: two misunderstood the request and returned irrelevant information, two returned criteria used for evaluating audiological skills, and one could not provide the requested information. The remaining 25 replies yielded 25 evaluation forms, totaling 382 criterion statements. A geographical cross-section of 18 states was represented by these returns. It was first necessary to reduce the original pool of criterion statements by eliminating all statements pertaining to parameters other than the provision of services through direct client contact. For example, pretherapy activities such as lesson plan preparation and posttherapy activities such as parent counseling and report writing were considered irrelevant to the present study. This procedure reduced the number of criterion statements to 132. Next it was necessary to eliminate all overlapping statements, reduce ambiguity, and ensure semantic purity. Statements that included more than one competency were rewritten by the experimenter as separate statements with each containing only one criterion, and all obviously redundant statements were omitted. This procedure reduced the number of criterion statements from 132 to 66. In order to verify the experimenter’s judgments, the 66 statements were submitted for inspection to three groups (N = 6 in each group) representing different levels of academic and professional experience: master’s degree candidates, doctoral candidates, and faculty. Each group was asked to inspect the 66 statements for overlap, ambiguity, and semantic purity and to provide both oral and written feedback. This information was used by the experimenter to rewrite the statements parsimoniously and clearly while retaining the vital concepts inherent in the original set of variables. This process further reduced the statements to a total of 40. These 40 items were used to construct a preliminary evaluative instrument (Table 1) for use in this study, and represent the variables that yielded data for factor analysis. Each item was written so that it could be scored by subjects on a seven point scale (l-7), with one being the most favorable rating and seven the least.
‘List of training request.
programs
and criteria
obtained
from each program
will be made available
upon
ALBERT
202
Forty Criterion
R. ORATIO
TABLE 1 Used to Construct the Prelimitran/ Evaluation Present Study _
Statements
Instrument
Used in the
Scale Number
Criterion
1. Organizes the physical environment responses 2.
Uses every opportunity
3.
Modifies planned strategy
4.
Correctly
5.
Presents a consistent
6.
Uses appropriate
7.
Uses a variety of materials
8. 9.
(Hi)
statement of the therapy room to maximize target
to obtain target responses to maximize
1234567
from the client
1234567
target responses
1234567
models the target response
Presents materials
1234567
model of the target response
materials
in relation to the therapeutic
1234567
goal
1234567 1234567
systematically
Uses materials creatively therapeutic activity
1234567
in an effort to stimulate
environmental
resources
1234567 1234567
Uses available
11.
Establishes teractions
rapport
12.
Maintains
a confident
13.
Shows respect for the client as evidenced behavior
14.
Uses appropriate with the client
15.
Communicates
16.
Demonstrates
17.
Changes therapeutic procedure when necessary in order to meet client needs
1234567
18.
Resolves unexpected
1234567
19.
Enforces limits on the client’s behavior,
20.
Shows emotional
21.
Maintains
22.
Shows appropriate look
23.
Demonstrates
24.
Utilizes therapeutic responses
25.
Maintains therapeutic
by productive
blackboard,
in the
10.
as evidenced
(furniture,
client interest
etc.)
verbal and nonverbal
in-
1234567
image in the clinical situation
speech characteristics:
at the client’s cognitive sensitivity
by both verbal and nonverbal
1234567
rate, pitch, and volume, in working
1234567
and linguistic
1234567
levels
1234567
to the client’s needs
problems
1234567
when necessary
1234567
stability during therapy
appropriate
personal
1234567
appearance
attitudes during therapy,
i.e., enthusiasm,
positive out-
activities which provide for maximum number of target clinician/client
1234567 1234567
punctuality
efficient session
1234567
verbalization
ratio
throughout
the
1234567 1234567
(LoI
A STUDY
OF CRITERIA
FOR EVALUATING TABLE
STUDENT
CLINICIANS
203
1 (continued) Scale
Number 26.
Criterion
Works toward therapeutic
statement
goals in an organized
21.
Verbalizes
the progression
28.
Presents accurate
29.
Shows willingness
30.
Uses techniques
31.
Utilizes various modalities
(I-Ii) 1234567
manner
1234567
of therapy to the client
instructions
1234567
to the client
to experiment
with new or original
techniques
(i.e., visual, auditory,
Encourages
33.
Shows appropriate
client’s self-evaluation
level of clinical involvement
34.
Applies theoretical
knowledge
35.
Uses effective reinforcers
36.
Appropriately
31.
Uses effective schedule
38.
Accomplishes
39.
Implements
40.
Utilizes appropriate
reinforces
tactile-kinesthetic)
1234567 1234567
of responses during therapy
of the disorder to therapeutic
practice
1234567 1234567 1234567
the client’s approximations of reinforcement
the session’s carryover
1234567 1234567
which are suitable to the client
32.
(Lo)
1234561 1234567 1234567
goals
procedures
to the target response
in therapy
closure of the therapy session
1234567 1234567
Subjects A total of 440 university supervisors, 5 from each of the 88 accredited training programs listed in the ASH_4 Directory (Johnson, 1975), were requested to serve as subjects. Data Collection A mail questionnaire method of obtaining information was used to collect data for the study. Five copies of the evaluation form (Table 1) were sent to each of the 88 accredited training programs, yielding a total of 440 forms. Subjects at each university were requested to use the rating instrument to evaluate a total of five student clinicians engaged in clinical practicum. To maximize supervisor-clinician participation in the study, the following three guidelines were indicated: (1) more than one supervisor from a given setting may participate in the evaluation of the five clinicians; (2) five supervisors, each evaluating one clinician, would provide the most favorable condition for the study;
204
ALBERT
R. ORATIO
and (3) repeated evaluations of the same clinician by one or more supervisors were considered unacceptable. Following completion of the evaluations the subjects were requested to indicate the number of supervisors who participated and to return all information in a self-addressed, stamped envelope within 21 days of receipt. Shortly after this deadline a follow-up letter was sent to all 88 programs urging participation in the study and return of the research materials.
Factor Analysis Computer program BMDX72 (Dixon, 1969) was used to execute the method of principal components for factor analysis. The factor analysis was used to determine the degree to which the variables contributed to the evaluation process and the intercorrelations of each variable. An eigenvalue cutoff of 1.0 was used in building the correlation matrix. Orthogonal rotation was requested for three factors using “normal-varimax” criterion (Kaiser, 1958). Criteria for using variables to define factors were set at 0.30 significance level or above. Results A total of 207 evaluation forms were completed and received from 152 university supervisors in 53 ASHA accredited training programs throughout the country. When transformed into percentages these returns indicated that the number of responses received represented 47% of the potential responses, the number of supervisors who participated represented 35% of the potential respondents, and the number of programs involved represented 60% of the total number of accredited training programs. Geographically, 20 states throughout the country were represented. Of the 152 subjects who participated, 23 reported their sole function within the university to be that of clinical supervisor. The remaining 129 subjects’ duties were distributed over teaching, research, and supervision. A principal-components factor analysis performed on the matrix for each of the 40 variables resulted in two factors that accounted for 58% of the total variance. The rotated factor matrix, means, and standard deviations for each of the 40 variables are presented in Table 2. The two factors that emerged from the analysis of the data are named, described below, and displayed in Table 3. Factor I (Technical Skills) was defined by a total of 10 variables, with high negative loadings on target response variables 2 and 3 and reinforcement variables 36 and 37. Therapeutic resourcefulness was characterized by variables 9, 17, and 18. Variable 18 had the lowest loading among these three variables, and also showed a lower mean rating by supervisors. Variables 34,38, and 39 seemed to reflect knowledge of therapeutic theory related to remediation. Variables 34 and 38 showed high negative loadings, while variable 39 showed a lower negative
A STUDY OF CRITERIA
FOR EVALUATING
STUDENT
CLINICIANS
205
TABLE 2 Factor Loadings, Means, and Standard Deviations for All 40 Variables. Based on 152 Supervisors’ Evaluations of 207 Clinicians Variable
2 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
I
II
Mean*
SD
-0.47 1 -0.719 -0.763 -0.516 -0.508 -0.540 -0.607 -0.590 -0.739 -0.547 -0.453 -0.462 -0.347 -0.202 -0.353 -0.507 -0.757 -0.681 -0.536 -0.398 -0.303 -0.290 -0.482 -0.654 -0.549 -0.641 -0.602 -0.619 -0.604 -0.577 -0.512 -0.582 -0.362 -0.762 -0.622 -0.730 -0.736 -0.734 -0.638 -0.500
0.379 0.298 0.246 0.496 0.555 0.528 0.344 0.406 0.344 0.350 0.710 0.656 0.763 0.737 0.728 0.622 0.334 0.295 0.340 0.654 0.598 0.778 0.423 0.496 0.491 0.544 0.384 0.530 0.431 0.582 0.451 0.385 0.597 0.359 0.553 0.387 0.393 0.340 0.311 0.382
2.33 2.71 2.74 1.99 2.03 2.05 2.49 2.37 2.60 2.50 1.97 2.11 1.85 2.01 2.17 2.11 2.53 2.72 2.43 1.80 1.69 1.85 1.76 2.48 2.47 2.30 2.70 2.33 2.32 2.17 2.50 2.79 1.93 2.63 2.28 2.44 2.55 2.34 2.64 2.29
1.25 1.31 1.45 1.26 1.23 1.16 1.46 1.33 1.49 1.49 1.23 1.23 1.15 1.12 1.21 1.33 1.44 1.45 1.19 1.11 1.04 1.15 1.10 1.36 1.30 1.35 1.57 1.24 1.54 1.26 1.42 1.60 1.27 1.50 1.40 1.43 1.47 1.33 1.49 1.38
*l constitutes the highest rating and 7 the lowest.
206
ALBERT
R. ORATIO
TABLE 3 Variables Defining Factors I and II with Loadings, Means, and Standard Deviations Table 2 Factors I-Technical Skills Factor (2) Uses every opportunity to obtain target responses from the client (3) Modifies planned strategy to maximize target responses (9) Uses materials creatively to stimulate client interest (17) Changes therapeutic procedure to meet client needs (18) Resolves unexpected problems in therapy (34) Applies theoretical knowledge of disorder to therapeutic practice (36) Appropriately reinforces client’s approximations to target response (37) Uses effective schedule of reinforcement (38) Accomplishes the goals of the session (39) Implements carryover procedures in therapy II-Interpersonal Relationship Factor (11) Establishes rapport as evidenced verbally and nonverbally (13) Shows respect for the client as evidenced verbally and nonverbally (14) Uses appropriate speech characteristics: rate, pitch, volume, etc. (15) Communicates at the client’s cognitive and linguistic levels (20) Shows emotional stability during therapy (2 1) Maintains appropriate personal appearance (22) Shows appropriate attitudes in therapy, i.e., enthusiasm, etc. (33) Shows appropriate level of clinical involvement *l constitutes
Extracted from
Loading
Mean*
SD
-0.719
2.71
1.31
-0.763 -0.739 -0.757 -0.681 -0.762
2.74 2.60 2.53 2.72 2.63
1.45 1.49 1.44 1.45 1.50
-0.730
2.44
1.43
-0.736 -0.734 -0.638
2.55 2.34 2.64
1.47 1.33 1.49
0.710 0.763
1.97 1.85
1.23 1.15
0.737
2.01
1.12
0.728 0.654 0.598 0.778
2.17 1.80 1.69 1.85
1.21 1.11 1.04 1.15
0.597
1.93
1.27
the highest rating and 7 the lowest.
loading and supervisory mean rating. Out of the 18 variables that define factors I and II, clinicians received the lowest mean rating from supervisors on variable 3. Factor II (Interpersonal Relationship) consisted of eight variables and was clearly defined by variables 11, 13, and 22, all of which showed high positive loadings. Interpersonal communication variables 14 and 15 also showed high positive loadings for variables within this factor. Three intrapersonal variables, 20,2 1, and 33, contained the lowest loadings in the second factor. Out of the 18 variables that define factors I and II, clinicians received the highest mean rating from supervisors on variable 21. Based on all 40 variables used in the evaluation process, university supervisors rated clinicians highest on variables 21 (maintains appropriate personal appearance) and 23 (demonstrates punctuality) and lowest on variables 3 (modifies planned strategy to maximize target responses) and 32 (encourages client’s selfevaluation of responses) as indicated by mean ratings listed in Table 2. Frequency distributions for these variables (Fig. 1) indicate that 80% of the clinicians were
A STUDY
OF CRITERIA
FOR EVALUATING
STUDENT
CLINICIANS
201
120 II5
-
110 IO5 -
\
+ -+l ----. -4
VARIABLE 21 23 32 3
loo
\ \ \
2 (HI)
3 SUPERVISORY
4
5 RATING
6
7 (Lo)
Fig. 1. Frequency distributions for variables on which clinicians were rated highest (variables 21 and 23) and lowest (variables 3 and 32) by university supervisors. Based on all 40 variables.
rated at levels 1 and 2 on variables 21 and 23, whereas less than 50% of the 207 clinicians were rated at these levels on variables 3 and 32.
The factor structure identified in the present study indicated that two major behavioral dimensions are regarded by supervisors as critical to the process of therapeutic treatment: technical skills and interpersonal relationship dimensions. This finding corroborates Shriberg et al.‘s (1975) division of evaluative criteria
208
ALBERT
R. ORATIO
into professional-technical and interpersonal skill domains, and suggests a restructuring of criteria based upon the 18 variables that were found to be correlated within these two dimensions. These variables contribute most significantly to supervisors’ evaluations of student clinicians, and therefore should be used to construct a revised instrument for evaluating the process of therapeutic treatment. The correlation of variables within factors I and II suggests that the greatest disparity in clinical rating will be on variables between factors rather than on variables within the first or second factor. The factor which has been labeled “Technical Skills” consists of 10 variables that are related to achieving target responses, utilizing effective reinforcement, exhibiting therapeutic resourcefulness, and an understanding of theory related to remediation. These criteria appear to have application for highly structured and direct as well as indirect therapeutic styles, and therefore justifiably allow for these individual differences. The factor labeled “Interpersonal Relationship” consists of eight variables that are related to rapport, respect, interpersonal communications, and various intrapersonal conditions. The establishment of rapport and other interpersonal behaviors has often been viewed as essential in the initial stages of therapy. The emergence of this dimension, however, suggests that interpersonal and affective behaviors constitute an ongoing dynamic process widely regarded as critical to therapeutic effectiveness. Furthermore, the emergence of these eight criteria serve to raise questions about the kinds of professional training required for competent therapeutic interaction. Course work in sensitivity training, group dynamics, and interpersonal communication may enhance interpersonal skills. Kaplan and Dreyer (1974) have reported an effective method for developing interpersonal skills and increasing the self-awareness of student clinicians during therapy. Based on supervisors’ mean ratings of student clinicians, however, there is greater need to develop technical therapeutic skill. Student clinicians were consistently rated higher on variables related to interpersonal skills than on those related to technical abilities. In fact, clinicians were rated highest on clinical dress and punctuality and lowest on modifying the therapeutic strategy to maximize target responses and encouraging the client’s self-evaluation of responses. These latter two components appear to be more sophisticated technical skills that may develop within the practicum experience, while the former are easily modifiable and may in fact serve as prerequisites to entering clinical practicum. The present study defines the evaluation of therapeutic treatment as tentatively consisting of 18 variables correlated within two dimensions. These dimensions were shown to account for 58% of the total variance in the therapeutic evaluation. It is interesting to speculate a third factor based on the remaining 22 variables combined with marker variables. Such a factor may consist of numerous techniques such as variables 27 (verbalizes the progression of therapy to the client), 31 (utilizes various modalities, i.e., visual, auditory, tactile-kinesthetic), and 32
A STUDY
OF CRITERIA
FOR EVALUATING
STUDENT
CLINICIANS
209
(encourages client’s self-evaluation of responses). Numerous investigations have shown attraction to have a positive influence on therapy-like learning (Sapolsky, 1960, 1965; Lott and Lott, 1966). Perhaps variables related to client attraction for the clinician, as conveyed verbally and nonverbally, would mark a third factor. An hypothesis-testing analysis would be necessary to determine whether such undiscovered dimensions exist. It remains to be determined whether the present factorial structure would be maintained in evaluating therapy for the disorder areas of articulation, voice, stuttering, and language, independently. As treatment procedures for each disorder become more highly specialized, evaluative criteria specific to therapeutic processes within each area may be defined. This hypothesis provides a direction for further research. Information is most needed on defining the construct of “clinical competence.” Anderson (1973, 1974b) indicated that the identification of clinical competencies is most important for structuring the operations of the clinical supervisor. The investigation of “clinical competence” must ultimately include study of the reciprocation of interaction between supervisor, clinician, and client. Clinical supervision, as an area of investigation, defies objective measure. The current findings, combined with those of past research, may be instrumental in studying the complexities of the construct of “clinical competence.” The author wishes to acknowledge the assistance of George Herman, William Hinkle, Stephen Hood, and Melvin Hyman in critiquing various portions of the manuscript. Appreciation is also expressed to Raymond Tucker who served as consultant for statistics and experimental design. Special thanks to Janet Watson for typing the manuscript and my wife, Suzanne Gratio, for her help in preparing and organizing the research materials.
References Anderson, J. L. Supervision: The neglected component of our profession. In: L. J. Turton (Ed.), Proceedings afa workshop on supervision in speech pathology.Ann Arbor, Mich .: University of Michigan, 1973. Anderson, J. L. Supervision of school speech, hearing, and language programs--an emerging role. ASHA, 1974a, 16, 7-10. Anderson, J. L. Supervision of the clinical process in speech pathology: Issues and practice. Short course presented at the Annual Convention of the American Speech and Hearing Association, Las Vegas, Nevada, 1974b. Boone, D., Prescott, T. Content and sequence analysis of speech and hearing therapy. ASHA, 1972, 14 58-62. Brown, E. A university’s approach to improving supervision. ASHA, 1967, 9, 476-479. Darley, F. L. Clinical training for full-time clinical services: a neglected obligation. ASHA, 1969,11, 143-148. Diedrich, W. M. Assessment of the clinical process. J. Kans. Speech Hearing Assoc., 1969, l-8. Dixon, W. .I. (Ed.). BMD Biomedical computer programs X-series supplement. Berkeley and Los Angeles, Calif.: University of California Press, 1969.
210
Halfond,
ALBERT
M. Clinical supervision-stepchild
R. ORATIO
in training.
ASHA,
1964, 6, 441-444.
Haller, R. M. Supervisor’s criteria for evaluating student’s performance in clinical practicum activities. ASHA, 1967, 9, 479-481, Johnson, K. 0. American speech and hearing associafion directory, Danville, Ill.: The Interstate Printers and Publishers, 1975. Johnson, T. The development of a multidimensional scoring system for obvserving the clinical process in speech pathology. Unpublished doctoral dissertation, University of Kansas, 1969. Kaiser, H. The Varimax criterion for analytic rotation in factor analysis. Psychometrika, 1958, 23, 187-200. Kaplan, N. R., Dreyer, D. E. The effect of self-awareness training on student speech pathologistclient relationships. J. Commun. Dis., 1974, 7, 329-342. Kleffner, F. Seminar on guidelines for fhe internship year. Washington, D. C.: American Speech and Hearing Association, 1964. Klevans, D., Volz, H. Development of a clinical evaluation procedure. ASHA, 1974,16,489-491. Lott, A. J., Lott, B. E. Group cohesiveness and individual learning. J. Educ. Psychol., 1966, 57, 61-73. MacLeatie, E. Appraisal form for speech and hearing therapists. J. Speech Hearing Dis., 1958,23, 612614. Miner, A. Standards for quality supervision of clinical practicum. ASHA, 1967, 9, 471-472. Perkins, W. Our professiorr_What is it? ASHA, 1962, 4, 339-344. Perkins, W., Shelton, R., Studebaker, G., Goldstein, R. The national examinations in speech pathology and audiology: philosophy and operation. ASHA, 1970, 12, 175181. Sapolsky, A. Effect of interpersonal relationships upon verbal conditioning. J. Abnorm. Sot. Psychol., 1960, 60, 241-246. Sapolsky, A. Relationship between patient-doctor compatability, mutual perception, and outcome of treatment. J. Abnorm. Psychol., 1965, 70, 7676. Schubert, G., Miner, A., Till, J. The analysis of behavior of clinicians (ABC) system. Grand Forks, North Dakota: University of North Dakota, 1973. Shriberg, L. D., Filley, F. S., Hayes, D. M., Kwiatkowski, J., Schatz, J. A., Simmons, K. M., Smith, M. E. The Wisconsin procedure for appraisal of clinical competence (w-pact): model and data. ASHA, 1975, 17, 158-165. Turton, L. J. (Ed.). Proceedings of a workshop on supervision in speech pathology. Ann Arbor, Mich.: University of Michigan, 1973. Van Riper, C. Supervision of clinical practice. ASHA, 1965, 3, 75-77. Villarreal, J. Seminar on guidelines for supervision of clinical practicum in programs of training for speech pathologists and audiologists. Washington, D.C.: American Speech and Hearing Association, 1964. Ward, L., Webster, E. The trainingof clinical personnel: I. issues in conceptualization. ASHA, 1965a, 7, 38-40. Ward, L., Webster, E. The training of clinical personnel: Il. a concept of clinical preparation. ASHA. 1965b, 7, 103-106.
OF COMMUNICATION
DISORDERS
9 (1976), 199-210
A FACTOR-ANALYTIC STUDY OF CRITERIA FOR EVALUATING STUDENT CLINICIANS IN SPEECH PATHOLOGY ALBERT
R. ORATIO,
MA
Division of Speech Pathology-Audiology, Bowling Green State University, Bowling Green, Ohio 43403
This study was designed to determine the most significant criteria used in university training programs to evaluate the therapeutic effectiveness of student clinicians enrolled in clinical practicum. A total of 152 supervisors in 53 ASHA accredited training programs used 40 criteria to evaluate 207 clinicians engaged in speech and language therapy. Factor analysis yielded two distinct dimensions, Technical Skills and Interpersonal Relationship Factors, and indicated that 18 variables contributed most significantly to the evaluative process. Student clinicians were consistently rated higher on variables related to the interpersonal dimension than on those related to technical skills. The identification and structure of the 18 variables suggests that both technical and interpersonal skills are widely regarded as essential to therapeutic effectiveness. The results provide insight into the nature of clinical evaluation and behavior and suggest a restructuring of criteria for evaluating the process of therapeutic treatment .
Introduction Developing clinical competencies in students is a major function of university training programs in speech pathology. Competencies begin to develop during enrollment in a formal academic program of study and continue following completion of that program. At least three orderly processes contribute to the development of clinical competence: (1) academic course work, (2) supervised clinical practice, and (3) independent work experience (Clinical Fellowship Year). Finally, after successful completion of the National Examination in Speech Pathology and Audiology, professionals become recognized by the American Speech and Hearing Association (ASHA) for national certification. Although each of the above areas presents its unique problems, supervised clinical practice has been considered a neglected, but vital area for fostering the development of clinical skills (Perkins, 1962; Halfond, 1964; Van Riper, 1965; Ward and Webster, 1965a,b; Darley, 1969). Since 1964, ASHA seminars have taken place in order to upgrade clinical supervision (Villarreal, 1964; Kleffner, 1964; Miner, 1967). During a conference on guidelines for supervision of clinical practicum (Vallarreal, 1964) it was reported that “Objective evidence of the validity of criteria for the evaluation of clinical competence is severely limited” (p. 15). This lack of criteria is still evident today. Anderson (1973, 1974a,b) suggested that the lack of objective criteria for evaluating student clinicians
o American
Elsevier Publishing
Company,
Inc.,
1976
199
ALBERT R. ORATIO
200
Haller, 1967; and Volz, Turton, 1973; et al., represent major supervisory tools for describing and evaluating clinical processes in speech pathology. Shriberg et al. (1975) developed a 3%item clinician appraisal instrument that yielded high reliability coefficients among supervisors who used the instrument. This evaluation scale represents one of the only tools of its kind to be subjected to statistical scrutiny. Subjective procedures used to construct descriptive and evaluative instruments in speech pathology have resulted in highly speculative and equivocal criteria. These criteria are primarily based upon each author’s a priori assumptions about the nature of clinical competence prior to clinical fact. Evaluative criteria derived by extracting data from clinical performance are presently nonexistent in the speech pathology literature. Objective identification and comprehension of the underlying constructs or competencies in the therapeutic process may enhance prediction and control of the therapeutic outcome. The division of competency-based criteria into discrete areas, such as diagnosis, treatment, interpersonal and professional relations, has been suggested by Diedrich (1969) and Perkins et al. (1970) and received statistical support by Shriberg et al. (1975). These content areas may serve as tentative parameters for defining the construct of “clinical competence. ” The present study explores evaluative criteria related to one aspect of clinical competence: the parameter of therapeutic treatment. Therapeutic treatment is considered the most important parameter of competence in clinical training, and therefore was selected as the focus of the present investigation. The purpose of this study was to determine the variables that contribute most significantly to supervisors’ evaluations of student clinicians during the direct provision of therapeutic treatment. This study represents an initial attempt in speech pathology to objectively validate and define evaluative criteria throughaposteriori experimentation. Method Preparation
of Evaluative
Instrument
A prerequisite to this study was the construction of an evaluation form that would be broadly representative of those used by clinical supervisors in university training programs throughout the country. In order to devise such a form, 46 university training programs accredited by the Education and Training Board of
A STUDY
OF CRITERIA
FOR EVALUATING
STUDENT
CLINICIANS
201
the American Boards of Examiners in Speech Pathology and Audiology were randomly chosen from the American Speech and Hearing Association Directory (Johnson, 1975). l This procedure allowed for a broad geographical representation of responses, which included the 33 states plus the District of Columbia listed in the directory under accredited training programs. Letters were sent to all 46 programs soliciting the criteria used by the department of speech pathology for evaluating student clinicians enrolled in clinical practicum. Of the 46 programs selected, 30 replies were received; five were eliminated for the following reasons: two misunderstood the request and returned irrelevant information, two returned criteria used for evaluating audiological skills, and one could not provide the requested information. The remaining 25 replies yielded 25 evaluation forms, totaling 382 criterion statements. A geographical cross-section of 18 states was represented by these returns. It was first necessary to reduce the original pool of criterion statements by eliminating all statements pertaining to parameters other than the provision of services through direct client contact. For example, pretherapy activities such as lesson plan preparation and posttherapy activities such as parent counseling and report writing were considered irrelevant to the present study. This procedure reduced the number of criterion statements to 132. Next it was necessary to eliminate all overlapping statements, reduce ambiguity, and ensure semantic purity. Statements that included more than one competency were rewritten by the experimenter as separate statements with each containing only one criterion, and all obviously redundant statements were omitted. This procedure reduced the number of criterion statements from 132 to 66. In order to verify the experimenter’s judgments, the 66 statements were submitted for inspection to three groups (N = 6 in each group) representing different levels of academic and professional experience: master’s degree candidates, doctoral candidates, and faculty. Each group was asked to inspect the 66 statements for overlap, ambiguity, and semantic purity and to provide both oral and written feedback. This information was used by the experimenter to rewrite the statements parsimoniously and clearly while retaining the vital concepts inherent in the original set of variables. This process further reduced the statements to a total of 40. These 40 items were used to construct a preliminary evaluative instrument (Table 1) for use in this study, and represent the variables that yielded data for factor analysis. Each item was written so that it could be scored by subjects on a seven point scale (l-7), with one being the most favorable rating and seven the least.
‘List of training request.
programs
and criteria
obtained
from each program
will be made available
upon
ALBERT
202
Forty Criterion
R. ORATIO
TABLE 1 Used to Construct the Prelimitran/ Evaluation Present Study _
Statements
Instrument
Used in the
Scale Number
Criterion
1. Organizes the physical environment responses 2.
Uses every opportunity
3.
Modifies planned strategy
4.
Correctly
5.
Presents a consistent
6.
Uses appropriate
7.
Uses a variety of materials
8. 9.
(Hi)
statement of the therapy room to maximize target
to obtain target responses to maximize
1234567
from the client
1234567
target responses
1234567
models the target response
Presents materials
1234567
model of the target response
materials
in relation to the therapeutic
1234567
goal
1234567 1234567
systematically
Uses materials creatively therapeutic activity
1234567
in an effort to stimulate
environmental
resources
1234567 1234567
Uses available
11.
Establishes teractions
rapport
12.
Maintains
a confident
13.
Shows respect for the client as evidenced behavior
14.
Uses appropriate with the client
15.
Communicates
16.
Demonstrates
17.
Changes therapeutic procedure when necessary in order to meet client needs
1234567
18.
Resolves unexpected
1234567
19.
Enforces limits on the client’s behavior,
20.
Shows emotional
21.
Maintains
22.
Shows appropriate look
23.
Demonstrates
24.
Utilizes therapeutic responses
25.
Maintains therapeutic
by productive
blackboard,
in the
10.
as evidenced
(furniture,
client interest
etc.)
verbal and nonverbal
in-
1234567
image in the clinical situation
speech characteristics:
at the client’s cognitive sensitivity
by both verbal and nonverbal
1234567
rate, pitch, and volume, in working
1234567
and linguistic
1234567
levels
1234567
to the client’s needs
problems
1234567
when necessary
1234567
stability during therapy
appropriate
personal
1234567
appearance
attitudes during therapy,
i.e., enthusiasm,
positive out-
activities which provide for maximum number of target clinician/client
1234567 1234567
punctuality
efficient session
1234567
verbalization
ratio
throughout
the
1234567 1234567
(LoI
A STUDY
OF CRITERIA
FOR EVALUATING TABLE
STUDENT
CLINICIANS
203
1 (continued) Scale
Number 26.
Criterion
Works toward therapeutic
statement
goals in an organized
21.
Verbalizes
the progression
28.
Presents accurate
29.
Shows willingness
30.
Uses techniques
31.
Utilizes various modalities
(I-Ii) 1234567
manner
1234567
of therapy to the client
instructions
1234567
to the client
to experiment
with new or original
techniques
(i.e., visual, auditory,
Encourages
33.
Shows appropriate
client’s self-evaluation
level of clinical involvement
34.
Applies theoretical
knowledge
35.
Uses effective reinforcers
36.
Appropriately
31.
Uses effective schedule
38.
Accomplishes
39.
Implements
40.
Utilizes appropriate
reinforces
tactile-kinesthetic)
1234567 1234567
of responses during therapy
of the disorder to therapeutic
practice
1234567 1234567 1234567
the client’s approximations of reinforcement
the session’s carryover
1234567 1234567
which are suitable to the client
32.
(Lo)
1234561 1234567 1234567
goals
procedures
to the target response
in therapy
closure of the therapy session
1234567 1234567
Subjects A total of 440 university supervisors, 5 from each of the 88 accredited training programs listed in the ASH_4 Directory (Johnson, 1975), were requested to serve as subjects. Data Collection A mail questionnaire method of obtaining information was used to collect data for the study. Five copies of the evaluation form (Table 1) were sent to each of the 88 accredited training programs, yielding a total of 440 forms. Subjects at each university were requested to use the rating instrument to evaluate a total of five student clinicians engaged in clinical practicum. To maximize supervisor-clinician participation in the study, the following three guidelines were indicated: (1) more than one supervisor from a given setting may participate in the evaluation of the five clinicians; (2) five supervisors, each evaluating one clinician, would provide the most favorable condition for the study;
204
ALBERT
R. ORATIO
and (3) repeated evaluations of the same clinician by one or more supervisors were considered unacceptable. Following completion of the evaluations the subjects were requested to indicate the number of supervisors who participated and to return all information in a self-addressed, stamped envelope within 21 days of receipt. Shortly after this deadline a follow-up letter was sent to all 88 programs urging participation in the study and return of the research materials.
Factor Analysis Computer program BMDX72 (Dixon, 1969) was used to execute the method of principal components for factor analysis. The factor analysis was used to determine the degree to which the variables contributed to the evaluation process and the intercorrelations of each variable. An eigenvalue cutoff of 1.0 was used in building the correlation matrix. Orthogonal rotation was requested for three factors using “normal-varimax” criterion (Kaiser, 1958). Criteria for using variables to define factors were set at 0.30 significance level or above. Results A total of 207 evaluation forms were completed and received from 152 university supervisors in 53 ASHA accredited training programs throughout the country. When transformed into percentages these returns indicated that the number of responses received represented 47% of the potential responses, the number of supervisors who participated represented 35% of the potential respondents, and the number of programs involved represented 60% of the total number of accredited training programs. Geographically, 20 states throughout the country were represented. Of the 152 subjects who participated, 23 reported their sole function within the university to be that of clinical supervisor. The remaining 129 subjects’ duties were distributed over teaching, research, and supervision. A principal-components factor analysis performed on the matrix for each of the 40 variables resulted in two factors that accounted for 58% of the total variance. The rotated factor matrix, means, and standard deviations for each of the 40 variables are presented in Table 2. The two factors that emerged from the analysis of the data are named, described below, and displayed in Table 3. Factor I (Technical Skills) was defined by a total of 10 variables, with high negative loadings on target response variables 2 and 3 and reinforcement variables 36 and 37. Therapeutic resourcefulness was characterized by variables 9, 17, and 18. Variable 18 had the lowest loading among these three variables, and also showed a lower mean rating by supervisors. Variables 34,38, and 39 seemed to reflect knowledge of therapeutic theory related to remediation. Variables 34 and 38 showed high negative loadings, while variable 39 showed a lower negative
A STUDY OF CRITERIA
FOR EVALUATING
STUDENT
CLINICIANS
205
TABLE 2 Factor Loadings, Means, and Standard Deviations for All 40 Variables. Based on 152 Supervisors’ Evaluations of 207 Clinicians Variable
2 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
I
II
Mean*
SD
-0.47 1 -0.719 -0.763 -0.516 -0.508 -0.540 -0.607 -0.590 -0.739 -0.547 -0.453 -0.462 -0.347 -0.202 -0.353 -0.507 -0.757 -0.681 -0.536 -0.398 -0.303 -0.290 -0.482 -0.654 -0.549 -0.641 -0.602 -0.619 -0.604 -0.577 -0.512 -0.582 -0.362 -0.762 -0.622 -0.730 -0.736 -0.734 -0.638 -0.500
0.379 0.298 0.246 0.496 0.555 0.528 0.344 0.406 0.344 0.350 0.710 0.656 0.763 0.737 0.728 0.622 0.334 0.295 0.340 0.654 0.598 0.778 0.423 0.496 0.491 0.544 0.384 0.530 0.431 0.582 0.451 0.385 0.597 0.359 0.553 0.387 0.393 0.340 0.311 0.382
2.33 2.71 2.74 1.99 2.03 2.05 2.49 2.37 2.60 2.50 1.97 2.11 1.85 2.01 2.17 2.11 2.53 2.72 2.43 1.80 1.69 1.85 1.76 2.48 2.47 2.30 2.70 2.33 2.32 2.17 2.50 2.79 1.93 2.63 2.28 2.44 2.55 2.34 2.64 2.29
1.25 1.31 1.45 1.26 1.23 1.16 1.46 1.33 1.49 1.49 1.23 1.23 1.15 1.12 1.21 1.33 1.44 1.45 1.19 1.11 1.04 1.15 1.10 1.36 1.30 1.35 1.57 1.24 1.54 1.26 1.42 1.60 1.27 1.50 1.40 1.43 1.47 1.33 1.49 1.38
*l constitutes the highest rating and 7 the lowest.
206
ALBERT
R. ORATIO
TABLE 3 Variables Defining Factors I and II with Loadings, Means, and Standard Deviations Table 2 Factors I-Technical Skills Factor (2) Uses every opportunity to obtain target responses from the client (3) Modifies planned strategy to maximize target responses (9) Uses materials creatively to stimulate client interest (17) Changes therapeutic procedure to meet client needs (18) Resolves unexpected problems in therapy (34) Applies theoretical knowledge of disorder to therapeutic practice (36) Appropriately reinforces client’s approximations to target response (37) Uses effective schedule of reinforcement (38) Accomplishes the goals of the session (39) Implements carryover procedures in therapy II-Interpersonal Relationship Factor (11) Establishes rapport as evidenced verbally and nonverbally (13) Shows respect for the client as evidenced verbally and nonverbally (14) Uses appropriate speech characteristics: rate, pitch, volume, etc. (15) Communicates at the client’s cognitive and linguistic levels (20) Shows emotional stability during therapy (2 1) Maintains appropriate personal appearance (22) Shows appropriate attitudes in therapy, i.e., enthusiasm, etc. (33) Shows appropriate level of clinical involvement *l constitutes
Extracted from
Loading
Mean*
SD
-0.719
2.71
1.31
-0.763 -0.739 -0.757 -0.681 -0.762
2.74 2.60 2.53 2.72 2.63
1.45 1.49 1.44 1.45 1.50
-0.730
2.44
1.43
-0.736 -0.734 -0.638
2.55 2.34 2.64
1.47 1.33 1.49
0.710 0.763
1.97 1.85
1.23 1.15
0.737
2.01
1.12
0.728 0.654 0.598 0.778
2.17 1.80 1.69 1.85
1.21 1.11 1.04 1.15
0.597
1.93
1.27
the highest rating and 7 the lowest.
loading and supervisory mean rating. Out of the 18 variables that define factors I and II, clinicians received the lowest mean rating from supervisors on variable 3. Factor II (Interpersonal Relationship) consisted of eight variables and was clearly defined by variables 11, 13, and 22, all of which showed high positive loadings. Interpersonal communication variables 14 and 15 also showed high positive loadings for variables within this factor. Three intrapersonal variables, 20,2 1, and 33, contained the lowest loadings in the second factor. Out of the 18 variables that define factors I and II, clinicians received the highest mean rating from supervisors on variable 21. Based on all 40 variables used in the evaluation process, university supervisors rated clinicians highest on variables 21 (maintains appropriate personal appearance) and 23 (demonstrates punctuality) and lowest on variables 3 (modifies planned strategy to maximize target responses) and 32 (encourages client’s selfevaluation of responses) as indicated by mean ratings listed in Table 2. Frequency distributions for these variables (Fig. 1) indicate that 80% of the clinicians were
A STUDY
OF CRITERIA
FOR EVALUATING
STUDENT
CLINICIANS
201
120 II5
-
110 IO5 -
\
+ -+l ----. -4
VARIABLE 21 23 32 3
loo
\ \ \
2 (HI)
3 SUPERVISORY
4
5 RATING
6
7 (Lo)
Fig. 1. Frequency distributions for variables on which clinicians were rated highest (variables 21 and 23) and lowest (variables 3 and 32) by university supervisors. Based on all 40 variables.
rated at levels 1 and 2 on variables 21 and 23, whereas less than 50% of the 207 clinicians were rated at these levels on variables 3 and 32.
The factor structure identified in the present study indicated that two major behavioral dimensions are regarded by supervisors as critical to the process of therapeutic treatment: technical skills and interpersonal relationship dimensions. This finding corroborates Shriberg et al.‘s (1975) division of evaluative criteria
208
ALBERT
R. ORATIO
into professional-technical and interpersonal skill domains, and suggests a restructuring of criteria based upon the 18 variables that were found to be correlated within these two dimensions. These variables contribute most significantly to supervisors’ evaluations of student clinicians, and therefore should be used to construct a revised instrument for evaluating the process of therapeutic treatment. The correlation of variables within factors I and II suggests that the greatest disparity in clinical rating will be on variables between factors rather than on variables within the first or second factor. The factor which has been labeled “Technical Skills” consists of 10 variables that are related to achieving target responses, utilizing effective reinforcement, exhibiting therapeutic resourcefulness, and an understanding of theory related to remediation. These criteria appear to have application for highly structured and direct as well as indirect therapeutic styles, and therefore justifiably allow for these individual differences. The factor labeled “Interpersonal Relationship” consists of eight variables that are related to rapport, respect, interpersonal communications, and various intrapersonal conditions. The establishment of rapport and other interpersonal behaviors has often been viewed as essential in the initial stages of therapy. The emergence of this dimension, however, suggests that interpersonal and affective behaviors constitute an ongoing dynamic process widely regarded as critical to therapeutic effectiveness. Furthermore, the emergence of these eight criteria serve to raise questions about the kinds of professional training required for competent therapeutic interaction. Course work in sensitivity training, group dynamics, and interpersonal communication may enhance interpersonal skills. Kaplan and Dreyer (1974) have reported an effective method for developing interpersonal skills and increasing the self-awareness of student clinicians during therapy. Based on supervisors’ mean ratings of student clinicians, however, there is greater need to develop technical therapeutic skill. Student clinicians were consistently rated higher on variables related to interpersonal skills than on those related to technical abilities. In fact, clinicians were rated highest on clinical dress and punctuality and lowest on modifying the therapeutic strategy to maximize target responses and encouraging the client’s self-evaluation of responses. These latter two components appear to be more sophisticated technical skills that may develop within the practicum experience, while the former are easily modifiable and may in fact serve as prerequisites to entering clinical practicum. The present study defines the evaluation of therapeutic treatment as tentatively consisting of 18 variables correlated within two dimensions. These dimensions were shown to account for 58% of the total variance in the therapeutic evaluation. It is interesting to speculate a third factor based on the remaining 22 variables combined with marker variables. Such a factor may consist of numerous techniques such as variables 27 (verbalizes the progression of therapy to the client), 31 (utilizes various modalities, i.e., visual, auditory, tactile-kinesthetic), and 32
A STUDY
OF CRITERIA
FOR EVALUATING
STUDENT
CLINICIANS
209
(encourages client’s self-evaluation of responses). Numerous investigations have shown attraction to have a positive influence on therapy-like learning (Sapolsky, 1960, 1965; Lott and Lott, 1966). Perhaps variables related to client attraction for the clinician, as conveyed verbally and nonverbally, would mark a third factor. An hypothesis-testing analysis would be necessary to determine whether such undiscovered dimensions exist. It remains to be determined whether the present factorial structure would be maintained in evaluating therapy for the disorder areas of articulation, voice, stuttering, and language, independently. As treatment procedures for each disorder become more highly specialized, evaluative criteria specific to therapeutic processes within each area may be defined. This hypothesis provides a direction for further research. Information is most needed on defining the construct of “clinical competence.” Anderson (1973, 1974b) indicated that the identification of clinical competencies is most important for structuring the operations of the clinical supervisor. The investigation of “clinical competence” must ultimately include study of the reciprocation of interaction between supervisor, clinician, and client. Clinical supervision, as an area of investigation, defies objective measure. The current findings, combined with those of past research, may be instrumental in studying the complexities of the construct of “clinical competence.” The author wishes to acknowledge the assistance of George Herman, William Hinkle, Stephen Hood, and Melvin Hyman in critiquing various portions of the manuscript. Appreciation is also expressed to Raymond Tucker who served as consultant for statistics and experimental design. Special thanks to Janet Watson for typing the manuscript and my wife, Suzanne Gratio, for her help in preparing and organizing the research materials.
References Anderson, J. L. Supervision: The neglected component of our profession. In: L. J. Turton (Ed.), Proceedings afa workshop on supervision in speech pathology.Ann Arbor, Mich .: University of Michigan, 1973. Anderson, J. L. Supervision of school speech, hearing, and language programs--an emerging role. ASHA, 1974a, 16, 7-10. Anderson, J. L. Supervision of the clinical process in speech pathology: Issues and practice. Short course presented at the Annual Convention of the American Speech and Hearing Association, Las Vegas, Nevada, 1974b. Boone, D., Prescott, T. Content and sequence analysis of speech and hearing therapy. ASHA, 1972, 14 58-62. Brown, E. A university’s approach to improving supervision. ASHA, 1967, 9, 476-479. Darley, F. L. Clinical training for full-time clinical services: a neglected obligation. ASHA, 1969,11, 143-148. Diedrich, W. M. Assessment of the clinical process. J. Kans. Speech Hearing Assoc., 1969, l-8. Dixon, W. .I. (Ed.). BMD Biomedical computer programs X-series supplement. Berkeley and Los Angeles, Calif.: University of California Press, 1969.
210
Halfond,
ALBERT
M. Clinical supervision-stepchild
R. ORATIO
in training.
ASHA,
1964, 6, 441-444.
Haller, R. M. Supervisor’s criteria for evaluating student’s performance in clinical practicum activities. ASHA, 1967, 9, 479-481, Johnson, K. 0. American speech and hearing associafion directory, Danville, Ill.: The Interstate Printers and Publishers, 1975. Johnson, T. The development of a multidimensional scoring system for obvserving the clinical process in speech pathology. Unpublished doctoral dissertation, University of Kansas, 1969. Kaiser, H. The Varimax criterion for analytic rotation in factor analysis. Psychometrika, 1958, 23, 187-200. Kaplan, N. R., Dreyer, D. E. The effect of self-awareness training on student speech pathologistclient relationships. J. Commun. Dis., 1974, 7, 329-342. Kleffner, F. Seminar on guidelines for fhe internship year. Washington, D. C.: American Speech and Hearing Association, 1964. Klevans, D., Volz, H. Development of a clinical evaluation procedure. ASHA, 1974,16,489-491. Lott, A. J., Lott, B. E. Group cohesiveness and individual learning. J. Educ. Psychol., 1966, 57, 61-73. MacLeatie, E. Appraisal form for speech and hearing therapists. J. Speech Hearing Dis., 1958,23, 612614. Miner, A. Standards for quality supervision of clinical practicum. ASHA, 1967, 9, 471-472. Perkins, W. Our professiorr_What is it? ASHA, 1962, 4, 339-344. Perkins, W., Shelton, R., Studebaker, G., Goldstein, R. The national examinations in speech pathology and audiology: philosophy and operation. ASHA, 1970, 12, 175181. Sapolsky, A. Effect of interpersonal relationships upon verbal conditioning. J. Abnorm. Sot. Psychol., 1960, 60, 241-246. Sapolsky, A. Relationship between patient-doctor compatability, mutual perception, and outcome of treatment. J. Abnorm. Psychol., 1965, 70, 7676. Schubert, G., Miner, A., Till, J. The analysis of behavior of clinicians (ABC) system. Grand Forks, North Dakota: University of North Dakota, 1973. Shriberg, L. D., Filley, F. S., Hayes, D. M., Kwiatkowski, J., Schatz, J. A., Simmons, K. M., Smith, M. E. The Wisconsin procedure for appraisal of clinical competence (w-pact): model and data. ASHA, 1975, 17, 158-165. Turton, L. J. (Ed.). Proceedings of a workshop on supervision in speech pathology. Ann Arbor, Mich.: University of Michigan, 1973. Van Riper, C. Supervision of clinical practice. ASHA, 1965, 3, 75-77. Villarreal, J. Seminar on guidelines for supervision of clinical practicum in programs of training for speech pathologists and audiologists. Washington, D.C.: American Speech and Hearing Association, 1964. Ward, L., Webster, E. The trainingof clinical personnel: I. issues in conceptualization. ASHA, 1965a, 7, 38-40. Ward, L., Webster, E. The training of clinical personnel: Il. a concept of clinical preparation. ASHA. 1965b, 7, 103-106.