Reactions of physicians and medical students to chronic illness

Reactions of physicians and medical students to chronic illness

J. &on. Dis. Vol. 15, pp. 785-794. Pergamon Press Ltd. Printed in Great Britain REACTIONS OF PHYSICIANS AND MEDICAL STUDENTS TO CHRONIC ILLNESS” AMA...

855KB Sizes 33 Downloads 80 Views

J. &on.

Dis. Vol. 15, pp. 785-794. Pergamon Press Ltd. Printed in Great Britain

REACTIONS OF PHYSICIANS AND MEDICAL STUDENTS TO CHRONIC ILLNESS” AMASA

B. FORD, M.D., RALPH E. LISKE, Ph.D. and ROBERTS. ORT M.D., Ph.D.

Department

of Medicine, Western Reserve University Medical School, Cleveland, Ohio (Received 30 October 1961)

THE

permanently disabled and the chronically ill tend to become objects of prejudice and discrimination in our society. These attitudes have been recognized and, to some extent, studied, by sociologists and psychologists [l-3] but have received scant attention from the medical profession. In most studies, the focus has been on the patient and the community. Only a few investigators have considered whether physicians and those in other health professions share such attitudes [4, 51, and no general study has been made of medical thinking about this increasingly important subject. The present investigation is an exploratory effort to describe and relate the feelings, thoughts and behavior of physicians toward people with chronic illness. In caring for patients with chronic illness, and in teaching medical students the principles of comprehensive care, the investigators have been struck by a reluctance on the part of physicians and students to participate in the care of such patients. Based on this experience, our preliminary hypotheses was: Physicians and medical students tend to react negatively to patients with chronic illness. We chose to approach this problem initially by studying th.e thought processes and behavior of medical students. We hoped that, by observing the transition from lay to professional orientation and behavior, we might be able to identify characteristics which belong specifically to the medical profession. A second reason for choosing the medical students as subjects was that they are involved in the actual treatment of patients with chronic illness in a special outpatient teaching clinic. Early in the investigation it became evident that it would also be necessary to study the medical faculty in order to define the professional models presented to students.? METHODS

Samples were drawn of the clinical faculty and the senior class of the medical school. Thirty faculty members were chosen as follows: ten each from the departments of medicine, surgery and psychiatry; within each department, four full-time, four part-time and two resident physicians; half the total sample were chosen as influential or representative members of the faculty and the other half were chosen at random. *This investigation was supported, in part, by a research grant from the Office of Vocational Rehabilitation, Department of Health, Education and Welfare, Washington, D.C. TSimilar data is currently being collected from physicians not associated with the medical school in order to determine differences which might exist between medical school faculty members and other practicing physicians. 785

186

AMASA Bl FORD, RALPH E. LISKEand ROBERTS. ORT

A substitute was chosen for one faculty member who refused to participate in the study. The average age in the faculty sample was 43 a3 years (range 25-67). A sample of twenty-seven medical students was drawn from the senior class of seventy-eight students: Twenty were male students selected at random. All seven of the female students in the class were ‘included in the sample to provide a group large enough for comparison with the men. The average age in the student sample was 26.5 years (range 23-38). A projective technique was selected because previous work on this subject has indicated that there are apt to be deeper feelings underlying commonly expressed, stereotyped and mildly favorable attitudes [l]. The specific technique chosen was the sentence completion method, since it permits the expression of basic thoughts and feelings. This technique makes possible not only a pro-con type of measurement but also measurement in other dimensions, combining the desirable features of projective testing and the advantages of open-ended questions [6]. In order to construct an appropriate completion form, stems were developed from two principal sources. The first source was descriptions of effective and ineffective clinical performance collected from the medical school faculty by the critical incidents technique [7]. A second source was the literature on attitudes toward individuals with chronic illness and disability. Thirty stems were constructed, sixteen of which refer to subjects relating specifically to chronic illness. Examples are: “Patients with chronic disease are . . . ,” “When I see a patient with relentlessly progressive disease . . . .” The remaining items have to do with general aspects of medical care, for example: “A doctor can do most for a patient when . . . ,” “If a patient ‘signs out’ against advice . . . .” The entire sentence completion form, with instructions, is given in Appendix I. Faculty members were approached in person, and students completed the forms individually in the research center. Subjects were requested to complete the form in order to provide information about the doctor-patient relationship for use in teaching and preparation of medical students for practice. Responses were handled anonymously by randomly assigned code numbers. The response given by each subject to each item was typed on a separate card, to facilitate analysis by sorting. The first approach to the data was an effort to classify the responses according to whether they showed a positive or negative reaction to the subject material. Some responses showed a clearly favorable feeling tone, for example, “A crucial factor in rehabilitation is . . . recognizing that the patient is a human being. All things follow from this.” Other responses expressed definitely negative feelings, such as, “When I see a patient with relentlessly progressive disease . . . I am depressed.” Only a small proportion of responses, however, gave as clear an expression of the respondents’ feelings as in the two examples. More commonly, the responses indicated a tendency to approach the situation by means of intellectual or physical action, for example, “When a patient persistently presents vague or inconsistent symptoms . . . I turn on my curiosity;” or, on the other hand, a tendency to avoid the situation, as, “Chronic disease is something I . . . prefer not to treat.” Since it was often impossible to distinguish between feeling-tone and approachavoidance tendency in a given response, a single index was constructed to include both.

Reactions

of Physicians and Medical Students to Chronic Illness

187

The concept of orientation was chosen to represent the combination of these two directional tendencies. The responses were then scored according to the seven-point index of orientation, which is described in Table 1. TABLE 1.

MODEL OF ORIENTATION INDEX (NUMBERS REPRESENTDISCRETESCORES)

1 2 Great Moderate unfavorable- unfavorableness ness

Great avoidance

Moderate avoidance

3 Slight unfavorableness Slight avoidance

4 Neutrality or ambivelence

5 Slight favorableness

6 Moderate favorableness

7 Great favorableness

Slight approach

Moderate approach

Great approach

Responses which faculty members gave to the sixteen chronic disease items were scored independently by two of the investigators. 93 *7 per cent of the scoring judgments made by these two raters were identical. Consequently, residual differences were resolved, and further scoring was done by a consensus of the three investigators. In this way, every response could be assigned an orientation score, with the exception of instances where a particular stem was rejected or no answer was given. Such instances amounted to 71, or 4.2 per cent of the total. In the course of scoring the responses, it became evident that some were simple expressions of a single idea, for example, “Most patients who complain very little are . . . stoical,” while some contained a wide variety of ideas, such as, “After multiple bilateral strokes, most patients . . . become constant care problems, burdens physically, mentally and financially to those who must assume care of them.” Since it seemed reasonable that reactions to chronic illness might be related to the breadth and depth of thought elicited by the subject matter, the next exploratory step was an attempt to measure the range and complexity of ideas manifest in the responses. Range, or breadth of thinking, appeared to be indicated by the number of general categories of thoughts, and complexity, or depth, by the elaboration within each category. Five medically related categories were recognized: physical, psychological, social, economic and institutional. These areas were further defined as : (1) Physical. Referring to the physical or organic problem, including diagnosis, treatment and other strictly medical aspects, for example, “The kind of patient I enjoy treating most is . . . the acutely and severely ill patient with a treatable disease.” (2) Psychological. Referring to psychological or emotional aspects, including the doctor-patient relationship, patients’ psychological characteristics, communications, or the physician’s emotional response, for example, “When I see a patient with relentlessly progressive disease . . . I try not to be sympathetic but to be reassuring and hopeful so as to aid the patient in reassuring himself.” (3) Social. Referring to the social problems related to illness, centering in relations to family, friends and other, for example, “If a 70-year-old aphasic hemiplegic came to my office . . . I’d want to know his family.” (4) Economic. Referring to medical cost, insurance, and the like, for example, “As I see it, comprehensive medical care means . . . that medical care is paid for by an insurance company from payments made by patients.”

AMASA

788

(5) Institutional.

B. FORD, RALPHE. LISKE and ROBERTS. ORT

Referring

to the way society

and medicine

relation to health, including cultural patterns, and professional organizations, for example, are . . . perhaps

“Patients with chronic our greatest public health problem today.”

An index was therefore Jn view of the current

are organized

constructed

emphasis

in

medical and related specialities,

to measure the breadth

on the comprehensive

disease

and depth of thinking.

approach

to chronic

illness,

greater weight was given to the numbers tion of thought within a given category,

of categories mentioned than to the elaboraalthough both were recognized. The index is therefore referred to as a comprehensiveness index. The scoring procedure is as follows: Each response is examined first, to identify which categories

are used, and second, to determine

how many ideas are expressed

in

each category. An ‘idea’ is defined as a statement which can stand alone as a response. A score is then assigned according to the comprehensiveness index shown in Table 2. TABLE2. 1 One idea in one category

MODELOF COMPREHENSIVENESS INDEX (NUMBERS REPRESENT DISCRETESCORES)

2 Two or more ideas in one category

3 One idea in each of two categories

4 5 Two or more One idea in one category, ideas in each plus two or of two or more more ideas in categories

6 One or more ideas in each of three categories

a second category More comprehensive

responses

on the number of categories

7 One or more ideas in each of four or

more categories

A seven-point scale is used, based and the number of ideas within each category.

receive higher scores.

mentioned,

The responses given by faculty members to thirty items were scored independently by all three investigators. Pairs of raters agreed on 92.6 per cent of the scoring judgments. Consequently, residual differences were resolved and student responses were scored by a consensus of the three investigators. a comprehensiveness score, with the exception

Thus, all responses could be assigned of the rejections and blanks.

Next, an effort was made to obtain an index of the clinical performance of the subNineteen raters were chosen from the professional jects in the area of chronic illness. superiors,

peers and subordinates

of the faculty

respondents.

Each

physician

was

asked to make confidential ratings of the respondents whom he believed he knew sufficiently well to judge in terms of ability to provide comprehensive medical care for medical care” was specipatients with chronic disease. “Providing comprehensive fically defined as “(1) identifying medical problems factors, (2) making a complete plan of treatment

and related social and emotional and executing his own part with

skill, and (3) following through and making good use of medical consultants and other services.” Each rater ranked those subjects he knew well by a card-sorting technique performed in the presence of one of the investigators [S]. A similar confidential performance rating, based on the same definition of comprehensive health care, was obtained for the sample of medical students from directors of clinical clerkships, medical preceptors and members of allied health professions who observed the students in the clinic. All members of the senior class were also requested to make anonymous Seventy-eight per cent of the class confidential ratings of their classmates.

and were

Reactions of Physicians and Medical Students to Chronic Illness

789

unable to make such ratings, giving reasons which will be considered later. Consequently, ratings by classmates were not included in the combined ratings. The rank order of ratings given by each rater were converted to scale scores for the faculty and for the students separately by the method of per cent position [9] in order to determine interrater agreement. RESULTS

AND

DISCUSSION

Orientation

The responses to the sentence completion forms were scored according to the orientation index (favorableness-approach tendencies versus unfavorablenessavoidance tendencies). Two items were omitted from this index because they required the respondent to reply from the point of view of the patient, (items 23 and 27). Chronic illness and general items were compared separately for the faculty and for the male and female student groups. TABLE 3.

AVERAGE SCORES

Orientation index (16 chronic disease items)

No.

Comprehensiveness (all items)

Average group scores

Range of average individual scores

Average group scores

index

Range of average individual scores

Students

27

4.3

3.1-5.3

2.5

1.34.5

Faculty

30

4.6

3.7-5.8

2.4

1.3-4.0

the distribution of individual scores were very 3). NO difference was found between male and female student groups; therefore, the student group is treated as a unit. Individual average scores on chronic illness items and general items were compared by a rank order method. Little correlation was demonstrated between the responses to chronic illness and to general items for both students and faculty (p=O.29 and O-52, respectively). For this reason, orientation is hereafter considered in relation to the sixteen chronic illness items alone. Orientation scores for students and faculty were compared by averages, rank order of item scores and extreme differences of item score, and no significance differences could be demonstrated. This analysis therefore suggests that, in terms of favorableness or unfavorableness of feeling and tendency to approach or avoid, the reactions of students and faculty members to ideas related to chronic illness are highly similar and generally neutral. Although the general reaction appears to be neutral, certain items consistently elicit generally positive or negative reactions. For both faculty and students, responses to five items show average orientation scores above 5.0 (items.3, 8, 12, 22 and 30), and to four items, below 4-O (items 7, 9, 17 and 21). All items were included in this part of the analysis in order to determine the preferences and dislikes of the respondents, whether related to chronic illness or not. Some of these items might be expected to elicit positive responses and do SO, for example item 22: “I enjoy treating patients because . . . .” Others, however, which seem to call for a negative reaction, produce a The

similar

average

orientation

for faculty

score

and students

and

(Table

AMASA B. FORD, RALPH E. LISKE and ROBERT S. ORT

790

positive one, for example item 30: “If a 70-year-old aphasic hemiplegic came to my office , . .” Thus, the phrasing of the stems does not necessarily dictate the reaction. Inspection of the positively oriented responses suggested that diagnosis and treatment were mentioned frequently. These references were counted, and it was demonstrated that the five items eliciting the most positive responses, which constitute 18 per cent of those scored, yield 171 of 518 such references, or 33 per cent. Conversely, items which elicit frequent references to diagnosis and treatment were also found to elicit positive orientation. Thus, positive orientation appears to be closely associated with the concepts of diagnosis and treatment. Analysis of the responses to the four items which elicited negative orientation showed frequent reference to demanding or hostile patients and to severely disabled or hopelessly ill patients. Forty-nine of sixty-six such references in the entire sample occur in these four items (forty-nine observed, nine predicted, P= x0.01 by chisquare). In these responses, the doctor is characterized as encountering difficulties, feeling helpless, rejecting the patient, or having mixed reactions. Patients, on the other hand, are described as having emotional problems, creating their own problems, being anxious, having intellectual limitations, social problems, or undesirable characteristics. From this analysis it appears that chronic illness is a complex situation, reactions to which cannot be characterized as uniformly positive or negative. The orientation of the respondents appears to be relatively positive when they perceive opportunities for coping with the situation (diagnosis and treatment), and relatively negative when they recognize frustrations (hostile, demanding patients, etc.). Although the average score suggests overall neutrality toward chronic illness, it includes discernible positive and negative reactions toward specific aspects of the whole situation. Comprehensiveness All responses were scored according to the comprehensiveness index (breadth and depth of thought). No difference was found between the male and female student groups; therefore the students are treated as a unit. The average comprehensiveness score and the distribution of individual scores was very similar for faculty and students (Table 3). The rank order correlation of student and faculty scores, item for item, is high (p=O.70). Thus, in terms of comprehensiveness of thought, students and faculty respond very much alike and in terms of only one or two categories. The scores on chronic illness and general medical items were compared for both TABLET.

DISTRIBUTIONof STUDENT AND

FACULTY RESPONSES TO THIRTY lTEMS,SCORED BY COMPREHENSIVENESSINDEX

Responses of 27 medical students

Psychological Physical Social Institutional Economic Total

SENTENCE

COMPLETION

Responses of 30 faculty members

No.

Per cent

No.

Per cent

706 367 77 55 17 1222

57-8 30.0 6.3 4.5 1.4 100.0

740 377 97 70 12 1296

57-l 29.1 7.5 5.4 0.9 100-O

Reactions of Physicians and Medical Students to Chronic Illness

791

students and faculty. Average comprehensiveness scores are nearly identical, and rank order correlation by individuals for both students and faculty is 0.74 and O-62, respectively. The comprehensiveness score does not appear to be greatly different when chronic illness items are compared with general medical items. Comprehensiveness scores on all thirty items are therefore used in further analysis of the data. The distribution of responses among the five categories is given in Table 4. The responses of faculty and students show remarkably similar patterns. The psychological category occurs twice as frequently as the physical in the responses of both groups, and both groups refer rarely to social, economic and institutional areas. The possibility was considered that the pattern of responses observed might have been a function of the stems. There is, in fact, a preponderence of ideas in the physical and psychological categories expressed in the stems. A scoring of the stems according to the comprehensiveness index yielded the following distribution: physical 44 per cent, psychological 38 per cent, social 9 per cent, institutional 6 per cent and economic 3 per cent. Efforts to predict the responses to specific stems, however, were completely unsuccessful, and responses in every category were given to stems which did not suggest that category. It is particularly noteworthy that the respondents reversed the proportions of physical and psychological ideas, giving greater emphasis to the psychological category. Next, the extremes of the comprehensiveness index were examined. The items which elicit the greatest breadth and depth of thought have to do with comprehensive medical care, the social and emotional problems of chronically ill patients, and the office treatment of a hemiplegic patient (items 4, 26 and 30). Responses to these items mention the physical (strictly medical) category more frequently than average, and the social, economic and institutional categories with from two to four times the average frequency. 83 per cent of the social, economic and institutional responses are expressed in terms of the individual doctor-patient relationship rather than of broad social issues. The items which elicit the least breadth and depth of thought have to do with the physically handicapped, malingering and blindness (items 10, 11 and 14). It appears that the respondents do not see themselves as professionally involved in relation to these situations, since in answer to these items, they give only half the average number of responses in the physical category and three times the average number of rejections or no answers. Social, economic and institutional ideas are almost completely absent in these responses. In summary, the analysis of comprehensiveness of thought brings out the central importance of the interpersonal relationship between the doctor and his patient. Students and faculty respond more frequently in psychological terms and less frequently in strictly medical terms than would be expected from the stimuli presented. It also appears that students and faculty tend to give little thought to situations in which they do not see themselves professionally involved. On the other hand, when they are able to take an active part in a situation, they are also able to think more broadly and deeply about it-though usually in terms of the doctor-patient relationship.

Orientation. and comprehensiveness Students and faculty members were ranked within their groups; first, according to average orientation scores, then, according to average comprehensiveness scores.

192

FORD, RALPH E.LISKE and ROBERT S. ORT

AMASAB.

A rank order correlation of 0.49 was found for the students, and of 0 *47 for the faculty, indicating a limited amount of common variance between orientation and comprehensiveness. A suggestion of what the common variance between these two indices might be, is found in the further observation that 47 per cent of the references to diagnosis and therapy found in the entire sample occurred in the 25 per cent of items which showed positive orientation (greater than 5 *O),high comprehensiveness (greater than 3 *O),or both. Thus, it appears that situations which permit the physician to think in terms of diagnosing and treating illness are associated with a strongly positive orientation and a relatively high comprehensiveness of thought. Clinical performance

ratings

For the performance ratings of the faculty, nineteen raters gave an average of twelve ratings for each respondent, the range varying from 5 to 19. The ratings assigned each respondent by varied numbers of raters were converted to a common scale and compared by the method of intraclass correlation [IO]. The ratings of faculty respondents show no correlation (R=O.05), indicating no agreement among faculty members as to the clinical performance of their colleagues, in terms of the rating procedure used. For the students, twenty-six raters gave an average of six ratings per student (range 2-8). The intraclass correlations for the student ratings show low correlation R=0*31). Thus, there appears to be but little more consistency shown in the ratings of the students by the faculty than in the ratings of the faculty by each other. In view of the lack of agreement among raters, comparisons with orientation and comprehensiveness scores were not considered justified. DISCUSSION

Sentence

completion

OF

METHODS

technique

The use of a sentence completion form has proved a valuable tool for exploration of attitudes and thought processes in regard to chronic illness. Cooperation was very high, in that only 4.2 per cent of 1710 answers were blanks or rejections. In addition, the technique permitted wide distribution of responses in terms of the two measures employed and there were many highly individualized responses. The responses could be analyzed easily according to both a reliable orientation index and a reliable index of comprehensiveness of thought. Performance

rating procedure

The performance rating technique yielded little agreement among raters. Two explanations are suggested. The first is the lack of visibility of relevant clinical performance. That is, medical practitioners and medical students are seldom in a position to observe each other in contact with patients. This explanation is supported by the fact that when the seventy-eight senior students were asked to rate their colleagues, fifty-four expressed inability to do so. All but one gave the reason that they did not feel qualified to do this because of their lack of opportunity to observe classmates in clinical situations. A second possible explanation for the lack of consistency is that the basis for rating is complex because of the three-part definition

Reactions of Physicians and Medical Students to

Chronic Illness

793

of comprehensive care. Two problems are involved here, namely, the lack of preciseness in the definition itself, and the probable lack of agreement on the concept of comprehensive care among physicians and students. Thus, it appears that before more accurate statements can be made about the relationship between clinical performance on the one hand and thoughts and feelings on the other, it will be necessary to devise more accurate and consistent measures of performance than those used in this study. From our observations and work thus far, we would suggest that a more adequate measure of clinical performance should ideally include the following items: (1) direct observation of the interaction between doctor and patient ; (2) evaluation of patient’s reaction to the doctor on a personal and professional level; (3) observation of coordinated efforts with physicians and other health professionals; (4) evaluation of communications, including such items as records, referrals and discharge summaries; and (5) a measurement of the effectiveness of treatment. SUMMARY

AND

CONCLUSIONS

This study describes the reactions of thirty clinical faculty members and twentyseven senior medical students to a sentence completion form based on chronic illness and general medical subject matter. The initial hypothesis of negative reactions to chronic illness is not borne out. Instead, the overall reaction of both faculty and students is relatively neutral. They do express favorable feelings about, and a tendency to approach, situations involving diagnosis and treatment, and they exhibit breadth and depth of thought in situations in which they see themselves professionally active. The interpersonal relationship between doctor and patient appears to be of primary importance, since the responses are cast in psychological terms more frequently than in any other, and since social, economic and institutional ideas are expressed mainly as they bear on the doctor-patient relationship. Certain aspects of chronic illness, on the other hand, do bring out negative reactions. Unfavorable feelings and a tendency to avoid the situation are associated with ideas of demanding, hostile, severely disabled and hopelessly ill patients. Physical handicaps, blindness and malingering do not engage the respondents’ thought or professional skill. These findings point to the hypothesis that reactions of physicians and medical students to patients with chronic illness are not necessarily negative but may actually be positive under two conditions: (1) when the physician or student feels he can exercise a professional skill, such as diagnosis or treatment, and (2) when he feels he can establish a satisfactory personal relationship with the patient.

SENTENCE

APPENDIX I COMPLETION

FORM

From your point of view as a physician, complete each of the following sentences with whatever thought comes to mind. You should try to make sense, though your sentences need not be long. The idea is to get down a complete thought as rapidly as you can. 1. Most patients who complain very little are 2. When a patient persistently presents vague or inconsistent symptoms

794

3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30.

AM.UAB. FORD,RALPHE. LISKEand ROBERT S. ORT

A doctor can do most for a patient when As I see it, comprehensive medical care means When patients can’t understand or remember instructions To me, dealing with non-medical aspects of a case seems I like most things about patients I see, except Where no specific treatment is available, the doctor Alcoholic patients are Condescension toward the physically handicapped is When I find that a patient is malingering The kind of patient I enjoy treating most is One reason patients ‘shop’ for doctors The striking thing about most blind patients Patients with chronic disease are Compared to visible deformity, non-apparent disability is After multiple bilateral strokes, most patients Motivation among outpatients When I see a patient with relentlessly progressive disease A crucial factor in rehabilitation is My idea of a ‘crock’ is a patient who I enjoy treating patients because If I were found to have inoperable cancer I think the social handicap that physical disability produces Chronic disease is something I As for social and emotional problems of chronically ill patients, I Being married to a physically handicapped person would If a patient ‘signs out’ against advice When colleagues joke about the ‘crocks’ they see in the clinic If a 70-year-old aphasic hemiplegic came to my office REFERENCES

1. BARKER, R. G., WRIGHT,B. A. and GONICK,M. R. : Adjustment to physical handicap andillness, Social Sci. Res. Council Bull. 55 (rev.), New York, 1953. 2. GELLMAN, W. : Roots of prejudice against the handicapped, J. Rehabil. 25,4, 1959. 3. ALLPORT,G. W.: The Nature of Prejudice, Doubleday, New York, 1954. 4. STOLLER, R. J. and GEERTSMA, R. H.: Measurementof medical students’ acceptance of emotionally ill patients, J. med. Educ. 33, 585, 1958. 5. CAPLOvrTZ,D.: Value orientation of medical students and facuky members: Paper given at American Sociological Society, Washington, D.C., August 1957. 6. ROTTER,J. B. and WILLERMAN, B.: The incomplete sentence test as a method of studying personality, J. consult. Psychol. 11, 43, 1947. 7. HERZBERG, F., INKLEY,S. and ADAMS,W. R.: Some effects on the clinical faculty of a critical incident study of the performance of students,J. med. Educ. 35, 666, 1960. 8. BIT~NER,R. H. and RUNDQUIST, E. A.: Rank-comparison rating method, J. appl. Psychol. 34, 171, 1950. 9. GARRET, H. E. : Statistics in Psychology and Education, (2nd Edition), pp. 169-173, Longmans Green, New York, 1941. 10. HAGGARD, E. A. : Intraclass Correlation and the Analysis of Variance, pp. 13-16, Dryden Press, New York, 1958.