The Board Examination Debate Continued

LETTERS TO THE EDITOR The Board Examination Debate Continued To the Editor: The Debate Forum (Fox and Herzog, 1991) regarding the pass rate for the Child and Adolescent Psychiatry Board Examination was framed in response to the wrong question. Although the high failure rate is the result of serious problems with the structure and content of the Board examination itself, the methods of administering and scoring the examination, the content and quality of child and adolescent training programs, the monitoring of residents' progress in assimilating relevant training into competent clinical skills, or combinations of the above, this letter will address the first and second possibilities only. Because there are no empirical data supporting the ability of the examination to discriminate between competent and incompetent clinicians, I propose that a high degree of face validity (examination format approximates "real" clinical practice) is essential. The adolescent examination, and to a lesser extent the videotape case, somewhat approximate clinical practice. In these sections, the examinee is expected to make clinical observations of real people and has opportunities to process additional information provided by the patient or through written materials. Both examiners and examinees start with the same observable data set, and the reasoning process used is the standard medical and psychiatric model for collecting history, mental status examination, formulation of diagnosis, and development of treatment plan that is used in clinical practice. This is in marked contrast to the examination components in which there is no clinically valid data set common to both the examinee and examiners. Written vignettes are abstractions, and interpretations of them are not directly related to the skills required to be a competent clinician. The problem is compounded in that the brief vignettes have a minimum of detail and require the examinee to interpret, fill in, and/ or elaborate on a hypothetical situation, with no understanding of the context or background in which the consultation might be performed. As no additional specificity or feedback regarding interpretation of the vignette is provided by the examiners, the examinee must immediately develop a coherent verbal presentation based on interpretation of the written abstractions and nuances from which one might infer a variety of meanings. Although the knowledge base required to competently perform consultation work may be demonstrated by some examinees in this examination format, the ability to reason and discuss a wide range of consultation problems and possibilities from a hypothetical sketchy vignette is not, itself, a skill required to competently practice child and adolescent psychiatric consultation. As face validity decreases, the examination is more susceptible to confounding variables, such as differences in the examinee's cognitive and personality style. Traits not problematic in actual practice may be very problematic in attempting to pass an examination of this type. For example, an examinee must rapidly read, process, organize, reason, and articulate a coherent presentation based on the written vignette while being observed by two to three examiners; all this occurring while the examinee may be quite anxious. Child psychiatry, unlike thoracic surgery, does not require instantaneous decision making or action. Because extremely rapid processing and organizing of data is not required even in emergency situations, a rather slower rate of processing is not problematic for psychiatrists. It may, however, be the critical factor in causing examination failure. An individual who organizes ideas more slowly and has difficulty with decoding or processing written language may have only a few seconds before overwhelming anxiety overrides the individual's cognitive processes. Although the Board has created some mechanisms that acknowl-

l.Am.Acad. Child Adolesc. Psychiatly, 31:4, luly 1992

edge the importance of interrater reliability, I believe they are insufficient to ensure that the goal is achieved. Although a limited amount of discussion among examiners regarding acceptable examinee responses does occur, no rigorous training for examiners exists. Such training might include a formal review of scoring by examiners who have viewed videotapes of examinees performing the examination, followed by additional training or retesting until a specified level of proficiency is reached. The use of multiple examiners is not a substitute for stringent interrater reliability, as examiners vary widely in experience, willingness to assert their opinions as different from other examiner(s), and knowledge of the examinees' performance (floaters). In my opinion, the Child and Adolescent Board Examination as presently constituted is not a valid test for the determination of clinical competency. In addition to the problems described above, the absence of assessment of the quality of actual clinical work, independently performed in the "real world," is a serious flaw of omission in the certification process. Failure to pass the examination, at least by implication in the eyes of the Board, many members of the public at large, and other health and mental health professionals, implies that the examinee is less than competent. As the failure to pass this examination may have profound effects on the child psychiatrist's career, this examination must be methodologically sound so that its results are, in fact, accurate. I believe that it is the Board's obligation to craft an examination with a high degree of discriminant validity. In the absence of data that demonstrate this, it is incumbent that the Board create an examination that has a high degree of face validity. Interrater reliability and other potential confounding variables must be more rigorously and systematically addressed. As the Child and Adolescent Board Examination is plagued with multiple serious problems in its present form, I would argue that it is not sufficiently valid or reliable to warrant its continued use as an official, discipline-sanctioned standard of clinical competence. As presently administered, the Board examination may have discriminant validity in identifying those examinees who are likely to do well in academic settings. It appears that the following personality and cognitive styles that are optimal for this examination are also optimal in many academic situations: (I) the ability to rapidly assess, organize, discuss and defend clinical and theoretical issues before an "audience," as in a grand rounds or clinical case conference format; (2) "quick thinking" responses to complex issues, as when responding to questions in a lecture situation; (3) a personality style that is comfortable with or even seeks to "perform" in challenging situations where the individual has an opportunity to "shine"; and (4) the physical constitution and personality style that maximizes the examinee's ability to perform under the stress of travel, strange surroundings, loss of sleep, anxiety, etc. The methodology for assessing clinical competence should be revaluated by the Board. In the' 'real world," clinical competence is judged by the quality and appropriateness of one's work-the product, not a "fact simile" assessed in unreal circumstances. Although an oral examination offers opportunities for the assessment of skills required for competent practice, it creates many pitfalls that must be more thoughtfully addressed if a high level of discriminant validity is to be realized. Louis P. Stein, M.D. Graham, North Carolina REFERENCE

Fox, G. & Herzog, D. B. (1991), Resolved: The pass rate for the child

751

LEITERS TO THE EDITOR

and adolescent psychiatry board examination should be higher than it is at present (Debate Forum). J. Am. Acad. Child Adolesc. Psychiatry, 30:685-691. Dr. Fox replies: [Dr. Fox argued for the affirmative -

Ed.]

I am less concerned than Dr. Stein about the use of written vignettes as a jumping-off point for an examination. Although the vignettes are abstractions from the live clinical situation, at least everyone starts with the same data. Assessment of written information is centrally relevant to clinical competence: clinicians must be able to correctly interpret patients' charts, school records, etc. By the same token, faceto-face clinical interactions are by no means immune from nuances of meaning, which can lead to differing interpretations. I wholeheartedly agree with Dr. Stein about the time demands of the examination being a problem. The examiner is attempting, within the constraint of a half-hour, to fairly assess the candidate's knowledge in a variety of areas. The candidate must observe and organize data; put together a formulation, differential diagnosis, treatment plan and prognosis; and demonstrate competence in the biological, developmental, psychological, and social realms. The examiner is often in the position of nudging candidates along from topic to topic to give them a chance to demonstrate their abilities in each area, which unfortunately makes candidates even more anxious. The vignettes are deliberately constructed to stimulate discussion on a wide range of topics, but it then becomes difficult to have a comprehensive discussion. So much information must be covered in such a limited amount of time that the resulting performance is of necessity superficial. Contrary to Dr. Stein's statement, child and adolescent psychiatrists do have to think and act very quickly at times, and it is a useful clinical skill to have a mental organizational checklist. However, the examination format favors the comprehensive checklist over the in-depth presentation. The Board certification process is an enormous undertaking. It is a logistical challenge to convene such a large number of professionals, patients, and staff at a specific time and place, at a manageable cost to the candidate. Given these demands, 3 hours of oral examination for each candidate in the areas of preschool, grade-school, consultationliaison, and adolescent psychiatry seem generous. However, I would probably vote for longer discussion times and greater depth, even at the expense of covering fewer age groups or specialty areas. Perhaps it would be useful to have only one vignette (preschool or gradeschool) that incorporated consultation-liaison techniques in its expanded discussion and retain the adolescent interview. Finally, I wish to assure Dr. Stein that most academics I know get anxious before speaking publicly, do not look forward to their own Board examinations, and do less well when sleep deprived, just like other people.

the past two examinations suggest that failures occur equally in all three orals (adolescent; consultation liaison; preschool) because of poor integration of data gathered either from the interview, video, or vignette. This failure of integration does not permit appropriate use of the data for clinical decision making, which becomes apparent in the inquiry that follows the data gathering. We also agree that we wish there was evidence that the examination is valid and discriminates better between good and poor clinicians. Such evidence is limited for Board examinations to the internal medicine predictive validity studies. No other Board has demonstrated validity, whether using written exams or clinical tasks. Why do we have Boards then? The answer is that Boards certify competence insofar as we can define its elements. Hence, we employ a limited peer review of representative samples of clinical skills and knowledge to guard the gates and give some recognition to those who can pass muster, thereby also assuring external reviewers that those who pass have undergone professional peer review. We recognize all too well that anxiety and performance variables interfere with many candidates' ability to demonstrate skills. Better reliability will not help this. Moreover, fewer observers will place too much responsibility on one examiner. In addition, as the number of candidates has increased from 160 to 275+ in recent years, we may be forced to examine at two sites. However, the cost of examining is huge, and our fees would be even larger if we were to expand. I would encourage those who are critical to constructively reconsider the process. Each year, we monitor our questions and turn over one-third of the short answer questions. We revise our examiner training and monitor the results of our efforts. We also have learned that training directors will not both teach and then judge their own residents incompetent. We also seem to be learning that competence in general psychiatry does not guarantee competence in child and adolescent psychiatry. We are, after all, the only special qualification of the ABPN for good reason. Recently, added qualifications in the new subspecialties of geriatrics and addiction psychiatry were added to the ABPN's roster. We ought to ask whether we have learned something in our child and adolescent psychiatry training that our psychiatric peers recognize as different. The practice effect derived from having been examined before does not address the new content of our training. Debates such as this one are useful to guide future discussion and to permit revision of our methods. My experience on the Committee and attendance at Board meetings has reassured me that the Board leadership is responsive and that the Research and Development Committees are working to make the best Board possible. Theodore Shapiro, M.D. Chairman, Committee on Certification in Child and Adolescent Psychiatry American Board of Psychiatry and Neurology, Inc.

Geri Fox, M.D. University of Illinois

Psychobiological Methodology The Board replies:

To Ihe Editor:

As the current Chair of the Committee on Certification in Child and Adolescent Psychiatry, let me join all those who have written about the Child and Adolescent Psychiatry Boards and say that I, too, wish that more child and adolescent psychiatrists would pass. Unfortunately, the results have been roughly parallel to those of general psychiatry Boards despite our not exercising a quota on passes or judging our candidates more stringently than they do. We have not rested easily and have been studying the matter. We have been looking at our candidate failures in a recent pilot study to see where our candidates or our examiners go wrong. It does not seem that we are catching many habitual failures. Preliminary results from

The September 1991 article by Jensen and colleagues, "Growth Hormone Response Patterns in Sexually or Physically Abused Boys," raises several interesting questions. The study examines growth hormone response to clonidine and L-dopa in a sample of 74 boys: 15 with purported physical abuse, seven with purported sexual abuse, 13 normal controls, and 39 psychiatric controls. The authors conclude that the physically abused children respond to these probes differently from the sexually abused children, and that both abused groups differ from the control subjects. This paper makes a significant contribution in focusing on the potential for psychobiological methodologies to increase our understanding of the effects of abuse and in proposing

752

J. Am. Acad. Child Adolesc. Psychiatry, 31 :4, July 1992