Continuing medical education and assessment of radiologists

Clinical Radiology (1988) 39, 575-577

Continuing Medical Education and Assessment of Radiologists G. W. STEVENSON and W. P. COCKSHOTT

Department of Radiology, Chedoke McMaster Hospitals and McMaster University, 1200 Main St West, Hamilton, Ontario L8N 3Z5

Once doctors have obtained their degree and licence they are free to practise for the rest of their working life unless negligence or inappropriate behaviour lead to disciplinary action. The medical profession has a duty to set and maintain standards but there are no formal mechanisms for assessing knowledge or performance once examination hurdles have been overcome. The need for continuing medical education (CME) has long been recognised; traditional journals and textbooks have been supplemented by courses and self-instruction packages using a wide variety of media forms. These voluntary CME activities developed by professional bodies and organisations are doubtless useful though the effects have rarely been formally documented. We believe that some measures to assess continued competence need to be developed by radiologists and other specialists, both for their own integrity and to satisfy pressure from the public who show declining satisfaction with doctors and the medical establishment. If the profession does not regulate itself, other remedies that may be inappropriate or suboptimal may be imposed. This has just come to pass in New York state where specialists are to be subject to periodic re-examination. This approach poses many problems and alternatives are needed. Radiologists in hospital practice are subject to forms of peer review inherent in daily consultation processes and feedback from laboratory and pathology tests as well as group educational sessions. However, those practising in an office without a hospital appointment have little feedback and patient referrals continue as long as physicians are satisfied with the speed of the reports. Once the formal training and education process of the specialist is completed, the assessment focus should change from 'performance in examination (what a man knows) to performance at work (what a man does)' (Mueller, 1971). Methods for assessing continued competence are many and varied and most fail to measure what they claim to evaluate. Though most radiologists subscribe to journals, it is not known how many read them, let alone how intensively. The effect of reading as an instrument of continuing education is unknown (Davis et al., 1984). Similarly, the effect of slide/tape video cassette and other self instruction packages has not been critically evaluated. The benefits derived from attending courses are not clear. Most evaluations have been based on the perceptions of the participants, cynically termed the satisfaction or happiness index. Such popularity scores may or may not reflect educational value. Sometimes courses encompass a pre- and posttest to assess cognitive or attitudinal changes, usually at the level of short-term memory recall. The preceding Reprint requests to: Dr Stevenson

two methods only skirt the issue and do not assess the effects on the performance of the doctor. Audit and utilisation studies accomplish this in radiology. Thus one can measure the decline (usually transient) in the use of skull radiographs for trauma following courses or articles that have discussed efficacy and high yield criteria. Ultimately, the most important determinant of educational benefit is a demonstrable effect on patient outcome. Such studies are expensive and difficult to administer. In a review of 238 studies of CME in the literature, only 13% considered patient outcome; with an improvement in outcome demonstrated in about half of the studies (Davis et al., 1984). Significantly, courses directed at a specific objective need were most likely to show a positive effect. In radiology, purpose designed courses to train and accredit mammographers are in this category (Fajardo et al., 1987). The principles of specialty certification and compulsory CME are far from new. From 1300 to 1801 both existed in_ Venice, with the latter provoking ingenious evasions over the centuries (Ell, 1984). However, 690 years of CME since 1300 have not led to any uniform view on how to assess educational needs, nor how to implement them. Assessment of needs in CME is a discipline in itself (Laxdal, 1982; Ward, 1988) yet few courses are planned around the identified needs of an intended audience. It is more usual to accommodate the interests of available speakers. Belsheim suggests that CME may be based on three different models: educational, problem based learning, or change orientated (Belsheim, 1986). Each may be useful for different purposes, but if a specific educational need or clinical deficiency has been identified the change orientated model will be more useful than the traditional CME day of lectures. If a course is to be successful in inducing a sustained change in behaviour, it will need to identify the specific change required, the present system of constraints in which the physician practices (financial, political or simply the climate of local medical opinion), the most effective format for the educational intervention, a method of helping those who decide to opt out or get discouraged, and finally a system of both short and longterm follow-up with carrots and/or sticks to lock in the new style of practice (Warren, 1977). Thus the establishment of an educational programme to produce a specific end point requires more planning and long-term involvement than the simple CME evening that informs the already converted. Because of the complexity and difficulties of assessing CME a simple scoring system of credits was established by the American Medical Association. Credits are provided for attending courses or partaking in approved CME self-instructional packages. A radiologist is expected to accumulate a specified number of credit points annually. It is noteworthy that credit points are

576

CLINICAL RADIOLOGY

not given for reading - the most universally used method of CME[ The traditional CME credit scheme, despite its imperfections, has undoubtedly been beneficial in maintaining standards and updating practice: it has also served to hold back those who favour recertification by periodic examinations which measure what is known rather than what is done. Radiologists differ widely in their practice: some are generalists while others have highly specialised practices. A 1983 survey reported that some 25% of clinicians in the USA believed radiologists to be inadequately subspecialised (Rockoff et al., 1983), and there is no doubt that greater specialisation is required within radiology if it is to remain useful to clinicians. The complexity of setting and scoring exams for general radiologists, let alone a variety of subspecialists, would be formidable, yet how can an examination be fair unless it is tailored to a radiologist's activities? What determines the pass level which if too high would disenfranchise a large number from practice but if too low would make the exercise meaningless. Should there be a single pass/fail result or should those with specific deficiencies be required to document that these have been addressed? All of the foregoing show how unsatisfactory the existing methods are of assessing competence. The credit system tabulates participation in some educational activities and when standard exams are used they assess knowledge rather than performance. Alternative strategies are needed. We propose three alternative approaches for consideration: audit, peer review and objective structured clinical examinations. Audit Audit consists of internal or external review of the activity of a hospital department or a private office practice. Some aspects may be mandatory, as in Ontario, where all X-ray facilities require licences to operate and are subject to inspections to determine conformity with calibration standards, and acceptable dose levels. In addition, certain quality assurance procedures relating to film processing and reject film analysis are required. But these only monitor compliance in producing radiographs, and do not assess interpretative aspects of practice. Hospital departments also have to conform to standards set up by a hospital accreditation committee; these requirements tend to be administrative in nature but may also encompass protocols for carrying out examinations, for instance by defining the number of projections deemed necessary etc. Some hospitals in addition require that departments regularly audit some aspects of diagnostic quality such as comparisons of sensitivity and specificity of two complementary types of examinations, for example ultrasonography or cholecystography for suspected cholelithiasis. Generally speaking, audit can monitor certain activities but does not tackle important but subjective analysis of the diagnostic quality of films. Peer Review Peer review is assessment of a medical activity by one's colleagues. The process has been extensively used

in Ontario in family practice. The licensing body, the Ontario College of Physicians and Surgeons, initiated over 7 years ago a programme to evaluate private practitioners: this group was chosen for review because it represents physicians practising outside hospital accreditation processes and sometimes operating alone in a solo practice. To date, over 1400 practices have been inspected by teams of two general practitioners who have themselves previously been subject to peer review. Assessment includes inspection of the facility and its records, and a sampling of management used for certain conditions. The large majority of practices get a satisfactory report; those that have deficiencies are graded on a scale so that appropriate corrective action may be taken. All findings are kept confidential to the peer review committee. Further reviews are carried out on practices that have deficiencies and progress is monitored. According to predefined criteria, some practices may be reported to the Ontario College or appropriate regulatory body. At the present time, development of a similar review scheme is being considered for private radiological practice. A voluntary survey has indicated that such a process is acceptable to radiologists and could be applied more widely. It goes beyond the mandatory Radiation Protection services inspection and calibration process to look at the operation of the practice. Have obsolete practices been abandoned and new procedures adopted? Are the films produced of diagnostic quality? Are protocols set up describing appropriate radiography for various clinical problems? Operating standards must be developed cautiously as local expertise and available equipment must be taken into account. There is no reason why such peer review should not be extended to hospital departments. The use of medical records for peer review has been extensively evaluated (Tugwell and Dok, 1985). Peer review of radiology reports has not been evaluated but might profitably be explored. The institution of regular well-publicised chart review does lead to a change in prescribing habits, but occasional chart review and evaluation had only a weak correlation with other measures of clinical competence. Objective Structured Clinical Examination (OSCE) This format of examination has been introduced to overcome the limitations of the multiple choice system which tests only detailed knowledge. It involves the construction of a number of stations designed to reproduce the practice of the specialists being examined. By spending 5 min at each of 24 stations, up to 24 candidates could be examined simultaneously in 2 h, or 48 in two consecutive examinations, with no opportunity for collusion. It is not a new idea, having been evaluated (Harden et al., 197'5) and used for surgery at Dundee (Cuschieri et al., 1979), and paediatrics in Manchester (Smith et al., 1984). More recently it has been implemented for Obstetric and ENT Fellowship examinations by the Canadian Royal College; and within the next 2 years will be used for radiology, initially as an adjunct to the MCQ and oral examinations. The OSCE can test a wide range of skills such as the ability to plan a sequence of investigations, practical handling of procedures and complications, reporting

CONTINUINGMEDICALEDUCATIONOF RADIOLOGISTS ability, c o n s u l t a t i v e skills, a n d u n d e r s t a n d i n g of radiation p r o t e c t i o n . F o r e x a m p l e , at o n e s t a t i o n m i g h t b e a simulated patient awaiting a screening m a m m o g r a m ; she is w o r r i e d a b o u t r a d i a t i o n dosage a n d asks the candidate what risk is i n v o l v e d . A t a n o t h e r is a n u r s e who tells the c a n d i d a t e t h a t the p a t i e n t w h o has just h a d a l u n g biopsy has b e c o m e a c u t e l y d y s p n o e i c , a n d asks for instructions. A t a n o t h e r s t a t i o n is a r a d i o g r a p h which has b e e n r e p o r t e d ; the c a n d i d a t e is a s k e d to s p e a k to an irate physician w h o is u n s a t i s f i e d with the report. A t a n o t h e r the c a n d i d a t e has to discuss s o m e i n a d e q u a t e skeletal films with a r a d i o g r a p h e r who asks for instructions o n p a t i e n t p o s i t i o n i n g . T h e possibilities are endless a n d the f o r m a t l e n d s itself to testing w h a t a radiologist does as well as w h a t is k n o w n , a l t h o u g h it still suffers from testing p e r f o r m a n c e u n d e r stress r a t h e r t h a n e v e r y d a y practice. F o r r e - e x a m i n a t i o n , the O S C E type of test could be m a d e m o r e r e l e v a n t t h a n a n M C Q , a l t h o u g h t h e r e is still the p r o b l e m of devising tests for the e n o r m o u s v a r i e t y of radiological s u b s p e c i a l t y practices. D i f f e r e n t levels of skill should n o t be a p r o b l e m b e c a u s e the p r i m a r y p u r pose of p e e r r e v i e w is n o t to r e c o g n i s e high p e r f o r m e r s b u t to detect u n a c c e p t a b l y p o o r levels, b e l o w which a p r a c t i t i o n e r is r e g a r d e d as d a n g e r o u s . I n c o n c l u s i o n , c o n t i n u i n g m e d i c a l e d u c a t i o n is necessary b u t d o c u m e n t a t i o n of its b e n e f i t s is p o o r ; prog r a m m e s d e s i g n e d to p r o d u c e a n d s u s t a i n specific changes in practice s h o u l d b e d e s i g n e d , i n t r o d u c e d a n d refined. T h e r e is a n e e d to devise m e t h o d s of assessing the c o n t i n u e d c o m p e t e n c e of radiologists. T h e F e l l o w s h i p or B o a r d s f o r m a t is p r o b a b l y i n a p p r o p r i a t e since it largely tests k n o w l e d g e a n d neglects p e r f o r m a n c e . A u d i t , p e e r review a n d objective s t r u c t u r e d e x a m i n -

577

ations are p r o b a b l y b e t t e r i n s t r u m e n t s for assessing o n g o i n g c o m p e t e n c e t h a n M C Q a n d oral e x a m i n a t i o n s . REFERENCES

Belsheim, DJ (1986). Models for continuing professional education. Journal of Medical Education, 61,971-978. Cuschieri, A, Gleeson, FA, Harden, RM & Wood, RAB (1979). A new approach to a final examination in surgery. Annals of the Royal College of Surgeons of England, 61, 400--405. Davis, D, Haynes, RB, Chambers, L, Neufeld, VR, McKibbon, A & Tugwell, P (1984). The impact of CME. A methodological review of the continuing medical education literature. Evaluation and the Health Professions, 7, 251-283. Ell, SR (1984). Five hundred years of specialty certification and compulsory continuing medical education. Venice 1300-1801. Journal of the American Medical Association, 251,752-753. Fajardo, LL, Hillman, BJ, Hunter, TB et al. (1987). The impact of focused instruction on learning mammography. Investigative Radiology, 22, 990-993. Harden, R McG, Stevenson, M, Downie, WW & Wilson, GM (1975). Assessment of clinical competence using objective structured examination. British Medical Journal, 1,447-451. Laxdal, OE (1982). Needs assessment in continuing medical education: a practical guide. Journal of Medical Education, 57, 827-834. Mueller, CB (1971). Continuing assessment of medical performance. New England Journal of Medicine, 284, 1378-1380. Rockoff, SD, Davis, DO & Gaskiull, JW (1983). Physician attitude toward the competence of general diagnostic radiologists: survey and implications. American Journal of Roentgenology, 140, 539648. Smith, LJ, Price, DA & Houston, IB (1984). Objective structured clinical examination compared with other forms of student assessment. Archives of Diseases of Childhood, 59, 1173-1176. Tugwell, P. & Dok, C (1985). Medical record review. In Assessing Clinical Competence, eds Neufeld, VR & Norman GR. Ch. 8. Springer Publishing Co., New York. Ward, J (1988). Continuing medical education. Part 2. Needs assessment in continuing medical education. Medical Journal of Australia, 148, 77-80. Warren, R (1977). Social Change and Human Purpose, Rand McNally, Illinois.

Book Reviews Nuclear Medicine Therapy By J. C. Harbert Georg Thieme Verlag,

Stuttgart, 1987, 340 pp., 117 figs, DM 198. This book is intended by its authors to be a comprehensive source of practical information for nuclear medicine physicians conducting therapy with radiopharmaceuticals. Accordingly it deals only with the therapeutic use of unsealed sources. The opening and longest chapters concern radioactive iodine therapy of thyrotoxicosis and thyroid carcinoma. There is a thorough, comprehensive discussion of these subjects with an extensive bibliography. Most of the remainder of the book discusses radio-iosotope treatments which are either rarely used or obsolete, mainly various applications of intracavitary radio-colloids. An exception is a brief discussion of tumour therapy with radio-labelled antibodies which is included as a hope for the future. This is a useful reference book for clinicians undertaking radioisotope therapy and should find a place on the shelves of nuclear medicine departments. J. M. Henk Gamuts in Nuclear Medicine 2nd Edn By F. E. Datz. Appleton &

Lange, Connecticut/California, 1987, 390 pp., £37.40. This book attempts to list the differential diagnosis of all the abnormal appearances and variants encountered in radionuclide imaging. It has increased in size by about 35% since the previous edition in 1983, due both to an increased number of diagnostic categories and to an

increase in the number of entries within each section. The causes of each abnormality are listed as 'common', 'uncommon' and 'rare'. These terms should be interpreted with some caution since they may reflect referral patterns as well as reported incidence in the literature. Crohn's disease is given as an uncommon or rare cause of gastrointestinal activity on Indium leucocyte imaging but some of us see several cases of this each week. Some abnormalities lead themselves to more precise description than others and some disease descriptions are imprecise or unclear (what does 'after seizures' mean as a cause for abnormal uptake on a brain scan?). Explanatory notes are given in some cases but more would be helpful. A list of references is given at the end of each section but it is frustrating that there is no indication of which references refer to each individual abnormality. (In this respect the book's chief rival, Differential Diagnosis in Nuclear Medicine by Silberstein and McAfee, is superior.) The references given are almost exclusivelyconfined to the American literature. This is not a book to which one will refer regularly when reporting nuclear medicine investigations. The lists of causes of some abnormalities are now so long that it would take a week's reading of the literature to look them all up. It is a useful reference work for any department providing a radionuclide imaging service. It will be helpful to track down the cause of an obscure abnormality. It will also be useful for trainees wishing to read up the background to conditions encountered and will provide them with a starting point to the American literature for different types of scan. E. P. Wright