- Email: [email protected]
A computer-assisted learning tool designed to improve clinical problem-solving skills
ORIGINAL CONTRIBUTION computer, clinical usage
A Computer-Assisted Learning Tool Designed to Improve Clinical Problem-Solving Skills A computer-based teaching and learning program was developed to supplement the case load of students on a one-month emergency medicine clinical rotation and interactively measure and support improvements in their acute chest pain differential diagnostic skills. Students using the program demonstrated a significantly higher level of diagnostic accuracy than a control group (one-tailed t test; N = 88; P < .018). The success of the program was attributed in part to its unique application of advanced interactive software. Specifically, students use the software (an expert system shell) to develop a computer-based acute chest pain differential diagnostic protocol. Students can subsequently challenge their protocol with a number and variety of training cases in the program's data bank. The software interactively apprises students of the performance of their protocol in terms of its diagnostic accuracy against the cases. Students make protocol modifications to improve its diagnostic accuracy against the training cases. Interactive reassessments of the protocol's performance against the cases follow each modification. This repetitive cycle of protocol modifications rapidly followed by interactive performance reassessments against training cases appears to result in efficient and effective differential diagnostic skills improvements. A new generation of computer-based teaching and learning tools may have a significant impact on undergraduate clinical education. [Papa FJ, Meyer S: A computer-assisted learning tool designed to improve clinical problem-solving skills. Ann Emerg Med March 1989;18:269-273.] INTRODUCTION A sufficient number, variety, and complexity of clinical case encounters during medical training are required to improve differential diagnostic skills. However, students on an emergency medicine clinical rotation may not experience a case load sufficient to improve their differential diagnostic skills for the problem of acute chest pain. Students directly experience only approximately 300 to 400 cases a m o n t h at our training sites. Of these, students report observing an average of ten to 12 acute chest pain encounters during the rotation. Thus, the students' variety of exposure to acute chest pain case presentations and etiologies is extremely limited. Case complexity also constitutes an important component in the improvement of diagnostic skills. ED triage, however, channels moderate- or highrisk patients (complex cases) to senior house officers or faculty for the preliminary assessment, while low-risk (low-complexity) cases might be initially worked up by the student. Students on emergency medicine clinical rotations elsewhere may similarly experience a limited number, variety, and complexity of acute chest pain cases. We propose the null hypothesis that the case load on an emergency medicine clinical rotation is insufficient for significantly improving the acute chest pain differential diagnostic skills of students (Hlo). If the null hypothesis cannot be rejected, emergency medicine faculty must develop a teaching and learning program capable of improving their students' acute chest pain differential diagnostic skills. We conducted a study to test the research hypothesis that the acute chest pain differential diagnostic performance levels of students on an emergency medicine clinical rotation using a computer-based teaching and learning program will be
18:3March 1989
Annals of Emergency Medicine
Frank J Papa, DO, FACEP* t Steve Meyer, MS* Fort Worth, Texas From the Department of Medical Education* and Division of Emergency Medicine,t Texas College of Osteopathic Medicine, Fort Worth, Texas. Received for publication May 24, 1988. Revision received August 18, 1988. Accepted for publication December 2, 1988. Presented at the University Association for Emergency Medicine Annual Meeting in Cincinnati, May 1988. This study was sponsored by the Association of American Colleges of Osteopathic Medicine 'Focus' Fund. Address for reprints: Frank J Papa, DO, FACER Department of Medical Education, Texas College of Osteopathic Medicine, Fort Worth, Texas 76107-2690.
269/65
COMPUTER PROGRAM Papa & Meyer
higher than the performance of students without the program (H2o).
EXPERT SYSTEM SHELLS: AN ADVANCED COMPUTER TECHNOLOGY A basis for the reliability of any assessment is dependent on adequacy of the sample size. 1 It follows that a reliable assessment of a student's differential diagnostic skills for a given medical problem (eg, acute chest pain) requires that the student be challenged with a number and variety of problem-specific (chest pain) test
cases.
Due to a lack of correlation bet w e e n w r i t t e n e x a m i n a t i o n s and clinical performance,2, 3 assessments of clinical skills such as differential diagnosis are dependent on real or simulated patient encounters as test stimuli. 4,5 Unfortunately, the time restraints associated w i t h real or simulated testing cases make it difficult to perform an in-depth and num e r i c a l l y a d e q u a t e (ie, reliable), problem-specific student assessment during a clinical rotation. As a result, faculty tend to render • "global" rather than problem-specific assessments of a student's clinical differential diagnostic knowledge and skills. Pragmatic and reliable, problem-specific performance assessment methodologies and criteria continue to remain an elusive goal of medical educators. In a variety of professions, including medicine, computer-based decision-making protocols (which shall be referred to as protocols) are closely approximating the decision-making accuracy of experts in well-defined problem areas. The programs that extract the decision-making protocols from professionals are termed "expert system shells." One conceivable application of these expert system shells is their potential to extract protocols that reflect the decisionmaking skills and accuracy of not only experts but novices as well. An i n h e r e n t advantage of computer-based protocols is that they can be rapidly, automatically, objectively, and extensively challenged by a data bank consisting of a large n u m b e r and variety of problem-specific criteria cases. The level of diagnostic accuracy that the protocol achieves against these criteria cases can subsequently be used as a measure of the experts' or novices' problem-specific 66/270
diagnostic k n o w l e d g e and skills. Faculty could theoretically replace n o r m a t i v e c u r v e s w i t h criteriareferenced clinical performance parameters by this technology. For the past five years, we have refined a computer-based expert system shell for the purpose of extracting decision-making protocols and measuring problem-specific differential diagnostic skills. In a pilot study, we demonstrated that this instrument is capable of extracting a problem-specific decision-making protocol from medical students. 6 In a yet unpublished study, we have gathered evidence that suggests the shell's potential use as an assessment instrument capable of demonstrating construct validity. Student-oriented, expert system shells have the potential to provide numerically reliable, criteria-referenced, problem-specific performance assessments of complex cognitive processes such as problem-solving skills. The purpose of this study was to focus on the instrument's apparent ability to not only measure problemspecific skills levels but also to support improvements in the differential diagnostic skills of medical students for the problem acute chest pain.
METHODS Eighty-eight medical students in their third and fourth years at the Texas College of Osteopathic Medicine (TCOM) were enrolled in the study. Participation in the study was a r e q u i r e m e n t for the o n e - m o n t h emergency medicine core clinical rotation and occurred as an added component to the rotation's 15-hour didactic program. Each student was given access to an IBM or IBM-compatible personal computer. We designed our own expert s y s t e m shell software using s t r u c t u r e d c o m p i l e d BASIC. The shell required 256K of memory and ran on MS-DOS, with an 8088 CPU.
Design The study used a two-gr0up design. At TCOM, all students are randomly assigned to clinical rotation slots before starting their clinical years. Eighty-eight consecutive students were chosen for the study and were assigned to either the control (47 students) or experimental (41 students) group. There were no statistically significant differences (P < .05) beAnnals of Emergency Medicine
tween the two groups in prestudy parameters (MCAT score, preclinical grade point average, number of clinical rotations before starting their emergency medicine rotation), and independent variable study parameters (case load, rotation didactics, and bedside teaching). The latter two were measured by finding no differences between the two groups in their final emergency medicine clinical grades. The treatment group was required to develop an acute chest pain diagnostic protocol on the first day of their four-week rotation (pretest). Treatment occurred at work sessions held during the second and third weeks of the rotation. The treatment given during these work sessions consisted of stimuli (access to 27 training cases) and feedback (interactive computer-based assessments of the diagnostic accuracy of their chest pain protocol against the training cases). At any time during these work sessions, students were free to modify the knowledge base of their diagnostic protocol to i m p r o v e its d i a g n o s t i c a c c u r a c y against the training cases. During the fourth week of the rotation, students were given one last opportunity to modify their protocol (post-test}. The control group was required to develop their first and only acute chest pain diagnostic protocol during the fourth week of their four-week rotation (post-test). They were not given access to the training cases or any interactive computer-based assessment of their protocols' diagnostic accuracy. Significance was set at P < .05. The following nine more common or important acute chest pain etiologies comprised the differentials under consideration in our study: myocardial infarction, myocardial a n g i n a or i s c h e m i a , p e r i c a r d i t i s , pneumonia, pneumothorax, pulmonary embolus, dissecting thoracic aortic aneurysm, esophageal or upper gastrointestinal disorders, and thoracic m u s c u l o s k e l e t a l etiologies. From a previous pilot study, the authors identified 67 h i s t o r i c a l and physical bedside findings that students believed were needed to form a clinical bedside impression that differentiated the nine previously mentioned etiologies (Figure 1). No laboratory, ancillary, or other technologybased findings were among the list of 18:3 March 1989
Historical Findings Demographics Male Female Age more than 40 Findings associated with chest pain Prodrome flu-like symptoms (excluding fever) Gradual dyspnea Sudden dyspnea Fever Rash Syncope Dysphagia Palpitations Hoarseness or voice changes Preceding trauma Onset of chest pain Sudden Gradual Duration of chest pain Lasting few seconds to minutes Lasting less than 20.to 30 minutes Lasting more than 20 to 30 minutes but less than 24 hours Lasting more than 24 hours Is pain present at time of arrival? Quality of chest pain Sharp, stabbing, fleeting Dull, pressure, squeezing Burning Location of chest pain Substernal, left precordial At or lateral to costochondral border Posterior thoracic Radiation of chest pain None To neck, arm, or jaw To back Factors that altered chest pain Food Medication Rest Movement, posturing Exertion Risk factors Drugs Immobility Previous surgery
findings. Historical findings included inquiries regarding acute chest pain (eg, quality, duration, onset). Physical findings included cardiac, respiratory, and abdominal, among others. The knowledge base that the students transferred to the expert system shell consisted of their understanding of the relationship between each of these 67 clinical findings and the nine differentials. More specifi18:3 March 1989
Physical Findings HEENT Palpably enlarged thyroid Respiratory Wheezes Rales Rhonchi Productive cough Hemoptysis Unilateral decreased breath sounds Unilateral hyperresonance Tachypnea Cardiovascular Blood pressure or pulses asymmetric Tachycardia S3/S4 gallops Precordial friction rub Accentuated pulmonic second sound Aortic regurgitant murmur Neck vein distension Superficial or deep vein thrombosis Pulsus paradox Hypertension Abdominal Ascites Pulsatile mass Subxyphoid, epigastric tenderness Muscuioskeletal Reproducible point tenderness Reprodbcible pain with movement/posturing Dermal Evidence of soft tissue trauma Diaphoresis Cool, pale, moist skin Cyanosis Neurologic Display of anxiety Pain relief at ED admission
cally, students were requested to estimate and input the percentage of patients with a given etiology likely to have a given finding (Figure 2). Therefore, for each of the 67 findings, the students were asked to estimate what percentage of patients with the disease (eg, myocardial infarction, angina or ischemia, pericarditis) had the finding. These estimates of relationships represented Annals of EmergencyMedicine
FIGURE 1. The 67 findings used in the study. the students' acute chest pain differential diagnostic knowledge base. Once a student's knowledge base was extracted,' the expert system shell would transform it into a decisionmaking diagnostic protocol. Stimuli consisted of a data bank of 271/67
COMPUTER PROGRAM Papa & Meyer
FIGURE 2. Students estimate the relationship between each of the nine diseases under consideration and a specific finding. 27 training cases (each case containing the presence or absence of the same 67 findings) that the student could b o t h a s s i m i l a t e and use to challenge the diagnostic accuracy of their computer-resident, diagnostic protocol. The n u m b e r of t r a i n i n g cases that a student's protocol correctly diagnosed constituted the students' immediate computer-based feedback. To improve the protocol's diagnostic accuracy, the student would need to m a k e changes in the protocol's k n o w l e d g e base. T h e objective of these knowledge base changes was to b e t t e r a p p r o x i m a t e the true incidence or relationship between given diseases and findings. Thus, improvements in the students' relationship approximations (ie, better approximations of the true incidence of findings associated with given diseases) would result in improvements of their computer-based protocol's diagnostic accuracy against the training cases. Improvements in the protocols' accuracy would reflect improved differential diagnostic knowledge and skills (learning). A potential criteria case contained real patient data associated with the previously m e n t i o n e d 67 findings. The clinical bedside diagnosis made by emergency medicine clinical faculty members was recorded for each case. A case was entered into the criteria case data bank if, on blinded review of the data, one of the authors concurred with the clinical impression. A total of 72 criteria cases were collected. At the conclusion of the study, each student's diagnostic protocol was prepared by the ESS to undergo a computer-based assessment of its diagnostic accuracy. The results of each student's protocol performance against the 72 criteria cases (ie, number of correct diagnoses attained by the protocol) was recorded. These results were transferred to SPSS/PC +, a standardized statistical package that performed all subsequent analysis. Higher p e r f o r m a n c e levels (ie, greater n u m b e r s of correctly diagnosed criteria cases) were interpreted as symbolizing greater acute chest 68/272
Developing the Acute Chest Pain Knowledge Base: Defining Relationships Given that we are dealing with patients presenting with acute chest pain, what percentage of patients with the following diseases have substernat or left precordial chest pain? Myocardial infarction 80 Myocardial angina or ischemia 70 Pericarditis ? Pneumonia Pneumothorax Pulmonary embolus Dissecting thoracic aortic aneurysm Esophageal or upper gastrointestinal disorders Thoracic musculoskeletal etiologies
pain diagnostic knowledge and skills.
RESULTS Hlo A m e a s u r e m e n t was made to det e r m i n e if no significant improvements in the students' acute chest pain diagnostic skills developed as a result of their emergency medicine clinical case load. Two-tailed Student's t test analysis of independent samples was performed. The performance of the protocol developed by the treatment group on the first day of their emergency medicine rotation served as the baseline level of students' acute chest pain diagnostic skills before an emergency medicine clinical rotation. The performance of the control group's protocol (their first and only protocol obtained during the fourth week of the rotation) was compared with the performance of the baseline protocol. T h e r e w e r e no s i g n i f i c a n t performance differences between the first day and last week protocols (N = 88; P < .34). The null hypothesis was retained.
H2o A m e a s u r e m e n t was made to det e r m i n e if t h e t r e a t m e n t g r o u p achieved a lower level of diagnostic performance than the control group. One-tailed, Student's t test analysis of independent samples was used. The performance of protocols (treatm e n t c o m p a r e d w i t h control) obtained during the last week of the e m e r g e n c y medicine rotation was compared. The performance of the Annals of Emergency Medicine
t r e a t m e n t group was significantly higher than the control group (N = 88; P < .018). The null hypothesis was rejected and the research hypothesis (that treatment resulted in s i g n i f i c a n t l y higher p e r f o r m a n c e levels) was accepted. A m e a s u r e m e n t was made to determine the magnitude of improvement in the treatment group. The individuals in this group served as their own controls. The p e r f o r m a n c e of the treatment group's initial first day protocol was compared with the performance of their own fourth-week protocol. Thus, a pretest-post-test, one-way, Student's t test analysis of related samples was conducted on the treatment group. Highly significant improvements were found comparing the performance of their posttest with their own pretest protocol (N = 47; P < .0001).
DISCUSSION Emergency medicine teaching faculty may question whether their students experience the number, variety, and complexity of case encounters sufficient for improving acute chest pain differential d i a g n o s t i c skills. Unfortunately, current assessm e n t methodologies do not readily support critically testing such an hypothesis. We developed a student-oriented expert s y s t e m shell, an advanced, c o m p u t e r - b a s e d t e c h n o l o g y , as a means of implementing a problemspecific performance assessment of s t u d e n t s ' knowledge and skills for the p r o b l e m of a c u t e c h e s t pain. 18:3 March 1989
W i t h this technology, w e a t t e m p t e d to assess t h e a d e q u a c y of s t u d e n t s ' a c u t e c h e s t p a i n c a s e l o a d on a n emergency medicine rotation. This was also used to s u p p l e m e n t the stud e n t s ' n u m b e r a n d v a r i e t y of a c u t e c h e s t p a i n case e n c o u n t e r s and, in addition, to help support the s t u d e n t in refining acute chest pain differential diagnostic skills. T h e r e s u l t s of our s t u d y suggest that this technology can be used to a s s e s s t h e a d e q u a c y of a s t u d e n t ' s c l i n i c a l c a s e load, s u p p l e m e n t t h e student's case load, and support efficient and effective i m p r o v e m e n t s in students' differential diagnostic skills for the p r o b l e m of chest pain. T h i s t e c h n o l o g y c o u l d have b e e n used in an effort to assess t h e adeq u a c y of t h e case load, s u p p l e m e n t t h e case load, and i m p r o v e s t u d e n t differential diagnostic skills for other m e d i c a l p r o b l e m s s u c h as dyspnea, cephalgia, j o i n t pain, or rash. Addit i o n a l a p p l i c a t i o n s for s i m i l a r l y designed and applied expert system shells could be in the area of resident
18:3 March 1989
s k i l l s a s s e s s m e n t s and Board certification e x a m i n a t i o n s .
by w h i c h teaching and learning occur on clinical rotations.
CONCLUSION T h e r e s u l t s of o u r s t u d y suggest that advanced c o m p u t e r - b a s e d technologies can provide a problemspecific p e r f o r m a n c e a s s e s s m e n t of diagnostic skills on s t u d e n t s during a c l i n i c a l r o t a t i o n ; h e l p t e a c h i n g facu l t y d e t e r m i n e if their s t u d e n t s ' clinical case load is sufficient for improving d i f f e r e n t i a l d i a g n o s t i c s k i l l s for selected high-impact patient problems; s u p p l e m e n t the s t u d e n t s ' clinical case load; i n t e r a c t i v e l y p r o v i d e s t u d e n t s w i t h c o n s t r u c t i v e feedback regarding their differential diagnostic skills; and in c o m b i n a t i o n w i t h case load s u p p l e m e n t a t i o n and c o n s t r u c tive, i n t e r a c t i v e feedback, r e s u l t in h i g h l y s i g n i f i c a n t i m p r o v e m e n t s in the students' differential diagnostic skills. A d v a n c e d c o m p u t e r - b a s e d technologies m a y e v e n t u a l l y play a role in redefining s t u d e n t a s s e s s m e n t m e t h odologies w h i l e m o d i f y i n g the m e a n s
REFERENCES
Annals of Emergency Medicine
1. Nitko AJ: Educational Test and Measurement: An Introduction. New York, Harcourt, Brace, and Jovanovich, Inc, 1983.
2. Goran MJ, Wiltiamson JW, Gonella JS: The validity of patient management problems, l
Med Educ 1973;48:171-177.
3. Wingard JR, Williamson JW: Grades as predictors of physicians' career performance: An evaluative literature review. J Med Educ 1973; 48:311-322. 4. Barrows H8, Williams RG, Moy RH: A comprehensive performance-based assessment of fourth-year students' clinical skills. J Med Educ 1987;62:805-809. 5. Norman GR, Tugwell P, Feightner JW: A comparison of resident performance on real and simulated patients. J Med Educ 1982;57: 708-715. 6. Papa FJ, Meyer S: An expert program shell designed for extracting "disease prototypes" and their use as models for exploring the "strong problem solving methods" employed in clinical reasoning, in Hart IR (ed): Further Developm e n t s in Assessing Clinical Competence.
Quebec, Heal Publications, 1987, p 354-364.
273/69
A Computer-Assisted Learning Tool Designed to Improve Clinical Problem-Solving Skills A computer-based teaching and learning program was developed to supplement the case load of students on a one-month emergency medicine clinical rotation and interactively measure and support improvements in their acute chest pain differential diagnostic skills. Students using the program demonstrated a significantly higher level of diagnostic accuracy than a control group (one-tailed t test; N = 88; P < .018). The success of the program was attributed in part to its unique application of advanced interactive software. Specifically, students use the software (an expert system shell) to develop a computer-based acute chest pain differential diagnostic protocol. Students can subsequently challenge their protocol with a number and variety of training cases in the program's data bank. The software interactively apprises students of the performance of their protocol in terms of its diagnostic accuracy against the cases. Students make protocol modifications to improve its diagnostic accuracy against the training cases. Interactive reassessments of the protocol's performance against the cases follow each modification. This repetitive cycle of protocol modifications rapidly followed by interactive performance reassessments against training cases appears to result in efficient and effective differential diagnostic skills improvements. A new generation of computer-based teaching and learning tools may have a significant impact on undergraduate clinical education. [Papa FJ, Meyer S: A computer-assisted learning tool designed to improve clinical problem-solving skills. Ann Emerg Med March 1989;18:269-273.] INTRODUCTION A sufficient number, variety, and complexity of clinical case encounters during medical training are required to improve differential diagnostic skills. However, students on an emergency medicine clinical rotation may not experience a case load sufficient to improve their differential diagnostic skills for the problem of acute chest pain. Students directly experience only approximately 300 to 400 cases a m o n t h at our training sites. Of these, students report observing an average of ten to 12 acute chest pain encounters during the rotation. Thus, the students' variety of exposure to acute chest pain case presentations and etiologies is extremely limited. Case complexity also constitutes an important component in the improvement of diagnostic skills. ED triage, however, channels moderate- or highrisk patients (complex cases) to senior house officers or faculty for the preliminary assessment, while low-risk (low-complexity) cases might be initially worked up by the student. Students on emergency medicine clinical rotations elsewhere may similarly experience a limited number, variety, and complexity of acute chest pain cases. We propose the null hypothesis that the case load on an emergency medicine clinical rotation is insufficient for significantly improving the acute chest pain differential diagnostic skills of students (Hlo). If the null hypothesis cannot be rejected, emergency medicine faculty must develop a teaching and learning program capable of improving their students' acute chest pain differential diagnostic skills. We conducted a study to test the research hypothesis that the acute chest pain differential diagnostic performance levels of students on an emergency medicine clinical rotation using a computer-based teaching and learning program will be
18:3March 1989
Annals of Emergency Medicine
Frank J Papa, DO, FACEP* t Steve Meyer, MS* Fort Worth, Texas From the Department of Medical Education* and Division of Emergency Medicine,t Texas College of Osteopathic Medicine, Fort Worth, Texas. Received for publication May 24, 1988. Revision received August 18, 1988. Accepted for publication December 2, 1988. Presented at the University Association for Emergency Medicine Annual Meeting in Cincinnati, May 1988. This study was sponsored by the Association of American Colleges of Osteopathic Medicine 'Focus' Fund. Address for reprints: Frank J Papa, DO, FACER Department of Medical Education, Texas College of Osteopathic Medicine, Fort Worth, Texas 76107-2690.
269/65
COMPUTER PROGRAM Papa & Meyer
higher than the performance of students without the program (H2o).
EXPERT SYSTEM SHELLS: AN ADVANCED COMPUTER TECHNOLOGY A basis for the reliability of any assessment is dependent on adequacy of the sample size. 1 It follows that a reliable assessment of a student's differential diagnostic skills for a given medical problem (eg, acute chest pain) requires that the student be challenged with a number and variety of problem-specific (chest pain) test
cases.
Due to a lack of correlation bet w e e n w r i t t e n e x a m i n a t i o n s and clinical performance,2, 3 assessments of clinical skills such as differential diagnosis are dependent on real or simulated patient encounters as test stimuli. 4,5 Unfortunately, the time restraints associated w i t h real or simulated testing cases make it difficult to perform an in-depth and num e r i c a l l y a d e q u a t e (ie, reliable), problem-specific student assessment during a clinical rotation. As a result, faculty tend to render • "global" rather than problem-specific assessments of a student's clinical differential diagnostic knowledge and skills. Pragmatic and reliable, problem-specific performance assessment methodologies and criteria continue to remain an elusive goal of medical educators. In a variety of professions, including medicine, computer-based decision-making protocols (which shall be referred to as protocols) are closely approximating the decision-making accuracy of experts in well-defined problem areas. The programs that extract the decision-making protocols from professionals are termed "expert system shells." One conceivable application of these expert system shells is their potential to extract protocols that reflect the decisionmaking skills and accuracy of not only experts but novices as well. An i n h e r e n t advantage of computer-based protocols is that they can be rapidly, automatically, objectively, and extensively challenged by a data bank consisting of a large n u m b e r and variety of problem-specific criteria cases. The level of diagnostic accuracy that the protocol achieves against these criteria cases can subsequently be used as a measure of the experts' or novices' problem-specific 66/270
diagnostic k n o w l e d g e and skills. Faculty could theoretically replace n o r m a t i v e c u r v e s w i t h criteriareferenced clinical performance parameters by this technology. For the past five years, we have refined a computer-based expert system shell for the purpose of extracting decision-making protocols and measuring problem-specific differential diagnostic skills. In a pilot study, we demonstrated that this instrument is capable of extracting a problem-specific decision-making protocol from medical students. 6 In a yet unpublished study, we have gathered evidence that suggests the shell's potential use as an assessment instrument capable of demonstrating construct validity. Student-oriented, expert system shells have the potential to provide numerically reliable, criteria-referenced, problem-specific performance assessments of complex cognitive processes such as problem-solving skills. The purpose of this study was to focus on the instrument's apparent ability to not only measure problemspecific skills levels but also to support improvements in the differential diagnostic skills of medical students for the problem acute chest pain.
METHODS Eighty-eight medical students in their third and fourth years at the Texas College of Osteopathic Medicine (TCOM) were enrolled in the study. Participation in the study was a r e q u i r e m e n t for the o n e - m o n t h emergency medicine core clinical rotation and occurred as an added component to the rotation's 15-hour didactic program. Each student was given access to an IBM or IBM-compatible personal computer. We designed our own expert s y s t e m shell software using s t r u c t u r e d c o m p i l e d BASIC. The shell required 256K of memory and ran on MS-DOS, with an 8088 CPU.
Design The study used a two-gr0up design. At TCOM, all students are randomly assigned to clinical rotation slots before starting their clinical years. Eighty-eight consecutive students were chosen for the study and were assigned to either the control (47 students) or experimental (41 students) group. There were no statistically significant differences (P < .05) beAnnals of Emergency Medicine
tween the two groups in prestudy parameters (MCAT score, preclinical grade point average, number of clinical rotations before starting their emergency medicine rotation), and independent variable study parameters (case load, rotation didactics, and bedside teaching). The latter two were measured by finding no differences between the two groups in their final emergency medicine clinical grades. The treatment group was required to develop an acute chest pain diagnostic protocol on the first day of their four-week rotation (pretest). Treatment occurred at work sessions held during the second and third weeks of the rotation. The treatment given during these work sessions consisted of stimuli (access to 27 training cases) and feedback (interactive computer-based assessments of the diagnostic accuracy of their chest pain protocol against the training cases). At any time during these work sessions, students were free to modify the knowledge base of their diagnostic protocol to i m p r o v e its d i a g n o s t i c a c c u r a c y against the training cases. During the fourth week of the rotation, students were given one last opportunity to modify their protocol (post-test}. The control group was required to develop their first and only acute chest pain diagnostic protocol during the fourth week of their four-week rotation (post-test). They were not given access to the training cases or any interactive computer-based assessment of their protocols' diagnostic accuracy. Significance was set at P < .05. The following nine more common or important acute chest pain etiologies comprised the differentials under consideration in our study: myocardial infarction, myocardial a n g i n a or i s c h e m i a , p e r i c a r d i t i s , pneumonia, pneumothorax, pulmonary embolus, dissecting thoracic aortic aneurysm, esophageal or upper gastrointestinal disorders, and thoracic m u s c u l o s k e l e t a l etiologies. From a previous pilot study, the authors identified 67 h i s t o r i c a l and physical bedside findings that students believed were needed to form a clinical bedside impression that differentiated the nine previously mentioned etiologies (Figure 1). No laboratory, ancillary, or other technologybased findings were among the list of 18:3 March 1989
Historical Findings Demographics Male Female Age more than 40 Findings associated with chest pain Prodrome flu-like symptoms (excluding fever) Gradual dyspnea Sudden dyspnea Fever Rash Syncope Dysphagia Palpitations Hoarseness or voice changes Preceding trauma Onset of chest pain Sudden Gradual Duration of chest pain Lasting few seconds to minutes Lasting less than 20.to 30 minutes Lasting more than 20 to 30 minutes but less than 24 hours Lasting more than 24 hours Is pain present at time of arrival? Quality of chest pain Sharp, stabbing, fleeting Dull, pressure, squeezing Burning Location of chest pain Substernal, left precordial At or lateral to costochondral border Posterior thoracic Radiation of chest pain None To neck, arm, or jaw To back Factors that altered chest pain Food Medication Rest Movement, posturing Exertion Risk factors Drugs Immobility Previous surgery
findings. Historical findings included inquiries regarding acute chest pain (eg, quality, duration, onset). Physical findings included cardiac, respiratory, and abdominal, among others. The knowledge base that the students transferred to the expert system shell consisted of their understanding of the relationship between each of these 67 clinical findings and the nine differentials. More specifi18:3 March 1989
Physical Findings HEENT Palpably enlarged thyroid Respiratory Wheezes Rales Rhonchi Productive cough Hemoptysis Unilateral decreased breath sounds Unilateral hyperresonance Tachypnea Cardiovascular Blood pressure or pulses asymmetric Tachycardia S3/S4 gallops Precordial friction rub Accentuated pulmonic second sound Aortic regurgitant murmur Neck vein distension Superficial or deep vein thrombosis Pulsus paradox Hypertension Abdominal Ascites Pulsatile mass Subxyphoid, epigastric tenderness Muscuioskeletal Reproducible point tenderness Reprodbcible pain with movement/posturing Dermal Evidence of soft tissue trauma Diaphoresis Cool, pale, moist skin Cyanosis Neurologic Display of anxiety Pain relief at ED admission
cally, students were requested to estimate and input the percentage of patients with a given etiology likely to have a given finding (Figure 2). Therefore, for each of the 67 findings, the students were asked to estimate what percentage of patients with the disease (eg, myocardial infarction, angina or ischemia, pericarditis) had the finding. These estimates of relationships represented Annals of EmergencyMedicine
FIGURE 1. The 67 findings used in the study. the students' acute chest pain differential diagnostic knowledge base. Once a student's knowledge base was extracted,' the expert system shell would transform it into a decisionmaking diagnostic protocol. Stimuli consisted of a data bank of 271/67
COMPUTER PROGRAM Papa & Meyer
FIGURE 2. Students estimate the relationship between each of the nine diseases under consideration and a specific finding. 27 training cases (each case containing the presence or absence of the same 67 findings) that the student could b o t h a s s i m i l a t e and use to challenge the diagnostic accuracy of their computer-resident, diagnostic protocol. The n u m b e r of t r a i n i n g cases that a student's protocol correctly diagnosed constituted the students' immediate computer-based feedback. To improve the protocol's diagnostic accuracy, the student would need to m a k e changes in the protocol's k n o w l e d g e base. T h e objective of these knowledge base changes was to b e t t e r a p p r o x i m a t e the true incidence or relationship between given diseases and findings. Thus, improvements in the students' relationship approximations (ie, better approximations of the true incidence of findings associated with given diseases) would result in improvements of their computer-based protocol's diagnostic accuracy against the training cases. Improvements in the protocols' accuracy would reflect improved differential diagnostic knowledge and skills (learning). A potential criteria case contained real patient data associated with the previously m e n t i o n e d 67 findings. The clinical bedside diagnosis made by emergency medicine clinical faculty members was recorded for each case. A case was entered into the criteria case data bank if, on blinded review of the data, one of the authors concurred with the clinical impression. A total of 72 criteria cases were collected. At the conclusion of the study, each student's diagnostic protocol was prepared by the ESS to undergo a computer-based assessment of its diagnostic accuracy. The results of each student's protocol performance against the 72 criteria cases (ie, number of correct diagnoses attained by the protocol) was recorded. These results were transferred to SPSS/PC +, a standardized statistical package that performed all subsequent analysis. Higher p e r f o r m a n c e levels (ie, greater n u m b e r s of correctly diagnosed criteria cases) were interpreted as symbolizing greater acute chest 68/272
Developing the Acute Chest Pain Knowledge Base: Defining Relationships Given that we are dealing with patients presenting with acute chest pain, what percentage of patients with the following diseases have substernat or left precordial chest pain? Myocardial infarction 80 Myocardial angina or ischemia 70 Pericarditis ? Pneumonia Pneumothorax Pulmonary embolus Dissecting thoracic aortic aneurysm Esophageal or upper gastrointestinal disorders Thoracic musculoskeletal etiologies
pain diagnostic knowledge and skills.
RESULTS Hlo A m e a s u r e m e n t was made to det e r m i n e if no significant improvements in the students' acute chest pain diagnostic skills developed as a result of their emergency medicine clinical case load. Two-tailed Student's t test analysis of independent samples was performed. The performance of the protocol developed by the treatment group on the first day of their emergency medicine rotation served as the baseline level of students' acute chest pain diagnostic skills before an emergency medicine clinical rotation. The performance of the control group's protocol (their first and only protocol obtained during the fourth week of the rotation) was compared with the performance of the baseline protocol. T h e r e w e r e no s i g n i f i c a n t performance differences between the first day and last week protocols (N = 88; P < .34). The null hypothesis was retained.
H2o A m e a s u r e m e n t was made to det e r m i n e if t h e t r e a t m e n t g r o u p achieved a lower level of diagnostic performance than the control group. One-tailed, Student's t test analysis of independent samples was used. The performance of protocols (treatm e n t c o m p a r e d w i t h control) obtained during the last week of the e m e r g e n c y medicine rotation was compared. The performance of the Annals of Emergency Medicine
t r e a t m e n t group was significantly higher than the control group (N = 88; P < .018). The null hypothesis was rejected and the research hypothesis (that treatment resulted in s i g n i f i c a n t l y higher p e r f o r m a n c e levels) was accepted. A m e a s u r e m e n t was made to determine the magnitude of improvement in the treatment group. The individuals in this group served as their own controls. The p e r f o r m a n c e of the treatment group's initial first day protocol was compared with the performance of their own fourth-week protocol. Thus, a pretest-post-test, one-way, Student's t test analysis of related samples was conducted on the treatment group. Highly significant improvements were found comparing the performance of their posttest with their own pretest protocol (N = 47; P < .0001).
DISCUSSION Emergency medicine teaching faculty may question whether their students experience the number, variety, and complexity of case encounters sufficient for improving acute chest pain differential d i a g n o s t i c skills. Unfortunately, current assessm e n t methodologies do not readily support critically testing such an hypothesis. We developed a student-oriented expert s y s t e m shell, an advanced, c o m p u t e r - b a s e d t e c h n o l o g y , as a means of implementing a problemspecific performance assessment of s t u d e n t s ' knowledge and skills for the p r o b l e m of a c u t e c h e s t pain. 18:3 March 1989
W i t h this technology, w e a t t e m p t e d to assess t h e a d e q u a c y of s t u d e n t s ' a c u t e c h e s t p a i n c a s e l o a d on a n emergency medicine rotation. This was also used to s u p p l e m e n t the stud e n t s ' n u m b e r a n d v a r i e t y of a c u t e c h e s t p a i n case e n c o u n t e r s and, in addition, to help support the s t u d e n t in refining acute chest pain differential diagnostic skills. T h e r e s u l t s of our s t u d y suggest that this technology can be used to a s s e s s t h e a d e q u a c y of a s t u d e n t ' s c l i n i c a l c a s e load, s u p p l e m e n t t h e student's case load, and support efficient and effective i m p r o v e m e n t s in students' differential diagnostic skills for the p r o b l e m of chest pain. T h i s t e c h n o l o g y c o u l d have b e e n used in an effort to assess t h e adeq u a c y of t h e case load, s u p p l e m e n t t h e case load, and i m p r o v e s t u d e n t differential diagnostic skills for other m e d i c a l p r o b l e m s s u c h as dyspnea, cephalgia, j o i n t pain, or rash. Addit i o n a l a p p l i c a t i o n s for s i m i l a r l y designed and applied expert system shells could be in the area of resident
18:3 March 1989
s k i l l s a s s e s s m e n t s and Board certification e x a m i n a t i o n s .
by w h i c h teaching and learning occur on clinical rotations.
CONCLUSION T h e r e s u l t s of o u r s t u d y suggest that advanced c o m p u t e r - b a s e d technologies can provide a problemspecific p e r f o r m a n c e a s s e s s m e n t of diagnostic skills on s t u d e n t s during a c l i n i c a l r o t a t i o n ; h e l p t e a c h i n g facu l t y d e t e r m i n e if their s t u d e n t s ' clinical case load is sufficient for improving d i f f e r e n t i a l d i a g n o s t i c s k i l l s for selected high-impact patient problems; s u p p l e m e n t the s t u d e n t s ' clinical case load; i n t e r a c t i v e l y p r o v i d e s t u d e n t s w i t h c o n s t r u c t i v e feedback regarding their differential diagnostic skills; and in c o m b i n a t i o n w i t h case load s u p p l e m e n t a t i o n and c o n s t r u c tive, i n t e r a c t i v e feedback, r e s u l t in h i g h l y s i g n i f i c a n t i m p r o v e m e n t s in the students' differential diagnostic skills. A d v a n c e d c o m p u t e r - b a s e d technologies m a y e v e n t u a l l y play a role in redefining s t u d e n t a s s e s s m e n t m e t h odologies w h i l e m o d i f y i n g the m e a n s
REFERENCES
Annals of Emergency Medicine
1. Nitko AJ: Educational Test and Measurement: An Introduction. New York, Harcourt, Brace, and Jovanovich, Inc, 1983.
2. Goran MJ, Wiltiamson JW, Gonella JS: The validity of patient management problems, l
Med Educ 1973;48:171-177.
3. Wingard JR, Williamson JW: Grades as predictors of physicians' career performance: An evaluative literature review. J Med Educ 1973; 48:311-322. 4. Barrows H8, Williams RG, Moy RH: A comprehensive performance-based assessment of fourth-year students' clinical skills. J Med Educ 1987;62:805-809. 5. Norman GR, Tugwell P, Feightner JW: A comparison of resident performance on real and simulated patients. J Med Educ 1982;57: 708-715. 6. Papa FJ, Meyer S: An expert program shell designed for extracting "disease prototypes" and their use as models for exploring the "strong problem solving methods" employed in clinical reasoning, in Hart IR (ed): Further Developm e n t s in Assessing Clinical Competence.
Quebec, Heal Publications, 1987, p 354-364.
273/69