- Email: [email protected]
A computer-assisted instruction (CAI) program in diseases of the thyroid gland (THYROID)
COMPUTERS
AND
BIOMEDICAL
RESEARCH
l&133-146
(1978)
A Computer-Assisted Instruction (CAI) Program Diseases of the Thyroid Gland (THYROID)
in
A. ARTHUR STEELE,* PAUL J. DAVIS, EDWARD P. HOFFER, AND KATHLEEN T. FAMIGLIETTI Endocrinology Division, Department of Medicine, Medical School of the State University of New York at Buffalo, Buffalo, New York 14222 and the Laboratory of Computer Science, Massachusetts General Hospital, Boston, Massachusetts 01915
Received April 18,1977 A computerized teaching program(THYROID)hasbeenauthored for thyroiddiseases which utilizesa medicaleducationdriver programof the Laboratoryof ComputerScienceat the Massachusetts GeneralHospital.Theoutputof thisteaching programhasfour components: (1) an inexhaustible generated pool of patients’protiles;(2) a glossaryof permissible clinical questions, diagnoses, andtherapies; (3) a teachingmessage for everyglossaryitem;and(4) an evaluationof userperformance. THYROID teaches problemrecognition andmanagement of 20 thyroiddiseases by presenting theuserwith a randomlygenerated “patient.”Theprogramwas constructed by writingquestion andresponse setsrelevantto thehistory,physicalexamination, andlaboratorytestsof patientswithvariousthyroiddisorders. In additionto teachingmessages for everyglossaryitem,the usermay requesta “consultant”who guidesthe userstepwise throughthe differentialdiagnosis to the final correctdiagnosis. THYROID canbe usedfor medicalstudentandhouse-staff educationand for reviewof thyroid diseases by practicing physicians.
Computer technology has greatest impact in medical education when applied to reinforcement of cognitive skills (Z-5). Computer programs have been written to assistclinicians in diagnosisand managementof actual patients with such endocrine and metabolic problems as acid-base disorders (6), hypercalcemia (7), and thyroid diseases(8,9). Instructional programs have been developed for patient case studies (IO), cardiopulmonary resuscitation (II, 12), and over 40 other subjects currently available nationally through a cooperative computer-assisted instruction (CAI) program network (13).’ In this paper we describea program which teachesdiseasesof the thyroid gland by synthesizing patient profiles. The program was remote authored, utilizing a c1inica.l data base developed at the State University of New York at Buffalo, and a driver program was written at the Laboratory of Computer Science, Massachusetts General Hospital. * Present address: 550CabotStreet,Beverly,Mass.01915 LTheHealthEducationNetwork. 133
001~809/78/0112-0133$02.oo/‘o Copyright @ 1978 by Academic Press. Inc. All rights of reproduction in any form reserved. Printed in Great Britain
134
STEELE,
DAVIS,
AND
FAMIGLIETTI
THE PROGRAM
Assembly of a data base for the teaching of thyroid disease was seen as a practical goal because sources of clinical statistics were available to define the incidence of certain symptoms and signs (14-19). In THYROID the data were assembled as items of a glossary (Table I) which included patient history, physical examination, results of laboratory tests, diagnostic possibilities, and therapeutic alternatives (Fig. 1). The format by which the data base was constructed is described below. TABLE THE COMPONENTS
I
OF THE THYROID
GLOSSARY LIST: HISTORY ITEMS PHYSICAL EXAMINATION LABORATORY TESTS THERAPEUTIC OPTIONS DIAGNOSES
GLOSSARY
ITEMS
Question and response sets were written for a total of 75 items of history, physical examination, laboratory testing, and patient management (Fig. 1). Each set consists of an inquiry-to be selected by the user-and several different responses to the inquiry. As an example, heat intolerance is the subject of item # 100. Patients with thyroid diseases who are eumetabolic or hypometabolic would not give a positive response; hence, as shown in Fig. 2, the question and response set includes responses appropriate for patients who are thyrotoxic and those who are euthyroid. The program author listed 20 disorders of the thyroid, assigned them numbers (see Fig. 1,900 series items), and for each of the 20 diagnoses estimated the probability of a given response being given to a particular history or physical examination item. Estimates were based on the published data (14-18) and modified in some instances to emphasize specific principles. The statistics upon which responses were based appear to be valid when applied to various patient populations (19,20). A list of response probabilities is presented in Fig. 3. As mentioned, some thyroid disorders do not alter the metabolic status of the patient (e.g., nontoxic goiter) and in these patients the responses to most questions are those of individuals without thyroid disease. The code numbers of these euthyroid disorders are listed at the bottom of Fig. 3, and their assigned response probabilities are denoted by the entry “*NL” midway through the figure. This procedure was followed for each item of the history and physical examination. Similarly, appropriate sets for all the laboratory tests were constructed. Finally, therapeutic options were written for the various diseases. For each of the 20 thyroid disorders, then, a response probability was assigned for every glossary item. The probability of many signs and symptoms for several thyroid disorders (e.g., hyperthyroidism, hypothyroidism) has been weighted in “indexes” (14-20). The
CA1
‘Tl-IYI~l:I:r
PROGRAM
FOR
THYROID
DISEASES
135
LI Ilk I ‘ILM!:;:
YCI! I IJNi:oMI~~ I’llu RBI I::‘I’ I:lCl ItIC r l/JI;A rt.II”I:) 1:11? it.II:i A I t!:Il ttiJ~l?l!: MAI\I 1l..lnN yi:lli 11:,1”\1 1’) rtt:“i’ ME! YOU I::Xi::ES!ii’i:UI.~I..Y ‘r~l.I::li:I.~ (‘III WV hltl:‘l’i rtC] Y(:](J RE(:;OHE:: $;HC)IIT (:)I::’ Rlil: A-rI..I Mi:lt?l:: I:::rr,!:;‘I I. Y’i’ I:10 E: Ci Y 13IJ li H Ei:A H ‘I’ li: V E!:I< R ti. A I ‘I I:) (.I 1:.n !,i I i 1I I ::;I’; 1:I B I : ,7 .I $: ‘? HOW ARE: YOUR NKRUFS” HAVE YOU l-Inn ANY CHANOI~. 1:N Yi.JlJtG nl:.‘l:‘li:. r ‘I.1 I:.‘* I-4AS Y 0 IJ R Wt. :I:(‘3 H ‘T i:: ti AN 0 E I:( M IJ I::I I IRI:: i:: I!: N I’ I...Y ‘1’ IHAVE!: YOU IHAD ANY i::tiANI;E:: :I:N Yl.lIJK I?~lWIzi:l... HfiIl’l: T’S’, IHAVE YOlJ HAD ANY Cl-IRNCiE:Ci IN Yi:llJli MEN!!; I’I:i(JAl. t’l-.ti.l i’lDS? HAVE.. YOU HAIt ONY TRi’JlIDLE W:C.fH YCtllFi F:Ylil:Ei’! HAVE: YOU NOT’l:i:Ei:D Yi3tIR SK1.N BEi:NG DFtY Clli i::l:tAI?:il-:‘i’ IS YCIiJII SKIN SOFTER OR F’I’NER; Cl K’ l-l AU I:: Y Cl 1.1 N ( I r ‘I: C:It!:I I l-l A 1: Ii I...0 Ci 5” I-10 YOU THINK YOU SWE:AT MlJRE OR I...I.‘EiCi ‘THAN YOIJ lISLi:I:l ‘TO’-’ HAVE YOU NOTTCI!D ANY CHANOE: 1:N YCIIJR UCll:C:L::‘? I-IAVE YCIU HAtI ANY SWEI.. i...:I:Nii :IN YIIIJR NECki’ I:~OE:!Fi YOUR NECK CIURT’? I-IOW IS YOlJli Hlli:AWINi~‘? FIRE: YOU TAKINi.3 f$NY I::l!iT~Oi:;EN--i:oNTn:I:NING Mli!:I:l’I:CINki:S’? HAVE: YOU F.:UE:l? HAI: ANY ‘rRtl:R TPfliNT Fi’lK THYRl’)I’I:l 1.1I:;FfiSI:i: RE:F’i:lRI;“?
I VI Y R 0 1 D I’ I::. I .I’ E::M S : GENEFrAl... ,?F)I::‘EAR,$NC:li. BL..OL?D F’RESSURE Pl!LSE RATE ‘rII:MF’ERATURE RESF~‘I:RA’rORY RATE:, HANI~CitiAKE SKSN HE AD E.:ypj Mi.3I.l.I t-l N Ei:C:Ii I..YMF:‘t.I NODI:!ii I...lJNi;!:; ANI:! C:Hf:Sl Ii 1:::AR ‘I A El I:1Cl M 11N Glii: N 1:‘I’ A 1..I: 13e NI.:lJRClI... Cl6 1 C : MI:N ‘I Al... ST ATlJ S [IX AM , N If:IJ I7 0 I,..Cl G :r c: : MCI~I’CIR ‘TE:‘; T’1:NI:i NE:: I.1li LI IL 0 1:;:L C : I:lF:F.:I” TE:NI:lClN REFI. I: XE:!:;
500 5 0 :I. 5 () 1,; 50 3 50 4 505 YO& $0:’ YQB
‘II-IYlii:l:I:I:I LAB :I: rEMS: Sli:l:ilJM ‘Tt~IYl3~3XINI.: (‘T’4) .r3 LJPTRIII: ( ‘T3l.J) 24 tK)lJli :I.31 ‘I: I.Ji::TAI
I”,;Efi:
SEl?lJM F; E:IIl.JM !j’li:RUM
T3 (R:l:A) ‘Tti Y RO X I: NE: ~rtvmi:ITtxor~~~N
‘r4
I.3:I NI:l 1: NO (Tt:iH)
[il.. 0 RI 1I. 1 N
( r Pf:; )
I-1‘I A
FIG. 1. The THYROID glossary. The program user may select any item from the glossary by entering the number for the item, and thereby carry out a history and physical examination, and appropriate laboratory studies. Eventually the diagnosis can be entered and proper therapy can be ordered for the “patient.”
36
STEELE,
DAVIS,
AND
FAMIGLIE’I-TI
I 3 :; f I F.‘F’ R IF:s s J,( t N T [::5; I I t~IYF;L.J’THCIF’I~II’N I-~ti:I...lii:A~i:l.Nl; t~lLltrMI.)Ni:. : I lci I ’ : I I I~I!I II , I i’il I:Ct.‘t’E::C rIUk:’ THYtIC)X.INFI: fiA’I:I Cl I’.‘I:\F:.[-’ ‘TtIYF;ClXTNF: :lNI:lt::X r I ,IY li (IX .I:N I:! .I..Fi t:.1.; 1.N 1 3 r N 1, I ‘, X ::;k.IZ(JM I::‘@:1 ::i I:.:I-.:l.J M T 4 U Y (‘: I.11. I.1M N t~‘I~~:t~~L:I~~I1..~:1~~‘1 ):: I::‘LI.!st.l Il.. !i 1 NE :I:!:X:1I I.:: A !:; P ‘r Ii A ‘r 1: 1:)N Gl? A M u rA IN OF NE: t. LII. ItI AI:; IF I. I:<;1 r :I ON CUL. VIJRE IJF i~l:;i:.X:~l.. 1::’ ‘R!jF:’ I I:i’A li: MA~rl!rl:):l Al (.I F’ Fl Fi A ‘I I v 1.: Ii J: 1) I::I$i y N t:.K II I...E If 110 IP !!i Y IPL A::IMA t.lCG ~~l:l:[(J-..F;li~~:~:l:‘l~~)1:\’ R!.;>jAY (I’IIK(‘;NRNl:Y t I !j; I 1
700 701 ‘7 () 2 703 704 705 706 70 7
700 709
GIVE: :L OR 2 MG I-‘RC)F:‘R~NC)L.ClI... .lU NI’JW SS6.J 2 DR(-JF’!j F’i) F.‘HOI:‘YL.Tt~I%OIJll~CI:l.. (F”rlJ) UK ME.IH l~“~A.ZI:ll...ii.. ~‘T~I~:‘Ic~Zl:)l...li-) BEGIN STEROIDS: F:‘l?EDN:l:SClNt: 40 MC;, CI tlAY I..:.-‘I 4 ‘THERAPY (OKAL.) K:IESSICATE:D ‘THYROID ‘T It.IEI?AF’Y BE:G 1. N bN I: U W :tTI-I Nl:)~MAl... 561. 1: NI:. THYRL?IDE:C’rOMY BEGIN AMF’ICI’I..LINV I”liLNK:i:CNU CIJI.. IlJ1:it: lil:‘!iUL I5 ADVISE EXF’I..ORAToRY SLJHGEF?Y OF’ ‘TldE!: ‘rHYI~:cI’l 1: A 11V I S E F’ S Y C H :I A T R :L’C It.1E I...I::’ I-‘ROPANOL.C~L :I:N I:lA~IlL.Ys ClF
T H Y FI’D I D OX ‘1 TEMS: THYROID STORM ~T#XIC DIFFUSE GOITlZFi roxLC Ai:lENOMf? .TOX:I:C MIJI...T:l:NODIJl.. Ali G1.1.l i I::12 904 I...Y PI F:’t ICl C Y T I C ‘I’ Id Y C1:I:1:) 1,.I’ .c $1 ( F:’A ‘T :I:I:: N 7 I-IY F:’CJfl E ‘I’ A 13i.1 IL I 1.: ) 1. Y MF’t.1 CJC Y ‘r I C 1 I.+Y IF?f:) :I:D :I:‘I I:S ( I+ ‘r :I.FI:N ‘I’ El.1 ME ‘I A I:ll:J I., :I:C i 90 5 906 ‘r I~IY F: cl :c D n L. ki Y F:’CI ‘I’ bi Y R (:I :I:D :r.:ii M YO7 P :I:‘I’ LJ I ‘Tn I? Y Cl I? H Y F’ Cl ‘T tj A IL.A M 1 c: It.1Y I:‘ Cl ‘I H Y I I I.1:1 u d. s M 90 8 I-I Y F’ u ‘r ti Y FZC)I 11I s M I::’I:)!ii .r :I :-I :I.:r ‘r cdw:i’ n F:*Y tiYF:‘I:lTt~IYHEJ:C~l:~~~~ IPIIL3’r I’I-IYRCl I:DII.C:TI:)MY 909 910 rHYI:i’Cl113 CARCI’NLJMA 91 1 NON _.’T (:1X .L’ C MU 1...I’ :I:NC) U I I I,. A I: i; I 1I r k::Fc 9 12 iiHnNIJl..oMA’rOU!ii .rblywI L’LI.rr:[:i 9 13 ACLJTE: Tt~YF2O:~I:l:l:T’l:Pi (V:l:I?AI..) 914 ACIJ IEi: THY Ri:) %D :I ‘r .[ !.i ( B A I.: ‘I F.:ti I Al... 1 9 :I.5 rHYfi’O’rOX:J:f::(?li1:S F’wr’r Tlfa 9 16 N 0 N -. ‘T 0 X I C I:1I’ F:’F: LJS E G D :l T E I< 9 1 .I NORMAL. FS’EGNANCY 9 1 IS IPAT [FNT UN 13RAl.. t!:STFr(lGl:N ‘THII-RAt’Y 91Y .T;3 .r I-l Y R Cl ‘1 (3 x :I:r: (1s :I s AF:‘R’rHE:TIC ( ‘MASKEI’LI * ) ‘I t-IYI:li.l’rc)X:IC(J~ 1’S 920 92 1 DEF’RESSILJN 922 ANXIEYTY NEIJROSIS
400 7 0 :I. 902 Y03
FIG.
I-Continued
;
t)lii
CA1 PROGRAM
i
,
,::! j ( ‘;
fg(.I!j
I N.!
‘I’!
II:,
I
!~;WI::A'I' 9
1:
PI I.. 'IICil:
FOR THYROID
137
DISEASES
.{ (I! j I i!l.l MII::W.i:i.!N VI’- .f /.j(.y,J.:i:ii;:Zf< 1+1’1’J{lpj’(l (tWtl,(,,‘) I.. I:iJ iJl:l:.:fL! J fAi~,iR~\ kli\kil:l )I 6111 $!I1i i! I I.::., :i:,,A:..I..:+ 'I I-II:
f:~i.!iltYIli:lii
L:(ll.
1.I
Wi...ii
I'l+II:.l?v
'II~lC)~.iI;~~l~~
ii\!
I !+,:-i'.,
Il,II
r 1ii!:
Pai
1 I~OI.IBI../El:I:i:UiF:NI:
.!
I I AU /II
probability that such responses will be obtained in other thyroid disorders or in disease-free individuals has not been formally determined. The previously published diagnostic indexes were used as a base upon which to make initial estimates of response probability for hypothyroidism and hyperthyroidism in THYROID; probabilities were arbitrarily assigned by the program author for other thyroid diseases. These assignments were independently reviewed by clinicians who were asked to flag response probabilities which appeared to vary by more than 510% YOU
Fi I.
YES, BY MID-AFTERNOON ClLL I W&NT T O I30 16 GO T O RED. NO MORE TIRE11 THAN USUFIL AT THE END OF THE DAY. I CAN’T DO ANYTHING ANYMORE. I FEEI.. TERRTHLY TIRE:Di NO, I FEEL.. OKAY MOST OF THE TINE.
R? I? 3
I.:4
Iix : 900 11x:901 ux : 902 11x : 903 I.1x : 0 0 4 1:1X :VNl EIX:Yod nx: 907 I.IX:YOH 11x ; YO9 Ii X : 9 1 5 11x : 9 I. 9 I) x : 9 L:?0 I.1x : Y 2 I
EXCESSIVELY
** 101 $1 OR WEAKER THAN
AN:
R:l:HO Fti:&lo Hltso FCl :I30 RI. : ‘I, 0 F
rf 3.: ‘70 R:I : 50
TIREn
JR3 : 5 113:60 113: s R3:hO R3:60 R 3 : II 0 Ii.5 : HO Fi . T : 5 11;3: 5 F?:i::1 :i t13 : :‘(I
YOU
USE’D
TO
BE?
R4:20 R4:65 R4:20 R4:20
R4: 5 R 4 : :I 0
11X:s W’J’J1.I NI PI!OIi:.;: Y 0 !fi 7 9 :/ 0 ., ! , ‘I I P ” I :‘! I 9 :, .< Y 0 J ‘l I 9 :, h * 9 I, 7 v 4 1 ! I t Y ::!I’ *
FIG. 3. A question and response set with probabilities assigned based on individual diagnoses. Refer to Fig. 1 for the diagnosis corresponding to each 900 series number. In the data base item shown in this figure, for diagnosis 900 (thyroid storm), it is estimated that response 1 would be given by 80% of patients in storm, and response 2 would be given by 20% of patients in storm. For diagnosis 901 (toxic diffuse goiter without storm) the likelihood of responses 1 and 2 are estimated to be similar, but there is also a slight chance that response 3 could be given instead, Several thyroid diseases are not characterized by complaints of heat intolerance. These are listed at the bottom of this data base item. In these diagnoses, the likelihood of a given response to this question would be the same as the likelihood for a healthy subject. The probabilities of responses for a normal subject are listed in the midportion of the figure (“DX*NL”).
138
STEELE, DAVIS, AND FAMIGLIETTI
from their own climcal experience. Areas of disagreement were then adjudicated by the clinicians and program author. In some cases, probabilities were purposely distorted in order to permit, for teaching purposes, the expression of certain unlikely responses.2 Armed with this information, the computer program selects one of the several responses to a question, in accordance with the probabilities assigned to the responses for the specific diagnosis. This selection process is repeated for every item chosen from the glossary by the user and a patient profile emerges which is unique for each patient simulation of the same diagnosis, and in addition, from diagnosis to diagnosis. Finally, a teaching message has been written for each item in the glossary, and can be obtained for each item at any time during the exercise (Fig. 4).
This formatting of information is compatible with a driver program written for the COMA, HYPERTENSION DIAGNOSIS, and JOINT PAIN programs of the Massachusetts General Hospital Laboratory of Computer Science. An option denoted “HELP” is provided in all these programs which allows the selection of a teaching message on any item found in the glossary (Fig. 4), or a “consultant” who will guide the user stepwise to the correct diagnosis, In addition, the user may ask for normal values for lab tests, or for a review of his data collected thus far (Table 11). TABLE THE “HELP”
II OPTION
The options available to a user when the user types “HELP” allow the user to continue with the exercise when the user might otherwise be unable to complete the case successfully. “HELP” options: (1) Consultant (2) Information (Teaching Messages) (3) Review of all the items (4) Normal values ‘For example: Some impairment in hearing is seen in about 35% of hypothyroid patients. Nonetheless, when the program was run, this symptom did not seem to be- given often enough to be assured that users would learn this point. Accordingly, the frequency of this response to a question about the patient’s hearing was increased to 50%.
CA1 PROGRAM
FOR THYROID
DISEASES
139
Figure 5 shows a typical “patient” generated by THYROID. It begins with an introductory message about the purpose of THYROID, and how to proceed with the exercise. Then, a clinical case presentation is given and the user proceeds to question
i..nR
ANI:I Tfa!:ATi4fi:NT CIF ~rtiY~~:l:xt ui:lixitu:irzx:i + rzvAuIAT:Ii3Nr I3IAGNixIst Yl1IJ MAY ASK t+xsTufzyr I:+IY8IC:AI... 9 OR LAB OlJEEiTTClN!3 ST.MF’I...Y I3Y E:N’rf::R:CNi:: ‘rtit: AF-pru3PliI:ATE: :ITl::M f3xln ‘THE: lxcl!iisAfi-y. ‘rt413l.11314 THIS C:i:IIJLX RI!: HAF’HAZARI) IN CiL:i:OIJE:NCE:, :I: ~~:I~.ILI:I II:N~~I~JRR~:~~~: YO~J .ru ‘rr
’ :LNF’l:)’
FCIR
*
ANIl
Rl.lN
IKtWN
:INf-i:lliMAT:l.i:lN
l...ATEI..Y
ABCIIJT
ANY
ANIl
I:II:tYe
e UL:)I::AI3UI...Al?Y
II () ‘9 -1
HANI:lCiHAl
‘7’14::
:[ r t:: M :I: :
I-1 00
i:rf3
-101
-k:
OPTICIN
WARI:
Al:Z: YCHJ IEXix8SI: 1 1:w3... wl:ai:~m.~ .I ‘r I:::1~ .R: :
I-‘1 0 a
I:li:l Y i:lIJ \JI:.. 6, I::‘E’w
RE:i::L:lME: :;;‘rA .f N:;
IiAND
UE:L..Y .r 1r3f3 i
I:-fxl~.8
N~~RMAI...I...Y
‘T‘:I:RE:I:l
ON
I
CAN’T
9Klli”r OF RFXATH :r :; I:I 1:r:p 1 (::l.lb.;r .
WARM
WII:AKEiX ‘THAN DO ANYTti:I:N13
MCIRE
EI:A!j:II...Y’i’
Yl3l.J USEIl ANYMi:lRf:,
YIZCi I
‘TCI HE?’
f:UI.:N
WAI..li’I:N13
FIG. 5. A typical user-computer interaction. Unknown to the user, all responses are being selected based on the likelihood of the response being given by a patient with hypothyroidism due to prior 13’1 therapy. This diagnosis has been selected at the beginning of the case by the computer, and the “patient” is constructed as the case proceeds. (User entries have been underlined).
140
STEELE,
DAVIS,
AND
FAMIGLIETTI
CA1 [ TE M:u: :
-is:1 3
,-1,::fif ‘, INTFKM:I:
rrwr
F’MI
J r I:M:4b: 3
53 +
Bl:)Wlii:l..
l..l3C:AI...I%EDi
THYROID
St tw
OF
141
DISEASES
F’L:!i)lI
0lJhl..
:I:TY
W L’ I’l-I
AN
SlllJNIlS
NORMAL.
i
NC)
C)f:(iANl:lMI:I:l:;Al
Y C1I.c T I:I’~!1’1I:‘l:iNiI-“1r:
+
I:N~rli:i...i...li:~~r11~1 f:i~~dI’:‘rI:iIM
3:Lh -
NE 1.1t2 i:) IL.(3 6 :I C: : SI:‘I.:E:C:ti F-jI...ClW i :c ‘1 IT:M 4: :
FOR
I.4
AH 1.101-iti N :[ I EMH::
F’OOl?I...Y
PROGRAM
MKNTAI.
mwx
STRTl.!Ei
lii:XAM
~i...E:fxmv~r,
IIIII
I G
-3 :I.7
Nli::lJRi:)l..C)lj:I:I::: Al I.. MIJSCI. EC; +
M Cl T I:) li ‘I’ I.1S ‘I :I N i:; f:iTl:;E:NGTti NC1 ~TI::F::MOR N(:lTE:U t
::iClMI:i:WI-IA’T
!‘I I M’l:N:l:li;t4I:I:I1
‘I fd
.[ 1 I,:M II: : -3 :L 8
:[‘Tt:i”iII::
-500
::;Ii I: II M ‘T t I Y I:2I:1X :I:N Fi: ( ‘I 4 )
2’. ‘I I.!l:;M ‘ill.
‘1‘r k:M41: -:.r0 :I. IF.‘? IJt’ rAI
(‘1’3l.1)
‘I :i ‘X FIG.
5-Continued
and examine the patient. In this example, the computer has randomly preselected diagnosis #908: hypothyroidism post 13rI therapy. Therefore, all responses generated during the exercise are based on the probability of those responses being given by a patient with hypothyroidism after 1311therapy. The use of the HELP option in this case is shown in Fig. 6. At the end of this patient presentation (Fig. 7) the user enters the correct diagnosis, orders the proper therapy, and in the example shown, declines to have the program give its evaluation of the user’s problem solving process. DISCUSSION
CA1 programs are of several types (21): (1) those which allow the user to make entries in his own words (“free text” entries) such as “CASE” from Ohio State University and “Orthopaedic Problems” from the Laboratory of Computer Science, Massachusetts General Hospital; (2) multiple-choice branching programs which cover many topics such as cardiopulmonary resuscitation, and hypertension; (3) complex evaluative and directive programs (12, 23); and (4) random generation programs which produce an inexhaustible supply of unique, synthesized patient profiles. Each of the latter is consistent with a diagnosis preselected by the computer,
STEELE, DAVIS, AND FAMIGLIETTI
142
1lE.M
I.:
t1 I:.:I..I:
iJl::T.r.i.Ji\‘:;: N I.! M H I. t< 3’
”
NI
t I:)12 4: TlzM
V&II Ill;
I )
i.;l.)NSLIi..:i
2)
# :
I:NF II
.I 1 tirv
I I:..w
41
I:i’ ri’ ‘?
vii1
IJI.
!I ) lie
IJN
-50 :I
W0 I.1I. rl Y CJ1.1 i:; I..1%iii !I; .KI:: Y I t.1li:. I::’A r :I:Fi.N i ’ ::i P I:CI:) 1.iI IL:h ri : I : 2) ~IYF’OME‘rAHClI...:I:C 3 1 li:lJMF:‘r~~l~~:tl.. ‘I:(:: 4) XII:1 lNi:li N ,,J J., H
NI.
I ! I .IY r.’ I: I:
‘2 .A
~1’1-11”‘; i::L..A!~;!li
GIN
HE
~LJk’rl-IEX
HI:~i:JlX’Ei:N
I:li.IWN
wi:xJi...u Yi31.1 i:l...fxi!j Off HYF’O’I’I-lhl...AM:l:i:: 3) I:KJ Ni:lT KNi:JW N”,.fF,.,:(‘;’ ‘:, 3;
:i: l:yY ‘r bit:. PA ‘1 1’ li:N .r ’ !:i ~l:Ii:Jul...li ‘M ix:i : I1YF”O’rI~lYl~i:lII:i:~~iM :’ 1 ‘7 t.1Y Ik 0 :I:1) RI.
,rl-i:l:!:;
Bl!
i::l A!:;R
i::AN
I::LJli”THli:R
BFCi:fliti:N
.I ! F :I:‘I’ LJ :L r A IFi Y t I 1 t l J‘I I .IY R CJ%1:1:I I:; M
UOWN
Wi.) 1.11,,rt Y I:)1J I : l...0 !i; 5 1: I:: Y ‘r I.4I:: k-A r I I:: N ‘r ’ s F-I:) L:)13I. I: M ,‘I ::; I i i r,>‘l l~oi.,r:.NJ ” ) 5 I:’ i:IN ‘r AN I:‘:L3LJ Pi I._Y Cl i: i:: I.1R I: N (3 I:1%::i Ei.A !; I): 3 ’ il0 NL’II I\Nl:JW N LJ M B I!: IF‘I’ 1 I~i.IliH
I:
1! !
FIG. 6. The “HELP” option. Here, the user asks for “HELP” and wants a consultant to assist in thinking through the meaning of the data the user has collected in Fig. 5. (User entries have been underlined).
CA1
wt1 fJIKli:. l..I::,nu.l:Nl; I:,’\ ( y /N ) II y
YOII
UE:
l:)liDE:RF.:I:l
I..AR
I’E:GTSi
CCIST
%:I.7
‘m
RE:ux~:w
r10
YC)I.I wcwr
ALL..
I::‘nT,rl:::N’r
YCIIJ YOIJR
GIVEN
I HP:
PROGRAM
FOR
Y W.I.L..I... ‘1’1111 Ill’1
AI'F'RUF'KIATE STEMS
THYROID
143
DISEASES
ANY
l.)l’HIkI.i
bI:-‘t~‘lif~Pli
I:KlF:
RX
W:[SEI..Y
,riiE
fxx:
(Y/N)'?N
FIG. 7. The end of the exercise. The user correctly diagnoses #908, hypothyroidism due to 1311 therapy. He orders L-T4 replacement; and then chooses not to have his performance evaluated by the computer. (User entries have been underlined).
such as in the “COMA” “HYPERTENSION DIAGNOSIS,” and “JOINT PAIN” programs from the Laboratory of Computer Science, Massachusetts General Hospital. Unlike these last three programs, which utilize the same driver program as THYROID, THYROID required a data base which accords unique attention to the sex and age of synthesized patients. For example, the possibility of the use of birth control pills must be limited to synthesized patients who are women of child-bearing age. Further, the possibility of administration of oral estrogens must be limited to synthesized patients who are postmenopausal women (and to the occasional elderly man undergoing treatment for prostatic cancer). Therefore, in order to ensure realistic patient profile generation, we provided a separate set of instructions for patient-response generation regarding the diagnosis which was designated “9 18-Patient on oral estrogen therapy.” Age and sex constraints also had to be provided for the diagnosis of pregnancy in synthesized patients. In addition to teaching the diagnosis of thyroid diseases, THYROID reinforces two desirable clinical skills. These are (1) performance of thorough history and physical examination and (2) concentration on relevant clinical information. One way to encourage thoroughness is to require that a minimum number of items
144
STEELE,
DAVIS,
AND
FAMIGLIETTI
be selected from the history and from the physical examination before permitting the user access to laboratory data. This requirement, however, ignores item-relevance, in that there is no proviso that items selected from the history or physical examination data base be appropriate to the diagnosis which is being presented. In order to encourage the user to select relevant history and physical examination items, the author defined a list of items necessary for each diagnosis. If these specitlc items were used as prerequisites, the computer would demand that an appropriate history be obtained and physical exam be carried out. Unfortunately, even if a user requested all but one or two of the items on the “necessaries” list, the computer would still refuse to allow the user to proceed to the laboratory test component. Therefore, a compromise was made lest the user become frustrated in his attempts to continue with the case solution when already elicited data have firmly suggested a correct diagnosis. The compromise chosen in THYROID was to set the absolute number of required items equal to the total number of history and physical examination items filed in the “necessaries” list, but to insist that only one-half of the items selected must match with items on the “necessaries” list. At the end of a case exercise, the user can request an assessment of his approach to the correct diagnosis. Evaluation of user performance is based on the definition of the necessary-items list mentioned above. In clinical practice there must be some critical mass of data, short of which wrong diagnoses are made far too often, and beyond which one’s diagnostic accuracy is not appreciably improved. Items which were considered relevant and necessary were listed and the user’s selections are compared to this list. The user is informed which of his selections were appropriate and which of the “necessary” items he omitted. While of limited scope, this user evaluation provides an insight into the efficiency of diagnostic process. THYROID represents an intensive instructional effort in a limited but extensively discussed area of endocrinology. Initial student acceptance of computer assisted instructional programs is usually high, but successful implementation of THYROID, and other CA1 programs, depends upon a variety of factors in addition to user acceptance. First, THYROID can be regarded only as a teaching adjunct, not as a replacement for more classical instruction. Acceptability of the program will be improved by providing the user with access to an instructor during problem-solving sessions, in order to obtain information not contained in THYROID. Second, there must be sustained intra-institutional financial support for maintenance of the CA1 program(s). The equipment which is required (computer terminals, telephones, and telephone lines) is expensive, must be adequate in amount, and situated so that access is convenient. Finally, centralized intra-institutional coordination of all CA1 materials and expertise is desirable, including a staff who are knowledgeable in computer terminal operation. Since THYROID can be used with or without extensive HELP, it can olfer practicing physicians an opportunity for continuing medical education as well as serve as an intensive, private tutor for the medical student or physician-in-training.
CA1 PROGRAM
Imposition systematic his overall recognition
FOR THYROID DISEASES
145
of prerequisites in the history and physical examination helps develop a and thorough approach, and the case review offers the user a critique of performance. THYROID can also serve to reinforce skills in problem and clinical problem solving when used in group sessions. ACKNOWLEDGEMENT
The authors gratefully acknowledge the comments of Jack Goldman, M.D. in reviewing the data base; John Naughton, M.D., Dean of the School of Medicine, for his interest and financial support; and Miss Mary Lorenz for preparation of the manuscript. REFERENCES 1. HOFFER, E. P., BARNETT, G. O., FARQLJAR, B. B., AND PRATHER, P. A. Computer-aided instruction in medicine. Ann. Rev. Biophys. Bioeng. 4, 103 (1975). 2. HOFFER, E. P. Experience with the use of computer simulation models in medical education. Comput. Biol. Med. 3,269 (1973). 3. WARNER, H. R., WOOLLEY, F. R., AND KANE, R. L. Computer assisted instruction for teaching clinical decision-making. Comput. Biomed. Res. 7,564 (1974). 4. ATTIA, R. R., MILLER, E. V., AND KITZ, R. J. Teaching effectiveness: Evaluation of computerassisted instruction for cardiopulmonary resuscitation. Anesth. Analg. Cleveland 54,308 (1975). 5. SAJID, A., LIPSON, L. F., AND TELDER, T. V. A simulation laboratory for medical education. J. Med. Educ. 50,970 (1975). 6. BLEICH, H. L. Computer evaluation of acid-base disorders. J. Clin. Invest. 48, 1689 (1969). 7. BRICETTI, A. B. AND BLEICH, H. L. A computer program that evaluates patients with hypercalcemia. J. Clin. Endocrin. Metab. 41,365 (1975). 8. BOUKAERT, A. Computer learning of the differential diagnosis of goiters. Znt. J. Bio-Med. Comput. 6,213 (1975). 9. OVERALL, J. E. AND WILLIAMS, C. M. Conditional probability program for diagnosis of thyroid function. J. Amer. Med. Assoc. 183,307 (1963). 10. HARLESS, W. G., DRENNON, G. G., MARXER, J. J., ROOT, J. A., Wilson, L. L., AND MILLER, G. E., CASE-A natural language computer model. Comput. Biol. Med. 3,227 (1973). 11. HOFFER, E. P., BARNETT, G. O., AND FARQUAR, B. B. Computer simulation model for teaching cardiopulmonary resuscitation. J. Med. Educ. 47,343 (1972). 12. HOFFER, E. P., MATHEWSON, H. O., LOUGHREY, A., AND BARNETT, G. 0. Use of computer-aided instruction in graduate nursing education: A controlled trial. J. Emer. Nurs., p. 27 (March/April 1975). 13. WOOSTER, H. AND LEWIS, J. F. Distribution of computer-assisted instructional materials in biomedicine through the Lister Hill Center Experimental Network. Comput. Biol. Med. 3, 319 (1973). 14. CROOKS, J., MURRAY, K. P. C., and WAYNE, E. J. Statistical methods applied to the clinical diagnosis of thyrotoxicosis. Quart. J. Med. Z&2 11 (1959). 15. GURNEY, C., HALL, R., HARPER, M., OWEN, S. G., ROTH, M., AND SMART, G. Newcastle thyrotoxicosis index. Lancer II, 1275 (1970). 16. WAYNE, E. J. The diagnosis of thyrotoxicosis. Brit. Med. J. 1,411 (1954). 2 7. HARVEY, R. F. Indices of thyroid function in thyrotoxicosis. Lancer II, 230 (1971).
146
STEELE,
DAVIS,
AND
FAMIGLIETTI
18. LAMBERG, B.-A., HEIONEN, V. M., ARO, A., LIEWENDAHL, K., KVIST, G., LAITINEN, O., AND KNEKT, P. Diagnosis of hyperthyroidism. Acta Endocrinol. Suppl. 146 (197b). 19. DAVIS, P. J. AND DAVIS, F. B. Hyperthyroidism in patients over the age of 60 years, Medicine 53, 161 (1974). 20. SPECTOR, D. A., DAVIS. P. J., HELDERMAN, J. H., BELL. B., AND UTIGEK, R. D. Thyroid function and metabolic state in chronic renal failure. Ann. Intern. Med. 85, 724 (1976). 21. BRIGHAM, C. R. AND KAMP, M. The current status of computer-assisted instruction in the health sciences. J. Med. Educ. 49,278 (1974). 22. PENGOV, R. E. The computer as an aid in medicai eduction. In “Proceedings of the EDUCOM Fall Concerence,” Chap. 14, October 1974. 23. JONES, L. A. AND SORLIE, W. E. Increasing medical student performance with an interactive computer-assisted appraisal system. Paper at 13th Annual Conference on Research in Medical Education, Univ. of Illinois, November 1974.
AND
BIOMEDICAL
RESEARCH
l&133-146
(1978)
A Computer-Assisted Instruction (CAI) Program Diseases of the Thyroid Gland (THYROID)
in
A. ARTHUR STEELE,* PAUL J. DAVIS, EDWARD P. HOFFER, AND KATHLEEN T. FAMIGLIETTI Endocrinology Division, Department of Medicine, Medical School of the State University of New York at Buffalo, Buffalo, New York 14222 and the Laboratory of Computer Science, Massachusetts General Hospital, Boston, Massachusetts 01915
Received April 18,1977 A computerized teaching program(THYROID)hasbeenauthored for thyroiddiseases which utilizesa medicaleducationdriver programof the Laboratoryof ComputerScienceat the Massachusetts GeneralHospital.Theoutputof thisteaching programhasfour components: (1) an inexhaustible generated pool of patients’protiles;(2) a glossaryof permissible clinical questions, diagnoses, andtherapies; (3) a teachingmessage for everyglossaryitem;and(4) an evaluationof userperformance. THYROID teaches problemrecognition andmanagement of 20 thyroiddiseases by presenting theuserwith a randomlygenerated “patient.”Theprogramwas constructed by writingquestion andresponse setsrelevantto thehistory,physicalexamination, andlaboratorytestsof patientswithvariousthyroiddisorders. In additionto teachingmessages for everyglossaryitem,the usermay requesta “consultant”who guidesthe userstepwise throughthe differentialdiagnosis to the final correctdiagnosis. THYROID canbe usedfor medicalstudentandhouse-staff educationand for reviewof thyroid diseases by practicing physicians.
Computer technology has greatest impact in medical education when applied to reinforcement of cognitive skills (Z-5). Computer programs have been written to assistclinicians in diagnosisand managementof actual patients with such endocrine and metabolic problems as acid-base disorders (6), hypercalcemia (7), and thyroid diseases(8,9). Instructional programs have been developed for patient case studies (IO), cardiopulmonary resuscitation (II, 12), and over 40 other subjects currently available nationally through a cooperative computer-assisted instruction (CAI) program network (13).’ In this paper we describea program which teachesdiseasesof the thyroid gland by synthesizing patient profiles. The program was remote authored, utilizing a c1inica.l data base developed at the State University of New York at Buffalo, and a driver program was written at the Laboratory of Computer Science, Massachusetts General Hospital. * Present address: 550CabotStreet,Beverly,Mass.01915 LTheHealthEducationNetwork. 133
001~809/78/0112-0133$02.oo/‘o Copyright @ 1978 by Academic Press. Inc. All rights of reproduction in any form reserved. Printed in Great Britain
134
STEELE,
DAVIS,
AND
FAMIGLIETTI
THE PROGRAM
Assembly of a data base for the teaching of thyroid disease was seen as a practical goal because sources of clinical statistics were available to define the incidence of certain symptoms and signs (14-19). In THYROID the data were assembled as items of a glossary (Table I) which included patient history, physical examination, results of laboratory tests, diagnostic possibilities, and therapeutic alternatives (Fig. 1). The format by which the data base was constructed is described below. TABLE THE COMPONENTS
I
OF THE THYROID
GLOSSARY LIST: HISTORY ITEMS PHYSICAL EXAMINATION LABORATORY TESTS THERAPEUTIC OPTIONS DIAGNOSES
GLOSSARY
ITEMS
Question and response sets were written for a total of 75 items of history, physical examination, laboratory testing, and patient management (Fig. 1). Each set consists of an inquiry-to be selected by the user-and several different responses to the inquiry. As an example, heat intolerance is the subject of item # 100. Patients with thyroid diseases who are eumetabolic or hypometabolic would not give a positive response; hence, as shown in Fig. 2, the question and response set includes responses appropriate for patients who are thyrotoxic and those who are euthyroid. The program author listed 20 disorders of the thyroid, assigned them numbers (see Fig. 1,900 series items), and for each of the 20 diagnoses estimated the probability of a given response being given to a particular history or physical examination item. Estimates were based on the published data (14-18) and modified in some instances to emphasize specific principles. The statistics upon which responses were based appear to be valid when applied to various patient populations (19,20). A list of response probabilities is presented in Fig. 3. As mentioned, some thyroid disorders do not alter the metabolic status of the patient (e.g., nontoxic goiter) and in these patients the responses to most questions are those of individuals without thyroid disease. The code numbers of these euthyroid disorders are listed at the bottom of Fig. 3, and their assigned response probabilities are denoted by the entry “*NL” midway through the figure. This procedure was followed for each item of the history and physical examination. Similarly, appropriate sets for all the laboratory tests were constructed. Finally, therapeutic options were written for the various diseases. For each of the 20 thyroid disorders, then, a response probability was assigned for every glossary item. The probability of many signs and symptoms for several thyroid disorders (e.g., hyperthyroidism, hypothyroidism) has been weighted in “indexes” (14-20). The
CA1
‘Tl-IYI~l:I:r
PROGRAM
FOR
THYROID
DISEASES
135
LI Ilk I ‘ILM!:;:
YCI! I IJNi:oMI~~ I’llu RBI I::‘I’ I:lCl ItIC r l/JI;A rt.II”I:) 1:11? it.II:i A I t!:Il ttiJ~l?l!: MAI\I 1l..lnN yi:lli 11:,1”\1 1’) rtt:“i’ ME! YOU I::Xi::ES!ii’i:UI.~I..Y ‘r~l.I::li:I.~ (‘III WV hltl:‘l’i rtC] Y(:](J RE(:;OHE:: $;HC)IIT (:)I::’ Rlil: A-rI..I Mi:lt?l:: I:::rr,!:;‘I I. Y’i’ I:10 E: Ci Y 13IJ li H Ei:A H ‘I’ li: V E!:I< R ti. A I ‘I I:) (.I 1:.n !,i I i 1I I ::;I’; 1:I B I : ,7 .I $: ‘? HOW ARE: YOUR NKRUFS” HAVE YOU l-Inn ANY CHANOI~. 1:N Yi.JlJtG nl:.‘l:‘li:. r ‘I.1 I:.‘* I-4AS Y 0 IJ R Wt. :I:(‘3 H ‘T i:: ti AN 0 E I:( M IJ I::I I IRI:: i:: I!: N I’ I...Y ‘1’ IHAVE!: YOU IHAD ANY i::tiANI;E:: :I:N Yl.lIJK I?~lWIzi:l... HfiIl’l: T’S’, IHAVE YOlJ HAD ANY Cl-IRNCiE:Ci IN Yi:llJli MEN!!; I’I:i(JAl. t’l-.ti.l i’lDS? HAVE.. YOU HAIt ONY TRi’JlIDLE W:C.fH YCtllFi F:Ylil:Ei’! HAVE: YOU NOT’l:i:Ei:D Yi3tIR SK1.N BEi:NG DFtY Clli i::l:tAI?:il-:‘i’ IS YCIiJII SKIN SOFTER OR F’I’NER; Cl K’ l-l AU I:: Y Cl 1.1 N ( I r ‘I: C:It!:I I l-l A 1: Ii I...0 Ci 5” I-10 YOU THINK YOU SWE:AT MlJRE OR I...I.‘EiCi ‘THAN YOIJ lISLi:I:l ‘TO’-’ HAVE YOU NOTTCI!D ANY CHANOE: 1:N YCIIJR UCll:C:L::‘? I-IAVE YCIU HAtI ANY SWEI.. i...:I:Nii :IN YIIIJR NECki’ I:~OE:!Fi YOUR NECK CIURT’? I-IOW IS YOlJli Hlli:AWINi~‘? FIRE: YOU TAKINi.3 f$NY I::l!iT~Oi:;EN--i:oNTn:I:NING Mli!:I:l’I:CINki:S’? HAVE: YOU F.:UE:l? HAI: ANY ‘rRtl:R TPfliNT Fi’lK THYRl’)I’I:l 1.1I:;FfiSI:i: RE:F’i:lRI;“?
I VI Y R 0 1 D I’ I::. I .I’ E::M S : GENEFrAl... ,?F)I::‘EAR,$NC:li. BL..OL?D F’RESSURE Pl!LSE RATE ‘rII:MF’ERATURE RESF~‘I:RA’rORY RATE:, HANI~CitiAKE SKSN HE AD E.:ypj Mi.3I.l.I t-l N Ei:C:Ii I..YMF:‘t.I NODI:!ii I...lJNi;!:; ANI:! C:Hf:Sl Ii 1:::AR ‘I A El I:1Cl M 11N Glii: N 1:‘I’ A 1..I: 13e NI.:lJRClI... Cl6 1 C : MI:N ‘I Al... ST ATlJ S [IX AM , N If:IJ I7 0 I,..Cl G :r c: : MCI~I’CIR ‘TE:‘; T’1:NI:i NE:: I.1li LI IL 0 1:;:L C : I:lF:F.:I” TE:NI:lClN REFI. I: XE:!:;
500 5 0 :I. 5 () 1,; 50 3 50 4 505 YO& $0:’ YQB
‘II-IYlii:l:I:I:I LAB :I: rEMS: Sli:l:ilJM ‘Tt~IYl3~3XINI.: (‘T’4) .r3 LJPTRIII: ( ‘T3l.J) 24 tK)lJli :I.31 ‘I: I.Ji::TAI
I”,;Efi:
SEl?lJM F; E:IIl.JM !j’li:RUM
T3 (R:l:A) ‘Tti Y RO X I: NE: ~rtvmi:ITtxor~~~N
‘r4
I.3:I NI:l 1: NO (Tt:iH)
[il.. 0 RI 1I. 1 N
( r Pf:; )
I-1‘I A
FIG. 1. The THYROID glossary. The program user may select any item from the glossary by entering the number for the item, and thereby carry out a history and physical examination, and appropriate laboratory studies. Eventually the diagnosis can be entered and proper therapy can be ordered for the “patient.”
36
STEELE,
DAVIS,
AND
FAMIGLIE’I-TI
I 3 :; f I F.‘F’ R IF:s s J,( t N T [::5; I I t~IYF;L.J’THCIF’I~II’N I-~ti:I...lii:A~i:l.Nl; t~lLltrMI.)Ni:. : I lci I ’ : I I I~I!I II , I i’il I:Ct.‘t’E::C rIUk:’ THYtIC)X.INFI: fiA’I:I Cl I’.‘I:\F:.[-’ ‘TtIYF;ClXTNF: :lNI:lt::X r I ,IY li (IX .I:N I:! .I..Fi t:.1.; 1.N 1 3 r N 1, I ‘, X ::;k.IZ(JM I::‘@:1 ::i I:.:I-.:l.J M T 4 U Y (‘: I.11. I.1M N t~‘I~~:t~~L:I~~I1..~:1~~‘1 ):: I::‘LI.!st.l Il.. !i 1 NE :I:!:X:1I I.:: A !:; P ‘r Ii A ‘r 1: 1:)N Gl? A M u rA IN OF NE: t. LII. ItI AI:; IF I. I:<;1 r :I ON CUL. VIJRE IJF i~l:;i:.X:~l.. 1::’ ‘R!jF:’ I I:i’A li: MA~rl!rl:):l Al (.I F’ Fl Fi A ‘I I v 1.: Ii J: 1) I::I$i y N t:.K II I...E If 110 IP !!i Y IPL A::IMA t.lCG ~~l:l:[(J-..F;li~~:~:l:‘l~~)1:\’ R!.;>jAY (I’IIK(‘;NRNl:Y t I !j; I 1
700 701 ‘7 () 2 703 704 705 706 70 7
700 709
GIVE: :L OR 2 MG I-‘RC)F:‘R~NC)L.ClI... .lU NI’JW SS6.J 2 DR(-JF’!j F’i) F.‘HOI:‘YL.Tt~I%OIJll~CI:l.. (F”rlJ) UK ME.IH l~“~A.ZI:ll...ii.. ~‘T~I~:‘Ic~Zl:)l...li-) BEGIN STEROIDS: F:‘l?EDN:l:SClNt: 40 MC;, CI tlAY I..:.-‘I 4 ‘THERAPY (OKAL.) K:IESSICATE:D ‘THYROID ‘T It.IEI?AF’Y BE:G 1. N bN I: U W :tTI-I Nl:)~MAl... 561. 1: NI:. THYRL?IDE:C’rOMY BEGIN AMF’ICI’I..LINV I”liLNK:i:CNU CIJI.. IlJ1:it: lil:‘!iUL I5 ADVISE EXF’I..ORAToRY SLJHGEF?Y OF’ ‘TldE!: ‘rHYI~:cI’l 1: A 11V I S E F’ S Y C H :I A T R :L’C It.1E I...I::’ I-‘ROPANOL.C~L :I:N I:lA~IlL.Ys ClF
T H Y FI’D I D OX ‘1 TEMS: THYROID STORM ~T#XIC DIFFUSE GOITlZFi roxLC Ai:lENOMf? .TOX:I:C MIJI...T:l:NODIJl.. Ali G1.1.l i I::12 904 I...Y PI F:’t ICl C Y T I C ‘I’ Id Y C1:I:1:) 1,.I’ .c $1 ( F:’A ‘T :I:I:: N 7 I-IY F:’CJfl E ‘I’ A 13i.1 IL I 1.: ) 1. Y MF’t.1 CJC Y ‘r I C 1 I.+Y IF?f:) :I:D :I:‘I I:S ( I+ ‘r :I.FI:N ‘I’ El.1 ME ‘I A I:ll:J I., :I:C i 90 5 906 ‘r I~IY F: cl :c D n L. ki Y F:’CI ‘I’ bi Y R (:I :I:D :r.:ii M YO7 P :I:‘I’ LJ I ‘Tn I? Y Cl I? H Y F’ Cl ‘T tj A IL.A M 1 c: It.1Y I:‘ Cl ‘I H Y I I I.1:1 u d. s M 90 8 I-I Y F’ u ‘r ti Y FZC)I 11I s M I::’I:)!ii .r :I :-I :I.:r ‘r cdw:i’ n F:*Y tiYF:‘I:lTt~IYHEJ:C~l:~~~~ IPIIL3’r I’I-IYRCl I:DII.C:TI:)MY 909 910 rHYI:i’Cl113 CARCI’NLJMA 91 1 NON _.’T (:1X .L’ C MU 1...I’ :I:NC) U I I I,. A I: i; I 1I r k::Fc 9 12 iiHnNIJl..oMA’rOU!ii .rblywI L’LI.rr:[:i 9 13 ACLJTE: Tt~YF2O:~I:l:l:T’l:Pi (V:l:I?AI..) 914 ACIJ IEi: THY Ri:) %D :I ‘r .[ !.i ( B A I.: ‘I F.:ti I Al... 1 9 :I.5 rHYfi’O’rOX:J:f::(?li1:S F’wr’r Tlfa 9 16 N 0 N -. ‘T 0 X I C I:1I’ F:’F: LJS E G D :l T E I< 9 1 .I NORMAL. FS’EGNANCY 9 1 IS IPAT [FNT UN 13RAl.. t!:STFr(lGl:N ‘THII-RAt’Y 91Y .T;3 .r I-l Y R Cl ‘1 (3 x :I:r: (1s :I s AF:‘R’rHE:TIC ( ‘MASKEI’LI * ) ‘I t-IYI:li.l’rc)X:IC(J~ 1’S 920 92 1 DEF’RESSILJN 922 ANXIEYTY NEIJROSIS
400 7 0 :I. 902 Y03
FIG.
I-Continued
;
t)lii
CA1 PROGRAM
i
,
,::! j ( ‘;
fg(.I!j
I N.!
‘I’!
II:,
I
!~;WI::A'I' 9
1:
PI I.. 'IICil:
FOR THYROID
137
DISEASES
.{ (I! j I i!l.l MII::W.i:i.!N VI’- .f /.j(.y,J.:i:ii;:Zf< 1+1’1’J{lpj’(l (tWtl,(,,‘) I.. I:iJ iJl:l:.:fL! J fAi~,iR~\ kli\kil:l )I 6111 $!I1i i! I I.::., :i:,,A:..I..:+ 'I I-II:
f:~i.!iltYIli:lii
L:(ll.
1.I
Wi...ii
I'l+II:.l?v
'II~lC)~.iI;~~l~~
ii\!
I !+,:-i'.,
Il,II
r 1ii!:
Pai
1 I~OI.IBI../El:I:i:UiF:NI:
.!
I I AU /II
probability that such responses will be obtained in other thyroid disorders or in disease-free individuals has not been formally determined. The previously published diagnostic indexes were used as a base upon which to make initial estimates of response probability for hypothyroidism and hyperthyroidism in THYROID; probabilities were arbitrarily assigned by the program author for other thyroid diseases. These assignments were independently reviewed by clinicians who were asked to flag response probabilities which appeared to vary by more than 510% YOU
Fi I.
YES, BY MID-AFTERNOON ClLL I W&NT T O I30 16 GO T O RED. NO MORE TIRE11 THAN USUFIL AT THE END OF THE DAY. I CAN’T DO ANYTHING ANYMORE. I FEEI.. TERRTHLY TIRE:Di NO, I FEEL.. OKAY MOST OF THE TINE.
R? I? 3
I.:4
Iix : 900 11x:901 ux : 902 11x : 903 I.1x : 0 0 4 1:1X :VNl EIX:Yod nx: 907 I.IX:YOH 11x ; YO9 Ii X : 9 1 5 11x : 9 I. 9 I) x : 9 L:?0 I.1x : Y 2 I
EXCESSIVELY
** 101 $1 OR WEAKER THAN
AN:
R:l:HO Fti:&lo Hltso FCl :I30 RI. : ‘I, 0 F
rf 3.: ‘70 R:I : 50
TIREn
JR3 : 5 113:60 113: s R3:hO R3:60 R 3 : II 0 Ii.5 : HO Fi . T : 5 11;3: 5 F?:i::1 :i t13 : :‘(I
YOU
USE’D
TO
BE?
R4:20 R4:65 R4:20 R4:20
R4: 5 R 4 : :I 0
11X:s W’J’J1.I NI PI!OIi:.;: Y 0 !fi 7 9 :/ 0 ., ! , ‘I I P ” I :‘! I 9 :, .< Y 0 J ‘l I 9 :, h * 9 I, 7 v 4 1 ! I t Y ::!I’ *
FIG. 3. A question and response set with probabilities assigned based on individual diagnoses. Refer to Fig. 1 for the diagnosis corresponding to each 900 series number. In the data base item shown in this figure, for diagnosis 900 (thyroid storm), it is estimated that response 1 would be given by 80% of patients in storm, and response 2 would be given by 20% of patients in storm. For diagnosis 901 (toxic diffuse goiter without storm) the likelihood of responses 1 and 2 are estimated to be similar, but there is also a slight chance that response 3 could be given instead, Several thyroid diseases are not characterized by complaints of heat intolerance. These are listed at the bottom of this data base item. In these diagnoses, the likelihood of a given response to this question would be the same as the likelihood for a healthy subject. The probabilities of responses for a normal subject are listed in the midportion of the figure (“DX*NL”).
138
STEELE, DAVIS, AND FAMIGLIETTI
from their own climcal experience. Areas of disagreement were then adjudicated by the clinicians and program author. In some cases, probabilities were purposely distorted in order to permit, for teaching purposes, the expression of certain unlikely responses.2 Armed with this information, the computer program selects one of the several responses to a question, in accordance with the probabilities assigned to the responses for the specific diagnosis. This selection process is repeated for every item chosen from the glossary by the user and a patient profile emerges which is unique for each patient simulation of the same diagnosis, and in addition, from diagnosis to diagnosis. Finally, a teaching message has been written for each item in the glossary, and can be obtained for each item at any time during the exercise (Fig. 4).
This formatting of information is compatible with a driver program written for the COMA, HYPERTENSION DIAGNOSIS, and JOINT PAIN programs of the Massachusetts General Hospital Laboratory of Computer Science. An option denoted “HELP” is provided in all these programs which allows the selection of a teaching message on any item found in the glossary (Fig. 4), or a “consultant” who will guide the user stepwise to the correct diagnosis, In addition, the user may ask for normal values for lab tests, or for a review of his data collected thus far (Table 11). TABLE THE “HELP”
II OPTION
The options available to a user when the user types “HELP” allow the user to continue with the exercise when the user might otherwise be unable to complete the case successfully. “HELP” options: (1) Consultant (2) Information (Teaching Messages) (3) Review of all the items (4) Normal values ‘For example: Some impairment in hearing is seen in about 35% of hypothyroid patients. Nonetheless, when the program was run, this symptom did not seem to be- given often enough to be assured that users would learn this point. Accordingly, the frequency of this response to a question about the patient’s hearing was increased to 50%.
CA1 PROGRAM
FOR THYROID
DISEASES
139
Figure 5 shows a typical “patient” generated by THYROID. It begins with an introductory message about the purpose of THYROID, and how to proceed with the exercise. Then, a clinical case presentation is given and the user proceeds to question
i..nR
ANI:I Tfa!:ATi4fi:NT CIF ~rtiY~~:l:xt ui:lixitu:irzx:i + rzvAuIAT:Ii3Nr I3IAGNixIst Yl1IJ MAY ASK t+xsTufzyr I:+IY8IC:AI... 9 OR LAB OlJEEiTTClN!3 ST.MF’I...Y I3Y E:N’rf::R:CNi:: ‘rtit: AF-pru3PliI:ATE: :ITl::M f3xln ‘THE: lxcl!iisAfi-y. ‘rt413l.11314 THIS C:i:IIJLX RI!: HAF’HAZARI) IN CiL:i:OIJE:NCE:, :I: ~~:I~.ILI:I II:N~~I~JRR~:~~~: YO~J .ru ‘rr
’ :LNF’l:)’
FCIR
*
ANIl
Rl.lN
IKtWN
:INf-i:lliMAT:l.i:lN
l...ATEI..Y
ABCIIJT
ANY
ANIl
I:II:tYe
e UL:)I::AI3UI...Al?Y
II () ‘9 -1
HANI:lCiHAl
‘7’14::
:[ r t:: M :I: :
I-1 00
i:rf3
-101
-k:
OPTICIN
WARI:
Al:Z: YCHJ IEXix8SI: 1 1:w3... wl:ai:~m.~ .I ‘r I:::1~ .R: :
I-‘1 0 a
I:li:l Y i:lIJ \JI:.. 6, I::‘E’w
RE:i::L:lME: :;;‘rA .f N:;
IiAND
UE:L..Y .r 1r3f3 i
I:-fxl~.8
N~~RMAI...I...Y
‘T‘:I:RE:I:l
ON
I
CAN’T
9Klli”r OF RFXATH :r :; I:I 1:r:p 1 (::l.lb.;r .
WARM
WII:AKEiX ‘THAN DO ANYTti:I:N13
MCIRE
EI:A!j:II...Y’i’
Yl3l.J USEIl ANYMi:lRf:,
YIZCi I
‘TCI HE?’
f:UI.:N
WAI..li’I:N13
FIG. 5. A typical user-computer interaction. Unknown to the user, all responses are being selected based on the likelihood of the response being given by a patient with hypothyroidism due to prior 13’1 therapy. This diagnosis has been selected at the beginning of the case by the computer, and the “patient” is constructed as the case proceeds. (User entries have been underlined).
140
STEELE,
DAVIS,
AND
FAMIGLIETTI
CA1 [ TE M:u: :
-is:1 3
,-1,::fif ‘, INTFKM:I:
rrwr
F’MI
J r I:M:4b: 3
53 +
Bl:)Wlii:l..
l..l3C:AI...I%EDi
THYROID
St tw
OF
141
DISEASES
F’L:!i)lI
0lJhl..
:I:TY
W L’ I’l-I
AN
SlllJNIlS
NORMAL.
i
NC)
C)f:(iANl:lMI:I:l:;Al
Y C1I.c T I:I’~!1’1I:‘l:iNiI-“1r:
+
I:N~rli:i...i...li:~~r11~1 f:i~~dI’:‘rI:iIM
3:Lh -
NE 1.1t2 i:) IL.(3 6 :I C: : SI:‘I.:E:C:ti F-jI...ClW i :c ‘1 IT:M 4: :
FOR
I.4
AH 1.101-iti N :[ I EMH::
F’OOl?I...Y
PROGRAM
MKNTAI.
mwx
STRTl.!Ei
lii:XAM
~i...E:fxmv~r,
IIIII
I G
-3 :I.7
Nli::lJRi:)l..C)lj:I:I::: Al I.. MIJSCI. EC; +
M Cl T I:) li ‘I’ I.1S ‘I :I N i:; f:iTl:;E:NGTti NC1 ~TI::F::MOR N(:lTE:U t
::iClMI:i:WI-IA’T
!‘I I M’l:N:l:li;t4I:I:I1
‘I fd
.[ 1 I,:M II: : -3 :L 8
:[‘Tt:i”iII::
-500
::;Ii I: II M ‘T t I Y I:2I:1X :I:N Fi: ( ‘I 4 )
2’. ‘I I.!l:;M ‘ill.
‘1‘r k:M41: -:.r0 :I. IF.‘? IJt’ rAI
(‘1’3l.1)
‘I :i ‘X FIG.
5-Continued
and examine the patient. In this example, the computer has randomly preselected diagnosis #908: hypothyroidism post 13rI therapy. Therefore, all responses generated during the exercise are based on the probability of those responses being given by a patient with hypothyroidism after 1311therapy. The use of the HELP option in this case is shown in Fig. 6. At the end of this patient presentation (Fig. 7) the user enters the correct diagnosis, orders the proper therapy, and in the example shown, declines to have the program give its evaluation of the user’s problem solving process. DISCUSSION
CA1 programs are of several types (21): (1) those which allow the user to make entries in his own words (“free text” entries) such as “CASE” from Ohio State University and “Orthopaedic Problems” from the Laboratory of Computer Science, Massachusetts General Hospital; (2) multiple-choice branching programs which cover many topics such as cardiopulmonary resuscitation, and hypertension; (3) complex evaluative and directive programs (12, 23); and (4) random generation programs which produce an inexhaustible supply of unique, synthesized patient profiles. Each of the latter is consistent with a diagnosis preselected by the computer,
STEELE, DAVIS, AND FAMIGLIETTI
142
1lE.M
I.:
t1 I:.:I..I:
iJl::T.r.i.Ji\‘:;: N I.! M H I. t< 3’
”
NI
t I:)12 4: TlzM
V&II Ill;
I )
i.;l.)NSLIi..:i
2)
# :
I:NF II
.I 1 tirv
I I:..w
41
I:i’ ri’ ‘?
vii1
IJI.
!I ) lie
IJN
-50 :I
W0 I.1I. rl Y CJ1.1 i:; I..1%iii !I; .KI:: Y I t.1li:. I::’A r :I:Fi.N i ’ ::i P I:CI:) 1.iI IL:h ri : I : 2) ~IYF’OME‘rAHClI...:I:C 3 1 li:lJMF:‘r~~l~~:tl.. ‘I:(:: 4) XII:1 lNi:li N ,,J J., H
NI.
I ! I .IY r.’ I: I:
‘2 .A
~1’1-11”‘; i::L..A!~;!li
GIN
HE
~LJk’rl-IEX
HI:~i:JlX’Ei:N
I:li.IWN
wi:xJi...u Yi31.1 i:l...fxi!j Off HYF’O’I’I-lhl...AM:l:i:: 3) I:KJ Ni:lT KNi:JW N”,.fF,.,:(‘;’ ‘:, 3;
:i: l:yY ‘r bit:. PA ‘1 1’ li:N .r ’ !:i ~l:Ii:Jul...li ‘M ix:i : I1YF”O’rI~lYl~i:lII:i:~~iM :’ 1 ‘7 t.1Y Ik 0 :I:1) RI.
,rl-i:l:!:;
Bl!
i::l A!:;R
i::AN
I::LJli”THli:R
BFCi:fliti:N
.I ! F :I:‘I’ LJ :L r A IFi Y t I 1 t l J‘I I .IY R CJ%1:1:I I:; M
UOWN
Wi.) 1.11,,rt Y I:)1J I : l...0 !i; 5 1: I:: Y ‘r I.4I:: k-A r I I:: N ‘r ’ s F-I:) L:)13I. I: M ,‘I ::; I i i r,>‘l l~oi.,r:.NJ ” ) 5 I:’ i:IN ‘r AN I:‘:L3LJ Pi I._Y Cl i: i:: I.1R I: N (3 I:1%::i Ei.A !; I): 3 ’ il0 NL’II I\Nl:JW N LJ M B I!: IF‘I’ 1 I~i.IliH
I:
1! !
FIG. 6. The “HELP” option. Here, the user asks for “HELP” and wants a consultant to assist in thinking through the meaning of the data the user has collected in Fig. 5. (User entries have been underlined).
CA1
wt1 fJIKli:. l..I::,nu.l:Nl; I:,’\ ( y /N ) II y
YOII
UE:
l:)liDE:RF.:I:l
I..AR
I’E:GTSi
CCIST
%:I.7
‘m
RE:ux~:w
r10
YC)I.I wcwr
ALL..
I::‘nT,rl:::N’r
YCIIJ YOIJR
GIVEN
I HP:
PROGRAM
FOR
Y W.I.L..I... ‘1’1111 Ill’1
AI'F'RUF'KIATE STEMS
THYROID
143
DISEASES
ANY
l.)l’HIkI.i
bI:-‘t~‘lif~Pli
I:KlF:
RX
W:[SEI..Y
,riiE
fxx:
(Y/N)'?N
FIG. 7. The end of the exercise. The user correctly diagnoses #908, hypothyroidism due to 1311 therapy. He orders L-T4 replacement; and then chooses not to have his performance evaluated by the computer. (User entries have been underlined).
such as in the “COMA” “HYPERTENSION DIAGNOSIS,” and “JOINT PAIN” programs from the Laboratory of Computer Science, Massachusetts General Hospital. Unlike these last three programs, which utilize the same driver program as THYROID, THYROID required a data base which accords unique attention to the sex and age of synthesized patients. For example, the possibility of the use of birth control pills must be limited to synthesized patients who are women of child-bearing age. Further, the possibility of administration of oral estrogens must be limited to synthesized patients who are postmenopausal women (and to the occasional elderly man undergoing treatment for prostatic cancer). Therefore, in order to ensure realistic patient profile generation, we provided a separate set of instructions for patient-response generation regarding the diagnosis which was designated “9 18-Patient on oral estrogen therapy.” Age and sex constraints also had to be provided for the diagnosis of pregnancy in synthesized patients. In addition to teaching the diagnosis of thyroid diseases, THYROID reinforces two desirable clinical skills. These are (1) performance of thorough history and physical examination and (2) concentration on relevant clinical information. One way to encourage thoroughness is to require that a minimum number of items
144
STEELE,
DAVIS,
AND
FAMIGLIETTI
be selected from the history and from the physical examination before permitting the user access to laboratory data. This requirement, however, ignores item-relevance, in that there is no proviso that items selected from the history or physical examination data base be appropriate to the diagnosis which is being presented. In order to encourage the user to select relevant history and physical examination items, the author defined a list of items necessary for each diagnosis. If these specitlc items were used as prerequisites, the computer would demand that an appropriate history be obtained and physical exam be carried out. Unfortunately, even if a user requested all but one or two of the items on the “necessaries” list, the computer would still refuse to allow the user to proceed to the laboratory test component. Therefore, a compromise was made lest the user become frustrated in his attempts to continue with the case solution when already elicited data have firmly suggested a correct diagnosis. The compromise chosen in THYROID was to set the absolute number of required items equal to the total number of history and physical examination items filed in the “necessaries” list, but to insist that only one-half of the items selected must match with items on the “necessaries” list. At the end of a case exercise, the user can request an assessment of his approach to the correct diagnosis. Evaluation of user performance is based on the definition of the necessary-items list mentioned above. In clinical practice there must be some critical mass of data, short of which wrong diagnoses are made far too often, and beyond which one’s diagnostic accuracy is not appreciably improved. Items which were considered relevant and necessary were listed and the user’s selections are compared to this list. The user is informed which of his selections were appropriate and which of the “necessary” items he omitted. While of limited scope, this user evaluation provides an insight into the efficiency of diagnostic process. THYROID represents an intensive instructional effort in a limited but extensively discussed area of endocrinology. Initial student acceptance of computer assisted instructional programs is usually high, but successful implementation of THYROID, and other CA1 programs, depends upon a variety of factors in addition to user acceptance. First, THYROID can be regarded only as a teaching adjunct, not as a replacement for more classical instruction. Acceptability of the program will be improved by providing the user with access to an instructor during problem-solving sessions, in order to obtain information not contained in THYROID. Second, there must be sustained intra-institutional financial support for maintenance of the CA1 program(s). The equipment which is required (computer terminals, telephones, and telephone lines) is expensive, must be adequate in amount, and situated so that access is convenient. Finally, centralized intra-institutional coordination of all CA1 materials and expertise is desirable, including a staff who are knowledgeable in computer terminal operation. Since THYROID can be used with or without extensive HELP, it can olfer practicing physicians an opportunity for continuing medical education as well as serve as an intensive, private tutor for the medical student or physician-in-training.
CA1 PROGRAM
Imposition systematic his overall recognition
FOR THYROID DISEASES
145
of prerequisites in the history and physical examination helps develop a and thorough approach, and the case review offers the user a critique of performance. THYROID can also serve to reinforce skills in problem and clinical problem solving when used in group sessions. ACKNOWLEDGEMENT
The authors gratefully acknowledge the comments of Jack Goldman, M.D. in reviewing the data base; John Naughton, M.D., Dean of the School of Medicine, for his interest and financial support; and Miss Mary Lorenz for preparation of the manuscript. REFERENCES 1. HOFFER, E. P., BARNETT, G. O., FARQLJAR, B. B., AND PRATHER, P. A. Computer-aided instruction in medicine. Ann. Rev. Biophys. Bioeng. 4, 103 (1975). 2. HOFFER, E. P. Experience with the use of computer simulation models in medical education. Comput. Biol. Med. 3,269 (1973). 3. WARNER, H. R., WOOLLEY, F. R., AND KANE, R. L. Computer assisted instruction for teaching clinical decision-making. Comput. Biomed. Res. 7,564 (1974). 4. ATTIA, R. R., MILLER, E. V., AND KITZ, R. J. Teaching effectiveness: Evaluation of computerassisted instruction for cardiopulmonary resuscitation. Anesth. Analg. Cleveland 54,308 (1975). 5. SAJID, A., LIPSON, L. F., AND TELDER, T. V. A simulation laboratory for medical education. J. Med. Educ. 50,970 (1975). 6. BLEICH, H. L. Computer evaluation of acid-base disorders. J. Clin. Invest. 48, 1689 (1969). 7. BRICETTI, A. B. AND BLEICH, H. L. A computer program that evaluates patients with hypercalcemia. J. Clin. Endocrin. Metab. 41,365 (1975). 8. BOUKAERT, A. Computer learning of the differential diagnosis of goiters. Znt. J. Bio-Med. Comput. 6,213 (1975). 9. OVERALL, J. E. AND WILLIAMS, C. M. Conditional probability program for diagnosis of thyroid function. J. Amer. Med. Assoc. 183,307 (1963). 10. HARLESS, W. G., DRENNON, G. G., MARXER, J. J., ROOT, J. A., Wilson, L. L., AND MILLER, G. E., CASE-A natural language computer model. Comput. Biol. Med. 3,227 (1973). 11. HOFFER, E. P., BARNETT, G. O., AND FARQUAR, B. B. Computer simulation model for teaching cardiopulmonary resuscitation. J. Med. Educ. 47,343 (1972). 12. HOFFER, E. P., MATHEWSON, H. O., LOUGHREY, A., AND BARNETT, G. 0. Use of computer-aided instruction in graduate nursing education: A controlled trial. J. Emer. Nurs., p. 27 (March/April 1975). 13. WOOSTER, H. AND LEWIS, J. F. Distribution of computer-assisted instructional materials in biomedicine through the Lister Hill Center Experimental Network. Comput. Biol. Med. 3, 319 (1973). 14. CROOKS, J., MURRAY, K. P. C., and WAYNE, E. J. Statistical methods applied to the clinical diagnosis of thyrotoxicosis. Quart. J. Med. Z&2 11 (1959). 15. GURNEY, C., HALL, R., HARPER, M., OWEN, S. G., ROTH, M., AND SMART, G. Newcastle thyrotoxicosis index. Lancer II, 1275 (1970). 16. WAYNE, E. J. The diagnosis of thyrotoxicosis. Brit. Med. J. 1,411 (1954). 2 7. HARVEY, R. F. Indices of thyroid function in thyrotoxicosis. Lancer II, 230 (1971).
146
STEELE,
DAVIS,
AND
FAMIGLIETTI
18. LAMBERG, B.-A., HEIONEN, V. M., ARO, A., LIEWENDAHL, K., KVIST, G., LAITINEN, O., AND KNEKT, P. Diagnosis of hyperthyroidism. Acta Endocrinol. Suppl. 146 (197b). 19. DAVIS, P. J. AND DAVIS, F. B. Hyperthyroidism in patients over the age of 60 years, Medicine 53, 161 (1974). 20. SPECTOR, D. A., DAVIS. P. J., HELDERMAN, J. H., BELL. B., AND UTIGEK, R. D. Thyroid function and metabolic state in chronic renal failure. Ann. Intern. Med. 85, 724 (1976). 21. BRIGHAM, C. R. AND KAMP, M. The current status of computer-assisted instruction in the health sciences. J. Med. Educ. 49,278 (1974). 22. PENGOV, R. E. The computer as an aid in medicai eduction. In “Proceedings of the EDUCOM Fall Concerence,” Chap. 14, October 1974. 23. JONES, L. A. AND SORLIE, W. E. Increasing medical student performance with an interactive computer-assisted appraisal system. Paper at 13th Annual Conference on Research in Medical Education, Univ. of Illinois, November 1974.