Inpatient diagnostic assessments: 3. Causes and effects of diagnostic imprecision

Inpatient diagnostic assessments: 3. Causes and effects of diagnostic imprecision

Psychiatry Research 111 (2002) 191–197 Inpatient diagnostic assessments: 3. Causes and effects of diagnostic imprecision夞 Paul R. Miller* Department ...

96KB Sizes 28 Downloads 73 Views

Psychiatry Research 111 (2002) 191–197

Inpatient diagnostic assessments: 3. Causes and effects of diagnostic imprecision夞 Paul R. Miller* Department of Psychiatry, School of Medical Sciences, University of California, Los Angeles, 2406 Astral Drive, Los Angeles, CA 90046, USA Received 18 October 2001; received in revised form 6 June 2002; accepted 17 June 2002

Abstract Preceding studies found that clinicians using the Traditional Diagnostic Assessment (TDA, the standard of clinical practice) often made imprecise diagnoses, compared with gold standards. Those same studies found excellent diagnostic agreement (kappa)0.75) between Computer Assisted Diagnostic Interview (CADI) and gold standards, thus warranting CADI’s use as the standard for data collection and diagnosis in this study. When TDA and CADI users independently examined 106 inpatient-subjects, TDA users agreed only 45.3% (48y106) with CADI’s primary diagnosis and found only 50.5% as many total diagnoses. This study searched for the causes and effects of those differences. To test the hypothesis that insufficient data collection was the cause, the 106 TDA write-ups were analyzed word-by-word. Only 46.2% (49y106) of the TDA write-ups listed enough symptom criteria (e.g. hallucinations, depression) to meet DSM-IV requirements for diagnosis, a likely cause of TDA’s inaccuracy. TDA write-ups evaluated only 52.9% of the 18 Key Criteria necessary to screen for 10 diagnostic groups, a likely cause of TDA’s incompleteness. TDA’s diagnostic imprecision had effects on (1) length of stay (LOS) for hospitalized patients and (2) associated costs. Patients evaluated with TDA had a mean LOS of 12.5 days versus 7.7 days for CADI patients, a reduction of 4.8 days (12.5–7.7). If CADI replaced TDA, then annual savings of $3 000 000 system-wide could be projected for inpatient care. Remedies for TDA’s diagnostic imprecision are proposed. 䊚 2002 Elsevier Science Ireland Ltd. All rights reserved. Keywords: DSM-IV; Accuracy; Validity; Completeness; Database; Data collection

夞 Similar findings with an overlapping sample were presented at the Annual Meetings, American Psychiatric Association, 6 May, 2000, in Chicago, IL, and 18 May, 2002, in Philadelphia, PA. *Tel.yfax: q1-323-876-2831. E-mail address: [email protected] (P.R. Miller). 0165-1781/02/$ - see front matter 䊚 2002 Elsevier Science Ireland Ltd. All rights reserved. PII: S 0 1 6 5 - 1 7 8 1 Ž 0 2 . 0 0 1 4 7 - 6

192

P.R. Miller / Psychiatry Research 111 (2002) 191–197

1. Introduction 1.1. The Traditional Diagnostic Assessment The Traditional Diagnostic Assessment (TDA) is the standard of practice for making initial psychiatric diagnoses. Textbooks (Othmer and Othmer, 1994; Hales et al., 1995; Sadock and Sadock, 2000) agree generally about the TDA’s processes and formats. Although textbooks accept its diagnostic precision as sufficient for clinical practice, research finds that TDA users often make inaccurate diagnoses (Hill et al., 1996; Lipton and Simon, 1985; McGorry et al., 1995; Miller, 2001; Miller et al., 2001; Mojtabai and Nicholson, 1995; Skodol et al., 1984; van Praag, 1997; Williams et al., 1996). Clinicians continue their universal use of the TDA in spite of contrary evidence, but investigators avoid using it as a solo diagnostic instrument in research (Basco et al., 2000; Shear et al., 2000; Widiger et al., 1994, 1996). 1.2. Computer Assisted Diagnostic Interview The Computer Assisted Diagnostic Interview (CADI) is a structured computer-based interview. Previous studies (Miller et al., 2001; Miller, 2001) found that CADI’s inpatient diagnoses agree with SCID-CV (kappa)0.75) and Consensus Diagnosis (kappa)0.81) and that CADI has excellent interrater reliability (kappa)0.91), thus warranting CADI’s use as the standard for data collection and diagnosis in this study. 1.3. Purpose of this study When TDA users and CADI users independently examined 106 inpatient-subjects, TDA users agreed only 45.3% (48y106) with the CADI diagnoses and named on average only 50.5% (1.53 versus 3.03) as many diagnoses per patient as did CADI. The purpose of this study is to research the causes and effects of those findings. 2. Methods 2.1. Subjects and evaluators Subjects came from two groups. Group A (Miller et al., 2001; Miller, 2001) included 56 inpa-

tient-subjects in an acute psychiatric unit of a publicly funded hospital affiliated with a medical school. Six experienced clinicians (five psychiatrists, one Ph.D. psychologist) made the TDA evaluations. Five research investigators (including the author) made the CADI evaluations blind (data came only from subjects). Group B included 50 inpatient-subjects admitted consecutively to a publicly funded sub-acute psychiatric unit housed in a skilled nursing facility. Of these 50, 41 were referred from hospital inpatient units, 7 from outpatient clinics, and 2 from hospital emergency rooms. The 50 TDA evaluations were made by 34 psychiatrists and one Ph.D. psychologist from 19 facilities (10 hospitals, 7 clinics, 2 emergency rooms) representing three sectors of medical care (7 public, 6 private, 6 medical school). The author made all 50 CADI evaluations blind. Most subjects had extensive psychiatric histories, multiple diagnoses, and dysfunctional adjustments. All subjects heard and read a description of the study and signed an informed consent. 2.2. Hypotheses and data analyses The first hypothesis was that insufficient data collection caused diagnostic inaccuracy and incompleteness. To research the cause for inaccuracy, TDA write-ups were searched word-by-word for symptom criteria (e.g. hallucinations, depression) to find whether enough were listed to meet DSM-IV requirements for the named diagnosis. To research the cause for incompleteness, TDA writeups were searched for Key Criteria. DSM-IV identifies Key Criteria as those criteria that must be assessed as positive to make the linked diagnosis—hence are ‘Key’ (author’s label). Table 1 shows 18 Key Criteria (Column B) for the 10 diagnostic groups (Column A) that covered all subjects in this study. Key Criteria, which are always listed first in DSM algorithms, function as Yes–No decision points: for example, 噛16 Elevatedyexpansive mood and 噛17 Irritability are Key Criteria for mania and must be evaluated. If either is positive (‘Yes’), then mania is possible

P.R. Miller / Psychiatry Research 111 (2002) 191–197

193

Table 1 Number (%) of Key Criteria evaluated by 41 TDA clinicians on 106 subjects A DSM-IV diagnostic groups

B Key Criteria

C % Evaluated

噛1. Attention-consciousness 54% (57y106) 噛2. Memory-orientation 74% (78y106) 噛3. Cognition 34% (36y106) (2) Disorders due to general medical condition 噛4. Evidence from medical history, physical, laboratory, imaging 76% (81y106) (3) Alcohol-induced disorders 噛5. Alcohol use 66% (70y106) (4) Drug-induced disorders 噛6. Drug use 80% (85y106) 噛7. Hallucinations 72% (76y106) (5) Schizophreniaa 噛8. Delusions 89% (94y106) (6) Schizoaffective disordera 噛9. Disorganized speech 78% (83y106) (7) Psychosis NOSa 噛10. Disorganized behavior 48% (51y106) 噛11. Flat affect 62% (66y106) 噛12. Alogia 29% (31y106) 噛13. Avolition 14% (15y106) (8) Depression 噛14. Depression 86% (91y106) 噛15. Loss of pleasure (anhedonia) 8% (8y106) (9) Mania 噛16. Elevatedyexpansive mood 22% (23y106) 噛17. Irritability 12% (13y106) (10) Anxiety (and PTSD) 噛18. Anxiety 48% (51y106)

(1) Cognitive impairment disorders

Number of Key Criteria evaluateds52.9% (1009y1908). Means9.52 (52.9%=18) per patient. a Groups 5–7 (schizophrenia, schizoaffective disorder, psychosis NOS) share Key Criteria 噛7–13.

and must be assessed fully; if both are absent (‘No’), then mania is ruled out. The second hypothesis was that the effects on patients evaluated with TDA, compared with CADI patients, would be (1) longer hospital LOS (Length of Stay) and (2) higher costs of care. Both effects were measured for inpatients from a previous study (Miller, 2001), by examining hospital records for LOS and calculating the projected costs.

3.2. TDA users’ diagnostic completeness DSM-IV requires that Key Criteria must be evaluated to make the linked diagnoses. Table 1 (bottom two lines) shows that TDA users assessed only 52.9% of Key Criteria, permitting only about half the 10 diagnostic groups to be screened. TDA Table 2 Number (%) of TDA users listing enough symptom criteria in write-ups to justify their Axis I primary diagnosis—106 subjects, 41 clinicians

3. Results 3.1. TDA users’ diagnostic accuracy Table 2, bottom line, shows that only 46.2% (49y106) of TDA write-ups assessed enough symptom criteria to meet DSM-IV diagnostic requirements. The consequent diagnoses agreed only 45.3% (48y106) with CADI diagnoses. CADI requires users to collect all data; the program then computes the diagnosis by matching the collected data exactly with DSM-IV algorithms.

A Disorder

B Symptom criteria

(1) Cognitive impairment (2) Medically induced (3) Alcohol-induced (4) Drug-induced (5) Schizophrenia (6) Schizoaffective (7) Psychosis NOS (8) Depression (9) Mania (10) Anxiety

– (0y0) – (0y0) 0% (0y1) 0% (0y1) 65% (13y20) 20% (3y15) 100% (17y17) 32% (11y34) 29% (5y17) 0% (0y1)

Totals46.2% (49y106).

194

P.R. Miller / Psychiatry Research 111 (2002) 191–197

users then found only 50.5% as many total diagnoses as CADI, which directs users to assess 100% of Key Criteria and to assess all linked diagnoses that have positive Key Criteria. 3.3. LOS for patients evaluated with TDA A preceding study (Miller, 2001) found that patients evaluated with TDA had a mean LOS of 12.5 days vs. 7.7 days for CADI patients, a reduction of 4.8 days (12.5–7.7). 3.4. Cost analysis for patients evaluated with TDA The Joint Commission to Accredit Hospital Organizations (JCAHO) requires clinical departments to conduct annual studies to improve patient care. Details of a cost analysis study for JCAHO, supervised by department administrators, are as follows: ● The psychiatric unit has 11 680 (32 beds=365 days) bed–days annually available. ● Annual inpatient admissions can be calculated as a function of the LOS that was found in the preceding study (Miller, 2001): – If mean LOSs12.5 days with TDA, then 11 680y12.5s934 annual admissions. – If mean LOSs7.7 days with CADI, then 11 680y7.7s1517 annual admissions. ● If CADI rather than TDA is used, then 583 (1517y934) more patients can be admitted annually, avoiding referral to private contract hospitals that charge $800yday. ● Annual projected savings resulting from patients being treated locally and not referred to contract hospitalss583 wpatientsx=$800 wcostydayx =7.7 wdays of stayxs$3 591 280. ● The same treatment teams performed both TDA and CADI evaluationsyhospitalizations (Miller, 2001). Thus, the same staff achieved shorter LOS with CADI, demonstrating that 583 more patients could be admitted annually with no increase in costs. 4. Discussion 4.1. Limitations These include regionalism (multi-ethnicity and urbanism of Los Angeles County) and non-repre-

sentativeness of subjects (inpatients with severe chronic disorders). Clinicians represented public, private and medical school sectors about equally. Results are provisional, awaiting other studies for wider validation. 4.2. Issues surrounding TDA’s data collection 4.2.1. Clinicians make the best diagnoses possible, given constraints of money and time Health care financing drives HMOs, insurance companies, and government agencies to restrict time for initial evaluations to a fraction of Klein’s (1995) recommendation for 90–120 min. With such limitations, clinicians often find that ‘clinical problem solving is the process of making adequate decisions with inadequate information’ (Jason, 1978, p. vii). Weed (1966) recognized this reality when he suggested that Problem 噛1 in the Problem Oriented Medical Record always be ‘incomplete database.’ Because psychiatric treatment plans are increasingly linked to diagnoses (Basco et al., 2000; Shear et al., 2000), clinicians need sufficient time to develop precise diagnoses. Legally, ‘‘the Medical Board of California stated that ‘adequate records’«contain, at a minimum, sufficient information to«support the diagnosis«wFailure to do sox constitutes unprofessional conduct’’ (Willick, 2001; my italics). To persuade administrators to make sufficient time available for diagnosis, psychiatrists can show that imprecise diagnoses lead to wrong treatment plans and medical–legal difficulties, while precise diagnoses lead to cost-effective practices and better patient care (see Sections 3.3 and 3.4 above). 4.2.2. Clinicians assess enough criteria to justify their diagnoses but simply neglect or forget to record all of them Even if true, that situation remains problematic: when part of an evaluation is unrecorded, it cannot be proved later that it was done, and that can cause medical–legal problems. Timeliness of recording is another factor: clinicians who do not have or take enough time to complete simultaneous written records may misremember 50–90% of the evaluation 1–8 h later (May and Miller, 1977).

P.R. Miller / Psychiatry Research 111 (2002) 191–197

195

Table 3 Correspondence of diagnoses between TDA and CADI—106 subjects, 41 TDA clinicians, 5 CADI clinicians CADI diagnoses (3–4) Substance induced TDA diagnoses (3–4) Substance induced (5) Schizophrenia (6) Schizoaffective (7) Psychosis NOS (8) Depression (9) Mania (10) Anxiety Total CADI Diagnoses

(5) Schizophrenia

(6) Schizoaffective

(7) Psychosis NOS

(8) Depression

7 5 4

11 10 11 8 4

0

1 18 1

1 8 11

16

44

0

22

21

1

(10) Anxiety

1

1 2

(9) Mania

2

1 1

Total TDA diagnoses

2 20 15 17 34 17 1 106

Boldface numbers indicate that TDA and CADI agreed: Substance induced 1 time, Schizophrenia 7 times, etc. Overall, TDA and CADI diagnoses agreed 45.3% (48y106). Italics show the totals for CADI (bottom line) and TDA (far right column). Lines and columns are numbered to be consistent with the numbers assigned diagnoses in Tables 1 and 2. Lines show TDA diagnoses. Line 5 shows that the TDA diagnosis of schizophrenia corresponded with CADI diagnoses of schizophrenia 7 times, schizoaffective 11 times, and depression 2 times, a total of 20. Columns show CADI diagnoses. Column 5 shows that the CADI diagnosis of schizophrenia corresponded with TDA diagnoses of schizophrenia 7 times, schizoaffective 5 times, and psychosis NOS 4 times, a total of 16.

4.2.3. Evaluating only a few criteria can achieve diagnostic accuracy, especially when aided by ‘clinical judgment’ This has been suggested as possible for DSMIII (Widiger et al., 1984; Fenton and McGlashan, 1989), but it has not been confirmed for DSM-IV. Clinical judgment is an admixture of knowledge, experience, and cognitive leaps (intuition, insight). DSM-IV (p. xxiii) states, ‘clinical judgment may justify giving a certain diagnosis to an individual even though the clinical presentation falls just short of meeting the full criteria.’ However, many TDA write-ups fell far short, not ‘just short.’ Lipton and Simon (1985, p. 371) stated, ‘the common practice of equating one or two basic symptoms with schizophrenia should be deplored.’ 4.2.4. Completing the facility’s TDA write-up format completes the diagnostic task Analysis of the 20 (8 public, 6 private, 6 medical school) TDA write-up formats used in this study found that on average they solicited only 33% (24y72; ranges8–41) of DSM symptom criteria and 61% (11y18; ranges6–17) of Key Criteria. Thus, incomplete data collection was nearly inevitable.

4.2.5. The diagnostic process can overload cognitive capacities, requiring adaptive responses Clinicians in this study needed to assess 72 symptom criteria and then compare them with many diagnostic algorithms. McDonald (1976) found that the ‘amount of data presented to the physician«is more than he can handle without error’ (p. 1354). Most persons cannot process more than a half dozen associations simultaneously (Hunt, 1977). Some clinicians avoid overload by stopping data collection early. Other clinicians use cognitive styles described by Elstein et al. (1978) that (1) focus on the first positive finding in the interview, (2) develop a quick related diagnosis, (3) seek selective confirmation for that diagnosis, (4) ignore contrary findings, and (5) ignore possible alternative or additional diagnoses. Such practices undermine empirical thinking and yield quick but imprecise diagnoses that superficially may appear correct. 4.3. Patterns of data deficiencies and specific diagnoses Researchers (Skodol et al., 1984; Robinson et al., 1985; Garb, 1998; Greist, 1998) have investi-

196

P.R. Miller / Psychiatry Research 111 (2002) 191–197

gated this general phenomenon: ‘documentation of DSM-III criteria for assigned chart diagnoses was not present in 80% of the 131 charts reviewed’ (Lipton and Simon, 1985, p. 370). This study found specific relationships. 4.3.1. Too few symptom criteria are evaluated Clinicians can find one or two psychotic symptoms, stop searching, and then diagnose Psychosis NOS, because ‘there is inadequate information to make a specific diagnosis’ (DSM-IV, p. 315). TDA users diagnosed Psychosis NOS 17 times (Table 3, line 7), while CADI users had to collect complete data, always made specific diagnoses, and never chose Psychosis NOS (Table 3, Column 7). 4.3.2. Too few Key Criteria are evaluated The diagnosis of Schizoaffective Disorder requires the user to examine 11 Key Criteria—7 for schizophrenia, 2 for depression, 2 for mania. TDA users diagnosed Schizoaffective Disorder only 34% (15y44) as frequently as CADI users, because they often failed to examine all 11. 4.3.3. One specific Key Criterion is omitted TDA users evaluated Key Criterion 噛17 Irritability only 12% of the time (Table 1). Therefore, they could not discover patients with ‘dysphoric’ Mania and missed 10 of the 21 manias that CADI found (Table 3, Column 9). 4.3.4. Several kinds of omission occur TDA users diagnosed Depression 34 times (Table 3, Line 8). They differed with CADI 16 times: (1) by not evaluating Key Criterion 噛17, they missed 8 dysphoric manias (line 8, column 9), and instead diagnosed depression; (2) by not evaluating all 11 necessary Key Criteria, they missed eight schizoaffective disorders (line 8, column 6), and instead diagnosed depression. 4.4. Explanation for LOS The preceding study (Miller, 2001) suggested that ‘early and precise diagnosis™early and precise«treatment wplansx™more rapid recovery™ shorter LOS.’

4.5. Remedies If it is accepted that improving data collection can improve diagnostic precision, then possible remedies should be considered. (1) Patients answer questionnaires about psychiatric symptoms with paper-and-pencil or the computer, and then clinicians use those data for screening and evaluation. Questionnaires are widely used in clinical medicine and can also work with psychiatric patients: many of these inpatientsubjects completed a Beck Depression Inventory on admission. If a psychiatric patient is too disturbed to complete a questionnaire, then a staff member can read the questions to the patient and record the answers. (2) Revise TDA formats so that they (1) list all symptom criteria in DSM algorithms and (2) list all requirements for diagnoses. The clinician can use these as checklists and guidelines. (3) Use a computerized system like CADI. A previous study (Miller, 2001) found that 22 clinicians (8 psychiatrists, 2 Ph.D. psychologists, and 12 second–fifth year resident psychiatrists) used CADI with 100 patients and had no major difficulties after minimal training. Such changes can enlarge data collection, format data for ready clinical review and research, increase diagnostic precision, foster new cognitive styles for clinicians, ‘operationalize clinical practice’ (Glazer, 1998), and improve patient care. References Basco, M.R., Bostic, J.Q., Davies, D., Rush, A.J., Witte, B., Hendrickse, W., Barnett, V., 2000. Methods to improve diagnostic accuracy in a community mental health setting. American Journal of Psychiatry 157, 1599–1605. Elstein, A.S., Shulman, L.S., Sprafka, S.A., 1978. Medical Problem Solving. An Analysis of Clinical Reasoning. Harvard University Press, Cambridge, MA. Fenton, W.S., McGlashan, T.H., 1989. Diagnostic efficiency of DSM-III schizophrenia. Journal of Nervous and Mental Disease 177, 690–694. Garb, H.N., 1998. Studying the Clinician: Judgment Research and Psychological Assessment. American Psychological Association, Washington, DC. Glazer, W.M., 1998. Defining best practices: a prescription for greater autonomy. Psychiatric Services 49, 1013–1016.

P.R. Miller / Psychiatry Research 111 (2002) 191–197 Greist, J.H., 1998. Clinical computing. The computer as clinician assistant: assessment made simple. Psychiatric Services 49, 467–472. Hales, R.E., Yudofsky, S.C., Talbott, J.A. (Eds.), 1995. Textbook of Psychiatry. Second ed. American Psychiatric Press, Washington, DC. Hill, C., Keks, N., Roberts, S., Opeskin, K., Dean, B., MacKinnon, A., Copolov, D., 1996. Problem of diagnosis in post-mortem brain studies of schizophrenia. American Journal of Psychiatry 153, 533–537. Hunt, M., 1977. The Universe Within. A New Science Explores the Human Mind. Simon and Schuster, New York. Jason, H., 1978. Foreword. In: Elstein, A.S., Shulman, L.S., Sprafka, S.A. (Eds.), Medical Problem Solving. An Analysis of Clinical Reasoning. Harvard University Press, Cambridge, MA. Klein, D.F., 1995. Psychopharmacological practice and health reform. Psychiatric Annals 25, 79–83. Lipton, A.A., Simon, F.S., 1985. Psychiatric diagnosis in a state hospital: Manhattan State revisited. Hospital and Community Psychiatry 36, 368–373. McDonald, C.J., 1976. Protocol-based computer reminders, the quality of care and the non-perfectibility of man. New England Journal of Medicine 295, 1351–1355. McGorry, P.D., Mihalopoulos, C., Henry, L., Dakis, J., Jackson, H.J., Flaum, M., Harrigan, S., McKenzie, D., Kulkarni, H., Karoly, R., 1995. Spurious precision: procedural validity of diagnostic assessment in psychotic disorders. American Journal of Psychiatry 152, 220–223. May, J., Miller, P.R., 1977. Note-taking and information recall. Journal of American Medical Education 52, 524–525. Miller, P.R., 2001. Inpatient diagnostic assessments: 2. Interrater reliability and outcomes of structured versus unstructured interviews. Psychiatry Research 105, 265–273. Miller, P.R., Dasher, R., Collins, R., Griffiths, P., Brown, F., 2001. Inpatient diagnostic assessments: 1. Accuracy of structured versus unstructured interviews. Psychiatry Research 105, 265–272. Mojtabai, R., Nicholson, R.A., 1995. Interrater reliability of ratings of delusion and bizarre delusions. American Journal of Psychiatry 152, 1804–1806.

197

Othmer, E., Othmer, S.C., 1994. The Clinical Interview Using DSM-IV. Vol I: Fundamentals. American Psychiatric Press, Washington, DC. Robinson, E.P., Asnis, G.M., Karkavy Friedman, J.M., 1985. Knowledge of the criteria for major depression: a survey of mental health professionals. Journal of Nervous and Mental Disease 76, 480–484. Sadock, B.J., Sadock, B.J., 2000. Kaplan and Sadock’s Comprehensive Textbook of Psychiatry. Lippincott Williams & Wilkins, Philadelphia. Shear, M.K., Breeno, C., Kang, J., Ludewig, D., Frank, E., Swartz, H., Hanekamp, M., 2000. Diagnosis of nonpsychotic patients in community clinics. American Journal of Psychiatry 157, 581–587. Skodol, A.E., Williams, J.B.W., Spitzer, R.L., Gibbon, M., Kass, F., 1984. Identifying common errors in the use of DSM-III through supervision. Hospital and Community Psychiatry 35, 251–255. van Praag, H.M., 1997. The future of biological psychiatry. CNS Spectrums. International Journal of Neuropsychiatric Medicine 2, 18–25. Weed, L., 1966. The Problem Oriented Medical Record. Case Western University Press, Cleveland. Widiger, T.A., Hunt, S.W., Frances, A., Clarkin, J.F., Gilmore, M., 1984. Diagnostic efficiency and DSM-III. Archives of General Psychiatry 41, 1005–1012. Widiger, T.A., Frances, A.J., Pincus, H.A., First, M.B., Ross, R., Davis, W. (Eds.), 1994. DSM-IV Sourcebook, vol. 1. American Psychiatric Association, Washington, DC. Widiger, T.A., Frances, A.J., Pincus, H.A., Ross, R., First, M.B., Davis, W.W., 1996. DSM-IV Sourcebook, vol. 2. American Psychiatric Association, Washington, DC. OPCRIT Reliability Research Group, Williams, J., Farmer, A.E., Ackenheil, M., Kaufmann, C.A., McGuffin, P., 1996. A multicentre inter-rater reliability study using the OPCRIT computerized diagnostic system. Psychological Medicine 26, 775–786. Willick, D.H., 2001. The obligation to keep medical records. California Psychiatrist 17, 3–12.