- Email: [email protected]
Administrative data used to identify patients with irritable bowel syndrome
Journal of Clinical Epidemiology 61 (2008) 617e621
Administrative data used to identify patients with irritable bowel syndrome Sarah L. Goffa, Andrew Feldb, Susan E. Andradea, Lisa Mahoneyb, Sarah J. Beatonc, Denise M. Boudreaub,d, Robert L. Davise, Michael Goodmanf, Cynthia L. Hartsfieldg,h, Richard Platti,j, Douglas Robline, David Smithk, Marianne Ulcickas Yoodl,m, Katherine Dodda, Jerry H. Gurwitza,* a
Meyers Primary Care Institute, University of Massachusetts Medical School, Fallon Clinic Foundation and Fallon Community Health Plan, 630 Plantation Street, Worcester, MA 01605, USA b Group Health, Center for Health Studies, Seattle, WA, USA c Lovelace Clinic Foundation, Albuquerque, NM, USA d University of Washington, Seattle, WA, USA e Kaiser Permanente Georgia, Center for Health Research/Southeast, Atlanta, GA, USA f HealthPartners Research Foundation, Minneapolis, MN, USA g Kaiser Permanente Colorado, Denver, CO, USA h University of Colorado at Denver and Health Sciences Center School of Pharmacy, Denver, CO, USA i Harvard Pilgrim Health Care, Boston, MA, USA j Harvard Medical School, Boston, MA, USA k Kaiser Permanente Northwest, Center for Health Research, Portland, OR, USA l Josephine Ford Cancer Center, Henry Ford Health System, Detroit, MI, USA m Epidemiology and Public Health, Yale University School of Medicine, New Haven, CI, USA Accepted 22 July 2007
Abstract Objective: To assess the usefulness of health plan administrative data for identifying patients with irritable bowel syndrome (IBS). Study Design and Setting: In this retrospective study of 442 medical records of patients in nine U.S. health plans, five sets of criteria that used administrative data were used to identify potential IBS patients. Physician reviewers provided an assessment of the likelihood of the diagnosis of IBS being present. IBS was considered to be present if the physician reviewer categorized the case as definite, probable, or possible based on medical record review. Analyses were also performed with cases categorized as possible placed in an ‘‘IBS not present’’ category. Results: The positive predictive value (PPV) for the five sets of criteria ranged from 63% to 83% with the highest PPV found with one of the most restrictive criteria. When cases characterized as possible were included in the ‘‘IBS not present’’ category, the PPV for each of the five sets of criteria decreased substantially, ranging from 33% to 63%. Conclusion: The PPV of different criteria used to identify patients with IBS from administrative data varies substantially based on the criteria that are used. Use of criteria with a higher PPV may come at the expense of generalizability. Ó 2008 Elsevier Inc. All rights reserved. Keywords: Irritable bowel syndrome; Health services research; Administrative databases; Diagnosis validation studies; pharmacoepidemiology; health maintenance organizations
1. Introduction Health planeautomated databases are frequently used to conduct epidemiologic studies, for economic analyses, to identify potential subjects for recruitment into clinical
* Corresponding author. Tel.: 508-791-7392; fax: 508-595-2200. E-mail address: [email protected] or jerry.gurwitz@ umassmed.edu (J.H. Gurwitz). 0895-4356/08/$ e see front matter Ó 2008 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2007.07.013
trials, and for other research purposes [1]. However, the validity of diagnoses derived from these data sources is often questioned. The results of validation studies can help gauge the usefulness of these data for research purposes. Irritable bowel syndrome (IBS) affects an estimated 10e15% of adults in the United States [2,3]. The subjective nature of the IBS diagnosis presents special challenges for both clinicians and researchers. Diagnostic criteria, such as the Rome II criteria (Appendix), have been used for
618
S.L. Goff et al. / Journal of Clinical Epidemiology 61 (2008) 617e621
What is New? Key findings 1. The positive predictive values (PPVs) of different criteria used to identify patients with irritable bowel syndrome (IBS) from administrative data vary substantially based on the components of the criteria that are used. 2. Use of criteria with a higher PPV will likely affect the generalizability of the patient population that would be identified. New addition Strict adherence to Rome II criteria for determining the presence of IBS is probably not practical for identifying patients with IBS based on information contained solely in medical records. Implications The use of administrative data alone may be unacceptable for epidemiologic and health services research studies where further validation of the IBS diagnosis may not be possible.
research purposes [4,5], and have been promoted for clinical use, but their role has not been adequately defined [6], and clinicians may not use or even be aware of these criteria when diagnosing and treating patients. In this study, we used information derived from medical record review to evaluate the positive predictive values (PPVs) of five different sets of criteria based solely on administrative data to identify patients with IBS. These criteria were developed using diagnostic and pharmacyadministrative data available from nine large health plans.
pseudo-membranous colitis, iron deficiency anemia, rectal bleeding, colon cancer, or weight loss. A multidisciplinary subgroup of the authors, with expertise in gastroenterology (A.F.), internal medicine (J.H.G.), and pharmacoepidemiology and use of administrative databases (S.E.A. and R.L.D.), convened to develop five sets of criteria based on IBS-related diagnoses and selected medications related to the treatment of IBS: (1) at least one diagnosis of IBS (ICD-9 564.1) identified from outpatient or inpatient derived administrative data; (2) at least one diagnosis of IBS and at least one dispensing of a laxative available by prescription (lactulose, PEG-electrolyte, and sorbitoldalone or in combination), antidiarrheals (diphenoxylate, kaolin, and pectindalone or in combination), antispasmodics (belladonna, scopolamine hydrobromide, opium, paregoric, dicyclomine hydrochloride, and hyoscamine sulfatedalone or in combination), 5-HT3 receptor antagonist (alosetron), or a 5-HT4 receptor agonist (tegaserod); (3) at least two IBS diagnoses occurring at least 6 months apart identified from outpatient or inpatient data; (4) at least one diagnosis of IBS identified from outpatient or inpatient data, at least one diagnosis of abdominal pain (ICD-9 789.0), and one other gastrointestinal symptom (either diarrhea [ICD-9 564.5, 787.91] or constipation [ICD-9 564.0]); and (5) at least one diagnosis of IBS and one diagnosis for abdominal pain with at least one other gastrointestinal symptom and at least one dispensing of any medication listed in criterion (2) above (Table 1). Each of the nine health plan sites selected a random sample of up to 25 patients for each of the five sets of criteria, with up to 10 samples of 25 obtained for medical record abstraction. If medical records were unavailable, additional patients were randomly chosen from the list of 25 to ultimately obtain the target of 10 medical records for each of the five sets of criteria for review at each participating health plan site, for a total of 450 medical records. Criteria were not mutually exclusive, meaning one patient could meet any or all of the five sets of criteria; however, in no instance was a patient included for medical record abstraction more than once.
2. Methods We conducted a retrospective study among patients enrolled in nine geographically dispersed health plans participating in the HMO Research Network Center for Education and Research on Therapeutics [7]. Institutional Review Board approval for this research project was obtained at each site. Patients aged 18 years or above with an International Classification of Diseases, 9th edition (ICD-9-CM) code for IBS (564.1) existing within the administrative databases of the health plans, relating to both inpatient and outpatient services between May 1, 2002 and September 15, 2002, were identified. Patients were excluded if they also had an ICD-9-CM code for a disease with symptoms that might overlap with IBS or which might suggest a different diagnosis, including Crohn’s disease, ulcerative colitis, ischemic colitis, microscopic or collagenous colitis,
Table 1 Chart review selection criteria for identification of potential cases of IBS Criterion 1: At least one diagnosis of IBS (ICD-9 564.1) identified from outpatient or hospital data. Criterion 2: At least one diagnosis of IBS and at least one prescription for an antispasmodic, laxative, antidiarrheal, 5HT3 receptor agonist, or 5-HT4 receptor antagonist. Criterion 3: At least two or more diagnoses of IBS occurring at least 6 months apart identified from outpatient or hospital data. Criterion 4: At least one diagnosis of IBS identified from outpatient or hospital data and at least one diagnosis of abdominal pain (ICD-9 789.0) and one other gastrointestinal symptom (either diarrhea [ICD-9 564.5, 787.91] or constipation [ICD-9 564.0]). Criterion 5: At least one diagnosis of IBS and one diagnosis for abdominal pain with at least one other gastrointestinal symptom and at least one prescription for any drug listed in criterion 4 above. Abbreviation: IBS, irritable bowel syndrome.
S.L. Goff et al. / Journal of Clinical Epidemiology 61 (2008) 617e621
Trained abstractors provided a summary of medical record information pertinent to the diagnosis of IBS to the physician reviewers. This information included all office visit notes from primary care physicians and relevant consultation reports by specialists as well as relevant diagnostic studies and tests including colonoscopy, sigmoidoscopy, abdominal x-rays, upper gastrointestinal series, computed tomography scans, ultrasounds, stool cultures, and relevant laboratory studies. The medical record abstraction covered a period from 6 months before the appearance of an IBS diagnosis associated with a clinical encounter to 1 year following that date. The time period of 18 months was chosen because IBS is often a diagnosis of exclusion requiring sufficient time for symptoms to be fully evaluated. This information was then reviewed by one of three physicians, including a gastroenterologist (A.F.) and two general internists (J.H.G. and S.L.G.). When assessing clinical information, each physician noted whether the case met Rome II criteria and also characterized the likelihood of the IBS diagnosis being present according to the following categories: unlikely, possible, probable, definite, or unable to assess. Categorization was based on clinical judgment, taking into account all information available in the medical record abstraction. Using the gastroenterologist’s (A.F.) assessment as the gold standard, the internists (J.H.G. and S.L.G.) underwent a training session before formally beginning to assess the medical record abstractions. During these training sessions, all three reviewers assessed 12 of the same medical record abstractions and when either of the internists’ (J.H.G. or S.L.G.) assessments differed from the gastroenterologist’s (A.F.) assessment, the case was discussed to maximize the consistency of physician assessments. IBS was considered to be present if the physician reviewer categorized the case as definite, probable, or possible. IBS was considered not to be present if the physician reviewer categorized the case as unlikely or unable to assess. Analyses were also performed with cases categorized as possible placed in the ‘‘IBS not present’’ category to assess the impact on the estimates of PPV. To assess inter-rater reliability between the general internists and the ‘‘gold standard’’ gastroenterologist reviewer, 10% (45) of the medical record abstractions reviewed by the gastroenterologist (A.F.) were also reviewed by each of the general internists (J.H.G. and S.L.G.). Reviewers were considered to be in agreement if they had rated the case unlikely or unable to assess (IBS not present) or if they rated the case possible, probable, or definite (IBS present). The kappa values for inter-rater reliability were calculated for each of the internists (J.H.G. and S.G.) compared with the gastroenterologist (A.F.) and were 0.61 (J.H.G.) and 0.80 (S.G.). A kappa score between 0.6 and 0.8 reflects ‘‘substantial’’ agreement and a kappa score between 0.8 and 1.0 is considered ‘‘almost perfect’’ [8]. The PPV was calculated for each of the five sets of criteria that were used to identify patients with IBS from the
619
nine participating health plans. Information was also collected regarding the presence of clinical information pertaining to the Rome II criteria in the medical record.
3. Results A total of 7,000 patients meeting one or more criteria were identified. More than half of these patients (56%) met only criterion 1 (one diagnosis of IBS), 2,338 (34%) met two criteria, and 718 patients (10%) met three or more criteria (Table 2). After sampling for each of the sets of criteria as described in Section 2, 453 medical records were abstracted and reviewed. Eleven medical record abstractions were removed from the analysis because the patient was less than 18 years old, leaving 442 medical record abstractions available for inclusion in the study. The mean age of these patients was 52 (range 18e92) years and 348 (79%) were female. Overall, the PPV for the 442 cases was 73%. The PPV for the five sets of criteria ranged from 63% to 83% (Table 3). The PPVs were higher for the criteria that included diagnoses in the administrative data supporting IBS (Table 2). When cases characterized as possible were included in the ‘‘IBS not present’’ category, the PPV for each of the five sets of criteria decreased substantially, ranging from 33% to 63%. Clinical information relating to the Rome II criteria was not consistently found in the medical record. Of the 321 cases determined to have IBS present based on review of the abstracted medical records by physician reviewers, 179 (56%) had no information on two or more of the three primary Rome II criteria used to establish the diagnosis of IBS. Of these 179, 8 had been categorized as definite, 76 as probable, and 95 as possible.
4. Discussion Our findings suggest that the utility of the five sets of criteria we used to identify patients with IBS varied substantially according to which criteria were used. When considering the implications of these findings, it is important to consider the purpose for which the information is being Table 2 Number of criteria met by potential study subjects Number of criteria met One criterion met Two criteria met
Criteria
Total
Criterion 1 3,944 (56%) 3,944 (56%) Criteria 1 & 2 1,750 (25%) 2,338 (33%) Criteria 1 & 3 330 (5%) Criteria 1 & 4 1,750 (4%) Three criteria met Criteria 1, 2, & 3 212 (3%) 292 (4%) Criteria 1, 2, & 5 34 (0.5%) Criteria 1, 3, & 4 46 (1%) Four criteria met Criteria 1, 2, 3, & 5 12 (0.2%) 360 (5%) Criteria 1, 2, 4, & 5 348 (5%) Five criteria met Criteria 1, 2, 3, 4, & 5 66 (1%) 66 (1%)
620
S.L. Goff et al. / Journal of Clinical Epidemiology 61 (2008) 617e621
Table 3 Validation of criteria based on diagnoses and pharmaceutical prescription data located in health plan administrative databases for identification of cases of IBS Patients meeting criterion (%)a
Criterion Criterion Criterion Criterion Criterion Criterion
1: 2: 3: 4: 5:
IBS IBS prescription IBS 2, 6 months apart IBS, abdominal pain, diarrhea, and constipation IBS, abdominal pain, diarrhea, and constipation prescription
Total
7,000 2,422 594 790 460
(100%) (35%) (8%) (11%) (7%)
7,000
Number of charts reviewed
Number of cases confirmed
92 88 87 87 88
58 59 65 72 67
442
321
PPV (95% CI) 63% 67% 75% 83% 76%
(53, (57, (66, (75, (67,
73) 77) 84) 91) 85)
Abbreviations: IBS, irritable bowel syndrome; PPV, positive predictive value. a Percentages do not add to 100% because patients could meet more than one criterion.
used. For example, a PPV of 63%, as found with a diagnosis of IBS alone, may be sufficient to identify patients for recruitment into clinical trials, where additional information that further supports the IBS diagnosis can be directly ascertained from the patient before study enrollment. That same PPV may be unacceptable for epidemiologic and health services research studies where further validation of the IBS diagnosis may not be possible. The criterion with the highest PPV (criterion 4, which included diagnosis codes for other gastrointestinal symptoms in addition to at least one diagnosis code for IBS) accounted for only 11% of the population identified with an IBS code in our study populations. Many of the medical records reviewed for this study lacked detailed information on Rome II criteria, suggesting both that these criteria may not be a useful way to identify IBS patients through medical record review and that physicians may not be aware of these criteria and may not be using them in making the diagnosis of IBS. Over half of the medical record abstractions lacking information in the Rome II criteria had been categorized as ‘‘possible’’ by physician reviewers, indicating that many of these cases may have had evidence supporting a diagnosis of IBS, but lacked sufficient detail to fully support the diagnosis. Studies evaluating the validity of other diagnoses such as gout [9], osteoarthritis [10], and peptic ulcer disease [11] as derived from administrative data also used chart review as a gold standard for confirming diagnoses and have reported PPVs lower than those identified in the present study. However, studies evaluating the validity of other diagnoses such as idiopathic thrombocytopenia [12], rhabdomyolysis [13], and ChurgeStrauss syndrome [14] have reported PPVs in the 70e80% range, indicating greater confidence in the use of the administrative data to identify patients with those conditions. Limitations of this study include reliance on the medical record as the sole source of information for determining the presence of a diagnosis of IBS. The study would have benefited greatly from comprehensive information relevant to the Rome criteria and a consistent approach to the assessment of all patients. Direct patient assessment would likely have provided an improved ‘‘gold standard’’ for this study.
In summary, the PPVs of different criteria used to identify patients with IBS from administrative data vary substantially based on the components of the criteria that are used. Use of criteria with a higher PPV will likely affect the generalizability of the patient population that would be identified. Strict adherence to Rome II criteria for determining the presence of IBS is probably not practical for identifying patients with IBS based on information contained solely in medical records. Appendix Rome II criteria IBS can be diagnosed based on at least 12 weeks (which need not be consecutive) in the preceding 12 months, of abdominal discomfort or pain that has two out of three of these features: 1. Relieved with defecation; and/or 2. Onset associated with a change in frequency of stool; and/or 3. Onset associated with a change in form (appearance) of stool. Symptoms that cumulatively support the diagnosis of IBS are as follows: 1. Abnormal stool frequency (may be defined as greater than three bowel movements per day and less than three bowel movements per week); 2. Abnormal stool form (lumpy/hard or loose/watery stool); 3. Abnormal stool passage (straining, urgency, or feeling of incomplete evacuation); 4. Passage of mucus; and 5. Bloating or feeling of abdominal distension. References [1] Strom Bl, editor. Pharmacoepidemiology. 4th edition. Chichester, UK: Wiley; 2005.
S.L. Goff et al. / Journal of Clinical Epidemiology 61 (2008) 617e621 [2] Thompson WG. Irritable bowel syndrome: prevalence, prognosis and consequences. CMAJ 1986;134:111e3. [3] Drossman DA, Li Z, Andruzzi E, Temple RD, Talley NJ, Thompson WG, et al. US householder survey of functional gastrointestinal disorders: prevalence, sociodemography and health impact. Dig Dis Sci 1993;38:1569e80. [4] Thompson WG. The road to Rome. Gastroenterology 2006;130: 1552e6. [5] Brandt LJ, Locke R, Olden K. An evidence based approach to the diagnosis of irritable bowel syndrome in North America. Am J Gastroenterol 2002;97:S1eS26. [6] Cash BD, Chey WD. Diagnosis of irritable bowel syndrome. Gastroenterol Clin North Am 2005;34:205e20. [7] Chan KA, Davis RL, Gunter MJ, Gurwitz JH, Herrinton LJ, Nelson WW, et al. The HMO research network. In: Strom B, editor. Pharmacoepidemiology. 4th edition. Chichester, UK: Wiley; 2005. p. 261e9. [8] Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical epidemiology: a basic science for clinical medicine. 2nd edition. Boston, MA: Little Brown and Co.; 1991.
621
[9] Harrold LR, Saag KG, Yood RA, Milkuls TR, Andrade SA, Fouaysi H, et al. Validity of gout diagnoses in administrative data. Arthritis Rheum 2007;57:103e8. [10] Harrold LR, Yood RA, Andrade SE, Reed JI, Cernieux J, Strauss W, et al. Evaluating the predictive value of osteoarthritis in administrative data. Arthritis Rheum 2000;43:1881e5. [11] Andrade SE, Gurwitz JH, Chan KA, Donahue JG, Beck A, Boles M, et al. Diagnoses of peptic ulcer and bleeding from administrative databases: a multi-health maintenance organization study. J Clin Epidemiol 2002;55:310e3. [12] Segal JP, Powe NR. Accuracy of identification of patients with idiopathic thrombocytopenia using administrative records: a data validation study. Am J Hematol 2004;75:12e7. [13] Andrade SE, Graham DJ, Staffa JA, Schech SD, Stetin D, Grenade LL, et al. Health plan administrative databases can efficiently identify serious myopathy and rhabdomyolysis. J Clin Epidemiol 2005;58:171e4. [14] Harrold LR, Andrade SE, Eisner M, Buist AS, Go A, Vollmer WM, et al. Identification of patients with Churg-Strauss syndrome (CSS) using administrative data. Pharmacoepidemiol Drug Saf 2004;13:661e7.
Administrative data used to identify patients with irritable bowel syndrome Sarah L. Goffa, Andrew Feldb, Susan E. Andradea, Lisa Mahoneyb, Sarah J. Beatonc, Denise M. Boudreaub,d, Robert L. Davise, Michael Goodmanf, Cynthia L. Hartsfieldg,h, Richard Platti,j, Douglas Robline, David Smithk, Marianne Ulcickas Yoodl,m, Katherine Dodda, Jerry H. Gurwitza,* a
Meyers Primary Care Institute, University of Massachusetts Medical School, Fallon Clinic Foundation and Fallon Community Health Plan, 630 Plantation Street, Worcester, MA 01605, USA b Group Health, Center for Health Studies, Seattle, WA, USA c Lovelace Clinic Foundation, Albuquerque, NM, USA d University of Washington, Seattle, WA, USA e Kaiser Permanente Georgia, Center for Health Research/Southeast, Atlanta, GA, USA f HealthPartners Research Foundation, Minneapolis, MN, USA g Kaiser Permanente Colorado, Denver, CO, USA h University of Colorado at Denver and Health Sciences Center School of Pharmacy, Denver, CO, USA i Harvard Pilgrim Health Care, Boston, MA, USA j Harvard Medical School, Boston, MA, USA k Kaiser Permanente Northwest, Center for Health Research, Portland, OR, USA l Josephine Ford Cancer Center, Henry Ford Health System, Detroit, MI, USA m Epidemiology and Public Health, Yale University School of Medicine, New Haven, CI, USA Accepted 22 July 2007
Abstract Objective: To assess the usefulness of health plan administrative data for identifying patients with irritable bowel syndrome (IBS). Study Design and Setting: In this retrospective study of 442 medical records of patients in nine U.S. health plans, five sets of criteria that used administrative data were used to identify potential IBS patients. Physician reviewers provided an assessment of the likelihood of the diagnosis of IBS being present. IBS was considered to be present if the physician reviewer categorized the case as definite, probable, or possible based on medical record review. Analyses were also performed with cases categorized as possible placed in an ‘‘IBS not present’’ category. Results: The positive predictive value (PPV) for the five sets of criteria ranged from 63% to 83% with the highest PPV found with one of the most restrictive criteria. When cases characterized as possible were included in the ‘‘IBS not present’’ category, the PPV for each of the five sets of criteria decreased substantially, ranging from 33% to 63%. Conclusion: The PPV of different criteria used to identify patients with IBS from administrative data varies substantially based on the criteria that are used. Use of criteria with a higher PPV may come at the expense of generalizability. Ó 2008 Elsevier Inc. All rights reserved. Keywords: Irritable bowel syndrome; Health services research; Administrative databases; Diagnosis validation studies; pharmacoepidemiology; health maintenance organizations
1. Introduction Health planeautomated databases are frequently used to conduct epidemiologic studies, for economic analyses, to identify potential subjects for recruitment into clinical
* Corresponding author. Tel.: 508-791-7392; fax: 508-595-2200. E-mail address: [email protected] or jerry.gurwitz@ umassmed.edu (J.H. Gurwitz). 0895-4356/08/$ e see front matter Ó 2008 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2007.07.013
trials, and for other research purposes [1]. However, the validity of diagnoses derived from these data sources is often questioned. The results of validation studies can help gauge the usefulness of these data for research purposes. Irritable bowel syndrome (IBS) affects an estimated 10e15% of adults in the United States [2,3]. The subjective nature of the IBS diagnosis presents special challenges for both clinicians and researchers. Diagnostic criteria, such as the Rome II criteria (Appendix), have been used for
618
S.L. Goff et al. / Journal of Clinical Epidemiology 61 (2008) 617e621
What is New? Key findings 1. The positive predictive values (PPVs) of different criteria used to identify patients with irritable bowel syndrome (IBS) from administrative data vary substantially based on the components of the criteria that are used. 2. Use of criteria with a higher PPV will likely affect the generalizability of the patient population that would be identified. New addition Strict adherence to Rome II criteria for determining the presence of IBS is probably not practical for identifying patients with IBS based on information contained solely in medical records. Implications The use of administrative data alone may be unacceptable for epidemiologic and health services research studies where further validation of the IBS diagnosis may not be possible.
research purposes [4,5], and have been promoted for clinical use, but their role has not been adequately defined [6], and clinicians may not use or even be aware of these criteria when diagnosing and treating patients. In this study, we used information derived from medical record review to evaluate the positive predictive values (PPVs) of five different sets of criteria based solely on administrative data to identify patients with IBS. These criteria were developed using diagnostic and pharmacyadministrative data available from nine large health plans.
pseudo-membranous colitis, iron deficiency anemia, rectal bleeding, colon cancer, or weight loss. A multidisciplinary subgroup of the authors, with expertise in gastroenterology (A.F.), internal medicine (J.H.G.), and pharmacoepidemiology and use of administrative databases (S.E.A. and R.L.D.), convened to develop five sets of criteria based on IBS-related diagnoses and selected medications related to the treatment of IBS: (1) at least one diagnosis of IBS (ICD-9 564.1) identified from outpatient or inpatient derived administrative data; (2) at least one diagnosis of IBS and at least one dispensing of a laxative available by prescription (lactulose, PEG-electrolyte, and sorbitoldalone or in combination), antidiarrheals (diphenoxylate, kaolin, and pectindalone or in combination), antispasmodics (belladonna, scopolamine hydrobromide, opium, paregoric, dicyclomine hydrochloride, and hyoscamine sulfatedalone or in combination), 5-HT3 receptor antagonist (alosetron), or a 5-HT4 receptor agonist (tegaserod); (3) at least two IBS diagnoses occurring at least 6 months apart identified from outpatient or inpatient data; (4) at least one diagnosis of IBS identified from outpatient or inpatient data, at least one diagnosis of abdominal pain (ICD-9 789.0), and one other gastrointestinal symptom (either diarrhea [ICD-9 564.5, 787.91] or constipation [ICD-9 564.0]); and (5) at least one diagnosis of IBS and one diagnosis for abdominal pain with at least one other gastrointestinal symptom and at least one dispensing of any medication listed in criterion (2) above (Table 1). Each of the nine health plan sites selected a random sample of up to 25 patients for each of the five sets of criteria, with up to 10 samples of 25 obtained for medical record abstraction. If medical records were unavailable, additional patients were randomly chosen from the list of 25 to ultimately obtain the target of 10 medical records for each of the five sets of criteria for review at each participating health plan site, for a total of 450 medical records. Criteria were not mutually exclusive, meaning one patient could meet any or all of the five sets of criteria; however, in no instance was a patient included for medical record abstraction more than once.
2. Methods We conducted a retrospective study among patients enrolled in nine geographically dispersed health plans participating in the HMO Research Network Center for Education and Research on Therapeutics [7]. Institutional Review Board approval for this research project was obtained at each site. Patients aged 18 years or above with an International Classification of Diseases, 9th edition (ICD-9-CM) code for IBS (564.1) existing within the administrative databases of the health plans, relating to both inpatient and outpatient services between May 1, 2002 and September 15, 2002, were identified. Patients were excluded if they also had an ICD-9-CM code for a disease with symptoms that might overlap with IBS or which might suggest a different diagnosis, including Crohn’s disease, ulcerative colitis, ischemic colitis, microscopic or collagenous colitis,
Table 1 Chart review selection criteria for identification of potential cases of IBS Criterion 1: At least one diagnosis of IBS (ICD-9 564.1) identified from outpatient or hospital data. Criterion 2: At least one diagnosis of IBS and at least one prescription for an antispasmodic, laxative, antidiarrheal, 5HT3 receptor agonist, or 5-HT4 receptor antagonist. Criterion 3: At least two or more diagnoses of IBS occurring at least 6 months apart identified from outpatient or hospital data. Criterion 4: At least one diagnosis of IBS identified from outpatient or hospital data and at least one diagnosis of abdominal pain (ICD-9 789.0) and one other gastrointestinal symptom (either diarrhea [ICD-9 564.5, 787.91] or constipation [ICD-9 564.0]). Criterion 5: At least one diagnosis of IBS and one diagnosis for abdominal pain with at least one other gastrointestinal symptom and at least one prescription for any drug listed in criterion 4 above. Abbreviation: IBS, irritable bowel syndrome.
S.L. Goff et al. / Journal of Clinical Epidemiology 61 (2008) 617e621
Trained abstractors provided a summary of medical record information pertinent to the diagnosis of IBS to the physician reviewers. This information included all office visit notes from primary care physicians and relevant consultation reports by specialists as well as relevant diagnostic studies and tests including colonoscopy, sigmoidoscopy, abdominal x-rays, upper gastrointestinal series, computed tomography scans, ultrasounds, stool cultures, and relevant laboratory studies. The medical record abstraction covered a period from 6 months before the appearance of an IBS diagnosis associated with a clinical encounter to 1 year following that date. The time period of 18 months was chosen because IBS is often a diagnosis of exclusion requiring sufficient time for symptoms to be fully evaluated. This information was then reviewed by one of three physicians, including a gastroenterologist (A.F.) and two general internists (J.H.G. and S.L.G.). When assessing clinical information, each physician noted whether the case met Rome II criteria and also characterized the likelihood of the IBS diagnosis being present according to the following categories: unlikely, possible, probable, definite, or unable to assess. Categorization was based on clinical judgment, taking into account all information available in the medical record abstraction. Using the gastroenterologist’s (A.F.) assessment as the gold standard, the internists (J.H.G. and S.L.G.) underwent a training session before formally beginning to assess the medical record abstractions. During these training sessions, all three reviewers assessed 12 of the same medical record abstractions and when either of the internists’ (J.H.G. or S.L.G.) assessments differed from the gastroenterologist’s (A.F.) assessment, the case was discussed to maximize the consistency of physician assessments. IBS was considered to be present if the physician reviewer categorized the case as definite, probable, or possible. IBS was considered not to be present if the physician reviewer categorized the case as unlikely or unable to assess. Analyses were also performed with cases categorized as possible placed in the ‘‘IBS not present’’ category to assess the impact on the estimates of PPV. To assess inter-rater reliability between the general internists and the ‘‘gold standard’’ gastroenterologist reviewer, 10% (45) of the medical record abstractions reviewed by the gastroenterologist (A.F.) were also reviewed by each of the general internists (J.H.G. and S.L.G.). Reviewers were considered to be in agreement if they had rated the case unlikely or unable to assess (IBS not present) or if they rated the case possible, probable, or definite (IBS present). The kappa values for inter-rater reliability were calculated for each of the internists (J.H.G. and S.G.) compared with the gastroenterologist (A.F.) and were 0.61 (J.H.G.) and 0.80 (S.G.). A kappa score between 0.6 and 0.8 reflects ‘‘substantial’’ agreement and a kappa score between 0.8 and 1.0 is considered ‘‘almost perfect’’ [8]. The PPV was calculated for each of the five sets of criteria that were used to identify patients with IBS from the
619
nine participating health plans. Information was also collected regarding the presence of clinical information pertaining to the Rome II criteria in the medical record.
3. Results A total of 7,000 patients meeting one or more criteria were identified. More than half of these patients (56%) met only criterion 1 (one diagnosis of IBS), 2,338 (34%) met two criteria, and 718 patients (10%) met three or more criteria (Table 2). After sampling for each of the sets of criteria as described in Section 2, 453 medical records were abstracted and reviewed. Eleven medical record abstractions were removed from the analysis because the patient was less than 18 years old, leaving 442 medical record abstractions available for inclusion in the study. The mean age of these patients was 52 (range 18e92) years and 348 (79%) were female. Overall, the PPV for the 442 cases was 73%. The PPV for the five sets of criteria ranged from 63% to 83% (Table 3). The PPVs were higher for the criteria that included diagnoses in the administrative data supporting IBS (Table 2). When cases characterized as possible were included in the ‘‘IBS not present’’ category, the PPV for each of the five sets of criteria decreased substantially, ranging from 33% to 63%. Clinical information relating to the Rome II criteria was not consistently found in the medical record. Of the 321 cases determined to have IBS present based on review of the abstracted medical records by physician reviewers, 179 (56%) had no information on two or more of the three primary Rome II criteria used to establish the diagnosis of IBS. Of these 179, 8 had been categorized as definite, 76 as probable, and 95 as possible.
4. Discussion Our findings suggest that the utility of the five sets of criteria we used to identify patients with IBS varied substantially according to which criteria were used. When considering the implications of these findings, it is important to consider the purpose for which the information is being Table 2 Number of criteria met by potential study subjects Number of criteria met One criterion met Two criteria met
Criteria
Total
Criterion 1 3,944 (56%) 3,944 (56%) Criteria 1 & 2 1,750 (25%) 2,338 (33%) Criteria 1 & 3 330 (5%) Criteria 1 & 4 1,750 (4%) Three criteria met Criteria 1, 2, & 3 212 (3%) 292 (4%) Criteria 1, 2, & 5 34 (0.5%) Criteria 1, 3, & 4 46 (1%) Four criteria met Criteria 1, 2, 3, & 5 12 (0.2%) 360 (5%) Criteria 1, 2, 4, & 5 348 (5%) Five criteria met Criteria 1, 2, 3, 4, & 5 66 (1%) 66 (1%)
620
S.L. Goff et al. / Journal of Clinical Epidemiology 61 (2008) 617e621
Table 3 Validation of criteria based on diagnoses and pharmaceutical prescription data located in health plan administrative databases for identification of cases of IBS Patients meeting criterion (%)a
Criterion Criterion Criterion Criterion Criterion Criterion
1: 2: 3: 4: 5:
IBS IBS prescription IBS 2, 6 months apart IBS, abdominal pain, diarrhea, and constipation IBS, abdominal pain, diarrhea, and constipation prescription
Total
7,000 2,422 594 790 460
(100%) (35%) (8%) (11%) (7%)
7,000
Number of charts reviewed
Number of cases confirmed
92 88 87 87 88
58 59 65 72 67
442
321
PPV (95% CI) 63% 67% 75% 83% 76%
(53, (57, (66, (75, (67,
73) 77) 84) 91) 85)
Abbreviations: IBS, irritable bowel syndrome; PPV, positive predictive value. a Percentages do not add to 100% because patients could meet more than one criterion.
used. For example, a PPV of 63%, as found with a diagnosis of IBS alone, may be sufficient to identify patients for recruitment into clinical trials, where additional information that further supports the IBS diagnosis can be directly ascertained from the patient before study enrollment. That same PPV may be unacceptable for epidemiologic and health services research studies where further validation of the IBS diagnosis may not be possible. The criterion with the highest PPV (criterion 4, which included diagnosis codes for other gastrointestinal symptoms in addition to at least one diagnosis code for IBS) accounted for only 11% of the population identified with an IBS code in our study populations. Many of the medical records reviewed for this study lacked detailed information on Rome II criteria, suggesting both that these criteria may not be a useful way to identify IBS patients through medical record review and that physicians may not be aware of these criteria and may not be using them in making the diagnosis of IBS. Over half of the medical record abstractions lacking information in the Rome II criteria had been categorized as ‘‘possible’’ by physician reviewers, indicating that many of these cases may have had evidence supporting a diagnosis of IBS, but lacked sufficient detail to fully support the diagnosis. Studies evaluating the validity of other diagnoses such as gout [9], osteoarthritis [10], and peptic ulcer disease [11] as derived from administrative data also used chart review as a gold standard for confirming diagnoses and have reported PPVs lower than those identified in the present study. However, studies evaluating the validity of other diagnoses such as idiopathic thrombocytopenia [12], rhabdomyolysis [13], and ChurgeStrauss syndrome [14] have reported PPVs in the 70e80% range, indicating greater confidence in the use of the administrative data to identify patients with those conditions. Limitations of this study include reliance on the medical record as the sole source of information for determining the presence of a diagnosis of IBS. The study would have benefited greatly from comprehensive information relevant to the Rome criteria and a consistent approach to the assessment of all patients. Direct patient assessment would likely have provided an improved ‘‘gold standard’’ for this study.
In summary, the PPVs of different criteria used to identify patients with IBS from administrative data vary substantially based on the components of the criteria that are used. Use of criteria with a higher PPV will likely affect the generalizability of the patient population that would be identified. Strict adherence to Rome II criteria for determining the presence of IBS is probably not practical for identifying patients with IBS based on information contained solely in medical records. Appendix Rome II criteria IBS can be diagnosed based on at least 12 weeks (which need not be consecutive) in the preceding 12 months, of abdominal discomfort or pain that has two out of three of these features: 1. Relieved with defecation; and/or 2. Onset associated with a change in frequency of stool; and/or 3. Onset associated with a change in form (appearance) of stool. Symptoms that cumulatively support the diagnosis of IBS are as follows: 1. Abnormal stool frequency (may be defined as greater than three bowel movements per day and less than three bowel movements per week); 2. Abnormal stool form (lumpy/hard or loose/watery stool); 3. Abnormal stool passage (straining, urgency, or feeling of incomplete evacuation); 4. Passage of mucus; and 5. Bloating or feeling of abdominal distension. References [1] Strom Bl, editor. Pharmacoepidemiology. 4th edition. Chichester, UK: Wiley; 2005.
S.L. Goff et al. / Journal of Clinical Epidemiology 61 (2008) 617e621 [2] Thompson WG. Irritable bowel syndrome: prevalence, prognosis and consequences. CMAJ 1986;134:111e3. [3] Drossman DA, Li Z, Andruzzi E, Temple RD, Talley NJ, Thompson WG, et al. US householder survey of functional gastrointestinal disorders: prevalence, sociodemography and health impact. Dig Dis Sci 1993;38:1569e80. [4] Thompson WG. The road to Rome. Gastroenterology 2006;130: 1552e6. [5] Brandt LJ, Locke R, Olden K. An evidence based approach to the diagnosis of irritable bowel syndrome in North America. Am J Gastroenterol 2002;97:S1eS26. [6] Cash BD, Chey WD. Diagnosis of irritable bowel syndrome. Gastroenterol Clin North Am 2005;34:205e20. [7] Chan KA, Davis RL, Gunter MJ, Gurwitz JH, Herrinton LJ, Nelson WW, et al. The HMO research network. In: Strom B, editor. Pharmacoepidemiology. 4th edition. Chichester, UK: Wiley; 2005. p. 261e9. [8] Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical epidemiology: a basic science for clinical medicine. 2nd edition. Boston, MA: Little Brown and Co.; 1991.
621
[9] Harrold LR, Saag KG, Yood RA, Milkuls TR, Andrade SA, Fouaysi H, et al. Validity of gout diagnoses in administrative data. Arthritis Rheum 2007;57:103e8. [10] Harrold LR, Yood RA, Andrade SE, Reed JI, Cernieux J, Strauss W, et al. Evaluating the predictive value of osteoarthritis in administrative data. Arthritis Rheum 2000;43:1881e5. [11] Andrade SE, Gurwitz JH, Chan KA, Donahue JG, Beck A, Boles M, et al. Diagnoses of peptic ulcer and bleeding from administrative databases: a multi-health maintenance organization study. J Clin Epidemiol 2002;55:310e3. [12] Segal JP, Powe NR. Accuracy of identification of patients with idiopathic thrombocytopenia using administrative records: a data validation study. Am J Hematol 2004;75:12e7. [13] Andrade SE, Graham DJ, Staffa JA, Schech SD, Stetin D, Grenade LL, et al. Health plan administrative databases can efficiently identify serious myopathy and rhabdomyolysis. J Clin Epidemiol 2005;58:171e4. [14] Harrold LR, Andrade SE, Eisner M, Buist AS, Go A, Vollmer WM, et al. Identification of patients with Churg-Strauss syndrome (CSS) using administrative data. Pharmacoepidemiol Drug Saf 2004;13:661e7.