- Email: [email protected]
Physician-derived asthma diagnoses made on the basis of questionnaire data are in good agreement with interview-based diagnoses and are not affected by objective tests
Physician-derived asthma diagnoses made on the basis of questionnaire data are in good agreement with interviewbased diagnoses and are not affected by objective tests Kathleen C. Barnes, PhD,a Linda R. Freidhoff, MS,a Edward M. Horowitz, BA,a Rasika A. Mathias, MS,b Dana M. Mulkern, BS,a Julia T. Bonacum, MD,c Michael H. Goldman, MD,a Albert J. Polito, MD,a,c Sarbjit S. Saini, MD,a David G. Marsh, PhD,a† Terri H. Beaty, PhD,b and Alkis Togias, MDa Baltimore, Md
Background: Defining the phenotype is critical for investigating the genetic etiology of asthma. As part of the Collaborative Study on the Genetics of Asthma (CSGA), the primary objective of which is to identify asthma susceptibility loci, an algorithm was designed to determine diagnoses of definite asthma, probable asthma, less than probable asthma, or no asthma. A respiratory questionnaire was designed to assist in the process of characterizing the asthma phenotype. Objective: This study was designed to determine the validity of the CSGA algorithm for the diagnosis of asthma, to determine agreement in assessing an asthma diagnosis between the information obtained by the CSGA questionnaire versus a patient interview by a panel of specialist physicians, and to determine the degree to which objective tests would alter the questionnaire-based certainty of asthma diagnosis. Methods: An expert panel of asthma clinicians (n = 4) indicated to what degree they were certain that a subject (n = 48) had asthma as determined by using a 6-point Likert scale based on a 20-minute interview (phase I), a review of the CSGA questionnaire (phase II), a review of the questionnaire plus skin test and peripheral blood eosinophilia data (phase III), and a review of phase III information plus pulmonary data (spirometry and methacholine-reversibility testing; IV). Intraclass correlation coefficients (ICCs) were calculated between the physicians’ interpretation of the likelihood of asthma based on the information they received during each of the phases and between the CSGA algorithm and each of the phases. Results: Interjudge reliability with regard to the degree of certainty with which an asthma diagnosis could be made by interview was excellent (ICC, 98; 95% confidence intervals [95% CIs], 0.87–0.99). We also found that the agreement between the
From athe Division of Clinical Immunology, Department of Medicine, bthe Department of Epidemiology, Johns Hopkins School of Hygiene and Public Health, and cthe Division of Pulmonary Medicine, Department of Medicine, The Johns Hopkins University School of Medicine, Baltimore. Supported by US Public Health Service Grant HL49612-06. †Dr. David G. Marsh was actively involved in the design and implementation of this study. He passed away on March 29, 1998, after a heroic battle with cancer. We dedicate this work and manuscript to his memory. Received for publication Feb 26, 1999; revised July 13, 1999; accepted for publication July 13, 1999. Reprint requests: Kathleen C. Barnes, PhD, Johns Hopkins Asthma & Allergy Center, 5501 Hopkins Bayview Circle, Baltimore, MD 21224. Copyright © 1999 by Mosby, Inc. 0091-6749/99 $8.00 + 0 1/1/101510
physicians’ interview with the patients (phase I) and the CSGA algorithm was good and at least as good with the addition of the CSGA questionnaire data and objective data (ICC, 0.65–0.75). Good agreement was also observed between the average certainty score from the interview and the CSGA questionnaire (ICC, 92; 95% CI, 0.76–0.93), and ICCs determining the agreement on asthma diagnosis between phase I and phases III and IV, in which objective data were introduced, did not change from the ICCs comparing phase I with phase II (ICC of 0.93 [95% CI, 0.79–0.96] and ICC of 0.91 [95% CI 0.73–0.95], respectively). Conclusion: We conclude that the CSGA algorithm is a valid tool for which the diagnosis of asthma can be made at an acceptable level of certainty and that the CSGA questionnaire, interpreted by an asthma specialist, is a useful tool for the diagnosis of asthma in clinical or epidemiologic studies. (J Allergy Clin Immunol 1999;104:791-6.) Key words: Asthma questionnaire, Collaborative Study on the Genetics of Asthma, interjudge reliability, intraclass correlation coefficient
The Collaborative Study on the Genetics of Asthma (CSGA), funded by the National Heart, Lung, and Blood Institute, is an ongoing study with the aim of identifying the genes controlling asthma. Data have been combined from 4 centers, and a 10 cM genome–wide screen in 261 affected sib pairs and their parents, representing 3 racialethnic groups (Caucasian, African American, and Hispanic), has been performed.1 Before attempting to dissect the genetic basis of asthma, investigators of the CSGA felt that it was necessary to establish a process by which the diagnosis of asthma could be achieved to the degree that the risk of false-positive results could be minimized. To accomplish this goal, an asthma algorithm designed by Panhuysen et al2 was modified for the CSGA to classify each proband and all family members participating in the study. The CSGA algorithm classifies subjects essentially into 4 categories: (1) definite asthma, (2) probable asthma, (3) less than probable asthma, (4) no asthma (eg, no evidence of asthma or other pulmonary diseases). Individuals with a greater than 5 pack-year smoking history are not considered in the algorithm. The algorithm is generated from asthma symptomatology, lung function data, a diagnostic history, and history of smoking, and there791
792 Barnes et al
Abbreviations used ATS: American Thoracic Society BHR: Bronchial hyperresponsiveness CSGA: Collaborative Study on the Genetics of Asthma ICC: Intraclass correlation coefficient SPT: Skin prick test
fore it relies on precise collection of subjective data (eg, symptoms and medical history) and pulmonary function testing data, including a measure of bronchial hyperresponsiveness (BHR), testing of reversibility of airway obstruction with a bronchodilator, or both. To collect the subjective data in a standardized fashion, investigators of the CSGA modified the American Thoracic Society (ATS)–Division of Lung Disease questionnaire3 to produce an asthma-specific instrument that could be used in combination with objective data to diagnose asthma. The CSGA respiratory questionnaire assesses frequency and duration of asthma and allergy symptoms. Modifications from the original ATS questionnaire included a change from a self-administered to an interviewer-administered format and a greater focus on the characteristics of asthma by deleting a number of questions about other chronic respiratory diseases; however, appropriate questions remained in the modified CSGA respiratory questionnaire to discriminate between asthma and conflicting diagnoses (eg, chronic obstructive pulmonary disease and congestive heart failure). The final asthma diagnosis protocol used in the CSGA included a number of criteria obtained from the questionnaire combined with a positive methacholine bronchial provocation. We designed a study to validate the CSGA algorithm for the diagnosis of asthma. The protocol we designed was based on a validation study of an asthma severity questionnaire that our group has recently performed.4 In brief, we analyzed the agreement between an asthma certainty score derived from a panel of expert clinicians in asthma and pulmonary medicine after they interviewed individuals with and without evidence of asthma, a score derived from the same physicians when they independently evaluated the answers of the same subjects to the CSGA questionnaire with and without objective data, and the asthma categorization according to the CSGA algorithm. In addition to our goal to validate the CSGA algorithm as a tool for the diagnosis of asthma, we also wanted to compare the degree of agreement on the likelihood of an asthma diagnosis that a group of specialist physicians would demonstrate between interviewing subjects and reviewing their responses to the CSGA questionnaire and to test whether objective information added to the results of the questionnaire would change the physicians’ original opinion on asthma diagnosis.
METHODS Study design Fifty-two subjects were selected from the Johns Hopkins University (Baltimore, Md) center as part of the CSGA, which has been
J ALLERGY CLIN IMMUNOL OCTOBER 1999
previously described.1,5 All subjects had previously visited the Johns Hopkins Asthma and Allergy Center and undergone full clinical evaluations according to the CSGA guidelines, which included the respiratory questionnaire, medical evaluation, bronchial reactivity or reversibility tests, skin prick tests (SPT), and blood collection for total IgE, peripheral blood eosinophil counts, and DNA typing, between November 1993 and August 1995. These subjects were asked to return for an interview with the panel of physicians between October and November 1995. Before participation, all subjects gave verbal and written consent as approved by the Johns Hopkins Bayview Hospital Institutional Review Board. The data collected from the subjects were used in the 4 phases of the study as follows: phase I, assignment of a physician-diagnosed asthma likelihood score based on a 20-minute interview; phase II, assignment of a physician-diagnosed asthma likelihood score based on the subjects’ responses to the CSGA respiratory questionnaire; phase III, assignment of a physician-diagnosed asthma likelihood score based on data from the CSGA respiratory questionnaire, SPTs, and peripheral blood eosinophil counts; and phase IV, assignment of a physician-diagnosed asthma likelihood score based on data from the CSGA respiratory questionnaire, SPTs, peripheral blood eosinophil counts, and pulmonary function tests. For phases II through IV, each physician received a packet containing the clinical data relevant to the particular phase for each subject. Random codes were assigned to all data forms to conceal subject identification. For each phase, each physician was asked to score the likelihood that the subject had asthma on the basis of a 6-point Likert scale (1, no asthma; 6, definite asthma). Physicians were also asked to complete an interview worksheet in which they recorded notes regarding the patient’s history and described which questions, responses, or data were most influential in making their diagnosis. The physicians were asked to refrain from discussing their decisions with each other and with any other clinicians and to make their own independent diagnoses. Physicians evaluated the packets for each phase, documented their scores within a 2-week period, and returned the information to the study coordinator.
Subject selection Complete clinical data were available for 277 subjects enrolled in the CSGA at the Johns Hopkins University School of Medicine site before August 1995. To recruit a range of asthmatic subjects classified from no asthma to definite asthma, a preliminary 5-category classification was created. Briefly, this classification factored the presence or absence of the following criteria: (1) the number of asthma-like symptoms (wheeze, cough, shortness of breath) experienced; (2) nonspecific bronchial reactivity to methacholine or a positive reversibility test after inhalation of a β-agonist (albuterol); and (3) the diagnosis of asthma by a clinician. Adjustments for cigarette smoking were made based on the reported number of pack-years. Because the diagnosis of asthma is more difficult in subjects who do not fulfill all of the conventional criteria (eg, hallmark symptoms and BHR), more subjects were recruited from the 3 middle asthma algorithm categories (n = 12 for each category) than from the 2 extreme (no asthma and definite asthma) categories (n = 8 for each category). A randomized list of subjects was generated for each of the 5 categories, and subjects (n = 52) were recruited by telephone. If a subject in a given category refused to participate, the study coordinator selected the next subject from the randomized list within that category. Four subjects failed to complete the validation study (2 from category 0 and 2 from category 3), resulting in a total of 48 participants.
The CSGA algorithm As part of the classification process for asthma, an algorithm described by Panhuysen et al2 was modified and applied to the
Barnes et al 793
J ALLERGY CLIN IMMUNOL VOLUME 104, NUMBER 4, PART 1
TABLE I. The CSGA asthma algorithm
Diagnosis
Pulmonary function testing*
No. of symptoms reported (wheeze, cough, shortness of breath)
Previous physician diagnosis or use of asthma medications
∆FEV1 ≥20% with < 25 mg ≥15% REV
≥2 ≥2
Optional Optional
∆FEV1 ≥20% with ≤10 mg ∆FEV1 <15% with 25 mg ∆FEV1 <15% with 25 mg ≥15% REV
1 ≥2 ≤1 1
Optional Yes Yes Optional
∆FEV1 ≥20% with ≤10 mg ∆FEV1 ≥20% with >10-25 mg ∆FEV1 ≥15% and <20% with ≤25 mg ≥15% REV
0 ≤1 ≥2 0
Optional Optional Optional Optional
∆FEV1 ≥15% and <20% with ≤25 mg ∆FEV1 <15% with 25 mg
≤1 ≥0
Optional No
Asthma or Probable asthma or or Less than probable asthma or or No asthma or
Adapted from Panhuysen et al.2 ∆FEV1, Change in forced expiratory volume; REV, reversibility of airway obstruction defined as the percentage of increase in FEV1 predicted after use of a bronchodilator. *Methacholine provocation or bronchodilator reversibility.
CSGA families (Table I). Briefly, the CSGA algorithm classifies subjects as having definite asthma, probable asthma, less than probable asthma, and no asthma. The “less than probable asthma” group includes subjects who are unlikely to have asthma but for whom a definitive diagnosis is not possible because of confounding factors (eg, a history of chronic obstructive pulmonary disease and unclassifiable airway disease because of smoking). The algorithm is based on respiratory symptoms, a prior diagnosis of asthma by a physician, smoking history (recorded in the CSGA questionnaire), and the presence or absence of BHR, bronchodilator reversibility of airway obstruction, or both.
interviewed in the presence of a parent or legal guardian. Recorded information included symptoms of respiratory disease, including asthma (cough, wheeze, shortness of breath, and chest tightness), triggers that provoked asthma symptoms, symptoms and triggers related to allergic rhinitis and eczema, previous diagnoses by a physician, asthma medication use (past and current), asthma-related hospitalizations and emergency department visits, and smoking history. Using the 6-point Likert scale, each physician produced an asthma certainty score for each patient on the basis of a review of the questionnaire.
Phases III and IV
Phase I: Patient interviews Using the methodology designed by Horowitz et al,4 2 expert clinicians from the Division of Pulmonary and Critical Care Medicine and 2 expert clinicians from the Division of Clinical Immunology at The Johns Hopkins Asthma and Allergy Center were selected to participate on the physician panel. The panel of physicians interviewed each patient for 20 minutes. Subjects under 18 years of age (n = 12) were accompanied by a parent or legal guardian. In the event that a physician had previously examined or had prior medical knowledge about the patient (2 subjects), the physician refrained from evaluating the patient. Physicians were allowed to ask any question necessary to determine the likelihood that a subject had asthma according to their own definition of the disease; no description of asthma or guidelines for diagnosing asthma were discussed with the physicians before the study. Evaluations were based strictly on historical data; no physical examination was performed, and there was no access to any objective data. Each physician produced an independent score (eg, certainty of asthma; 0 [no asthma] to 5 [definite asthma]) for each patient on the basis of the abovedescribed 6-point Likert scale.
Phase II: Respiratory questionnaire The CSGA respiratory questionnaire had been previously administered to all subjects by trained technicians during the initial CSGA clinical evaluation. Subjects under 18 years of age had been
SPTs. Semiquantitative SPT procedures6 were performed on all subjects by using bifurcated smallpox vaccination needles (Wyeth Laboratories, Marietta, Pa) and standardized glycerinated extracts (ALK Laboratories Hørsholm, Denmark, and Greer Laboratories, Lenoir, NC) of 30 common inhalant allergens (arthropod-animal, pollen, and mold). Positive reactions were recorded by tracing the perimeter of the wheal 15 minutes after puncturing the skin and transferring the tracing with transparent surgical tape (Transpore; 3M Co, Minneapolis, Minn) to paper labeled for each extract, creating a permanent record of the skin test reactions to each extract for all subjects. Eosinophilia. Plasma was collected in EDTA tubes from each patient at the time of clinical evaluation. Routine hematologic testing with differential was performed at the Laboratory Corporation of America (Baltimore, Md) for the purpose of measuring eosinophils by using the automated STKS (Coulter Inc, Fort Lauderdale, Fla). For an absolute value, eosinophils were counted manually on a slide after automatic staining (Hematek; Miles Diagnostics, Kankakee, Ill) with 22% wt/vol Wright and 0.50% wt/vol Giemsa stain in 220 mL of methanol, 540 mL of phosphate buffer (pH 6.8), and 990 mL of methyl alcohol solution in 10% H2O (Fisher Science, Pittsburgh, Pa). Results were presented as a per-
794 Barnes et al
J ALLERGY CLIN IMMUNOL OCTOBER 1999
TABLE II. ICCs and 95% CIs between asthma categorization as defined by the CSGA algorithm and the mean physician scores Phase
ICC
95% CI
I II III IV
0.63 0.65 0.69 0.75
0.15-0.77 0.17-0.78 0.21-0.81 0.28-0.84
TABLE III. ICCs and 95% CIs between the mean physician scores generated by the patient interview (phase I) and the mean physician scores generated in the other phases Phase
ICC
95% CI
II III IV
0.92 0.93 0.91
0.76-0.93 0.79-0.96 0.73-0.95
Physician scores were generated by the patient interview (phase I); a review of the questionnaire (phase II); a review of the questionnaire, SPT data, and eosinophil counts (phase III); and a review of the questionnaire, SPT data, eosinophil counts, and complete pulmonary data (spirometry and methacholine-reversibility testing; phase IV).
The mean physician scores were generated from review of the questionnaire (phase II); review of the questionnaire, SPT data, and eosinophil counts (phase III); and review of the questionnaire, SPT data, eosinophil counts, and complete pulmonary data (spirometry and methacholine-reversibility testing; phase IV).
centage of white blood cells or as an absolute number (value × 10–3 µL). Pulmonary function testing. Baseline pulmonary function (forced vital capacity [FVC], FEV1, FEV1/FVC, peak expiratory flow rate, and FEF25-75) was measured with a KOKO (Pulmonary Data Services, Inc, Louisville, Colo) pneumotach connected to a desktop or laptop computer in accordance with the ATS standards of spirometry.7 Subjects performed at least 3 forced expiratory maneuvers until 2 repeatable FEV1 values were obtained (within 10% of each other), and the highest value was recorded. If the patient’s FEV1 was less than 70% of the predicted value, a reversibility test was performed by delivering 2 inhalations of a β-agonist (albuterol) by means of a metered-dose inhaler and measuring FEV1 after 10 minutes. A positive reversibility test response was recorded if the patient’s FEV1 was increased by 15% from baseline. If the patient’s FEV1 was 70% or greater of the predicted value, a quantitative bronchoprovocation challenge was conducted. Methacholine challenge was performed as described by Chai et al.8 A breath-actuated dosimeter was used to deliver a diluent control and successive half-log incremental concentrations of methacholine (0.025, 0.075, 0.25, 0.75, 2.5, 7.5, and 25 mg/mL). The provocation ended after reaching a 20% reduction in FEV1 from the postdiluent value or after the highest concentration of methacholine had been inhaled. The PD20 and the PC20 were determined by interpolation of the dose-response curve (log PD20 cumulative breath units = 1 breath of a 1 mg/mL concentration and PC20 = mg/mL). Patients were asked to refrain from taking β-agonists and theophylline 24 hours before testing. For phase III, physicians were given blinded packages, including the following for each subject: (1) the questionnaires (as in phase II), (2) the record of SPT reactions to each of the 30 extracts, and (3) eosinophil counts both as a percentage of white blood cells and as an absolute number. For phase IV, physicians were given blinded packages with all material for phase III plus a blinded summary of pulmonary testing (FEV1/FVC, FVC percentage predicted, FEV1 percentage predicted, and PC20). As with phases I and II, each physician scored each patient’s phase III and phase IV packages by using the 6-point Likert scale.
of the information they received during each of the 4 phases of the study and the diagnosis of our study subjects according to the algorithm. The Likert scale scores generated by the 4 physicians in each phase were averaged, and the average was then used in the calculation of the ICC. Calculation of ICC was based on a 2-way ANOVA model wherein the between-targets (subjects) mean square (variance of interest) and the residual error mean square were obtained. This statistical approach is appropriate because each target is rated by each judge, and the intraclass correlation can be derived by the following formula as described by Shrout and Fleiss12: BMS – EMS/BMS, where BMS is the between-targets mean squares and EMS is the residual or error of the mean square. A 95% confidence interval (CI) was calculated for each ICC by using the same methodology.12 To determine the degree of agreement between the physicians’ interpretation of the likelihood of asthma on the basis of the information they received during the interview and the questionnaire and from the objective data, ICCs between phase I and phases II through IV were calculated by using the mean asthma certainty scores of the clinicians for each phase.
Data analysis Similarity of diagnosis among the 4 physicians (interjudge reliability) in their rating of the 48 patients was assessed by calculating the intraclass correlation coefficient (ICC), which is the ratio of the variance of interest over the sum of the variance of interest plus the error.9-12 To validate the CSGA algorithm, ICCs were calculated between the physicians’ interpretation of the likelihood of asthma on the basis
RESULTS Complete data on questionnaire, SPT, eosinophilia, spirometry, and PD20 reversibility data were obtained from 48 subjects (35% male, 35% African American, and 65% Caucasian). The ICC between the physician scores generated by the patient interview (phase I) was 0.98 (95% CI, 0.86–0.99), indicating that only 2% of the variability in the measurement of asthma certainty was caused by differences between the judges (physicians), whereas the remaining 98% was caused by differences between the patients. ICCs among the physician scores generated in phases II through IV were equally high (0.95 [95% CI, 0.71–0.97], 0.95 [95% CI, 0.73–0.97], and 0.89 [95% CI, 0.52–0.93], respectively). These data allowed us, at each phase, to use the average physician score as an independent variable. To validate the CSGA approach of the diagnosis of asthma, we compared the average scores generated by 4 physicians on the basis of the 20-minute interview with the category of asthma certainty derived by the CSGA algorithm. Table II indicates that the agreement between the physicians’ interview with the patients (phase I) and the CSGA algorithm was good and that the ICC was at least as good with the addition of objective information, especially with SPT-eosinophil counts (ICC, 0.69 [95%
Barnes et al 795
J ALLERGY CLIN IMMUNOL VOLUME 104, NUMBER 4, PART 1
TABLE IV. Components of the CSGA respiratory questionnaire that influenced the clinicians’ diagnosis of asthma (phase II) Percentage response (phase II)* Item or information
Physician 1
Physician 2
Physician 3
62 32 18 53 12 6 12 0
65 44 24 15 32 9 15 3
58 56 38 17 19 31 8 2
History of asthma symptoms Relief by asthma medications Symptoms associated with triggers History of allergies History of asthma Asthma symptoms associated with upper respiratory infection History of other pulmonary diseases Emergency department visits or hospitalizations
Physician 4
38 13 25 25 0 0 0 0
Mean
56 36 26 28 16 12 9 1
Values represent the percentage of patients for whom the particular component of the questionnaire was considered important by each clinician. *The percentage response is calculated as the number of patients for whom the item or information was considered relevant to the diagnosis divided by the number of patients for whom the clinician documented notes. The number of patients for whom each clinician documented notes are as follows: physician 1, 34 patients; physician 2, 34 patients; physician 3, 48 patients; physician 4, 8 patients.
CI, 0.21–0.81]) and with BHR-reversibility (ICC, 0.75 [95% CI, 0.28–0.84]). Because the CSGA algorithm is based on specific information collected from the CSGA questionnaire (symptomatology, physician diagnosis, and smoking history; see the “Methods” section), we then examined the relationship between the asthma diagnosis derived from the patient interviews (phase I) and that derived from the CSGA respiratory questionnaire. The average physicians’ score derived from reviewing the blinded questionnaire (phase II) showed excellent agreement with the average score derived from the patient interview (phase I), with an ICC of 0.92 (95% CI, 0.76–0.93; Table III). The ICC between phase I and phases III and IV closely matched that between phases I and II; in fact, there was virtually no change in the ICCs with the addition of the objective data from phases III and IV to the questionnaire. The descriptive data taken from the interview worksheets, in which each clinician noted which questions, responses, or data were most influential in making their diagnosis of asthma, are summarized in Table IV. The most important elements in the CSGA respiratory questionnaire for determining the likelihood that a patient has asthma were the questions related to a history of asthma symptoms. For the 4 clinicians, the responses to these questions influenced their asthma certainty score in 38% to 65% (mean, 56%) of the patients reviewed. Also of great importance in influencing the physicians’ scores were the questions associated with the efficacy of asthma medications; these questions were influential in over one third (36%) of the patients reviewed.
DISCUSSION As the primary goal of this study, we established that the CSGA algorithm is a valid tool for which the diagnosis of asthma can be made at an acceptable level of certainty. Both epidemiologic and genetic studies of asthma are dependent on the accurate classification of subjects as asthmatic versus nonasthmatic. This dilemma was
recently discussed by Britton13 and Postma et al14 at “the Workshop on The Genetics of Asthma: Methodological Approaches” (Brussels, 1998). It was further agreed upon that the most contentious area of disagreement with regard to the asthma phenotype is bronchial challenge testing.15 In our study we demonstrated a high degree of agreement among an expert panel of physicians in phases I through IV (ICCs of 0.98, 0.95, 0.95, and 0.89, respectively). Furthermore, when that same panel of physicians is provided with the same data collected as part of the CSGA (eg, CSGA questionnaire and pulmonary function testing), their level of certainty for the diagnosis of asthma closely matches the categorization derived from the CSGA algorithm (ICC, 0.75), thus demonstrating that the CSGA algorithm is a valid tool (eg, it measures what it is supposed to measure) for diagnosing asthma. The feasibility of performing pulmonary function testing, especially testing for BHR, in large populations is problematic and potentially represents one of the most limiting factors in conducting population studies of asthma. After determining high interjudge reliability (ICC, 0.98) among a panel of expert clinicians who specialize in asthma management and care, we subsequently observed that the 4 physicians, on average, had a high degree of agreement (ICC, 0.92) between the asthma scores derived from a patient interview (phase I) and the scores derived from the CSGA questionnaire data only (phase II). In the process of identifying to what degree specific objective parameters contribute in the diagnosis of asthma by clinicians who specialize in the disease, we found that the addition of objective parameters (SPT data, eosinophil counts, and pulmonary function data) added little to the agreement between interviews and questionnaires. We therefore established the usefulness of the interviewer-administered CSGA respiratory questionnaire in the diagnosis of asthma at an acceptable level of certainty, even in the absence of conventional objective data (eg, BHR). One would then envision the following approach for asthma diagnosis in future epidemiologic-genetic studies: (1) administer the CSGA questionnaire to all subjects by a trained field-
796 Barnes et al
worker; (2) organize a panel of expert clinicians; (3) provide the blinded questionnaires to each physician, including a Likert scale for scoring the likelihood that the patient has asthma; and (4) use the mean score derived from the physicians’ reviews for diagnosing asthma in the patient population. One could even go further and use only one clinician if the panel agrees (high ICC) on a random source of subjects. It has been argued that one approach in enhancing the likelihood of identifying genes that contribute to complex traits is to narrow the definition of a disease to one that is as Mendelian in nature as possible. For this reason, many genetic studies of asthma have sought evidence for linkage to certain regions of the human genome by using intermediate asthma-related phenotypes. The most commonly used measures of asthma in epidemiologic and clinical studies are symptom data, specifically cough, wheeze, shortness of breath, chest tightness, and sputum production.16 The CSGA respiratory questionnaire includes specific symptom questions but also focuses on specific triggers, although this particular component of the questionnaire has yet to be validated. Descriptive data recorded by the clinicians indicating which parameters of the questionnaire were most useful in determining their diagnosis suggested that the questions related to a history of asthma symptoms were the most important (Table IV). An unexpected finding was that objective test data, especially pulmonary function testing data (phase IV), did not improve the degree of agreement with the interview-generated asthma certainty score when compared with the questionnaire alone. In other words, what influenced these specialists the most was the information from the questionnaire rather than the objective data. One may argue that because the agreement between the interview and the questionnaire alone (phase I vs phase II) was excellent (ICC, 0.92), there was not much unexplained variance left for improvement. This is supported by our primary observation, when the physicians’ average asthma certainty score was evaluated for agreement against the CSGA algorithm-based asthma categorization. In this analysis we observed progressive improvement in the agreement between the physicians’ opinion and the CSGA outcome from the interview (ICC, 0.63) to the questionnaire plus all objective data (ICC, 0.75). Even in this analysis, however, it is evident that the physician interview alone or the physician interpretation of each patient’s responses to the questionnaire were by far the most important factors in determining agreement with the CSGA algorithm–based asthma certainty categorization. Findings from this study illustrate that the CSGA res-
J ALLERGY CLIN IMMUNOL OCTOBER 1999
piratory questionnaire is a useful instrument for identifying subjects with asthma and that historical elements obtained through the CSGA respiratory questionnaire are equally as good in the diagnosis of asthma as objective parameters (SPT data, eosinophil counts, and pulmonary function data). The methods designed in this study have already proven beneficial in genetics studies of asthma, but the practicality and usefulness of these tools in epidemiologic studies have not yet been investigated. We thank the families in Baltimore for their generous participation in this study. We thank the members of the Collaborative Study on the Genetics of Asthma for their kind contributions to this project and development of the CSGA Respiratory Questionnaire. We also thank Maria Stockton and Kimberly Donnelly for assistance with the clinical evaluations. Finally, we are grateful for the useful comments of 3 anonymous reviewers. REFERENCES 1. The Collaborative Study on the Genetics of Asthma. The Collaborative Study on the Genetics of Asthma: A genome-wide search for asthma susceptibility loci in ethnically diverse populations. Nat Genet 1997;15:38992. 2. Panhuysen CIM, Bleecker ER, Koeter GH, Meyers DA, Postma DS. Characterization of obstructive airway disease in family members of probands with asthma. Am J Respir Crit Care Med 1998;157:1734-42. 3. Ferris BG. Epidemiology standardization project. II. Recommended respiratory disease questionnaires for use with adults and children in epidemiologic research. Am Rev Respir Dis 1978;118(Suppl):7-53. 4. Horowitz, E, Joyner D, Guydon L, Malveaux F, Philip G. Peebles S, et al. Validation of an asthma severity questionnaire for adolescents. J Allergy Clin Immunol 1995;95:269. 5. Blumenthal MN, Banks-Schlegel S, Bleecker ER, Marsh DG, Ober C. Collaborative studies on the genetics of asthma—National Heart, Lung, and Blood Institute. Clin Exp Allergy 1995;25:29-32. 6. Freidhoff LR, Meyers DA, Bias WB, Chase GA, Hussain R, Marsh DG. A genetic-epidemiologic study of human immune responsiveness to allergens in an industrial population. I. Epidemiology of reported allergy and skin-test positivity. Am J Med Genet 1981;9:323-40. 7. American Thoracic Society. Standardization of spirometry. 1994 update. Am J Respir Crit Care Med 1995;152:1107-36. 8. Chai H, Farr RS, Froehlich LA, Mathison DA, McLean JA, Rosenthal RR, et al. Antigen and methacholine challenge in children with asthma. J Allergy Clin Immunol 1975;56:323-7. 9. Jeyaseelan L, Rao PSS. Statistical measures of clinical agreement. Natl J India 1992;5:286-90. 10. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1:307-10. 11. Bartko JJ. The intraclass correlation coefficient as a measure of reliability. Psychol Rep 1966;19:3-11. 12. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979;86:420-8. 13. Britton J. Symptoms and objective measures to define the asthma phenotype. Clin Exp Allergy 1998;28(Suppl 1):2-7. 14. Postma DS, Meijer GG, Koppelman GH. Definition of asthma: possible approaches in genetic studies. Clin Exp Allergy 1998;28(Suppl 1):62-4. 15. Weiss ST, Paré P. Report of the Working Group on phenotype approaches. Clin Exp Allergy 1998;28(Suppl 1):112. 16. O’Connor GT, Weiss ST. Clinical and symptom measures. Am J Respir Crit Care Med 1994;149:S21-8.
Background: Defining the phenotype is critical for investigating the genetic etiology of asthma. As part of the Collaborative Study on the Genetics of Asthma (CSGA), the primary objective of which is to identify asthma susceptibility loci, an algorithm was designed to determine diagnoses of definite asthma, probable asthma, less than probable asthma, or no asthma. A respiratory questionnaire was designed to assist in the process of characterizing the asthma phenotype. Objective: This study was designed to determine the validity of the CSGA algorithm for the diagnosis of asthma, to determine agreement in assessing an asthma diagnosis between the information obtained by the CSGA questionnaire versus a patient interview by a panel of specialist physicians, and to determine the degree to which objective tests would alter the questionnaire-based certainty of asthma diagnosis. Methods: An expert panel of asthma clinicians (n = 4) indicated to what degree they were certain that a subject (n = 48) had asthma as determined by using a 6-point Likert scale based on a 20-minute interview (phase I), a review of the CSGA questionnaire (phase II), a review of the questionnaire plus skin test and peripheral blood eosinophilia data (phase III), and a review of phase III information plus pulmonary data (spirometry and methacholine-reversibility testing; IV). Intraclass correlation coefficients (ICCs) were calculated between the physicians’ interpretation of the likelihood of asthma based on the information they received during each of the phases and between the CSGA algorithm and each of the phases. Results: Interjudge reliability with regard to the degree of certainty with which an asthma diagnosis could be made by interview was excellent (ICC, 98; 95% confidence intervals [95% CIs], 0.87–0.99). We also found that the agreement between the
From athe Division of Clinical Immunology, Department of Medicine, bthe Department of Epidemiology, Johns Hopkins School of Hygiene and Public Health, and cthe Division of Pulmonary Medicine, Department of Medicine, The Johns Hopkins University School of Medicine, Baltimore. Supported by US Public Health Service Grant HL49612-06. †Dr. David G. Marsh was actively involved in the design and implementation of this study. He passed away on March 29, 1998, after a heroic battle with cancer. We dedicate this work and manuscript to his memory. Received for publication Feb 26, 1999; revised July 13, 1999; accepted for publication July 13, 1999. Reprint requests: Kathleen C. Barnes, PhD, Johns Hopkins Asthma & Allergy Center, 5501 Hopkins Bayview Circle, Baltimore, MD 21224. Copyright © 1999 by Mosby, Inc. 0091-6749/99 $8.00 + 0 1/1/101510
physicians’ interview with the patients (phase I) and the CSGA algorithm was good and at least as good with the addition of the CSGA questionnaire data and objective data (ICC, 0.65–0.75). Good agreement was also observed between the average certainty score from the interview and the CSGA questionnaire (ICC, 92; 95% CI, 0.76–0.93), and ICCs determining the agreement on asthma diagnosis between phase I and phases III and IV, in which objective data were introduced, did not change from the ICCs comparing phase I with phase II (ICC of 0.93 [95% CI, 0.79–0.96] and ICC of 0.91 [95% CI 0.73–0.95], respectively). Conclusion: We conclude that the CSGA algorithm is a valid tool for which the diagnosis of asthma can be made at an acceptable level of certainty and that the CSGA questionnaire, interpreted by an asthma specialist, is a useful tool for the diagnosis of asthma in clinical or epidemiologic studies. (J Allergy Clin Immunol 1999;104:791-6.) Key words: Asthma questionnaire, Collaborative Study on the Genetics of Asthma, interjudge reliability, intraclass correlation coefficient
The Collaborative Study on the Genetics of Asthma (CSGA), funded by the National Heart, Lung, and Blood Institute, is an ongoing study with the aim of identifying the genes controlling asthma. Data have been combined from 4 centers, and a 10 cM genome–wide screen in 261 affected sib pairs and their parents, representing 3 racialethnic groups (Caucasian, African American, and Hispanic), has been performed.1 Before attempting to dissect the genetic basis of asthma, investigators of the CSGA felt that it was necessary to establish a process by which the diagnosis of asthma could be achieved to the degree that the risk of false-positive results could be minimized. To accomplish this goal, an asthma algorithm designed by Panhuysen et al2 was modified for the CSGA to classify each proband and all family members participating in the study. The CSGA algorithm classifies subjects essentially into 4 categories: (1) definite asthma, (2) probable asthma, (3) less than probable asthma, (4) no asthma (eg, no evidence of asthma or other pulmonary diseases). Individuals with a greater than 5 pack-year smoking history are not considered in the algorithm. The algorithm is generated from asthma symptomatology, lung function data, a diagnostic history, and history of smoking, and there791
792 Barnes et al
Abbreviations used ATS: American Thoracic Society BHR: Bronchial hyperresponsiveness CSGA: Collaborative Study on the Genetics of Asthma ICC: Intraclass correlation coefficient SPT: Skin prick test
fore it relies on precise collection of subjective data (eg, symptoms and medical history) and pulmonary function testing data, including a measure of bronchial hyperresponsiveness (BHR), testing of reversibility of airway obstruction with a bronchodilator, or both. To collect the subjective data in a standardized fashion, investigators of the CSGA modified the American Thoracic Society (ATS)–Division of Lung Disease questionnaire3 to produce an asthma-specific instrument that could be used in combination with objective data to diagnose asthma. The CSGA respiratory questionnaire assesses frequency and duration of asthma and allergy symptoms. Modifications from the original ATS questionnaire included a change from a self-administered to an interviewer-administered format and a greater focus on the characteristics of asthma by deleting a number of questions about other chronic respiratory diseases; however, appropriate questions remained in the modified CSGA respiratory questionnaire to discriminate between asthma and conflicting diagnoses (eg, chronic obstructive pulmonary disease and congestive heart failure). The final asthma diagnosis protocol used in the CSGA included a number of criteria obtained from the questionnaire combined with a positive methacholine bronchial provocation. We designed a study to validate the CSGA algorithm for the diagnosis of asthma. The protocol we designed was based on a validation study of an asthma severity questionnaire that our group has recently performed.4 In brief, we analyzed the agreement between an asthma certainty score derived from a panel of expert clinicians in asthma and pulmonary medicine after they interviewed individuals with and without evidence of asthma, a score derived from the same physicians when they independently evaluated the answers of the same subjects to the CSGA questionnaire with and without objective data, and the asthma categorization according to the CSGA algorithm. In addition to our goal to validate the CSGA algorithm as a tool for the diagnosis of asthma, we also wanted to compare the degree of agreement on the likelihood of an asthma diagnosis that a group of specialist physicians would demonstrate between interviewing subjects and reviewing their responses to the CSGA questionnaire and to test whether objective information added to the results of the questionnaire would change the physicians’ original opinion on asthma diagnosis.
METHODS Study design Fifty-two subjects were selected from the Johns Hopkins University (Baltimore, Md) center as part of the CSGA, which has been
J ALLERGY CLIN IMMUNOL OCTOBER 1999
previously described.1,5 All subjects had previously visited the Johns Hopkins Asthma and Allergy Center and undergone full clinical evaluations according to the CSGA guidelines, which included the respiratory questionnaire, medical evaluation, bronchial reactivity or reversibility tests, skin prick tests (SPT), and blood collection for total IgE, peripheral blood eosinophil counts, and DNA typing, between November 1993 and August 1995. These subjects were asked to return for an interview with the panel of physicians between October and November 1995. Before participation, all subjects gave verbal and written consent as approved by the Johns Hopkins Bayview Hospital Institutional Review Board. The data collected from the subjects were used in the 4 phases of the study as follows: phase I, assignment of a physician-diagnosed asthma likelihood score based on a 20-minute interview; phase II, assignment of a physician-diagnosed asthma likelihood score based on the subjects’ responses to the CSGA respiratory questionnaire; phase III, assignment of a physician-diagnosed asthma likelihood score based on data from the CSGA respiratory questionnaire, SPTs, and peripheral blood eosinophil counts; and phase IV, assignment of a physician-diagnosed asthma likelihood score based on data from the CSGA respiratory questionnaire, SPTs, peripheral blood eosinophil counts, and pulmonary function tests. For phases II through IV, each physician received a packet containing the clinical data relevant to the particular phase for each subject. Random codes were assigned to all data forms to conceal subject identification. For each phase, each physician was asked to score the likelihood that the subject had asthma on the basis of a 6-point Likert scale (1, no asthma; 6, definite asthma). Physicians were also asked to complete an interview worksheet in which they recorded notes regarding the patient’s history and described which questions, responses, or data were most influential in making their diagnosis. The physicians were asked to refrain from discussing their decisions with each other and with any other clinicians and to make their own independent diagnoses. Physicians evaluated the packets for each phase, documented their scores within a 2-week period, and returned the information to the study coordinator.
Subject selection Complete clinical data were available for 277 subjects enrolled in the CSGA at the Johns Hopkins University School of Medicine site before August 1995. To recruit a range of asthmatic subjects classified from no asthma to definite asthma, a preliminary 5-category classification was created. Briefly, this classification factored the presence or absence of the following criteria: (1) the number of asthma-like symptoms (wheeze, cough, shortness of breath) experienced; (2) nonspecific bronchial reactivity to methacholine or a positive reversibility test after inhalation of a β-agonist (albuterol); and (3) the diagnosis of asthma by a clinician. Adjustments for cigarette smoking were made based on the reported number of pack-years. Because the diagnosis of asthma is more difficult in subjects who do not fulfill all of the conventional criteria (eg, hallmark symptoms and BHR), more subjects were recruited from the 3 middle asthma algorithm categories (n = 12 for each category) than from the 2 extreme (no asthma and definite asthma) categories (n = 8 for each category). A randomized list of subjects was generated for each of the 5 categories, and subjects (n = 52) were recruited by telephone. If a subject in a given category refused to participate, the study coordinator selected the next subject from the randomized list within that category. Four subjects failed to complete the validation study (2 from category 0 and 2 from category 3), resulting in a total of 48 participants.
The CSGA algorithm As part of the classification process for asthma, an algorithm described by Panhuysen et al2 was modified and applied to the
Barnes et al 793
J ALLERGY CLIN IMMUNOL VOLUME 104, NUMBER 4, PART 1
TABLE I. The CSGA asthma algorithm
Diagnosis
Pulmonary function testing*
No. of symptoms reported (wheeze, cough, shortness of breath)
Previous physician diagnosis or use of asthma medications
∆FEV1 ≥20% with < 25 mg ≥15% REV
≥2 ≥2
Optional Optional
∆FEV1 ≥20% with ≤10 mg ∆FEV1 <15% with 25 mg ∆FEV1 <15% with 25 mg ≥15% REV
1 ≥2 ≤1 1
Optional Yes Yes Optional
∆FEV1 ≥20% with ≤10 mg ∆FEV1 ≥20% with >10-25 mg ∆FEV1 ≥15% and <20% with ≤25 mg ≥15% REV
0 ≤1 ≥2 0
Optional Optional Optional Optional
∆FEV1 ≥15% and <20% with ≤25 mg ∆FEV1 <15% with 25 mg
≤1 ≥0
Optional No
Asthma or Probable asthma or or Less than probable asthma or or No asthma or
Adapted from Panhuysen et al.2 ∆FEV1, Change in forced expiratory volume; REV, reversibility of airway obstruction defined as the percentage of increase in FEV1 predicted after use of a bronchodilator. *Methacholine provocation or bronchodilator reversibility.
CSGA families (Table I). Briefly, the CSGA algorithm classifies subjects as having definite asthma, probable asthma, less than probable asthma, and no asthma. The “less than probable asthma” group includes subjects who are unlikely to have asthma but for whom a definitive diagnosis is not possible because of confounding factors (eg, a history of chronic obstructive pulmonary disease and unclassifiable airway disease because of smoking). The algorithm is based on respiratory symptoms, a prior diagnosis of asthma by a physician, smoking history (recorded in the CSGA questionnaire), and the presence or absence of BHR, bronchodilator reversibility of airway obstruction, or both.
interviewed in the presence of a parent or legal guardian. Recorded information included symptoms of respiratory disease, including asthma (cough, wheeze, shortness of breath, and chest tightness), triggers that provoked asthma symptoms, symptoms and triggers related to allergic rhinitis and eczema, previous diagnoses by a physician, asthma medication use (past and current), asthma-related hospitalizations and emergency department visits, and smoking history. Using the 6-point Likert scale, each physician produced an asthma certainty score for each patient on the basis of a review of the questionnaire.
Phases III and IV
Phase I: Patient interviews Using the methodology designed by Horowitz et al,4 2 expert clinicians from the Division of Pulmonary and Critical Care Medicine and 2 expert clinicians from the Division of Clinical Immunology at The Johns Hopkins Asthma and Allergy Center were selected to participate on the physician panel. The panel of physicians interviewed each patient for 20 minutes. Subjects under 18 years of age (n = 12) were accompanied by a parent or legal guardian. In the event that a physician had previously examined or had prior medical knowledge about the patient (2 subjects), the physician refrained from evaluating the patient. Physicians were allowed to ask any question necessary to determine the likelihood that a subject had asthma according to their own definition of the disease; no description of asthma or guidelines for diagnosing asthma were discussed with the physicians before the study. Evaluations were based strictly on historical data; no physical examination was performed, and there was no access to any objective data. Each physician produced an independent score (eg, certainty of asthma; 0 [no asthma] to 5 [definite asthma]) for each patient on the basis of the abovedescribed 6-point Likert scale.
Phase II: Respiratory questionnaire The CSGA respiratory questionnaire had been previously administered to all subjects by trained technicians during the initial CSGA clinical evaluation. Subjects under 18 years of age had been
SPTs. Semiquantitative SPT procedures6 were performed on all subjects by using bifurcated smallpox vaccination needles (Wyeth Laboratories, Marietta, Pa) and standardized glycerinated extracts (ALK Laboratories Hørsholm, Denmark, and Greer Laboratories, Lenoir, NC) of 30 common inhalant allergens (arthropod-animal, pollen, and mold). Positive reactions were recorded by tracing the perimeter of the wheal 15 minutes after puncturing the skin and transferring the tracing with transparent surgical tape (Transpore; 3M Co, Minneapolis, Minn) to paper labeled for each extract, creating a permanent record of the skin test reactions to each extract for all subjects. Eosinophilia. Plasma was collected in EDTA tubes from each patient at the time of clinical evaluation. Routine hematologic testing with differential was performed at the Laboratory Corporation of America (Baltimore, Md) for the purpose of measuring eosinophils by using the automated STKS (Coulter Inc, Fort Lauderdale, Fla). For an absolute value, eosinophils were counted manually on a slide after automatic staining (Hematek; Miles Diagnostics, Kankakee, Ill) with 22% wt/vol Wright and 0.50% wt/vol Giemsa stain in 220 mL of methanol, 540 mL of phosphate buffer (pH 6.8), and 990 mL of methyl alcohol solution in 10% H2O (Fisher Science, Pittsburgh, Pa). Results were presented as a per-
794 Barnes et al
J ALLERGY CLIN IMMUNOL OCTOBER 1999
TABLE II. ICCs and 95% CIs between asthma categorization as defined by the CSGA algorithm and the mean physician scores Phase
ICC
95% CI
I II III IV
0.63 0.65 0.69 0.75
0.15-0.77 0.17-0.78 0.21-0.81 0.28-0.84
TABLE III. ICCs and 95% CIs between the mean physician scores generated by the patient interview (phase I) and the mean physician scores generated in the other phases Phase
ICC
95% CI
II III IV
0.92 0.93 0.91
0.76-0.93 0.79-0.96 0.73-0.95
Physician scores were generated by the patient interview (phase I); a review of the questionnaire (phase II); a review of the questionnaire, SPT data, and eosinophil counts (phase III); and a review of the questionnaire, SPT data, eosinophil counts, and complete pulmonary data (spirometry and methacholine-reversibility testing; phase IV).
The mean physician scores were generated from review of the questionnaire (phase II); review of the questionnaire, SPT data, and eosinophil counts (phase III); and review of the questionnaire, SPT data, eosinophil counts, and complete pulmonary data (spirometry and methacholine-reversibility testing; phase IV).
centage of white blood cells or as an absolute number (value × 10–3 µL). Pulmonary function testing. Baseline pulmonary function (forced vital capacity [FVC], FEV1, FEV1/FVC, peak expiratory flow rate, and FEF25-75) was measured with a KOKO (Pulmonary Data Services, Inc, Louisville, Colo) pneumotach connected to a desktop or laptop computer in accordance with the ATS standards of spirometry.7 Subjects performed at least 3 forced expiratory maneuvers until 2 repeatable FEV1 values were obtained (within 10% of each other), and the highest value was recorded. If the patient’s FEV1 was less than 70% of the predicted value, a reversibility test was performed by delivering 2 inhalations of a β-agonist (albuterol) by means of a metered-dose inhaler and measuring FEV1 after 10 minutes. A positive reversibility test response was recorded if the patient’s FEV1 was increased by 15% from baseline. If the patient’s FEV1 was 70% or greater of the predicted value, a quantitative bronchoprovocation challenge was conducted. Methacholine challenge was performed as described by Chai et al.8 A breath-actuated dosimeter was used to deliver a diluent control and successive half-log incremental concentrations of methacholine (0.025, 0.075, 0.25, 0.75, 2.5, 7.5, and 25 mg/mL). The provocation ended after reaching a 20% reduction in FEV1 from the postdiluent value or after the highest concentration of methacholine had been inhaled. The PD20 and the PC20 were determined by interpolation of the dose-response curve (log PD20 cumulative breath units = 1 breath of a 1 mg/mL concentration and PC20 = mg/mL). Patients were asked to refrain from taking β-agonists and theophylline 24 hours before testing. For phase III, physicians were given blinded packages, including the following for each subject: (1) the questionnaires (as in phase II), (2) the record of SPT reactions to each of the 30 extracts, and (3) eosinophil counts both as a percentage of white blood cells and as an absolute number. For phase IV, physicians were given blinded packages with all material for phase III plus a blinded summary of pulmonary testing (FEV1/FVC, FVC percentage predicted, FEV1 percentage predicted, and PC20). As with phases I and II, each physician scored each patient’s phase III and phase IV packages by using the 6-point Likert scale.
of the information they received during each of the 4 phases of the study and the diagnosis of our study subjects according to the algorithm. The Likert scale scores generated by the 4 physicians in each phase were averaged, and the average was then used in the calculation of the ICC. Calculation of ICC was based on a 2-way ANOVA model wherein the between-targets (subjects) mean square (variance of interest) and the residual error mean square were obtained. This statistical approach is appropriate because each target is rated by each judge, and the intraclass correlation can be derived by the following formula as described by Shrout and Fleiss12: BMS – EMS/BMS, where BMS is the between-targets mean squares and EMS is the residual or error of the mean square. A 95% confidence interval (CI) was calculated for each ICC by using the same methodology.12 To determine the degree of agreement between the physicians’ interpretation of the likelihood of asthma on the basis of the information they received during the interview and the questionnaire and from the objective data, ICCs between phase I and phases II through IV were calculated by using the mean asthma certainty scores of the clinicians for each phase.
Data analysis Similarity of diagnosis among the 4 physicians (interjudge reliability) in their rating of the 48 patients was assessed by calculating the intraclass correlation coefficient (ICC), which is the ratio of the variance of interest over the sum of the variance of interest plus the error.9-12 To validate the CSGA algorithm, ICCs were calculated between the physicians’ interpretation of the likelihood of asthma on the basis
RESULTS Complete data on questionnaire, SPT, eosinophilia, spirometry, and PD20 reversibility data were obtained from 48 subjects (35% male, 35% African American, and 65% Caucasian). The ICC between the physician scores generated by the patient interview (phase I) was 0.98 (95% CI, 0.86–0.99), indicating that only 2% of the variability in the measurement of asthma certainty was caused by differences between the judges (physicians), whereas the remaining 98% was caused by differences between the patients. ICCs among the physician scores generated in phases II through IV were equally high (0.95 [95% CI, 0.71–0.97], 0.95 [95% CI, 0.73–0.97], and 0.89 [95% CI, 0.52–0.93], respectively). These data allowed us, at each phase, to use the average physician score as an independent variable. To validate the CSGA approach of the diagnosis of asthma, we compared the average scores generated by 4 physicians on the basis of the 20-minute interview with the category of asthma certainty derived by the CSGA algorithm. Table II indicates that the agreement between the physicians’ interview with the patients (phase I) and the CSGA algorithm was good and that the ICC was at least as good with the addition of objective information, especially with SPT-eosinophil counts (ICC, 0.69 [95%
Barnes et al 795
J ALLERGY CLIN IMMUNOL VOLUME 104, NUMBER 4, PART 1
TABLE IV. Components of the CSGA respiratory questionnaire that influenced the clinicians’ diagnosis of asthma (phase II) Percentage response (phase II)* Item or information
Physician 1
Physician 2
Physician 3
62 32 18 53 12 6 12 0
65 44 24 15 32 9 15 3
58 56 38 17 19 31 8 2
History of asthma symptoms Relief by asthma medications Symptoms associated with triggers History of allergies History of asthma Asthma symptoms associated with upper respiratory infection History of other pulmonary diseases Emergency department visits or hospitalizations
Physician 4
38 13 25 25 0 0 0 0
Mean
56 36 26 28 16 12 9 1
Values represent the percentage of patients for whom the particular component of the questionnaire was considered important by each clinician. *The percentage response is calculated as the number of patients for whom the item or information was considered relevant to the diagnosis divided by the number of patients for whom the clinician documented notes. The number of patients for whom each clinician documented notes are as follows: physician 1, 34 patients; physician 2, 34 patients; physician 3, 48 patients; physician 4, 8 patients.
CI, 0.21–0.81]) and with BHR-reversibility (ICC, 0.75 [95% CI, 0.28–0.84]). Because the CSGA algorithm is based on specific information collected from the CSGA questionnaire (symptomatology, physician diagnosis, and smoking history; see the “Methods” section), we then examined the relationship between the asthma diagnosis derived from the patient interviews (phase I) and that derived from the CSGA respiratory questionnaire. The average physicians’ score derived from reviewing the blinded questionnaire (phase II) showed excellent agreement with the average score derived from the patient interview (phase I), with an ICC of 0.92 (95% CI, 0.76–0.93; Table III). The ICC between phase I and phases III and IV closely matched that between phases I and II; in fact, there was virtually no change in the ICCs with the addition of the objective data from phases III and IV to the questionnaire. The descriptive data taken from the interview worksheets, in which each clinician noted which questions, responses, or data were most influential in making their diagnosis of asthma, are summarized in Table IV. The most important elements in the CSGA respiratory questionnaire for determining the likelihood that a patient has asthma were the questions related to a history of asthma symptoms. For the 4 clinicians, the responses to these questions influenced their asthma certainty score in 38% to 65% (mean, 56%) of the patients reviewed. Also of great importance in influencing the physicians’ scores were the questions associated with the efficacy of asthma medications; these questions were influential in over one third (36%) of the patients reviewed.
DISCUSSION As the primary goal of this study, we established that the CSGA algorithm is a valid tool for which the diagnosis of asthma can be made at an acceptable level of certainty. Both epidemiologic and genetic studies of asthma are dependent on the accurate classification of subjects as asthmatic versus nonasthmatic. This dilemma was
recently discussed by Britton13 and Postma et al14 at “the Workshop on The Genetics of Asthma: Methodological Approaches” (Brussels, 1998). It was further agreed upon that the most contentious area of disagreement with regard to the asthma phenotype is bronchial challenge testing.15 In our study we demonstrated a high degree of agreement among an expert panel of physicians in phases I through IV (ICCs of 0.98, 0.95, 0.95, and 0.89, respectively). Furthermore, when that same panel of physicians is provided with the same data collected as part of the CSGA (eg, CSGA questionnaire and pulmonary function testing), their level of certainty for the diagnosis of asthma closely matches the categorization derived from the CSGA algorithm (ICC, 0.75), thus demonstrating that the CSGA algorithm is a valid tool (eg, it measures what it is supposed to measure) for diagnosing asthma. The feasibility of performing pulmonary function testing, especially testing for BHR, in large populations is problematic and potentially represents one of the most limiting factors in conducting population studies of asthma. After determining high interjudge reliability (ICC, 0.98) among a panel of expert clinicians who specialize in asthma management and care, we subsequently observed that the 4 physicians, on average, had a high degree of agreement (ICC, 0.92) between the asthma scores derived from a patient interview (phase I) and the scores derived from the CSGA questionnaire data only (phase II). In the process of identifying to what degree specific objective parameters contribute in the diagnosis of asthma by clinicians who specialize in the disease, we found that the addition of objective parameters (SPT data, eosinophil counts, and pulmonary function data) added little to the agreement between interviews and questionnaires. We therefore established the usefulness of the interviewer-administered CSGA respiratory questionnaire in the diagnosis of asthma at an acceptable level of certainty, even in the absence of conventional objective data (eg, BHR). One would then envision the following approach for asthma diagnosis in future epidemiologic-genetic studies: (1) administer the CSGA questionnaire to all subjects by a trained field-
796 Barnes et al
worker; (2) organize a panel of expert clinicians; (3) provide the blinded questionnaires to each physician, including a Likert scale for scoring the likelihood that the patient has asthma; and (4) use the mean score derived from the physicians’ reviews for diagnosing asthma in the patient population. One could even go further and use only one clinician if the panel agrees (high ICC) on a random source of subjects. It has been argued that one approach in enhancing the likelihood of identifying genes that contribute to complex traits is to narrow the definition of a disease to one that is as Mendelian in nature as possible. For this reason, many genetic studies of asthma have sought evidence for linkage to certain regions of the human genome by using intermediate asthma-related phenotypes. The most commonly used measures of asthma in epidemiologic and clinical studies are symptom data, specifically cough, wheeze, shortness of breath, chest tightness, and sputum production.16 The CSGA respiratory questionnaire includes specific symptom questions but also focuses on specific triggers, although this particular component of the questionnaire has yet to be validated. Descriptive data recorded by the clinicians indicating which parameters of the questionnaire were most useful in determining their diagnosis suggested that the questions related to a history of asthma symptoms were the most important (Table IV). An unexpected finding was that objective test data, especially pulmonary function testing data (phase IV), did not improve the degree of agreement with the interview-generated asthma certainty score when compared with the questionnaire alone. In other words, what influenced these specialists the most was the information from the questionnaire rather than the objective data. One may argue that because the agreement between the interview and the questionnaire alone (phase I vs phase II) was excellent (ICC, 0.92), there was not much unexplained variance left for improvement. This is supported by our primary observation, when the physicians’ average asthma certainty score was evaluated for agreement against the CSGA algorithm-based asthma categorization. In this analysis we observed progressive improvement in the agreement between the physicians’ opinion and the CSGA outcome from the interview (ICC, 0.63) to the questionnaire plus all objective data (ICC, 0.75). Even in this analysis, however, it is evident that the physician interview alone or the physician interpretation of each patient’s responses to the questionnaire were by far the most important factors in determining agreement with the CSGA algorithm–based asthma certainty categorization. Findings from this study illustrate that the CSGA res-
J ALLERGY CLIN IMMUNOL OCTOBER 1999
piratory questionnaire is a useful instrument for identifying subjects with asthma and that historical elements obtained through the CSGA respiratory questionnaire are equally as good in the diagnosis of asthma as objective parameters (SPT data, eosinophil counts, and pulmonary function data). The methods designed in this study have already proven beneficial in genetics studies of asthma, but the practicality and usefulness of these tools in epidemiologic studies have not yet been investigated. We thank the families in Baltimore for their generous participation in this study. We thank the members of the Collaborative Study on the Genetics of Asthma for their kind contributions to this project and development of the CSGA Respiratory Questionnaire. We also thank Maria Stockton and Kimberly Donnelly for assistance with the clinical evaluations. Finally, we are grateful for the useful comments of 3 anonymous reviewers. REFERENCES 1. The Collaborative Study on the Genetics of Asthma. The Collaborative Study on the Genetics of Asthma: A genome-wide search for asthma susceptibility loci in ethnically diverse populations. Nat Genet 1997;15:38992. 2. Panhuysen CIM, Bleecker ER, Koeter GH, Meyers DA, Postma DS. Characterization of obstructive airway disease in family members of probands with asthma. Am J Respir Crit Care Med 1998;157:1734-42. 3. Ferris BG. Epidemiology standardization project. II. Recommended respiratory disease questionnaires for use with adults and children in epidemiologic research. Am Rev Respir Dis 1978;118(Suppl):7-53. 4. Horowitz, E, Joyner D, Guydon L, Malveaux F, Philip G. Peebles S, et al. Validation of an asthma severity questionnaire for adolescents. J Allergy Clin Immunol 1995;95:269. 5. Blumenthal MN, Banks-Schlegel S, Bleecker ER, Marsh DG, Ober C. Collaborative studies on the genetics of asthma—National Heart, Lung, and Blood Institute. Clin Exp Allergy 1995;25:29-32. 6. Freidhoff LR, Meyers DA, Bias WB, Chase GA, Hussain R, Marsh DG. A genetic-epidemiologic study of human immune responsiveness to allergens in an industrial population. I. Epidemiology of reported allergy and skin-test positivity. Am J Med Genet 1981;9:323-40. 7. American Thoracic Society. Standardization of spirometry. 1994 update. Am J Respir Crit Care Med 1995;152:1107-36. 8. Chai H, Farr RS, Froehlich LA, Mathison DA, McLean JA, Rosenthal RR, et al. Antigen and methacholine challenge in children with asthma. J Allergy Clin Immunol 1975;56:323-7. 9. Jeyaseelan L, Rao PSS. Statistical measures of clinical agreement. Natl J India 1992;5:286-90. 10. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1:307-10. 11. Bartko JJ. The intraclass correlation coefficient as a measure of reliability. Psychol Rep 1966;19:3-11. 12. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979;86:420-8. 13. Britton J. Symptoms and objective measures to define the asthma phenotype. Clin Exp Allergy 1998;28(Suppl 1):2-7. 14. Postma DS, Meijer GG, Koppelman GH. Definition of asthma: possible approaches in genetic studies. Clin Exp Allergy 1998;28(Suppl 1):62-4. 15. Weiss ST, Paré P. Report of the Working Group on phenotype approaches. Clin Exp Allergy 1998;28(Suppl 1):112. 16. O’Connor GT, Weiss ST. Clinical and symptom measures. Am J Respir Crit Care Med 1994;149:S21-8.