Systematic Review and Metaanalysis

CHEST

Original Research RESPIRATORY INFECTIONS

Systematic Review and Metaanalysis Urinary Antigen Tests for Legionellosis Toshihiko Shimada, MD; Yoshinori Noguchi, MD, MPH; Jeffrey L. Jackson, MD, MPH; Jun Miyashita, MD, MPH; Yasuaki Hayashino, MD, PhD; Toru Kamiya, MD; Shin Yamazaki, PhD; Tadashi Matsumura, MD; and Shunichi Fukuhara, MD, MSc, DMSc

Background: Urinary antigen assays offer simplicity and rapidity in diagnosing Legionnaires’ disease, though studies report a range of sensitivities. We conducted a systematic review to assess the test characteristics of Legionella urinary antigen. Methods: We searched Medline, Excerpta Medica Database, and bibliographies of retrieved articles. English-language studies were used and included if the absolute number of true-positive, false-negative, true-negative, and false-positive observations were available, and the “gold standards” were described clearly. Two investigators independently reviewed articles and extracted data. Quality was assessed with the Quality Assessment for Diagnostic Accuracy Studies (QUADAS). Sensitivities and specificities were pooled using a random-effects model weighted with the inverse of the SE calculated through the Wald method. Results: Fifty articles were retrieved for detailed evaluation, and 30 met the inclusion criteria. All but two studies focused on serotype 1 Legionella. Forty assays were reported using six different methodologies, whereas 26 assays used commercial tests, and 14 assays used in-house tests. Study quality was generally low, with average QUADAS scores of 4.4 of a total of 14 points (range, 1 to 9 points). The pooled sensitivity was 0.74 (95% CI, 0.68 to 0.81), and the specificity was 0.991 (95% CI, 0.984 to 0.997). Higher quality studies had lower sensitivity, and there was evidence of publication bias. Conclusions: Legionella urinary antigen for serotype 1 appears to have excellent specificity, though modest sensitivity. However, the poor quality of the included studies and the presence of publication bias suggest an overestimation of test performance. High-quality studies are needed. (CHEST 2009; 136:1576 –1585) Abbreviations: LR ⫽ likelihood ratio; QUADAS ⫽ Quality Assessment for Diagnostic Accuracy Studies

pneumonia is increasing in incidence in L egionella the United States and other countries. In Ger-

American Legion convention in Pennsylvania in 1976.3 Whether the incidence of the disease is truly

many, Legionella pneumophila is a common cause of community-acquired pneumonia.2 Legionnaires’ disease was first described after an outbreak at an

For Commentary see page 1618

1

Manuscript received October 31, 2008; revision accepted February 18, 2009. Affiliations: From the Department of Epidemiology and Healthcare Research (Drs. Shimada, Miyashita, Hayashino, Yamazaki, and Fukuhara), Graduate School of Medicine and Public Health, Kyoto University, Kyoto, Japan; the Department of General Internal Medicine (Drs. Shimada, Miyashita, Kamiya, and Matsumura), Rakuwakai Otowa Hospital, Kyoto, Japan; the Department of General Internal Medicine (Dr. Noguchi), Nagoya Daini Red Cross Hospital, Nagoya, Japan; and the Uniformed Services University (Dr. Jackson), Bethesda MD. Funding/Support: This study was supported by a grant (No. H18-001) from the Ministry of Health, Labor and Welfare in 1576

increasing or the increased detection reflects greater awareness and the availability of newer diagnostic Japan, “Development of Clinical Research Fellowship” (Principal Investigator, Shunichi Fukuhara). Correspondence to: Toshihiko Shimada, MD, Department of Epidemiology and Healthcare Research, Graduate School of Medicine and Public Health, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto, 606-8501, Japan; e-mail: [email protected] © 2009 American College of Chest Physicians. Reproduction of this article is prohibited without written permission from the American College of Chest Physicians (www.chestjournal.org/site/ misc/reprints.xhtml). DOI: 10.1378/chest.08-2602 Original Research

tests is unclear. Before the development of rapid assays for Legionella urinary antigen, the diagnosis was made based on a constellation of clinical suspicion, and culture and serology results. It took days for results to return.4 Currently, there are a number of commercially available urinary antigen tests for Legionnaires’ disease that are reported to have high sensitivity and specificity,5 and test results can be available within 15 min. Although numerous studies have reported on the usefulness of urinary antigens, many have reported wide variance in urinary antigen test characteristics, and we are unaware of any published systematic reviews on this topic. The purpose of our review was to assess the test characteristics of Legionella urinary antigen for the diagnosis of clinical pneumonia.

Materials and Methods Search Strategy We searched Medline from 1966 to August 2008 using Medical Subject Headings and a free-text search strategy, using the terms “urinary antigens”; “antigen, urine”; “antigens, bacterial”; “Legionellosis”; “Legionella”; and “Legionnaire.” In addition, we searched Excerpta Medica Database from 1968 to August 2008, using the search strategy (“Legionnaires disease” OR “Legionella” OR “Legionellosis”) AND (“urinary antigens” OR “bacterial antigens” OR Legionella urinary antigen“). Both searches were limited to human studies and studies published in the English language. We also restricted our search to published studies and hand-searched bibliographies of retrieved articles. Two of the investigators searched independently.

in the following 14 domains: patient representativeness, selection criteria clarity, reference standard, duration between test and reference standard, verification bias, completeness of verification, consistency of verification, completeness of index, completeness of reference test descriptions, blinding of reference, index test results, similarity to practice, uninterpretable tests, and withdrawals. We performed both a scored and a component analysis. Quality was rated in duplicate but not independently. In addition, we rated whether the total number of reference standard results was equal to the number of patients.7 Disagreements were resolved by consensus. Data Synthesis and Analysis We pooled sensitivity and specificity with a random-effects model.8 Study weights were taken to be the inverse of the square of the SE, calculated using the Wald method. For studies reporting sensitivities or specificities of 1, we calculated the SE after the Agresti-Coull (Wald) adjustment9 was applied. Although we planned to pool likelihood ratios (LRs), many of the studies reported specificities of 1; therefore, rather than dropping the studies, we calculated LR⫹ and LR⫺ from the pooled sensitivities and specificities. We tested for a correlation between sensitivity and specificity using Spearman rank correlation to assess the need to pool using summary receiver operating characteristic methods10 and found no correlation (Spearman ␳, 0.39). This lack of association was confirmed by examination of the scatterplot. We explored heterogeneity using a combination of stratified analyses and metaregression.11 Although the appropriate test statistic for assessing publication bias in metaanalyses of diagnostic test metaanalysis is uncertain,12,13 we assessed publication bias using the Egger test,14 which evaluates the likelihood of missing studies based on funnel-plot asymmetry. In addition to the Egger test, we assessed publication bias using the method of Deeks et al12 and funnel-plot asymmetry by sample size and variance as suggested by Song et al.13 All statistical analyses were performed using a statistical software package (STATA, version 9.2; STATA Corp; College Station, TX).

Study Selection Studies relevant to the diagnostic value of urine antigen tests for Legionellosis were included if the following criteria were met: absolute numbers of true-positive, false-negative, true-negative, and false-positive observations were available or could be derived from the reported data; and a “gold standard” and criteria for diagnosis were described explicitly. We excluded studies if cases were diagnosed by urinary antigen testing alone and had not been confirmed by other methods.

Results Studies Identified In the first stage, we identified 68 possibly relevant articles from the Medline search. Fifty full articles

Data Extraction and Assessment of Study Quality We planned to extract the following variables: publication year; name and institution of the authors; information on the original sample source; study design; patient demographics and comorbidities; type and severity of pneumonia; serogroup of Legionellosis; clinical setting (outbreak or not); characteristics of control subjects; numbers of true-positive, false-negative, true-negative, and false-positive observations; type and brand name of the antigen test; and the reference standard used. Data were extracted and analyzed based on the number of cases, not the number of samples. For studies that partially used the urinary antigen as the reference test, we excluded cases diagnosed by urine antigen testing only. We also excluded duplicate data. We rated quality using the Quality Assessment for Diagnostic Accuracy Studies (QUADAS) method.6 The tool assesses quality www.chestjournal.org

Figure 1. Flow of studies through the retrieval and inclusion process in the metaanalysis. FN ⫽ false-negative; FP ⫽ falsepositive; PCR ⫽ polymerase chain reaction; TN ⫽ true-negtive; TP ⫽ true-positive. CHEST / 136 / 6 / DECEMBER, 2009

1577

Table 1—Characteristics of Included Studies Duration of Illness Test, Commercial Before Reference Study/Year/Country Brand/Methods Assay 15

16

17

Diederen SAS Legionella/ and Peeters/2007/ ICT the Netherlands BinaxNow/ICT

Diederen and Peeters/2006/the Netherlands

Diederen and Peeters/2006/the Netherlands

NS

LP1

19 20

21

Koide et al/2006/ Japan

99 169

3

21

86 162

2

50

87 162

0

6

89 162

0

6

91 162

4

6

87 162

0

0

40

75

0

0

40

75

0

4

40

75

0

38

279 466

6

0

1

181 262

2

Culture, serology (IIF) Culture, serology (IIF) Culture, serology (IIF, MA)

34

0

8

122 164

4

27

0

15

122 164

11

1

14

19

45

Culture, serology (IIF, MA) Serology (IIF)

10

1

15

19

45

5

1

1

77

84

8

Culture, serology 110 (unknown details), PCR, DFA LP1 Culture, serology 46 (IIF) Other SG, other Culture 0 SP (SG2, 3, 4, 10, Legionella longbeachae)

0

55

123 288

6

1

13

208 268

5

1

12

208 221

LP1

LP1

LP1

LP1, other SG, other SP

BinaxNow/ICT

LP1, other SG, other SP

Biotest/EIA

LP1, other SG, other SP

Helbig et al/2001/ United Kingdom Harrison and Doshi/2001/ United Kingdom

BinaxNow/ICT

Dominguez et al/ 2001/Spain

Bartels/EIA

Franzin and Cabodi/2000/ Italy

6

BinaxNow/ICT

Bartels/EIA

1–33 d

Binax/EIA

LP Case: 10–52 d Control: 3–63 d

Biotest/EIA 23

Socan et al/1999/ Slovenia

Binax/ELISA*

24

Harrison et al/1998/ Biotest/EIA United Kingdom

25

Kazandjian et al/ 1997/Australia

Binax/EIA Binax/EIA

LP1, other SG

“Most” LP1 ⬍ 24 h after hospital admission NS LP

Binax/EIA 22

1

LP1

2–8 d

LP

LP First day of LP1 hospital admission NS LP1, other SG

NS

QUADAS Score

98 169

SD Bioline/ICT

Binax/EIA

FP TN FN No. 12

LP1

NS

TP

1

Rapid U NS Legionella/ICT

BinaxNow/ICT

Reference Standard

Culture, PCR, 58 serology (ELISA) Culture, PCR, 64 serology (ELISA) Culture, PCR, 52 serology (ELISA) Culture, PCR, 23 serology (ELISA) Culture, PCR, 67 serology (ELISA) Culture, PCR, 65 serology (ELISA) Culture, PCR, 65 serology (ELISA) Culture, serology 35 (unknown details) Culture, serology 35 (unknown details) Culture, serology 31 (unknown details) Culture, serology 149 (IIF) Culture, serology 81 (unknown details)

LP1

Rapid U Legionella/ICT 18

Legionellosis Serogroup

5

5

5

2

6

(Continued) 1578

Original Research

Table 1—Continued Duration of Illness Test, Commercial Before Reference Study/Year/Country Brand/Methods Assay 26

Dominguez et al/ 1997/Spain

Binax/EIA

NS

Binax/RIA 27

28 29

Hackman Binax/EIA et al/1996/United States Dominguez et al/ Binax/RIA 1996/Spain Plouffe et al/1995/ Binax/RIA United States

Legionellosis Serogroup LP LP

NS

LP1

NS

LP1

Reference Standard

0

15

68 114

28

0

18

68 114

20

0

6

33

59

2

Culture, serology (IIF) Culture, serology (IIF)

25

0

16

58

99

6

38

6

30

630 704

9

Culture, serology (IIF)

9

2

8

115 134

7

Culture, serology (IIF)

76

89

64

252 481

4

Culture, serology (IIF) Culture, serology (IIF) Culture, DFA

28

0

3

93

0

13

Bernander et al/ 1994/Sweden

Binax/RIA

31

Leland and Kohler/ 1991/United States Samuel et al/1990/ United States Birtles et al/1990/ United States Aguero-Rosenfeld and Edelstein/ 1988/United States Kohler et al/1987/ United States Tang and Toma/ 1986/Canada Kohler et al/1985/ Canada Sathapatayavongs et al/1983/United States Kohler and Sathapatayavongs/ 1983/ United States Kohler and Wheat/ 1982/United States Tang et al/1982/ United States Sathapatayavongs et al/1982/United States

L-Clone/LA

In-house ELISA

NS

LP1

In-house ELISA

NS

LP1

DuPont/RIA

NS

LP1, other SG

In-house RIA

NS

In-house ELISA

NS

In-house RIA

NS

In-house LA

NS

Other SG (SG4, 10) LP1, other SG, other SP LP1, other SG, other SP LP1

Culture, serology (IIF), DFA Culture, serology (IIF), DFA Culture, serology (IIF), DFA Culture, serology (IIF), DFA

In-house RIA

NS

LP1

In-house RIA

NS

In-house RPA

NS

In-house ELISA

Before Rx: LP 6 cases 1–3 d After Rx: 33 cases After 3 d: 22 cases Unclear: 4 cases

34

35 36 37 38

39

40

41 42

QUADAS Score

31

30

33

FP TN FN No.

Culture, serology (IIF) Culture, serology (IIF) Culture, serology (IIF)

First several LP1 days after hospital admission 7 cases: LP1 0–5 d 3 cases: 6–14 d 7 cases: 30–60 d NS LP1

32

TP

8

6

39

4

27

80 200

4

0

10

346 369

4

4

0

6

100 110

4

25

0

10

251 286

4

40

0

72

347 459

4

46

1

20

160 227

4

Culture, serology (IIF), DFA

47

0

11

431 489

4

LP1

Culture, serology (IIF), DFA

26

0

8

185 219

4

LP1

Culture, serology (IIF), DFA Culture, serology (IIF), DFA

14

0

1

263 278

3

35

0

12

178 225

4

(Continued)

www.chestjournal.org

CHEST / 136 / 6 / DECEMBER, 2009

1579

Table 1—Continued Duration of Illness Test, Commercial Before Reference Study/Year/Country Brand/Methods Assay 43 44

Kohler et al/1981/ United States White et al/1981/ United States

In-house RIA In-house ELISA

1–2 d after Rx NS

Legionellosis Serogroup LP1 LP

Reference Standard Culture, serology (IIF), DFA Culture, serology (IIF), DFA

TP

FP TN FN No.

QUADAS Score

9

0

0

241 250

4

35

3

12

175 225

4

We judged serogroups of Legionellosis for inclusion in 2 ⫻ 2 tables. DFA ⫽ direct fluorescent antibody; EIA ⫽ enzyme immunoassay; ELISA ⫽ enzyme-linked immunosorbent assay; ICT ⫽ immunochromatographic test; IIF ⫽ indirect immunofluorescence; LA ⫽ latex agglutination; LP ⫽ L pneumophila; LP1 ⫽ L pneumophila serogroup 1; MA ⫽ microagglutination; NS ⫽ not stated; other SG ⫽ other serogroup; other SP ⫽ other Legionella species; RIA ⫽ radioimmunoassay; RPA ⫽ reverse passive agglutination; Rx ⫽ treatment. See Figure 1 for abbreviations not used in text. *The existence of the Binax ELISA could not be confirmed independently of this article. It may be an erroneous reference to Binax EIA.

among these were selected for detailed analysis on the basis of title or abstract; 30 articles met the inclusion criteria. Retrieval and inclusion flow is shown in Figure 1, and the study characteristics of the 30 included studies15– 44 are outlined in Table 1. Patient demographic information generally was not provided. One study29 reported that subjects were “adults,” one study23 reported that subjects were ⬎ 15 years of age, and two studies reported mean ages of 58 years28 and 60 years.26 No study provided information on pneumonia severity or subjects’ immune status. Three studies20,29,30 reported that the subjects were hospitalized, and nine studies18 –20,22,23,29,30,42,43 provided information on the duration of illness prior to patients undergoing the Legionella assay (Table 1). Two studies26,28 reported subjects to have a mixture of communityacquired and nosocomial infections, and two studies23,29 reported that all subjects had community-acquired infection. Among the 30 included studies, 40 assays were reported using six different methodologies, with 26 assays being commercial tests and 14 assays being in-house tests (Table 1). Studies originated from 10 different countries. The majority of assays were performed in the United States (n ⫽ 15); other countries were the Netherlands (n ⫽ 7), Spain (n ⫽ 5), Japan (n ⫽ 3), the United Kingdom (n ⫽ 3), Italy (n ⫽ 2), Australia (n ⫽ 2), Canada (n ⫽ 1), Slovenia (n ⫽ 1), and Sweden (n ⫽ 1). Two studies25,35 focused on the diagnosis of serogroups or species other than L pneumophila serogroup 1 and were excluded from the pooled analysis. Both found poor sensitivity (0% and 40%, respectively) but high specificity (100% and 99.5%, respectively). An additional study45 focusing on Pontiac fever was excluded because of an unclear description of the “gold standard.” A combination of serology and culture was the most commonly used reference standard, albeit with many variations in cutoff and antibodies. Two studies31,40 purely targeted outbreak cases. 1580

Assessment of Study Quality Table 1 shows that the average QUADAS score of the 30 studies was 4.4 (range, 1 to 9) of a maximum score of 14. Only two studies23,29 investigated a clinical population, as opposed to a case-controlled population, and used the same reference standards and complete verification. Three studies23,29,30 described the spectrum of patients, four studies22,23,26,28 clearly described the patient selection criteria, and one study24 reported the inclusion of a blinded interpretation of the index test. Other quality problems are listed in Table 2. Weighted Pooled Analysis Among the 40 assays reported, there were three groups of authors (Diederen and Peeters,15–17 Dominguez and colleagues,21,26,28 and Kohler and colleagues35,37,39,40,43) who published reports at different time points, with increasing numbers of patients at subsequent time points. Some of these reports were on different types of tests, but others reported16,17,28,40,43 on the use of a particular test on increasing numbers of patients over time. Because this increase was concerning for the possibility of partial duplication of data, an attempt was made to contact the authors. Authors of only one study (Diederen and Peeters16,17) from among the three groups replied, confirming that patients from the earlier studies were part of the sample for the subsequent larger ones. To be conservative and to ensure that duplication did not occur, we excluded 6 assays, yielding a final sample of 32 assays for pooling. Some of these reports15–18,20 –22,24 –26,36,38,41,42 included additional arms with different types of assays (ie, radioimmunoassay vs enzyme immunoassay). In those reports, the additional assays were included in our pooled analysis. We chose for inclusion either the most recent report or the one with the largest patient cohort. Original Research

www.chestjournal.org

CHEST / 136 / 6 / DECEMBER, 2009

1581

No No No No No No No No Yes No No No No No Yes Yes No No No No No No No No No No No No No No

Reference

15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

No No No No No No No Yes Yes No No Yes No Yes No No No No No No No No No No No No No No No No

Clearly Described Selection Criteria

Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

Reference Standard Classifies Target Correctly

*Index did not form part of reference.

Patients Represented in Clinical Practice Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

Period Between Reference and Index Tests Short Enough No No No No No No No No Yes No No No No No Yes No No No No No No No No No No No No No No No

Whole or Random Samples Receive Verification No No No No No No No No Yes No No No No No Yes Yes No No No No No No No No No No No No No No

Same Reference Regardless of Index Yes Yes Yes No Yes No No Yes No Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

Reference Independence Index* Yes Yes Yes Yes Yes No Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

Detailed Described Index Yes Yes Yes No Yes No Yes Yes Yes Yes Yes Yes No Yes Yes Yes No No No No No No No No No No No No No No

Detailed Described Reference No No No No No No No No No Yes No No No No No No No No No No No No No No No No No No No No

Blind Interpretation of Index Test

Table 2—Quality Problems According to QUADS

No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No

Blind Interpretation of Reference Test No No No No No No No No Yes No No No No No Yes No No No No No No No No No No No No No No No

Same Clinical Data Available in Practice

No No No No Yes No No No No No No No No No No No No No No No No No No No No No No No No No

Intermediate Result Report

No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No

Withdrawals Explained

0.57 (0.36-0.77) 0.53 (0.29-0.77) 0.77 (0.70-0.85) 0.71 (0.61-0.82) 0.92 (0.85-0.98) 0.83 (0.74-0.92) 0.91 (0.85-0.98) 0.61 (0.47-0.75) 0.67 (0.54-0.81) 0.64 (0.50-0.79) 0.81 (0.69-0.93) 0.44 (0.25-0.63) 0.40 (0.21-0.59) 0.77 (0.61-0.93) 0.67 (0.59-0.74) 0.99 (0.96-1.0) 0.80 (0.74-0.85) 0.78 (0.67-0.89) 0.81 (0.71-0.91) 0.36 (0.27-0.45) 0.89 (0.78-0.99) 1.00 (0.91-1.00) 1.00 (0.91-1.00) 0.54 (0.46-0.63) 0.56 (0.44-0.68) 0.90 (0.80-1.0) 0.74 (0.62-0.87) 0.70 (0.59-0.81) 0.83 (0.54-1.00) 0.93 (0.81-1.00) 0.71 (0.56-0.86) 0.74 (0.62-0.87) 0.74 (0.68-0.81)

Aguero-Rosenfeld (1988) Bernander (1994) Birtles (1990) Diederen (2006) Diederen (2006) Diederen (2007) Diederen (2007) Dominguez (1997) Dominguez (1997) Dominguez (2001) Dominguez (2001) Franzin (2000) Franzin (2000) Hackman (1996) Harrison (1998) Harrison (2001) Helbig (2001) Kazandjian (1997) Kohler (1983) Kohler (1985) Koide (2005) Koide (2005) Koide (2005) Leland (1991) Plouffe (1995) Samuel (1990) Sathapatayavongs (1982) Sathapatayavongs (1983) Socan (1999) Tang (1982) Tang (1986) White (1981) Overall

0

Sensitivity

1

Figure 2. Pooled sensitivity of Legionella urinary antigen serogroup 1. Studies reporting sensitivities of 1 were 100% sensitive. For these studies, we calculated the SE using the Agresti-Coull (Wald) adjustment.9

Sensitivity The pooled sensitivity of Legionella urinary assays was 0.74 (95% CI, 0.68 to 0.81) [Fig 2].

These results were heterogeneous (Q ⫽ 507.74; degrees of freedom, 31; p ⬍ 0.005; I2 ⫽ 93.9%). This heterogeneity was not explained by publication year, sample size, study design, country of origin, or the particular reference standard used. Overall, no difference was found between the performance of commercial and in-house tests (commercial test sensitivity, 0.74 [95% CI, 0.67 to 0.82]; in-house test sensitivity, 0.74 [95% CI, 0.62 to 0.86]), and no statistically significant differences were found among the various brands. Higher QUADAS scores were associated with lower reported sensitivities (metaregression ␤, ⫺0.05; p ⫽ 0.002). Assays using radioimmunoassay had lower sensitivity (metaregression ␤, ⫺0.1642503; p ⫽ 0.03) [Table 3]. Quality scores and specific assays explained 57% of the heterogeneity, although the residual heterogeneity was still significant (Q ⫽ 220.39; degrees of freedom, 29; p ⬍ 0.005; I2 ⫽ 86.8%). Specificity The pooled estimate for specificity was 0.991 (95% CI, 0.984 to 0.997) [Fig 3]. This result was also heterogeneous (Q ⫽ 137.14; degrees of freedom, 31; p ⬍ 0.005; I2 ⫽ 77.4%). Heterogeneity was not explained by publication year, sample size, country of origin, study design, or particular reference standard used. There was no association between specificity and QUADAS scores. Overall, no difference was found in the performance between commercial and in-house tests (commercial test specificity, 0.98 [95% CI, 0.97 to 1.00]; in-house test specificity, 0.99 [95% CI, 0.991 to 1.00]), and no differences were found among brands. One hundred percent of the heterogeneity in specificity was due to a single study,31 which reported a specificity of 73.9%. This study

Table 3—Weighted Pooled Analysis Variables

Number of Studies

Sensitivity (95% CI)

Specificity (95% CI)

LR⫹

LR–

Overall Brand-name assays Bartels BinaxNow (ICT) Binax (EIA) Binax (RIA) Biotest DuPont L-Clone Rapid U SAS In-house test Test EIA ELISA ICT LA RIA RPA

32

0.74 (0.680–0.80)

0.991 (0.98–0.997)

82

0.26

2 3 7 3 3 1 1 2 1 9

0.91 (0.74–1.08) 0.90 (0.79–1.00) 0.74 (0.59–0.89) 0.57 (0.49–0.66) 0.67 (0.45–0.88) 0.57 (0.36–0.77) 0.54 (0.46–0.63) 0.82 (0.62–1.00) 0.83 (0.74–0.92) 0.74 (0.62–0.86)

1.0 (0.975–1.00) 0.99 (0.98–0.997) 0.99 (0.98–1.00) 0.99 (0.98–1.00) 0.99 (0.97–1.00) 1.0 (0.98–1.00) 0.74 (0.69–0.79) 0.96 (0.93–0.99) 0.99 (0.97–1.00) 0.99 (0.97–1.00)

90 74 57 67

0.10 0.26 0.44 0.33

2.1 20.5 83.0 74.0

0.46 0.19 0.17 0.26

12 5 6 2 6 1

0.75 (0.64–0.86) 0.78 (0.72–0.85) 0.86 (0.79–0.94) 0.62 (0.47–0.77) 0.57 (0.41–0.73) 0.93 (0.81–1.00)

0.99 (0.98–1.00) 0.99 (0.98–1.00) 0.99 (0.98–1.00) 0.87 (0.62–1.00) 0.997 (0.993–1.00) 1.00 (0.98–1.00)

75 67 86 4.8 114

0.25 0.22 0.14 0.44 0.43

See Table 1 for abbreviations not used in the text. 1582

Original Research

1.00 (0.99-1.00) 0.98 (0.96-1.00) 1.00 (0.97-1.00) 0.97 (0.94-0.99) 0.96 (0.92-0.99) 0.99 (0.97-1.00) 0.99 (0.98-1.00) 1.00 (0.96-1.00) 1.00 (0.96-1.00) 1.00 (0.98-1.00) 1.00 (0.98-1.00) 0.95 (0.89-1.00) 0.95 (0.89-1.00) 0.95 (0.89-1.00) 1.00 (0.92-1.00) 1.00 (0.98-1.00) 1.00 (0.99-1.00) 1.00 (0.98-1.00) 1.00 (0.99-1.00) 1.00 (0.99-1.00) 1.00 (0.94-1.00) 1.00 (0.94-1.00) 1.00 (0.94-1.00) 0.74 (0.70-0.78) 0.99 (0.98-1.00) 1.00 (0.74-1.00) 1.00 (0.98-1.00) 0.99 (0.98-1.00) 0.99 (0.96-1.00) 1.00 (0.99-1.00) 1.00 (0.99-1.00) 0.98 (0.97-1.00) 0.991 (0.98-0.997)

Aguero-Rosenfeld (1988) Bernander (1994) Birtles (1990) Diederen (2006) Diederen (2006) Diederen (2007) Diederen (2007) Dominguez (1997) Dominguez (1997) Dominguez (2001) Dominguez (2001) Franzin (2000) Franzin (2000) Hackman (1996) Harrison (1998) Harrison (2001) Helbig (2001) Kazandjian (1997) Kohler (1983) Kohler (1985) Koide (2005) Koide (2005) Koide (2005) Leland (1991) Plouffe (1995) Samuel (1990) Sathapatayavongs (1982) Sathapatayavongs (1983) Socan (1999) Tang (1982) Tang (1986) White (1981) Overall

0

Specificity

1

Figure 3. Pooled specificity of Legionella urinary antigen serogroup type 1. Studies reporting specificities of 1 were 100% specific. For these studies, we calculated the SE using the Agresti-Coull (Wald) adjustment.9

used a commercial assay (L-Clone; Access Medical Systems, Inc; Branford, CT) that used latex agglutination. Exclusion of this study eliminated heterogeneity (Q ⫽ 19.91; degrees of freedom, 30; p ⫽ 0.92; I2 ⫽ 0.0%). Component Analysis The overall QUADAS score was associated with sensitivity but not specificity. On component analysis, the following two components were associated with sensitivity: whether the reference test was independent of the index test (p ⫽ 0.001); and whether the period between reference and index tests was short (p ⫽ 0.01). Studies that met these quality criteria had lower pooled sensitivities than those that did not (Table 2). None of the QUADAS components were associated with specificity. Publication Bias Evidence pointed to publication bias of diagnostic odds ratios based on the Egger test (p ⫽ 0.03). There was no evidence of asymmetry of the funnel www.chestjournal.org

plots for either sensitivity or specificity given the constraint that neither could exceed 1.0 in value, although areas were missing in the cloud that could suggest possible publication bias. Discussion Our pooled analysis found that Legionella urinary antigen has very good specificity but low sensitivity for L pneumophila serogroup 1; thus, it is better for ruling in than ruling out disease. One of our most interesting findings is the 26% false-negative rate for Legionella urinary antigen. Practically speaking, this finding suggests that stopping therapy with antiLegionella antibiotics in patients with a clinical suspicion for L pneumophila after a negative urinary antigen test result, especially in endemic areas and in cases of severe pneumonia, could be a mistake. A positive urinary antigen test result, in the appropriate clinical setting, virtually rules in Legionellosis, but a negative urinary antigen test result does not rule out the presence of the disease, as 26% of patients with confirmed Legionellosis have a negative urinary antigen test result. The various test brands performed well in general, although radioimmunoassay seemed less sensitive and latex agglutination less specific than other types of assays. However, the study quality for the Legionella diagnostic reports was low (average QUADAS score, 4.4 of 14) [Table 2], and most trials included relatively small numbers of patients. Higher quality studies reported lower sensitivities.22,26,29,30 This bias in which low-quality studies overestimate test performance has been seen previously in studies46,47 of diagnostic tests. Nearly all the studies we reviewed used a case-control design; casecontrol studies are particularly susceptible to biased results. Currently, Legionnaires’ disease due to L pneumophila serogroup 1 is now more frequently diagnosed by urinary antigen testing rather than culture or other serologic tests.4 The advantages of urinary antigen testing over culture include the simplicity of collection and the rapidity of test results. Urinary antigen tests also are less influenced by antibiotic use than sputum cultures. The major disadvantage is that these tests are largely restricted to L pneumophila serogroup 1. The limited data available suggest that the current urinary antigens are not helpful for testing serogroups other than type 1; the diagnosis of those other serotypes can only occur with culture. Fortunately, most cases (85 to 90%) of Legionnaires’ disease are caused by serogroup 1.48 One current guideline49 recommends trying to obtain a sputum culture for epidemiologic tracking. Several limitations in this study warrant mention. First, we limited our search to published EnglishCHEST / 136 / 6 / DECEMBER, 2009

1583

language reports. Second, we were unable to contact two author groups to confirm the duplication of data, so we had to drop their data. Third, there was evidence for publication bias; therefore, it is possible that our results are an overestimation of the performance of the test. Fourth, the subgroup analyses, such as specific brand or type of assay, often included relatively few studies and should not be used as the basis for selecting a particular brand or assay. Fifth, we found evidence of heterogeneity in both sensitivity and specificity. Although we were able to explain the majority of the heterogeneity in sensitivity with the specific assay and quality scores, there was still significant heterogeneity. For specificity, all the heterogeneity was due to a single study.31 In diagnostic metaanalyses, heterogeneity may be overstated because of the weighting commonly used. In this metaanalysis, factors that could be important contributors to heterogeneity were the severity of the illness, whether subjects were immunosuppressed, the duration of illness prior to laboratory diagnosis, and whether the pneumonia was community acquired or hospital acquired. Unfortunately, the included studies provided little or no information, rendering our ability to explore these potentially important sources of heterogeneity impossible. Finally, we were unable to assess test performance for serogroups of Legionella other than type 1. The two studies that focused on other serogroups reported that the test performed poorly. Although type 1 is the most common serogroup in pneumonia, there are other strains. How well the antigen will perform in other serotypes is uncertain. In conclusion, the Legionella urine antigen appears to have excellent specificity, though only modest sensitivity for serogroup 1. Pragmatically, this finding means that in the appropriate clinical context, a positive antigen test result virtually rules in L pneumophila. However, the 26% rate of false-negative findings should make the clinician cautious about deciding to withdraw therapy with anti-Legionella antibiotics. In general, the quality of studies in this metaanalysis was low with regard to reporting important clinical information such as pneumonia severity or where the pneumonia was acquired. Well-designed prospective studies on urinary antigen testing are needed. Studies that include serogroup and species other than L pneumophila serogroup 1 also would be of value.

Acknowledgments Author contributions: Drs. Shimada and Noguchi contributed to the conception and design of the study. Drs. Shimada, Jackson, and Miyashita contributed to acquisition, analysis, and interpretation of the data. Drs. Shimada, Noguchi, and Jackson drafted the article. Drs. Noguchi, Jackson, Hayashino, Kamiya, and 1584

Yamazaki critically revised the article for important intellectual content. Drs. Shimada, Jackson, and Fukuhara contributed to the final approval of the article. Dr. Jackson provided statistical expertise. Drs. Matsumura and Fukuhara supervised the study. Financial/nonfinancial contributions: The authors have reported to the ACCP that no significant conflicts of interest exist with any companies/organizations whose products or services may be discussed in this article. Role of sponsors: The funding source had no role in data collection or interpretation or in manuscript preparation.

References 1 Neil K, Berkelman R. Increasing incidence of Legionellosis in the United States, 1990 –2005: changing epidemiologic trends. Clin Infect Dis 2008; 47:591–599 2 von Baum H, Ewig S, Marre R, et al. Community-acquired Legionella pneumonia: new insights from the German competence network for community acquired pneumonia. Clin Infect Dis 2008; 46:1356 –1364 3 Tilton RC. Legionnaires’ disease antigen detected by enzyme-linked immunosorbent assay. Ann Intern Med 1979; 90:697– 698 4 Benin AL, Benson RF, Besser RE. Trends in Legionnaires disease, 1980 –1998: declining mortality and new patterns of diagnosis. Clin Infect Dis 2002; 35:1039 –1046 5 Inverness Medical Professional Diagnostics. Legionella. Available at: http://www.binax.com/products/now_legionella_urinary. aspx. Accessed July 13, 2009 6 Whiting P, Rutjes AW, Reitsma JB, et al. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol 2003; 3:25 7 Kelly S, Berry E, Roderick P, et al. The identification of bias in studies of the diagnostic performance of imaging modalities. Br J Radiol 1997; 70:1028 –1035 8 DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986; 7:177–188 9 Agresti A, Coull B. Approximate is better than exact for interval estimates of binominal proportions. Am Stat 1998; 52:119 –126 10 Littenberg B, Moses LE. Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method. Med Decis Making 1993; 13:313–321 11 van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and metaregression. Stat Med 2002; 21:589 – 624 12 Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol 2005; 58:882– 893 13 Song F, Khan KS, Dinnes J, et al. Asymmetric funnel plots and publication bias in meta-analyses of diagnostic accuracy. Int J Epidemiol 2002; 31:88 –95 14 Egger M, Davey Smith G, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315:629 – 634 15 Diederen BM, Peeters MF. Evaluation of the SAS Legionella test: a new immunochromatographic assay for the detection of Legionella pneumophila serogroup 1 antigen in urine. Clin Microbiol Infect 2007; 13:86 – 88 16 Diederen BM, Peeters MF. Evaluation of two new immunochromatographic assays (Rapid U Legionella antigen test and SD Bioline Legionella antigen test) for detection of Legionella pneumophila serogroup 1 antigen in urine. J Clin Microbiol 2006; 44:2991–2993 Original Research

17 Diederen BM, Peeters MF. Evaluation of Rapid U Legionella Plus Test, a new immunochromatographic assay for detection of Legionella pneumophila serogroup 1 antigen in urine. Eur J Clin Microbiol Infect Dis 2006; 25:733–735 18 Koide M, Higa F, Tateyama M, et al. Detection of Legionella species in clinical samples: comparison of polymerase chain reaction and urinary antigen detection kits. Infection 2006; 34:264 –268 19 Helbig JH, Uldum SA, Luck PC, et al. Detection of Legionella pneumophila antigen in urine samples by the BinaxNOW immunochromatographic assay and comparison with both Binax Legionella urinary enzyme immunoassay (EIA) and Biotest Legionella urinary antigen EIA. J Med Microbiol 2001; 50:509 –516 20 Harrison TG, Doshi N. Evaluation of the Bartels Legionella urinary antigen enzyme immunoassay. Eur J Clin Microbiol Infect Dis 2001; 20:738 –740 21 Dominguez J, Gali N, Blanco S, et al. Assessment of a new test to detect Legionella urinary antigen for the diagnosis of Legionnaires’ disease. Diagn Microbiol Infect Dis 2001; 41:199 –203 22 Franzin L, Cabodi D. Comparative evaluation of two commercially available antigen enzyme immunoassays (EIA) for the detection of Legionella pneumophila urinary antigen in frozen non-concentrated urine samples. New Microbiol 2000; 23:383–389 23 Socan M, Marinic-Fiser N, Kese D. Comparison of serologic tests with urinary antigen detection for diagnosis of Legionnaires’ disease in patients with community-acquired pneumonia. Clin Microbiol Infect 1999; 5:201–204 24 Harrison T, Uldum S, Alexiou-Daniel S, et al. A multicenter evaluation of the Biotest Legionella urinary antigen EIA. Clin Microbiol Infect 1998; 4:359 –365 25 Kazandjian D, Chiew R, Gilbert GL. Rapid diagnosis of Legionella pneumophila serogroup 1 infection with the Binax enzyme immunoassay urinary antigen test. J Clin Microbiol 1997; 35:954 –956 26 Dominguez JA, Matas L, Manterola JM, et al. Comparison of radioimmunoassay and enzyme immunoassay kits for detection of Legionella pneumophila serogroup 1 antigen in both concentrated and nonconcentrated urine samples. J Clin Microbiol 1997; 35:1627–1629 27 Hackman BA, Plouffe JF, Benson RF, et al. Comparison of Binax Legionella urinary antigen EIA kit with Binax RIA urinary antigen kit for detection of Legionella pneumophila serogroup 1 antigen. J Clin Microbiol 1996; 34:1579 –1580 28 Dominguez JA, Manterola JM, Blavia R, et al. Detection of Legionella pneumophila serogroup 1 antigen in nonconcentrated urine and urine concentrated by selective ultrafiltration. J Clin Microbiol 1996; 34:2334 –2336 29 Plouffe JF, File TM Jr, Breiman RF, et al. Reevaluation of the definition of Legionnaires’ disease: use of the urinary antigen assay; Community Based Pneumonia Incidence Study Group. Clin Infect Dis 1995; 20:1286 –1291 30 Bernander S, Gastrin B, Lofgren S, et al. Legionella urinary antigen in early disease. Scand J Infect Dis 1994; 26:777–778 31 Leland DS, Kohler RB. Evaluation of the L-CLONE Legionella pneumophila serogroup 1 urine antigen latex test. J Clin Microbiol 1991; 29:2220 –2223 32 Samuel D, Harrison TG, Taylor AG. Detection of Legionella pneumophila serogroup 1 urinary antigen using an enhanced

www.chestjournal.org

33

34

35

36

37

38 39

40 41 42

43

44 45 46 47 48 49

chemiluminescence ELISA. J Biolumin Chemilumin 1990; 5:183–185 Birtles RJ, Harrison TG, Samuel D, et al. Evaluation of urinary antigen ELISA for diagnosing Legionella pneumophila serogroup 1 infection. J Clin Pathol 1990; 43:685– 690 Aguero-Rosenfeld ME, Edelstein PH. Retrospective evaluation of the Du Pont radioimmunoassay kit for detection of Legionella pneumophila serogroup 1 antigenuria in humans. J Clin Microbiol 1988; 26:1775–1778 Kohler RB, Allen SD, Wheat LJ, et al. Development and evaluation of diagnostic radiometric assays for serogroup 4 L pneumophila urinary antigens. Diagn Microbiol Infect Dis 1987; 6:101–107 Tang PW, Toma S. Broad-spectrum enzyme-linked immunosorbent assay for detection of Legionella soluble antigens. J Clin Microbiol 1986; 24:556 –558 Kohler RB, Wheat LJ, French ML, et al. Cross-reactive urinary antigens among patients infected with Legionella pneumophila serogroups 1 and 4 and the Leiden 1 strain. J Infect Dis 1985; 152:1007–1012 Sathapatayavongs B, Kohler RB, Wheat LJ, et al. Rapid diagnosis of Legionnaires’ disease by latex agglutination. Am Rev Respir Dis 1983; 127:559 –562 Kohler RB, Sathapatayavongs B. Recent advances in the diagnosis of serogroup 1 L pneumophila pneumonia by detection of urinary antigen. Zentralbl Bakteriol Mikrobiol Hyg A 1983; 255:102–107 Kohler R, Wheat LJ. Rapid diagnosis of pneumonia due to Legionella pneumophila serogroup 1. J Infect Dis 1982; 146:444 Tang PW, de Savigny D, Toma S. Detection of Legionella antigenuria by reverse passive agglutination. J Clin Microbiol 1982; 15:998 –1000 Sathapatayavongs B, Kohler RB, Wheat LJ, et al. Rapid diagnosis of Legionnaires’ disease by urinary antigen detection: comparison of ELISA and radioimmunoassay. Am J Med 1982; 72:576 –582 Kohler RB, Zimmerman SE, Wilson E, et al. Rapid radioimmunoassay diagnosis of Legionnaires’ disease: detection and partial characterization of urinary antigen. Ann Intern Med 1981; 94:601– 605 White A, Kohler RB, Wheat LJ, et al. Rapid diagnosis of Legionnaires’ disease. Trans Am Clin Climatol Assoc 1981; 93:50 – 62 Edelstein PH. Urine antigen tests positive for Pontiac fever: implications for diagnosis and pathogenesis. Clin Infect Dis 2007; 44:229 –231 Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 1999; 282:1061–1066 Rutjes AW, Reitsma JB, Di Nisio M, et al. Evidence of bias and variation in diagnostic accuracy studies. CMAJ 2006; 174:469 – 476 Muder RR, Yu VL. Infection due to Legionella species other than L pneumophila. Clin Infect Dis 2002; 35:990 –998 Mandell LA, Wunderink RG, Anzueto A, et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of communityacquired pneumonia in adults. Clin Infect Dis 2007; 44(suppl):S27–S72

CHEST / 136 / 6 / DECEMBER, 2009

1585