- Email: [email protected]
Evaluation of microbial identification systems
L
Editorial Evaluation of Microbial Identification Systems Stephen C. Edberg, Ph.D. Director, Clinical Microbiology Laboratory Department of La~bor~tory Medicine Yale University School of Medicine New Haven, Connecticut 06520 Every civilization has its myths and fables. They exist to provide support and tranquility in an unsettled environment. A serum bacteriocidal level of 1"8 or above, designating a favorable outcome in a patient with subacute bacterial endocarditis, and the acceptance of the geometric titer, the MIC, as an absolute number belong to this category. This article will address another prominent legend, that "95% agreement" legitimizes a microbial identification system. Unlike tests in other clinical laboratories, clinical microbiology does not have the luxury of defining its procedures in terms of accuracy plus or minus standard deviations. This restriction is largely due to the fact that whereas we use a genus and species name, a species is strictly a grouping of living entities that sexually reproduce to yield viable offspring. Sexual exclusion defines the individual species. Because bacteria do not reproduce by sexual means the implication is that there are no true species in the Linnean lexicon. (Please note "speciate" means to produce new species in the Darwinian sense and does not mean identification of species.) Our prime reference laboratory, the Centers for Disease Control, has circumvented this sexual component by defining a species as an organism that shares >170% DNA homology with a second organism. Unfortunately, ii is not now possible for the individual clinical microbiology laboratory to routinely perform DNA homology testing, and we must rely on phenotypic expressions to define a species name. This factor brings additional problems: not all phenotypic expressions are stable
(they may be transferred among different species), some phenotypic expressions are constitutive and some are inducible, and our means to achieve a positive or negative test can significantly vary with substrate concentration, buffer, and base medium used. Many methods, including the Centers for Disease Control conventional series of biochemical tests, homegrown "conventional" biochemical tests, and commercial tests, are available for the identification of members of the family Enterobacteriaceae. Based on a microbe's reacting (positive test) or not reacting (negative test) in a series of biochemical tests, the isolate is named. In order to evaluate a microbial identification system accurately, I believe certain basic criteria must be fulfilled: l) a comparison of identification systems must be based on uniform levels of identification (e.g., species to species); 2) any new bacterial identification system must be compared to a standard reference method; 3) epithets (such as "first X number of species possible") must be taken into account. Microbial identification systems must be compared based on the names of the organisms and not individual biochemical tests. The activity of an individual biochemical test in an identification system solely relates to its usefulness in that particular identification system. In naming isolates two principles must be considered: typicality, or "likelihood fraction" (LFR), which is a measure of the resemblance of the isolate to the typical culture of the species and relatedness, or "normalized probability," which is a me~sure of the degree of separation of a particular clinical isolate from other identities. The standard system must be a recognized reference system used by an agency that is an authority by virtue of its ability to name and have new species accepted. We have available to us several reference systems, the most common in the United States
being the Centers for Disease Control and the Bergey's Manual. One should not assume that because an investigator inoculates agar media that his system is "conventional." Once a standard method is chosen as reference and a series of biochemical tests have been established that will identify isolates in accordance with the standard methods, the investigator must decide what his data base will be. The literature reports evaluations that contain from fewer than 100 to'over I000 isolates. An evaluation can be severely influenced by the proportions of different species chosen in the evaluation. For example, if system B performs poorly with Klebsiella ozaenae, the inclusion of inordinantly large numbers of this species could weaken any overall measure of accuracy (i.e., "the system agreed 87% with system A " ) . In the same vein, organisms from each species should be chosen by likelihood fraction analysis so that there is a standard distribution around average (i.e., do not include large numbers of "atypicals"). I believe the data base must contain, for valid statistical comparison, 30 isolates of each species. We have successfully utilized two computer programs to statistically describe the accuracy of bacterial identification systems. The first generates a crosstabs table that presents, in a single chart, the strengths and weaknesses of identification systems and obviates the need for many accessory tables (Figure 1). After the crosstabs table is generated analysis of each species is made by subjecting the data to statistical analysis by the Cochran Q method. The Cochran Q is a statistic that allows one to compare factors such as population groups. Subjecting microbial data to computer analysis (Figure 2) demonstrates that one requires at least 30 isolates of a particular species to achieve statistical significance. The Cochran Q can be used if the organisms composing the data base have a normal distribution around the typical culture (i.e., LFR
MICRO-ID IDENTIFICATION
/ t/
_ I~,/'~/,~/o/~/~/~/~/~/~,/~/~,/~,/~/~/~/~
~,
E,ooh Sh,gena
157
1~
4
-
sp
31 4 -m
$
s~ner
3
-
E
f~d~
1
-
s
&p~
1
~
S e~tertt~s A Z 0 I-. < C)
23
-
2
-
I ~ t s
3 I 1
C lrbund,
kZ uJ Q
C ~versus
~
K p,'~ecrr,~n~ae
1~
-
K ozaena~
15
-
E aeroger)es
25
-
E ck~acae
Z
E. haf,~e
15
°
E ~jg~ne~s
9
-
Z UJ
P rn~rab~s
.
.
.
-
_
_
-n
.
.
.
.
.
.
-
_
_
-
-
-
-
-
2
-
-
-
-
33
I
.
.
i-
-
-
.
.
.
.
1
.
.
-m _
-u
_
-i
71 -
.
.
.
.
.
.
.
.
.
.
8
-
.
17
-
.
11~
-
.
12
-
2
-
2
-
.
.
.
.
.
.
.
44
.
.
.
.
.
.
.
.
.
.
1
.
11 -
n
.
.
.
.
.
.
91
.
.
.
.
7
-
.
.
.
-
-
1
.
.
.
.
.
.
.
2
.
-
'
-i
-
I ,
-
- i -
I
-
21 21
-
-
-
-
-
2
-
-n
-
5
-
-u
-
4
-
II
.
.
Y e~erocokt~ca
.
.
.
.
.
.
8
.
.
. .
. .
.
.
.
.
.
.
2C
.
.
.
11
1
Figure 2. The relationship of sample size to predictability of the Cochran Q test. IJ
11
II 0
I0
~
.15
• . . . . .
~
/ 15
10/20
, ~ . ~ . ~
N/tO
.
I
g5 .es
03
02
OI
.000
"~r
=
. . . .
: - 1
i
I i IO
i It
I 10
I 25
I 1tl
I 15
SampleSize
I 11
I 45
I 50
I $$
i H
-
-
:[ -
.
i
2
Figure 1. Crosstabs table. (Reprinted with permission from Edberg, S., et al. 1979. Clinical evaluation of the MICRO-ID, API 20E, and conventional media systems for identification of Enterobacteriaceae. J. Clin. Microbiol. 10:161-167.)
S
-
-
-
!
- i
P stuart.
U
°
!
P akcaMec~errs
~
.
1~
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
.
.
.
.
.
.
.
.
.
P tettgen
,,e-
°
-n
.
P mo~gav~
::7
n
-I
-n
-
S kquefaoe~s
.
-
.
-
S ~ce$cen~
Z 0 0
.
6
..J
o_
.
--n
6 .
.
.
--I
= 1.000). Like the chi-square test, on which it is based, the Q test will accurately yield a P value to predict if two population groups are the same or different. Therefore, by creating a crosstabs table and using statistical analysis, the clinical microbiologist can accurately describe the performance of a microbial identification system on a species by species basis in a single table. However, I do not imply the case is closed. Other statistics may also be, and should be, developed to help expand the scientific basis of clinical microbiology. We cannot describe the accuracy of any microbial identification system by a single number (e.g., "95% agreement"). Such a number is subject to considerable manipulation, both objectively and subjectively. Although much of the joy of clinical microbiology and the separation of clinical microbiology from other laboratory sciences rest with the dynamic and occasionally unpredictable activity of the living microbe, our tests must be defendable on mathematical principles if they are to have any meaning.
Editorial Evaluation of Microbial Identification Systems Stephen C. Edberg, Ph.D. Director, Clinical Microbiology Laboratory Department of La~bor~tory Medicine Yale University School of Medicine New Haven, Connecticut 06520 Every civilization has its myths and fables. They exist to provide support and tranquility in an unsettled environment. A serum bacteriocidal level of 1"8 or above, designating a favorable outcome in a patient with subacute bacterial endocarditis, and the acceptance of the geometric titer, the MIC, as an absolute number belong to this category. This article will address another prominent legend, that "95% agreement" legitimizes a microbial identification system. Unlike tests in other clinical laboratories, clinical microbiology does not have the luxury of defining its procedures in terms of accuracy plus or minus standard deviations. This restriction is largely due to the fact that whereas we use a genus and species name, a species is strictly a grouping of living entities that sexually reproduce to yield viable offspring. Sexual exclusion defines the individual species. Because bacteria do not reproduce by sexual means the implication is that there are no true species in the Linnean lexicon. (Please note "speciate" means to produce new species in the Darwinian sense and does not mean identification of species.) Our prime reference laboratory, the Centers for Disease Control, has circumvented this sexual component by defining a species as an organism that shares >170% DNA homology with a second organism. Unfortunately, ii is not now possible for the individual clinical microbiology laboratory to routinely perform DNA homology testing, and we must rely on phenotypic expressions to define a species name. This factor brings additional problems: not all phenotypic expressions are stable
(they may be transferred among different species), some phenotypic expressions are constitutive and some are inducible, and our means to achieve a positive or negative test can significantly vary with substrate concentration, buffer, and base medium used. Many methods, including the Centers for Disease Control conventional series of biochemical tests, homegrown "conventional" biochemical tests, and commercial tests, are available for the identification of members of the family Enterobacteriaceae. Based on a microbe's reacting (positive test) or not reacting (negative test) in a series of biochemical tests, the isolate is named. In order to evaluate a microbial identification system accurately, I believe certain basic criteria must be fulfilled: l) a comparison of identification systems must be based on uniform levels of identification (e.g., species to species); 2) any new bacterial identification system must be compared to a standard reference method; 3) epithets (such as "first X number of species possible") must be taken into account. Microbial identification systems must be compared based on the names of the organisms and not individual biochemical tests. The activity of an individual biochemical test in an identification system solely relates to its usefulness in that particular identification system. In naming isolates two principles must be considered: typicality, or "likelihood fraction" (LFR), which is a measure of the resemblance of the isolate to the typical culture of the species and relatedness, or "normalized probability," which is a me~sure of the degree of separation of a particular clinical isolate from other identities. The standard system must be a recognized reference system used by an agency that is an authority by virtue of its ability to name and have new species accepted. We have available to us several reference systems, the most common in the United States
being the Centers for Disease Control and the Bergey's Manual. One should not assume that because an investigator inoculates agar media that his system is "conventional." Once a standard method is chosen as reference and a series of biochemical tests have been established that will identify isolates in accordance with the standard methods, the investigator must decide what his data base will be. The literature reports evaluations that contain from fewer than 100 to'over I000 isolates. An evaluation can be severely influenced by the proportions of different species chosen in the evaluation. For example, if system B performs poorly with Klebsiella ozaenae, the inclusion of inordinantly large numbers of this species could weaken any overall measure of accuracy (i.e., "the system agreed 87% with system A " ) . In the same vein, organisms from each species should be chosen by likelihood fraction analysis so that there is a standard distribution around average (i.e., do not include large numbers of "atypicals"). I believe the data base must contain, for valid statistical comparison, 30 isolates of each species. We have successfully utilized two computer programs to statistically describe the accuracy of bacterial identification systems. The first generates a crosstabs table that presents, in a single chart, the strengths and weaknesses of identification systems and obviates the need for many accessory tables (Figure 1). After the crosstabs table is generated analysis of each species is made by subjecting the data to statistical analysis by the Cochran Q method. The Cochran Q is a statistic that allows one to compare factors such as population groups. Subjecting microbial data to computer analysis (Figure 2) demonstrates that one requires at least 30 isolates of a particular species to achieve statistical significance. The Cochran Q can be used if the organisms composing the data base have a normal distribution around the typical culture (i.e., LFR
MICRO-ID IDENTIFICATION
/ t/
_ I~,/'~/,~/o/~/~/~/~/~/~,/~/~,/~,/~/~/~/~
~,
E,ooh Sh,gena
157
1~
4
-
sp
31 4 -m
$
s~ner
3
-
E
f~d~
1
-
s
&p~
1
~
S e~tertt~s A Z 0 I-. < C)
23
-
2
-
I ~ t s
3 I 1
C lrbund,
kZ uJ Q
C ~versus
~
K p,'~ecrr,~n~ae
1~
-
K ozaena~
15
-
E aeroger)es
25
-
E ck~acae
Z
E. haf,~e
15
°
E ~jg~ne~s
9
-
Z UJ
P rn~rab~s
.
.
.
-
_
_
-n
.
.
.
.
.
.
-
_
_
-
-
-
-
-
2
-
-
-
-
33
I
.
.
i-
-
-
.
.
.
.
1
.
.
-m _
-u
_
-i
71 -
.
.
.
.
.
.
.
.
.
.
8
-
.
17
-
.
11~
-
.
12
-
2
-
2
-
.
.
.
.
.
.
.
44
.
.
.
.
.
.
.
.
.
.
1
.
11 -
n
.
.
.
.
.
.
91
.
.
.
.
7
-
.
.
.
-
-
1
.
.
.
.
.
.
.
2
.
-
'
-i
-
I ,
-
- i -
I
-
21 21
-
-
-
-
-
2
-
-n
-
5
-
-u
-
4
-
II
.
.
Y e~erocokt~ca
.
.
.
.
.
.
8
.
.
. .
. .
.
.
.
.
.
.
2C
.
.
.
11
1
Figure 2. The relationship of sample size to predictability of the Cochran Q test. IJ
11
II 0
I0
~
.15
• . . . . .
~
/ 15
10/20
, ~ . ~ . ~
N/tO
.
I
g5 .es
03
02
OI
.000
"~r
=
. . . .
: - 1
i
I i IO
i It
I 10
I 25
I 1tl
I 15
SampleSize
I 11
I 45
I 50
I $$
i H
-
-
:[ -
.
i
2
Figure 1. Crosstabs table. (Reprinted with permission from Edberg, S., et al. 1979. Clinical evaluation of the MICRO-ID, API 20E, and conventional media systems for identification of Enterobacteriaceae. J. Clin. Microbiol. 10:161-167.)
S
-
-
-
!
- i
P stuart.
U
°
!
P akcaMec~errs
~
.
1~
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
.
.
.
.
.
.
.
.
.
P tettgen
,,e-
°
-n
.
P mo~gav~
::7
n
-I
-n
-
S kquefaoe~s
.
-
.
-
S ~ce$cen~
Z 0 0
.
6
..J
o_
.
--n
6 .
.
.
--I
= 1.000). Like the chi-square test, on which it is based, the Q test will accurately yield a P value to predict if two population groups are the same or different. Therefore, by creating a crosstabs table and using statistical analysis, the clinical microbiologist can accurately describe the performance of a microbial identification system on a species by species basis in a single table. However, I do not imply the case is closed. Other statistics may also be, and should be, developed to help expand the scientific basis of clinical microbiology. We cannot describe the accuracy of any microbial identification system by a single number (e.g., "95% agreement"). Such a number is subject to considerable manipulation, both objectively and subjectively. Although much of the joy of clinical microbiology and the separation of clinical microbiology from other laboratory sciences rest with the dynamic and occasionally unpredictable activity of the living microbe, our tests must be defendable on mathematical principles if they are to have any meaning.