- Email: [email protected]
Microbiological collaborative studies for quality control in food laboratories: Reference material and evaluation of analyst's errors
International Journal of Food Microbiology ELSEVIER
International Journal of Food Microbiology 24 (1994) 41-52
Review Paper
Microbiological collaborative studies for quality control in food laboratories: Reference material and evaluation of analyst's errors C. Berg *, S. Dahms, G. Hildebrandt, S. Klaschka, H. Weiss Institut fiir Lebensmittelhygiene, Institut fiir Biometrie und Informationsverarbeitung des Fachbereiches Veteriniirmedizin der Freien Universitiit Berlin, K6nigsweg 69, D-14163 Berlin, Germany
Abstract Working groups of the Federal Health Office in Germany and of the International Dairy Federation (IDF) have developed a quality assurance system to assess the analyst performance for colony count methods. The experiment design consists of several dilution series produced from a homogeneous sample suspension. Each series contains a number of twofold dilution steps with parallel plates on each level. The structure of this design permits a detailed analysis of the total variance and identification of analysts' methodological errors as single effects. A computer-based interpretation aid may classify the laboratory as working in correspondence with good laboratory practice ('acceptable') or, otherwise, furnishing 'to good' or 'unacceptable' results. Some examples are given for deviation depending on individual faults. Both, internal quality assurance and collaborative studies demand suitable reference samples. The reference material used has to meet certain requirements concerning homogeneity, contamination level and microbiological stability during a period of storage and transport. A naturally contaminated sample material was prepared, which has been used successfully in microbiological collaborative studies. Keywords: Quality assurance; Good laboratory practice; Analyst performance; Plate count
method; Analysis of variance; Reference material
1. Introduction I n o r d e r to a t t a i n r e l i a b l e e x a m i n a t i o n results in m i c r o b i o l o g i c a l quality control, it is n e c e s s a r y to a p p l y s t a n d a r d i z e d m e t h o d s . A d r a f t o f t h e s t a n d a r d ' Q u a l i t y
* Corresponding author. Tel. (30) 8108 2550. Fax (30) 8108 2552. 0168-1605/94/$07.00 © 1994 Elsevier Science B.V. All rights reserved SSDI 0168-1605(94)00132-4
42
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
control in the microbiological laboratory - the spread plate technique' was elaborated by the working group 'Statistical Evaluation of Microbiological Methods' of the German Federal Health Office according to official methods (Weiss et al., 1991). Nearly the same concept has been worked out for the pour-plate method which is part of the collection of official methods of the German Federal Health Office (BGA, 1991). It will also become International Dairy Federation (IDF) Standard ~ for quality control in the microbiological laboratory. Results of microbiological analyses are influenced by various factors apart from the material to be analysed. Two important factors are the analytical method that should be applied and the analyst's ability to perform it according to good laboratory practice. This quality assurance system permits to detect and to quantify methodological error components such as material influences and analyst's performance. Statistically analysed pilot studies have been carried out to test the performance of this quantitative technique. Both, collaborative studies and internal quality assurance demand suitable reference samples. Reference materials for quantitative microbiological tests have to meet certain requirements: 1. a sufficiently high level of contamination along with a low portion of aerobic spore-forming bacteria; 2. a homogeneous distribution of organisms both among the samples of a lot and in a homogenized sample; and 3. a stable contamination level and vari~,nce among the reference samples during a period of storage and transport. Whereas studies of the Community Bureau of Reference have dealt with artificially contaminated reference material (Mooijman et al., 1992; Heisterkamp et al., 1993), in the present study the suitability of a naturally contaminated material was tested. Results of homogeneity tests of a specially prepared material will be discussed. Furthermore, the suitability of this material for proficiency assessment and collaborative studies will be illustrated.
2. Material and method
Naturally contaminated lean meat emulsion was manufactured to carry out these experiments. The material was prepared according to the following recipe: 1.5 kg minced lean beef, 1.5 kg minced pork, 60 g NaC1 and 1.0 kg shattered ice. The lean meat emulsion was produced according to customary processing of frankfurter type sausage. Directly after production, the lean meat emulsion was subdivided into portions of 50 g, deep frozen and stored at - 18°C. Thus, four lots were produced. From each of the four lots of lean meat emulsion 50 parallel single
1 IDF-Questionnaire 1492/E: Statistical control in the microbiological laboratory: the plate count.
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
43
Structure of the Total Variance
Design of Trial Material: "Lean Meat Emulsion"
Variance among the Samples of a Lot -> Sample Error
Random Sampling: 50 Samples / Lot
1 10 g Sample Material + 90 ml Diluent Stomacher
+
Sample Homogenate Dilution 1:10
1
1
Series A Decimal Series
Series B / Decimal Series
l
1
Parall:l Plates
J
Variance between the Parallel Plates => Random Error (unavoidable)
Parallel Plates
Spread Plate Technique Plate-Count-Agar ncubat on: A e r o b / 3 0 ° C / 7 2 h
Variance between the Dilution S e r i e s of a Single Sample -) Methodological Error
/ Methodological Error
Determination of the Total Count ~ Total Variance Ideal Values= s2/~ • 1 (Untransformed Data) SZsr/~ • 0,25 (Square Root Transformed Data)
Fig. 1. Test of homogeneity.
samples of 10 g were analysed. The preparation of the samples was carried out according to the standards of §35 Lebensmittel- und Bedarfsgegenst~indegesetz. A special experimental design was developed in order to test the homogeneity of reference materials (Fig. 1). The hierarchical structure with parallel samples, parallel dilution series and parallel plates permits a detailed statistical evaluation. An analysis of variance of the square root transformed colony counts was performed in order to estimate the influence of the following three components on the total variance of the results. The sample error consists of the variance among the single samples of a lot additional to the unavoidable random error. This component of variance characterizes the homogeneity of a lot. The methodological error arises from technical inaccuracy and microbiological factors like overcrowding. This effect is detectable as a difference between the parallel dilution series of a single sample. If the analysis is carried out with a high level of proficiency, the value of this variance component will be nearly 0. -
-
44
C. Berg et aL / International Journal o f Food Microbiology 24 (1994) 41-52
The unauoidable (sampling or) random error appears as variation between the parallel plates of a single sample. If the microorganisms are randomly distributed in a homogenized sample the variance of the square root transformed colony counts on parallel plates will be 0.25. This value is considered as the ideal variance if perfect homogeneity of the prepared and diluted sample can be assumed.
3. Results
Collaborative studies can be evaluated statistically if plate counts of at least five successive twofold dilution stages of four independent dilution series are available. To get a sufficient large number colonies in different dilution stages, for methodological reasons a contamination level of the sample material no less than 1 million c f u / g would be necessary. Since the counts ranged from 6.3 to 9.8 × 106 cfu/g, the tested lots comply with this requirement (Table 1). After a storage of 6 weeks at -18°C, the microflora of a sample e.g. was composed as follows (cfu/g): Lactobacillaceae 1.8 × 10 4, Micrococcaceae 7.6 x l0 s, Pseudomonaceae 5.0 X 10 4, Enterobacteriaceae 6.2 x 103, Coliforms 3.4 × 103, Entercoccaceae 2.0 × 102, yeasts 2.0 × 102; the number of Staphylococcaceae, aerobic spore-forming bacteria and moulds was below the detectable level of 100 c f u / g . Standard deviations showed relatively low values for all four lots of lean meat emulsion ranging from 15% to 25%. Amongst the measures of localisation and dispersion the relation variance divided by the arithmetic mean gives considerable information about the variance among the samples of a lot. This value is independent of the level of the average colony count. If the random error is the only effect the total variance will correspond with the arithmetic mean. In this case the quotient is 1, which is equivalent to the ideal variance of 0.25 of the square root transformed data mentioned above. Quotients of variance and arithmetic mean ranged from 2.3 to 4.0. These results indicate that perfect homogeneity of the lots cannot be assumed, because the values were larger than the ideal quotient of 1. An additional variance among the single samples of a lot became noticeable.
Table 1 Measures of localisation and dispersion of the test of homogeneity of lean meat emulsion Lot
n
a 1 2 3 4
50 50 50 25
£
s
s2
s / ~ (%)
s2/y: b
Log-transformed colony counts
64 61 96 64
14.5 15.8 15.0 14.1
210.3 249.6 225.0 202.0
22 25 15 21
3.1 4.0 2.3 3.0
6.8 6.8 7.0 6.8
U n t r a n s f o r m e d colony counts
67 x 105 63 x 105 98 X 105 68 x 105
a Contamination level. b S / 2 , ratio: s 2 / £ = 1, if there is the random error only.
0.09 0A 1 0.06 0.10
C. Berg et aL / International Journal of Food Microbiology 24 (1994) 41-52
45
Total variance (absolute)
1,4 /
0,8 0,6 0,4 0,2 1
2
3
4 Lot
Verlanoe ot samples
Varlanae of series
Variance of plates Fig. 2. Effect of the variance components 'sample', 'dilution series' and 'parallel plates' on the total variance. Estimates for the variance of the square root transformed data.
The effect of each component of variation on the total variance calculated by analysis of variance is shown in Fig. 2. The absolute variation among the single samples of a lot ranged from 0.41 to 0.92 that corresponds with 51% to 70% of the total variance. So the variance component 'sample' has the most important effect on the total variance. The variance between the parallel dilution series of a single sample amounts to 0.04 to 0.11 from square root transformed data. So the methodological error has no significant influence on the total variance. Greater methodological effects of 0.41 in the last trial are due to the fact that the fourth lot was tested by different analysts, which were not so familiar with the experimental design. The variance of the transformed colony counts of parallel plates ranged from 0.26 to 0.32. Since these results correspond with the ideal variance of 0.25, the POISSON-distribution of the microorganisms in a homogenized sample of lean meat emulsion can be assumed. After testing the homogeneity of the four lots of lean meat emulsion two national collaborative studies were undertaken. At the same time the applicability of the draft of the quality assurance system was tested.
46
C. Berg et aL / International Journal of Food Microbiology 24 (1994) 41-52 Reference Material= Lean Meat Emulsion 10 g Sample Material Dilution 1;10
* 90 ml Diluent Stomacher 1 rain.
o
:o
10 ml Homogenized Sample
g cb o b 6
c5~5cb ~ ]
~ c5~6cb
. ,, ~:bcbcb ~
cbdbcb
Fig. 3. Quality assurance in the microbiological laboratory. The spread plate technique. Design of pilot studies. Twofold dilution steps.
The experimental design of the quality assurance method requires the preparation of four parallel dilution series from a given homogenized sample suspension, each series consisting of twelve twofold dilution steps (Fig. 3). From each dilution level three parallel plates should be inoculated. The colony counts on these plates after 72 h of incubation form the data set that will be analysed statistically. Concerning the counting procedure some remarks are necessary: experience shows that analysts knowing the origin of a parallel plate tend to come up with too similar counting results. Consequently, this would lead to an underestimation of the variation between the parallel plates. Therefore the plates should not be evaluated in the order in which they were prepared but they have to be coded and
C. Berget aL / InternationalJournal of Food Microbiology24 (1994) 41-52
47
arranged in a random order before counting by a person not involved in that task. Counting results than are entered in a given application form. From the decoded counting results five to six dilution levels are chosen for the statistical evaluation. The criteria for the exclusion of dilution steps are: (a) that there should be no levels with more than 300 colonies per plate (pour plate method), respectively, 400 colonies per plate (spread plate method) on the average in one series; if this would be the case microbiological reasons like overcrowding could lead to false results; (b) that the expected colony count per plate should not be lower than 5, since this would cause problems for the application of the statistical procedures. Thus, a single data set consists of 60 to 72 independent plate counts belonging to four independent dilution series with five to six successive twofold dilution steps. The statistical evaluation of the data sets can be performed using the computer programm ' G L P Analyst' (Fig. 4), a short description of the single steps is following. After entering the counting results the completeness and plausibility of the data set has to be checked. If the data set contains too many missing values an analysis and evaluation is not possible. The analyst performance assessment has to be repeated in such a case. Otherwise the homogeneity of the parallel plate results is checked using a likelihood-ratio-test (G2-test). If the counting results are 'too similar', the experiment has to be repeated as well and no further statistical evaluation is done. This may be a hint that the coding and counting in a random sequence of the plates did not happen according to the instructions. If results are not 'too similar', a succeeding overall G2-test examines the homogeneity of the whole dataset, this time under the aspect wether there is too much deviation in the counting results or not. In case of acceptable results the evaluation terminates, because the analyst's performance can be judged as 'satisfactory'. Otherwise an analysis of variance components is carried out using square root transformed data. The total variance is calculated as the sum of the following components: - the variance between the four parallel dilution series; - the variance between the twelve twofold dilution steps of each dilution series; and - the variance between the three parallel plates. The sum of the variance components is regarded as the criterion for the assessment of analyst's performance: o-2(total) = ~r2(dilution series) + ~r2(dilution steps) + ~r2(parallel plates) If the analyst's performance meets a good proficiency standard, the total variance will consist only of unavoidable random effects. In that ideal case parallel dilution series and dilution steps will be prepared without any methodical inaccuracy. Thus, the variance components 'dilution series' and 'dilution steps' amount approximately to 0. If the variance of the parallel plates is solely caused by the random error, the ideal value for square root transformed data will amount to 0.25. Additional methodological errors result in variance components of 'series' and 'steps' larger than 0 a n d / o r the variance component for 'plates' exceeds 0.25. According to the quality assessment procedure the proficiency standard will be judged as 'acceptable' if the total variance is less than 1. The value 1 - four times
48
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
(start)
1 data entry
l enough data for statistical analysis?
no )~
repeatthe triall
1
(end)
yes
are parallel plates too similar?
yes
results are "too good" ]
repeat the triall
] (end)
no are dilution series and dilution steps homogeneous?
[ res~ aro"acoepta~le"] (end)
no
t analysisof variance components: total variance_~ 1
[results~e ,=pO~le" ] (end)
no ~ results are "not tolerable" ]
i
exploratory analysis of structure: interpretation of results, detection of day-to-day errors / potential sources of failure
l (end) Fig. 4. Flow chart of the analysis (using the computer program 'GLP Analyst').
the ideal variance of 0.25 of the square root transformed data - is not a mathematically but an empirically determined limit. If the sum of variance components is greater than 1, the proficiency standard will be judged as 'unacceptable'. An exploratory analysis of the variance structure follows in order to determine the methodological weak points of the analyst's performance. The whole experiment should be repeated after investigation and correction of the analyst's errors.
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
49
Total v a r i a n c e (absolute) 2,5
,
n
o
o
l
I i1 , lll,l,,,il, Ii,,iI,ll ,i,, I
1
3 2
5 4
7
6
g
8
,
ii,
=
,,
t
,
i
11 43 45 47 19 24 23 25 27 29 31 33 35 37 3g 41 43 45 10 12 14 16 18 20 22 24 26 2 8 : 3 0 32 34 36 38 40 42 44 46
Laboratory m
Variance of plates
W
Variance of steps
~
Variance of series
Fig. 5. Results of the collaborative study in 4 / 9 3 with lean meat emulsion as reference material. Estimates for the variance of the square root transformed data. Results of the laboratories 18, 24, 33, 35 and 43 are unevaluable.
In the whole 46 laboratories took part in the second spread plate technique collaborative study in april 1993 (Fig. 5). Every participant received two samples of the standardized lean meat emulsion in order to analyse them according to the internal quality assurance method. If the samples arrived in a thawed condition, the laboratory should request another shipment. Twenty-nine laboratories met an 'acceptable' standard. The sums of the variance components of nine laboratories were greater than 1, the analyst's performance has to be judged as 'unacceptable'. The results of three laboratories were 'too good', results of further five laboratories were unevaluable. So the proficiency of these eight laboratories could not be assessed. In order to detect sources of errors the additional variations can also be illustrated by diagrams (Figs. 6-8). Provided that the colony forming units are homogeneous (randomly) distributed in the processed and diluted sample, it is possible to prepare four parallel dilution series starting from the final decimal dilution without methodological inaccuracy (Fig. 6). In the left diagram no differences between the parallel dilution series are detectable, whereas one can see significant variations in the right figure. Source of variation may be an insufficient preparation of the sample material or the material can't be homogenized properly. Improper mixing of the last preliminary dilution before preparing binary dilution series is also considered. As illustrated in the left diagram, exact serial twofold dilution steps were prepared (Fig. 7). In comparison the right figure shows additionally systematical differences between the dilution steps. Here the colony counts on the plates of the first dilution levels were lower than expected due to overcrowding effect's. Insuffi-
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
50
\
4,
J
|00
"~
5O
o
20
2,
200
-
lO0
•
20
-
JO 5
5"
I 2
I 3
I 4
i ~
i 5
I 7
Dilution
X
~ 3
I
r •
2""
I 5
i 6
Dilution
L 7
X 2 -=
Variance between the
Ideal
Dilution
Series
Fig. 6. Preparation of the parallel dilution series. Untransformed colony counts. Weighed averages.
400
2OO
ZOO •
a:
100
~ o
so
50
-
20
20
-~
8 lO 5 -
5
2
~
,
5 Dilution
Ideal
6,
x
'~
~
3
2 "=
,
56 DHu'tlon
Variance
between
Dilution
x 2""
the
Steps
Fig. 7. Preparation of the serial twofold dilution steps. Untransformed colony counts. Parallel plate averages.
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
51
400
,Ioo
20O
•
--q
100
2,0o "6 ~9
ao 10
1o" 4
I S
I 6
I 7
I B
I 9
Dilution
Ideal ->
[ 1
I a
I 3
2 "=
I
~ ~
I 6
Dilution
X 2"'
Additional Variance
Only Random Effects
between the Parallel Plates
Fig. 8. Inoculation of the parallel plates. Untransformed colony counts.
ciently mixing the suspension before preparing the next dilution step could be a reason for variation. Also sterile pipettes should be used for each dilution step. The colony counts of the three parallel plates of each evaluable dilution level are illustrated as the points in Fig. 8. As shown before on the left-hand side one sees the results of the accurate procedure - the discernible variation consists only of unavoidable r a n d o m effects. Additional differences between the parallel plates as visible on the right-hand side result if analysts didn't use a sterile glass spreader for each plate. Also counting errors can be assumed. In order to identify the reasons for the effects of variance a close co-operation of the supervisor and the laboratory t e a m is necessary.
4. Conclusions The following conclusions can be drawn due to the results of this testing: - The hierachical design of the trial is suitable to test the homogeneity of reference materials. - The quality assurance system permits to assess analyst's performance with respect to the standardized spread plate technique. - The sum of the variance components is regarded as the criterion for the assessment. - The limit for an acceptable standard of proficieny is 1, four times the ideal variance of 0.25. Using lean m e a t emulsion as naturally contaminated reference material reveals further advantages as follows: - Samples of this material can be p r e p a r e d and investigated as day-to-day sampies. - Sufficiently high levels of contamination can be reckoned with. - Easy manufacturing and storage allow internal quality control at every time.
52
C Berg et al. /International Journal of Food Microbiology 24 (1994) 41-52
The disadvantages of this material are as follows: - It is still difficult to compare the results of laboratories which take part in collaborative studies because a certain variance among the samples of a lot is unavoidable. - The extent of the sample variance has to be known. It may not be added to the methodological error. - Furthermore the contamination level of naturally contaminated material cannot be regulated. - The transport of deep frozen samples is complicated and expensive. Despite these critical points this naturally contaminated reference material can be used in order to test the precision of quantiative methods as well as laboratory proficiency under routine conditions. Since the distribution of microorganisms in a homogenized single sample of lean meat emulsion follows a POISSON-distribution, the tested material guarantees to keep an acceptable standard of proficiency in routine laboratories. The use of naturally or artificially contaminated material depends on whether the analyst's performance should be tested under routine or test conditions. The decision must be taken with regard to the intention of the collaborative study respectively the internal proficiency test.
Acknowledgements
Authors wish to thank the Deutsche Forschungsgemeinschaft (DFG) which has sponsored the project 'sample-variance'. For the evaluation and interpretation of the analyst performance assessment inside the laboratory the computer program ' G L P analyst' may be used. Further information concerning the program is provided by the authors.
References
BGA (Bundesgesundheitsamt, 1991) L. 01.00-00 Bestimmung der Keimzahl in Milch. GuBverfahren. Verfahren zur Qualitiitssicherung im Laboratorium. Amtliche Sammlung von Untersuchungsverfahren nach §35 LMBG. Beuth, Berlin/K61n. Heisterkamp, S.H., Hoekstra, J.A., Van Strijp-Lockefeer, N.G.W.M., Havelaar, A.H., Mooijman, K.A., In 't Veld, P.H., Notermans, S.H.W., Maier, E.A. and Griepink, B. (1993) Statistical analysis of certification trials for microbiological reference materials. BCR information, Report EUR 15008 EN (ISSN 1018-5593). Mooijman, K.A., In 't Veld, P.H., Hoekstra, J.A., Heisterkamp, S.H., Havelaar, A.H., Notermans, S.H.W., Roberts, D., Griepink, B. and Maier, E.A. (1992) Development of microbiologicalreference materials. BCR information, Report EUR 14375 EN (ISSN 1018-5593). Well3, H., Niemelii, S. and Arndt, G. (1991) Sicherung der Priizision standardisierter mikrobiologischer Untersuchungsverfahren. Biometrie und Informatik in Medizin und Biologie 22 (3), 116-135.
International Journal of Food Microbiology 24 (1994) 41-52
Review Paper
Microbiological collaborative studies for quality control in food laboratories: Reference material and evaluation of analyst's errors C. Berg *, S. Dahms, G. Hildebrandt, S. Klaschka, H. Weiss Institut fiir Lebensmittelhygiene, Institut fiir Biometrie und Informationsverarbeitung des Fachbereiches Veteriniirmedizin der Freien Universitiit Berlin, K6nigsweg 69, D-14163 Berlin, Germany
Abstract Working groups of the Federal Health Office in Germany and of the International Dairy Federation (IDF) have developed a quality assurance system to assess the analyst performance for colony count methods. The experiment design consists of several dilution series produced from a homogeneous sample suspension. Each series contains a number of twofold dilution steps with parallel plates on each level. The structure of this design permits a detailed analysis of the total variance and identification of analysts' methodological errors as single effects. A computer-based interpretation aid may classify the laboratory as working in correspondence with good laboratory practice ('acceptable') or, otherwise, furnishing 'to good' or 'unacceptable' results. Some examples are given for deviation depending on individual faults. Both, internal quality assurance and collaborative studies demand suitable reference samples. The reference material used has to meet certain requirements concerning homogeneity, contamination level and microbiological stability during a period of storage and transport. A naturally contaminated sample material was prepared, which has been used successfully in microbiological collaborative studies. Keywords: Quality assurance; Good laboratory practice; Analyst performance; Plate count
method; Analysis of variance; Reference material
1. Introduction I n o r d e r to a t t a i n r e l i a b l e e x a m i n a t i o n results in m i c r o b i o l o g i c a l quality control, it is n e c e s s a r y to a p p l y s t a n d a r d i z e d m e t h o d s . A d r a f t o f t h e s t a n d a r d ' Q u a l i t y
* Corresponding author. Tel. (30) 8108 2550. Fax (30) 8108 2552. 0168-1605/94/$07.00 © 1994 Elsevier Science B.V. All rights reserved SSDI 0168-1605(94)00132-4
42
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
control in the microbiological laboratory - the spread plate technique' was elaborated by the working group 'Statistical Evaluation of Microbiological Methods' of the German Federal Health Office according to official methods (Weiss et al., 1991). Nearly the same concept has been worked out for the pour-plate method which is part of the collection of official methods of the German Federal Health Office (BGA, 1991). It will also become International Dairy Federation (IDF) Standard ~ for quality control in the microbiological laboratory. Results of microbiological analyses are influenced by various factors apart from the material to be analysed. Two important factors are the analytical method that should be applied and the analyst's ability to perform it according to good laboratory practice. This quality assurance system permits to detect and to quantify methodological error components such as material influences and analyst's performance. Statistically analysed pilot studies have been carried out to test the performance of this quantitative technique. Both, collaborative studies and internal quality assurance demand suitable reference samples. Reference materials for quantitative microbiological tests have to meet certain requirements: 1. a sufficiently high level of contamination along with a low portion of aerobic spore-forming bacteria; 2. a homogeneous distribution of organisms both among the samples of a lot and in a homogenized sample; and 3. a stable contamination level and vari~,nce among the reference samples during a period of storage and transport. Whereas studies of the Community Bureau of Reference have dealt with artificially contaminated reference material (Mooijman et al., 1992; Heisterkamp et al., 1993), in the present study the suitability of a naturally contaminated material was tested. Results of homogeneity tests of a specially prepared material will be discussed. Furthermore, the suitability of this material for proficiency assessment and collaborative studies will be illustrated.
2. Material and method
Naturally contaminated lean meat emulsion was manufactured to carry out these experiments. The material was prepared according to the following recipe: 1.5 kg minced lean beef, 1.5 kg minced pork, 60 g NaC1 and 1.0 kg shattered ice. The lean meat emulsion was produced according to customary processing of frankfurter type sausage. Directly after production, the lean meat emulsion was subdivided into portions of 50 g, deep frozen and stored at - 18°C. Thus, four lots were produced. From each of the four lots of lean meat emulsion 50 parallel single
1 IDF-Questionnaire 1492/E: Statistical control in the microbiological laboratory: the plate count.
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
43
Structure of the Total Variance
Design of Trial Material: "Lean Meat Emulsion"
Variance among the Samples of a Lot -> Sample Error
Random Sampling: 50 Samples / Lot
1 10 g Sample Material + 90 ml Diluent Stomacher
+
Sample Homogenate Dilution 1:10
1
1
Series A Decimal Series
Series B / Decimal Series
l
1
Parall:l Plates
J
Variance between the Parallel Plates => Random Error (unavoidable)
Parallel Plates
Spread Plate Technique Plate-Count-Agar ncubat on: A e r o b / 3 0 ° C / 7 2 h
Variance between the Dilution S e r i e s of a Single Sample -) Methodological Error
/ Methodological Error
Determination of the Total Count ~ Total Variance Ideal Values= s2/~ • 1 (Untransformed Data) SZsr/~ • 0,25 (Square Root Transformed Data)
Fig. 1. Test of homogeneity.
samples of 10 g were analysed. The preparation of the samples was carried out according to the standards of §35 Lebensmittel- und Bedarfsgegenst~indegesetz. A special experimental design was developed in order to test the homogeneity of reference materials (Fig. 1). The hierarchical structure with parallel samples, parallel dilution series and parallel plates permits a detailed statistical evaluation. An analysis of variance of the square root transformed colony counts was performed in order to estimate the influence of the following three components on the total variance of the results. The sample error consists of the variance among the single samples of a lot additional to the unavoidable random error. This component of variance characterizes the homogeneity of a lot. The methodological error arises from technical inaccuracy and microbiological factors like overcrowding. This effect is detectable as a difference between the parallel dilution series of a single sample. If the analysis is carried out with a high level of proficiency, the value of this variance component will be nearly 0. -
-
44
C. Berg et aL / International Journal o f Food Microbiology 24 (1994) 41-52
The unauoidable (sampling or) random error appears as variation between the parallel plates of a single sample. If the microorganisms are randomly distributed in a homogenized sample the variance of the square root transformed colony counts on parallel plates will be 0.25. This value is considered as the ideal variance if perfect homogeneity of the prepared and diluted sample can be assumed.
3. Results
Collaborative studies can be evaluated statistically if plate counts of at least five successive twofold dilution stages of four independent dilution series are available. To get a sufficient large number colonies in different dilution stages, for methodological reasons a contamination level of the sample material no less than 1 million c f u / g would be necessary. Since the counts ranged from 6.3 to 9.8 × 106 cfu/g, the tested lots comply with this requirement (Table 1). After a storage of 6 weeks at -18°C, the microflora of a sample e.g. was composed as follows (cfu/g): Lactobacillaceae 1.8 × 10 4, Micrococcaceae 7.6 x l0 s, Pseudomonaceae 5.0 X 10 4, Enterobacteriaceae 6.2 x 103, Coliforms 3.4 × 103, Entercoccaceae 2.0 × 102, yeasts 2.0 × 102; the number of Staphylococcaceae, aerobic spore-forming bacteria and moulds was below the detectable level of 100 c f u / g . Standard deviations showed relatively low values for all four lots of lean meat emulsion ranging from 15% to 25%. Amongst the measures of localisation and dispersion the relation variance divided by the arithmetic mean gives considerable information about the variance among the samples of a lot. This value is independent of the level of the average colony count. If the random error is the only effect the total variance will correspond with the arithmetic mean. In this case the quotient is 1, which is equivalent to the ideal variance of 0.25 of the square root transformed data mentioned above. Quotients of variance and arithmetic mean ranged from 2.3 to 4.0. These results indicate that perfect homogeneity of the lots cannot be assumed, because the values were larger than the ideal quotient of 1. An additional variance among the single samples of a lot became noticeable.
Table 1 Measures of localisation and dispersion of the test of homogeneity of lean meat emulsion Lot
n
a 1 2 3 4
50 50 50 25
£
s
s2
s / ~ (%)
s2/y: b
Log-transformed colony counts
64 61 96 64
14.5 15.8 15.0 14.1
210.3 249.6 225.0 202.0
22 25 15 21
3.1 4.0 2.3 3.0
6.8 6.8 7.0 6.8
U n t r a n s f o r m e d colony counts
67 x 105 63 x 105 98 X 105 68 x 105
a Contamination level. b S / 2 , ratio: s 2 / £ = 1, if there is the random error only.
0.09 0A 1 0.06 0.10
C. Berg et aL / International Journal of Food Microbiology 24 (1994) 41-52
45
Total variance (absolute)
1,4 /
0,8 0,6 0,4 0,2 1
2
3
4 Lot
Verlanoe ot samples
Varlanae of series
Variance of plates Fig. 2. Effect of the variance components 'sample', 'dilution series' and 'parallel plates' on the total variance. Estimates for the variance of the square root transformed data.
The effect of each component of variation on the total variance calculated by analysis of variance is shown in Fig. 2. The absolute variation among the single samples of a lot ranged from 0.41 to 0.92 that corresponds with 51% to 70% of the total variance. So the variance component 'sample' has the most important effect on the total variance. The variance between the parallel dilution series of a single sample amounts to 0.04 to 0.11 from square root transformed data. So the methodological error has no significant influence on the total variance. Greater methodological effects of 0.41 in the last trial are due to the fact that the fourth lot was tested by different analysts, which were not so familiar with the experimental design. The variance of the transformed colony counts of parallel plates ranged from 0.26 to 0.32. Since these results correspond with the ideal variance of 0.25, the POISSON-distribution of the microorganisms in a homogenized sample of lean meat emulsion can be assumed. After testing the homogeneity of the four lots of lean meat emulsion two national collaborative studies were undertaken. At the same time the applicability of the draft of the quality assurance system was tested.
46
C. Berg et aL / International Journal of Food Microbiology 24 (1994) 41-52 Reference Material= Lean Meat Emulsion 10 g Sample Material Dilution 1;10
* 90 ml Diluent Stomacher 1 rain.
o
:o
10 ml Homogenized Sample
g cb o b 6
c5~5cb ~ ]
~ c5~6cb
. ,, ~:bcbcb ~
cbdbcb
Fig. 3. Quality assurance in the microbiological laboratory. The spread plate technique. Design of pilot studies. Twofold dilution steps.
The experimental design of the quality assurance method requires the preparation of four parallel dilution series from a given homogenized sample suspension, each series consisting of twelve twofold dilution steps (Fig. 3). From each dilution level three parallel plates should be inoculated. The colony counts on these plates after 72 h of incubation form the data set that will be analysed statistically. Concerning the counting procedure some remarks are necessary: experience shows that analysts knowing the origin of a parallel plate tend to come up with too similar counting results. Consequently, this would lead to an underestimation of the variation between the parallel plates. Therefore the plates should not be evaluated in the order in which they were prepared but they have to be coded and
C. Berget aL / InternationalJournal of Food Microbiology24 (1994) 41-52
47
arranged in a random order before counting by a person not involved in that task. Counting results than are entered in a given application form. From the decoded counting results five to six dilution levels are chosen for the statistical evaluation. The criteria for the exclusion of dilution steps are: (a) that there should be no levels with more than 300 colonies per plate (pour plate method), respectively, 400 colonies per plate (spread plate method) on the average in one series; if this would be the case microbiological reasons like overcrowding could lead to false results; (b) that the expected colony count per plate should not be lower than 5, since this would cause problems for the application of the statistical procedures. Thus, a single data set consists of 60 to 72 independent plate counts belonging to four independent dilution series with five to six successive twofold dilution steps. The statistical evaluation of the data sets can be performed using the computer programm ' G L P Analyst' (Fig. 4), a short description of the single steps is following. After entering the counting results the completeness and plausibility of the data set has to be checked. If the data set contains too many missing values an analysis and evaluation is not possible. The analyst performance assessment has to be repeated in such a case. Otherwise the homogeneity of the parallel plate results is checked using a likelihood-ratio-test (G2-test). If the counting results are 'too similar', the experiment has to be repeated as well and no further statistical evaluation is done. This may be a hint that the coding and counting in a random sequence of the plates did not happen according to the instructions. If results are not 'too similar', a succeeding overall G2-test examines the homogeneity of the whole dataset, this time under the aspect wether there is too much deviation in the counting results or not. In case of acceptable results the evaluation terminates, because the analyst's performance can be judged as 'satisfactory'. Otherwise an analysis of variance components is carried out using square root transformed data. The total variance is calculated as the sum of the following components: - the variance between the four parallel dilution series; - the variance between the twelve twofold dilution steps of each dilution series; and - the variance between the three parallel plates. The sum of the variance components is regarded as the criterion for the assessment of analyst's performance: o-2(total) = ~r2(dilution series) + ~r2(dilution steps) + ~r2(parallel plates) If the analyst's performance meets a good proficiency standard, the total variance will consist only of unavoidable random effects. In that ideal case parallel dilution series and dilution steps will be prepared without any methodical inaccuracy. Thus, the variance components 'dilution series' and 'dilution steps' amount approximately to 0. If the variance of the parallel plates is solely caused by the random error, the ideal value for square root transformed data will amount to 0.25. Additional methodological errors result in variance components of 'series' and 'steps' larger than 0 a n d / o r the variance component for 'plates' exceeds 0.25. According to the quality assessment procedure the proficiency standard will be judged as 'acceptable' if the total variance is less than 1. The value 1 - four times
48
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
(start)
1 data entry
l enough data for statistical analysis?
no )~
repeatthe triall
1
(end)
yes
are parallel plates too similar?
yes
results are "too good" ]
repeat the triall
] (end)
no are dilution series and dilution steps homogeneous?
[ res~ aro"acoepta~le"] (end)
no
t analysisof variance components: total variance_~ 1
[results~e ,=pO~le" ] (end)
no ~ results are "not tolerable" ]
i
exploratory analysis of structure: interpretation of results, detection of day-to-day errors / potential sources of failure
l (end) Fig. 4. Flow chart of the analysis (using the computer program 'GLP Analyst').
the ideal variance of 0.25 of the square root transformed data - is not a mathematically but an empirically determined limit. If the sum of variance components is greater than 1, the proficiency standard will be judged as 'unacceptable'. An exploratory analysis of the variance structure follows in order to determine the methodological weak points of the analyst's performance. The whole experiment should be repeated after investigation and correction of the analyst's errors.
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
49
Total v a r i a n c e (absolute) 2,5
,
n
o
o
l
I i1 , lll,l,,,il, Ii,,iI,ll ,i,, I
1
3 2
5 4
7
6
g
8
,
ii,
=
,,
t
,
i
11 43 45 47 19 24 23 25 27 29 31 33 35 37 3g 41 43 45 10 12 14 16 18 20 22 24 26 2 8 : 3 0 32 34 36 38 40 42 44 46
Laboratory m
Variance of plates
W
Variance of steps
~
Variance of series
Fig. 5. Results of the collaborative study in 4 / 9 3 with lean meat emulsion as reference material. Estimates for the variance of the square root transformed data. Results of the laboratories 18, 24, 33, 35 and 43 are unevaluable.
In the whole 46 laboratories took part in the second spread plate technique collaborative study in april 1993 (Fig. 5). Every participant received two samples of the standardized lean meat emulsion in order to analyse them according to the internal quality assurance method. If the samples arrived in a thawed condition, the laboratory should request another shipment. Twenty-nine laboratories met an 'acceptable' standard. The sums of the variance components of nine laboratories were greater than 1, the analyst's performance has to be judged as 'unacceptable'. The results of three laboratories were 'too good', results of further five laboratories were unevaluable. So the proficiency of these eight laboratories could not be assessed. In order to detect sources of errors the additional variations can also be illustrated by diagrams (Figs. 6-8). Provided that the colony forming units are homogeneous (randomly) distributed in the processed and diluted sample, it is possible to prepare four parallel dilution series starting from the final decimal dilution without methodological inaccuracy (Fig. 6). In the left diagram no differences between the parallel dilution series are detectable, whereas one can see significant variations in the right figure. Source of variation may be an insufficient preparation of the sample material or the material can't be homogenized properly. Improper mixing of the last preliminary dilution before preparing binary dilution series is also considered. As illustrated in the left diagram, exact serial twofold dilution steps were prepared (Fig. 7). In comparison the right figure shows additionally systematical differences between the dilution steps. Here the colony counts on the plates of the first dilution levels were lower than expected due to overcrowding effect's. Insuffi-
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
50
\
4,
J
|00
"~
5O
o
20
2,
200
-
lO0
•
20
-
JO 5
5"
I 2
I 3
I 4
i ~
i 5
I 7
Dilution
X
~ 3
I
r •
2""
I 5
i 6
Dilution
L 7
X 2 -=
Variance between the
Ideal
Dilution
Series
Fig. 6. Preparation of the parallel dilution series. Untransformed colony counts. Weighed averages.
400
2OO
ZOO •
a:
100
~ o
so
50
-
20
20
-~
8 lO 5 -
5
2
~
,
5 Dilution
Ideal
6,
x
'~
~
3
2 "=
,
56 DHu'tlon
Variance
between
Dilution
x 2""
the
Steps
Fig. 7. Preparation of the serial twofold dilution steps. Untransformed colony counts. Parallel plate averages.
C. Berg et al. / International Journal of Food Microbiology 24 (1994) 41-52
51
400
,Ioo
20O
•
--q
100
2,0o "6 ~9
ao 10
1o" 4
I S
I 6
I 7
I B
I 9
Dilution
Ideal ->
[ 1
I a
I 3
2 "=
I
~ ~
I 6
Dilution
X 2"'
Additional Variance
Only Random Effects
between the Parallel Plates
Fig. 8. Inoculation of the parallel plates. Untransformed colony counts.
ciently mixing the suspension before preparing the next dilution step could be a reason for variation. Also sterile pipettes should be used for each dilution step. The colony counts of the three parallel plates of each evaluable dilution level are illustrated as the points in Fig. 8. As shown before on the left-hand side one sees the results of the accurate procedure - the discernible variation consists only of unavoidable r a n d o m effects. Additional differences between the parallel plates as visible on the right-hand side result if analysts didn't use a sterile glass spreader for each plate. Also counting errors can be assumed. In order to identify the reasons for the effects of variance a close co-operation of the supervisor and the laboratory t e a m is necessary.
4. Conclusions The following conclusions can be drawn due to the results of this testing: - The hierachical design of the trial is suitable to test the homogeneity of reference materials. - The quality assurance system permits to assess analyst's performance with respect to the standardized spread plate technique. - The sum of the variance components is regarded as the criterion for the assessment. - The limit for an acceptable standard of proficieny is 1, four times the ideal variance of 0.25. Using lean m e a t emulsion as naturally contaminated reference material reveals further advantages as follows: - Samples of this material can be p r e p a r e d and investigated as day-to-day sampies. - Sufficiently high levels of contamination can be reckoned with. - Easy manufacturing and storage allow internal quality control at every time.
52
C Berg et al. /International Journal of Food Microbiology 24 (1994) 41-52
The disadvantages of this material are as follows: - It is still difficult to compare the results of laboratories which take part in collaborative studies because a certain variance among the samples of a lot is unavoidable. - The extent of the sample variance has to be known. It may not be added to the methodological error. - Furthermore the contamination level of naturally contaminated material cannot be regulated. - The transport of deep frozen samples is complicated and expensive. Despite these critical points this naturally contaminated reference material can be used in order to test the precision of quantiative methods as well as laboratory proficiency under routine conditions. Since the distribution of microorganisms in a homogenized single sample of lean meat emulsion follows a POISSON-distribution, the tested material guarantees to keep an acceptable standard of proficiency in routine laboratories. The use of naturally or artificially contaminated material depends on whether the analyst's performance should be tested under routine or test conditions. The decision must be taken with regard to the intention of the collaborative study respectively the internal proficiency test.
Acknowledgements
Authors wish to thank the Deutsche Forschungsgemeinschaft (DFG) which has sponsored the project 'sample-variance'. For the evaluation and interpretation of the analyst performance assessment inside the laboratory the computer program ' G L P analyst' may be used. Further information concerning the program is provided by the authors.
References
BGA (Bundesgesundheitsamt, 1991) L. 01.00-00 Bestimmung der Keimzahl in Milch. GuBverfahren. Verfahren zur Qualitiitssicherung im Laboratorium. Amtliche Sammlung von Untersuchungsverfahren nach §35 LMBG. Beuth, Berlin/K61n. Heisterkamp, S.H., Hoekstra, J.A., Van Strijp-Lockefeer, N.G.W.M., Havelaar, A.H., Mooijman, K.A., In 't Veld, P.H., Notermans, S.H.W., Maier, E.A. and Griepink, B. (1993) Statistical analysis of certification trials for microbiological reference materials. BCR information, Report EUR 15008 EN (ISSN 1018-5593). Mooijman, K.A., In 't Veld, P.H., Hoekstra, J.A., Heisterkamp, S.H., Havelaar, A.H., Notermans, S.H.W., Roberts, D., Griepink, B. and Maier, E.A. (1992) Development of microbiologicalreference materials. BCR information, Report EUR 14375 EN (ISSN 1018-5593). Well3, H., Niemelii, S. and Arndt, G. (1991) Sicherung der Priizision standardisierter mikrobiologischer Untersuchungsverfahren. Biometrie und Informatik in Medizin und Biologie 22 (3), 116-135.