- Email: [email protected]
Harmonized protocol for the design and interpretation of collaborative studies
118
trends in analytical chemistry, vql. 7, no. 4,1988
Harmonized protocol for the design and interpretation of collaborative studies William Hotwitz Washington,DC, U.S.A.
Representatives of twenty international organizations have agreed on a harmonized protocol for the design of collaborative studies that utilizes at least eight laboratories, and five materials. Withinlaboratory repeatability is obtained preferably by using split-level and/or blind duplicate test samples. The components of variance are obtained by one-way analysis of variance after removal of outHers by the Cochran extreme variance test and the Grubbs extreme value tests at a probability level of 0.01. Estimates of the mean, relative standard deviations (within- and between-laboratories), and repeatability and reproducibility values are reported before and after removal of outliers.
Introduction A collaborative study is a set of analyses conducted by a number of laboratories on identical portions of the same materials by exactly the same method in order to determine the operating characteristics of that method. Unfortunately, the term has been applied indiscriminately to all types of interlaboratory studies, making it necessary to distinguish collaborative studies from two other major types of interlaboratory studies. A proficiency study is a set of analyses conducted by a number of laboratories on identical portions of the same material by any method for the purpose of demonstrating the performance of the laboratories (or analysts). A certification study is a set of analyses conducted by a number of laboratories for the purpose of assigning a value for an analyte or a property of a material. Although these three types of interlaboratory studies have much in common, each emphasizes a different factor: the collaborative study places the emphasis on the method; the proficiency study places the emphasis on the laboratory or the analysts; and the certification study places the emphasis on the material.
Collaborative studies are of primary importance to laboratories undertaking analyses that check for compliance with statutes dealing with public health, agricultural commodities, and environmental concerns. The pertinent law or regulation often specifies the method to be used for the analysis. Proficiency studies are important to those laboratories engaged in clinical chemistry and in determining the economic value of various materials. These areas often have extensive voluntary or compulsory quality assurance programs in operation. The choice of method is left up to the laboratory. Certification studies are usually performed by official or professional organizations with a responsibility to produce reference materials. They usually require the participation of several laboratories applying different test methods in order to establish the reference value of an analyte or property* International workshop on harmonization of collaborative studies A group of 27 individuals representing twenty international organizations with major programs using
interlaboratory studies met in Geneva on May 4-5, 1987, to harmonize their protocols for the design, conduct, and interpretation of collaborative studies. This working group was organized by the International Union of Pure and Applied Chemistry (IUPAC), and the meeting was conducted on the premises of the International Organization for Standardization (ISO) in Geneva, Switzerland. A discussion paper previously developed and circulated by William Horwitz of the Food and Drug Ad(Washington, DC, ministration U.S.A.), who serves as Chairman of the Committee on Interlaboratory Studies of the Association of the Official Analytical Chemists (AOAC), was used as the basis for the final document. The discussion paper identified thirteen points that required agreement in order to obtain a harmonized protocol for the design and interpretation of collaborative studies. Background The purpose of arriving at a harmonized protocol was to permit a collaborative study conducted by any of the adhering organizations to be valid for any other organizations. Because collaborative studies require an investment of $25 OOO75 000, the organizations have a considerable economic incentive to avoid duplication of effort in their programs that validate methods of analysis. This meeting was the culmination of a program initiated by the late Harold Egan, Government Chemist of the United Kingdom. Egan and Horwitz, as representatives of the United Kingdom and the United States, respectively, to the Codex Committee on Milk and Milk Products and later to the corresponding Codex Committee on Methods of Analysis and Sampling, were frustrated by the constant submission of unvalidated methods of analysis for approval by the Codex Alimentarius
119
trends in analytical chemistry, vol. 7, no. 4, I988
Commission of the Food and Agriculture Organization (FAO)/W&ld Health Organization (WHO) Food Standards Program. They subsequently developed terms of reference for the Codex Committee on Methods of Analysis and Sampling which required the submission of the results of a successful collaborative study as a prerequisite for acceptance of the method of analysis by the Codex Committee. The imposition of this requirement provided the impetus for international organizations to supply collaboratively studied methods of analysis for enforcement of the provisions of Codex commodity standards. Egan organized the first meeting of 20 interested international organizations, which took place at the Royal Society in London on March 9-10, 1978, and was entitled the International Symposium on Harmonization of Collaborative Analytical Studies. The organizations outlined their ongoing programs to validate analytical methods through collaborative studies. This meeting was followed by an international symposium on harmonization of collaborative analytical studies in Helsinki, Finland, on August 20-21, 1981’. A second symposium2 was held in Washington, DC, U.S.A., on the premises of the National Academy of Sciences on October 25-27, 1984, in connection with the 100th Anniversary of the founding of the AOAC. This symposium too was planned by Egan but his untimely death in June of 1984 required that his plans be implemented by L. Coles, H. Frehse, and W. Horwitz. The conclusions reached at this symposium resulted in the Geneva meeting and the acceptance of the recommendations required for harmonization. Details of the protocol (1) Preliminary intralaboratory work
The group emphasized the importance of properly validating and optimizing a candidate method within a single laboratory before organizing a collaborative study. It recommended the use of a long-term within-laboratory precision parameter which is intermediate between the customary
repeatability within-laboratory (minimum variability) and the between-laboratory reproducibility (maximum variability) precision parameters as defined by IS03. (2) Symbols, ventions
terminology,
and con-
The group provided a set of symbols and terms for common use. It recommended the term “relative standard deviation” (RSD) in preference to “coefficient of variation” and provided a set of mnemonic (in English) symbols for number of replicates (k), laboratories (L), and materials (m). The number of significant figures to be reported should be based on the output of the measuring instrument. Standard deviations and RSDs should ordinarily be reported with only two significant figures.
(3) Minimum size of a collaborative study
The minimum number of materials in a collaborative study should be five. If a method is used only at a single level specification, this may be reduced to an absolute minimum of three. The minimum number of participating laboratories is eight. If it is impossible to obtain this many, as in the case of specialized experience or instruments, the study may be conducted with five, with the accompanying penalty that the confidence limits of the precision are expanded. The group preferred obtaining within-laboratory precision by means of a split-level design, using two test samples of nearly the same concentration level to prevent any censoring of the results by wellmeaning analysts. Other acceptable designs include a mixture of split levels and blind duplicates, blind duplicates alone (but this fact should not be announced beforehand), and the double split-level (two known duplicates at each of the split levels). As a final resort, known duplicates and quality control materials analyzed within each laboratory, independent of the collaborative study, may be used.
(4) Statistical analysis
The results must be analyzed by a one-way analysis of variance (ANOVA) (material-by-material). A more complex statistical analysis is not precluded, but the components of variance are to be estimated from the fundamental one-way ANOVA. The precision estimates are to be calculated in two ways: (a) with no outliers removed; and (b) with outliers removed by the following specific harmonized protocol. Calculate the mean, repeatability (within-laboratory) relative standard deviation (RSD,), and reproducibility (between-laboratory) relative standard deviation (RSD,), with no outlying laboratories removed, but using only valid data. (Data are valid if the analyst has no reason to suspect that anything has gone wrong with the analysis. Invalid results may result from malfunctioning instruments, not following the method, the occurrence of unexpected reactions, not meeting systems suitability tests, and similar disqualifying events.) Remove any laboratories that are flagged by the Cochran extreme variance test and by the Grubbs or paired Grubbs (two outliers flagged simultaneously) extreme value tests at the probability level of 1%. If outliers are removed, repeat the outlier tests, but do not remove more than 22.2% (2 of 9) of the laboratories. Recalculate and report the parameters with and without the flagged laboratories removed. In addition, report the IS0 quality control parameters of repeatability (r) and reproducibility (R) values, which indicate how closely two values from the same laboratory (r) or different laboratories (R) should agree with each other 95% of the time. Other specific recommendations required for harmonization are the following: if the estimate of the within-laboratory standard deviation (s,) is greater than the between-laboratory standard deviation (sR), make them equal. Calculate percent recovery as (total analyte found - analyte originally present) x 100 divided by the amount of analyte added. Report analytical results uncorrected for recovery. Report recoveries separately.
trends in analytical chemistry, vol. 7, no. 4, 1988
120
References
Future work The participants recommended that the task of harmonization be extended to other aspects of the practice of analytical chemistry, such as quality control. Copies of the report may be obtained from Dr. William Horwitz, HFF-7, Food and Drug Administration, Washington, DC 20204, U.S.A. A protocol for the design, conduct, and interpretation of collaborative studies based on these recommendations will be published in Pure and Applied
Chemistry.
H. Egan and T. S. West (Editors), Cob laborative Interlaboratory Studies in Chemical Analyses, Pergamon Press,
Oxford, 1982, p. 171. J. Assoc. Off. Anal. Chem., 69 (1986)
391-436. IS0 5725-1986, International zation for Standardization, Switzerland.
OrganiGeneva,
William Horwitz is the Scientific Advisor to the Director, Center for Food Safety and Applied Nutrition, Food and Drug Administration, HFF-7, Washington, DC 20204, U.S.A.
in the
news News from the Chemometrics Society
J
THECHEMOMETR’CSSOC’ETY ~
In 1987 elections were held to choose a new President and a new Secretary of the Chemometrics Society accord-
l Secretary: Dr. Wolfhard Wegscheider, Institut fur Analytische Chemie, Mikro- und Radiochemie, Technische Universitat Graz, Technikerstrasse 4, A-8010 Graz, Austria. Participation in the vote was very high, and both officers were almost unanimously elected. As the elections were held after a considerable delay, I therefore suggest that the officers are appointed for the trienniurn 1988-1990. It is my pleasure to welcome the new officers, and I am certain that through their scientific prestige and their organizational capacities, the Chemometrics Society will enjoy a very profitable trienniurn. I congratulate the newly formed National Sections; some news I have received recently makes me hopeful that new national groups will very soon join the Chemometrics Society, thereby increasing its standing and professional base, and bringing it more in accordance with the aims for which it has been constituted.
ing to the decision of the Assembly of the Society at the Lertcl Meeting in 1986. The results . Prof.of the polls were:
prasi;ent: Dr; ;.,,,,,,.
sart, Vrije Universiteit Brussel, Faculteit der Geneeskunde en Farmatie, Farmaceutisch Instituut, Laarbeeklaan 103, B-1090 Brussels, Belgium;
MICHELE FORINA
Professor Michele Forina is the outgoing President of the Chemometrics Society.
FORADVERTISINGINFORMATIONPLEASECONTACTOUR ADVERTISINGREPRESENTATIVES JAPAN
USA/CANADA
ESP- Tokyo Branch
Michael Baer 50 East 42nd NEW
YORK,
Street,
Suite
504
Mr H. Ogura 28-1 Yushtma.
NY 10017
3-chome,
Tel: t2121682-2200
TOKYO
Telex:
Tel: 103) 836 0810
226000
ur m.haer/synergtstic
Telex:
GREAT
Mr M. White 30-32
Son Ltd.
or MS A. Malcolm
Southampton
LONDON
WCZE
Street 7HR
Tel: 101) 240 2032 Telex:
299181
02657617
REST OF WORLD
BRITAIN
T.G. Scott 8
Bunkyo-Ku
113
adsale/g
Fax: 1011 379 7155
ELSEVIER SCIENCEPUBLISHERS MS W. van Cattenhurch p.0. Box 211 1000 AE AMSTERDAM The Netherlands Tel: 120) 5803.714/715/721 Telex:
18582 espa/nl
Fax: (20) 5803.769
trends in analytical chemistry, vql. 7, no. 4,1988
Harmonized protocol for the design and interpretation of collaborative studies William Hotwitz Washington,DC, U.S.A.
Representatives of twenty international organizations have agreed on a harmonized protocol for the design of collaborative studies that utilizes at least eight laboratories, and five materials. Withinlaboratory repeatability is obtained preferably by using split-level and/or blind duplicate test samples. The components of variance are obtained by one-way analysis of variance after removal of outHers by the Cochran extreme variance test and the Grubbs extreme value tests at a probability level of 0.01. Estimates of the mean, relative standard deviations (within- and between-laboratories), and repeatability and reproducibility values are reported before and after removal of outliers.
Introduction A collaborative study is a set of analyses conducted by a number of laboratories on identical portions of the same materials by exactly the same method in order to determine the operating characteristics of that method. Unfortunately, the term has been applied indiscriminately to all types of interlaboratory studies, making it necessary to distinguish collaborative studies from two other major types of interlaboratory studies. A proficiency study is a set of analyses conducted by a number of laboratories on identical portions of the same material by any method for the purpose of demonstrating the performance of the laboratories (or analysts). A certification study is a set of analyses conducted by a number of laboratories for the purpose of assigning a value for an analyte or a property of a material. Although these three types of interlaboratory studies have much in common, each emphasizes a different factor: the collaborative study places the emphasis on the method; the proficiency study places the emphasis on the laboratory or the analysts; and the certification study places the emphasis on the material.
Collaborative studies are of primary importance to laboratories undertaking analyses that check for compliance with statutes dealing with public health, agricultural commodities, and environmental concerns. The pertinent law or regulation often specifies the method to be used for the analysis. Proficiency studies are important to those laboratories engaged in clinical chemistry and in determining the economic value of various materials. These areas often have extensive voluntary or compulsory quality assurance programs in operation. The choice of method is left up to the laboratory. Certification studies are usually performed by official or professional organizations with a responsibility to produce reference materials. They usually require the participation of several laboratories applying different test methods in order to establish the reference value of an analyte or property* International workshop on harmonization of collaborative studies A group of 27 individuals representing twenty international organizations with major programs using
interlaboratory studies met in Geneva on May 4-5, 1987, to harmonize their protocols for the design, conduct, and interpretation of collaborative studies. This working group was organized by the International Union of Pure and Applied Chemistry (IUPAC), and the meeting was conducted on the premises of the International Organization for Standardization (ISO) in Geneva, Switzerland. A discussion paper previously developed and circulated by William Horwitz of the Food and Drug Ad(Washington, DC, ministration U.S.A.), who serves as Chairman of the Committee on Interlaboratory Studies of the Association of the Official Analytical Chemists (AOAC), was used as the basis for the final document. The discussion paper identified thirteen points that required agreement in order to obtain a harmonized protocol for the design and interpretation of collaborative studies. Background The purpose of arriving at a harmonized protocol was to permit a collaborative study conducted by any of the adhering organizations to be valid for any other organizations. Because collaborative studies require an investment of $25 OOO75 000, the organizations have a considerable economic incentive to avoid duplication of effort in their programs that validate methods of analysis. This meeting was the culmination of a program initiated by the late Harold Egan, Government Chemist of the United Kingdom. Egan and Horwitz, as representatives of the United Kingdom and the United States, respectively, to the Codex Committee on Milk and Milk Products and later to the corresponding Codex Committee on Methods of Analysis and Sampling, were frustrated by the constant submission of unvalidated methods of analysis for approval by the Codex Alimentarius
119
trends in analytical chemistry, vol. 7, no. 4, I988
Commission of the Food and Agriculture Organization (FAO)/W&ld Health Organization (WHO) Food Standards Program. They subsequently developed terms of reference for the Codex Committee on Methods of Analysis and Sampling which required the submission of the results of a successful collaborative study as a prerequisite for acceptance of the method of analysis by the Codex Committee. The imposition of this requirement provided the impetus for international organizations to supply collaboratively studied methods of analysis for enforcement of the provisions of Codex commodity standards. Egan organized the first meeting of 20 interested international organizations, which took place at the Royal Society in London on March 9-10, 1978, and was entitled the International Symposium on Harmonization of Collaborative Analytical Studies. The organizations outlined their ongoing programs to validate analytical methods through collaborative studies. This meeting was followed by an international symposium on harmonization of collaborative analytical studies in Helsinki, Finland, on August 20-21, 1981’. A second symposium2 was held in Washington, DC, U.S.A., on the premises of the National Academy of Sciences on October 25-27, 1984, in connection with the 100th Anniversary of the founding of the AOAC. This symposium too was planned by Egan but his untimely death in June of 1984 required that his plans be implemented by L. Coles, H. Frehse, and W. Horwitz. The conclusions reached at this symposium resulted in the Geneva meeting and the acceptance of the recommendations required for harmonization. Details of the protocol (1) Preliminary intralaboratory work
The group emphasized the importance of properly validating and optimizing a candidate method within a single laboratory before organizing a collaborative study. It recommended the use of a long-term within-laboratory precision parameter which is intermediate between the customary
repeatability within-laboratory (minimum variability) and the between-laboratory reproducibility (maximum variability) precision parameters as defined by IS03. (2) Symbols, ventions
terminology,
and con-
The group provided a set of symbols and terms for common use. It recommended the term “relative standard deviation” (RSD) in preference to “coefficient of variation” and provided a set of mnemonic (in English) symbols for number of replicates (k), laboratories (L), and materials (m). The number of significant figures to be reported should be based on the output of the measuring instrument. Standard deviations and RSDs should ordinarily be reported with only two significant figures.
(3) Minimum size of a collaborative study
The minimum number of materials in a collaborative study should be five. If a method is used only at a single level specification, this may be reduced to an absolute minimum of three. The minimum number of participating laboratories is eight. If it is impossible to obtain this many, as in the case of specialized experience or instruments, the study may be conducted with five, with the accompanying penalty that the confidence limits of the precision are expanded. The group preferred obtaining within-laboratory precision by means of a split-level design, using two test samples of nearly the same concentration level to prevent any censoring of the results by wellmeaning analysts. Other acceptable designs include a mixture of split levels and blind duplicates, blind duplicates alone (but this fact should not be announced beforehand), and the double split-level (two known duplicates at each of the split levels). As a final resort, known duplicates and quality control materials analyzed within each laboratory, independent of the collaborative study, may be used.
(4) Statistical analysis
The results must be analyzed by a one-way analysis of variance (ANOVA) (material-by-material). A more complex statistical analysis is not precluded, but the components of variance are to be estimated from the fundamental one-way ANOVA. The precision estimates are to be calculated in two ways: (a) with no outliers removed; and (b) with outliers removed by the following specific harmonized protocol. Calculate the mean, repeatability (within-laboratory) relative standard deviation (RSD,), and reproducibility (between-laboratory) relative standard deviation (RSD,), with no outlying laboratories removed, but using only valid data. (Data are valid if the analyst has no reason to suspect that anything has gone wrong with the analysis. Invalid results may result from malfunctioning instruments, not following the method, the occurrence of unexpected reactions, not meeting systems suitability tests, and similar disqualifying events.) Remove any laboratories that are flagged by the Cochran extreme variance test and by the Grubbs or paired Grubbs (two outliers flagged simultaneously) extreme value tests at the probability level of 1%. If outliers are removed, repeat the outlier tests, but do not remove more than 22.2% (2 of 9) of the laboratories. Recalculate and report the parameters with and without the flagged laboratories removed. In addition, report the IS0 quality control parameters of repeatability (r) and reproducibility (R) values, which indicate how closely two values from the same laboratory (r) or different laboratories (R) should agree with each other 95% of the time. Other specific recommendations required for harmonization are the following: if the estimate of the within-laboratory standard deviation (s,) is greater than the between-laboratory standard deviation (sR), make them equal. Calculate percent recovery as (total analyte found - analyte originally present) x 100 divided by the amount of analyte added. Report analytical results uncorrected for recovery. Report recoveries separately.
trends in analytical chemistry, vol. 7, no. 4, 1988
120
References
Future work The participants recommended that the task of harmonization be extended to other aspects of the practice of analytical chemistry, such as quality control. Copies of the report may be obtained from Dr. William Horwitz, HFF-7, Food and Drug Administration, Washington, DC 20204, U.S.A. A protocol for the design, conduct, and interpretation of collaborative studies based on these recommendations will be published in Pure and Applied
Chemistry.
H. Egan and T. S. West (Editors), Cob laborative Interlaboratory Studies in Chemical Analyses, Pergamon Press,
Oxford, 1982, p. 171. J. Assoc. Off. Anal. Chem., 69 (1986)
391-436. IS0 5725-1986, International zation for Standardization, Switzerland.
OrganiGeneva,
William Horwitz is the Scientific Advisor to the Director, Center for Food Safety and Applied Nutrition, Food and Drug Administration, HFF-7, Washington, DC 20204, U.S.A.
in the
news News from the Chemometrics Society
J
THECHEMOMETR’CSSOC’ETY ~
In 1987 elections were held to choose a new President and a new Secretary of the Chemometrics Society accord-
l Secretary: Dr. Wolfhard Wegscheider, Institut fur Analytische Chemie, Mikro- und Radiochemie, Technische Universitat Graz, Technikerstrasse 4, A-8010 Graz, Austria. Participation in the vote was very high, and both officers were almost unanimously elected. As the elections were held after a considerable delay, I therefore suggest that the officers are appointed for the trienniurn 1988-1990. It is my pleasure to welcome the new officers, and I am certain that through their scientific prestige and their organizational capacities, the Chemometrics Society will enjoy a very profitable trienniurn. I congratulate the newly formed National Sections; some news I have received recently makes me hopeful that new national groups will very soon join the Chemometrics Society, thereby increasing its standing and professional base, and bringing it more in accordance with the aims for which it has been constituted.
ing to the decision of the Assembly of the Society at the Lertcl Meeting in 1986. The results . Prof.of the polls were:
prasi;ent: Dr; ;.,,,,,,.
sart, Vrije Universiteit Brussel, Faculteit der Geneeskunde en Farmatie, Farmaceutisch Instituut, Laarbeeklaan 103, B-1090 Brussels, Belgium;
MICHELE FORINA
Professor Michele Forina is the outgoing President of the Chemometrics Society.
FORADVERTISINGINFORMATIONPLEASECONTACTOUR ADVERTISINGREPRESENTATIVES JAPAN
USA/CANADA
ESP- Tokyo Branch
Michael Baer 50 East 42nd NEW
YORK,
Street,
Suite
504
Mr H. Ogura 28-1 Yushtma.
NY 10017
3-chome,
Tel: t2121682-2200
TOKYO
Telex:
Tel: 103) 836 0810
226000
ur m.haer/synergtstic
Telex:
GREAT
Mr M. White 30-32
Son Ltd.
or MS A. Malcolm
Southampton
LONDON
WCZE
Street 7HR
Tel: 101) 240 2032 Telex:
299181
02657617
REST OF WORLD
BRITAIN
T.G. Scott 8
Bunkyo-Ku
113
adsale/g
Fax: 1011 379 7155
ELSEVIER SCIENCEPUBLISHERS MS W. van Cattenhurch p.0. Box 211 1000 AE AMSTERDAM The Netherlands Tel: 120) 5803.714/715/721 Telex:
18582 espa/nl
Fax: (20) 5803.769