‘Oncocheck’: an international external quality assessment scheme for immunoassays of tumor markers

Nucl. Med. Biol. Vol. 21, No. 3, pp. 4X3-493,

1994

Copyright 0 1994 Elsevier Science Ltd Printed in Great Britain. All rights reserved 0969-8051194 $6.00 + 0.00

Pcrgamon

‘Oncocheck’ : an international external quality assessment scheme for immunoassays of tumor markers

Richard COHEN*, Gian-Carlo Marie-Yolande RIGAULTm, * * -

ZUCCHELLF, Marc FRAYSSE*, Alessandro PILO*, Serge GRILLETM and Charles-Albert BKOLLON*

Service de Radiopharmacie et Radioanalyse, H6pital Neuro-Cardiologlque, France Institute of Clinical Physiology, CNR, Pisa, ltaly CIS bio International, Gif-sur-Yvette, France

Lyon,

ABSTRACT

Starting from November for immmunoassays France, Germany,

1990, an international

of tumor markers has been organized. Italy, Japan and Spain participate

features of the EQAS and data processing cycle allow evaluation immunoassays. can be classified variability

Presently,

in the scheme.

Scheme (EGAS)

238 laboratories

According

In this report the main

are outlined. Results collected

during the 1992-

to their analytical performances, the 6 tumor marker immunoassays

into several groups, the first including AFP and CA 15-3 for with

even

from

of the state of the art of AFP, CEA, CA 19-9, CA 15-3, CA 125 and PSA

and within-kit

as satisfactory

External Quality Assessment

agreement are good. For CEA assay, performance though

further

improvements

of between-lab

welcome. For the 3 other tumor markers, the higher total variability a better standardization (CA 19-9) or between-kit

by improvement

of either both within-kit

both total

can be considered

agreement

would

indicates an urgent need for and between-kit

agreements

agreement mainly (PSA, CA 125).

All correspondence should be addressed to: R. Cohen, Service de Radiopharmacie Radioanalyse, Hopital Neuro-Cardiologique, BP Lyon Montchat, 69394 Lyon Cedex 03, FRANCE. 4x3

be

et

484

RICHARD

Conm et al.

The tumor marker immunoassays have reached, in a few years, a large volume in routine laboratories. The need for an effective tool of testing quality in this field was obvious. For this reason an EQAS for immunoassays of tumor markers : the 'ONCOCHECK' program, has been set up since November 1990. It involves the collaboration of the "Service de Radiopharmacie et Radioanalyse", the Institute of Clinical Physiology from the Italian National Research Council (CNR) and CIS bio international. The "Service de Radiopharmacie et Radioanalyse" is in charge of the preparation of quality control samples, of receiving and processing data sent each month by the participating laboratories and of editing and mailing monthly and six-monthly reports. The Institute of Clinical Physiology coordinates this program for Italian participants. CIS bio international is mainly responsible for the diffusion of the program and the management of registrations. OUTLINE OF THE EQAS 'ONCOCHECK' is an EQA program based on the use of 24 samples every year ; each month the participating laboratories assay two sera containing different concentration of the tumor markers. The tumor markers included in the scheme are : AFP, CEA, CA 19-9, CA 15-3, CA 125 and PSA. The main feature is the use of real sera containing native antigens for the preparation of QC-samples. Another feature is that reports are edited on an anonymous basis, laboratory codes being known only by the "Service de Radiopharmacie et Radioanalyse" and the Institute of Clinical Physiology. The organization timing flowchart for a month is shown in Fig. 1. The testing period is between the fifth and the fifteenth day of the month. Then, results should be sent quickly so that they reach the "Service de Radiopharmacie et Radioanalyse" by the twentieth of the month. Results from Italian laboratories reach us through the Institute of Clinical Physiology. The complete monthly report is, usually, sent back to the participants at the beginning of each month.

i

&

4

I

I

1

5

15

20

_

,

!

30

5

Day OF

themonth

Fig. 1. Organisation timing flowchart of the 'ONCOCHECK' program

This EQAS has an international dimension. The number of participants has increased rapidly since the beginning of the scheme and is indicated in Table I for the 1992 cycle. The majority of contributions come from Italy, France and Spain. It can be expected that some changes will occur in the statistical sample of laboratories since almost fourty more labs actively participated in the EQAS during the second six-month period.

Euro-Immunoanalyse

485

‘93

Table 1. Number of 'ONCOCHECK' participants during the 1992 cycle for the first and second semesters. The two figures indicate actively participating or registered laboratories.

France Germany

1992/l

199212

40148

59161

515

Italy

616

751113

Japan

719

Spain

19129

Miscellaneous

851128 618 24129

l/4

All

516

1471208

1851238

DATA PROCESSING The monthly report is a personalized report on which the laboratory code number, the kit used and the results for QC-samples A and B are reminded for each analyte. It comprises several plots on one side of the sheet. Firstly, a histogram for each QC-sample allows the laboratory to know its position among the distribution of all data. Figure 2 shows an example for PSA of such a histogram with a bimodal distribution.

Laboratoiresexclus dugmphique:

x10

PSA

1

Program

1203AIB

3

DATE 03/04/1992 KIT

Your Lab isF409

:CIS bio int ELSA

Votre Rhultat A : 8.7 ng/mI Your Result B

Fig.2.

: 2.2 ng/mI

Histogram on the 'ONCOCHECK' monthly report. The result from the concerned laboratory is indicated by the darker bar

RHXARD Conm

486

et al.

Secondly, a Youden-plot (Fig. 3) drawn with data from QC-samples A and B is included in the monthly report in order to give rough estimates of random and systematic components of the variability. If only a bias is present in results from a laboratory then the point representative of these data will be located on the diagonal line which joins the origin to the point of which coordinates are the means for QC-samples A and B. It is necessary to point out that this is true only under the assumption that biases are proportional to the concentration when two different level QC-samples are used. Moreover, if random errors are present the point will move away from the diagonal line. Now, according to the quadratic addition of the two components of error and to the Pythagoreas theorem it is possible, from any point representative of laboratory data, to draw a right-angled triangle of which the hypotenuse, the side perpendicular to the diagonal line and the side lying on it are total error, random and bias components respectively. Ul/ml xlOE2

A

3.34 --

7

Youden /

B

1.18 2.19

Fig. 3.

4.k2

6.45

UI/ml 10El

x

Youden-plot on the 'ONCOCHECK' monthly report. The point of which coordinates are QC-samples A and B trimmed means is indicated by a cross. The point representative of results from the concerned laboratory lies at the intersection of the dotted lines. The two vertical and horizontal lines inside the plot are drawn two standard deviations away from the means of QC-samples A and B respectively

The third kind of plot on the monthly report is a control chart for the follow-up of Z-score over a twelve-month period (Fig. 4). Z-score is computed, for each QC-sample, by subtracting the overall trimmed mean from the lab result and by dividing this difference by the overall trimmed standard deviation. In this way, Z-score is an expression in standard deviation units of the lab bias in respect to the overall mean. This plot is very useful for within-laboratory quality control since it can indicate changes in bias as when a laboratory changes one kit for another.

481

Euro-Immunoanalyse'93 ET Tr

Z SCORE

Echantillons I Samples A=

l

, B= o

>3

lab. result - overall trimmed mean overall trimmed standard deviation

Fig. 4

Control chart for Z-score on the ETTr : trimmed standard deviation

'ONCOCHECK' monthly

report.

Lastly, on the back of the sheet are printed parametric and non-parametric statistics for all data and for data subdivided according to the kit : numbers of labs, means and standard deviations before and after rejection of 2 standard deviation outliers, ranges and 2.5, 16, 50, 64, 97.5 percentiles which are plotted on a diagram (Fig. 5).

Fig. 5

Parametric and non-parametric statistics on the 'ONCOCHECK' monthly report. MoyTr, ETTr, CVTr : trimmed mean, standard deviation and coefficient of variation. For the right-hand side diagram, only the median (50 percentile) is indicated if kit users are less than 12.

RICHARD COHEN et al.

488

In addition, all data accumulated during six months are used to prepare a cumulative report containing for each laboratory, a recapitulative board and estimates of bias and precision. Bias estimate is obtained from Z-scores and precision estimate from the data reported by the lab for hidden replicate QC-samples. Laboratories are ranked according to their performance.

RESULTS AND DISCUSSION Before discussing results obtained from the 'ONCOCHECK' program 1992 cycle, it is necessary to outline some characteristics of the statistical sample of participating laboratories, namely the number of different kits and the kind of tracer used.

Table 2.

Kits used by the 'ONCOCHECK' participants for CEA and CA 15-3 assays. Number of kits (n>, number of kits including a radioactive label (IRMA kits), percentage of labs using IRMA kits (% IRMA) and maximum percentage of users of the same kit (% Max) are indicated for each country and for all the participating labs.

n

CEA

IRMA kits

CA 15-3

CEA

CA 15-3

France

9

3

4

1

Germany

4

2

2

1

1

2

1

2

%IRMA

CEA

% Max

CA 15-3

CEA

CA 15-3

78

93

45

93

60

80

24

65

Italy Japan

2 All

16

5

8

3

Table 2 shows two opposite cases which exist for the six tumor marker assays considered. The first applies to AFP, CEA and PSA to a lesser extent and is characterized by a great (more than 10) number of available kits. Consequently the maximum percentage of users of the same kit is relatively small ; therefore, the consensus mean does not predominantly reflect the analytical performances of one kit only. On the other hand, for CA 15-3, CA 125 and CA 19-9 assays the number of various kits is between 5 and 10 and the maximum percentage of users of the same kit could reach 65 % ; in all cases this percentage is greater for French labs than for others. In addition, half the kits are immunoradiometric assays used by more than 50 % labs.

489

Euro-Immunoanalyse '93

In order to assess the state of the art of tumor marker immunoassays, precision estimates were obtained from data reported by the labs assaying hidden replicate QC-samples. Coefficients of variation (CV) were computed from results of samples prepared from the same pool and assayed on two or more occasions. During the 'ONCOCHECK' 1992-cycle several pools at different concentration levels were distributed. Data shown were taken from a pool with a concentration in the range between normal and pathological values and assayed three times.

Table 3.

Variability of tumor marker assays estimated from one hidden replicate (n = 3) pool. N, number of labs, r, consensus mean

N

z

CVWL (%I

CVSL (%I

CVT (%I

CVTR (%I

AFP (IlJ/ml)

154

12

20

15

25

20

CEA (ng/ml)

163

5

15

23

27

25

CA 19-9 (U/ml> 163

35

CA 15-3 (U/ml) 161

25

16

12

20

16

CA 125 (U/ml>

160

30

15

26

30

27

PSA (ng/ml)

155

6

19

28

34

31

In Table 3 are shown for the six tumor markers the total variability (CVT> and its within-lab (CVwI) and between-lab (CVRI) components obtained by one-way analysis of variance (Pilo et al., 1990). The greater contribution to the latter component are kit discrepancies. Another estimate of the total variability is given by the trimmed coefficient of variation (CVTR) computed and printed on the monthly report. Slight differences in the two estimates of total variability (CVT and CVTR) are explained by the rejection of outliers. From these data, it can be said that for AFP and CA 15-3 both between-lab agreement and within-lab precision are relatively satisfactory in respect to the variability of the other four tumor markers. These figures should be interpreted, bearing in mind that the number of available kits for CA 15-3 is very smaller in respect to those for AFP assay. For PSA, CA 125 and CEA it is the between-kit component of variance which needs improvements. The worst variability is observed for CA 19-9 due to both larger systematic between-kit differences and to poor within-lab precision. Now, regarding the means obtained for the QC-samples distributed during the 1992 cycle, with the different kits and expressed as a ratio according to the overall mean (Fig. 6) we can conclude that the traditional IRMA techniques produce, with some exceptions, results similar to each other whatever the QC-sample, while determinations produced by non-isotopic methods are more different from each other and from IRMAs, the discrepancies depending on the QC-sample used. For example, it is noteworthy that kits IRMA1 and NIAl on the one hand and IRMA2 and NIA2 on the other hand are produced by the same manufacturer and that results reported

490

RlCHARD

et al.

COHEN

5

4

----_-_-_--_--_-_-_____------__-----_-__--*-_-__ l l

3

--------_____-_-_______-_______r____,_____~___ 8 : :

2

:

.

:

m _

----_--_-______________________o_____~___ -c__-_! l

0

L

0

8

; 1

----

i -_--

----

+

t t ----

i -_c_--a---t ----*-_-______ l

0 I I

IRMA1

Fig. 6

I 1

IRMA2

I I

IRMA3

I I

IRMA4

I 1

IRMA5

I I

NlAl

I I

NIA2

I I

NIA3

a ;

3

5 I NIA4

Means obtained with five IRMA and four non-radioactive (NIA) kits during the 'ONCOCHECK' 1992-cycle. Each mean is expressed as a ratio (R) according to the overall trimmed mean.

by the users of non-isotopic kits are generally two to three times higher compared to IRMAs. The increasing use of these new immunoassay techniques represents the major cause of the sudden worsening of the between-lab agreement of CA 19-9 determinations (Zucchelli et al., 1993). The reasons for these discrepancies are unclear at the present time ; it is, however, conceivable that different kits for CA 19-9, even based on the same monoclonal antibody and the same antigen for standard preparation, show a different degree of specificity against the 19-9 determinant due to differences in the tracer, in the solid-phase and/or in the experimental assay conditions (pH, time, temperature of the antigen/antibody reaction). In addition, the presence in serum of different molecular forms of CA 19-9 which may be differently recognized by the kits can contribute to the discrepancies observed. Now,

the picture for CA 125 is somewhat similar. The total variability observed in the QC-samples distributed in 1992 is depicted as a function of concentration in Fig. 7 ; the CVs found in the first semester (closed circles) are lower than those observed in the second semester (open squares). Excluding QC-samples at low concentration, the between-lab agreement of about lo-15 % increased to about 20-25 % in the second part of 1992. The reasons for this decrease of agreement remain unclear at present.

The third tumor marker immunoassay which needs improvements in standardization is PSA assay. During the 1992 cycle, the total variability ranged from 30 % to 40 % over the entire concentration range explored (l-60 ng/ml). This large variability is mainly attributable to systematic between-kit differences. To better analyze them, Table 4 reports the mean values (expressed as a ratio according to the overall mean> obtained by the users of the most popular methods for one 'ONCOCHECK' pool and for a QC-sample with a similar concentration, distributed in the course

491

Euro-Immunoanalyse '93 70

-

60

~-

50

--

8 . F e

40 .

.-

.

: 2

30 0

0

0

20

--

10

--

0

T-

m-i--

0

I 40

20

60

60

---m-----i

100

120

140

160

160

Moy.Tr (U/ml)

Fig. 7

CA 125 between-lab agreement during the first semester (closed circles) and the second semester (open squares) of the 'ONCOCHECK' 1992-cycle.

of a mandatory French EQAS (Cohen et al., 19911, which encompass a great number (N = 2692) of labs. The discrepancies are largely accounted for by the use of local kit standards calibrated versus two different reference preparations. Therefore, it is highly advisable that an expert panel decide on the choice of an international standard and that kit producers refer the kit calibrators to it or at least indicate the conversion factors against it. Table 4.

Means for the most used PSA kits, expressed as a ratio (R) according to the overall mean and obtained during 'ONCOCHECK' program (QC-sample 1211 B) or during a mandatory French EQAS (QC-sample LNS G015). 'ONCOCHECK' 1211 B N

Abbott IMX

R

LNS GO15 N

R

15

0.64

811

0.81

5

1.61

409

1.15

CIS bio int ELSA

45

1.31

39

1.16

Pharmacia DELFIA

4

0.80

32

0.68

Stratus System

3

1.14

340

1.61

Tosoh AIA 600-1200

4

0.67

139

0.66

CIS bio int EIA

PSA

156

Z = 6.15 ng/ml CVTR = 33 %

2692

t = 6.17 ng/ml CVTR = 37 %

492

RICHARD COHEN etal.

The last point to address concerns the origin of antigens used to prepare QC-samples. Commercial sera are generally spiked with extractive tumor markers. As already mentioned, 'ONCOCHECK' pools are prepared with native antigens from pooled sera of patients containing high concentrations. Figure 8 shows the correlation between the results of two IRMA kits for CEA assay according to the origin of QC-samples. It can be seen that the correlation is very good (r = 0.98) when QC-samples from one commercial source and from 'ONCOCHECK' program are used while it drastically worsens (r = 0.83) when sera from another commercial source are included. Therefore, it was decided to prepare 'ONCOCHECK' pools from native antigens to avoid heterogeneity of results due to the source of commercial sera chosen.In any case when the analyte may be present -in the specimens of patients- in different molecular forms, the use of control material from various sources can affect the results of the EQAS.

0 0

0

10

20 IRMA1

Fig.8.

30

40

50

nglml

Correlation between the results of two CEA IRMA kits obtained with two different commercial sources (open and closed circles) and from 'ONCOCHECK' (closed squares) QC-samples.

In conclusion, the state of the art of immunoassays of tumor markers can be assessed from the results of the 'ONCOCHECK' 1992-cycle. The performance achieved for AFP, CA 15-3 and CEA is satisfactory and not different from that observed in the EQAS of proteic hormones. As for the results reported for CA 19-9, CA 125 and PSA, a higher between-lab variability due to larger systematic between-kit differences and to worse precision of the methods is observed. In particular for CA 19-9 the new non-isotopic methods produce higher values than those found by traditional IRMAS ; for PSA assay the use of different local kit standards is responsible for the large variability observed.

Euro-Immunoanalyse '93 REFERENCES PILO, A., G.C. ZUCCHELLI, S. MASINI, G.C. TORRE and A.M. BALLESTA (1990). Progress report on external quality assessment program for immunoassays of tumor markers. J. Nucl. Med. Allied Sci., -34 (suppl n"3), 75-82. COHEN, R. and Ch. A. BIZOLLON Immunoassay external (1991). quality assessment schemes in France. Ann. 1st. Super. Sanita, 27, 3, 503-510. ZUCCHELLI, G.C., A. PILO, M.R. CHIESA, R. COHEN Ch. A. BIZOLLON and (1993). The growing use of non-isotopic techniques for CA 19-9 immunoassay increases the between-laboratory variability. Clin. Chem.,in press.

493