- Email: [email protected]
Early conception factor lateral flow assays for pregnancy in the mare
Available online at www.sciencedirect.com
Theriogenology 71 (2009) 877–883 www.theriojournal.com
Early conception factor lateral flow assays for pregnancy in the mare E. Marino a,*, W.R. Threlfall b,1, R.A. Schwarze b,1 a
Department of Veterinary Clinical Sciences, College of Veterinary Medicine, The Ohio State University, 30425 McPeck Road, Richwood, OH 43344, United States b Department of Veterinary Clinical Sciences, College of Veterinary Medicine, The Ohio State University, 601 Vernon L. Tharp Street, Columbus, OH 43210, United States Received 3 August 2007; received in revised form 11 March 2008; accepted 3 June 2008
Abstract The ECFTM lateral flow assay test is marketed to detect non-pregnancy in mares. The objectives of the present study were to determine the accuracy of the ECF test, the accuracy of the electronic reader accompanying the ECF test, and agreement between two human readers and the electronic reader. Serum samples were collected from anestrus, cycling but not inseminated, and inseminated mares, and were evaluated with the ECFTM test (EDP Biotech Company, Knoxville, TN, USA) at The Ohio State University and at the EDP Biotech Laboratory. Specificity ranged from 0.07 to 0.16, the negative predictive value ranged from 0.15 to 0.33, and accuracy ranged from 0.43 to 0.52. The electronic reader did not add improve the accuracy or predictive values of the test. Based on the electronic reader, 80.0% of the serum samples collected from the anestrus mares were false positives; Readers 1 and 2 had 60.0 and 33.3% false positives, respectively. For samples collected during the estrous cycle, 83.9% were false positives by the electronic reader, whereas Readers 1 and 2 had 43.7 and 26.4% false positives. We concluded that, regardless of whether the test strips were evaluated by a human or electronic reader, this assay was not accurate for determination of the non-pregnant mare. Published by Elsevier Inc. Keywords: Pregnancy diagnosis; Early conception factor; Mare; Equine
1. Introduction A transrectal examination (manual palpation or ultrasonography) is the most reliable method of early pregnancy diagnosis in the mare. Unfortunately, these methods do not allow reliable detection of the nonpregnant mare much earlier than the onset of spontaneous luteolysis in these mares (approximately
* Corresponding author. Tel.: +1 614 307 9880. E-mail addresses: [email protected] (E. Marino), [email protected] (W.R. Threlfall), [email protected] (R.A. Schwarze). 1 Tel.: +1 614 292 6661; fax: +1 614 292 0895. 0093-691X/$ – see front matter. Published by Elsevier Inc. doi:10.1016/j.theriogenology.2008.06.003
13 or 14 d after ovulation). However, if a non-pregnant mare could be accurately identified 6 d after ovulation, she could be immediately given prostaglandin (PGF2a) to initiate luteolysis [1], thereby shortening interestrus intervals and increasing opportunities for breeding [1]. The discovery of early pregnancy factor (EPF) and the rosette inhibition test enabled detection of pregnancy soon after breeding [2]. However, this method was time consuming, expensive, and tedious [2], and could not be readily performed under field conditions [2]. More recently, the early conception factor (ECFTM) test, originally released in 1998 by Concepto Diagnostics, Knoxville, TN, USA, was marketed as an immunoassay capable of diagnosing the non-pregnant cow within 12–48 h after ovulation
878
E. Marino et al. / Theriogenology 71 (2009) 877–883
[3]. Although this test was reported as inaccurate [3–6], the company released a similar test in 2003 for horses. The manufacturer claimed that the test could detect the ECF glycoprotein in bred mares between 3 and 30 d after ovulation. According to the manufacturer, this is a lateral flow assay that uses monoclonal and polyclonal antibodies with a colloidal gold indicator. There is apparently only a single report regarding the specificity and sensitivity of this test; in that report, it was suggested that the test was inaccurate [5]. The present study examined the improved equine ECF test (marketed in 2006) to determine if changes made to the previous marketed test (2004) improved the specificity, accuracy, and negative predictive value, and to determine if the added electronic reader (2006) improved the results of the test. The electronic reader values were also compared to two human ‘‘readers’’. The objectives of this project were to determine the accuracy of the ECF test, the accuracy of the electronic reader accompanying the ECF test, and agreement between two human readers and the electronic reader. 2. Materials and methods 2.1. Horses This study was performed during the 2007 breeding season (February 1 to May 31), using 61 Standardbred mares housed at a farm in central Ohio. Data regarding age, parity, previous breeding history, and lactation were not specifically recorded.
2.2.2. Experiment 2 Fifteen mares that had ovulated and had at least one apparently normal estrous cycle (with continued cyclicity) were used. For two consecutive weeks, transrectal examinations (to monitor ovarian activity) were done as described in Experiment 1 and blood collection was done twice weekly (Monday and Friday). None of the mares were bred until after all blood samples had been collected. 2.2.3. Experiment 3 This experiment used 61 cycling mares, including the 26 previously used in Experiments 1 and 2. All mares in estrus with an ovarian follicle 35 mm in diameter were artificially inseminated with fresh extended semen. Mares were bred every 48 h until ovulation occurred; most were bred at least twice and some three times. Ovulation (Day 0) was confirmed by transrectal palpation or ultrasonography after insemination. Blood samples were collected on Day 6 (range, Days 4–7), Day 10 (range, Days 10–12), Day 14 (range, Days 13–15), Day 18 (range, Days 17–19), and Day 35. Day 6 was chosen as the critical time to give PGF2a to induce premature luteolysis and estrus in non-pregnant mares [7]. Pregnancy diagnosis was conducted with transrectal ultrasonography on Day 18; mares that were pregnant on Day 18 were re-confirmed as pregnant on Day 35. The Day-35 blood samples were collected only on mares deemed pregnant on the basis of a transrectal ultrasonographic examination on Day 18. 2.3. Blood samples
2.2. Experimental groups There were three experiments that varied by reproductive status. In these three experiments, mares were anestrus, cycling but not inseminated, and cycling and inseminated. For these experiments, the objectives were to determine if anestrus or cycling but non-bred mares would yield false positive results, and to determine the reliability of the test to detect pregnant mares.
Samples were collected (jugular venipuncture) with a 20-ga, 38 mm vacutainer needle into a 10-mL red top vacutainer tube containing no anticoagulant (Becton Dickinson, Franklin Lakes, NJ, USA). The tubes were kept cool from collection to centrifugation (refrigeration of the blood samples did not affect test results [5]) and were centrifuged (2800 g for 10 min) within 12 h after collection. 2.4. Test procedure
2.2.1. Experiment 1 Eleven anestrus mares were used. Anestrus was defined as no follicles >20 mm in diameter and no luteal tissue was detected in the ovaries by transrectal palpation and ultrasonography (Pie Medical Scanner 480, 5 MHz, linear-array transducer; Pie Medical Imaging, Netherlands) conducted every Monday, Wednesday, and Friday for three consecutive weeks. Blood samples were collected every Monday.
Testing was done immediately after centrifugation. Serum and test kits were allowed to reach room temperature before the tests were performed. The ECF lateral flow assay test was performed in accordance with the manufacturer’s instructions. Lateral flow assays are designed to rapidly detect a particular molecule; typically the molecule of interest is isolated based on binding to a specific antibody, and this
E. Marino et al. / Theriogenology 71 (2009) 877–883
binding is detected with markers that can be visualized. The test strips were comprised of nitrocellulose and overlayed with an absorption pad to carry the sample over the testing reagents. One drop of mare serum was added to a well, where it immediately contacted a reagent pad. The reagent pad contained colloidal gold-labeled monoclonal antibody for ECF. Following addition of the sample, two drops of buffer were added to help carry the sample and the monoclonal antibody across the length and width of the absorption pad. Two lines of reagents (approximately 1 mm wide) were set up (by the manufacturer) in the nitrocellulose. These lines were oriented perpendicular to the flow of the sample and carrying buffer. The control line was the farthest from the sample well. It contained an antibody, generated against the monoclonal antibody for ECF. If this line was visible following the test, it confirmed that the colloidal goldlabeled monoclonal antibody in the reagent pad was carried throughout the entire absorption pad, and thus, beyond the test line. Binding of the antibody for ECF, which was colloidal gold-labeled, to the antibody in the control line isolated and concentrated the colloidal gold particles so that they were visible along the control line.
879
Binding of these two antibodies did not depend on the presence of ECF in the sample. The test line, or indicator line, likely contained a polyclonal antibody generated against ECF. This line was positioned on the nitrocellulose strip between the sample well and the control line. It was oriented parallel to the control line and perpendicular to the flow of the sample. If the ECF glycoprotein was present in the blood serum, the ECF molecules would have bound with the gold labeled monoclonal antibodies in the reagent pad. As the sample and reagent mixture passed over the test line, the ECF molecules bound to the polyclonal antibodies in the test strip. Since the ECF was also bound to colloidal goldlabeled monoclonal antibody, this concentrated the colloidal gold particles so that they were visible along the test line. Binding along the test line was dependent on the presence of ECF in the sample, and should have not occurred in the absence of ECF. 2.5. Test interpretation The ‘C’ region was the control line and the ‘T’ region was the positive reading (Fig. 1). Therefore, if two lines
Fig. 1. Interpretation of the ECFTM test strips. One line appearing in the control region (‘‘C’’) indicated the mare was non-pregnant, whereas one line appearing in the control region and one line appearing in the test region (‘‘T’’) indicated the mare was pregnant. Two readers interpreted test strip number 65 as non-pregnant and pregnant, respectively. Any line in the test region, no matter how faint it was, was considered a positive test reading. Test strip 63 was determined pregnant by all readers and test strip 8 was determined non-pregnant by the two readers and negative (non-pregnant) by the electronic reader.
880
E. Marino et al. / Theriogenology 71 (2009) 877–883
appeared, one in the ‘C’ region and one in the ‘T’ region, the conclusion was that the mare was pregnant. However, if only one line in the ‘C’ region appeared, the conclusion was that the mare was non-pregnant. The manufacturer also provided an electronic reader (Fig. 2) that was new to the assay procedure and being tested for the first time in this study. Therefore, each ECF test was read three times by the electronic reader (it was turned off and restarted between successive readings). The reader did not consistently give the same reading on all three tests; when there was a discrepancy, the reading recorded was based on the results of the two tests that were
in agreement. For example, if the electronic reader recorded pregnant, pregnant, and non-pregnant, the final result was recorded as pregnant. Two humans (Reader 1 and Reader 2) also read the tests; their assessments were done independent of each other. Serum was also sent to the manufacturer for interpretation. The samples were stored at 4 8C between serum removal and shipping. The time elapsed between centrifugation and shipment was 2– 7 d. The serum was chilled and held at 4 8C until shipment. During shipment, the samples were maintained chilled by placing two frozen shipping cold packs with the serum and sending the package by overnight express. On the day that the samples arrived at the manufacturer, they were tested (consistently by the same operator). 2.6. Statistical analysis Each experiment was analyzed separately. The accuracy of the ECF test was evaluated by a two-bytwo table (Chi square) [8]. Specificity, accuracy, and negative predictive value (NPV) were calculated for the electronic reader and for the two human readers (Table 1). Specificity, or the true negative rate, is the proportion of non-pregnant mares diagnosed by transrectal ultrasonographic examination with a negative result (nonpregnant) at Day 18. Using the two-by-two table, there was a square for true positives, a square for true negatives, a square for false positives, and a square for false negatives. From these values, the proportions were calculated. The two-by-two table was used to calculate the specificity, the NPV and the kappa statistic [8]. The NPV was the probability that a negative ECF test result was from a non-pregnant mare. The kappa statistic represented the proportion of potential agreement, beyond chance, that was achieved. The kappa statistic usually ranges from 0.0 to 1.0, with 0.0 being no agreement (beyond chance) and 1.0 one being perfect agreement. A kappa of 1.0 is uncommon, whereas kappa of 0.6–0.8 is considered to be substantial agreement. Very rarely is a kappa statistic below zero; it means that the agreement is weaker than expected by chance. Kappa was calculated using the K-test of FREQ procedure of SAS (Microsoft Windows Release 9, 2005; SAS Institute, Cary, NC, USA). 3. Results 3.1. ECF test performance
Fig. 2. Displays of the electronic reader screen displaying a positive and a negative reading.
Based on the electronic reader, 80.0% of the serum samples collected from the anestrus mares were false positives. Furthermore, Readers 1 and 2 had 60.0 and
E. Marino et al. / Theriogenology 71 (2009) 877–883
881
Table 1 Performance of the ECFTM test, conducted at OSU and the manufacturer, on blood samples collected at various times (ovulation = Day 0) in inseminated mares. Reader 1
Reader 2
Day 6 Sensitivity Specificity Positive predictive value Negative predictive value Accuracy Kappa
0.62 0.48 0.62 0.52 0.57 0.07
0.18 0.79 0.5 0.45 0.46 0.016
0.79 0.07 0.5 0.22 0.46 0.065
0.65 0.55 0.64 0.55 0.61 0.1
Day 10 Sensitivity Specificity Positive predictive value Negative predictive value Accuracy Kappa
0.6 0.58 0.66 0.52 0.59 0.099
0.17 0.84 0.6 0.41 0.44 0.0032
0.77 0.077 0.53 0.2 0.48 0.069
0.69 0.58 0.69 0.58 0.64 0.15
Day 14 Sensitivity Specificity Positive predictive value Negative predictive value Accuracy Kappa
0.6 0.62 0.68 0.53 0.12 0.61
0.23 0.88 0.73 0.46 0.51 0.059
0.69 0.077 0.5 0.15 0.43 0.1
0.63 0.61 0.69 0.55 0.62 0.14
Day 18 Sensitivity Specificity Positive predictive value Negative predictive value Accuracy Kappa
0.64 0.64 0.72 0.55 0.64 0.16
0.22 0.84 0.67 0.43 0.44 0.0027
0.78 0.16 0.57 0.33 0.52 0.022
0.66 0.58 0.68 0.55 0.62 0.13
Day 35 Sensitivity Specificity Positive predictive value Negative predictive value Accuracy Kappa
0.52 0.4 0.83 0.13 0.5 0.02
0.17 0.6 0.71 0.11 0.24 0.054
0.86 0.4 0.89 0.33 0.79 0.071
N/Aa N/Aa N/Aa N/Aa N/Aa N/Aa
a
Electronic reader
Manufacturer
N/A values were not available.
33.3% false positives, respectively. For samples collected during the estrous cycle, 83.9% were false positives by the electronic reader, whereas Readers 1 and 2 had 43.7 and 26.4% false positives. The specificity of the electronic reader ranged from 0.07 to 0.4, whereas for Readers 1 and 2, it ranged from 0.40 to 0.64, and from 0.60 to 0.88. Negative predictive values for the electronic reader, Reader 1, and Reader 2 ranged from 0.15 to 0.33, 0.13 to 0.55, and 0.11 to 0.46. 3.2. Manufacturer results Results received from the manufacturer were similar between days. Specificity ranged from 0.55
to 0.62. Negative predictive values ranged from 0.55 to 0.58. The kappa statistic ranged from 0.10 to 0.15. The manufacturer results were similar to OSU results. Day 35 results were not available from the manufacturer. 3.3. Agreement between readers Each test was read by an electronic reader provided by the manufacturer of the test and two human readers. Between the two human readers, the kappa coefficient was 0.049. Between the electronic reader and Reader 1, the kappa coefficient was 0.013, and between the electronic reader and Reader 2, the kappa coefficient
882
E. Marino et al. / Theriogenology 71 (2009) 877–883
was 0.029. For all three readers, the kappa coefficient was 0.011. The electronic reader was used to read each test three times. If the readings did not agree, the two readings that were the same were taken to be the electronic reader’s reading. Combined for all three experiments, the electronic reader did not give the same result for all three readings of a single sample on 63 of 396 tests (16% of the time). The kappa statistics for the electronic reader ranged from 0.10 to 0.071. For Readers 1 and 2, the kappa statistic ranged from 0.02 to 0.16, and from 0.016 to 0.059, respectively. The kappa statistic suggested no agreement between Readers 1 and 2, nor between the electronic reader and the human readers. 3.4. Agreement between Reader 1 and the manufacturer Combined for all samples, the kappa coefficient between Reader 1 and the manufacturer was 0.55. 4. Discussion In the current experiment, the accuracy of the ECF test results was compared to the actual pregnancy status (transrectal palpation and ultrasonography on Day 18), with a follow-up examination on Day 35. The electronic reader had 80% and 83.3% false positive for anestrus mares and cycling mares that were not inseminated, respectively. Based on the assumption that ECF is only present upon fertilization, the positive test results before breeding and in anestrus mares were unexpected. These false positive results during estrus were attributed to cross-reactivity with other proteins present in the serum [5]. However, this does not explain false positives in anestrus mares that were not cycling nor having follicular activities. Detection of ECF in anestrus mares may indicate that the test detects ECF produced by nonembryonic tissue. During the first 18 d after breeding, there were still approximately 40% false positives. According to the electronic reader, the rate of false positives decreased after breeding. The performance of the ECF test was similar among Days 6, 10, 14, and 18. The main objective of this study was to determine the pregnancy status of mares at Day 6 after ovulation. Pregnancy was not determined until Day 18, therefore the main statistical error for Day 6 is whether the mare was truly pregnant or non-pregnant. Due to embryonic loss, a mare that was pregnant on Day 6 may not have been pregnant on Day 18. However, comparing the statistics for Days 6 and 18, the values
were relatively similar for Reader 1. We inferred that even though the true pregnancy status on Day 6 was not known, based on the statistical information provided, it would be similar to the results reported. For the test to be beneficial in the equine breeding industry, these values need to be close to the values for transrectal ultrasonographic examination on Day 15 (0.98 and 0.90) [5]. If these values are reached, then only a 2% risk of iatrogenic early embryonic loss due to treatment with PGF2a to induce premature luteolysis and estrus. For most breeders, a 2% risk would be acceptable. Results for Day 35 varied from the other days, with more accurate results from the electronic reader. The 0.79 accuracy of the electronic reader on Day 35 suggests that it does not matter if the mare is pregnant or non-pregnant, it will always read close to 80% pregnant. The overall results were not similar among readers. Furthermore, the present results were not similar to those reported in other studies [3,5]. The specificity in Horteloup et al. [5] ranged from 0.14 to 0.33. In Cordoba et al. [3], this value was 0.04 for specificity for the cow. The accuracy in Cordoba et al. [3] was not higher than 0.67 and in Horteloup et al. [5], the accuracy was not higher than 0.60. Cordoba et al. [3] concluded that induced embryonic loss due to PGF2a administration would occur frequently if this test was used as a means of pregnancy detection in the cow. The observed specificity for the electronic reader ranged from 0.07 to 0.40. Therefore, the electronic reader was not capable of accurately detecting nonpregnant mares. The electronic reader was most accurate with a specificity of 0.40 for Day 35. The range of Reader 1’s specificity, 0.40–0.64 seemed better than the results in Cordoba et al. [3] and in Horteloup et al. [5]. However, Reader 1’s values were still well below the target value of 0.98. Based on all of these data, we concluded that neither the test nor the electronic reader were accurate. The negative predictive values for the electronic reader ranged from 0.15 to 0.33. This suggests that only 15–33 out of 100 mares diagnosed non-pregnant were truly non-pregnant. If mares diagnosed as non-pregnant using the ECF test are given PGF2a to hasten the onset of estrus, 67–85% of the mares would suffer from early embryonic loss due to administration of PGF2a. The negative predictive value range for Reader 1 was 0.13– 0.55, meaning that 45–87% of the mares in this study could have iatrogenic early embryonic losses if PGF2a was administered. A kappa statistic was calculated to determine overall agreement of the electronic reader versus the observed results (pregnant or non-pregnant). All of the kappa
E. Marino et al. / Theriogenology 71 (2009) 877–883
coefficients were below zero, except for Day 35, indicating that agreement of actual results was weaker than expected by chance. For Day 35, the kappa coefficient was 0.071, meaning there was no agreement. Kappa coefficients below zero are rare. Not only was there no agreement between the reader and the actual results, but also the agreement was weaker than expected by chance. Comparing the kappa statistic of Reader 1, the range was 0.02 to 0.16. These numbers were still very small, indicating almost no agreement between test results and actual pregnancy status. Results from the manufacturer were very similar to Reader 1, suggesting that both interpreted the tests similarly. The kappa statistic between Reader 1 and the manufacturer also suggested that interpretation of the tests was similar. This was the highest kappa statistic reported in this study. Following the guidelines for kappa statistics, a 0.55-kappa falls in the middle range of good agreement [8]. Although this agreement was not excellent, it was the best agreement between any of the other readers. There are many possible reasons the kappa statistics are so small. Perhaps the antibody used in the test was non-specific, or at least it was not specific against the antigen ECF. The test strips may not have been working properly, too much or too little gold was attached to the antibody, or the amount of antibody in the test strip was inappropriate. Perhaps there was a pregnancy variant that was detectable by Day 6; however, ECF may not be that variant. Reasons for the electronic reader’s poor results are not addressable here, since this product is sold to make the reading of the test more accurate. The electronic reader may have read any discoloration in the test strip; in that regard, it may have read the control line instead of the indicator line. In conclusion, although this test would be of great value to equine reproductive management if it worked properly, it is clear that there would be considerable loss of pregnancies if the ECF test was used to detect non-
883
pregnant mares at Day 6 (or later), and based on that information, the mares were treated to induce luteolysis and estrus. Furthermore, the electronic reader did not add value to the test; neither a human-evaluated ECF test nor the electronic reader were effective at diagnosing non-pregnant mares. Therefore, transrectal palpation and ultrasonography examination remain the preferred methods of pregnancy diagnosis by Day 15 in the mare. Acknowledgements Appreciation is extended to Drs. Mossbarger and Wittum for their help in this study. References [1] Allen WR. Use of prostaglandin for synchronization of oestrus and treatment of prolonged dioestrus in mares. Acta Vet Scand Suppl 1981;77:227–39. [2] Clarke FM, Wilson S, McCarthy R, Perkins T, Orozco C. Early pregnancy factor: large scale isolation of rosette inhibition testactive polypeptides from ovine placental extracts. J Reprod Immunol 1987;2:133–56. [3] Cordoba MC, Sartori R, Fricke PM. Assessment of a commercially available early conception factor (ECF) test for determining pregnancy status of dairy cattle. J Dairy Sci 2001;84:1884–9. [4] Threlfall WR. Immunosuppressive early pregnancy factor (ISEPF) determination for pregnancy diagnosis in dairy cows. Theriogenology 1994;41:317. Abstract. [5] Horteloup MP, Threlfall WR, Funk JA. The early conception factor (ECF) lateral flow assay for non-pregnancy determination in the mare. Theriogenology 2005;5:1061–71. [6] Gandy B, Tucker W, Ryan P, Williams A, Tucker A, Moore A, et al. Evaluation of the early conception factor (ECF) test for the detection of nonpregnancy in dairy cattle. Theriogenology 2001;4:637–47. [7] Beg MA, Gastal EL, Gastal MO, Ji S, Wiltbank MC, Ginther OJ. Changes in steady-state concentrations of messenger ribonucleic acids in luteal tissue during prostaglandin F2alpha induced luteolysis in mares. Anim Reprod Sci 2005;3/4:273–85. [8] Sackett D, Haynes R, Tugwell P. Clinical epidemiology: a basic science for clinical medicine. Little Brown; 1991.
Theriogenology 71 (2009) 877–883 www.theriojournal.com
Early conception factor lateral flow assays for pregnancy in the mare E. Marino a,*, W.R. Threlfall b,1, R.A. Schwarze b,1 a
Department of Veterinary Clinical Sciences, College of Veterinary Medicine, The Ohio State University, 30425 McPeck Road, Richwood, OH 43344, United States b Department of Veterinary Clinical Sciences, College of Veterinary Medicine, The Ohio State University, 601 Vernon L. Tharp Street, Columbus, OH 43210, United States Received 3 August 2007; received in revised form 11 March 2008; accepted 3 June 2008
Abstract The ECFTM lateral flow assay test is marketed to detect non-pregnancy in mares. The objectives of the present study were to determine the accuracy of the ECF test, the accuracy of the electronic reader accompanying the ECF test, and agreement between two human readers and the electronic reader. Serum samples were collected from anestrus, cycling but not inseminated, and inseminated mares, and were evaluated with the ECFTM test (EDP Biotech Company, Knoxville, TN, USA) at The Ohio State University and at the EDP Biotech Laboratory. Specificity ranged from 0.07 to 0.16, the negative predictive value ranged from 0.15 to 0.33, and accuracy ranged from 0.43 to 0.52. The electronic reader did not add improve the accuracy or predictive values of the test. Based on the electronic reader, 80.0% of the serum samples collected from the anestrus mares were false positives; Readers 1 and 2 had 60.0 and 33.3% false positives, respectively. For samples collected during the estrous cycle, 83.9% were false positives by the electronic reader, whereas Readers 1 and 2 had 43.7 and 26.4% false positives. We concluded that, regardless of whether the test strips were evaluated by a human or electronic reader, this assay was not accurate for determination of the non-pregnant mare. Published by Elsevier Inc. Keywords: Pregnancy diagnosis; Early conception factor; Mare; Equine
1. Introduction A transrectal examination (manual palpation or ultrasonography) is the most reliable method of early pregnancy diagnosis in the mare. Unfortunately, these methods do not allow reliable detection of the nonpregnant mare much earlier than the onset of spontaneous luteolysis in these mares (approximately
* Corresponding author. Tel.: +1 614 307 9880. E-mail addresses: [email protected] (E. Marino), [email protected] (W.R. Threlfall), [email protected] (R.A. Schwarze). 1 Tel.: +1 614 292 6661; fax: +1 614 292 0895. 0093-691X/$ – see front matter. Published by Elsevier Inc. doi:10.1016/j.theriogenology.2008.06.003
13 or 14 d after ovulation). However, if a non-pregnant mare could be accurately identified 6 d after ovulation, she could be immediately given prostaglandin (PGF2a) to initiate luteolysis [1], thereby shortening interestrus intervals and increasing opportunities for breeding [1]. The discovery of early pregnancy factor (EPF) and the rosette inhibition test enabled detection of pregnancy soon after breeding [2]. However, this method was time consuming, expensive, and tedious [2], and could not be readily performed under field conditions [2]. More recently, the early conception factor (ECFTM) test, originally released in 1998 by Concepto Diagnostics, Knoxville, TN, USA, was marketed as an immunoassay capable of diagnosing the non-pregnant cow within 12–48 h after ovulation
878
E. Marino et al. / Theriogenology 71 (2009) 877–883
[3]. Although this test was reported as inaccurate [3–6], the company released a similar test in 2003 for horses. The manufacturer claimed that the test could detect the ECF glycoprotein in bred mares between 3 and 30 d after ovulation. According to the manufacturer, this is a lateral flow assay that uses monoclonal and polyclonal antibodies with a colloidal gold indicator. There is apparently only a single report regarding the specificity and sensitivity of this test; in that report, it was suggested that the test was inaccurate [5]. The present study examined the improved equine ECF test (marketed in 2006) to determine if changes made to the previous marketed test (2004) improved the specificity, accuracy, and negative predictive value, and to determine if the added electronic reader (2006) improved the results of the test. The electronic reader values were also compared to two human ‘‘readers’’. The objectives of this project were to determine the accuracy of the ECF test, the accuracy of the electronic reader accompanying the ECF test, and agreement between two human readers and the electronic reader. 2. Materials and methods 2.1. Horses This study was performed during the 2007 breeding season (February 1 to May 31), using 61 Standardbred mares housed at a farm in central Ohio. Data regarding age, parity, previous breeding history, and lactation were not specifically recorded.
2.2.2. Experiment 2 Fifteen mares that had ovulated and had at least one apparently normal estrous cycle (with continued cyclicity) were used. For two consecutive weeks, transrectal examinations (to monitor ovarian activity) were done as described in Experiment 1 and blood collection was done twice weekly (Monday and Friday). None of the mares were bred until after all blood samples had been collected. 2.2.3. Experiment 3 This experiment used 61 cycling mares, including the 26 previously used in Experiments 1 and 2. All mares in estrus with an ovarian follicle 35 mm in diameter were artificially inseminated with fresh extended semen. Mares were bred every 48 h until ovulation occurred; most were bred at least twice and some three times. Ovulation (Day 0) was confirmed by transrectal palpation or ultrasonography after insemination. Blood samples were collected on Day 6 (range, Days 4–7), Day 10 (range, Days 10–12), Day 14 (range, Days 13–15), Day 18 (range, Days 17–19), and Day 35. Day 6 was chosen as the critical time to give PGF2a to induce premature luteolysis and estrus in non-pregnant mares [7]. Pregnancy diagnosis was conducted with transrectal ultrasonography on Day 18; mares that were pregnant on Day 18 were re-confirmed as pregnant on Day 35. The Day-35 blood samples were collected only on mares deemed pregnant on the basis of a transrectal ultrasonographic examination on Day 18. 2.3. Blood samples
2.2. Experimental groups There were three experiments that varied by reproductive status. In these three experiments, mares were anestrus, cycling but not inseminated, and cycling and inseminated. For these experiments, the objectives were to determine if anestrus or cycling but non-bred mares would yield false positive results, and to determine the reliability of the test to detect pregnant mares.
Samples were collected (jugular venipuncture) with a 20-ga, 38 mm vacutainer needle into a 10-mL red top vacutainer tube containing no anticoagulant (Becton Dickinson, Franklin Lakes, NJ, USA). The tubes were kept cool from collection to centrifugation (refrigeration of the blood samples did not affect test results [5]) and were centrifuged (2800 g for 10 min) within 12 h after collection. 2.4. Test procedure
2.2.1. Experiment 1 Eleven anestrus mares were used. Anestrus was defined as no follicles >20 mm in diameter and no luteal tissue was detected in the ovaries by transrectal palpation and ultrasonography (Pie Medical Scanner 480, 5 MHz, linear-array transducer; Pie Medical Imaging, Netherlands) conducted every Monday, Wednesday, and Friday for three consecutive weeks. Blood samples were collected every Monday.
Testing was done immediately after centrifugation. Serum and test kits were allowed to reach room temperature before the tests were performed. The ECF lateral flow assay test was performed in accordance with the manufacturer’s instructions. Lateral flow assays are designed to rapidly detect a particular molecule; typically the molecule of interest is isolated based on binding to a specific antibody, and this
E. Marino et al. / Theriogenology 71 (2009) 877–883
binding is detected with markers that can be visualized. The test strips were comprised of nitrocellulose and overlayed with an absorption pad to carry the sample over the testing reagents. One drop of mare serum was added to a well, where it immediately contacted a reagent pad. The reagent pad contained colloidal gold-labeled monoclonal antibody for ECF. Following addition of the sample, two drops of buffer were added to help carry the sample and the monoclonal antibody across the length and width of the absorption pad. Two lines of reagents (approximately 1 mm wide) were set up (by the manufacturer) in the nitrocellulose. These lines were oriented perpendicular to the flow of the sample and carrying buffer. The control line was the farthest from the sample well. It contained an antibody, generated against the monoclonal antibody for ECF. If this line was visible following the test, it confirmed that the colloidal goldlabeled monoclonal antibody in the reagent pad was carried throughout the entire absorption pad, and thus, beyond the test line. Binding of the antibody for ECF, which was colloidal gold-labeled, to the antibody in the control line isolated and concentrated the colloidal gold particles so that they were visible along the control line.
879
Binding of these two antibodies did not depend on the presence of ECF in the sample. The test line, or indicator line, likely contained a polyclonal antibody generated against ECF. This line was positioned on the nitrocellulose strip between the sample well and the control line. It was oriented parallel to the control line and perpendicular to the flow of the sample. If the ECF glycoprotein was present in the blood serum, the ECF molecules would have bound with the gold labeled monoclonal antibodies in the reagent pad. As the sample and reagent mixture passed over the test line, the ECF molecules bound to the polyclonal antibodies in the test strip. Since the ECF was also bound to colloidal goldlabeled monoclonal antibody, this concentrated the colloidal gold particles so that they were visible along the test line. Binding along the test line was dependent on the presence of ECF in the sample, and should have not occurred in the absence of ECF. 2.5. Test interpretation The ‘C’ region was the control line and the ‘T’ region was the positive reading (Fig. 1). Therefore, if two lines
Fig. 1. Interpretation of the ECFTM test strips. One line appearing in the control region (‘‘C’’) indicated the mare was non-pregnant, whereas one line appearing in the control region and one line appearing in the test region (‘‘T’’) indicated the mare was pregnant. Two readers interpreted test strip number 65 as non-pregnant and pregnant, respectively. Any line in the test region, no matter how faint it was, was considered a positive test reading. Test strip 63 was determined pregnant by all readers and test strip 8 was determined non-pregnant by the two readers and negative (non-pregnant) by the electronic reader.
880
E. Marino et al. / Theriogenology 71 (2009) 877–883
appeared, one in the ‘C’ region and one in the ‘T’ region, the conclusion was that the mare was pregnant. However, if only one line in the ‘C’ region appeared, the conclusion was that the mare was non-pregnant. The manufacturer also provided an electronic reader (Fig. 2) that was new to the assay procedure and being tested for the first time in this study. Therefore, each ECF test was read three times by the electronic reader (it was turned off and restarted between successive readings). The reader did not consistently give the same reading on all three tests; when there was a discrepancy, the reading recorded was based on the results of the two tests that were
in agreement. For example, if the electronic reader recorded pregnant, pregnant, and non-pregnant, the final result was recorded as pregnant. Two humans (Reader 1 and Reader 2) also read the tests; their assessments were done independent of each other. Serum was also sent to the manufacturer for interpretation. The samples were stored at 4 8C between serum removal and shipping. The time elapsed between centrifugation and shipment was 2– 7 d. The serum was chilled and held at 4 8C until shipment. During shipment, the samples were maintained chilled by placing two frozen shipping cold packs with the serum and sending the package by overnight express. On the day that the samples arrived at the manufacturer, they were tested (consistently by the same operator). 2.6. Statistical analysis Each experiment was analyzed separately. The accuracy of the ECF test was evaluated by a two-bytwo table (Chi square) [8]. Specificity, accuracy, and negative predictive value (NPV) were calculated for the electronic reader and for the two human readers (Table 1). Specificity, or the true negative rate, is the proportion of non-pregnant mares diagnosed by transrectal ultrasonographic examination with a negative result (nonpregnant) at Day 18. Using the two-by-two table, there was a square for true positives, a square for true negatives, a square for false positives, and a square for false negatives. From these values, the proportions were calculated. The two-by-two table was used to calculate the specificity, the NPV and the kappa statistic [8]. The NPV was the probability that a negative ECF test result was from a non-pregnant mare. The kappa statistic represented the proportion of potential agreement, beyond chance, that was achieved. The kappa statistic usually ranges from 0.0 to 1.0, with 0.0 being no agreement (beyond chance) and 1.0 one being perfect agreement. A kappa of 1.0 is uncommon, whereas kappa of 0.6–0.8 is considered to be substantial agreement. Very rarely is a kappa statistic below zero; it means that the agreement is weaker than expected by chance. Kappa was calculated using the K-test of FREQ procedure of SAS (Microsoft Windows Release 9, 2005; SAS Institute, Cary, NC, USA). 3. Results 3.1. ECF test performance
Fig. 2. Displays of the electronic reader screen displaying a positive and a negative reading.
Based on the electronic reader, 80.0% of the serum samples collected from the anestrus mares were false positives. Furthermore, Readers 1 and 2 had 60.0 and
E. Marino et al. / Theriogenology 71 (2009) 877–883
881
Table 1 Performance of the ECFTM test, conducted at OSU and the manufacturer, on blood samples collected at various times (ovulation = Day 0) in inseminated mares. Reader 1
Reader 2
Day 6 Sensitivity Specificity Positive predictive value Negative predictive value Accuracy Kappa
0.62 0.48 0.62 0.52 0.57 0.07
0.18 0.79 0.5 0.45 0.46 0.016
0.79 0.07 0.5 0.22 0.46 0.065
0.65 0.55 0.64 0.55 0.61 0.1
Day 10 Sensitivity Specificity Positive predictive value Negative predictive value Accuracy Kappa
0.6 0.58 0.66 0.52 0.59 0.099
0.17 0.84 0.6 0.41 0.44 0.0032
0.77 0.077 0.53 0.2 0.48 0.069
0.69 0.58 0.69 0.58 0.64 0.15
Day 14 Sensitivity Specificity Positive predictive value Negative predictive value Accuracy Kappa
0.6 0.62 0.68 0.53 0.12 0.61
0.23 0.88 0.73 0.46 0.51 0.059
0.69 0.077 0.5 0.15 0.43 0.1
0.63 0.61 0.69 0.55 0.62 0.14
Day 18 Sensitivity Specificity Positive predictive value Negative predictive value Accuracy Kappa
0.64 0.64 0.72 0.55 0.64 0.16
0.22 0.84 0.67 0.43 0.44 0.0027
0.78 0.16 0.57 0.33 0.52 0.022
0.66 0.58 0.68 0.55 0.62 0.13
Day 35 Sensitivity Specificity Positive predictive value Negative predictive value Accuracy Kappa
0.52 0.4 0.83 0.13 0.5 0.02
0.17 0.6 0.71 0.11 0.24 0.054
0.86 0.4 0.89 0.33 0.79 0.071
N/Aa N/Aa N/Aa N/Aa N/Aa N/Aa
a
Electronic reader
Manufacturer
N/A values were not available.
33.3% false positives, respectively. For samples collected during the estrous cycle, 83.9% were false positives by the electronic reader, whereas Readers 1 and 2 had 43.7 and 26.4% false positives. The specificity of the electronic reader ranged from 0.07 to 0.4, whereas for Readers 1 and 2, it ranged from 0.40 to 0.64, and from 0.60 to 0.88. Negative predictive values for the electronic reader, Reader 1, and Reader 2 ranged from 0.15 to 0.33, 0.13 to 0.55, and 0.11 to 0.46. 3.2. Manufacturer results Results received from the manufacturer were similar between days. Specificity ranged from 0.55
to 0.62. Negative predictive values ranged from 0.55 to 0.58. The kappa statistic ranged from 0.10 to 0.15. The manufacturer results were similar to OSU results. Day 35 results were not available from the manufacturer. 3.3. Agreement between readers Each test was read by an electronic reader provided by the manufacturer of the test and two human readers. Between the two human readers, the kappa coefficient was 0.049. Between the electronic reader and Reader 1, the kappa coefficient was 0.013, and between the electronic reader and Reader 2, the kappa coefficient
882
E. Marino et al. / Theriogenology 71 (2009) 877–883
was 0.029. For all three readers, the kappa coefficient was 0.011. The electronic reader was used to read each test three times. If the readings did not agree, the two readings that were the same were taken to be the electronic reader’s reading. Combined for all three experiments, the electronic reader did not give the same result for all three readings of a single sample on 63 of 396 tests (16% of the time). The kappa statistics for the electronic reader ranged from 0.10 to 0.071. For Readers 1 and 2, the kappa statistic ranged from 0.02 to 0.16, and from 0.016 to 0.059, respectively. The kappa statistic suggested no agreement between Readers 1 and 2, nor between the electronic reader and the human readers. 3.4. Agreement between Reader 1 and the manufacturer Combined for all samples, the kappa coefficient between Reader 1 and the manufacturer was 0.55. 4. Discussion In the current experiment, the accuracy of the ECF test results was compared to the actual pregnancy status (transrectal palpation and ultrasonography on Day 18), with a follow-up examination on Day 35. The electronic reader had 80% and 83.3% false positive for anestrus mares and cycling mares that were not inseminated, respectively. Based on the assumption that ECF is only present upon fertilization, the positive test results before breeding and in anestrus mares were unexpected. These false positive results during estrus were attributed to cross-reactivity with other proteins present in the serum [5]. However, this does not explain false positives in anestrus mares that were not cycling nor having follicular activities. Detection of ECF in anestrus mares may indicate that the test detects ECF produced by nonembryonic tissue. During the first 18 d after breeding, there were still approximately 40% false positives. According to the electronic reader, the rate of false positives decreased after breeding. The performance of the ECF test was similar among Days 6, 10, 14, and 18. The main objective of this study was to determine the pregnancy status of mares at Day 6 after ovulation. Pregnancy was not determined until Day 18, therefore the main statistical error for Day 6 is whether the mare was truly pregnant or non-pregnant. Due to embryonic loss, a mare that was pregnant on Day 6 may not have been pregnant on Day 18. However, comparing the statistics for Days 6 and 18, the values
were relatively similar for Reader 1. We inferred that even though the true pregnancy status on Day 6 was not known, based on the statistical information provided, it would be similar to the results reported. For the test to be beneficial in the equine breeding industry, these values need to be close to the values for transrectal ultrasonographic examination on Day 15 (0.98 and 0.90) [5]. If these values are reached, then only a 2% risk of iatrogenic early embryonic loss due to treatment with PGF2a to induce premature luteolysis and estrus. For most breeders, a 2% risk would be acceptable. Results for Day 35 varied from the other days, with more accurate results from the electronic reader. The 0.79 accuracy of the electronic reader on Day 35 suggests that it does not matter if the mare is pregnant or non-pregnant, it will always read close to 80% pregnant. The overall results were not similar among readers. Furthermore, the present results were not similar to those reported in other studies [3,5]. The specificity in Horteloup et al. [5] ranged from 0.14 to 0.33. In Cordoba et al. [3], this value was 0.04 for specificity for the cow. The accuracy in Cordoba et al. [3] was not higher than 0.67 and in Horteloup et al. [5], the accuracy was not higher than 0.60. Cordoba et al. [3] concluded that induced embryonic loss due to PGF2a administration would occur frequently if this test was used as a means of pregnancy detection in the cow. The observed specificity for the electronic reader ranged from 0.07 to 0.40. Therefore, the electronic reader was not capable of accurately detecting nonpregnant mares. The electronic reader was most accurate with a specificity of 0.40 for Day 35. The range of Reader 1’s specificity, 0.40–0.64 seemed better than the results in Cordoba et al. [3] and in Horteloup et al. [5]. However, Reader 1’s values were still well below the target value of 0.98. Based on all of these data, we concluded that neither the test nor the electronic reader were accurate. The negative predictive values for the electronic reader ranged from 0.15 to 0.33. This suggests that only 15–33 out of 100 mares diagnosed non-pregnant were truly non-pregnant. If mares diagnosed as non-pregnant using the ECF test are given PGF2a to hasten the onset of estrus, 67–85% of the mares would suffer from early embryonic loss due to administration of PGF2a. The negative predictive value range for Reader 1 was 0.13– 0.55, meaning that 45–87% of the mares in this study could have iatrogenic early embryonic losses if PGF2a was administered. A kappa statistic was calculated to determine overall agreement of the electronic reader versus the observed results (pregnant or non-pregnant). All of the kappa
E. Marino et al. / Theriogenology 71 (2009) 877–883
coefficients were below zero, except for Day 35, indicating that agreement of actual results was weaker than expected by chance. For Day 35, the kappa coefficient was 0.071, meaning there was no agreement. Kappa coefficients below zero are rare. Not only was there no agreement between the reader and the actual results, but also the agreement was weaker than expected by chance. Comparing the kappa statistic of Reader 1, the range was 0.02 to 0.16. These numbers were still very small, indicating almost no agreement between test results and actual pregnancy status. Results from the manufacturer were very similar to Reader 1, suggesting that both interpreted the tests similarly. The kappa statistic between Reader 1 and the manufacturer also suggested that interpretation of the tests was similar. This was the highest kappa statistic reported in this study. Following the guidelines for kappa statistics, a 0.55-kappa falls in the middle range of good agreement [8]. Although this agreement was not excellent, it was the best agreement between any of the other readers. There are many possible reasons the kappa statistics are so small. Perhaps the antibody used in the test was non-specific, or at least it was not specific against the antigen ECF. The test strips may not have been working properly, too much or too little gold was attached to the antibody, or the amount of antibody in the test strip was inappropriate. Perhaps there was a pregnancy variant that was detectable by Day 6; however, ECF may not be that variant. Reasons for the electronic reader’s poor results are not addressable here, since this product is sold to make the reading of the test more accurate. The electronic reader may have read any discoloration in the test strip; in that regard, it may have read the control line instead of the indicator line. In conclusion, although this test would be of great value to equine reproductive management if it worked properly, it is clear that there would be considerable loss of pregnancies if the ECF test was used to detect non-
883
pregnant mares at Day 6 (or later), and based on that information, the mares were treated to induce luteolysis and estrus. Furthermore, the electronic reader did not add value to the test; neither a human-evaluated ECF test nor the electronic reader were effective at diagnosing non-pregnant mares. Therefore, transrectal palpation and ultrasonography examination remain the preferred methods of pregnancy diagnosis by Day 15 in the mare. Acknowledgements Appreciation is extended to Drs. Mossbarger and Wittum for their help in this study. References [1] Allen WR. Use of prostaglandin for synchronization of oestrus and treatment of prolonged dioestrus in mares. Acta Vet Scand Suppl 1981;77:227–39. [2] Clarke FM, Wilson S, McCarthy R, Perkins T, Orozco C. Early pregnancy factor: large scale isolation of rosette inhibition testactive polypeptides from ovine placental extracts. J Reprod Immunol 1987;2:133–56. [3] Cordoba MC, Sartori R, Fricke PM. Assessment of a commercially available early conception factor (ECF) test for determining pregnancy status of dairy cattle. J Dairy Sci 2001;84:1884–9. [4] Threlfall WR. Immunosuppressive early pregnancy factor (ISEPF) determination for pregnancy diagnosis in dairy cows. Theriogenology 1994;41:317. Abstract. [5] Horteloup MP, Threlfall WR, Funk JA. The early conception factor (ECF) lateral flow assay for non-pregnancy determination in the mare. Theriogenology 2005;5:1061–71. [6] Gandy B, Tucker W, Ryan P, Williams A, Tucker A, Moore A, et al. Evaluation of the early conception factor (ECF) test for the detection of nonpregnancy in dairy cattle. Theriogenology 2001;4:637–47. [7] Beg MA, Gastal EL, Gastal MO, Ji S, Wiltbank MC, Ginther OJ. Changes in steady-state concentrations of messenger ribonucleic acids in luteal tissue during prostaglandin F2alpha induced luteolysis in mares. Anim Reprod Sci 2005;3/4:273–85. [8] Sackett D, Haynes R, Tugwell P. Clinical epidemiology: a basic science for clinical medicine. Little Brown; 1991.