- Email: [email protected]
What is required for the validation of in vitro assays for predicting contaminant relative bioavailability? Considerations and criteria
Environmental Pollution 180 (2013) 372e375
Contents lists available at SciVerse ScienceDirect
Environmental Pollution journal homepage: www.elsevier.com/locate/envpol
Commentary
What is required for the validation of in vitro assays for predicting contaminant relative bioavailability? Considerations and criteria Albert L. Juhasz a, *, Nicholas T. Basta b, Euan Smith a a
Centre for Environmental Risk Assessment and Remediation (CERAR), University of South Australia, Building X1-17, Mawson Lakes Campus, Adelaide, SA 5095, Australia b School of Environmental and Natural Resources, The Ohio State University, Columbus, OH 43210-1085, USA
a r t i c l e i n f o
a b s t r a c t
Article history: Received 19 February 2013 Received in revised form 18 April 2013 Accepted 1 May 2013
A number of studies have shown the potential of in vitro assays to predict contaminant in vivo relative bioavailability in order to refine human health exposure assessment. Although the term ‘validated’ has been used to describe the goodness of fit between in vivo and in vitro observations, its misuse has arisen from semantic considerations in addition to the lack of defined criteria for establishing performance validation. While several internal validation methods may be utilised, performance validation should preferably focus on assessing the agreement of model predictions with a set of data which are independent of those used to construct the model. In order to achieve robust validated predictive models, a number of parameters (e.g. size of data set, source of independent soils, contaminant concentration range, animal model, relative bioavailability endpoint) need to be considered in addition to defined criteria for establishing performance validation which are currently lacking. Ó 2013 Elsevier Ltd. All rights reserved.
Keywords: Bioaccessibility Bioavailability Correlation Human health risk assessment Validation
1. Introduction In order to define remediation goals for As-, Cd- and Pbcontaminated sites, site-specific data are required to ensure an accurate assessment of potential environmental and human health risk. Site specific data are also warranted to refine default risk variables which, as a result of their conservative nature, may result in unnecessarily low clean up levels, use of additional remediation resources and deliver unwarranted remediation costs. A parameter that may be utilised to refine site specific remediation goals is contaminant relative bioavailability (RBA). RBA is a measure of the amount of the contaminant that is absorbed into systemic circulation (with comparison to a highly soluble reference dose) as a result of contaminated soil exposure (i.e. incidental ingestion of soil). It is dependent on mineralogy, the influence of soil properties (e.g. particle size distribution, pH, OC, Fe content) and the residence time of the contaminant in the soil (Ruby et al., 1999). As a result of these factors, RBA is often less than the 100% contaminant bioavailability value which is often used as a default when estimating human contaminant exposure in the risk assessment process. Adjustments to RBA may be achieved by conducting in vivo bioavailability or in vitro bioaccessibility studies. Currently, in vivo * Corresponding author. E-mail address: [email protected] (A.L. Juhasz). 0269-7491/$ e see front matter Ó 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.envpol.2013.05.008
assays (e.g. swine, primate, rodent) are the method of choice, however, these assays are complicated, expensive and time consuming (Rees et al., 2009). Due to their simplicity, speed and affordability, in vitro assays that simulate conditions in the gastrointestinal tract may be an attractive alternative for predicting contaminant RBA (Juhasz et al., 2007a). However, in order for in vitro assays to be used as a surrogate measurement of contaminant RBA, the correlation between RBA and bioaccessibility, and the validation of the relationship, should be a prerequisite for scientific as well as regulatory acceptance. Over the past 2 decades, a limited number of studies have established the relationship between in vivo RBA and in vitro bioaccessibility for As (Basta et al., 2007; Bradham et al., 2011; Denys et al., 2012; Juhasz et al., 2007b, 2009a,b, 2011; Rodriguez et al., 1999), Cd (Denys et al., 2012; Juhasz et al., 2010; Schroder et al., 2003) and Pb (Denys et al., 2012; Drexler and Brattin, 2007; Juhasz et al., 2009b; Ruby et al., 1996; Schroder et al., 2004; Smith et al., 2011). The aforementioned studies demonstrated the potential of in vitro assays (IVG, PBET, Rel SBRC-I, RBALP, SBRC-G, UMB) to predict As and/or Cd and/or Pb RBA using swine, primates or rodents (as measured by contaminant concentrations in blood/urine or accumulation in organs/bone). However, these studies were conducted on a limited number of soils and the ability of in vitro methods to accurately predict RBA for soils outside of the correlation has not been determined.
A.L. Juhasz et al. / Environmental Pollution 180 (2013) 372e375
2. Defining in vivoein vitro validation Although the term ‘validated’ has been used to describe the goodness of fit between in vivo and in vitro observations, its misuse has arisen from semantic considerations in addition to the lack of defined criteria for establishing performance validation. There are many definitions of validation, however, in the context of environmental science it may be viewed as ‘a demonstration that a model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model’ (Sargent, 1984). Although several internal validation methods are available that aim at providing a more accurate estimate of model performance (e.g. split-sample, cross-validation and bootstrapping methods), assumptions can never be fully met with empirical data. As a result, the performance of the predictive model may be overestimated when simply determined on the samples that were used to construct the model. Performance validation should preferably focus on assessing the agreement of model predictions with a set of data which are independent to those used to construct the model. In this way, the predictive performance of the linear regression models can be evaluated (Emami, 2006). This is currently lacking for in vivoein vitro linear regression models for the prediction of contaminant RBA. Although USEPA (2007) ‘Guidance for Evaluating Oral Bioavailability of Metals in Soils for Use in Human Health Risk Assessment’ provides nine criteria for the ‘validation’ of test methods (adapted from ICCVAM, 1997), criteria do not explicitly state that validation requires the evaluation of the ability of in vitro methods to predict RBA for soils independent of the initial study. In other words, criteria do not include the evaluation of predictive relationships (i.e. in vivoein vitro linear regressions) using a set of soils/materials outside of the empirical dataset used in the correlation study. Considering the aforementioned As, Cd and Pb in vivoein vitro correlations, the next step in the process to validate these relationships is to assess an independent set of soils, using both in vivo and in vitro methodologies, and to evaluate the robustness of the RBA-bioaccessibility relationships in the context of the original (correlations) predictive models. 3. Considerations for in vivoein vitro validation A number of points need to be considered prior to the implementation of the validation phase. Firstly, how many data points (independent soils) are required in order to establish the in vivoe in vitro relationship for comparison to the original linear regression model. USEPA (2007) recommend ‘sufficient data to permit a comparison of the performance of a proposed substitute test with that of the test it is designed to replace’. In vivoein vitro models to date have utilised between 5 and 19 contaminated soils. From the literature it is unclear the minimum number of soils necessary to develop in vivoein vitro models and how many independent soils should be utilised to assess the predictability of the correlation. The greater the number of independent soils utilised to validate the model the better, however, the number of soils utilised is often limited by either the research outcomes to be achieved and/or the cost to undertake the studies. Secondly, many of the in vivoein vitro relationships developed to date have utilised soil predominantly contaminated as a result of mining activities where contaminant mineralogy is a critical factor influencing RBA and bioaccessibility. Conceivably, in vivoein vitro relationships may differ for other (non-mine site) source materials where discrete minerals are less abundant (i.e. contaminants present as sorbed species). Validation of linear regression models for predicting contaminant RBA in mine impacted materials is a noteworthy pursuit in the context of refining human health exposure for populations in the vicinity of mining activities. However,
373
the applicability of such models needs to be established for other contaminated matrices or linear regression models developed using multiple source materials. A third consideration for in vivoein vitro predictive models is what contaminant concentration range should be utilised for model development and validation. Arsenic, Cd and Pb contaminated soils that have been included in in vivoein vitro predictive models are, in some cases, up to three orders of magnitude higher than the relevant health investigation levels. As detailed during a recent workshop on the ‘Bioavailability and bioaccessibility of inorganic contaminants in soil’ (BARC, February 2011), it may be appropriate for a bioavailability adjustment factor to be included for the refinement of exposure for soils that are up to ten-fold higher than the relevant health investigation levels as this may impact human health risk assessment outcomes. Conceivably, inclusion of a bioavailability adjustment factor for highly contaminated soils (orders of magnitude higher than the relevant health investigation levels) may not influence the decision to remediate or manage a contaminated site. In order to ensure the applicability of in vivoein vitro predictive models, their development and validation should utilise soils with ‘environmentally relevant’ contaminant concentrations. A fourth point for consideration is the cost and ethical considerations associated with the in vivo assessment of contaminant soils. Although both are major factors restricting the quantity of RBA data being generated, there has been a slow but steady increase in data over recent years in both peer and non-peer reviewed (grey) literature. As a result of the collegiality of the RBA-bioaccessibility research community, contaminated soils that have undergone in vivo assessment may be provided to other groups for in vitro research. While information generated from additional studies may add to the body of knowledge on RBAbioaccessibility for specific datasets, an issue associated with the utilisation of these soils for model validation is the potential difference in methodologies for deriving in vivo data compared to the primary dataset (i.e. animal model; endpoint for the assessment of RBA). To date, there is a lack of information regarding the relationship between contaminant RBA generated using different animal models (swine, primate, rodent etc.) and different endpoints (e.g. area under the blood As time curve versus urinary As excretion). In order to overcome these potential issues, in vivo data used for model validation should be generated using the same methodology as the initial in vivoein vitro correlation or the in vivo RBA relationship established for different methodologies.
4. Criteria for establishing performance validation In addition to the above mentioned considerations, defined criteria for establishing performance validation are yet to be defined. What specified performance standards are required in order to validate in vivoein vitro correlations? Wragg et al. (2011) detailed a number of criteria for the development of a harmonised in vitro bioaccessibility methodology (Unified BARGE Method; UMB) for predicting As, Cd and Pb RBA in contaminated soil and acceptability criteria for in vivoein vitro correlation. The proposed acceptability criteria were based on practices used in the pharmaceutical industry as detailed in guidance developed by the U.S. Department of Health and Human Services Food and Drug Administration (1997) in addition to data from soil extraction literature. Criteria included: 1. A linear relationship between in vivo and in vitro data with a correlation of coefficient (r) > 0.8 and a slope > 0.8 and <1.2; 2. A within-laboratory repeatability of 10% RSD; and
374
A.L. Juhasz et al. / Environmental Pollution 180 (2013) 372e375 100
A In Vivo Relative Bioavailability (%)
In Vivo Relative Bioavailability (%)
100
80
60
40
20
0
0
20
40
60
80
80
60
40
20
0
100
B
0
20
In Vitro Bioaccessibility (%) 100
C In Vivo Relative Bioavailability (%)
In Vivo Relative Bioavailability (%)
100
80
60
40
20
0
0
20
40
60
80
40
60
80
100
In Vitro Bioaccessibility (%)
100
In Vitro Bioaccessibility (%)
D
80
60
40
20
0
0
20
40
60
80
100
In Vitro Bioaccessibility (%)
Fig. 1. Hypothetic example of an in vivoein vitro correlation (A; -) and its comparison to independent data sets (,) for model validation (BeD). B shows the example where there is no significant difference in the slopes and y intercepts of the original in vivoein vitro correlation and the independent data set (i.e. model validation). C shows partial validation of the in vivoein vitro linear regression model whereby there is no significant difference in the slopes but a significant difference in the y intercepts. In contrast, D shows significant differences in the slopes and y intercepts of the two in vivoein vitro linear regression models (i.e. validation criteria are not met).
3. A between-laboratory reproducibility of 20% RSD (Wragg et al., 2011) Criterion 1 refers to the strength of the correlation between RBA and bioaccessibility data while criteria 2 and 3 refer specifically to repeatability and reproducibility conditions for the in vitro methodology. Although criteria 2 and 3 could be equally applied as performance conditions for the within-laboratory repeatability and between-laboratory reproducibility of in vivo data, acceptability criteria are lacking for the validation of in vivoein vitro correlations. In the context of model validation, criterion 1 should also be applied to the in vivoein vitro relationship generated following the assessment of the independent soil set while an additional criterion is required in order to determine the performance of the independent observations to those observations used to construct the model. For example, when the slope and y intercepts of the in vivoein vitro relationships (original versus independent correlations) are compared, do they differ significantly (P < 0.05 or some other probability level) or are they in agreement (see Fig. 1)? Conceivably, some soils may not fit model predictions due to the influence of matrix and or contaminant properties which vary from the norm. Identification of these outlier soils is important in order to facilitate research to enhance our understanding of parameters, both physico-chemical and biological, that influence contaminant release and uptake. Gaining a greater understanding of factors that influence in vivoein vitro relationships will lead to the development of more robust (validated) models and greater confidence in
the used of these predictive models for estimating contaminant RBA for the refinement of human health exposure. 5. Conclusions Contaminant RBA is an important parameter for the refinement of exposure for human health risk assessment. However, in order to overcome the time, expense, reproducibility and ethical issues associated with animal assays, in vitro bioaccessibility methodologies have been developed and shown to have the potential to act as a surrogate for in vivo RBA assessment. While strong in vivoein vitro correlations have been demonstrated, the next phase of research needs to focus on the validation of these RBA-bioaccessibility relationships. However, prior to the initiation of the validation phase, consensus is required between researchers, risk assessors, practitioners and regulators regarding a number of key operational and sample-defined parameters for RBA-bioaccessibility assessment. Consensus is also required for defining and establishing performance validation criteria (which are currently lacking) in order to ensure robust, defensible in vivoein vitro linear regression models for refining human health exposure. Acknowledgements The authors would like to acknowledge the support of the Centre for Environmental Risk Assessment and Remediation, University of South Australia and the School of Environmental and Natural Resources, The Ohio State University.
A.L. Juhasz et al. / Environmental Pollution 180 (2013) 372e375
References Basta, N.T., Foster, J.N., Dayton, E.A., Rodriguez, R.R., Casteel, S.W., 2007. The effect of dosing vehicle on arsenic bioaccessibility in smelter-contaminated soil. Journal of Environmental Health Science Part A 42, 1275e1281. Bradham, K.D., Scheckel, K.G., Nelson, C.M., Seales, P.E., Lee, G.E., Hughes, M.F., Miller, B.W., Yeow, A., Gilmore, T., Harper, S., Thomas, D.J., 2011. Relative bioavailability and bioaccessibility and speciation of arsenic in contaminated soils. Environmental Health Perspectives 119, 1629e1634. Denys, S., Caboche, J., Tack, K., Rychen, G., Wragg, J., Cave, M., Jondreville, C., Feidt, C., 2012. In vivo validation of the unified BARGE method to assess the bioaccessibility of arsenic, antimony, cadmium, and lead in soils. Environmental Science and Technology 46, 6252e6260. Drexler, J.W., Brattin, W.J., 2007. An in vitro procedure for the estimation of lead relative bioavailability: with validation. Human and Ecological Risk Assessment 13, 383e401. Emami, J., 2006. In vitroein vivo correlation: from theory to application. Journal of Pharmacy and Pharmaceutical Science 9, 169e189. ICCVAM (Interagency Coordinating Committee on the Validation of Alternate Methods), 1997. Validation and Regulatory Acceptance of Toxicological Test Methods: a Report of the Ad Hoc Coordinating Committee on the Validation of Alternative Methods. NIH Publication 97-3981. National Institute of Environmental Health Sciences, Research Triangle Park, NC. Juhasz, A.L., Weber, J., Naidu, R., Gancarz, D., Rofe, A., Todor, D., Smith, E., 2010. Determination of relative cadmium bioavailability in contaminated soils and it prediction using in vitro methodologies. Environmental Science and Technology 44, 5240e5247. Juhasz, A.L., Weber, J., Smith, E., 2011. Influence of saliva, gastric and intestinal phases on the prediction of As relative bioavailability using the unified Bioaccessibility Research Group of Europe method (UBM). Journal of Hazardous Materials 197, 161e168. Juhasz, A.L., Smith, E., Weber, J., Naidu, R., Rees, M., Rofe, A., Kuchel, T., Sansom, L., 2009a. Assessment of four commonly employed in vitro arsenic bioaccessibility assays for predicting in vivo arsenic relative bioavailability in contaminated soils. Environmental Science and Technology 43, 9887e9894. Juhasz, A.L., Smith, E., Weber, J., Naidu, R., Rees, M., Rofe, A., Kuchel, T., Sansom, L., 2009b. Evaluation of SBRC-gastric and SBRC-intestinal methods for the prediction of in vivo relative lead bioavailability in contaminated soils. Environmental Science and Technology 43, 4503e4509. Juhasz, A.L., Smith, E., Weber, J., Rees, M., Rofe, A., Kuchel, T., Sansom, L., Naidu, R., 2007a. In vitro assessment of arsenic bioaccessibility in contaminated (anthropogenic and geogenic) soils. Chemosphere 69, 69e78.
375
Juhasz, A.L., Smith, E., Weber, J., Rees, M., Rofe, A., Kuchel, T., Sansom, L., Naidu, R., 2007b. Comparison of in vivo and in vitro methodologies for the assessment of arsenic bioavailability in contaminated soils. Chemosphere 69, 961e966. Rees, M., Sansom, L., Rofe, A., Juhasz, A.L., Smith, E., Weber, J., Naidu, R., Kuchel, T., 2009. Principles and application of an in vivo swine assay for the determination of arsenic bioavailability in contaminated matrices. Environmental Geochemistry and Health 31, 167e177. Rodriguez, R.R., Basta, N.T., Casteel, S.W., Pace, L.W., 1999. An in-vitro gastro-intestinal method to assess bioavailable arsenic in contaminated soils and solid media. Environmental Science and Technology 33, 642e649. Ruby, M.V., Davis, A., Schoof, R., Eberle, S., Sellstone, C.M., 1996. Estimation of lead and arsenic bioavailability using a physiologically based extraction test. Environmental Science and Technology 30, 422e430. Ruby, M.V., Schoof, R., Brattin, W., Goldade, M., Post, G., Harnois, M., Mosby, D.E., Casteel, S.W., Berti, W., Carpenter, M., Edwards, D., Cragin, D., Chappell, W., 1999. Advances in evaluating the oral bioavailability of inorganics in soil for use in human health risk assessment. Environmental Science and Technology 33, 3697e3705. Sargent, R.G., 1984. A tutorial on verification and validation of simulation models. In: Sheppard, S., Pooch, U., Pegden, D. (Eds.), Proceedings of the 1984 Winter Simulation Conference. IEEE 84CH2098-2. Schroder, J.L., Basta, N.T., Casteel, S.W., Evans, T.J., Payton, M.E., Si, J., 2004. Validation of the in vitro gastrointestinal (IVG) method to estimate relative bioavailable lead in contaminated soils. Journal of Environmental Quality 33, 513e521. Schroder, J.L., Basta, N.T., Si, J., Casteel, S.W., Evans, T., Payton, M., 2003. In vitro gastrointestinal method to estimate relative bioavailable cadmium in contaminated soil. Environmental Science and Technology 37, 1365e1370. Smith, E., Kempson, I.M., Juhasz, A.L., Weber, J., Rofe, A., Gancarz, D., Naidu, R., McLaren, R.G., Grafe, M., 2011. In vivoein vitro and XANES spectroscopy assessments of lead bioavailability in contaminated peri-urban soils. Environmental Science and Technology 45, 6145e6152. U.S, Department of Health and Human Services Food and Drug Administration, 1997. Guidance for Industry Extended Release Oral Dosage Forms: Development, Evaluation and Application of In Vitro/In Vivo Correlations. U.S. Department of Health and Human Services Food and Drug Administration. September 1997. U.S. Environmental Protection Agency, 2007. Guidance for Evaluating the Oral Bioavailability of Metals in Soils for Use in Human Health Risk Assessment. OSWER 9285, pp. 7e80. May, 2007. Wragg, J., Cave, M., Basta, N., Brandon, E., Casteel, S., Denys, S., Gron, C., Oomen, A., Reimer, K., Tack, K., Van de Wiele, T., 2011. An inter-laboratory trial of the unified BARGE bioaccessibility method for arsenic, cadmium and lead in soil. Science of the Total Environment 409, 4016e4030.
Contents lists available at SciVerse ScienceDirect
Environmental Pollution journal homepage: www.elsevier.com/locate/envpol
Commentary
What is required for the validation of in vitro assays for predicting contaminant relative bioavailability? Considerations and criteria Albert L. Juhasz a, *, Nicholas T. Basta b, Euan Smith a a
Centre for Environmental Risk Assessment and Remediation (CERAR), University of South Australia, Building X1-17, Mawson Lakes Campus, Adelaide, SA 5095, Australia b School of Environmental and Natural Resources, The Ohio State University, Columbus, OH 43210-1085, USA
a r t i c l e i n f o
a b s t r a c t
Article history: Received 19 February 2013 Received in revised form 18 April 2013 Accepted 1 May 2013
A number of studies have shown the potential of in vitro assays to predict contaminant in vivo relative bioavailability in order to refine human health exposure assessment. Although the term ‘validated’ has been used to describe the goodness of fit between in vivo and in vitro observations, its misuse has arisen from semantic considerations in addition to the lack of defined criteria for establishing performance validation. While several internal validation methods may be utilised, performance validation should preferably focus on assessing the agreement of model predictions with a set of data which are independent of those used to construct the model. In order to achieve robust validated predictive models, a number of parameters (e.g. size of data set, source of independent soils, contaminant concentration range, animal model, relative bioavailability endpoint) need to be considered in addition to defined criteria for establishing performance validation which are currently lacking. Ó 2013 Elsevier Ltd. All rights reserved.
Keywords: Bioaccessibility Bioavailability Correlation Human health risk assessment Validation
1. Introduction In order to define remediation goals for As-, Cd- and Pbcontaminated sites, site-specific data are required to ensure an accurate assessment of potential environmental and human health risk. Site specific data are also warranted to refine default risk variables which, as a result of their conservative nature, may result in unnecessarily low clean up levels, use of additional remediation resources and deliver unwarranted remediation costs. A parameter that may be utilised to refine site specific remediation goals is contaminant relative bioavailability (RBA). RBA is a measure of the amount of the contaminant that is absorbed into systemic circulation (with comparison to a highly soluble reference dose) as a result of contaminated soil exposure (i.e. incidental ingestion of soil). It is dependent on mineralogy, the influence of soil properties (e.g. particle size distribution, pH, OC, Fe content) and the residence time of the contaminant in the soil (Ruby et al., 1999). As a result of these factors, RBA is often less than the 100% contaminant bioavailability value which is often used as a default when estimating human contaminant exposure in the risk assessment process. Adjustments to RBA may be achieved by conducting in vivo bioavailability or in vitro bioaccessibility studies. Currently, in vivo * Corresponding author. E-mail address: [email protected] (A.L. Juhasz). 0269-7491/$ e see front matter Ó 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.envpol.2013.05.008
assays (e.g. swine, primate, rodent) are the method of choice, however, these assays are complicated, expensive and time consuming (Rees et al., 2009). Due to their simplicity, speed and affordability, in vitro assays that simulate conditions in the gastrointestinal tract may be an attractive alternative for predicting contaminant RBA (Juhasz et al., 2007a). However, in order for in vitro assays to be used as a surrogate measurement of contaminant RBA, the correlation between RBA and bioaccessibility, and the validation of the relationship, should be a prerequisite for scientific as well as regulatory acceptance. Over the past 2 decades, a limited number of studies have established the relationship between in vivo RBA and in vitro bioaccessibility for As (Basta et al., 2007; Bradham et al., 2011; Denys et al., 2012; Juhasz et al., 2007b, 2009a,b, 2011; Rodriguez et al., 1999), Cd (Denys et al., 2012; Juhasz et al., 2010; Schroder et al., 2003) and Pb (Denys et al., 2012; Drexler and Brattin, 2007; Juhasz et al., 2009b; Ruby et al., 1996; Schroder et al., 2004; Smith et al., 2011). The aforementioned studies demonstrated the potential of in vitro assays (IVG, PBET, Rel SBRC-I, RBALP, SBRC-G, UMB) to predict As and/or Cd and/or Pb RBA using swine, primates or rodents (as measured by contaminant concentrations in blood/urine or accumulation in organs/bone). However, these studies were conducted on a limited number of soils and the ability of in vitro methods to accurately predict RBA for soils outside of the correlation has not been determined.
A.L. Juhasz et al. / Environmental Pollution 180 (2013) 372e375
2. Defining in vivoein vitro validation Although the term ‘validated’ has been used to describe the goodness of fit between in vivo and in vitro observations, its misuse has arisen from semantic considerations in addition to the lack of defined criteria for establishing performance validation. There are many definitions of validation, however, in the context of environmental science it may be viewed as ‘a demonstration that a model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model’ (Sargent, 1984). Although several internal validation methods are available that aim at providing a more accurate estimate of model performance (e.g. split-sample, cross-validation and bootstrapping methods), assumptions can never be fully met with empirical data. As a result, the performance of the predictive model may be overestimated when simply determined on the samples that were used to construct the model. Performance validation should preferably focus on assessing the agreement of model predictions with a set of data which are independent to those used to construct the model. In this way, the predictive performance of the linear regression models can be evaluated (Emami, 2006). This is currently lacking for in vivoein vitro linear regression models for the prediction of contaminant RBA. Although USEPA (2007) ‘Guidance for Evaluating Oral Bioavailability of Metals in Soils for Use in Human Health Risk Assessment’ provides nine criteria for the ‘validation’ of test methods (adapted from ICCVAM, 1997), criteria do not explicitly state that validation requires the evaluation of the ability of in vitro methods to predict RBA for soils independent of the initial study. In other words, criteria do not include the evaluation of predictive relationships (i.e. in vivoein vitro linear regressions) using a set of soils/materials outside of the empirical dataset used in the correlation study. Considering the aforementioned As, Cd and Pb in vivoein vitro correlations, the next step in the process to validate these relationships is to assess an independent set of soils, using both in vivo and in vitro methodologies, and to evaluate the robustness of the RBA-bioaccessibility relationships in the context of the original (correlations) predictive models. 3. Considerations for in vivoein vitro validation A number of points need to be considered prior to the implementation of the validation phase. Firstly, how many data points (independent soils) are required in order to establish the in vivoe in vitro relationship for comparison to the original linear regression model. USEPA (2007) recommend ‘sufficient data to permit a comparison of the performance of a proposed substitute test with that of the test it is designed to replace’. In vivoein vitro models to date have utilised between 5 and 19 contaminated soils. From the literature it is unclear the minimum number of soils necessary to develop in vivoein vitro models and how many independent soils should be utilised to assess the predictability of the correlation. The greater the number of independent soils utilised to validate the model the better, however, the number of soils utilised is often limited by either the research outcomes to be achieved and/or the cost to undertake the studies. Secondly, many of the in vivoein vitro relationships developed to date have utilised soil predominantly contaminated as a result of mining activities where contaminant mineralogy is a critical factor influencing RBA and bioaccessibility. Conceivably, in vivoein vitro relationships may differ for other (non-mine site) source materials where discrete minerals are less abundant (i.e. contaminants present as sorbed species). Validation of linear regression models for predicting contaminant RBA in mine impacted materials is a noteworthy pursuit in the context of refining human health exposure for populations in the vicinity of mining activities. However,
373
the applicability of such models needs to be established for other contaminated matrices or linear regression models developed using multiple source materials. A third consideration for in vivoein vitro predictive models is what contaminant concentration range should be utilised for model development and validation. Arsenic, Cd and Pb contaminated soils that have been included in in vivoein vitro predictive models are, in some cases, up to three orders of magnitude higher than the relevant health investigation levels. As detailed during a recent workshop on the ‘Bioavailability and bioaccessibility of inorganic contaminants in soil’ (BARC, February 2011), it may be appropriate for a bioavailability adjustment factor to be included for the refinement of exposure for soils that are up to ten-fold higher than the relevant health investigation levels as this may impact human health risk assessment outcomes. Conceivably, inclusion of a bioavailability adjustment factor for highly contaminated soils (orders of magnitude higher than the relevant health investigation levels) may not influence the decision to remediate or manage a contaminated site. In order to ensure the applicability of in vivoein vitro predictive models, their development and validation should utilise soils with ‘environmentally relevant’ contaminant concentrations. A fourth point for consideration is the cost and ethical considerations associated with the in vivo assessment of contaminant soils. Although both are major factors restricting the quantity of RBA data being generated, there has been a slow but steady increase in data over recent years in both peer and non-peer reviewed (grey) literature. As a result of the collegiality of the RBA-bioaccessibility research community, contaminated soils that have undergone in vivo assessment may be provided to other groups for in vitro research. While information generated from additional studies may add to the body of knowledge on RBAbioaccessibility for specific datasets, an issue associated with the utilisation of these soils for model validation is the potential difference in methodologies for deriving in vivo data compared to the primary dataset (i.e. animal model; endpoint for the assessment of RBA). To date, there is a lack of information regarding the relationship between contaminant RBA generated using different animal models (swine, primate, rodent etc.) and different endpoints (e.g. area under the blood As time curve versus urinary As excretion). In order to overcome these potential issues, in vivo data used for model validation should be generated using the same methodology as the initial in vivoein vitro correlation or the in vivo RBA relationship established for different methodologies.
4. Criteria for establishing performance validation In addition to the above mentioned considerations, defined criteria for establishing performance validation are yet to be defined. What specified performance standards are required in order to validate in vivoein vitro correlations? Wragg et al. (2011) detailed a number of criteria for the development of a harmonised in vitro bioaccessibility methodology (Unified BARGE Method; UMB) for predicting As, Cd and Pb RBA in contaminated soil and acceptability criteria for in vivoein vitro correlation. The proposed acceptability criteria were based on practices used in the pharmaceutical industry as detailed in guidance developed by the U.S. Department of Health and Human Services Food and Drug Administration (1997) in addition to data from soil extraction literature. Criteria included: 1. A linear relationship between in vivo and in vitro data with a correlation of coefficient (r) > 0.8 and a slope > 0.8 and <1.2; 2. A within-laboratory repeatability of 10% RSD; and
374
A.L. Juhasz et al. / Environmental Pollution 180 (2013) 372e375 100
A In Vivo Relative Bioavailability (%)
In Vivo Relative Bioavailability (%)
100
80
60
40
20
0
0
20
40
60
80
80
60
40
20
0
100
B
0
20
In Vitro Bioaccessibility (%) 100
C In Vivo Relative Bioavailability (%)
In Vivo Relative Bioavailability (%)
100
80
60
40
20
0
0
20
40
60
80
40
60
80
100
In Vitro Bioaccessibility (%)
100
In Vitro Bioaccessibility (%)
D
80
60
40
20
0
0
20
40
60
80
100
In Vitro Bioaccessibility (%)
Fig. 1. Hypothetic example of an in vivoein vitro correlation (A; -) and its comparison to independent data sets (,) for model validation (BeD). B shows the example where there is no significant difference in the slopes and y intercepts of the original in vivoein vitro correlation and the independent data set (i.e. model validation). C shows partial validation of the in vivoein vitro linear regression model whereby there is no significant difference in the slopes but a significant difference in the y intercepts. In contrast, D shows significant differences in the slopes and y intercepts of the two in vivoein vitro linear regression models (i.e. validation criteria are not met).
3. A between-laboratory reproducibility of 20% RSD (Wragg et al., 2011) Criterion 1 refers to the strength of the correlation between RBA and bioaccessibility data while criteria 2 and 3 refer specifically to repeatability and reproducibility conditions for the in vitro methodology. Although criteria 2 and 3 could be equally applied as performance conditions for the within-laboratory repeatability and between-laboratory reproducibility of in vivo data, acceptability criteria are lacking for the validation of in vivoein vitro correlations. In the context of model validation, criterion 1 should also be applied to the in vivoein vitro relationship generated following the assessment of the independent soil set while an additional criterion is required in order to determine the performance of the independent observations to those observations used to construct the model. For example, when the slope and y intercepts of the in vivoein vitro relationships (original versus independent correlations) are compared, do they differ significantly (P < 0.05 or some other probability level) or are they in agreement (see Fig. 1)? Conceivably, some soils may not fit model predictions due to the influence of matrix and or contaminant properties which vary from the norm. Identification of these outlier soils is important in order to facilitate research to enhance our understanding of parameters, both physico-chemical and biological, that influence contaminant release and uptake. Gaining a greater understanding of factors that influence in vivoein vitro relationships will lead to the development of more robust (validated) models and greater confidence in
the used of these predictive models for estimating contaminant RBA for the refinement of human health exposure. 5. Conclusions Contaminant RBA is an important parameter for the refinement of exposure for human health risk assessment. However, in order to overcome the time, expense, reproducibility and ethical issues associated with animal assays, in vitro bioaccessibility methodologies have been developed and shown to have the potential to act as a surrogate for in vivo RBA assessment. While strong in vivoein vitro correlations have been demonstrated, the next phase of research needs to focus on the validation of these RBA-bioaccessibility relationships. However, prior to the initiation of the validation phase, consensus is required between researchers, risk assessors, practitioners and regulators regarding a number of key operational and sample-defined parameters for RBA-bioaccessibility assessment. Consensus is also required for defining and establishing performance validation criteria (which are currently lacking) in order to ensure robust, defensible in vivoein vitro linear regression models for refining human health exposure. Acknowledgements The authors would like to acknowledge the support of the Centre for Environmental Risk Assessment and Remediation, University of South Australia and the School of Environmental and Natural Resources, The Ohio State University.
A.L. Juhasz et al. / Environmental Pollution 180 (2013) 372e375
References Basta, N.T., Foster, J.N., Dayton, E.A., Rodriguez, R.R., Casteel, S.W., 2007. The effect of dosing vehicle on arsenic bioaccessibility in smelter-contaminated soil. Journal of Environmental Health Science Part A 42, 1275e1281. Bradham, K.D., Scheckel, K.G., Nelson, C.M., Seales, P.E., Lee, G.E., Hughes, M.F., Miller, B.W., Yeow, A., Gilmore, T., Harper, S., Thomas, D.J., 2011. Relative bioavailability and bioaccessibility and speciation of arsenic in contaminated soils. Environmental Health Perspectives 119, 1629e1634. Denys, S., Caboche, J., Tack, K., Rychen, G., Wragg, J., Cave, M., Jondreville, C., Feidt, C., 2012. In vivo validation of the unified BARGE method to assess the bioaccessibility of arsenic, antimony, cadmium, and lead in soils. Environmental Science and Technology 46, 6252e6260. Drexler, J.W., Brattin, W.J., 2007. An in vitro procedure for the estimation of lead relative bioavailability: with validation. Human and Ecological Risk Assessment 13, 383e401. Emami, J., 2006. In vitroein vivo correlation: from theory to application. Journal of Pharmacy and Pharmaceutical Science 9, 169e189. ICCVAM (Interagency Coordinating Committee on the Validation of Alternate Methods), 1997. Validation and Regulatory Acceptance of Toxicological Test Methods: a Report of the Ad Hoc Coordinating Committee on the Validation of Alternative Methods. NIH Publication 97-3981. National Institute of Environmental Health Sciences, Research Triangle Park, NC. Juhasz, A.L., Weber, J., Naidu, R., Gancarz, D., Rofe, A., Todor, D., Smith, E., 2010. Determination of relative cadmium bioavailability in contaminated soils and it prediction using in vitro methodologies. Environmental Science and Technology 44, 5240e5247. Juhasz, A.L., Weber, J., Smith, E., 2011. Influence of saliva, gastric and intestinal phases on the prediction of As relative bioavailability using the unified Bioaccessibility Research Group of Europe method (UBM). Journal of Hazardous Materials 197, 161e168. Juhasz, A.L., Smith, E., Weber, J., Naidu, R., Rees, M., Rofe, A., Kuchel, T., Sansom, L., 2009a. Assessment of four commonly employed in vitro arsenic bioaccessibility assays for predicting in vivo arsenic relative bioavailability in contaminated soils. Environmental Science and Technology 43, 9887e9894. Juhasz, A.L., Smith, E., Weber, J., Naidu, R., Rees, M., Rofe, A., Kuchel, T., Sansom, L., 2009b. Evaluation of SBRC-gastric and SBRC-intestinal methods for the prediction of in vivo relative lead bioavailability in contaminated soils. Environmental Science and Technology 43, 4503e4509. Juhasz, A.L., Smith, E., Weber, J., Rees, M., Rofe, A., Kuchel, T., Sansom, L., Naidu, R., 2007a. In vitro assessment of arsenic bioaccessibility in contaminated (anthropogenic and geogenic) soils. Chemosphere 69, 69e78.
375
Juhasz, A.L., Smith, E., Weber, J., Rees, M., Rofe, A., Kuchel, T., Sansom, L., Naidu, R., 2007b. Comparison of in vivo and in vitro methodologies for the assessment of arsenic bioavailability in contaminated soils. Chemosphere 69, 961e966. Rees, M., Sansom, L., Rofe, A., Juhasz, A.L., Smith, E., Weber, J., Naidu, R., Kuchel, T., 2009. Principles and application of an in vivo swine assay for the determination of arsenic bioavailability in contaminated matrices. Environmental Geochemistry and Health 31, 167e177. Rodriguez, R.R., Basta, N.T., Casteel, S.W., Pace, L.W., 1999. An in-vitro gastro-intestinal method to assess bioavailable arsenic in contaminated soils and solid media. Environmental Science and Technology 33, 642e649. Ruby, M.V., Davis, A., Schoof, R., Eberle, S., Sellstone, C.M., 1996. Estimation of lead and arsenic bioavailability using a physiologically based extraction test. Environmental Science and Technology 30, 422e430. Ruby, M.V., Schoof, R., Brattin, W., Goldade, M., Post, G., Harnois, M., Mosby, D.E., Casteel, S.W., Berti, W., Carpenter, M., Edwards, D., Cragin, D., Chappell, W., 1999. Advances in evaluating the oral bioavailability of inorganics in soil for use in human health risk assessment. Environmental Science and Technology 33, 3697e3705. Sargent, R.G., 1984. A tutorial on verification and validation of simulation models. In: Sheppard, S., Pooch, U., Pegden, D. (Eds.), Proceedings of the 1984 Winter Simulation Conference. IEEE 84CH2098-2. Schroder, J.L., Basta, N.T., Casteel, S.W., Evans, T.J., Payton, M.E., Si, J., 2004. Validation of the in vitro gastrointestinal (IVG) method to estimate relative bioavailable lead in contaminated soils. Journal of Environmental Quality 33, 513e521. Schroder, J.L., Basta, N.T., Si, J., Casteel, S.W., Evans, T., Payton, M., 2003. In vitro gastrointestinal method to estimate relative bioavailable cadmium in contaminated soil. Environmental Science and Technology 37, 1365e1370. Smith, E., Kempson, I.M., Juhasz, A.L., Weber, J., Rofe, A., Gancarz, D., Naidu, R., McLaren, R.G., Grafe, M., 2011. In vivoein vitro and XANES spectroscopy assessments of lead bioavailability in contaminated peri-urban soils. Environmental Science and Technology 45, 6145e6152. U.S, Department of Health and Human Services Food and Drug Administration, 1997. Guidance for Industry Extended Release Oral Dosage Forms: Development, Evaluation and Application of In Vitro/In Vivo Correlations. U.S. Department of Health and Human Services Food and Drug Administration. September 1997. U.S. Environmental Protection Agency, 2007. Guidance for Evaluating the Oral Bioavailability of Metals in Soils for Use in Human Health Risk Assessment. OSWER 9285, pp. 7e80. May, 2007. Wragg, J., Cave, M., Basta, N., Brandon, E., Casteel, S., Denys, S., Gron, C., Oomen, A., Reimer, K., Tack, K., Van de Wiele, T., 2011. An inter-laboratory trial of the unified BARGE bioaccessibility method for arsenic, cadmium and lead in soil. Science of the Total Environment 409, 4016e4030.