Impact of abundance data errors on the uncertainty of an ecological water quality assessment index

Ecological Indicators 60 (2016) 746–753

Contents lists available at ScienceDirect

Ecological Indicators journal homepage: www.elsevier.com/locate/ecolind

Impact of abundance data errors on the uncertainty of an ecological water quality assessment index Sacha Gobeyn ∗ , Elina Bennetsen, Wout Van Echelpoel, Gert Everaert, Peter L.M. Goethals Ghent University, Laboratory of Environmental Toxicology and Aquatic Ecology, J. Plateaustraat 22, B-9000 Ghent, Belgium

a r t i c l e

i n f o

Article history: Received 15 May 2015 Received in revised form 28 July 2015 Accepted 31 July 2015 Keywords: Ecological water quality assessment Uncertainty analysis Abundance data Macroinvertebrates Virtual experiments River management

a b s t r a c t Increased awareness about the uncertainty of ecological water quality (EWQ) assessment tools in river management has led to the identification of the underlying uncertainty sources and the quantification of their effect on assessment. More specifically, with respect to macroinvertebrate-based EWQ assessment, use of erroneous abundance data has been identified as a (possible) source of uncertainty. In this paper, the effect of erroneous abundance data on the uncertainty of an EWQ assessment index was investigated. A model simulation based method, the virtual ecologist approach, was used to estimate the impact of abundance data errors on the uncertainty of the Multimetric Macroinvertebrate Index Flanders (MMIF). The results of this study show that the effects of relative small errors on the MMIF and assessment are limited. Additionally, it is observed that uncertainties due to abundance errors increase with decreasing EWQ (i.e. lower MMIF). This is important, since decision-makers typically formulate management actions for rivers with a low EWQ. In short, the innovative virtual ecologist approach proved to be very successful to research the index uncertainty and present a unique insight in the functioning of the assessment index. © 2015 Elsevier Ltd. All rights reserved.

1. Introduction Ecological water quality (EWQ) assessment of freshwater ecosystems is subject to uncertainty and errors. Decision-makers are increasingly aware of the importance of including uncertainty in the assessment of the aquatic environment (Uusitalo et al., 2015). Consequently, it is a matter of concern that the sources of the uncertainty in the assessment are identified and effects are quantified (Clarke et al., 2002). For instance, macroinvertebrate-based EWQ assessment is often based on presence-absence and abundance data to calculate a multimetric index (Hering et al., 2003, 2004). As such, errors in these data will influence the precision of the multimetric index calculation, EWQ assessment and decisions in river management. Clarke and Hering (2006) state that variations in macroinvertebrate field data are due to sampling variations, sampling method, natural temporal variation (i.e. variations caused by reasons other than stress or pollution), sample processing and errors in taxonomic identification. Several studies have researched the effect of these variations and errors on EWQ assessment (Clarke et al., 2002, ˇ 2006; Haase et al., 2006; Lorenz and Clarke, 2006; Sporka et al., 2006; Vlek et al., 2006; Johnson et al., 2012). Sampling variations

∗ Corresponding author. Tel.: +32 9 264 39 96; fax: +32 9 244 44 10. E-mail address: [email protected] (S. Gobeyn). http://dx.doi.org/10.1016/j.ecolind.2015.07.031 1470-160X/© 2015 Elsevier Ltd. All rights reserved.

are observed variations within one site, caused by a spatial heterogeneity in microhabitats and distribution of macroinvertebrate species in these habitats. Lorenz and Clarke (2006) introduced the term “sample coherence” with the aim to assess within site variability. They concluded that sample coherence, expressed in several similarity indices, between replicates in one site is high. Clearly, the precision of the used sampling method will be influenced by the range of habitats (i.e. area) which are sampled and the number of sampling units (Barbour et al., 2003). With respect to number of sampling units, Vlek et al. (2006) investigated the effect of sample size (i.e. physical size, expressed as the length over which the river is sampled) on metric uncertainty by comparing the variance of several metrics for an increased sample size. As expected, the precision (uncertainty) of all metrics increased (decreased) as the sample size increased. The sample size needed to gain a certain degree of precision was, however, different for every metric. Furthermore, with respect to natural variations, a high “seasonal coherence” was observed, indicating that samples taken in the same season show higher similarity than samples taken in different seasons (Lorenz and Clarke, 2006). This was confirmed by ˇ Sporka et al. (2006) and Johnson et al. (2012), who observed an effect of seasonality on the samples and calculated metrics. In a final example, Clarke et al. (2006) researched the uncertainty due to sample processing by quantifying the effect of sub-sampling on a number of metrics. Sub-sampling is carried out by selecting a representative part of the sample and is required according to the

S. Gobeyn et al. / Ecological Indicators 60 (2016) 746–753

747

STAR-AQEM method (Clarke and Hering, 2006). Clarke et al. (2006) took replicates of sub-samples to assess the effect of this procedure on the metric uncertainty. They concluded that sub-sampling of the field sample can cause variations in several metrics up to 50%. Apart from sampling variations and method, temporal variations and sample processing, errors in the taxonomic identification are a source of uncertainty in EWQ assessment (Clarke and Hering, 2006). The taxonomic identification covers the identification of species and the abundance of each unique species. The latter data are often used to calculate metrics which account for community evenness and/or diversity (for example, the Shannon Wiener Diversity index). As indicated by Haase et al. (2006) and Jones (2008), the errors in species identification will influence the assessment results. Additionally, the variations and errors in the identification of species abundance could also have an impact on the uncertainty of the EWQ assessment. However, to the authors’ knowledge, this impact has yet to be quantified. In this research, we investigated the impact of errors in species abundance data on a multimetric index and the coupled EWQ assessment. To do so, we used the virtual ecologist approach of Zurell et al. (2010) which has received increasing attention to assess the quality of sampling protocols, data collection and model evaluation. In a virtual experiment, data are simulated by an observer model and used as input for a simulation model, which, in our case, is the multimetric index. We used an observer model to simulate errors on abundance data and researched the effect of these errors on the multimetric index. The paper is structured as follows: in Section 2, we present the set-up of the virtual experiment, the used data and multimetric index. In Section 3, we present the results of the experiment. Finally, in Section 4, we discuss and summarise the findings of the experiment and the implications for EWQ assessment.

of the multimetric index (Section 2.4). In order to estimate the effect of the abundance error on the assessment, the true metric results are compared with the MC simulations (Section 2.5).

2. Materials and methods

The multimetric index used in this study was the Multimetric Macroinvertebrate Index Flanders (MMIF) (Gabriels et al., 2010). The MMIF is a multimetric index that aggregates five metrics accounting for evenness, species richness and sensitivity properties of the macroinvertebrate community. The included metrics are the taxa richness (TAX), the number of Ephemeroptera, Plecoptera and Trichoptera (EPT), Number of other (i.e. non-EPT) Sensitive Taxa (NST), the Shannon-Wiener Diversity (SWD) index and the Mean Tolerance Score (MTS). A scoring system is appointed to each metric based on defined reference values. These score values range from zero to four with four being assigned to the metric values nearest to the reference value. The five scores are summed and subsequently divided by 20 to obtain a discrete value for the MMIF between 0 and 1, representing an ecologically unfavourable and favourable status of the water body, respectively. These values are then classified in 5 classes – Bad, Poor, Moderate, Good and High – based on quality class ranges which are defined for every type of water body through an intercalibration exercise (Buffagni and Furse, 2006; Gabriels et al., 2010). Since abundance data are only used to calculate the SWD, we only analysed the SWD metric and its related score (SWDs ), as well as the effect of this metric on the numerical MMIF value and MMIF class (MMIFc ).

A schematic representation of the virtual experiment is presented in Fig. 1. In the top panel of the figure, the methodology of a field experiment is shown. A sample of the river’s macroinvertebrate community is taken, the sample is processed and the present taxa and abundance are identified. The taxa and abundance data (Section 2.1) are used as input for the multimetric index (Section 2.2), so that the EWQ of the river can be assessed. In the lower panel of Fig. 1, simulated data are generated with an observer model and the original data (Section 2.3). In this experiment, the observer model accounts for errors in the abundance data due to miscounts, misidentification and erroneous estimates. The simulated data are used to calculate the multimetric index and to assess the EWQ. Monte Carlo (MC) simulations are used to estimate the uncertainty

2.1. Data The dataset consists of samples of macroinvertebrates collected by the Flemish Environment Agency (VMM). The macroinvertebrate samples were collected by the VMM over a period of 20 years. Throughout this period, the VMM has monitored the EWQ at more than 2500 locations in Flanders spread over different water bodies (Boets et al., 2013). All data were collected using the sampling methodology described by De Pauw and Vanhooren (1983) and Gabriels et al. (2010). The macroinvertebrate community was sampled using kick sampling with hand nets (De Pauw and Vanhooren, 1983) or artificial substrates (De Pauw et al., 1986, 1994). In the laboratory, the macroinvertebrate species were picked and identified based upon the determination key of De Pauw and Vannevel (1991) and the reference taxa list of Gabriels et al. (2010). The abundance of all present taxa was counted or estimated in a tray. An estimate of the abundance was done when more than 10 instances of the species were present. Typically, this was done by dividing the tray in a number of subsections and counting the species abundance in one subsection of the tray. Finally, the count in this one subsection was multiplied by the number of equal subsections in the tray (internal communication and VMM (2014)). In this study, abundance data between 2000 and 2012 of the VMM data base were used. The dataset comprises 7260 unique samples collected in 2682 sampling locations. 2.2. Multimetric index

2.3. Observer model Fig. 1. Illustration of the virtual experiment. In the top panel, the procedure of a typical field experiment is shown, in which data are collected by sampling the river’s macroinvertebrate community. The data are used to calculate the multimetric index and assess the EWQ of the river. In the virtual experiment (lower panel), simulated data are generated with an observer model. This model simulates data by perturbing the original dataset with an error rate (see also Section 2.3). Monte Carlo (MC) simulations are used to estimate the uncertainty of the multimetric index. The MC simulations are evaluated by comparing the simulations with the true metric values.

The observer model simulates data from the original abundance data (Section 2.1). It is assumed that abundance errors increase with increasing values of the abundance. The simulated data are generated from the original dataset by adding a multiplicative normal distributed (N) error model: A = A + A ∗ N(0, 2 )

(1)

748

S. Gobeyn et al. / Ecological Indicators 60 (2016) 746–753

Fig. 2. Mean (left) and standard deviation (right) over the RMSE values of n MC simulations for an assumed error rate  equal to 10%. In this case 400–500 MC simulations are needed for a converging solution.

with A , the vector of the simulated data, A, the vector of the original data and , the assumed error rate (%). In this experiment, different percentages for  were assumed; 5, 10, 20, 30, 50, 100%.

categorical items. The Kappa statistic normalises the overall accuracy by the accuracy that might occur by chance alone (Fielding and Bell, 1997). The calculation of this performance index is based on the confusion matrix (see Table 1).

2.4. Uncertainty analysis Monte Carlo (MC) Simulation was used to assess the error propagation and metric uncertainty (Um ). Monte Carlo Simulation can be defined as a statistical technique for stochastic model simulations and analysis for error propagation through simulations (Refsgaard et al., 2007). Monte Carlo Simulation repeatedly samples a distribution (see Eq. (1)) in order to generate a large number of simulations. In this study, the observer model was applied to generate n simulations of the original dataset. An output distribution was generated by applying the MMIF on the n simulated datasets. Finally, the metric uncertainty Um was calculated by computing the variance of this distribution. The standard deviation and range were respectively used for continuous (i.e. the SWD) and discrete (i.e. SWDs , MMIF and MMIFc ) metrics as measure for distribution variance. The difference between the maximum and minimum value of the distribution was used to define the range. To compare if the median values of two distributions were equal, the Mood’s Median test was used. To determine the number of MC simulations n needed for a statistically sound solution, the standard deviation and mean on root mean square error (RMSE) between the simulated (A’) and original data (A) were plotted (Fig. 2). It was found that for all assumed error rates, a converging solution was reached for 400–500 MC simulations. The latter was used in the remainder of the paper. 2.5. Model evaluation The MC simulations were evaluated by computing the coefficient of determination (R2 , based on the Spearman correlation coefficient) and the Cohen’s Kappa (K). The Cohen’s Kappa coefficient is a statistic which measures inter-rater agreement for Table 1 Confusion matrix. TP is true positive, FP, false positive, FN, false negative, TN, true negative.

Simulated metric Presence Absence

True metric Presence

Absence

TP FN

FP TN

3. Results 3.1. Impact of abundance data errors on the MMIF The objective of this section is to analyse the effect of the abundance error on the SWD, SWDs , MMIF and coupled class. Therefore, the MMIF described in Section 2.2 and the observer model (Section 2.3) were used with the dataset described in Section 2.1. The MC metric simulations were compared with the original metric calculation (with the original dataset) by calculating the evaluation criteria described in Section 2.5. In Fig. 3, the performance distributions for all considered metrics are plotted as a function of the error rates . These performance distributions are generated from the evaluation of the n Monte Carlo simulations. The regression lines are plotted through the median values of the performance distributions. All regression lines have a significant (p < 0.01) linear decrease of median performance for increasing error rates. It is important to note that there is a difference in the regression slope between the metrics. For instance, the R2 slope of SWD regression is steeper (20% performance decrease, going from a 0 to 100% error rate) than the slope of the MMIF regression (±3–4% decrease). This is as expected, since the SWD is one of the five metrics determining the final value of the MMIF. Overall, from this analysis, it can be concluded that the effects of abundance errors on the EWQ assessment are rather limited for small error rates (<50%). 3.2. Uncertainty analysis Although it was shown what the effects of abundance errors on the multimetric index and assessment are, it remains unclear how these abundance errors propagate through the MMIF. Furthermore, there is no indication if the uncertainties are biased as a function of the SWD, MMIF values and/or MMIF class. In this section, we want to test whether the SWD and MMIF uncertainties are equal over the range of SWD and MMIF values. To do so, we performed an uncertainty analysis (see Section 2.4) on the results generated by the observer model with a multiplicative error rate of 50%. In Fig. 4, the density plot of the SWD uncertainty as a function of SWD value is shown. Most samples have a SWD value in the

S. Gobeyn et al. / Ecological Indicators 60 (2016) 746–753

749

Uncertainty of MMIF (-)

0.20

0.15

0.10

0.05

0 0

1

2

3

SWD value (-) Fig. 5. Boxplots of SWD values for every value of uncertainty in the MMIF (UMMIF ∈ {0, 0.05, 0.10, 0.15, 0.20}). Table 2 Results of Mood’s Median test. The test compares the median of the distributions of the SWD mean (boxplots) in Fig. 5. Fig. 3. Performance distributions (R2 , upper panel, K, lower panel) for the MC simulations of SWD, SWDs , MMIF and MMIFc as a function of the abundance error rate (, %). The regression lines through the median of the performance distributions are also shown.

SWD uncertainty (-)

0.6 σ[SWD ] = -0.05SWD + 0.25 (R2: 0.19 )

0.4

0.2

0 0

1

2 SWD value (-)

3

Fig. 4. Density plot of the SWD uncertainty ([SWD]) as a function of the SWD values. The red colour indicates a high density of points, while the blue colour indicates a low density. The equation of the first order regression is also shown. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

range of 1–2.5 with an uncertainty of 0.15. Going from a low to high value, the uncertainty and the range on the uncertainty decreases. At low values, the introduced errors will result in a lower precision compared to high values. Note that there is a collection of samples – with in general lower SWD values – that have a higher uncertainty (>0.3). The samples have a relative high number of unique taxa. Therefore, more abundance records in one sample are perturbed by the observer model, which may in turn increase the uncertainty of the SWD. The question remains whether the observed bias in SWD uncertainty will propagate through the MMIF. It is expected that the bias observed in Fig. 4 will attenuate, since the SWD is only one of the five indices defining the MMIF. In the remainder of this section, the MMIF uncertainty will be analysed and discussed as a function of the SWD and MMIF class. The distribution of the SWD values (boxplot) for every possible value of MMIF uncertainty is shown in Fig. 5. It is observed

Sequential distributions

p-value

Significant

0–0.05 0.05–0.10 0.10–0.15 0.15–0.20

<0.01 0.15 <0.01 0.48

* *

that the MMIF uncertainty declines for an increasing median of the SWD distributions. In other words, the uncertainty on the MMIF will on average be high (low) when the SWD values are low (high). Additionally, for high uncertainties (=0.20), the spread on the SWD distribution is small, while the distribution range increases towards lower uncertainties. Finally, the distribution range is again smaller for an uncertainty value of 0. In order to compare the median values of the distributions for equality, the Mood’s Median test between the sequential distributions (P[SWD|UMMIF ∈ U] with U = {0, 0.05, 0.10, 0.15, 0.20}) are calculated (Table 2). The results show that SWD distributions of uncertainty 0 and 0.05 are significantly different, which is also true for 0.10 and 0.15, this on a 1% significance level. The uncertainties of the MMIF are also analysed as a function of the MMIF class (Fig. 6). The frequency distributions of the MMIF uncertainty grouped per MMIF class show a clear bias in the MMIF uncertainty as a function of the EWQ class. For instance, the first panel in Fig. 6 shows the frequency distribution of MMIF uncertainty for all samples classified under the class “Bad”. In this panel, an uncertainty value of 0.05 is the most frequent (±60%), followed by the uncertainty of 0.1 (±30%), 0 (<10%) and finally 0.15 (<0.05%). In case of a “High” class, the value of 0.05 is also the most frequent (±70%), however, it is followed by the uncertainty value of 0 (±20%), 0.1 (<0.10%), while the uncertainty of 0.15 is absent. The distribution of the MMIF uncertainty clearly shifts to lower values when the MMIF and EWQ is higher. The results presented in Figs. 5 and 6 are unexpected, since it is assumed that the bias in the SWD uncertainty would attenuate in the MMIF. In contrast, the patterns observed in the MMIF uncertainty are even more biased than the ones observed in the SWD uncertainty. The reason for this bias is explained in Appendix A and summarised here. Assume that the SWD uncertainty is equal over the range of SWD values. In this case, the MMIF uncertainty over the MMIF values will be uniform over the MMIF values, if the contribution of the SWD value to the final MMIF is equal to 20%. In other words, 20% of the variability in the MMIF is explained by the variability in the SWD values. If the MMIF variability is explained

S. Gobeyn et al. / Ecological Indicators 60 (2016) 746–753

Frequency (%)

Frequency (%)

Frequency (%)

750

80

Bad

80

60

60

40

40

20

20

0 80

0

0.05

0.1

0.15

0.2

Moderat e

0 80

60

60

40

40

20

20

0 80

0

0.05

0.1

0.15

0.2

0

Poor

0

0.05

0.1

0.15

0.2

0.05

0.1

0.15

0.2

Good

0

Uncertainty of MMIF (-)

High

60 40 20 0

0

0.05

0.1

0.15

0.2

Uncertainty of MMIF (-) Fig. 6. Frequency distribution of MMIF uncertainty as a function of the EWQ class (MMIFc ∈{Bad: 0 ≤ MMIF < 0.2; Poor: 0.2 ≤ MMIF < 0.4; Moderate: 0.4 ≤ MMIF < 0.6; Good: 0.6 ≤ MMIF < 0.8; High: 0.8 ≤ MMIF ≤ 1.0})

by a higher percentage of SWD variability (i.e. higher contribution rate SWD, see Appendix A), the SWD uncertainty will propagate substantially stronger through the MMIF model, explaining the patterns observed in Fig. 6. Additionally, lower MMIF values generally have lower SWD values (see Appendix A), therefore explaining the patterns observed in Fig. 5. 4. Discussion Uncertainty analysis of the ecological water quality indices is of major importance to river management. Typically, the uncertainty of EWQ assessment based on macroinvertebrates, is caused by sampling variations and method, natural temporal variations, sample processing and errors in taxonomic identification (Clarke and Hering, 2006). In this paper, the effect of erroneous abundance data on the multimetric index MMIF was tested by means of a simulation based approach, i.e. the virtual ecologist approach. The results of this study show that the effect of relative small errors are rather limited. However, the results of the uncertainty analysis are rather unexpected, because the error attenuation through the MMIF is lower than expected. Closer inspection shows that the current MMIF formulation assigns larger weights to abundance errors than the expected 20% benchmark. The error weights increase with decreasing MMIF values, resulting in a bias in the distribution of the errors. In addition, the virtual experiment proved to be a useful tool to thoroughly research the uncertainty propagation and the MMIF response to varying input. To the knowledge of the authors, no studies have focused on the effect of erroneous species abundance identification on the EWQ assessment or the typical abundance errors in laboratory practices. Apart from the Central Plains Center for BioAssessment (2009), suggesting a maximum error rate of 10% on the abundance identification, no clear guidelines (knowledge) exist about the acceptable (common) errors. Based on the presented results, we have shown that, for the considered case, relative small errors rates (<50%) are acceptable for a precise assessment. The presented results could therefore be used to set priorities in the time-consuming laboratory procedure (e.g. sample processing, taxonomic identification, counting (Haase et al., 2004; Vlek et al., 2006)). In this context,

priority could be given to sample processing and determination of the species, rather than precisely identifying the number of instances in a sample. The present research indicates that MMIF-based water quality assessment is robust to erroneous quantification of macroinvertebrate occurrences. These results are rather straightforward, since the species abundance are accounted for in one of the five metrics of the MMIF. Actually, the robustness of multimetric indices is one of the main reasons to use these tools as measures to monitor ecosystem health or human disturbances (Morais et al., 2004; Gabriels et al., 2010; Schoolmaster et al., 2012). However, the results of the uncertainty propagation are unexpected, since the weights assigned to the abundance errors in the MMIF are larger than the 20% benchmark (see Appendix A). In other words, the relative importance of the SWD in the MMIF is larger than expected. In addition, the observed MMIF uncertainties are in general higher when samples with a lower MMIF EWQ class are analysed; over 30% of the samples of the bad class have an uncertainty over 0.10, while for the high water quality class this percentage is three times lower (10% of the samples). A thorough analyses shows that the MMIF scoring system causes larger weights to be assigned to the SWD. Gabriels et al. (2010) developed the scoring system by equally dividing the interval between an expert-based target reference value and a value corresponding to a bad ecological water quality into five intervals. This exercise did, however, not account for the distribution of the index values in the Flemish data, resulting in an unequal increase of the five index scores for improved water quality. This causes the weights assigned to the scores and coupled uncertainties in the MMIF to be unequally divided. Whether the conclusions of this study are valid for other ecological water quality indices (used in other countries) will depend on the structure of the used multimetric index. With respect to the structure of the used index, it is important to know how many metrics of the multimetric index use abundance data to calculate the metric value. For instance, in the multimetric index I2 M2 for invertebrate-based ecological assessment of French wadeable streams, abundance data are used to calculate three of a total of five metrics (i.e. SWD, relative abundance of polyvoltine taxa and relative abundance of ovoviviparous taxa)

S. Gobeyn et al. / Ecological Indicators 60 (2016) 746–753

(Mondy et al., 2012). Consequently, it is possible that abundance errors propagate stronger through the I2 M2 than through the MMIF. An example of a multimetric index which could respond to abundance data in a same manner as the MMIF, is the multimetric index for ecological assessment of Mediterranean flatland ponds (Trigal et al., 2009). In the study of Trigal et al. (2009), five metrics with only the SWD accounting for abundance data, are identified as the main explanatory indices for ecological disturbances. Given this metric set-up, it is expected that the influence of abundance errors in Mediterranean flatland ponds multimetric index will be limited, depending on the values of the other metrics and, if implemented, scoring system. Multimetric indices are useful and robust instruments to assess the ecological state of complex river systems (Clark, 2002; Schoolmaster et al., 2012). These indices provide cost-effective information needed by policy makers to ground their management decisions (Lorenz et al., 2001). However, the accompanying index uncertainty can hamper the translation of the assessment to decisions (Bradshaw and Borchers, 2000). Awareness of this issue has caused decision-makers and scientist to map and visualise the uncertainties in a number of decision support tools (Landuyt et al., 2015; Spiegelhalter et al., 2011; Uusitalo et al., 2015). With respect to the MMIF, policy makers should take care when implementing a restoration action based on the MMIF assessment, since river systems with a lower EWQ are typically the systems for which management actions are formulated. Therefore, over- or underestimating the EWQ could result in cost-inefficient river management. In this paper, we used a simulation based method, the virtual ecologist, to research the response of the MMIF to index inputs. A series of example applications of using an observer model to test certain hypothesis exist (see Zurell et al. (2010) for an extensive list) and are not limited to the field of ecology (e.g. hydrology (Walker and Houser, 2004)). With respect to the (uncertainty) analysis of multimetric indices, this study is the first to apply the approach in a case study. This article has validated the potential of the approach to analyse in-depth the relation between multimetric index model in- and output. More importantly, the methodology was able to uncover less obvious patterns with respect to the propagation of errors. In addition to the existing tools (e.g. sampling methods, statistical analysis, modelling tools), this approach can help in better understanding the structure of multimetric indices, the relation between index in- and output and its implications for ecosystem assessment. Future research could focus on the use of the presented methodology to map uncertainties of other EWQ management support tools. To discuss one example, the simulation based method could aid to provide insight in the integrated tool of Holguin-Gonzalez et al. (2013). The tool of Holguin-Gonzalez et al. (2013) aims to identify the main priorities in EWQ management of the Cauca river (Colombia) by coupling a hydraulic and physicochemical water quality model (process based models) with aquatic ecological models (data-driven model). The model integration makes it difficult to identify how certain model inputs and coupled uncertainties propagate through the model. Therefore, the in this paper presented methodology could be used to analyse different uncertainties and possible implications for the EWQ of the Cauca river. Identifying the sources and quantifying the effect of the uncertainty on the EWQ assessment is of major importance (Clarke et al., 2002). In this study, it was shown that uncertainties in the MMIF due to abundance errors are limited, however, care should be taken when analysing samples with a lower EWQ, since uncertainties are higher. This is important, since decision-makers typically formulate management actions for rivers with a low EWQ. The results were found by applying the virtual ecologist approach, which has shown to be particularly useful to analyse the functioning of the MMIF. Consequently, this makes the simulation based method, in

751

addition to the available tools (i.e. statistical analyses, modelling approaches,...), an interesting approach to research other multimetric index formulations and their response to uncertainties. To conclude, the research presented in this paper gives a unique insight in the uncertainties of a multimetric index caused by abundance data errors and the implications for ecological water quality assessment. Acknowledgements This study was carried out within the “Ecological Modelling” project funded by the Flemish Environment Agency (VMM). Sacha Gobeyn is supported by a Bijzonder Onderzoeksfonds (BOF) project related to the Ecuador Biodiversity Network of the Vlaamse Interuniversitaire Raad–Universitaire Ontwikkelingssamenwerking (VLIR-UOS). Elina Bennetsen is supported by a BOF project (8/09924/02). The authors would like to acknowledge the Flemish Environment Agency and in particular thank Tom D’Heygere and Rob Laethem for making available the data and giving their critical insights for this research. The authors would also like to thank two anonymous reviewers for their valuable comments, which improved the manuscript considerably. Appendix A. SWD contribution rate The propagation of SWD uncertainty through the MMIF will depend on the contribution rate of the SWD value and score to the MMIF. For instance, if the variability in the MMIF is mainly explained by the variability in the SWD, the uncertainties in the SWD will propagate stronger in the MMIF. If the final MMIF variability is mainly defined by the variability of other indices, the uncertainties in the SWD will be observed to a limited extent in the MMIF. In this appendix, we define and research the contribution rate of the SWD score to the MMIF. In order to do so, the data in Section 2.1 are used. There are five scores that contribute to the value of the MMIF, i.e. the score of TAX, EPT, NST, MTS and SWD. The scores of these individual indices depend on the values of the indices and the scoring system defined for these indices (Gabriels et al., 2010). It can be assumed that all index scores contribute equally to the MMIF for the whole domain of the MMIF values, therefore, a contribution rate of 20% would be expected. The contribution rate C[SWDs ] for every sample in the dataset is calculated as follows: C[SWDs ] =

SWDs SWDs + EPT s + TAX s + MTS s + NTS s

(A.1)

with the subscript s referring to the score of the metric. In Fig. A.1, the SWD contribution rates of all samples are classified per MMIF class. The SWD contribution rates generally exceed the 20% benchmark. This is especially striking for lower MMIF classes; for instance, the median contribution rate for the class “Bad” is equal to 40%. For higher ecological water quality classes, this value decreases to the 20% benchmark. Additionally, the variance over the contribution rates decreases as a function of the increased class. The reason for this decreased median contribution and variability is the maximum possible contribution rate of the SWD as a function of the MMIF value. Closer inspection of the MMIF formula shows that a total score of 20 can be obtained from 5 indices. The maximum SWD score can only be 4. Given a “Bad” water quality class, the MMIF variability (∈[0/20,4/20]) can mainly be explained by the variability in the SWD (∈[0,4]). However in case of a “High” quality (∈[16/20,20/20]), the variability in SWD (∈[0,4]) can only account for ±20–25% to the MMIF variability. The contribution rates of the SWD score will depend on the score of the other indices and the scoring system of the indices. For the first case, the median contribution rates of the five indices are

752

S. Gobeyn et al. / Ecological Indicators 60 (2016) 746–753

the scoring system of the SWD is researched in-depth. In Fig. A.2, the distribution of the SWD values are shown, classified per MMIF class. The boxplots show an increase of the SWD value and score for an increasing value and class of the MMIF. Additionally, a decreased spread on the SWD values is observed as the MMIF class increases. The boundaries of the scoring system show that most of the values are categorised in the score class 2 and 3. It is also important to note that for a “Bad” water quality, the median value of the SWD is close to the boundary of SWD score 2. Therefore, the SWD values observed in this class are relatively high, which also confirms the findings derived from Fig. A.1 and Table A.1.

100

Cont ribut ion rates (%)

80

60

40

References 20

0

Bad

Poo r

Moderat e

Good

High

Ecolo gical wat er qualit y cla ss (MMIF) Fig. A.1. Boxplots of the contribution rates of the SWD classified for every MMIF class (see Eq. (A.1)).

Table A.1 Median contribution rates for the indices of the MMIF classified for every MMIF class. Index/Class

Bad

Poor

Moderate

Good

High

TAX EPT NST MTS SWD

0.20 0.00 0.00 0.40 0.40

0.22 0.11 0.11 0.29 0.33

0.25 0.10 0.11 0.27 0.25

0.25 0.13 0.21 0.20 0.21

0.24 0.20 0.21 0.19 0.21

Fig. A.2. Boxplots of the SWD values for every MMIF class. The scoring system is also indicated. The scoring system is water body type depend, therefore, the most used score boundaries in Flanders are shown here.

summarised in Table A.1. The table shows a large contribution of both the MTS and SWD score to the MMIF value for the MMIF class “Bad”. The contribution rates of these two scores are lower for the higher MMIF classes. An inverse pattern is observed for the contribution of the EPT and NST scores. The contribution rates for TAX is relatively constant. As indicated in the former analysis, the scoring system of the SWD will influence the contribution rates. In this analysis,

Barbour, M.T., Gerritsen, J., Snyder, B.D., Stribling, J.B., 2003. Rapid Bioassessment Protocols For Use in Streams and Wadeable Rivers: Periphyton, Benthic Macroinvertebrates, and Fish, 2nd edition. U.S. Environmental Protection Agency, Office of Water, Washington, D.C. Boets, P.L.M., Lock, K., Goethals, P., 2013. Modelling habitat preference, abundance and species richness of alien macrocrustaceans in surface waters in Flanders (Belgium) using decision trees. Ecol. Informat. 17, 73–81. Bradshaw, G.A., Borchers, J., 2000. Uncertainty as information: narrowing the science – policy gap. Ecol. Soc. 4, 1–14. Buffagni, A., Furse, M., 2006. Intercalibration and comparison – major results and conclusions from the STAR project. Hydrobiologia 566, 357–364. Central Plains Center for BioAssessment, K.B.S., 2009. Standard operating procedure for the benthic macroinvertebrate laboratory. Technical Report. University of Kansas, Kansas. Clark, M.J., 2002. Dealing with uncertainty: adaptive approaches to sustainable river management. Aquat. Conserv.: Mar. Freshw. Ecosyst. 12, 347–363. Clarke, R.T., Furse, M.T., Gunn, R.J.M., Winder, J.M., Wright, J.F., 2002. Sampling variation in macroinvertebrate data and implications for river quality indices. Freshw. Biol. 47, 1735–1751. Clarke, R.T., Hering, D., 2006. Errors and uncertainty in bioassessment methods – major results and conclusions from the STAR project and their application using STARBUGS. Hydrobiologia 566, 433–439. Clarke, R.T., Lorenz, A., Sandin, L., Schmidt-Kloiber, A., Strackbein, J., Kneebone, N.T., Haase, P., 2006. Effects of sampling and sub-sampling variation using the STAR-AQEM sampling protocol on the precision of macroinvertebrate metrics. Hydrobiologia 566, 441–459. De Pauw, N., Lambert, V., Van Kenhove, A., De Vaate, A.B., 1994. Performance of two artificial substrate samplers for macroinvertebrates in biological monitoring of large and deep rivers and canals in Belgium and the Netherlands. Environ. Monit. Assess. 30, 25–47. De Pauw, N., Roels, D., Fontoura, A.P., 1986. Use of artificial substrates for standardized sampling of macroinvertebrates in the assessment of water quality by the Belgian Biotic Index. Hydrobiologia 133, 237–258. De Pauw, N., Vanhooren, G., 1983. Method for biological quality assessment of watercourses in Belgium. Hydrobiologia 100, 153–168. De Pauw, N., Vannevel, R., 1991. Macroinvertebraten en waterkwaliteit. Determineersleutels van macroinvertebraten en beoordelingsmethoden van de waterkwaliteit, 2nd edition. Stichting Leefmilieu, Antwerpen. Fielding, A.H., Bell, J.F., 1997. A review of methods for the assessment of prediction errors in conservation presence/absence models. Environ. Conserv. 24, 38–49. Gabriels, W., Lock, K., De Pauw, N., Goethals, P.L.M., 2010. Multimetric Macroinvertebrate Index Flanders (MMIF) for biological assessment of rivers and lakes in Flanders (Belgium). Limnologica 40, 199–207. Haase, P., Lohse, S., Pauls, S., Schindehütte, K., Sundermann, A., Rolauffs, P., Hering, D., 2004. Assessing streams in Germany with benthic invertebrates: development of a practical standardised protocol for macroinvertebrate sampling and sorting. Limnologica 34, 349–365. Haase, P., Murray-Bligh, J., Lohse, S., Pauls, S., Sundermann, A., Gunn, R., Clarke, R., 2006. Assessing the impact of errors in sorting and identifying macroinvertebrate samples. Hydrobiologia 566, 505–521. Hering, D., Moog, O., Sandin, L., Verdonschot, P.F.M., 2004. Overview and application of the AQEM assessment system. Hydrobiologia 516 (1–3), 1–20. Hering, D., Buffagni, A., Moog, O., Sandin, L., Sommerhäuser, M., Stubauer, I., Feld, C., Johnson, R., Pinto, P., Skoulikidis, N., Verdonschot, P.F.M., Zahrádková, S., 2003. The development of a system to assess the ecological quality of streams based on macroinvertebrates – design of the sampling programme within the AQEM project. Int. Rev. Hydrobiol. 88, 345–361. Holguin-Gonzalez, J.E., Everaert, G., Boets, P., Galvis, A., Goethals, P.L.M., 2013. Development and application of an integrated ecological modelling framework to analyze the impact of wastewater discharges on the ecological water quality of rivers. Environ. Model. Softw. 48, 27–36. Johnson, R.C., Carreiro, M.M., Jin, H.S., Jack, J.D., 2012. Within-year temporal variation and life-cycle seasonality affect stream macroinvertebrate community structure and biotic metrics. Ecol. Indic. 13, 206–214. Jones, F.C., 2008. Taxonomic sufficiency: the influence of taxonomic resolution on freshwater bioassessments using benthic macroinvertebrates. Environ. Rev. 16, 45–69.

S. Gobeyn et al. / Ecological Indicators 60 (2016) 746–753 Landuyt, D., Van der Biest, K., Broekx, S., Staes, J., Meire, P., Goethals, P.L., 2015. A GIS plug-in for Bayesian belief networks: towards a transparent software framework to assess and visualise uncertainties in ecosystem service mapping. Environ. Model. Softw. 71, 30–38. Lorenz, A., Clarke, R.T., 2006. Sample coherence – a field study approach to assess similarity of macroinvertebrate samples. Hydrobiologia 566, 461–476. Lorenz, C.M., Gilbert, A.J., Cofino, W.P., 2001. Indicators for transboundary river management. Environ. Manage. 28, 115–129. Mondy, C.P., Villeneuve, B., Archaimbault, V., Usseglio-Polatera, P., 2012. A new macroinvertebrate-based multimetric index (I2M2) to evaluate ecological quality of French wadeable streams fulfilling the WFD demands: a taxonomical and trait approach. Ecol. Indic. 18, 452–467. Morais, M., Pinto, P., Guilherme, P., Rosado, J., Antunes, I., 2004. Assessment of temporary streams: the robustness of metric and multimetric indices under different hydrological conditions. Hydrobiologia 516, 229–249. Refsgaard, J.C., van der Sluijs, J.P., Højberg, A.L., Vanrolleghem, P.A., 2007. Uncertainty in the environmental modelling process – a framework and guidance. Environ. Model. Softw. 22, 1543–1556. Schoolmaster, D.R., Grace, J.B., Schweiger, E.W., 2012. A general theory of multimetric indices and their properties. Methods Ecol. Evol. 3, 773–781. Spiegelhalter, D., Pearson, M., Short, I., 2011. Visualizing uncertainty about the future. Science 333, 1393–1400.

753

Trigal, C., García-Criado, F., Fernández-Aláez, C., 2009. Towards a multimetric index for ecological assessment of Mediterranean flatland ponds: the use of macroinvertebrates as bioindicators. Hydrobiologia 618, 109–123. Uusitalo, L., Lehikoinen, A., Helle, I., Myrberg, K., 2015. An overview of methods to evaluate uncertainty of deterministic models in decision support. Environ. Model. Softw. 63, 24–31. ˇ Vlek, H.E., Sporka, F., Krno, I., 2006. Influence of macroinvertebrate sample size on bioassessment of streams. Hydrobiologia 566, 523–542. VMM, 2014. Beoordeling van de ecologische en chemische toestand in natuurlijke, sterk veranderde en kunstmatige oppervlaktewaterlichamen in Vlaanderen conform de Europese Kaderrichtlijn Water. Technical Report. Vlaamse Milieu Maatschappij. ˇ Sporka, F., Vlek, H.E., Bulánková, E., Krno, I., 2006. Influence of seasonal variation on bioassessment of streams using macroinvertebrates. Hydrobiologia 566, 543–555. Walker, J.P., Houser, P.R., 2004. Requirements of a global near-surface soil moisture satellite mission: accuracy, repeat time, and spatial resolution. Adv. Water Resour. 27, 785–801. Zurell, D., Berger, U., Cabral, J.S., Jeltsch, F., Meynard, C.N., Münkemüller, T., Nehrbass, N., Pagel, J., Reineking, B., Schröder, B., Grimm, V., 2010. The virtual ecologist approach: simulating data and observers. Oikos 119, 622–635.