An interactive web app for retrieval, visualization, and analysis of hydrologic and meteorological time series data

Accepted Manuscript An interactive web app for retrieval, visualization, and analysis of hydrologic and meteorological time series data Conrad E. Brendel, Randel L. Dymond, Marcus F. Aguilar PII:

S1364-8152(18)30734-5

DOI:

https://doi.org/10.1016/j.envsoft.2019.03.003

Reference:

ENSO 4407

To appear in:

Environmental Modelling and Software

Received Date: 24 July 2018 Revised Date:

25 February 2019

Accepted Date: 6 March 2019

Please cite this article as: Brendel, C.E., Dymond, R.L., Aguilar, M.F., An interactive web app for retrieval, visualization, and analysis of hydrologic and meteorological time series data, Environmental Modelling and Software (2019), doi: https://doi.org/10.1016/j.envsoft.2019.03.003. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT

Title An interactive web app for retrieval, visualization, and analysis of hydrologic and meteorological time series data

Conrad E. Brendel a*, Randel L. Dymond a, Marcus F. Aguilarb a

RI PT

Author Names & Affiliations

Via Department of Civil & Environmental Engineering, Virginia Tech; Blacksburg, VA 24060, USA

b

*Correspondence: [email protected] Tel.: +1-540-231-3478

M AN U

Highlights

SC

City of Roanoke Department of Public Works; Roanoke, VA 24012, USA

•

Web app synthesizes data from government, commercial, and local sources

•

Web app integrates data retrieval, visualization, and analysis tools

•

Rapid analyses provide valuable insights to watershed hydrologic processes

•

App structure permits implementation in other locations

TE D

Abstract

As wireless sensor networks become more prevalent tools for analyzing watershed dynamics, the integration and management of the myriad of available data streams presents a unique challenge. Webbased platforms provide tools to retrieve and visualize hydrologic and meteorologic data from various

EP

sources while addressing syntactic and semantic differences in data formats. However, current platforms are limited by a lack of data analysis tools. The Stream Hydrology And Rainfall Knowledge

AC C

System (SHARKS) app was developed to expand upon existing platforms by providing a suite of exploratory data analysis features including the ability to calculate the total precipitation depth recorded for any period, interpolate the annual recurrence interval for precipitation events, perform hydrograph separations, and calculate the volume of runoff for any period. A case study provides a conceptual description of the types of rapid analysis that can be performed using the SHARKS app. Keywords Hydrologic Monitoring; Smart Watershed; Environmental Observations; Data Visualization; Data Analysis

1

ACCEPTED MANUSCRIPT

Software Availability Software name: Stream Hydrology And Rainfall Knowledge System (SHARKS) Developers: SHARKS team

RI PT

Contact information: [email protected] Hardware required: Any web-enabled device with a modern web browser Software required: Internet browser Program languages: R, HTML, and CSS

SC

Availability: The SHARKS app is available at https://bigbadcrad.shinyapps.io/SHARKS/ and the underlying R code may be obtained on GitHub at https://github.com/bigbadcrad/SHARKS or on Zenodo (Brendel et al., 2018). Code is released under the MIT License.

dashboardthemes

1.0.2

data.table

1.11.4

dataRetrieval

2.7.3

dplyr

0.7.5

DT

0.4

EcoHydRology

0.4.12

ggplot2

2.2.1

googleAuthR

0.6.2

googleID

0.0.9001

googlesheets

0.2.2

gridExtra

2.3

leaflet

2.0.2

lubridate

1.7.4

pracma

2.1.4

Rcurl

1.95-4.10

rgdal

1.3-4

TE D

Version

AC C

EP

R Package

M AN U

Dependencies:

rwunderground

0.1.8

shadowtext

0.0.3

shiny

1.1.0

shinyalert

1.0

shinydashboard

0.7.0

shinyjs

1.0

shinyWidgets

0.4.3

stringr

1.3.1

zoo

1.8-2

2

ACCEPTED MANUSCRIPT

1. Introduction As a result of the integration of low-cost sensors and wireless communications with web services, large sensor networks are increasingly being used in environmental monitoring (Bartos et al., 2018).

RI PT

Sensor networks have been implemented in projects including habitat monitoring (Biagioni and Bridges, 2002; Mainwaring et al., 2002), environmental observation and forecasting systems (Keefer et al., 1987; Steere et al., 2000), forest fire detection (Hefeeda and Bagheri, 2009; Soliman et al., 2010), agriculture (Kim et al., 2008), and glacier research (Padhy et al., 2005). In urban areas, sensor networks in “smart” watersheds can also provide information on watershed dynamics and provide insights to issues such as

SC

flooding, runoff pollution, and degradation of aquatic ecosystems (Bartos et al., 2018). Furthermore, a new generation of intelligent stormwater networks will allow cities to expand real-time monitoring and

M AN U

control of stormwater systems to entire watersheds through the implementation of sensors and dynamic controls (Bartos et al., 2018; Kerkez et al., 2016; Lefkowitz et al., 2016; Mullapudi et al., 2017; Muschalla et al., 2014). As hydrologic sensor networks become more prevalent, the effective integration and management of the multitude of data streams — including those from government, commercial, and local sources — to produce coherent results presents a unique challenge. Synthesizing data from different sources can be difficult because each source can have a different way of navigating through

TE D

pages, menus, and files to access the data. Furthermore, syntactic and semantic differences in data formats can make it difficult to find, organize, and interpret data (Horsburgh et al., 2009). Finally, data retrieval and processing can be hindered by the sheer quantity of available data (Vitolo et al., 2015). Several platforms have been created to gather and visualize hydrologic data from various sources.

EP

Notably, the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) HydroClient (http://data.cuahsi.org/) accesses hydrologic data from over 95 sources stored in the

AC C

CUAHSI Hydrologic Information System (HIS). This system was developed to address syntactic and semantic heterogeneity in environmental data by the use of a standard Observations Data Model (ODM) database format for data storage and the WaterOneFlow web services for data communication (Horsburgh et al., 2009). The CUAHSI HydroClient also provides a basic data series viewer to plot datasets. A related platform, the Time Series Analyst (TSA) (http://data.iutahepscor.org/tsa/) is able to retrieve any dataset published to the CUAHSI HIS and provides more advanced tools for data visualization including a map-based interface, faceted filters, and the ability to display data using time series, box-and-whisker, or histogram plots. Furthermore, the TSA includes the calculation of data summary statistics (Horsburgh et al., 2016). The Great Lakes Dashboard (GLD) platform 3

ACCEPTED MANUSCRIPT

(https://www.glerl.noaa.gov/data/dashboard/GLD_HTML5.html) was specifically designed as a tool to visualize and download datasets from the North American Laurentian Great Lakes and displays plots for a multitude of different time series datasets (Smith et al., 2016). Finally, the Virtual Observatory and Ecological Informatics System (VOEIS) Data Hub (https://voeis.msu.montana.edu/) provides a suite of

RI PT

tools for users to compile, manage, visualize, and redistribute datasets. The VOEIS Data Hub allows users to upload data or retrieve data from networked sensors and includes QA/QC capabilities. Furthermore, the VOEIS Data Hub includes data access restriction features and allows users to publish data to a CUAHSI HIS HydroServer (Mason et al., 2014).

SC

Despite the advances made by these platforms, they are limited by the shortage of data analysis tools. A web application, called the Stream Hydrology And Rainfall Knowledge System (SHARKS) app

M AN U

(https://bigbadcrad.shinyapps.io/SHARKS/), was developed to expand upon existing platforms by integrating data analysis tools with data retrieval and visualization capabilities. The SHARKS app is tailored to stormwater managers, emergency managers, hydrologists, and/or meteorologists that need to understand in real-time and for retrospective analysis (1) what is the location, intensity, and average annual recurrence interval of rainfall? (2) what is the corresponding depth/discharge in streams and large storm drains? (3) what is the weather forecast? (4) what is the proportion of precipitation leaving a

TE D

watershed as runoff? (5) how does water quality change during events? The app ties together data from multiple, syntactically different sources and provides immediately useful information for decisionmaking and retrospective study (e.g. water balance, H&H modeling). The web-based interface of the SHARKS app was designed with the goal of providing an accessible platform for users possessing an

EP

understanding of basic hydrology. Although the SHARKS app was conceptualized and implemented to perform data analysis for the City of Roanoke, Virginia, the app was designed to also retrieve and analyze data for other locations. This paper describes the SHARKS app workflow and presents a case

AC C

study demonstrating how the SHARKS app can be used to provide valuable insights to urban hydrologic processes.

Analysis features were chosen for inclusion in SHARKS based on needs identified by City of Roanoke staff and SHARKS developers. The information needed from the app fit into two broad categories: (1) rapidly available rainfall and streamflow/depth information during intense precipitation events and (2) hydrologic summary information organized systematically for retrospective analysis. As a result, the design of the app was borne out of the basic need to know, report, and respond to rainfall (e.g. depth, intensity, average annual recurrence interval) and stream (e.g. depth, discharge) conditions across the 4

ACCEPTED MANUSCRIPT

City’s service area, as well as to use more complex data (e.g. separated hydrographs, direct runoff volumes, runoff volume coefficients) to better understand long-term hydrological processes. 2. Software Implementation

RI PT

2.1. Architecture SHARKS is implemented as a web-based application so that users can access the app from a web browser without installing any additional software. In addition, deploying the app online ensures that all users are using the most recent version of SHARKS. SHARKS was programmed using the open-source R

SC

language with the Shiny web application framework (https://shiny.rstudio.com/). R and Shiny were chosen for development over alternatives, such as Python and Dash, because R is generally considered to have superior data visualization features and more sophisticated statistical libraries. In contrast,

M AN U

Python is a general-purpose language which facilitates integration among different software components. Because SHARKS is focused on data analysis and visualization and was designed to be a stand-alone platform, the advantages of R’s data visualization features were decided to outweigh the advantages of Python’s ability to integrate with other web components. In addition, the Shiny framework allows developers to create web apps using entirely R code and uses a reactive programming model in which outputs update instantly as inputs are modified, thus eliminating the need for messy

TE D

event handling code (Chang et al., 2018). Furthermore, Shiny includes a library of pre-built input and output widgets for controlling apps and displaying outputs. R also includes an extensive set of packages and available code that facilitate the retrieval of hydrologic and meteorological data from the United States Geological Survey (USGS) and the National Oceanic and Atmospheric Administration (NOAA).

EP

Finally, R and Shiny are free and open-source so others can modify and adapt the publicly available SHARKS source code (Brendel et al., 2018).

AC C

2.1.1. Mechanics

The SHARKS Shiny framework consists of two components: a user interface object and a reactive server function. The app’s user interface is built primarily of R code, but was customized using HTML, CSS, and JavaScript code directly. SHARKS is controlled using pre-built Shiny widgets (e.g. Date Range Inputs, Selection Boxes, and Switches) placed in the user interface. Selections made using the control widgets are passed to the Shiny server function which performs the computations and creates output objects (e.g. Graphs, Maps, and Tables) that are passed back to the Shiny user interface for display. The Shiny server is based on the concept of reactivity, in which output objects are updated automatically as 5

ACCEPTED MANUSCRIPT

inputs are changed. Therefore, when users make selections using the control widgets in the SHARKS user interface, the server performs the computations and updates the app outputs immediately, unless told otherwise. The mechanics of the SHARKS app are described as follows.

RI PT

Data is retrieved from sources, based on the stations and time series datasets specified using control widgets, and joined by date/time to create one master data frame. The master data frame is structured with separate columns for each time series dataset and rows for each unique date/time. The various SHARKS analysis features use and reshape the data from the master data frame as necessary. Some SHARKS analysis features are performed automatically and do not require any user input. These

SC

automated analysis features include performing hydrograph separations (Section 2.4.1), calculating direct runoff volume/depth and runoff volume coefficients (Section 2.4.2), calculating total precipitation

M AN U

depth and maximum precipitation intensity, and plotting hyetographs and hydrographs. Due to the reactive Shiny framework, if users make changes using the user interface control widgets, then these analyses are re-executed and their user interface outputs are updated automatically. Other SHARKS analysis features — calculating the storm average annual recurrence interval calculation (Section 2.4.4), the interactive maps (Section 2.4.5), and the interactive plots (Section 2.4.6) — require user input. When users click on these interactive analysis features, the user inputs (e.g. date/time range selected on the

TE D

interactive plots) are passed to the server to query data from the master data frame and perform any necessary computations before updating the user interface outputs. 2.1.2. Implementation

The SHARKS app has been implemented for the City of Roanoke – a medium sized urbanized area in

EP

southwest Virginia. In order to more effectively manage the quality and quantity of stormwater runoff from City watersheds, and to improve compliance with the City’s municipal separate storm sewer

AC C

system (MS4) permit and total maximum daily load (TMDL) requirements, the City has installed a number of hydrologic and water quality instruments throughout the service area. These instruments provide the empirical basis for data-driven capital improvement spending and will also allow for the evaluation and iterative adjustment of watershed improvements over long periods of time (i.e. adaptive management).

The instruments installed throughout the City’s service area, summarized in Table 1, provide observations of hydrology and water quality at sub-hourly time steps, which creates a wealth of highquality environmental data for management and decision-making. However, this also presents a data

6

ACCEPTED MANUSCRIPT

management problem because of the large and always growing amount of raw data. In addition, semantic and syntactic differences between the data sources make it challenging to join and organize datasets because they may use different codes for the same parameter (e.g. “p01i” vs. “00045” for precipitation) or use the same name for multiple parameters (e.g. using “temperature” to describe both

RI PT

air temperature and water temperature), encode data using different file types, and have different ways of structuring data. The SHARKS app provides the necessary integration of data streams into a dynamic user interface so that City stormwater managers, emergency managers, hydrologists and/or meteorologists can easily gather information and requisite analysis and communicate it to the necessary

SC

stakeholders. The SHARKS app was also implemented as a pre-processing tool to collate rainfall and runoff time series data for calibration and validation of a forthcoming hydrologic/hydraulic modeling

Table 1: City of Roanoke Data Sources

M AN U

study.

Source

Meteorological, 5minute interval

NOAA – Roanoke Regional Airport Automated Surface Observing System (ASOS) USGS – Roanoke Nine-Gauge Network from National Water Information Service (NWIS) USGS – Lick Run Station from National Water Information Service (NWIS) USGS – Roanoke River Station from National Water Information Service (NWIS) City of Roanoke from Onset HOBOWare proprietary software.

Storm Sewer Flow Depth, 5-minute interval

Since 1/1/1948

Figure Reference Figure 2

Since 2/21/2018 Since 8/8/2016

Figure 2

Since 10/1/1990 Since 8/10/2017

Figure 2 and 8

EP

Discharge & Water Quality, 5-minute interval

Availability

TE D

Parameter

AC C

SHARKS is currently deployed for the City online at https://bigbadcrad.shinyapps.io/SHARKS/ using the shinyapps.io platform (http://www.shinyapps.io/). Hosting SHARKS online not only ensures that all users are utilizing the most recent version of the app, but also allows for remote updates to the app. The shinyapps.io platform was chosen as a low-cost alternative to configuring and maintaining an expensive server for SHARKS. In addition, shinyapps.io offers scalable hosting plans ranging from a free option to a professional option, with increasing performance with higher tier plans. The shinyapps.io platform hosts SHARKS on a virtualized server, termed an “instance”, which is served by “worker” processes which service requests to the app (RStudio, 2015). With the current hosting plan, each SHARKS instance is limited to 1024 MB of memory for computations. Because R is a single threaded application, SHARKS 7

ACCEPTED MANUSCRIPT

cannot serve two users at exactly the same time. Typically, this is not an issue because computations take place in the order of tens or hundreds of milliseconds, enabling a single R process to serve 5-30 requests per second (RStudio, 2015). However, as more users interact with a Shiny app simultaneously, the demand on the app’s resources is increased. To maximize the performance of SHARKS as additional

RI PT

users access the app, the shinyapps.io server has been configured to trigger the addition of new worker processes and application instances more rapidly to spread the computational load. In addition, SHARKS utilizes the Shiny reactive framework to compartmentalize computations to ensure that they are not reperformed unnecessarily. As usage of SHARKS increases, the hosting plan can be upgraded to increase

SC

the available memory per app instance as well as the limit of available app instances and worker processes in order to keep the app responsive for large-scale implementation. Alternatively, the City could choose to deploy SHARKS on their own server. The SHARKS source code is publicly available online

M AN U

and is released under the MIT License (Brendel et al., 2018). SHARKS can be launched locally in any R environment (Console R, Rgui, RStudio, etc.) or deployed to a local or remote server so that users may access the app from a website without needing to install R. 2.2. Graphical User Interface

The SHARKS app user interface consists of a Sidebar Menu containing the app’s inputs and options

TE D

and a Main Panel containing the visualization capabilities (Figure 1). The Sidebar Menu and Main Panel are dynamically linked, and selections on the Sidebar Menu determine the meteorological and hydrological stations, parameters, and date range for which data is retrieved. A set of eight navigational tabs — the Summary, Map, Forecast, Interactive Plots, Tables, Storm Sewer, Real-Time Flood Stages,

EP

and Help tabs (Figure 1) — are also included at the top of the Sidebar Menu and control which outputs and features are displayed in the Main Panel. The Help tab provides user with the developer contact

AC C

information and a link to download the SHARKS user manual but is not discussed further in this paper. Help buttons have been placed throughout the user interface and, when clicked, display messages providing information regarding the various app inputs, outputs, and options. The aforementioned components are described in more detail in Sections 2.2.1 through 2.2.8.

8

TE D

M AN U

SC

RI PT

ACCEPTED MANUSCRIPT

EP

Figure 1: SHARKS User Interface structure with dynamically linked Sidebar Menu and Main Panel. Sidebar Menu contains Navigational Tabs and app inputs and options. Main Panel contains app visualization capabilities.

AC C

2.2.1. Sidebar Menu

The SHARKS app Sidebar menu (Figure 1) contains inputs and options that control the content displayed in the Main Panel. The Main Panel is reactive to the Sidebar menu, and content in the Main Panel is updated immediately as changes are made to the inputs and options. Inputs located in the Sidebar menu allow users to specify the NOAA Automated Surface Observing System (ASOS) meteorological station, USGS meteorological stations and stream stations, and the date range from which data is retrieved. Although Roanoke’s NOAA ASOS and USGS stations (Figure 2) are preloaded into the SHARKS app, the app can retrieve data for any NOAA ASOS or USGS station. NOAA ASOS stations are

9

ACCEPTED MANUSCRIPT

specified by the station’s Federal Aviation Administration (FAA) ID and USGS stations are specified by the station’s USGS site ID. A drop-down list on the sidebar allows users to select which parameters are retrieved for the USGS stream stations. Available parameters are Discharge (cfs), Gauge Height (ft), Water Temperature (°C), Specific Conductance (μS/cm @ 25°C), Dissolved Oxygen (mg/L), pH, and

RI PT

Turbidity (FNU). Finally, an input is provided to allow users to specify the upstream watershed area for each stream station so that the app can calculate the direct runoff depth.

Plotting options included in the Sidebar menu control which meteorological station is used to create the app’s hyetographs and which parameter, if any, is displayed on a secondary y-axis in the app’s

SC

hydrographs. Additional options control which time zone is used to retrieve and display data, how datasets are symbolized in the SHARKS app hydrographs, and if the app should calculate the Average

AC C

EP

TE D

M AN U

Annual Recurrence Interval (ARI) for precipitation events occurring during the specified date range.

Figure 2: Roanoke meteorological stations, stream stations, and storm sewer sensor locations. Storm sewer sensor locations shown in detail in Figure 8.

10

ACCEPTED MANUSCRIPT

2.2.2. Summary Tab The “Summary” tab was designed to provide a suite of tools for data comparison including tables of summary data, runoff volume coefficients, and calculated ARIs as well as a stacked

RI PT

hyetograph/hydrograph and bar graphs of maximum flood stages. The “Data Summary” table (Supplemental Figure S1) provides an overview of the precipitation and discharge data for the meteorological and stream stations and facilitates quick comparisons between sites. The location and station ID of each station are displayed in the table, and the table provides a summary of the total precipitation depth and maximum precipitation intensity for each meteorological station and the

Data in the table can be sorted by clicking the column headers.

SC

maximum discharge, total direct runoff volume, and total direct runoff depth for each stream station.

M AN U

Runoff volume coefficients, which represent the proportion of precipitation leaving a watershed as runoff, are calculated for each stream station and meteorological station combination (Section 2.4.2) and displayed in the “Runoff Volume Coefficient Summary” table (Supplemental Figure S2). Values in the table indicate the runoff volume coefficients for the stream station and meteorological station in the respective rows and columns.

The “ARI Summary” table (Supplemental Figure S3) displays the interpolated ARI (Section 2.4.4) for

TE D

precipitation events occurring during the specified date range. The table includes the station location, station ID, and calculated ARI for each event. In addition, the table includes the start time and the duration of the window for which the ARI was calculated as well as the total precipitation depth and average precipitation intensity occurring during the window. Although the table is sorted by ARI in

EP

descending order by default, the table can be sorted by any other parameter by clicking on the column

file.

AC C

headers. A download button is also included to download the table as a comma-separated values (CSV)

The stacked hyetograph/hydrograph (Supplemental Figure S4) provides a visual display of the selected time series datasets, allowing users to identify temporal trends in data and the occurrence of precipitation events. Although only one meteorological station may be displayed in the hyetograph, there is no limit on the number of stream stations that are displayed in the hydrograph. Parameters which can be plotted in the hydrograph are discharge with baseflow, gauge height, water temperature, specific conductance, dissolved oxygen, pH, and turbidity. Discharge and baseflow are plotted on the primary y-axis of the hydrograph by default. However, the Sidebar menu can be used to plot any other

11

ACCEPTED MANUSCRIPT

parameter on a secondary y-axis. The color palette used for the hydrograph can be adjusted in the Sidebar menu to either use the same color to symbolize all datasets from a stream station or to use a different color to symbolize each dataset from every stream station.

RI PT

If users select to retrieve gauge height data from a Roanoke USGS stream station, then a bar graph of the maximum gauge height measured during the specified date range compared to the United States National Weather Service (NWS) flood thresholds is displayed (Figure 3). Colors of the bars change from green (no flooding) to yellow (minor flooding), red (moderate flooding), and black (major flooding) depending on the flood stage exceeded. Data labels identify the maximum gauge height as well as the

EP

TE D

M AN U

SC

date/time in which it occurred.

Figure 3: Summary Tab – Bar graphs displaying the current or maximum recorded gauge height (for a specified date range) in comparison to the National Weather Service flood thresholds for the RealTime Flood Stages and Maximum Flood Stages analysis features, respectively.

AC C

2.2.3. Map Tab

The “Map” tab was included to allow users to view summary meteorological and hydrologic data from each specified NOAA and USGS station on an interactive map (Figure 4, Section 2.4.5). Clicking on the map markers displays a label with the total precipitation depth and maximum precipitation intensity during the specified date range for meteorological stations and the maximum stream discharge during the specified date range for stream stations. For stations measuring both precipitation and stream flow, the labels display the total precipitation depth, maximum precipitation intensity, and maximum stream discharge measured during the specified date range. For USGS stations, clicking the station name in the

12

ACCEPTED MANUSCRIPT

label will open a new browser tab to the USGS National Water Information System (NWIS) website for

AC C

EP

TE D

M AN U

SC

RI PT

the respective station.

Figure 4: Map Tab – Interactive Map displays summary meteorological and hydrologic data for all NOAA and USGS stations. 2.2.4. Forecast Tab

The “Forecast” tab was designed to allow users to view the current weather conditions and 10-Day weather forecast from Weather Underground (https://www.wunderground.com/) for any user-specified location (Supplemental Figure S5, Section 2.4.3). Current weather conditions for the specified location are displayed using a Weather Underground weather sticker (a widget that displays a visual representation of the current weather conditions), and clicking on the sticker will open a new browser 13

ACCEPTED MANUSCRIPT

window to the Weather Underground website for the location. The 10-Day forecast for the specified location is displayed in a tabular format and includes the forecasted high and low temperatures, weather condition, precipitation probability, and precipitation depth. Both the weather sticker and the

RI PT

forecast table respond dynamically to location inputs made in an input box located on the Forecast tab. 2.2.5. Interactive Plots Tab

The “Interactive Plots” tab was designed to allow users to investigate and download data from specific time periods within the specified date range (Section 2.4.6). A switch on the tab toggles the

SC

display between an interactive hyetograph and an interactive hydrograph created using the ggplot2 R package (Wickham, 2016). Users may study specific time periods within the hyetograph (Figure 5) by clicking and dragging a box around the desired data. The duration of this “brushed” time period, as well

M AN U

as the total precipitation depth and average precipitation intensity (total precipitation depth/duration) for the period, is summarized below the hyetograph. Below this, the maximum precipitation intensity observed during the period and the date/time(s) when it was recorded are also displayed. Furthermore, a “Brushed Data” table is included of all incremental precipitation depth and intensity data for the displayed meteorological station during the brushed period. Data in the table can be sorted by clicking

AC C

EP

TE D

on the column titles, and a button is included to download the table as a CSV file.

14

AC C

EP

TE D

M AN U

SC

RI PT

ACCEPTED MANUSCRIPT

Figure 5: Interactive Plots Tab – Interactive Hyetograph with brushed data. Like the hyetograph, users may interact with the hydrograph (Supplemental Figure S6) by brushing data for a specific time period. A “Volume Summary” table displays the total discharge volume, baseflow volume, and stormflow runoff volume, calculated via trapezoidal integration, for each stream station for the brushed time period. In addition, all discharge and water quality data from the stream stations for the brushed period is displayed in “Brushed Data” table. Data in the table can be sorted by clicking on the column headers, and a button is included to download the table as a CSV file. Users may also interact with the hydrograph by clicking any spot on the hydrograph to display data from the nearest 15

ACCEPTED MANUSCRIPT

point, including the location and date/time from which the data was recorded as well as the baseflow, discharge, and any downloaded water quality parameters corresponding to that station and time. 2.2.6. Tables Tab

RI PT

While the “Interactive Plots” tab is useful for downloading precipitation or discharge/water quality data for specific time periods, the “Tables” tab was included in the SHARKS app to allow users to download all time series data from the app as one file for further offline use. To allow users to sort the tables to identify the maximum and minimum recorded measurements of all stations for discharge and

SC

water quality, data in the interactive hydrograph “Brushed Data” table was not joined by date/time, resulting in duplicate date/times if measurements were taken at the same time at different stream stations. In contrast, the “Tables” tab contains a “Combined Data Table” in which the time series data

M AN U

for each of the meteorological stations and stream stations are joined by date/time (Supplemental Figure S7). Thus, there are separate columns for each time series dataset, and every data point recorded for a specific time is shown on the same row within the table. An additional benefit of joining all of the time series data streams into one table is that all of the data are correctly aligned, regardless of any missing data points. For meteorological stations, the presented datasets include incremental precipitation and precipitation intensity and for stream stations, the presented datasets include

TE D

baseflow discharge, total discharge, direct runoff depth, direct runoff volume, and the selected water quality parameters. Data in the table can be sorted by clicking on the column headers, and a button is included to download the table as a CSV file.

The NOAA Precipitation Frequency Data Server (PFDS) tables for all specified NOAA and USGS

EP

meteorological stations are also displayed on the “Tables” tab (Supplemental Figure S8) and can be downloaded as CSV files. Currently, the NOAA PFDS server (https://hdsc.nws.noaa.gov/hdsc/pfds/) only

AC C

allows users to retrieve the PFDS tables by latitude/longitude, NOAA station, address, or geocoded address. However, the SHARKS app automatically retrieves the latitude and longitude for any specified meteorological station and fetches the NOAA PFDS table for that location (Section 2.3.1). This automated process not only saves users time, but also eliminates the possibility of transcription errors associated with manually entering station latitude/longitudes into the NOAA server. 2.2.7. Storm Sewer Tab The “Storm Sewer” tab (Figure 6) displays data from nine Roanoke storm sewer flow depth sensors (Section 2.3.3). To access the tab’s content, users must log in with an authorized Google account. Upon 16

ACCEPTED MANUSCRIPT

logging in, an interactive map of the location of each sensor, a stacked hyetograph/hydrograph, and a data table will appear on the tab. The interactive map (Figure 6, Section 2.4.5) displays the spatial location of each selected Roanoke storm sewer sensor, and clicking on the map markers displays a label with the sensor name and the maximum relative depth (Section 2.3.3) measured at that location during

M AN U

SC

RI PT

the specified date range.

TE D

Figure 6: Storm Sewer Tab – Interactive Map displays the spatial location and the maximum relative depth measured during the specified date range for selected Roanoke storm sewer sensors. The stacked hyetograph/hydrograph (Figure 7) plots precipitation and relative depth (Section 2.3.3) time series for the specified date range. Drop down menus above the stacked plot allow users to select

EP

the storm sewer sensor and meteorological station data sources displayed in the plot. The “Storm Sewer Sensor Data” table (Figure 7) displays the sensor stage and relative depth (Section 2.3.3) measurements from the specified date range as well as the date/time and sensor at which they were recorded. Data in

AC C

the table can be sorted by clicking on the column headers, and a button is included to download the table as a CSV file.

17

TE D

M AN U

SC

RI PT

ACCEPTED MANUSCRIPT

EP

Figure 7: Storm Sewer Tab – Stacked Hyetograph/Relative Depth Hydrograph and Storm Sewer Sensor Data table. 2.2.8. Real-Time Flood Stages Tab

AC C

The “Real-Time Flood Stages” tab consists of a set of bar graphs display the most recent gauge height measurements for five Roanoke stream stations compared to their respective NWS flood thresholds (Figure 3, Section 2.4.3). Colors of the bars change from green (no flooding) to yellow (minor flooding), red (moderate flooding), and black (major flooding) depending on the flood stage exceeded. Data labels identify the most recent gauge height measurement as well as the date/time in which it occurred.

18

ACCEPTED MANUSCRIPT

2.3. Data Retrieval & Pre-Processing 2.3.1. Meteorological Data

RI PT

The SHARKS app was designed to retrieve meteorological data from any NOAA ASOS station. The NOAA ASOS network is the flagship automated observing network and provides observations for the NWS, FAA, and Department of Defense. Meteorological data for the user-specified ASOS station is obtained

via

the

Iowa

Environmental

Mesonet

maintained

by

Iowa

State

University

SC

(https://mesonet.agron.iastate.edu/request/download.phtml). Data is retrieved from the Mesonet via a programmatically created URL based on user-input state, time zone, ASOS station/airport FAA ID, and date range. The ASOS network reports precipitation data as a cumulative precipitation depth that

on when the precipitation depth resets.

M AN U

generally resets on the hour. The time step for the ASOS data is typically 5-minutes but varies depending

The SHARKS app was also designed to retrieve precipitation data from any USGS meteorological station. Precipitation data for the user-specified USGS meteorological stations is obtained from USGS NWIS via the readNWISuv() function in the dataRetrieval R package (Hirsch and De Cicco, 2015) based

TE D

on the user-specified date range and time zone. USGS precipitation data is reported as an incremental precipitation depth, so no pre-processing of the data was required. Timesteps for the USGS precipitation data vary by station but are typically either 5 or 15 minutes. To calculate the ARI of storm events, the SHARKS app retrieves the NOAA Atlas 14 point

EP

precipitation frequency estimates table (https://hdsc.nws.noaa.gov/hdsc/pfds/) via a programmatically created URL for each user-specified NOAA ASOS and USGS meteorological station based on latitude and

AC C

longitude of the station. Although the station latitude and longitude are included with the meteorological data for NOAA ASOS stations, this information is not included with the precipitation data for USGS stations. Thus, the readNWISsite() function in the dataRetrieval R package (Hirsch and De Cicco, 2015) was used to retrieve the latitude and longitude for each USGS meteorological station. 2.3.2. Stream Discharge & Water Quality Data The SHARKS app was designed to retrieve gauge height, stream discharge, and water quality data from any USGS stream station. This data is obtained from the USGS NWIS via the readNWISuv() function in the dataRetrieval R package (Hirsch and De Cicco, 2015) based on the user-specified date range and 19

ACCEPTED MANUSCRIPT

time zone. Water quality parameters available for download within the app are Water Temperature (°C), Specific Conductance (μS/cm @ 25°C), Dissolved Oxygen (mg/L), pH, and Turbidity (FNU). Time steps for the USGS gauge height, stream discharge, and water quality data vary by station but are typically either

RI PT

5 or 15 minutes. 2.3.3. Storm Sewer Flow Depth Data

In August 2017, the City of Roanoke installed HOBO U20 water-level loggers, built by Onset Corp. (http://www.onsetcomp.com/), to monitor storm sewer flow depth in nine critical downtown pipe

SC

network locations (Figure 8) with the goal of identifying choke points within the network (Aguilar et al., Under Review). The sensors record flow depth data at 5-minute increments, and data is manually retrieved from each sensor by a City of Roanoke employee. Because the level loggers are installed above

M AN U

the baseflow flow depth, negative flow depths are recorded when the water level is below the sensor and the pressure transducers measure the atmospheric pressure instead of the water pressure. Thus, negative flow depth datapoints are removed from the datasets because the measurements are inaccurate during these conditions. To prevent “floating” data when graphing, flow depth values immediately before/after removed data points are set to 0.001 feet below the sensor position to indicate that the water level starts/ends below the sensor position. Since the storm sewer slope and

TE D

roughness at each of the nine locations are subject to considerable uncertainty, the flow depth data is divided by the full-flow depth at the respective location to calculate a “relative depth” metric to identify choke points (Aguilar et al., Under Review). Then, the relative depth time series for each of the nine locations are uploaded to Google Sheets. Storing the data in a Google Sheet has two primary benefits.

EP

First, it enables the SHARKS app to retrieve the data without relying on any local files. Second, it allows for control of user access to the data, so access can be limited to authorized accounts. To access the

AC C

storm sewer data, the SHARKS app requires users to login with a Google Account that has access to the Google Sheets which store the data. The googleAuthR R package (Edmondson, 2018) handles Google authentication and retrieves the Google access token. Then, the SHARKS app retrieves the relative depth data from the Google Sheets using the Google Sheets application program interface (API) (https://developers.google.com/sheets/api/).

20

Figure 8: Roanoke Storm Sewer Sensor Locations

SC

RI PT

ACCEPTED MANUSCRIPT

M AN U

2.3.4. Current Weather Conditions & 10-Day Weather Forecast

The Google Geocoding API (https://developers.google.com/maps/documentation/geocoding/start) is used to retrieve the latitude and longitude of the user-specified location for which they wish to retrieve forecast data. Then, the current weather conditions for the location are retrieved from Weather Underground as a Weather Sticker (https://www.wunderground.com/stickers/), and the 10-Day weather forecast for the location is obtained from Weather Underground via the forecast10day()

AC C

EP

TE D

function in the rwunderground R package (Shum, 2018).

21

ACCEPTED MANUSCRIPT

2.4. Analysis Features The SHARKS app includes both automated analysis features, which do not require any user inputs, and interactive analysis features, which require user input from the user interface. Automated analysis

RI PT

features include: performing hydrograph separations, calculating direct runoff volume/depth and runoff volume coefficients, retrieving the current weather conditions and 10-day forecast, and retrieving the current flood stages. Due to the reactive Shiny framework, the automated analyses are re-executed and their outputs are immediately updated upon changes to the selections made with the SHARKS control widgets. Interactive analysis features include calculating storm ARI, interactive maps, and interactive

SC

plots. The interactive analyses require inputs from users such as clicking on a station in the interactive maps or selecting a date/time range on the interactive plots. This information is then passed to the

M AN U

server to query data from the master data frame and perform any necessary computations before updating the user interface outputs. All of the SHARKS analysis features, with the exception of the forecast and real-time flood stages features, pull data from the master data frame. Thus, the SHARKS implementation is generic such that any additional exploratory data analysis features can easily be incorporated by adding code to query data from the master data frame, perform any necessary computations, and display outputs in the user interface.

TE D

2.4.1. Hydrograph Baseflow Separation

The SHARKS app performs hydrograph separations automatically for each user-specified USGS stream flow station. Partitioning a hydrograph into the stormflow and baseflow components can provide valuable insight into a watershed’s response to precipitation or snowmelt events, particularly

EP

the proportion of added water that becomes runoff (runoff volume coefficient) and the timing of the hydrologic response. Results of the hydrograph separations are used in the calculations for the direct

AC C

runoff volume/depth and runoff volume coefficients analysis features detailed in Section 2.4.2. Hydrograph separations are performed using the BaseflowSeparation() function in the EcoHydRology R package (Fuka et al., 2014). This function uses the recursive digital filter from Nathan and McMahon (1990) to partition total streamflow into baseflow and stormflow components based on inputs of the flow time series, number of passes, and filter parameter alpha. The number of passes controls the degree of smoothing applied to the hydrograph separation, and a value of three was used similar to Nathan and McMahon (1990), who concluded that an alpha value in the range of 0.90-0.95 produced the most acceptable hydrograph separations. To determine the optimal alpha value for

22

ACCEPTED MANUSCRIPT

hydrograph separations for Roanoke’s Lick Run and Roanoke River USGS stream stations, hydrograph separations performed using the conductivity mass-balance (CMB) method were compared to hydrograph separations performed using the digital filter method with alpha values ranging from 0.900-

RI PT

0.995. The CMB hydrograph separation method has been used to calibrate other hydrograph separation methods and is detailed in Nathan and McMahon (1990), Pilgrim et al. (1979), Stewart et al. (2007), and Bhaskar and Welty (2015). During precipitation events, stream flow conductance decreases as flow is diluted. The CMB method uses this effect to partition total stream flow into baseflow and stormflow

SC

components based on measured stream discharge, measured stream specific conductance, estimated stormflow specific conductance, and estimated baseflow specific conductance.

M AN U

To compare the CMB and digital filter hydrograph separation methods, stream discharge and specific conductance data was downloaded from the Lick Run and Roanoke River stations for a library of precipitation events. For the Lick Run station, 11 events from 2017 and 2018 were chosen for analysis. Specific conductivity data was unavailable from 2012-2017 for the Roanoke River station, so 10 events from 2011 and 2018 were chosen for analysis. For the CMB method, stormflow specific conductance was estimated as the minimum measured stream specific conductance observed during the event, and

TE D

baseflow specific conductance was estimated as the average stream specific conductance at the stations during extreme low-flow periods when it can be assumed that stream flow is entirely baseflow (Stewart et al., 2007).

Hydrograph separations were performed for each of the 11 Lick Run and 10 Roanoke River events

EP

using the CMB method and the digital filter method with alpha values ranging from 0.900 to 0.995. Then, the Spearman’s rho, Percent Bias (PBIAS), and Nash-Sutcliffe Efficiency (NSE) model-fit statistics

AC C

were calculated with the CMB method baseflow estimates as the target values. Spearman’s rho correlations were considered significant for p-values ≤0.05, PBIAS was considered acceptable for values between -25% and 25%, and NSE values ≥ 0.0 were considered acceptable. Overall, the optimal digital filter alpha values for the Lick Run and Roanoke River stations were selected as the alpha values which resulted in the greatest number of events with significant Spearman’s Rho p-values and acceptable PBIAS and NSE values. For the Lick Run station, the optimal alpha value was 0.9875 with nine events with p-values ≤ 0.05, ten events with acceptable PBIAS, and three events with acceptable NSE. Likewise, for the Roanoke River station, the optimal value was 0.9250 with ten events with p-values ≤0.05, three events with acceptable PBIAS, and three events with acceptable NSE. For any other user-specified USGS 23

ACCEPTED MANUSCRIPT

stream station, the Nathan and McMahon (1990) recommended alpha value of 0.925 will be used for hydrograph separations. 2.4.2. Calculate Direct Runoff Volume, Direct Runoff Depth, and Runoff Volume Coefficients

RI PT

The SHARKS app automatically calculates the direct runoff volume, direct runoff depth, and the proportion of precipitation leaving a watershed (runoff volume coefficient) for each user-specified USGS stream flow station, and the results from these analyses are displayed in data tables on the SHARKS Summary Tab (Section 2.2.2). Increases in impervious area correspond to increases in runoff volume,

SC

peak flow rates, flooding frequency, and runoff volume coefficients (Bonta et al., 2003; Goldshleger et al., 2009). Furthermore, as percent impervious area increases from 20-40%, the range of percent impervious area corresponding to the transformation from pre-urban to urban land use, there is a sharp

M AN U

increase in runoff volume coefficient (Goldshleger et al., 2009). Therefore, since this land use transformation is prevalent in areas of high urbanization (Goldshleger et al., 2009), analysis of runoff volume coefficients provides a useful metric for evaluating the threat of storm runoff-flooding. The direct runoff volume/depth and runoff volume coefficient analysis features were included in SHARKS to aid users in understanding watershed responses to precipitation and to evaluate the threat of flooding from storm runoff. The runoff volume coefficient is a particularly important hydrologic

TE D

parameter, as the City will be using it for long-term evaluation of trends in hydrologic function as investment in stormwater control measures to restore pre-development hydrology continues to increase over time. This parameter also allows for comparison across watersheds as a method of

same region.

EP

comparing one of the City’s more highly urbanized watersheds to a less developed watershed in the

The volume of direct runoff is calculated for each specified stream station using trapezoidal

AC C

integration of stormflow with respect to time. The direct runoff volume is then normalized by the userspecified watershed area to calculate a direct runoff depth for each stream station. Runoff volume coefficients represent the proportion of precipitation leaving a watershed and are calculated for each stream station and meteorological station combination by dividing direct runoff depth by total precipitation depth. 2.4.3. Forecast & Real-Time Flood Stages The SHARKS app automatically retrieves the current weather conditions and 10-day weather forecast for a user-specified location (Section 2.3.4) and displays this information on the “Forecast” tab 24

ACCEPTED MANUSCRIPT

(Section 2.2.4). In addition, SHARKS automatically retrieves the most recent gauge height measurements from USGS NWIS for five Roanoke stream stations and displays this information relative to the NWS flood thresholds on the “Real-Time Flood Stages” tab (Section 2.2.8). Currently, there is no way to programmatically retrieve flood thresholds from the NWS, so only the five Roanoke stream stations

RI PT

have been included in SHARKS. Combined, the SHARKS forecast and flood stage analysis features provide information to enable users to make informed decisions regarding how an area may respond to future precipitation and weather conditions.

Leading up to and during flood events, City of Roanoke stormwater and emergency management

SC

staff rely on hydrologic measurements and forecasts to make decisions about road-closures, deployment of flood proofing measures, and evacuations. In particular, staff monitor how close rivers

M AN U

are to NWS flood stages, current and forecasted weather conditions, and field observations as the basic information used to make rapid and sometimes life-saving decisions. The real-time flood stages tab condenses river information from the salient USGS stations, with NWS flood thresholds, so that local officials have the information needed in a single “dashboard” type interface. In addition, the SHARKS forecast analysis features informs local officials if the area is likely to receive more precipitation, which could exacerbate flooding.

TE D

2.4.4. Calculate Average Annual Recurrence Interval (ARI) Users can calculate the ARI for storm events at each specified meteorological station by clicking a switch on the SHARKS sidebar menu (Section 2.2.1). As an interactive analysis feature, the ARI computations will only be performed if users set the switch to calculate the ARIs. A data table of all ARIs

EP

calculated for the specified date range (Supplemental Figure S3) is displayed on the SHARKS Summary tab. This analysis feature was included in SHARKS to provide a useful metric for users to study the spatial

AC C

distribution of event precipitation among separate stations as well as to compare separate storm events. In addition, expressing the rarity of precipitation in terms of an ARI provides an objective criterion that can be easily conveyed to decision makers and the public. City of Roanoke staff have used the SHARKS ARI analysis feature to study the variability of local precipitation and distribute the results to news stations, emergency managers and the public in response to flooding that occurred in the southwest part of the City (Section 3.2). Furthermore, quantifying ARIs allows users to compare actual events to design storms to determine if infrastructure is functioning according to current design standards, or if it needs to be upsized.

25

ACCEPTED MANUSCRIPT

To calculate storm ARIs, a sliding window analysis is performed to first calculate the measured precipitation depth at the station for every possible time interval during the specified date range. The sliding window analysis divides the date range into a set of intervals, or windows, with lengths t, 2t, 3t, …, nt where t is the time increment between precipitation data points and n is the number of

RI PT

precipitation data points in the specified date range (Figure 9). For each window length, a window is created starting at every precipitation data point (i, i+1, i+2, …) to obtain every possible window of every possible length within the specified date range (Figure 9). Then, the total precipitation depth for each window is calculated as the sum of the incremental precipitation depths within the window. Next, the

SC

sliding window dataset is subset to only include windows with lengths equal to the event durations in NOAA PFDS tables. Although sliding windows could have been created only for lengths equal to the event durations in the PFDS tables, creating windows of every possible length does not drastically

M AN U

increase program run time, and it facilitates the ability to implement interpolation between event durations in addition to between ARIs. Finally, the ARI for each sliding window is calculated by using the window’s precipitation depth to interpolate the ARI between the precipitation depths from the station’s NOAA PFDS table for the event duration corresponding to the window length. If the precipitation depth for a window is less than the depth of the 1-year storm, then the ARI for the window is set to not

TE D

available (NA).

Window Length

Instantaneous Precipitation Depth i i+1 i+2 i+3

EP

t

AC C

2t 3t

nt Figure 9: Sliding window example

2.4.5. Interactive Maps Interactive maps are included on the SHARKS Map (Figure 4) and Storm Sewer (Figure 6) tabs. These maps are automatically populated with markers indicating the spatial location of each selected

26

ACCEPTED MANUSCRIPT

station/sensor. Users can zoom and pan the map or click on the markers to view summary data (e.g. precipitation depth, storm ARI, maximum stream discharge, storm sewer flow depth, etc.) for that location. Interactive mapping in SHARKS is handled by the Leaflet R package (Cheng et al., 2018). Each interactive map in the SHARKS user interface is supported by a reactive data frame that contains

RI PT

information about the station/sensor’s name, latitude, longitude, and map marker attributes. The data frame also includes the programmatically built label for each station/sensor containing the summary data queried from the master data frame.

Interactive mapping is a critical component of any spatial web application and provides spatial

SC

context to the data for users who may not be familiar with the locations of the stations/sensors based on their names alone. The interactive mapping analysis features were included in SHARKS to allow users

M AN U

to identify spatial patterns across a network of stations. In addition, the interactive maps were designed to provide a brief summary of the data for users who may not have backgrounds in hydrology and could be intimidated by the other SHARKS analysis features. In Roanoke, the SHARKS interactive maps have been used by City staff to study the variability of local rainfall events (Section 3.2). 2.4.6. Interactive Plots

The SHARKS Interactive Plots tab contains two interactive analysis features: an interactive

TE D

hyetograph (Figure 5) and an interactive hydrograph (Supplemental Figure S6). These plots are created using the ggplot2 R package (Wickham, 2016) and allow users to view individual data points — by clicking on the data point on the plot — or summarize the data from a specific time period — by clicking and dragging a selection box around data on the plot, termed “brushing”. The Shiny framework provides

EP

built-in support for the click and brush interactions. When users click or brush data in the interactive plots, this data is subset from the master data frame and stored into a separate data frame for analysis.

AC C

During testing of SHARKS, it was decided that clicking on individual incremental precipitation data points on the hyetograph was difficult and therefore viewing individual incremental precipitation data points was not a useful feature. Thus, the SHARKS interactive hyetograph only supports brushing whereas the interactive hydrograph supports both brushing and clicking. Using the interactive hyetograph, users can view the duration of a brushed time period as well as the total precipitation depth, average precipitation intensity (total precipitation depth/duration), and maximum precipitation intensity observed during the period. In addition, SHARKS displays all the incremental precipitation depth and intensity data from the brushed period in a sortable and

27

ACCEPTED MANUSCRIPT

downloadable data table. These analysis features were included in SHARKS so users could identify how much precipitation was received at a meteorological station for a precise time period, unlike other platforms which may only report precipitation in certain increments (e.g. 1, 6, 12, or 24-hours).

RI PT

The interactive hydrograph can be used to determine the total discharge volume, baseflow volume, and stormflow runoff volume, calculated via trapezoidal integration, at a stream station for a brushed time period. Users may also click on data points in the interactive hydrograph to view the baseflow, discharge, and water quality parameters corresponding to that station and time. These analysis features

SC

were included in SHARKS to allow users to investigate the temporal variation in the hydrologic datasets. 3. Case Study – City of Roanoke

The purpose of this section is to demonstrate the immediate added value of the SHARKS

M AN U

application to the City of Roanoke and to describe some initial lessons-learned that were made possible by the availability of this app. Although the initial use of the SHARKS app has been site specific to the City of Roanoke, the authors reiterate that any USGS or NOAA ASOS site can be used in the application, and the underlying R code is publicly available, to be tailored to a different site as needed (Brendel et al., 2018). The following sub-sections therefore provide a conceptual description of the types of rapid

TE D

analysis that this app provide, that can be applied to other jurisdictions or watersheds. 3.1. Variability of Local Rainfall Events

An ancillary observation that was made during the analysis of storm events was that the intensity and depth of rainfall varies dramatically across the City’s service area. This had been observed

EP

anecdotally for many years in this region, though the installation of the rain gage network, and viewing capabilities of the SHARKS app allowed for empirical evidence of this phenomenon. For example, during

AC C

a storm event that occurred on 5/27/18, the total observed rainfall depth at two stations varied by an order of magnitude across a distance of approximately two miles. This variability was further demonstrated by estimating the ARI at each station using the SHARKS app for another storm event that occurred on 5/17/18 and plotting this variability across the City’s service area. To do this, a grid of ARIs was created by performing inverse distance weighting (IDW) interpolation between the maximum ARI during the storm at each location as calculated from SHARKS, and the results are shown in Figure 10. The maximum ARI recorded at each station during this storm ranged from 1.7 years in the northeast to 35.5 years in the southwest part of the City, and it is clear that this event was centered in a highly localized area. 28

ACCEPTED MANUSCRIPT

The short-term benefit of the app for this event, was that City staff were able to characterize this event within hours of the end of the rainfall when the memory of the event was still fresh, tying anecdotal observations to gage measurements. These data were then distributed to news stations, emergency managers, and the public in response to flooding that occurred in the southwest part of the

RI PT

City. The long-term benefit of these data is that the high noted variability in rainfall across the City may lead to a more localized rainfall design paradigm for stormwater structures than the regional approach that has historically been taken based on NOAA PFDS methodology. Continued monitoring of the spatial distribution of rainfall using the SHARKs app will allow the City to determine if the high variability

SC

observed on 5/17/18 was an outlier, or if order-of-magnitude variability across the City’s service area is

AC C

EP

TE D

M AN U

a typical pattern for rainfall in this area.

29

EP

TE D

M AN U

SC

RI PT

ACCEPTED MANUSCRIPT

AC C

Figure 10: Spatial variation of event annual recurrence interval (ARI) for May 17th, 2018 event

30

ACCEPTED MANUSCRIPT

3.2. Storm Sewer Response to Precipitation One issue of considerable interest in the City, was the recurring flooding that occurs in the City’s Central Business District (CBD) during brief, intense rainfall (see e.g. (Chittum, 2017)). The cause of this flooding was unclear, as the stream draining the watershed was buried in large tunnels around the turn

RI PT

of the 20th century, and it was therefore not possible to visually observe the stream during storm events. To address this issue, the nine storm sewer flow depth sensors previously described were installed at critical locations throughout the tunnel system (Figure 8), and the data from these sensors was integrated with precipitation data in the SHARKS app. The primary interest was to evaluate the hydraulic

SC

grade line at these nine locations with respect to the tunnel soffit and manhole rims during a range of different storm event depths, durations, intensities, and corresponding annual recurrence intervals

M AN U

(ARIs). Pipes that surcharge under ARI events that are smaller than a typical 10-year ARI design storm could be targeted for hydraulic improvements, such as replacement or sediment clearing. The preliminary results of this work show that several of the tunnels are surcharging under 5 – 10 yr. ARI rainfall events. The cause of this surcharging, as revealed by the SHARKS app, appears to be undersized local arterial drainage conveyances and sediment build-up, but not downstream capacity. However, it has been observed that during larger storm events, runoff from an adjacent watershed

TE D

creates a backwater effect that could prevent the tunnel system from efficient draining. Furthermore, one of the tunnels appears to be flowing in two directions, depending on the location of rainfall with respect to the contributing drainage areas. The recommended engineering improvement is highly contingent on understanding flow direction through this tunnel, and a new velocity sensor will be added

EP

to the SHARKS app to help better understand the complex hydraulics in this reach. A second reason for the integration of tunnel hydraulics data into the SHARKS app was to develop a

AC C

relationship between rainfall characteristics and the relative fullness of the tunnels under the CBD to serve as an early flood warning system for owners and tenants. This would allow for City officials to alert stakeholders of a potential flash flood situation based on NOAA’s quantitative precipitation estimate, so that temporary flood-proofing measures could be put in place. The SHARKS app has provided the necessary data analysis tools to develop these simple rainfall-runoff relationships, though the preliminary results have only shown a tentative correlation, and more storm event data is needed.

31

ACCEPTED MANUSCRIPT

3.3. Stream Specific Conductance During Snow Events During the March 12th, 2018 snow event, two large spikes in specific conductance were observed at the Lick Run stream station (Figure 11). The initial peak in specific conductance, observed around

RI PT

5:30am, corresponded to only a minor increase in stream discharge. Upon analysis of the SHARKS app output, the City of Roanoke concluded that the initial spike in specific conductance was likely the result of application of de-icer on an adjacent highway prior to the peak commute time and that the second specific conductance peak was due to the flushing of salts from the watershed during snow melt. Measured precipitation at the Lick Run station was only 0.06 inches during the initial specific

SC

conductance peak and 0.25 inches during the second specific conductance peak. In the long-term, the ability of snow-removal operators to track in-stream specific conductance at near-real time, while

M AN U

viewing accumulated and predicted snowfall amounts through the SHARKS app will provide valuable

AC C

EP

TE D

information for decisions about additional application of de-icing agents.

Figure 11: Stream discharge and specific conductance during March 12th, 2018 snow event

32

ACCEPTED MANUSCRIPT

4. Conclusions Overall, the SHARKS app expands the capabilities of existing web-based visualization platforms beyond exclusively data retrieval and visualization through the inclusion of a suite of data analysis tools.

RI PT

The combination of data retrieval, visualization, and analysis tools allows users to quickly compare and export data from any USGS stream station and any NOAA ASOS or USGS meteorological station. In addition, the app expedites the data analysis process with dynamically linked data visualization and analysis tools which allow users to refine the datasets and time periods analyzed based on the analysis results, before exporting the data for further use. Furthermore, the app introduces a framework for

SC

integrating time series data from local sources with data from government and commercial sources, while providing options for restricting user access to the data, through the use of Google Sheets. No

M AN U

specialized coding, software, or expertise are needed to utilize the SHARKS app, and the web interface ensures that the app can be accessed from a web browser without installing any additional software. As implemented for the City of Roanoke, the SHARKS app has proven its usefulness as a tool for investigating hydrologic processes in urban watersheds. However, the SHARKS app is easily adapted to include preloaded monitoring stations for implementation in other locations and will be useful for other cities and organizations that need to retrieve and analyze hydrologic data from a variety of sources. A

TE D

streamlined version of the SHARKS app, tailored towards use by the general public, is currently under development and will provide a subset of the app’s analysis features in a simplified user interface. Abbreviations

Acknowledgements

EP

ARI, annual recurrence interval; CMB, conductivity mass balance

AC C

This work was supported by the City of Roanoke, Virginia and the Virginia Tech Via Department of Civil & Environmental Engineering. Special thanks to David Woodson and the City of Roanoke Stormwater Division for their valuable insights and helpful feedback during the development and testing of this app. Competing Interests

Declarations of interest: none

33

ACCEPTED MANUSCRIPT

References 1. Aguilar, M.F., Dymond, R.L., Cooper, D.R., Under Review. Hydrologic diagnosis of a buried stream under a central business district. Journal of Water Resources Planning and Management.

RI PT

2. Bartos, M., Wong, B., Kerkez, B., 2018. Open storm: a complete framework for sensing and control of urban watersheds. Environmental Science: Water Research & Technology 4(3) 346358. 3. Bhaskar, A.S., Welty, C., 2015. Analysis of subsurface storage and streamflow generation in urban watersheds. Water Resources Research 51(3) 1493-1513.

SC

4. Biagioni, E.S., Bridges, K., 2002. The application of remote sensor technology to assist the recovery of rare and endangered species. The International Journal of High Performance Computing Applications 16(3) 315-324.

M AN U

5. Bonta, J., Shuster, W., Warnemuende, E., Thurston, H., Smith, D., Goss, M., Cabezas, H., 2003. Quantification of urbanization in experimental watersheds, Conference on Research in the Watersheds. 6. [dataset] Brendel, C., Dymond, R.L., Aguilar, M.F., 2018. SHARKS, v1.0.0 ed. Zenodo. http://doi.org/10.5281/zenodo.1977491 7. Chang, W., Cheng, J., Allaire, J., Xie, Y., McPherson, J., 2018. shiny: Web Application Framework for R. R package version 1.1.0.

TE D

8. Cheng, J., Karambelkar, B., Xie, Y., Wickham, H., Russell, K., Johnson, K., 2018. Leaflet: create interactive web maps with the JavaScript “Leaflet” Library. R package version 2.0.2.

EP

9. Chittum, M., 2017. After 4 floods swamp downtown in a year, Roanoke officials look for answers, The Roanoke Times: https://www.roanoke.com/news/after-floods-swamp-downtownin-a-year-roanoke-officials-look/article_ac03c944-6dfc-5c9c-9cdf-2edb79aef3f1.html (accessed 24 July 2018). 10. Edmondson, M., 2018. googleAuthr: Authenticate and Create Google APIs, 0.6.3 ed.

AC C

11. Fuka, D., Walter, M., Archibald, J., Steenhuis, T., Easton, Z., 2014. EcoHydRology: A community modeling foundation for Eco-Hydrology, 0.4.12 ed. 12. Goldshleger, N., Shoshany, M., Karnibad, L., Arbel, S., Getker, M., 2009. Generalising relationships between runoff–rainfall coefficients and impervious areas: an integration of data from case studies in Israel with data sets from Australia and the USA. Urban Water Journal 6(3) 201-208. 13. Hefeeda, M., Bagheri, M., 2009. Forest Fire Modeling and Early Detection using Wireless Sensor Networks. Ad Hoc & Sensor Wireless Networks 7(3-4) 169-224. 14. Hirsch, R.M., De Cicco, L.A., 2015. User guide to Exploration and Graphics for RivEr Trends (EGRET) and dataRetrieval: R packages for hydrologic data. US Geological Survey. 34

ACCEPTED MANUSCRIPT

15. Horsburgh, J.S., Jones, A.S., Ramírez, M., Caraballo, J., 2016. Time Series Analyst: Interactive Online Visualization of Standards Based Environmental Time Series Data, 8th International Congress on Environmental Modelling and Software.

RI PT

16. Horsburgh, J.S., Tarboton, D.G., Piasecki, M., Maidment, D.R., Zaslavsky, I., Valentine, D., Whitenack, T., 2009. An integrated system for publishing environmental observations data. Environmental Modelling & Software 24(8) 879-888. 17. Keefer, D.K., Wilson, R.C., Mark, R.K., Brabb, E.E., Brown, W.M., Ellen, S.D., Harp, E.L., Wieczorek, G.F., Alger, C.S., Zatkin, R.S., 1987. Real-time landslide warning during heavy rainfall. Science 238(4829) 921-925.

SC

18. Kerkez, B., Gruden, C., Lewis, M., Montestruque, L., Quigley, M., Wong, B., Bedig, A., Kertesz, R., Braun, T., Cadwalader, O., 2016. Smarter stormwater systems. Environ. Sci. Technol. 50(14) 7267-7273.

M AN U

19. Kim, Y., Evans, R.G., Iversen, W.M., 2008. Remote sensing and control of an irrigation system using a distributed wireless sensor network. IEEE transactions on instrumentation and measurement 57(7) 1379-1387. 20. Lefkowitz, J., Sarmanian, A., Quigley, M., 2016. Continuous Monitoring and Adaptive Control— The Internet of Things Transforms Stormwater Management. JN Engl. Water Environ. Assoc 50 44-51.

TE D

21. Mainwaring, A., Culler, D., Polastre, J., Szewczyk, R., Anderson, J., 2002. Wireless sensor networks for habitat monitoring, Proceedings of the 1st ACM international workshop on Wireless sensor networks and applications. Acm, pp. 88-97. 22. Mason, S.J., Cleveland, S.B., Llovet, P., Izurieta, C., Poole, G.C., 2014. A centralized tool for managing, archiving, and serving point-in-time data in ecological research laboratories. Environmental Modelling & Software 51 59-69.

EP

23. Minnesota Pollution Control Agency, 2015. Precipitation, runoff and runoff ratio maps: https://www.pca.state.mn.us/sites/default/files/wq-cm1-11.pdf (accessed 24 July 2018).

AC C

24. Mullapudi, A., Wong, B.P., Kerkez, B., 2017. Emerging investigators series: building a theory for smart stormwater systems. Environmental Science: Water Research & Technology 3(1) 66-77. 25. Muschalla, D., Vallet, B., Anctil, F., Lessard, P., Pelletier, G., Vanrolleghem, P.A., 2014. Ecohydraulic-driven real-time control of stormwater basins. Journal of Hydrology 511 82-91. 26. Nathan, R., McMahon, T., 1990. Evaluation of automated techniques for base flow and recession analyses. Water Resources Research 26(7) 1465-1473. 27. Padhy, P., Martinez, K., Riddoch, A., Ong, H., Hart, J.K., 2005. Glacial environment monitoring using sensor networks, Real-World Sensor Networks.

35

ACCEPTED MANUSCRIPT

28. Pilgrim, D.H., Huff, D.D., Steele, T.D., 1979. Use of specific conductance and contact time relations for separating flow components in storm runoff. Water Resources Research 15(2) 329339.

RI PT

29. RStudio, 2015. Scaling and Performance Tuning with shinyapps.io: https://shiny.rstudio.com/articles/scaling-and-tuning.html (accessed 23 December 2018). 30. Shum, A., 2018. rwunderground: R Interface to Weather Underground API, 0.1.8 ed.

31. Smith, J.P., Hunter, T.S., Clites, A.H., Stow, C.A., Slawecki, T., Muhr, G.C., Gronewold, A.D., 2016. An expandable web-based platform for visually analyzing basin-scale hydro-climate time series data. Environmental Modelling & Software 78 97-105.

SC

32. Soliman, H., Sudan, K., Mishra, A., 2010. A smart forest-fire early detection sensory system: Another approach of utilizing wireless sensor and neural networks, Sensors, 2010 IEEE. IEEE, pp. 1900-1904.

M AN U

33. Steere, D.C., Baptista, A., McNamee, D., Pu, C., Walpole, J., 2000. Research challenges in environmental observation and forecasting systems, Proceedings of the 6th annual international conference on Mobile computing and networking. ACM, pp. 292-299. 34. Stewart, M., Cimino, J., Ross, M., 2007. Calibration of base flow separation methods with streamflow conductivity. Groundwater 45(1) 17-27.

TE D

35. Vitolo, C., Elkhatib, Y., Reusser, D., Macleod, C.J., Buytaert, W., 2015. Web technologies for environmental Big Data. Environmental Modelling & Software 63 185-198.

AC C

EP

36. Wickham, H., 2016. ggplot2: elegant graphics for data analysis. Springer.

36

ACCEPTED MANUSCRIPT

M AN U

SC

RI PT

Supplementary Material

TE D

Supplemental Figure S1: Summary Tab – Data Summary Table providing an overview of precipitation and discharge data for selected meteorological and stream stations.

AC C

EP

Supplemental Figure S2: Summary Tab – Runoff Volume Coefficient Summary Table presenting runoff volume coefficients for each stream station and meteorological station combination.

37

SC

RI PT

ACCEPTED MANUSCRIPT

AC C

EP

TE D

M AN U

Supplemental Figure S3: Summary Tab – Average Annual Recurrence Interval (ARI) Summary Table displaying the interpolated ARI for meteorological stations.

Supplemental Figure S4: Summary Tab – Stacked Hyetograph/Hydrograph displaying meteorological and hydrologic data for specified stations.

38

AC C

EP

TE D

M AN U

SC

RI PT

ACCEPTED MANUSCRIPT

Supplemental Figure S5: Forecast Tab – Weather Underground weather sticker widget displaying current weather conditions and data table displaying the 10-Day weather forecast.

39

EP

TE D

M AN U

SC

RI PT

ACCEPTED MANUSCRIPT

AC C

Supplemental Figure S6: Interactive Plots Tab – Interactive Hydrograph with brushed data and volume summary.

40

SC

RI PT

ACCEPTED MANUSCRIPT

AC C

EP

TE D

M AN U

Supplemental Figure S7: Tables Tab – Combined Data table presents time series data for all meteorological and stream stations joined by date/time.

Supplemental Figure S8: Tables Tab – PFDS tables can be displayed for all meteorological stations.

41