Assessing Quality of Surgical Real-World Data from an Automated Electronic Health Record Pipeline

ORIGINAL SCIENTIFIC ARTICLE

Assessing Quality of Surgical Real-World Data from an Automated Electronic Health Record Pipeline Kristin M Corey, BA, Joshua Helmkamp, BS, Morgan Simons, BS, Lesley Curtis, PhD, Keith Marsolo, PhD, Suresh Balu, MBA, MS, Michael Gao, BS, Marshall Nichols, MS, Joshua Watson, MD, Leila Mureebe, MD, MPH, FACS, Allan D Kirk, MD, PhD, FACS, Mark Sendak, MD, MPP Significant analysis errors can be caused by nonvalidated data quality of electronic health records data. To determine surgical data fitness, a framework of foundational and study-specific data analyses was adapted and assessed using conformance, completeness, and plausibility analyses. STUDY DESIGN: Electronic health records-derived data from a cohort of 241,695 patients undergoing 412,182 procedures from October 1, 2014 to August 31, 2018 at 3 hospital sites was evaluated. Data quality analyses tested CPT codes, medication administrations, vital signs, provider notes, labs, orders, diagnosis codes, medication lists, and encounters. RESULTS: Foundational checks showed that all encounters had procedures within the inclusion period, all admission dates occurred before discharge dates, and race was missing for 1% of patients. All procedures had associated CPT codes, 69% had recorded blood pressure, pulse, temperature, respiration rate, and oxygen saturation. After curation, all medication matched RxNorm medication naming standards, 84% of procedures had current outpatient medication lists, and 15% of procedures had missing procedure notes. Study-specific checks temporally validated CPT codes, intraoperative medication doses were in conventional units, and of the 13,500 patients who received blood pressure medication intraoperatively, 93% had a systolic blood pressure >140 mmHg. All procedure notes were completed within less than 30 days of the procedure and 93% of patients after total knee arthroplasty had postoperative physical therapy notes. All patients with postoperative troponin-T lab values
0.10 ng/mL had more than 1 ECG with relevant diagnoses. Postoperative opioid prescription decreased by 8.8% and nonopioid use increased by 8.8%. CONCLUSIONS: High levels of conformance, completeness, and clinical plausability demonstrate higher quality of real-world data fitness and low levels demonstrate less-fit-for-use data. (J Am Coll Surg 2020; -:1e11.
2020 Published by Elsevier Inc. on behalf of the American College of Surgeons.)

BACKGROUND:

to transform healthcare. RWD, as described by the US FDA, includes EHR-derived data, medical claims, and billing data, as well as registry data and patientgenerated data. In essence, RWD are any gathered data

Data from electronic health records (EHR) are used to support studies to improve the quality and efficiency of healthcare delivery. Curated within a single site or multiple sites, these real-world data (RWD) have the potential Disclosure Information: Nothing to disclose. Support: This study was funded in part by a Duke Institute for Health Innovation (https://dihi.org/) pilot grant. K Corey, J Helmkamp, and M Simons were partially supported by the Duke Institute for Health Innovation Clinical Research and Innovation Scholarship. No funders had a role in the decision to publish. Drs Kirk and Sendak contributed equally to this work.

From the Duke University School of Medicine (Corey, Helmkamp, Simons, Balu), Duke Clinical Research Institute (Curtis, Marsolo), Department of Surgery (Watson, Mureebe, Kirk), Duke Institute for Health Innovation (Corey, Helmkamp, Simons, Balu, Gao, Nichols, Sendak), and Department of Population Health Sciences (Curtis), Duke University, Durham, NC. Correspondence address: Kristin M Corey, BA, Duke University School of Medicine, 300 Erwin Rd, Durham, NC 27707. email: corey006@duke. com; [email protected]

Received October 2, 2019; Revised December 19, 2019; Accepted December 19, 2019.

ª 2020 Published by Elsevier Inc. on behalf of the American College of Surgeons.

1

https://doi.org/10.1016/j.jamcollsurg.2019.12.005 ISSN 1072-7515/20

2

Corey et al

J Am Coll Surg

Assessing Quality of Surgical Real-World Data

that have the ability to inform health status.1,2 Distributed research networks, such as the Patient-Centered Outcomes Research Network (PCORnet), American College of Surgeon NSQIP, and the Society of Thoracic Surgeons National Registries, are developing real-world evidence to answer key questions on health services, quality and safety, and fundamental research.3-6 Despite the use of RWD, standard data quality-assurance processes are not uniformly adopted, and the rigor of many RWD projects is questionable. Statistical editors have recently voiced their concerns that many submissions do not meet the threshold for peer review, given flaws in the analytic approach or data quality.7 Premier journals have recently published statistical considerations for research conducted on pooled RWD8 and a series of data use guidelines for the 11 most widely used datasets in surgical research.9-18 Technologies enabling automated sourcing of EHR data for research are emerging, but continue to face challenges.19-23 Although EHRs are ubiquitous across hospitals in the US24 and can be a cost-effective way of obtaining RWD, considerable investments are required to ensure data quality.25,26 The PCORnet confronted these challenges by engaging experts to create a unified data quality framework.27 This framework has been applied to pooled EHR data from PCORnet sites,28 but has yet to be adapted and validated at a specific site for surgical research. To our knowledge, no other data quality framework has been validated to support surgery-specific quality and research studies conducted on RWD from EHRs. We sought to adapt and apply a previously described data quality framework to assess the quality of our sitespecific RWD repository, which automatically curates EHR data to support surgical research. We propose several adaptations to the framework to enhance quality assessment. We also demonstrate how surgical research questions can be mapped to 2 different kinds of data quality checks (foundational and study-specific) to ensure that the RWD are fit for use for a proposed analysis.

METHODS The Duke IRB approved this study with a waiver of informed consent. This was a single-center, retrospective study at a large quaternary, multisite hospital system with 70,000 operations and 68,000 inpatient hospitalizations in 2017, and is reported as per the Strengthening the Reporting of Observational Studies in Epidemiology guidelines (eDocument 1). Cohort An electronic data pipeline was developed to automatically source EHR data from adult surgical patients.23

Key Points Question: How can researchers systematically assess the data quality and fitness of a real-world data (RWD) repository for surgical research questions? Findings: By adapting and applying a previously described data quality framework, we assessed the quality of our site-specific RWD and its fitness in supporting surgical research. Defining 2 different types of data quality analysis (foundational and study-specific), investigators can evaluate RWD on multiple levels of complexity tailored to their specific hypothesis. Meaning: Assessing the quality and fitness of RWD before research investigation is critical. Using our methodology, researchers can apply this framework to reliably test whether the RWD is fit for use.

Procedural encounters with a CPT code for invasive procedures as defined by Surgery Flags Software for ICD-9CM (Healthcare Cost and Utilization Project) were included from both inpatient and outpatient encounters. The data model, presented in eTable 1, included features identified by an interdisciplinary group of local surgical researchers and those in the PCORnet common data model.29 Data quality analyses were based on a cohort of 241,695 patients undergoing 412,182 procedures (as defined by 1 CPT code) between October 1, 2014 and August 31, 2018. Data quality checks To assess the quality of the RWD repository, we built on the harmonized data quality framework containing the following groups of quality checks: conformance, completeness, and plausibility.27 We developed a series of surgical-specific “queries” that drove each data quality check. These types of queries represented either a research hypothesis or quality-improvement project. Conformance checks tested that variable type, value, range, and computational output matched prespecified expectations.27 For example, we assessed the degree to which cleaned medication names in the data matched an external standard by comparing spelling. This was completed by tokenizing raw medication names and matching strings to RxNorm30 medication name strings. Names that did not match were cleaned manually by experts. Completeness checks reported percent missingness of data and plausibility checks

Vol.

-,

No.

-, -

2020

Corey et al

tested data variable credibility within clinical contexts. Plausibility checks also tested the validity of variables across data tables and were further split into the following types: uniqueness, a-temporal, and temporal. Uniqueness plausibility checks determined whether variables that represented individual entities were unique (eg each patient had a unique medical record number). A-temporal plausibility checks tested the validity of variable relationships without regard to time (eg all surgical encounters had an associated documented diagnosis code). Temporal plausibility checks tested the validity of variable relationships with regard to time (eg all inpatient discharge dates occurred after inpatient procedure dates). Conformance, completeness, and plausibility checks were further divided into foundational and study-specific subtypes. Foundational checks were general quality analyses encompassing all data points, and study-specific checks targeted morespecific clinical questions. Queries represent research or quality-improvement questions that assess an unknown, and data quality checks are confirmation of a known quality within a data asset. We adapted the harmonized data quality framework by developing the following new types of temporal plausibility checks: data shift and data drift checks. These checks require institutional knowledge of an expected change in data due to a modification in either healthcare delivery or data codification/capture. Data shift checks tested whether a sudden change in clinical practice could be observed. For example, demonstrating the expected change from ICD-9 to ICD-10 codes after implemention of ICD-10 is a data shift check. Our study-specific experiment for this data shift check used new CPT codes for back procedures (22853, 22854, and 22859) introduced in January 2017 compared with the old CPT code 22851. However, data drift checks tested whether an expected gradual change in clinical practice could be observed. For example, whether site-specific policy changes could be replicated in the EHR data. We analyzed provider response to state/hospital policy recommended changes for pain prescription. This known institutional shift should be visualized within the RWD in the expected time. Identifying changes in policy and practice up front and explicitly stating expected data shifts and drifts can help prevent misinterpretation of findings.31,32 Confirming that these expected systems changes are observed illustrates how well a data source represents the dynamic reality of how care is delivered within the institution. To demonstrate how surgical inquiries interface with data quality checks, 5 prototypical surgical queries were created spanning preoperative, intraoperative, and postoperative time frames. Each query was used to drive the

Assessing Quality of Surgical Real-World Data

3

types of foundational (Table 1) and study-specific (Table 2) data quality checks. Data shift and data drift temporal plausibility checks were included for queries 1 and 5 (Table 2). Foundational checks were necessary for determining RWD quality, but were not sufficient in determining data fitness for any specific question. Some study-specific checks required institutional policy knowledge to complete. Taken together, foundational and study-specific conformance, completeness, and plausibility checks ensure the RWD are rigorously evaluated or “fit for use” to answer the query.

RESULTS Demographic, comorbidity, and surgical characteristics are shown in Table 3. The data elements from the 5 queries were evaluated with foundational completeness, conformance, and plausibility checks. Study-specific plausibility checks were completed to ensure that the RWD were fit for use. Foundational checks Standard foundational checks across all variables were studied. Example results from these checks are shown in eTable 2. These types of foundational checks are not research question-specific and are applied across a data source for baseline quality measures. The majority of these checks ensure conformance with the accompanying data dictionary (eTable 1). Preoperative Foundational checks for query 1 demonstrated that all procedures within an inpatient and outpatient encounter had an associated CPT code matching numerical values defined by the American Medical Association. Intraoperative Foundational checks for query 2 revealed all vital sign values and intraoperative medication doses were numeric. Further analysis demonstrated that 99% of procedural encounters had vital signs recorded for the same encounter, either preoperatively or postoperatively. However, only 69% of procedures had vital signs recorded within the correct EHR encounter intraoperatively. After tokenization, all intraoperative blood pressure medication string names matched RxNorm30 spelling standards. The entire string, however, was not an exact match. For example, a raw medication administration in our EHR is “HYDRALAZINE 20 MG/ML INJECTION SOLUTION,” which contains the correct spelling of “HYDRALAZINE” and in turn will map to RxNorm anatomical therapeutic chemical level 2 class “ANTIHYPERTENSIVES.”30

1

Preoperative

2

Intraoperative

3

Postoperative

4

Postoperative

5

Postoperative

How rapidly is a new back surgery (identified via CPT code) adopted? Which intraoperative medication is most effective for intraoperative hypertension?

Do patients who undergo total knee replacement have better outcomes if they receive physical therapy before discharge? What is the rate of postoperative MI?

What pain medication is highly correlated with the incidence of postoperative ileus?

Foundational completeness check

Foundational temporal plausibility check

Data table required

CPT codes match numerical value defined by AMA

Describe the missingness of CPT code for procedure encounter

CPT coding is observed over entire cohort time period

Procedure

Vital sign value and intraoperative medication administration doses are numerical; blood pressure medication names match RxNorm naming convention Provider note type conforms to local care practice (eg patient history, progress note, consult note, and nursing)

Describe the missingness of intraoperative vital sign value for inpatient procedure

Intraoperative vital signs occur on the same data as an associated procedure

Procedure, medication administration, flowsheet

Describe the missingness of provider note for procedure encounter

Procedure notes are written within 1 wk after procedure

Provider notes, procedure

Lab result value is numeric

Describe the missingness of lab value for completed lab

Procedure, inpatient encounter, lab, order, and diagnosis

Medication name conforms to local care practice

Describe the missingness of outpatient medication list

Inpatient encounter has ordered lab during date of admission; patient death date occurs after procedure date that is performed Outpatient medication are available for procedure encounter date

Assessing Quality of Surgical Real-World Data

Operative time frame

Foundational conformance check

Corey et al

Query no.*

Example query question

4

Table 1. Examples of Foundational Surgical Data Quality Checks Within Perioperative Time Frame for a Specific Site

Procedure, outpatient medication list, outpatient encounter

J Am Coll Surg

*Query 1 was preoperative and assessed temporal changes in CPT codes. Query 2 was intraoperative and examined administration of medication to reduce blood pressure during procedure. The last 3 queries were postoperative; query 3 was focused around postoperative clinician notes and services; query 4 examined surgical complication and mortality; and query 5 assessed postoperative pain management. AMA, American Medical Association.

Vol. -,

Operative time frame

Example query question

Study-specific completeness

Study-specific temporal plausibility check

Describe the missingness of new back surgery CPT code for procedure that represents new back surgery after code implementation Describe the missingness of intraoperative blood pressure value recorded and intraoperative medication dosage value

Data shift: new CPT code introduced January 2017

Procedure

Intraoperative IV blood pressure medication administration is associated with systolic blood pressure of >140 mmHg recorded Postoperative physical therapy note present after total knee replacement

Procedure, medication administration, flowsheet

How rapidly is a new back surgery (identified via CPT code) adopted?

New back surgery CPT code conforms to numerical value defined by AMA

2

Intraoperative

Which intraoperative medication is most effective for intraoperative hypertension?

3

Postoperative

4

Postoperative

Do patients who undergo total knee replacement have better outcomes if they receive physical therapy before discharge? What is the rate of postoperative MI?

Systolic blood pressure is greater than diastolic blood pressure; units for IV medication dose conform to conventional unit measurement All notes tagged as physical therapy notes were written by physical therapist

5

Postoperative

What pain medication is highly correlated with the incidence of postoperative ileus?

Describe the missingness of patient physical therapy note after inpatient procedure

Troponin-T result value is numeric

Describe the missingness of troponin-T value in patients with postoperative MI

Opioid medication name conforms to proper prescription expectation

Describe the missingness of expected pain medication on outpatient list

Troponin-T value means are stable over the cohort time period; illustrate data quality of established institutional policy: all postoperative readmission with high troponin levels also have orders placed for ECG; these encounters are associated with diagnosis causing high troponin levels Data drift: (illustrate data quality of established institutional policy); postoperative opioid prescription decreased after state and hospital policy implementation to curb prescription

Data table required

Provider note, procedure

Procedure, inpatient encounter, lab, order, and diagnosis

Procedure, outpatient medication list, outpatient encounter

Assessing Quality of Surgical Real-World Data

Preoperative

Corey et al

1

2020

Study-specific conformance

-, -

Query no.

No.

Table 2. Examples of Study-Specific Surgical Data Quality Checks Within Perioperative Time Frame for a Specific Site

5

6

Corey et al

Assessing Quality of Surgical Real-World Data

Table 3. Baseline Characteristics of the Surgical Cohort Represented in the Dataset

Baseline characteristic

Patients, n Encounter, n Age, y, mean  SD Female sex in patient population, n (%) Inpatient encounter, n (%) Procedure code, n (%) Unique procedural class Top 5 most common procedural classes Colonoscopy and biopsy Upper gastrointestinal endoscopy, biopsy Lens and cataract procedure Other diagnostic radiology and related technique Diagnostic bronchoscopy and biopsy of bronchus Race, n (%)
2 races American Indian or Alaska Native Black or African American Caucasian/white Asian Native Hawaiian or Pacific Islander Other/missing Ethnicity, n (%) Non-Hispanic Hispanic Other/missing Top 10 diagnoses codes in inpatient population, n (%) Sepsis, unspecified organism Unilateral primary osteoarthritis, right knee Unilateral primary osteoarthritis, right hip Unilateral primary osteoarthritis, left knee Spinal stenosis, lumbar region Malignant neoplasm of prostate Morbid (severe) obesity due to excess calories Unilateral primary osteoarthritis, left hip Infection after procedure, initial encounter Nonrheumatic aortic (valve) stenosis Hospital length of stay for inpatient population, d, mean (SD) Mortality, n (%)

Surgical cohort (October 2014 to August 2018)

241,659 412,182 57.8  15.8 138,130 (60.1) 98,821 (24) 211 79,054 (19.2) 45,281 (11.0) 33,211 (8.1) 11,790 (2.9) 11,463 (2.6)

1,297 617 25,618 66,111 1,475 62 3,576

(1.31) (0.62) (26) (66.9) (1.50) (0.06) (3.62)

93,490 (94.6) 3,061 (3.12) 2,271 (2.30)

1,917 (1.94) 1,702 (1.72) 1,624 1,590 1,492 1,417 1,406

(1.64) (1.61) (1.51) (1.43) (1.42)

1,303 (1.32) 1,244 (1.26) 882 (0.89) 11 (22) 14,036 (5.8)

J Am Coll Surg

Postoperative Foundational checks for query 3 showed a low rate of missingness in operating room case identifiers and dates for procedural encounters. However, specifically coded “procedure note” in note type fields for procedural encounters were missing in approximately 15% of cases. Note types demonstrated stability over time (Fig. 1A) with “progress note” and “discharge summary” being the 2 most common notes. For query 4, a foundational plausibility check assessed whether inpatient encounter dates had laboratory values occurring between admission and discharge dates. Analysis to determine whether patient death timestamps were recorded after procedures was also performed. Every inpatient admission had at least 1 laboratory result occurring during the inpatient stay. Eighty deaths were observed before a procedure. Chart review confirmed that 96.3% of these patients were organ donors undergoing organ procurement. Foundational checks for query 5 demonstrated proper medication naming. After tokenization and manual cleaning, all medication names matched RXnorm conventions.30 All of these procedures had current outpatient medication lists linked to the date of the procedure. Eighty-four percent (n ¼ 345,620) of procedures defined as “invasive” by Healthcare Cost and Utilization Project’s Surgery Flags Software for ICD-9-CM had an outpatient medication list corresponding to the procedure hospital encounter dates and 15% (n ¼ 61,718) of these procedures had a pain medication prescribed postoperatively. A list of these medications is provided in eTable 3. Study-specific checks Preoperative Additional study-specific plausibility checks were performed (Table 2). Data shift analyses tested data capture of new CPT codes for back procedures. These specific CPT codes were implemented immediately; the codes were first observed on January 3, 2017 and 687 procedures were associated with the codes through August 31, 2018. Earlier CPT code was not seen after implementation of new codes. Intraoperative Query 2 tested both completeness and plausibility of intraoperative vital signs and medication. All units of intraoperative medication doses for IV blood pressure medication used conventional dosing units (eg mg, mg, mg/min, mg/min, and mg/h). For 13,500 patients who received IV blood pressure-lowering medication (eTable 4), 93% (12,644) had a recorded intraoperative

Vol.

-,

No.

-, -

2020

Corey et al

Assessing Quality of Surgical Real-World Data

7

Figure 1. (A) Analysis of note type variable input stabilities over time. Progess notes are the most common type of provider note, followed by discharge summaries. (B) Postoperative procedure note completion over time. All notes with a procedural note are completed by 30 days after the corresponding procedure

systolic blood pressure >140 mmHg. All medication administrations had a recorded corresponding dose value and all patients who received intraoperative blood pressure-lowering medication had a recorded intraoperative blood pressure. Postoperative Query 3 demonstrated that all notes with “procedure note” type were written within 30 days of the corresponding procedure (Fig. 1B). Institutional policies required that all patients with total knee arthroplasty received physical therapy. The data demonstrated that of the 2,477 patients who underwent total knee arthroplasty, 93% had associated postoperative physical therapy notes. Data quality checks for query 4 demonstrated that troponin-T laboratory values were clinically plausible and stable (eFig. 1). Another plausibility check based on institutional policy demonstrated that all patients (n ¼ 1,357) who had a high troponin-T lab level (
0.10 ng/ mL) within 90 days postoperatively had at least 1 ECG ordered on the same day. In addition, all 1,357 patients with a high troponin-T level had a diagnosis code associated with high troponin levels. These diagnosis codes data

were associated with the correct hospital encounter; the top diagnoses and counts are shown in eTable 5. Lastly, data quality checks for query 5 demonstrated a data drift. Based on statewide and institutional policy, local providers have decreased opioid prescription for postoperative pain management. This known healthcare delivery trend was used to check the expected change. Data demonstrated that the rates of opioid analgesic prescription within 30 days postoperatively gradually decreased as the rate of nonopioid analgesic prescription gradually increased, illustrated in Figure 2. Between October and December 2014, 31.2% of postoperative patients were prescribed opioids and 68.8% of them were prescribed nonopioid pain medication. Four years later, in 2018, these changed to 22.4% and 77.6%, respectively.

DISCUSSION Most data quality analyses focus on specific foundational checks evaluating the structure of the data.33,34 Conformance and completeness checks explore variable types, ranges, unique table keys, and percent missingness,27 and plausibility checks ensure that patient procedure dates

8

Corey et al

Assessing Quality of Surgical Real-World Data

J Am Coll Surg

Figure 2. Postoperative pain medication trends over time. Postoperative opioid prescription decreased by 8.8% and nonopioid prescription increased by 8.8% between 2014 and 2018.

do not occur before hospital admission dates or that laboratory values have clinically expected results. If numerical outliers are detected,35 they are subsequently excluded. These checks have the ability to uncover many data inaccuracies. It is common for sites participating in a distributed research network to learn about local data anomalies from a data coordinating center, raising the possibility for data quality-improvement programs within EHR systems themselves.36 However, these checks remain generic with regard to the suitability of a dataset to answer specific surgical questions. When designing data quality checks, it is important to distinguish analyses that demonstrate data quality from those that reflect the quality of delivered care. Although difficult, this distinction is necessary. We constructed our data quality checks to demonstrate either established stabilities or changes that are known to have occurred within our health system. For example, knowing when system implementation of new billing codes was necessary in determining whether our RWD reflected that anticipated change. Care delivery practices are dynamic and the more investigators explicitly state plausibility checks, expected data shifts, and expected data drifts, the better prepared the research team can be to conduct analyses and interpret results. We demonstrate how specific query-driven data quality checks, performed alongside foundational checks, can determine whether RWD are fit for use to address surgical research questions. Study-specific checks differ from foundational checks in that they are more in-depth and require specific medical and even hospital knowledge. For example, plausibility checks from query 3 (Tables 1 and 2) assess the data quality of provider notes during a procedural encounter. Broad, foundational plausibility checks demonstrated that of the procedure notes that are captured within our database, 100% were written

within 30 days after the procedure. However, applying knowledge of an institutional patient care policy allowed us to generate a study-specific plausibility check for patients undergoing hip arthroplasty (Table 2). Focusing on the specific cohort relevant to a study question demonstrated a different level of data quality compared with the plausibility of note date timestamps alone. The second example set of plausibility checks was conducted to determine how many postoperative MIs result in a readmission within 30 days after a specific procedure. This type of surgical query is fairly common, as it relates to healthcare quality measures and requires institutional knowledge about how postoperative patients are evaluated.37,38 This check assessed troponin lab values, associated ECG orders, and billing diagnosis codes. Troponin-T results appear stable in the RWD repository over time (eFig. 1). Combining data on ECG orders and diagnosis codes (eTable 5) tested relationships across 4 different tables in the RWD repository. In addition to designing data quality checks for proposed surgical quality and research studies, we developed additional types of temporal plausibility checks that enhanced our data quality assessment. “Data shift” and “data drift” (Fig. 2) quality checks were developed to assess whether anticipated local changes in clinical practice could be observed within the local RWD repository. These types of changes in underlying data are too often recognized after completion of analyses and can have significant implications. For example, the benefit of the hospital readmission reduction program might have been overstated due to a data shift,31 and predictive model performance has been shown to deteriorate during data shifts or drifts.32,39 In conducting these analyses, we observed an institutional change in the data on newly adopted CPT codes immediately after their official introduction.

Vol.

-,

No.

-, -

2020

Corey et al

However, not all changes in care delivery data are abrupt. Data drift plausibility checks were able to test whether more gradual changes in healthcare were being captured. To determine whether our RWD repository captured these more subtle changes, we assessed systemwide opioid prescription up to 30 days postoperatively. Opioid addiction is a well-documented long-term end point after inpatient surgical procedure,40,41 and several policies have emerged to curb postoperative opioid prescription.42 Figure 2 illustrates this data drift as a gradual downward trend in postoperative opioid medication and a subsequent upward trend of nonopioid medication. Future research can better characterize changes in opioid-prescribing practices, but the confirmation of the observed trend provides investigators with meaningful insight into the data quality to complete the analysis. To date, most data quality assessment has been uncoupled from local research conducted using EHR data. Data quality assessment for distributed research networks attest to the integrity of their observations from pooled data. However, researchers using local EHR data from a single site do not regularly perform and report such analyses, potentially undermining validity of their findings. Within surgery specifically, many projects are conducted on data sources that cannot robustly answer the proposed research question.7 Our application of this data quality framework on local surgical data demonstrates how a local RWD repository can be assessed as fit for use to address surgical quality and research queries. Performing in-depth data quality assessment locally can identify opportunities to improve data-capture processes and improve data quality for future research. For example, 3 patient encounters were identified for which a death date occurred before a procedure date not for postmortem organ procurement. Chart reviews revealed that patient birth dates were being captured as death dates; the errors were subsequently corrected. We propose that this process be iterated so that local systems can aggregate different data quality assessment not only for different fields of research, but over time. Continual data quality assessment should be embedded into local EHR RWD research practices so that changes in data capture can be noted and resolved as they occur. In addition, sites that have similar EHR quality constructs will be able to pool their data for research efforts with more confidence in data quality. Limitations to this work include analyses involving specific data features from a single healthcare system. In addition, the depth and complexity of analysis for each query example were limited. The purpose of the study was to apply a generalizable framework for assessing the quality of an RWD repository derived from EHR data. Five hypothetical surgery queries were used as test cases for

Assessing Quality of Surgical Real-World Data

9

quality assessment; we did not rigorously test the query hypotheses, but rather presented examples of how to ensure the data can be used for formal hypothesis testing. The study-specific check sets that were developed span specific surgical use cases and do not generalize across the hospital or to other use cases. New research questions will necessitate the development of additional studyspecific and foundational data quality checks. Future efforts include requiring these types of checks to be performed internally by researchers using this RWD repository. Documentation and publication of data quality check results should be paired with the main study analysis results and best practices, for such reporting are emerging.43,44 Although the current study assesses data quality of 4 years’ of data, new assessment must be conducted as new data are captured. Changes in the data model, collection processes, and care delivery provide opportunities for new data quality issues to emerge.28 Lastly, electronic data are entered by individual providers and a key issue in the data integrity is the degree of internal consistency. Healthcare institutions can reduce the inherent variability in this by providing training to providers on billing, coding, and documentation, as well as by ensuring that information technology systems are optimized to structure and annotate data that are entered into EHR systems. This type of check can only be resolved by periodic manual adjudication on a subset of the data.

CONCLUSIONS We developed a set of foundational and study-specific checks to test a data quality framework for an institutional RWD repository. These checks included various conformance, completeness, and plausibility checks based on plausible local and national data drifts and shifts. These checks were developed to assess the evidence generation capabilities of the data for future surgical quality assessment and other research use. By combining foundational and study-specific checks, as well as data shift and DATA drift checks, the quality of surgical studies performed on RWD data can continue to improve. Author Contributions Study conception and design: Corey, Helmkamp, Simons, Curtis, Marsolo, Balu, Gao, Nichols, Watson, Mureebe, Kirk, Sendak Acquisition of data: Corey, Helmkamp, Simons, Balu, Gao, Nichols, Sendak Analysis and interpretation of data: Corey, Helmkamp, Simons, Curtis, Marsolo, Gao, Nichols, Watson, Mureebe, Kirk, Sendak

10

Corey et al

Assessing Quality of Surgical Real-World Data

Drafting of manuscript: Corey, Helmkamp, Simons, Mureebe, Kirk, Sendak Critical revision: Corey, Curtis, Marsolo, Balu, Sendak Acknowledgment: Kristin M Corey and Morgan Simons had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Duke Institute for Health Innovation employees participated in this study and were involved in the study design, data collection and analysis, and preparation of the manuscript. REFERENCES 1. US FDA. Framework for FDA’s Real-World Evidence Program. Silver Spring, MD: US FDA; 2018. 2. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidencedwhat is it and what can it tell us? N Engl J Med 2016; 375:2293e2297. 3. Sun E, Mello MM, Rishel CA, et al. Association of overlapping surgery with perioperative outcomes. JAMA 2019;321: 762e772. 4. Presley CJ, Tang D, Soulos PR, et al. Association of broadbased genomic sequencing with survival among patients with advanced nonesmall cell lung cancer in the community oncology setting. JAMA 2018;320. 469e469. 5. Singal G, Miller PG, Agarwala V, et al. Association of patient characteristics and tumor genomics with clinical outcomes among patients with nonesmall cell lung cancer using a clinicogenomic database. JAMA 2019;321:1391e1399. 6. Horvath MM, Winfield S, Evans S, et al. The DEDUCE Guided Query tool: providing simplified access to clinical data for research and quality improvement. J Biomed Inform 2011;44:266e276. 7. Haider AH, Bilimoria KY, Kibbe MR. A checklist to elevate the science of surgical database research. JAMA Surg 2018; 153:505e507. 8. Kaji AH, Rademaker AW, Hyslop T. Tips for analyzing large data sets from the JAMA Surgery Statistical Editors. JAMA Surg 2018;153:508e509. 9. Raval MV, Pawlik TM. Practical guide to surgical data sets: National Surgical Quality Improvement Program (NSQIP) and Pediatric NSQIP. JAMA Surg 2018;153. 764e762. 10. Telem DA, Dimick JB. Practical guide to surgical data sets: Metabolic and Bariatric Surgery Accreditation and Quality Program (MBSAQIP). JAMA Surg 2018;153. 766e762. 11. Massarweh NN, Kaji AH, Itani KMF. Practical guide to surgical data sets: Veterans Affairs Surgical Quality Improvement Program (VASQIP). JAMA Surg 2018;153. 768e762. 12. Farjah F, Kaji AH, Chu D. Practical guide to surgical data sets: Society of Thoracic Surgeons (STS) National Database. JAMA Surg 2018;153. 955e952. 13. Merkow RP, Rademaker AW, Bilimoria KY. Practical guide to surgical data sets: National Cancer Database (NCDB). JAMA Surg 2018;153:850e852. 14. Doll KM, Rademaker A, Sosa JA. Practical guide to surgical data sets: Surveillance, Epidemiology, and End Results (SEER) Database. JAMA Surg 2018;153:588e589.

J Am Coll Surg

15. Ghaferi AA, Dimick JB. Practical guide to surgical data sets: Medicare claims data. JAMA Surg 2018;153. 677e672. 16. Stulberg JJ, Haut ER. Practical guide to surgical data sets: Healthcare Cost and Utilization Project National Inpatient Sample (NIS). JAMA Surg 2018;153:586e587. 17. Schoenfeld AJ, Kaji AH, Haider AH. Practical guide to surgical data sets: Military Health System Tricare Encounter Data. JAMA Surg 2018;153. 679e672. 18. Hashmi ZG, Kaji AH, Nathens AB. Practical guide to surgical data sets: National Trauma Data Bank (NTDB). JAMA Surg 2018;153. 852e852. 19. Hersh WR, Weiner MG, Embi PJ, et al. Caveats for the use of operational electronic health record data in comparative effectiveness research. Med Care 2013;51[Suppl 3]:S30eS37. 20. Bayley KB, Belnap T, Savitz L, et al. Challenges in using electronic health record data for CER: experience of 4 learning organizations and solutions applied. Med Care 2013;51[Suppl 3]:S80eS86. 21. Fleurence RL, Curtis LH, Califf RM, et al. Launching PCORnet, a national patient-centered clinical research network. J Am Med Inform Assoc 2014;21:578e582. 22. Behrman RE, Benner JS, Brown JS, et al. Developing the Sentinel Systemda national resource for evidence development. N Engl J Med 2011;364:498e499. 23. Corey KM, Kashyap S, Lorenzi E, et al. Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): a retrospective, single-site study. PLoS Med 2018;15[11]:e1002701. 24. The Office of the National Coordinator for Health Information Technology. Percent of hospitals, by type, that possess certified health IT. Available at: https://dashboard.healthit.gov/ quickstats/pages/certified-electronic-health-record-technologyin-hospitals.php. Accessed May 7, 2019. 25. Massarweh NN, Chang GJ. Translating research findings into practicedthe unfulfilled and unclear mission of observational data. JAMA Surg 2019;154:103e104. 26. Sendak MP, Balu S, Schulman KA. Barriers to achieving economies of scale in analysis of ehr data: a cautionary tale. applied clinical informatics. Appl Clin Inform 2017;8: 826e831. 27. Kahn MG, Callahan TJ, Barnard J, et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC) 2016;4:1e18. 28. Qualls LG, Phillips TA, Hammill BG, et al. Evaluating foundational data quality in the national Patient-Centered Clinical Research Network (PCORnet). EGEMS (Wash DC) 2018;6 [1]:3. 29. The National Patient-Centered Clinical Research Network. PCORnet Common Data Model (CDM). Available at: https://pcornet.org/pcornet-common-data-model/. Accessed May 20, 2019. 30. US National Library of Medicine. Unified Medical Language System. RxNorm. Available at: https://www.nlm.nih.gov/ research/umls/rxnorm/. Accessed May 18, 2018. 31. Ody C, Msall L, Dafny LS, et al. Decreases in readmissions credited to Medicare’s program to reduce hospital readmissions have been overstated. Health Aff (Millwood) 2019;38: 36e43.

Vol.

-,

No.

-, -

2020

Corey et al

32. Nestor B, McDermott MB, Chauhau G, et al. Rethinking clinical prediction: why machine learning must consider year of care and feature aggregation. Presnted at the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montre´al, Canada, November 2018. 33. Raebel MA, Haynes K, Woodworth TS, et al. Electronic clinical laboratory test results data tables: lessons from MiniSentinel. Pharmacoepidemiol Drug Saf 2014;23:609e618. 34. Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform 2013;46:830e836. 35. Khare R, Utijian L, Ruth BJ, et al. A longitudinal analysis of data quality in a large pediatric data research network. J Am Med Inform Assoc 2017;24:1072e1079. 36. Brown JS, Kahn M, Toh S. Data quality assessment for comparative effectiveness research in distributed data networks. Med Care 2013;51[Suppl 3]:S22eS29. 37. Writing Committee for the VISION Study Investigators. Association of postoperative high-sensitivity troponin levels with myocardial injury and 30-day mortality among patients undergoing noncardiac surgery. JAMA 2017;317: 1642e1651.

Assessing Quality of Surgical Real-World Data

11

38. Bertges D, Neal D, Schanzer A, et al. The Vascular Quality Initiative Cardiac Risk Index for prediction of myocardial infarction after vascular surgery. J Vasc Surg 2016;64:1411e1421. 39. Davis SE, Lasko TA, Chen G, et al. Calibration drift in regression and machine learning models for acute kidney injury. J Am Med Inform Assoc 2017;24:1052e1061. 40. Brummett CM, Waljee JF, Goesling J, et al. New persistent opioid use after minor and major surgical procedures in US adults. JAMA Surg 2017;152[6]. e170504. 41. Sun EC, Darnall BD, Baker LC, Mackey S. Incidence of and risk factors for chronic opioid use among opioid-naive patients in the postoperative period. JAMA Intern Med 2016;176[9]: 1286e1293. 42. N.C. Legis. S. Strengthen Opioid Misuse Prevention (STOP) Act (House Bill 243), January 2018. 43. Gebru T, Morgenstern J, Veccione B, et al. Datasheets for datasets. Available at: https://arxiv.org/pdf/1803.09010.pdf. Accessed May 2, 2019. 44. Holland S, Ahmed H, Newman S, et al. The dataset nutrition label: a framework to drive higher data quality standards. Available at: https://arxiv.org/ftp/arxiv/papers/1805/1805. 03677.pdf. Accessed May 2, 2019.

Corey et al

J Am Coll Surg

Assessing Quality of Surgical Real-World Data

3

Troponin T

2

Mean Value

11.e1

1

0

2015

2016

2017

2018

Time

eFigure 1. Analysis of troponin-T mean during cohort time period.

Vol.

-,

No.

-, -

Corey et al

2020

Assessing Quality of Surgical Real-World Data

11.e2

eTable 1. Data Dictionary of Real-World Data Pipeline table

adt adt adt adt adt adt adt adt adt adt adt adt adt adt adt adt adt adt allergy allergy allergy allergy allergy allergy allergy allergy allergy allergy allergy allergy analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte

Variable

encounter_id patient_id event_id event_occurred_at event_occured_at_raw event_type_id event_type_label event_subtype_id event_subtype_label from_base_class_id from_base_class_label to_base_class_id to_base_class_label department_id department_label hospital_service location_id location_label patient_id allergen_name reaction_label reaction_line reaction_updated_at reaction_updated_at_raw allergy_updated_at allergy_updated_at_raw allergy_entered_at allergy_entered_at_raw severity_label allergy_status_label encounter_id patient_id component_id component_label common_name test_id test_label proc_id proc_label order_proc_id order_label ordered_at ordered_at_raw result_lab_id result_lab_label specimen_source_label collected_at collected_at_raw

Variable definition

Variable type

Encounter identifier that uniquely identified patient encounters ID used to uniquely identify a patient Event identifier for ADT Datetime of ADT event in event type Datetime of ADT event in event type Integer representation of the ADT event type ADT event type describing the type of transfer occurring at event time Integer representation of the ADT event type ADT event type describing the type of transfer occurring at event time Integer representation of the class of group being transferred FROM The class of group being transferred FROM Integer representation of the class of group being transferred TO The class of group being transferred TO Integer representation of the department being transferred to or from Name of the department being transferred to or from Name of the hospital service being transferred to or from Integer ID for the ADT location Physical location of the transfer to or from ID used to uniquely identify a patient Name of the allergen to which the patient has a reaction Reaction to the medication Record line number of the reaction details Time of reaction (UTC conversion) Time of reaction (raw time) Time of allergy (UTC conversion) Time of allergy (raw time) Time of allergy being recorded (UTC conversion) Time of allergy being recorded (raw time) Severity of allergy Status of the allergy Encounter identifier that uniquely identifies patient encounters ID used to uniquely identify a patient Integer representation components of the order set Name of component Name of component matched to component ID Integer representation of test labels Test labels Unique integer ID for specific procedure Procedure name Unique integer ID for specific procedure Same as procedure label Time procedure ordered Time procedure ordered Unique integer ID for specific lab Laboratory name Source of specimen Time procedure ordered (UTC conversion) Time procedure ordered (raw time)

Integer String Integer Datetime Datetime Integer String Integer String Integer String Integer String Integer String String Integer String String String String Integer Datetime Datetime Datetime Datetime Datetime Datetime String String Integer String Integer String String Integer String Integer String Integer String Datetime Datetime Integer String String Datetime Datetime (Continued)

11.e3

Corey et al

Assessing Quality of Surgical Real-World Data

J Am Coll Surg

eTable 1. Continued Pipeline table

Variable

analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte analyte demographic demographic demographic demographic demographic demographic demographic demographic demographic demographic demographic demographic demographic encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter

resulted_at resulted_at_raw ordering_provider_id ordering_provider_npi ordering_location_id order_dept_name line raw_name raw_value_txt raw_value raw_unit normal_lower_bound normal_upper_bound atom_name atom_value_txt atom_value atom_unit patient_id updated_at update_at_raw patient_name sex_label race_line race_label ethnicity_label address city state_label county_label zip patient_id patient_mrn contacted_at contacted_at_raw encounter_effective_at encounter_effective_at_raw enc_type_id provider_id enc_type_label appt_sched_at appt_sched_at_raw checked_in_at checked_in_at_raw hospital_admitted_at hospital_admitted_at_raw hospital_discharged_at hospital_discharged_at_raw admission_type_id

Variable definition

Time lab resulted (UTC conversion) Time lab resulted (raw time) Integer ID for name of provider Provider NPI code Integer for location Department name Number of row associated with unique key of the table Name of lab collected Value of lab collected Integer value of lab collected Unit of measurement of lab collected Lower bound of lab collected if normal Upper bound of lab collected if normal Lab name Lab value if text Lab value if numeric Lab value unit ID used to uniquely identify a patient Date pulled Date pulled Patient name Sex type Race filled out Race type Ethnicity type Actual address City name State name County name ZIP code ID used to uniquely identify a patient Patient MRN Time contacted before encounter (UTC conversion) Time contacted before encounter (raw time) Encounter open (UTC conversion) Encounter open (raw time) Integer representation of encounter type Integer representation of provider Name of encounter type Appointment time (UTC conversion) Appointment time (raw time) Check-in time (UTC conversion) Check-in time (raw time) Hospital admit time (UTC conversion) Hospital admit time (raw time) Hospital discharge time (UTC conversion) Hospital discharge time (raw time) Admission type

Variable type

Datetime Datetime String String Integer String Integer String float String String String String String String float String String Datetime String Integer String String String String String String String String String String String Datetime Datetime Datetime Datetime String String String Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime String (Continued)

Vol.

-,

No.

-, -

Corey et al

2020

Assessing Quality of Surgical Real-World Data

11.e4

eTable 1. Continued Pipeline table

encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter encounter flowsheet flowsheet flowsheet flowsheet flowsheet flowsheet flowsheet flowsheet flowsheet flowsheet flowsheet flowsheet flowsheet flowsheet flowsheet icd10_diagnosis icd10_diagnosis icd10_diagnosis icd10_diagnosis icd10_diagnosis icd10_diagnosis icd10_diagnosis icd10_diagnosis icd10_diagnosis icd9_diagnosis icd9_diagnosis icd9_diagnosis icd9_diagnosis icd9_diagnosis icd9_diagnosis icd9_diagnosis icd9_diagnosis icd9_diagnosis ip_encounter

Variable

admission_type_label account_base_class dept_id dept_name chief_complaint location_id location_label visit_type_id visit_type_label location_group_id location_group_label payor_id payor_label inpatient_data_id inpatient_data_id patient_id recorded_at recorder_at_raw val_type_label flowsheet_data_id flowsheet_line flo_meas_id disp_name raw_name raw_value_txt atom_name atom_value_txt atom_value atom_unit encounter_id patient_id line contacted_at contacted_at_raw encounter_effective_at encounter_effective_at_raw enc_dx_line code encounter_id patient_id line contacted_at contacted_at_raw encounter_effective_at encounter_effective_at_raw enc_dx_line code patient_id

Variable definition

Variable type

Integer representation for admission type Billing code for type of encounter Integer representation for department Name of department Text of chief complaint Integer representation of location Location name Integer representation of visit type Name of visit type Integer representation of location of clinic/hospital Name of location of clinic/hospital Integer representation of insurance type Name of insurance type Identifier used to link encounters to flowsheet data Identifier used to link flowsheets to encounter table data ID used to uniquely identify a patient Time measurement recorded (UTC conversion) Time measurement recorded (raw time) Type of value measured Unique key of flowsheet observation Number of row associated with unique key of the table ID for the type of observation Name of the type of observation Further description of the type of observation with location information The observation value Lab name Lab value Lab value Lab unit Encounter identifier that uniquely identifies patient encounters ID used to uniquely identify a patient Number of row associated with unique key of the table Time of contact (UTC conversion) Time of contact (raw time) Time of encounter (UTC conversion) Time of encounter (raw time) Number of row associated with unique encounter diagnosis key ICD-10 code Encounter identifier that uniquely identifies patient encounters ID used to uniquely identify a patient Number of row associated with unique key of the table Time of contact (UTC conversion) Time of contact (raw time) Time of encounter (UTC conversion) Time of encounter (raw time) Number of row associated with unique encounter diagnosis key ICD-9 code ID used to uniquely identify a patient

Integer Integer Integer String String Integer String Integer String Integer String Integer String String Integer String Datetime Datetime String String Integer String String String String String String float String Integer String Integer Datetime Datetime Datetime Datetime Integer String Integer String Integer Datetime Datetime Datetime Datetime Integer String String (Continued)

11.e5

Corey et al

Assessing Quality of Surgical Real-World Data

J Am Coll Surg

eTable 1. Continued Pipeline table

Variable

ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter ip_encounter med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_admin med_list med_list med_list med_list med_list

location_id location_label hospital_admitted_at hospital_admitted_at_raw emergency_admitted_at emergency_admitted_at_raw inpatient_admitted_at inpatient_admitted_at_raw outpatient_admitted_at outpatient_admitted_at_raw discharged_at discharged_at_raw admission_source_label admission_type_label disch_dispo_label drg_id_type_label drg_name drg_mpi_name diagnosis current_icd9_list current_icd10_list inpatient_data_id encounter_id patient_id order_med_id taken_at taken_at_raw mar_time_source_label mar_action_label route_label infusion_rate infusion_rate_unit duration duration_unit department_name raw-name raw-volume_dose raw-volume_dose_unit atom_name atom_dose atom_dose_unit atom_volume atom_volume_unit encounter_id patient_id contacted_at contacted_at_raw encounter_effective_at

Variable definition

Integer representation of location Name of location Time of admission (UTC conversion) Time of admission (raw time) Time of emergency admission (UTC conversion) Time of emergency admission (raw time) Time of inpatient admission (UTC conversion) Time of inpatient admission (raw time) Time of admission if direct from outpatient (UTC conversion) Time of admission if direct from outpatient (raw time) Time of discharge (UTC conversion) Time of discharge (raw time) From where patient was admitted Type of admission To where discharged DRG ID type name DRG name DRG MPI name Diagnosis name ICD-9 list ICD-10 list Identifier used to link encounters to flowsheet data Encounter identifier that uniquely identifies patient encounters ID used to uniquely identify a patient Integer representation of medication Time medication taken (UTC conversion) Time medication taken (raw time) Source of medication administration record Action of medication administration Name of route of administration Rate of infusion Unit of rate infusion Integer of duration Unit of duration Name of department Name of drug Volume amount of drug Volume unit of drug Medication name Medication dose Medication unit Medication volume Medication volume unit Unique ID for each encounter Unique ID for patient Timedate of contact (UTC conversion) Timedate of contact (raw time) Timedate of encounter (UTC conversion)

Variable type

Integer String Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime String String String String String String String String String String Integer String Integer Datetime Datetime String String String String String String String String String String String String String String float String Integer String Datetime Datetime Datetime (Continued)

Vol.

-,

No.

-, -

Corey et al

2020

Assessing Quality of Surgical Real-World Data

11.e6

eTable 1. Continued Pipeline table

med_list med_list med_list med_list med_list med_list med_list med_list med_list med_list med_list med_list note note note note note note note note note note note note note note note note note note note note note note note note note note note order order order order order order order order order

Variable

encounter_effective_at_raw line is_active name strength form route atom_name atom_dose atom_dose_unit atom_volume atom_volume_unit encounter_id patient_id note_id contact_serial_num note_csn_id auth_linked_prov_id author_service_label author_provider_type_label note_status_label contact_num ip_note_type_label entered_at entered_at_raw note_filed_at note_filed_at_raw entered_at_local entered_at_local_raw note_filed_at_local note_filed_at_local_raw created_at created_at_raw serviced_at serviced_at_raw last_filed_at last_filed_at_raw line note_text encounter_id patient_id order_proc_id proc_type_label proc_name description authorizing_provider_id ordered_at ordered_at_raw

Variable definition

Timedate of encounter (raw time) Number of row associated with unique key of the table Medication active Medication name Medication strength Medication form Medication route of administration Medication atomic name Medication atomic dose Medication atomic dose unit Medication atomic volume Medication atomic volume unit Unique ID for each encounter Unique ID for patient Unique ID for each note Same value as NOTE_CSN_ID Unique CSN_ID for each note Unique provider ID number Service of note author Author type Status of the note Contact number for the record Note type Datetime note was entered (UTC conversion) Datetime note was entered (raw time) Datetime note was filed (UTC conversion) Datetime note was filed (raw time) Local datetime note was entered (UTC conversion) Local datetime note was entered (raw time) Local datetime note was filed (UTC conversion) Local datetime note was filed (raw time) Datetime of note creation (UTC conversion) Datetime of note creation (raw time) Datetime note was serviced at (UTC conversion) Datetime note was serviced at (raw time) Datetime not was last filed (UTC conversion) Datetime not was last filed (raw time) Number of row associated with unique key of the table Note text Unique ID for each encounter Unique ID for patient Unique ID for order or procedure Procedure type Procedure name Order or procedure description Unique ID of authorizing provider Datetime order placed at (UTC conversion) Datetime order placed at (raw time)

Variable type

Datetime Integer String String String String String String String String float String Integer String String Integer Integer String String String String String String Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Integer String Integer String Integer String String String String Datetime Datetime (Continued)

11.e7

Corey et al

Assessing Quality of Surgical Real-World Data

J Am Coll Surg

eTable 1. Continued Pipeline table

order pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx pat_loc_hx patient patient patient patient patient problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list problem_list procedure procedure procedure procedure procedure procedure procedure

Variable

status_label patient_id encounter_id event_id event_type_id event_type_label transferred_in_at transferred_in_at_raw transferred_out_at transferred_out_at_raw department_id department_label room_name bed_label location_id location_label service_area_id service_area_label born_at born_at_raw died_at died_at_raw patient_id encounter_id patient_id contacted_at contacted_at_raw encounter_effective_at encounter_effective_at_raw started_at started_at_raw resolved_at resolved_at_raw problem_updated_at problem_updated_at_raw status_changed_at status_changed_at_raw status_label diagnosis_name current_icd9_list current_icd10_list encounter_id patient_id or_case_id or_proc_id status_id status_label total_length

Variable definition

Order status Unique ID for patient Unique ID for each encounter Unique ID for event Numeric ID of the event type Event type Datetime patient transferred in to location (UTC conversion) Datetime patient transferred in to location (raw time) Datetime patient transferred out of location (UTC conversion) Datetime patient transferred out of location (raw time) Unique ID for department Department name Room name Bed label Unique ID for location Location name Unique ID for service area Service area name Date of birth Date of birth raw Date of death Date of death raw Unique ID for patient Unique ID for each encounter Unique ID for patient Datetime patient contacted at (UTC conversion) Datetime patient contacted at (raw time) Datetime of encounter with effective problem (UTC conversion) Datetime of encounter with effective problem (raw time) Datetime problem started (UTC conversion) Datetime problem started (raw time) Datetime problem resolved (UTC conversion) Datetime problem resolved (raw time) Datetime problem updated (UTC conversion) Datetime problem updated (raw time) Datetime status of problem changed (UTC conversion) Datetime status of problem changed (raw time) Status of the problem Diagnosis name ICD-9 code ICD-10 code Unique ID for each encounter Unique ID for patient Unique ID for OR case Unique ID for procedure type Numeric ID of the status of the procedure Status of procedure Length of time for procedure in minutes

Variable type

String String Integer String Integer String Datetime Datetime Datetime Datetime Integer String String String Integer String Integer String Datetime Datetime Datetime Datetime String Integer String Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime Datetime String String String String Integer String String String Integer String Integer (Continued)

Vol.

-,

No.

-, -

Corey et al

2020

Assessing Quality of Surgical Real-World Data

11.e8

eTable 1. Continued Pipeline table

procedure procedure procedure procedure procedure procedure procedure procedure procedure procedure procedure procedure provider provider provider provider social_hx social_hx social_hx social_hx social_hx social_hx social_hx social_hx social_hx social_hx social_hx social_hx social_hx social_hx social_hx

Variable

panel surgery_occurred_at surgery_occurred_at_raw cpt_code proc_name case_class_label service_label contacted_at contacted_at_raw encounter_effective_at encounter_effective_at_raw encounter_type name type title npi encounter_id patient_id contacted_at contacted_at_raw tob_user_label alc_user_label drg_user_label sexually_active_label tob_pack_per_day tob_used_years started_smoking_at quit_smoking_at alcohol_oz_per_wk female_partner_yn male_partner_yn

Variable definition

Number of procedure in surgery Datetime surgery occurred at (UTC conversion) Datetime surgery occurred at (raw time) CPT code Procedure name Type of surgery Surgical service Datetime when instance was created (UTC conversion) Datetime when instance was created (raw time) Datetime of encounter (UTC conversion) Datetime of encounter (raw time) Type of hospital encounter Provider name Provider type Provider title National provider identifier Unique ID for each encounter Unique ID for patient Datetime patient contacted at (UTC conversion) Datetime patient contacted at (raw time) Indicator for tobacco use Indicator for tobacco use Indicator for tobacco use Indicator if patient sexually active Number of packs per day Number of years of tobacco use Datetime patient started smoking Datetime patient quit smoking Alcohol per week in ounces Sexual partner(s) female Y/N Sexual partner(s) male Y/N

Variable type

Integer Datetime Datetime String String String String Datetime Datetime Datetime Datetime String String String String Integer Integer String Datetime Datetime String String String String String String Datetime Datetime String String String

ADT, admit, discharge, transfer; DRG, diagnosis-related group; MPI, multiple procedures index; MRN, medical record number; N, no; NPI, National Provider Identifier; OR, operating room; UTC, Coordinated Universal Time; Y, yes.

11.e9

Corey et al

J Am Coll Surg

Assessing Quality of Surgical Real-World Data

eTable 2. Examples of Generic Foundational Quality Checks and Results Check

Check type

Example result

Cohort inclusion and exclusion criteria

Conformance

Value set Value format Missing data

Conformance Conformance Completeness

Datetime validity

Plausibility

All patients in surgical database had a procedure date within cohort inclusion period (October 1, 2014 to August 31, 2018) Standard concept for populating sex (male, female) in patient table Datetime fields are in standard MM/DD/YYYY format 223 patients (1%) in the surgical database were missing race data in patient table All inpatient encounters for the surgical database have an admission datetime that occurs before the discharge datetime

eTable 3. List of Pain Medication Variable

Medication name

Opioid medication

Fentanyl Hydromorphone Morphine Oxycodone Tramadol Acetaminophen Aspirin Ibuprofen Lidocaine

Nonopioid medication

Vol.

-,

No.

-, -

2020

Corey et al

eTable 4. List of Intraoperative Blood Pressure Medication Blood pressure medication name

Esmolol Labetalol Metoprolol tartrate Hydralazine Clevidipine Nitroglycerin Nicardipine

eTable 5. Diagnosis Codes (ICD-9 and ICD-10) for Patients with Postoperative Elevated Troponin-T Levels Diagnosis description

Acute respiratory failure with hypoxia Acute renal failure, unspecified acute renal failure type Hypotension, unspecified hypotension type Acute kidney injury, unspecified Non-ST elevated MI Sepsis, due to unspecified organism, unspecified Acute blood loss anemia Shock, cardiogenic End-stage renal disease on dialysis Cardiogenic shock Postoperative cardiogenic shock, subsequent encounter Postoperative anemia due to acute blood loss

n

488 460 408 404 308 292 271 228 214 190 183 171

Assessing Quality of Surgical Real-World Data

11.e10

11.e11

Corey et al

Assessing Quality of Surgical Real-World Data

J Am Coll Surg

eDocument 1. Strengthening the Reporting of Observational Studies in Epidemiology Statementd Checklist of items that should beincluded in reports of cohort studies Item no.

Recommendation

Page no.

Title and abstract

1

(a) Indicate the study’s design with a commonly used term in the title or the abstract. (b) Provide in the abstract an informative and balanced summary of what was done and what was found.

2e3

Introduction Background/rationale

2

Explain the scientific background and rationale for the investigation being reported. State specific objectives, including any prespecified hypotheses.

Objectives Methods Study design Setting

3

Participants

6

Variables

7

Data sources/measurement

8*

4 5

Bias Study size Quantitative variables

9 10 11

Statistical methods

12

Results Participants

13*

Descriptive data

14*

Outcomes data Main results

15* 16

Present key elements of study design early in the paper. Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection. (a) Give the eligibility criteria, and the sources and methods of selection of participants. Describe methods of follow-up. (b) For matched studies, give matching criteria and number of exposed and unexposed. Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers. Give diagnostic criteria, if applicable. For each variable of interest, give sources of data and details of methods of assessment (measurement). Describe comparability of assessment methods if there is more than 1 group. Describe any efforts to address potential sources of bias. Explain how the study size was arrived at. Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen and why. (a) Describe all statistical methods, including those used to control for confounding. (b) Describe any methods used to examine subgroups and interactions. (c) Explain how missing data were addressed. (d) If applicable, explain how loss to follow-up was addressed. (e) Describe any sensitivity analyses.

4 4 5 5 5

6

5e6

6 5 5e6 Tables 1 and 2 5e6 Tables 1 and 2

(a) Report numbers of individuals at each stage of study, eg numbers 5 potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analyzed. (b) Give reasons for nonparticipation at each stage. (c) Consider use of a flow diagram. (a) Give characteristics of study participants (eg demographic, clinical, Table 3 social) and information on exposures and potential confounders. (b) Indicate number of participants with missing data for each variable of interest. (c) Summarize follow-up time (eg average and total amount) Report numbers of outcomes events or summary measures over time 6e8 Figures 1 and 2 (a) Give unadjusted estimates and, if applicable, confounder-adjusted estimates and their precision (eg 95% CI). Make clear which confounders were adjusted for and why they were included. (b) Report category boundaries when continuous variables were categorized. (c) If relevant, consider translating estimates of relative risk into absolute risk for a meaningful time period. (Continued)

Vol.

-,

No.

-, -

Corey et al

2020

Assessing Quality of Surgical Real-World Data

11.e12

eDocument 1. Continued Other analyses Discussion Key results Limitations

Item no.

Recommendation

17

Report other analyses done, eg analyses of subgroups and interactions, and sensitivity analyses.

18 19

Summarize key results with reference to study objectives. Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias. Give a cautious overall interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence. Discuss the generalizability (external validity) of the study results.

Interpretation

20

Generalizability Other information Funding

21 22

Page no.

Give the source of funding and the role of the funders for the current study and, if applicable, for the original study on which the present article is based.

An Explanation and Elaboration article discusses each checklist item and gives methodological background and published examples of transparent reporting. The STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist is best used in conjunction with this article (freely available on the web sites of PLoS Medicine at http://www.plosmedicine.org/, Annals of Internal Medicine at http://www.annals.org/, and Epidemiology at http://www. epidem.com/). Information on the STROBE Initiative is available at www.strobe-statement.org. *Give information separately for exposed and unexposed groups.