- Email: [email protected]
Structured data quality reports to improve EHR data quality
International Journal of Medical Informatics 84 (2015) 1094–1098
Contents lists available at ScienceDirect
International Journal of Medical Informatics journal homepage: www.ijmijournal.com
Structured data quality reports to improve EHR data quality Jane Taggart a , Siaw-Teng Liaw a,b,c,∗ , Hairong Yu a a
Centre for Primary Health Care& Equity, UNSW Australia, Sydney, Australia School of Public Health & Community Medicine, UNSW Australia, Sydney, Australia c General Practice Unit, South Western Sydney Local Health District, NSW, Australia b
a r t i c l e
i n f o
Article history: Received 15 April 2015 Received in revised form 14 September 2015 Accepted 30 September 2015 Keywords: Electronic health records Data quality Quality improvement Structured reports Feedback Quality of care
a b s t r a c t Objective: To examine whether a structured data quality report (SDQR) and feedback sessions with practice principals and managers improve the quality of routinely collected data in EHRs. Methods: The intervention was conducted in four general practices participating in the Fairfield neighborhood electronic Practice Based Research Network (ePBRN). Data were extracted from their clinical information systems and summarised as a SDQR to guide feedback to practice principals and managers at 0, 4, 8 and 12 months. Data quality (DQ) metrics included completeness, correctness, consistency and duplication of patient records. Information on data recording practices, data quality improvement, and utility of SDQRs was collected at the feedback sessions at the practices. The main outcome measure was change in the recording of clinical information and level of meeting Royal Australian College of General Practice (RACGP) targets. Results: Birth date was 100% and gender 99% complete at baseline and maintained. DQ of all variables measured improved significantly (p < 0.01) over 12 months, but was not sufficient to comply with RACGP standards. Improvement was greatest with allergies. There was no significant change in duplicate records. Conclusions: SDQRs and feedback sessions support general practitioners and practice managers to focus on improving the recording of patient information. However, improved practice DQ, was not sufficient to meet RACGP targets. Randomised controlled studies are required to evaluate strategies to improve data quality and any associated improved safety and quality of care. © 2015 Elsevier Ireland Ltd. All rights reserved.
1. Introduction Current Australian and international health policies emphasise the importance of electronic health records (EHRs), information sharing and the smart use of data, information and communication technologies in improving the coordination, quality and efficiency of healthcare through the secure use and sharing of information [1–3]. The use of EHRs can assist with clinical decision making, improve adherence to guidelines and reduce errors in prescribing and most of the evidence for improvements in quality of care are made in primary and secondary prevention. [4] The routine collection of clinical information from EHRs is increasingly being stored in large data repositories and used for research and quality improvement. Quantifying, understanding and improving the quality of routinely collected clinical information in EHRs is cru-
∗ Correspondence author at: General Practice Unit GPO Box 5, Fairfield, NSW 1650, Australia. Fax: +61 3 96168400. E-mail address: [email protected] (S.-T. Liaw). http://dx.doi.org/10.1016/j.ijmedinf.2015.09.008 1386-5056/© 2015 Elsevier Ireland Ltd. All rights reserved.
cial if they are to support effective clinical care, monitor safety and quality and be useful for audit and research purposes. The Royal Australian College of General Practice (RACGP) Standards for General Practice 4th edition [5] were developed by a National Expert Committee in consultation with key stakeholders in primary care. The indicators, explanations and practical resources for general practices include key elements of a National Accreditation Scheme and e-health initiatives: patient records, health summaries, patient identification, clinical handover, governance and quality use of medicines. Improving the quality of data in EHRs requires the use of consistent coding systems and terminology, the recording of ethnicity and Aboriginal or Torres Strait Islander status, minimum requirements for health summaries such as relevant family and social history and current medications, preventive care (e.g. smoking and blood pressure) and the documentation of consultations so they include the reason for visit, referrals, clinical findings and other information necessary for good decision making, clinical handover and integrated care. The University of New South Wales (UNSW) electronic Practice Based Research Network (ePBRN) extracts and links data from information systems (IS) and electronic health records
J. Taggart et al. / International Journal of Medical Informatics 84 (2015) 1094–1098
(EHR) in general practices, hospitals (ED and admissions) and ambulatory/community health services in the Fairfield health neighborhood in South Western Sydney, Australia. The quality of data and information from participating general practices is routinely examined for completeness, correctness, consistency and duplication of records [6]. Generally, the quality of demographic (except for ethnicity), service, prescribing and investigation data were better than those of clinical measures such as BMI and smoking status. Completeness of records and data varied between the practices. There was a lack of consistent coding rules and standards for data entry within and across the practices. These findings are consistent with other studies on the quality of data in primary care Clinical Information Systems (CIS) [7,8]. A literature review included twelve studies which reported some change in quality of data after feedback, training or audit type interventions. However, it was not clear whether it was the intervention, participation or time that explained the changes in data quality [9]. We report on a study to examine whether a structured data quality report (SDQR) and feedback sessions with practice principals and managers improved data quality of EHRs amongst general practices participating in the Fairfield neighborhood ePBRN.
2. Materials and methods The GRHANITETM extraction and linkage software pseudonymised patient records, using a standard hashing technique [10]. Clinical and managerial data were extracted, encrypted and sent securely to the ePBRN data repository at UNSW. Patient records were linked within and between the practices, allowing the identification of patients with duplicate records or clinical records at more than one practice. For this study, we used data extracted at four time points (baseline, 4, 8 and 12 months) from the four practices, generating the SDQR using an automated process. The SDQRs emphasized data quality metrics for each practice, compared with the previous SDQR and aggregate of all 4 practices, and benchmarked against the RACGP standards. Generating the SDQR: The data repository was deployed by Microsoft SQL Server. Its query output was cross-checked with IBM SPSS Statistics V20. Differences were resolved by discussion and query adjustments (HY, JT, STL). The output tables from the Structured Query Language (SQL) queries were copied across to a report template. Once the SDQRs were produced they were re-checked (JT) for accuracy and queries were rerun if any unusual results were identified (HY). We considered three practice denominators: 1) all patients that had been entered into the CIS; 2) EHR-active patients (patients recorded as active by the practice); and 3) RACGP-active patients (patients who had 3 or more visits in the two years prior to the data extraction). All SDQR information in this paper used RACGP-active patients as the denominator except for duplicated records that used EHR-active as the denominator (this was more useful for the practices in cleaning duplicate records) and included:
1. risk profile of all patients for cardiovascular disease and diabetes (For example, the proportions of patients who’s last BMI recorded is >30, have a systolic BP > 140, have total cholesterol >4 mmol/L or fasting blood glucose ≥7.0 mmol/L; 2. clinical status of diabetes patients; 3. completeness and correctness of patient records for gender, date of birth, Aboriginal and Torres Strait Islander status, smoking, alcohol, height, weight, waist circumference, Body Mass Index (BMI), blood pressure (BP). Country of birth and allergies were included in the third feedback report and the final results; 4. possible duplication of records in the EHR; and 5. number of patients shared with other participating practices.
1095
Completeness was defined as a patient having at least one record for a particular attribute; and correctness as a valid and appropriate record (e.g. height is in metres, centimetres and/or millimetres and is within an appropriate range for age). The number and rates (denominator = RACGP-active patients) of completeness for each of the attributes were provided in the SDQR alongside the RACGP targets for clinical records. For our study we defined ”working towards” the RACGP Standards as showing improvement in recording over time and “routinely recorded” as achieving near 100% recording. (Note: Allergies data were not extracted for the first two timepoints as allergies had not been included in the extraction configuration file at the time of the earlier two extractions. Country of birth was not available for Practice #2 as they did not use the appropriate software.) SDQR feedback sessions: These half-hour sessions between 1 or 2 researchers (JT, STL or OI) and the participating general practitioners and practice managers were recorded, transcribed, coded and thematically analysed using NVivo. The broad area of questioning was whether the data quality was expected, possible reasons, what information was entered into the clinical records and how, enhancements for the SDQR, and strategies to improve data quality and utility. Where relevant, the researcher demonstrated where clinical information could be added into the information system in a more structured and consistent way. Practices were also given the opportunity to re-identify the patients with diabetes and those with possible duplicated patient records. [Note: we have included additional qualitative feedback to add to the richness of the discourse from 6 other practices that joined the ePBRN around the time the pilot practices received their second or third practice SDQR. Participating general practitioners qualified for 40 clinical audit points in the compulsory RACGP Quality Improvement and Continuing Professional Development (QI&CPD) Programme. Australian general practitioners need a total of 130 points for the QI&CPD Triennium, of which at least 40 points must be for a clinical audit activity. Ethics approval was obtained from the South Western Sydney Local Health District and University of New South Wales Human Research Ethics Committees.
3. Results Seven GPs (including practice principals) and two practice managers from the four practices consented to participate in the SDQR feedback study. These practices varied in size with the number of GPs ranging from 4 to 7; the two largest practices employed practice nurses and full time practice managers. The aggregate RACGP-active population for the four ePBRN practices at baseline was 27,042. The size of the RACGP-active populations varied across the practices (Practice #1: 870; Practice #2 4,452; Practice #3: 8, 337; Practice #4:13,303). To demonstrate the implication of using different denominators, RACGP-active patients was 26–30% and EHR-active patients 70–80% of all patients (Table 1). The change in the completeness of records for the combined ePBRN practices is shown in Table 1 and for the individual practices in Figs. 1–7 . At baseline there were high rates of completeness for gender (99%) and date of birth (100%) and relatively high rates for smoking (68%), but the recording of Aboriginal and Torres Strait Islander (44%), alcohol (8%), height (32%), weight (37%), waist circumference (5%) and BMI (17%) were low. Allergies (84%) had a relatively high rate when it was first included in the reports while the recording of country of birth (2%) was low. Apart from 100% for date of birth, all other variables at baseline were below the RACGP targets,
1096
J. Taggart et al. / International Journal of Medical Informatics 84 (2015) 1094–1098
Table 1 Change in completeness and duplicate records over 12 months (ePBRN practices combined). Completeness (%) over time for RACGP active patients
Three possible practice populations for use as denominators
Gender DOB Aboriginal or Torres Strait Islander status Country of birtha Allergies Smoking status Alcohol assessment Height Weight BMI Number of duplicate records
RACGP target (%)
0 months (%)
4 months (%)
8 months (%)
12 months (%)
Change over time (%)
All patients
91702
95056
97358
99750
+8.8
RACGP-active only EHR-active only 100 100 Routinely recorded (100)
27042 (29.5) 73115 (79.7) 99.99 √ 100 44.71
25918 (27.3) 64770 (68.1) 99.9 √ 100 43.2
26559 (27.3) 66993 (68.8) 99.98 √ 100 43.19
26321 (26.4) 69663 (69.8) 99.99 √ 100 41.53
−3.1 −9.9 0.02 0 −3.19
<.001 <.001 NS NS <.001
2.69 84.68 71.66 10.61 35.69 41.69 19.67
5.06 √ 95.70 73.04 11.52 √ 37.45 √ 43.41 √ 20.78
<.001 <.001 <.001 <.001 <.001 <.001 <.001
759 (1.1)
780 (1.1)
2.13 11.02 5.06 1.24 5.34 5.71 3.84 p-value (t-test) 0.9
75 NA NA 90 NA NA 75 68.25 70.8 75 8.48 11.7 Working towards 32.27 34.3 Working towards 37.82 40.4 Working towards 17.12 18.5 Duplicates (%) over time for EHR-active patients 1432 (1.9) 1432 (1.9) NA
p-value for change (t-test)
√ NA: not available; NS: not significant; meets RACGP standards. a Practice #2 not included as not using relevant software.
Fig. 1. Recording trend of Aboriginal or Torres Strait Islander status.
Fig. 4. Recording trend of smoking.
Fig. 5. Recording trend of height. Fig. 2. Recording trend of allergies.
Fig. 6. Recording trend of weight.
Fig. 3. Recording trend of alcohol consumption.
particularly for the recording of alcohol consumption, alcohol assessment, Aboriginal or Torres Strait Islander status, height, weight and BMI. Over the 12-months, the recording of date of birth remained perfect (100%) and gender near perfect (99.99) along with significant positive changes in all other study variables (p < .001). However,
Fig. 7. Recording trend of BMI.
J. Taggart et al. / International Journal of Medical Informatics 84 (2015) 1094–1098
only date of birth (100%) and allergies (95%) met the RACGP targets for all practices at the end of the 12-months (Fig. 2). Smoking was almost at target (73%), but the recording of alcohol assessment and consumption were well below (Figs. 3 and 4). Most practices were working on improving their recording of height (37%), weight (43%) and BMI (21%) (Figs. 5–7). The exception was a negative change for the recording of Aboriginal or Torres Strait Islander(<.001). This highlights the importance of the denominator (all RACGP-active), which changed significantly in the two largest practices (#3 and #4), while the numerator (Aboriginal & Torres Strait Islander) remained relatively constant. Practice #1 was engaged in a cultural respect project at that time and had a significant positive change in the recording of Aboriginal or Torres Strait Islander (p < .001), while Practice #2 had no significant change. While there was a reduction in the proportion of patients with duplicate records it was not significant. Data correctness for all the variables remained high (99–100%) across the study period. Poor data completeness was reported to be due to time or staff constraints, patients declining or failing to provide the information, difficulty with recording information and needing to focus on the reason for the patient’s attendance. “I think it is all about time constraints... if the government gives us income like $150,000 or $200,000 to do the right thing then I think we will do a good job rather than thinking about income and worrying about expenditure and all” General Practitioner “Computerisation . . . does help but it takes the focus off why the patient is here and to get that balance I am still struggling with it and I will still struggle until I finish working as a general practitioner. We are all in the same boat we have to learn how to do it” General Practitioner Recording information as text in progress notes during the consultation, rather than as coded data in a structured field, is preferred by some as they believed that it provided more details and made the records more useful at point of care. “Because if we do the weight we just put it down in the notes. . .. . .. I think the way we approach things is that we type notes for us to understand so we don’t really think about the other things.” General Practitioner Practicality and perceived utility of information was another factor determining if a piece of information was collected/measured or documented. One GP reported discomfort in measuring waist circumference in very obese patients, describing the process as “like I was cuddling the patient”. Another was alcohol assessment when the patient was not forthcoming with information on their drinking habits or did not seem motivated to change if risk reduction counselling was to be provided. General practitioners and practice managers were proactive about implementing change in response to the SDQR and feedback, initiating discussions about strategies to improve and achieve goals such as RACGP standards for completeness of records. Benchmarking against their peers was a motivator for quality improvement, especially when their performance was lower than the average for the ePBRN. “It is good to know that we are doing the similar things and we are using the tools the right way”. General Practitioner “It is still lower than other practices. So maybe we need to improve on that one”. General Practitioner On the other hand, where performances were low but similar to the ePBRN average, they felt it was a common problem amongst the practices and that improvements in those “gaps” would require a
1097
broader and more generic approach such as better education about the quality of EHRs or remuneration for GPs to compensate for time constraints during clinical consultations. The practice principals were interested in getting other GPs and the nurses in the practice to improve data completeness. “We will try to complete all the records like smoking.. . ..For me that is a big challenge. . ..It goes back to us about how to educate the other GPs”. General Practitioner “But we can ask for weight and height, and may be waist,. . . get the nurse to do it” General Practitioner
4. Discussion The recording of risk factor information improved significantly with the intervention, but it was not sufficient to meet the RACGP targets. The negative trend found with the recording of Aboriginal and Torres Strait Islander status reflected a changing RACGP-active practice population as well as multiple factors such as staff inertia and patients not volunteering information. GPs confirmed they do not routinely collect this information as they believe they know their patients and only ask those who they thought were Aboriginal or Torres Strait Islander. This complex multifactorial issue involving vulnerable populations may require a different approach to improving data quality. The SDQRs and feedback sessions encouraged the GPs and practice managers to reflect and consider strategies to improve completeness, and perhaps, other dimensions of data quality. Peercomparisons motivated the competitive GPs and practice managers to improve their performance in relation to other general practices. It is important to note that these improvements occurred with the involvement of the practice principals (in all practices) and with the practice manager (in two practices)—without all practice staff participating. The practice principal appears to be the driver in leading the practice team to improve data quality. The variations between practices for completeness in each of the attributes require further investigation but could be due to a number of reasons. For example, the priority the practice principal decides to focus on, engagement of the practice manager, size of practice and staff turnover particularly of general practitioners, registrars and nurses. The RACGP Standards for General Practice state that general practices need to demonstrate they are “working towards” the recording of preventive care (includes height, weight and BMI) and “routinely recording” Aboriginal and Torres Strait Islander status. For the purposes of our study we defined “working towards” as improving recording over time and “routinely recorded” as aiming for 100%. The RACGP could consider including specific targets that are more useful and measurable so that practices have a clearer understanding of what they should be aiming to achieve in order to provide quality care. The improvement in data quality without meeting the RACGP targets suggests that these targets may be unrealistic. Taken together, these findings suggest that a multi-pronged and ecological approach across the data production cycle is required to improve the quality of data in EHRs. Limitations of the study: The lack of a control group made it difficult to suggest a causal relationship or exclude other causal factors. We may not have excluded other activities that may have improved data quality. For instance, the practices also had another Clinical Audit Tool installed by the Medicare Local (provides support to general practices). The practices only used this tool occasionally to generate reports for purposes other than improving the recording of clinical data. For example, providing data to the Medicare Local and monitoring diabetes patients for the annual
1098
J. Taggart et al. / International Journal of Medical Informatics 84 (2015) 1094–1098
Summary Points What is known about the topic • Improving the quality of routinely collected clinical data is important to support effective care and research. • There is variation in the quality of clinical data with demographic, prescribing and investigation data being generally better than clinical measures. • Feedback, training and audits have been reported in the literature to improve data quality but it is unclear what actually explains the change. What this study adds to our knowledge • Significant improvements can be made in the recording of risk factors but not indigenous status after Structured Data Quality Report sessions. • Peer comparisons can motivate change, particularly where practice data quality was below the level of peers. • The findings suggest a multi-pronged and ecological approach across the data production cycle is required to improve data quality.
cycle of care. The Medicare Local Practice Support Officers were also aware of this study and may have influenced it unwittingly. 5. Conclusion SDQRs with benchmarks and peer-comparisons and feedback sessions with practice principals and practice managers support general practitioners and practice managers to focus on improving the recording of patient information. However, improvements did not result in compliance to RACGP targets. It may be that the targets are unrealistic, but further mixed methods studies are needed to understand the reasons why in various contexts, target populations and practice organization models these are not met. Randomised controlled studies are required to evaluate strategies to improve data quality and any associated improvements in safety and quality of care. Conflict of interest No conflict of interest Contribution of authors Jane Taggart contributed to the design of the study, the collection and interpretation of extracted and qualitative data, the
drafting and revising of the paper and final approval for submission. Siaw-Teng Liaw contributed to the conception and design of the study, the interpretation of extracted and qualitative data, drafting and revising the paper and final approval for submission. Hairong Yu contributed to the management, manipulation and interpretation of the extracted data and revising the paper and final approval for submission. Acknowledgements All participating general practices who contributed their data and perspectives and Oluyemisi Ijamkinwa (OI) who assisted with the feedback sessions and data synthesis. References [1] Australian Government Department of Health. Primary Health Care reform in Australia. Report to support Australia’s first National Primary Health Care Strategy. 2009 ISBN: 1-74186-936-6. [2] National Health & Hospital Commission. A Healthier Future For All Australians—Final Report of the National Health and Hospitals Reform Commission. Commonwealth of Australia, Department of Health and Ageing, 2009. [3] D. Blumenthal, Wiring the Health System—origins and provisions of a new federal program, New Engl. J. Med. 365 (24) (2011) 2323–2329, PubMed PMID.:22168647. [4] B. Chaudhry, J. Wang, S. Wu, M. Maglione, W. Mojica, E. Roth, et al., Systematic review: impact of health information technology on quality, efficiency, and costs of medical care, Ann. Intern. Med. 144 (10) (2006) 742–752. [5] Royal Australian College of General Practitioners. Standards for General Practices (4th Edition) 2013. http://www.racgp.org.au/download/documents/ Standards/standards4thedition.pdf. [6] S.T. Liaw, J. Taggart, S. Dennis, A. Yeo, Data quality and fitness for purpose of routinely collected data—a general practice case study from an electronic Practice-Based Research Network (ePBRN), AMIA Annual Symposium Proc. (2011) 785–794. [7] S.K.K. de Lusignan, J. Belsey, A. Hattersley, J. vanVlymen, H. Gallagher, et al., A method of identifying and correcting miscoding, misclassification and misdiagnosis in diabetes: a pilot and validation study of routinely collected data, Diabet. Med. 27 (2010) 203–209. [8] K. Thiru, A. Hassey, F. Sullivan, Systematic review of scope and quality of electronic patient record data in primary care, BMJ (2003), 2003-05-15 00:00:00;326(7398):1070. [9] H. Brouwer, P. Bindels, H. Weert, Data quality improvement in general practice, Fam. Pract. 23 (October 2006 (5)) (2006) 529–536. [10] D.I.R. Boyle, S.T. Liaw, A. Crowden, GRHANITETM : generic software demonstrating advanced security, ethical consent and confidentiality processes for clinical data sharing, audit and research In The Australasian Bioethics Association/Australian and New Zealand Institute of Health Law and Ethics, 2007.
Contents lists available at ScienceDirect
International Journal of Medical Informatics journal homepage: www.ijmijournal.com
Structured data quality reports to improve EHR data quality Jane Taggart a , Siaw-Teng Liaw a,b,c,∗ , Hairong Yu a a
Centre for Primary Health Care& Equity, UNSW Australia, Sydney, Australia School of Public Health & Community Medicine, UNSW Australia, Sydney, Australia c General Practice Unit, South Western Sydney Local Health District, NSW, Australia b
a r t i c l e
i n f o
Article history: Received 15 April 2015 Received in revised form 14 September 2015 Accepted 30 September 2015 Keywords: Electronic health records Data quality Quality improvement Structured reports Feedback Quality of care
a b s t r a c t Objective: To examine whether a structured data quality report (SDQR) and feedback sessions with practice principals and managers improve the quality of routinely collected data in EHRs. Methods: The intervention was conducted in four general practices participating in the Fairfield neighborhood electronic Practice Based Research Network (ePBRN). Data were extracted from their clinical information systems and summarised as a SDQR to guide feedback to practice principals and managers at 0, 4, 8 and 12 months. Data quality (DQ) metrics included completeness, correctness, consistency and duplication of patient records. Information on data recording practices, data quality improvement, and utility of SDQRs was collected at the feedback sessions at the practices. The main outcome measure was change in the recording of clinical information and level of meeting Royal Australian College of General Practice (RACGP) targets. Results: Birth date was 100% and gender 99% complete at baseline and maintained. DQ of all variables measured improved significantly (p < 0.01) over 12 months, but was not sufficient to comply with RACGP standards. Improvement was greatest with allergies. There was no significant change in duplicate records. Conclusions: SDQRs and feedback sessions support general practitioners and practice managers to focus on improving the recording of patient information. However, improved practice DQ, was not sufficient to meet RACGP targets. Randomised controlled studies are required to evaluate strategies to improve data quality and any associated improved safety and quality of care. © 2015 Elsevier Ireland Ltd. All rights reserved.
1. Introduction Current Australian and international health policies emphasise the importance of electronic health records (EHRs), information sharing and the smart use of data, information and communication technologies in improving the coordination, quality and efficiency of healthcare through the secure use and sharing of information [1–3]. The use of EHRs can assist with clinical decision making, improve adherence to guidelines and reduce errors in prescribing and most of the evidence for improvements in quality of care are made in primary and secondary prevention. [4] The routine collection of clinical information from EHRs is increasingly being stored in large data repositories and used for research and quality improvement. Quantifying, understanding and improving the quality of routinely collected clinical information in EHRs is cru-
∗ Correspondence author at: General Practice Unit GPO Box 5, Fairfield, NSW 1650, Australia. Fax: +61 3 96168400. E-mail address: [email protected] (S.-T. Liaw). http://dx.doi.org/10.1016/j.ijmedinf.2015.09.008 1386-5056/© 2015 Elsevier Ireland Ltd. All rights reserved.
cial if they are to support effective clinical care, monitor safety and quality and be useful for audit and research purposes. The Royal Australian College of General Practice (RACGP) Standards for General Practice 4th edition [5] were developed by a National Expert Committee in consultation with key stakeholders in primary care. The indicators, explanations and practical resources for general practices include key elements of a National Accreditation Scheme and e-health initiatives: patient records, health summaries, patient identification, clinical handover, governance and quality use of medicines. Improving the quality of data in EHRs requires the use of consistent coding systems and terminology, the recording of ethnicity and Aboriginal or Torres Strait Islander status, minimum requirements for health summaries such as relevant family and social history and current medications, preventive care (e.g. smoking and blood pressure) and the documentation of consultations so they include the reason for visit, referrals, clinical findings and other information necessary for good decision making, clinical handover and integrated care. The University of New South Wales (UNSW) electronic Practice Based Research Network (ePBRN) extracts and links data from information systems (IS) and electronic health records
J. Taggart et al. / International Journal of Medical Informatics 84 (2015) 1094–1098
(EHR) in general practices, hospitals (ED and admissions) and ambulatory/community health services in the Fairfield health neighborhood in South Western Sydney, Australia. The quality of data and information from participating general practices is routinely examined for completeness, correctness, consistency and duplication of records [6]. Generally, the quality of demographic (except for ethnicity), service, prescribing and investigation data were better than those of clinical measures such as BMI and smoking status. Completeness of records and data varied between the practices. There was a lack of consistent coding rules and standards for data entry within and across the practices. These findings are consistent with other studies on the quality of data in primary care Clinical Information Systems (CIS) [7,8]. A literature review included twelve studies which reported some change in quality of data after feedback, training or audit type interventions. However, it was not clear whether it was the intervention, participation or time that explained the changes in data quality [9]. We report on a study to examine whether a structured data quality report (SDQR) and feedback sessions with practice principals and managers improved data quality of EHRs amongst general practices participating in the Fairfield neighborhood ePBRN.
2. Materials and methods The GRHANITETM extraction and linkage software pseudonymised patient records, using a standard hashing technique [10]. Clinical and managerial data were extracted, encrypted and sent securely to the ePBRN data repository at UNSW. Patient records were linked within and between the practices, allowing the identification of patients with duplicate records or clinical records at more than one practice. For this study, we used data extracted at four time points (baseline, 4, 8 and 12 months) from the four practices, generating the SDQR using an automated process. The SDQRs emphasized data quality metrics for each practice, compared with the previous SDQR and aggregate of all 4 practices, and benchmarked against the RACGP standards. Generating the SDQR: The data repository was deployed by Microsoft SQL Server. Its query output was cross-checked with IBM SPSS Statistics V20. Differences were resolved by discussion and query adjustments (HY, JT, STL). The output tables from the Structured Query Language (SQL) queries were copied across to a report template. Once the SDQRs were produced they were re-checked (JT) for accuracy and queries were rerun if any unusual results were identified (HY). We considered three practice denominators: 1) all patients that had been entered into the CIS; 2) EHR-active patients (patients recorded as active by the practice); and 3) RACGP-active patients (patients who had 3 or more visits in the two years prior to the data extraction). All SDQR information in this paper used RACGP-active patients as the denominator except for duplicated records that used EHR-active as the denominator (this was more useful for the practices in cleaning duplicate records) and included:
1. risk profile of all patients for cardiovascular disease and diabetes (For example, the proportions of patients who’s last BMI recorded is >30, have a systolic BP > 140, have total cholesterol >4 mmol/L or fasting blood glucose ≥7.0 mmol/L; 2. clinical status of diabetes patients; 3. completeness and correctness of patient records for gender, date of birth, Aboriginal and Torres Strait Islander status, smoking, alcohol, height, weight, waist circumference, Body Mass Index (BMI), blood pressure (BP). Country of birth and allergies were included in the third feedback report and the final results; 4. possible duplication of records in the EHR; and 5. number of patients shared with other participating practices.
1095
Completeness was defined as a patient having at least one record for a particular attribute; and correctness as a valid and appropriate record (e.g. height is in metres, centimetres and/or millimetres and is within an appropriate range for age). The number and rates (denominator = RACGP-active patients) of completeness for each of the attributes were provided in the SDQR alongside the RACGP targets for clinical records. For our study we defined ”working towards” the RACGP Standards as showing improvement in recording over time and “routinely recorded” as achieving near 100% recording. (Note: Allergies data were not extracted for the first two timepoints as allergies had not been included in the extraction configuration file at the time of the earlier two extractions. Country of birth was not available for Practice #2 as they did not use the appropriate software.) SDQR feedback sessions: These half-hour sessions between 1 or 2 researchers (JT, STL or OI) and the participating general practitioners and practice managers were recorded, transcribed, coded and thematically analysed using NVivo. The broad area of questioning was whether the data quality was expected, possible reasons, what information was entered into the clinical records and how, enhancements for the SDQR, and strategies to improve data quality and utility. Where relevant, the researcher demonstrated where clinical information could be added into the information system in a more structured and consistent way. Practices were also given the opportunity to re-identify the patients with diabetes and those with possible duplicated patient records. [Note: we have included additional qualitative feedback to add to the richness of the discourse from 6 other practices that joined the ePBRN around the time the pilot practices received their second or third practice SDQR. Participating general practitioners qualified for 40 clinical audit points in the compulsory RACGP Quality Improvement and Continuing Professional Development (QI&CPD) Programme. Australian general practitioners need a total of 130 points for the QI&CPD Triennium, of which at least 40 points must be for a clinical audit activity. Ethics approval was obtained from the South Western Sydney Local Health District and University of New South Wales Human Research Ethics Committees.
3. Results Seven GPs (including practice principals) and two practice managers from the four practices consented to participate in the SDQR feedback study. These practices varied in size with the number of GPs ranging from 4 to 7; the two largest practices employed practice nurses and full time practice managers. The aggregate RACGP-active population for the four ePBRN practices at baseline was 27,042. The size of the RACGP-active populations varied across the practices (Practice #1: 870; Practice #2 4,452; Practice #3: 8, 337; Practice #4:13,303). To demonstrate the implication of using different denominators, RACGP-active patients was 26–30% and EHR-active patients 70–80% of all patients (Table 1). The change in the completeness of records for the combined ePBRN practices is shown in Table 1 and for the individual practices in Figs. 1–7 . At baseline there were high rates of completeness for gender (99%) and date of birth (100%) and relatively high rates for smoking (68%), but the recording of Aboriginal and Torres Strait Islander (44%), alcohol (8%), height (32%), weight (37%), waist circumference (5%) and BMI (17%) were low. Allergies (84%) had a relatively high rate when it was first included in the reports while the recording of country of birth (2%) was low. Apart from 100% for date of birth, all other variables at baseline were below the RACGP targets,
1096
J. Taggart et al. / International Journal of Medical Informatics 84 (2015) 1094–1098
Table 1 Change in completeness and duplicate records over 12 months (ePBRN practices combined). Completeness (%) over time for RACGP active patients
Three possible practice populations for use as denominators
Gender DOB Aboriginal or Torres Strait Islander status Country of birtha Allergies Smoking status Alcohol assessment Height Weight BMI Number of duplicate records
RACGP target (%)
0 months (%)
4 months (%)
8 months (%)
12 months (%)
Change over time (%)
All patients
91702
95056
97358
99750
+8.8
RACGP-active only EHR-active only 100 100 Routinely recorded (100)
27042 (29.5) 73115 (79.7) 99.99 √ 100 44.71
25918 (27.3) 64770 (68.1) 99.9 √ 100 43.2
26559 (27.3) 66993 (68.8) 99.98 √ 100 43.19
26321 (26.4) 69663 (69.8) 99.99 √ 100 41.53
−3.1 −9.9 0.02 0 −3.19
<.001 <.001 NS NS <.001
2.69 84.68 71.66 10.61 35.69 41.69 19.67
5.06 √ 95.70 73.04 11.52 √ 37.45 √ 43.41 √ 20.78
<.001 <.001 <.001 <.001 <.001 <.001 <.001
759 (1.1)
780 (1.1)
2.13 11.02 5.06 1.24 5.34 5.71 3.84 p-value (t-test) 0.9
75 NA NA 90 NA NA 75 68.25 70.8 75 8.48 11.7 Working towards 32.27 34.3 Working towards 37.82 40.4 Working towards 17.12 18.5 Duplicates (%) over time for EHR-active patients 1432 (1.9) 1432 (1.9) NA
p-value for change (t-test)
√ NA: not available; NS: not significant; meets RACGP standards. a Practice #2 not included as not using relevant software.
Fig. 1. Recording trend of Aboriginal or Torres Strait Islander status.
Fig. 4. Recording trend of smoking.
Fig. 5. Recording trend of height. Fig. 2. Recording trend of allergies.
Fig. 6. Recording trend of weight.
Fig. 3. Recording trend of alcohol consumption.
particularly for the recording of alcohol consumption, alcohol assessment, Aboriginal or Torres Strait Islander status, height, weight and BMI. Over the 12-months, the recording of date of birth remained perfect (100%) and gender near perfect (99.99) along with significant positive changes in all other study variables (p < .001). However,
Fig. 7. Recording trend of BMI.
J. Taggart et al. / International Journal of Medical Informatics 84 (2015) 1094–1098
only date of birth (100%) and allergies (95%) met the RACGP targets for all practices at the end of the 12-months (Fig. 2). Smoking was almost at target (73%), but the recording of alcohol assessment and consumption were well below (Figs. 3 and 4). Most practices were working on improving their recording of height (37%), weight (43%) and BMI (21%) (Figs. 5–7). The exception was a negative change for the recording of Aboriginal or Torres Strait Islander(<.001). This highlights the importance of the denominator (all RACGP-active), which changed significantly in the two largest practices (#3 and #4), while the numerator (Aboriginal & Torres Strait Islander) remained relatively constant. Practice #1 was engaged in a cultural respect project at that time and had a significant positive change in the recording of Aboriginal or Torres Strait Islander (p < .001), while Practice #2 had no significant change. While there was a reduction in the proportion of patients with duplicate records it was not significant. Data correctness for all the variables remained high (99–100%) across the study period. Poor data completeness was reported to be due to time or staff constraints, patients declining or failing to provide the information, difficulty with recording information and needing to focus on the reason for the patient’s attendance. “I think it is all about time constraints... if the government gives us income like $150,000 or $200,000 to do the right thing then I think we will do a good job rather than thinking about income and worrying about expenditure and all” General Practitioner “Computerisation . . . does help but it takes the focus off why the patient is here and to get that balance I am still struggling with it and I will still struggle until I finish working as a general practitioner. We are all in the same boat we have to learn how to do it” General Practitioner Recording information as text in progress notes during the consultation, rather than as coded data in a structured field, is preferred by some as they believed that it provided more details and made the records more useful at point of care. “Because if we do the weight we just put it down in the notes. . .. . .. I think the way we approach things is that we type notes for us to understand so we don’t really think about the other things.” General Practitioner Practicality and perceived utility of information was another factor determining if a piece of information was collected/measured or documented. One GP reported discomfort in measuring waist circumference in very obese patients, describing the process as “like I was cuddling the patient”. Another was alcohol assessment when the patient was not forthcoming with information on their drinking habits or did not seem motivated to change if risk reduction counselling was to be provided. General practitioners and practice managers were proactive about implementing change in response to the SDQR and feedback, initiating discussions about strategies to improve and achieve goals such as RACGP standards for completeness of records. Benchmarking against their peers was a motivator for quality improvement, especially when their performance was lower than the average for the ePBRN. “It is good to know that we are doing the similar things and we are using the tools the right way”. General Practitioner “It is still lower than other practices. So maybe we need to improve on that one”. General Practitioner On the other hand, where performances were low but similar to the ePBRN average, they felt it was a common problem amongst the practices and that improvements in those “gaps” would require a
1097
broader and more generic approach such as better education about the quality of EHRs or remuneration for GPs to compensate for time constraints during clinical consultations. The practice principals were interested in getting other GPs and the nurses in the practice to improve data completeness. “We will try to complete all the records like smoking.. . ..For me that is a big challenge. . ..It goes back to us about how to educate the other GPs”. General Practitioner “But we can ask for weight and height, and may be waist,. . . get the nurse to do it” General Practitioner
4. Discussion The recording of risk factor information improved significantly with the intervention, but it was not sufficient to meet the RACGP targets. The negative trend found with the recording of Aboriginal and Torres Strait Islander status reflected a changing RACGP-active practice population as well as multiple factors such as staff inertia and patients not volunteering information. GPs confirmed they do not routinely collect this information as they believe they know their patients and only ask those who they thought were Aboriginal or Torres Strait Islander. This complex multifactorial issue involving vulnerable populations may require a different approach to improving data quality. The SDQRs and feedback sessions encouraged the GPs and practice managers to reflect and consider strategies to improve completeness, and perhaps, other dimensions of data quality. Peercomparisons motivated the competitive GPs and practice managers to improve their performance in relation to other general practices. It is important to note that these improvements occurred with the involvement of the practice principals (in all practices) and with the practice manager (in two practices)—without all practice staff participating. The practice principal appears to be the driver in leading the practice team to improve data quality. The variations between practices for completeness in each of the attributes require further investigation but could be due to a number of reasons. For example, the priority the practice principal decides to focus on, engagement of the practice manager, size of practice and staff turnover particularly of general practitioners, registrars and nurses. The RACGP Standards for General Practice state that general practices need to demonstrate they are “working towards” the recording of preventive care (includes height, weight and BMI) and “routinely recording” Aboriginal and Torres Strait Islander status. For the purposes of our study we defined “working towards” as improving recording over time and “routinely recorded” as aiming for 100%. The RACGP could consider including specific targets that are more useful and measurable so that practices have a clearer understanding of what they should be aiming to achieve in order to provide quality care. The improvement in data quality without meeting the RACGP targets suggests that these targets may be unrealistic. Taken together, these findings suggest that a multi-pronged and ecological approach across the data production cycle is required to improve the quality of data in EHRs. Limitations of the study: The lack of a control group made it difficult to suggest a causal relationship or exclude other causal factors. We may not have excluded other activities that may have improved data quality. For instance, the practices also had another Clinical Audit Tool installed by the Medicare Local (provides support to general practices). The practices only used this tool occasionally to generate reports for purposes other than improving the recording of clinical data. For example, providing data to the Medicare Local and monitoring diabetes patients for the annual
1098
J. Taggart et al. / International Journal of Medical Informatics 84 (2015) 1094–1098
Summary Points What is known about the topic • Improving the quality of routinely collected clinical data is important to support effective care and research. • There is variation in the quality of clinical data with demographic, prescribing and investigation data being generally better than clinical measures. • Feedback, training and audits have been reported in the literature to improve data quality but it is unclear what actually explains the change. What this study adds to our knowledge • Significant improvements can be made in the recording of risk factors but not indigenous status after Structured Data Quality Report sessions. • Peer comparisons can motivate change, particularly where practice data quality was below the level of peers. • The findings suggest a multi-pronged and ecological approach across the data production cycle is required to improve data quality.
cycle of care. The Medicare Local Practice Support Officers were also aware of this study and may have influenced it unwittingly. 5. Conclusion SDQRs with benchmarks and peer-comparisons and feedback sessions with practice principals and practice managers support general practitioners and practice managers to focus on improving the recording of patient information. However, improvements did not result in compliance to RACGP targets. It may be that the targets are unrealistic, but further mixed methods studies are needed to understand the reasons why in various contexts, target populations and practice organization models these are not met. Randomised controlled studies are required to evaluate strategies to improve data quality and any associated improvements in safety and quality of care. Conflict of interest No conflict of interest Contribution of authors Jane Taggart contributed to the design of the study, the collection and interpretation of extracted and qualitative data, the
drafting and revising of the paper and final approval for submission. Siaw-Teng Liaw contributed to the conception and design of the study, the interpretation of extracted and qualitative data, drafting and revising the paper and final approval for submission. Hairong Yu contributed to the management, manipulation and interpretation of the extracted data and revising the paper and final approval for submission. Acknowledgements All participating general practices who contributed their data and perspectives and Oluyemisi Ijamkinwa (OI) who assisted with the feedback sessions and data synthesis. References [1] Australian Government Department of Health. Primary Health Care reform in Australia. Report to support Australia’s first National Primary Health Care Strategy. 2009 ISBN: 1-74186-936-6. [2] National Health & Hospital Commission. A Healthier Future For All Australians—Final Report of the National Health and Hospitals Reform Commission. Commonwealth of Australia, Department of Health and Ageing, 2009. [3] D. Blumenthal, Wiring the Health System—origins and provisions of a new federal program, New Engl. J. Med. 365 (24) (2011) 2323–2329, PubMed PMID.:22168647. [4] B. Chaudhry, J. Wang, S. Wu, M. Maglione, W. Mojica, E. Roth, et al., Systematic review: impact of health information technology on quality, efficiency, and costs of medical care, Ann. Intern. Med. 144 (10) (2006) 742–752. [5] Royal Australian College of General Practitioners. Standards for General Practices (4th Edition) 2013. http://www.racgp.org.au/download/documents/ Standards/standards4thedition.pdf. [6] S.T. Liaw, J. Taggart, S. Dennis, A. Yeo, Data quality and fitness for purpose of routinely collected data—a general practice case study from an electronic Practice-Based Research Network (ePBRN), AMIA Annual Symposium Proc. (2011) 785–794. [7] S.K.K. de Lusignan, J. Belsey, A. Hattersley, J. vanVlymen, H. Gallagher, et al., A method of identifying and correcting miscoding, misclassification and misdiagnosis in diabetes: a pilot and validation study of routinely collected data, Diabet. Med. 27 (2010) 203–209. [8] K. Thiru, A. Hassey, F. Sullivan, Systematic review of scope and quality of electronic patient record data in primary care, BMJ (2003), 2003-05-15 00:00:00;326(7398):1070. [9] H. Brouwer, P. Bindels, H. Weert, Data quality improvement in general practice, Fam. Pract. 23 (October 2006 (5)) (2006) 529–536. [10] D.I.R. Boyle, S.T. Liaw, A. Crowden, GRHANITETM : generic software demonstrating advanced security, ethical consent and confidentiality processes for clinical data sharing, audit and research In The Australasian Bioethics Association/Australian and New Zealand Institute of Health Law and Ethics, 2007.