- Email: [email protected]
Audit of voice recognition system report accuracy
Abstract / Clinical Radiology 69 (2014) S11eS22
Methodology: A prospective audit was undertaken over a 6-week period across the department. Operators were unaware and were assumed to be cognisant of the 3 patient point ID check procedure from local training. Results of 1st audit round: A total of 154 procedures were audited from various imaging modalities and procedures were performed by different operators. Only 47/154 procedures had a full 3-point identification performed prior to commencement of examination. 1st action plan: Regular mandatory training of all staff and a mandatory 10 second time out pre procedure led by the operator was proposed. Results of 2nd round: A total of 178 procedures from various modalities and different level of operators were audited. 158/178 procedures had a full 3-point identification performed prior to commencement of examination. 2nd action plan: Although a positive improvement has been identified, further re-iteration is needed of the importance of the check prior to commencement of any radiological procedure. Audit of the proportion of CTs informally double reported at West Suffolk Hospital
S19
Grade given Correct Descriptor used DEXA recommendation Methodology: Reporting of vertebral fractures in our institution was evaluated for the following examinations performed on females over the age of 45 years, against the set standard. 406 thoracic and lumbar plain films 163 thorax/abdomen/pelvis CTs Results of 1st audit round: The vertebral fracture identification rate was 94.2% on plain film but only 13.5% on CT. However when fractures were identified, there was generally consistent use of the word ’fracture’ rather than ’wedging’ or ’compression’ in both modalities. 1st action plan: The enormous discrepancy in CT implies that radiologists are not routinely using sagittal reconstructions to evaluate the spine. The findings were presented in a local audit meeting to encourage the use of 3D reconstruction capability of CT to actively look for spinal fractures even on general CT scans. An audit of voice recognition errors in neuroradiology reports
Authors: Zosia Rodak*, E.R. Darrah Background: There is increasing recognition that double reporting, particularly of cross-sectional imaging is important for governance and the RCR has recognised its value for revalidation evidence. The literature and current guidance relates to “formal, blinded double reporting”, but a significant amount of more informal double reporting occurs in all radiology departments, as second opinions, MDT review etc. This audit is to quantify this and assess its potential value. Standard, indicator and target: Independent providers of radiology services double report 5-10% of scans as part of standard governance and the RCR includes double reporting in its toolkit for revalidation. Methodology: From 19/03/12 - 25/03/12 and 22/04/13 - 28/04/13, all radiologists at West Suffolk Hospital recorded when they double reported a CT, the reason for doing so, and if any change was noted. This was compared to the total number of CTs. Results of 1st audit round: 82 of 247 of the scans were double reported. 86% were reviewed in MDTs. 3.6% of scans were reported differently. 1st action plan: As a large proportion of scans are already being doublereported informally day-to-day, this should continue. The plan is to formalise this into a report for our governance meeting and to make it an annual process that can feed into ISAS outcomes. Results of 2nd round: 74 of 287 (25.7%) of the scans were double reported. 85% were reviewed in MDTs. 2.7% of scans were reported differently. 2nd action plan: To continue with this as an annual process to contribute to ISAS outcomes.
Authors: Mark Igra*, Stuart Currie, Jeremy Macmullen-Price, Daniel J. Warren, Ian Craven Background: Voice recognition (VR) use has increased since the 1980s. This increase has lead to more errors in reports.1,2 The Royal College of Radiologists states that wording of reports should be clear.3 Errors can have significant clinical consequences if not corrected promptly.4 Errors are more likely in noisy departments, those with high workloads or if English is not the reporter’s first language. Standard, indicator and target: Our aims were to determine the frequency and type of errors and whether errors affected report quality. 100% of reports should be free of errors affecting report interpretation. Image interpretation was not assessed. Methodology: 300 reports from 6 consultants (50 consecutive each) were analysed. Errors were categorized as wrong word substitutions, nonsense phrases, deletions, insertions or punctuation errors. Whether the error affected report interpretation was determined. Results of 1st audit round: Average words per report varied from 62 to 190, and average words per error was 929. Total number of individual errors per consultant varied from 2 to 37. 88% of reports were completely error free. 99.4% of reports were free of significant errors. 1st action plan: Data was presented to the department with advice on reducing grammatical errors. Re-audit will take place annually to ensure we are maintaining or improving standards. VR is an effective reporting tool but checking reports prior to verifying remains essential. Reminders on reporting software to check may help further reduce errors.
Standards of spinal fractures reports in plain radiograph and CT Audit of voice recognition system report accuracy Authors: Ayano Tachibana*, H’ssein Al-Chalabi, Clare Groves Authors: Ai-Lee Chang*, Dorothy Keane, Lance Cope Background: Osteoporotic vertebral fractures are associated with significant risk of further spinal and hip fractures, with subsequent increase in morbidity and mortality. Secondary prevention strategies are dependent on accurate reporting of spinal fractures. The International Osteoporosis Foundation recommends the use of clear, unambiguous terminology using the word ‘fracture’, and grading the fracture, based on the semi-quantitative system outlined by Genant and Wu (1993). Standard, indicator and target: Standard of 100% was set for criteria including:Fractures identified Term “Fracture” used if identified
* * * *
Guarantor Guarantor Guarantor Guarantor
and and and and
correspondent: correspondent: correspondent: correspondent:
Zosia Rodak Ayano Tachibana Mark Igra Ai-Lee Chang
Background: Audit to assess errors in voice recognition system (VRS) generated reports due to failure of software to correctly identify words and failure of user to spot error in text. Standard, indicator and target: VRS generated reports should be accurate and clear. Overall error rate <5%VRS reports. There should be no major errors where the report makes no sense, target 0%. Indicator:% of reports containing an error. Methodology: 50CR and 20CT consecutive reports for each VRS user Error classified as minor, moderate, major Clinical details and body of report reviewed VRS users: CT6 CR8.
S20
Abstract / Clinical Radiology 69 (2014) S11eS22
Results of 1st audit round: CT- avg 27%, median 15%, range 0-90% CR- avg 11%, median 8%, range 2-38% 1 CR report made no sense. Targets not met. Types of errors - similar sounding words, lack of punctuations, words cut off, extra words. 1st action plan: 1. Presented at departmental and hospital audit meeting 2. Recommend RIS that would support auto-transcription of ’clinical details’ 3. Feedback to reporter 4. Repeat audit in 6 months Results of 2nd round: March - April 2014. CTe avg 23%; median 5%; range 0-100% CReavg18%; median 8%; range 2e84% Targets not met. Reporter A continues to have high VR error rate. Other reporters have good improvement. Overall improvement in CT errors. CR errors have deteriorated. Errors are mainly of voice recognition and punctuation. Majority of errors are mild and in clinical details. 2nd action plan:1. Present at departmental audit meeting 2. Urgent recommendation of RIS that would support auto-transcription of ‘clinical details’ 3. Feedback to reporter
Standard, indicator and target: 100% of patients with unexpected findings should either: Have their report faxed to the referring clinician or Be added to a relevant MDT meeting for discussion. Methodology: Data Set: All emails regarding unexpected findings that were sent to radiology secretaries during a one month period (November 2013) were retrieved from the radiology information system. These emails may request for the report to be urgently faxed to the referring clinician OR request that the patient is added to the relevant MDT meeting. Hospital records were then analysed to ensure these patients had been followed up as appropriate. Results of 1st audit round: Of the email requests for a report to be faxed, 94% (50/53) had records confirming that a fax had been sent. Average time lag ¼ 3 days Of the email requests for MDT discussion, 90% (43/48) of patient records confirmed they were discussed at MDT. Average time lag ¼ 7 days All remaining patients were accounted for and followed up. 1st action plan: 1. Standardise the format of email requests and of the electronic record showing the request was actioned 2. Departmental meeting to suggest a more robust and failsafe method of communicating urgent findings to clinicians
Radiology registrar on-call reporting accuracy
Patient satisfaction with day case provision of interventional radiology procedures
Authors: Victoria Chan*, Heiko Parvizi, James Briggs
Peschl, Matthew
Smedley, Nassim Authors: Jessie Gil*, James H. Briggs, Brenda Shanahan, Mark Bratby, Raman Uberoi
Background: At our Trust, on- call registrars (ST3+) issue verified reports, subsequently checked by consultants within 24 hours. Clinical teams should be contacted when there are discrepancies to allow for changes in management. Standard, indicator and target: An RCR AuditLive template proposed a 2-5% error rate for on-call reporting (2% for significant errors,5% for all discrepancies affecting management). Methodology: All on-call registrar reports were compared with consultant addendums for February 2013.Discrepancies were categorised according to grading system, 1 ¼ minor error with no significance,2 ¼ minor error affecting long-term outcome, 3 ¼ major error affecting long-term outcome,4 ¼ major error affecting immediate outcome. Results of 1st audit round: Total of 646 reports,77(11.9%) had corrections:22 scored 1(3.4%),25 scored 2(3.9%),22 scored 3(3.4%),8 scored 4(1.2%). 1st action plan: This audit demonstrates that issuing on-call reports by registrars is safe with only a small proportion of acutely significant discrepancies.It also emphasises the value of Consultant review, with subsequent amendments impacting upon management. Results of 2nd round: Follow-up audit for February 2014 had total of 770 reports,66(8.6%) had corrections:34 scored 1(4.4%),17 scored 2(2.2%),13 scored 3(1.7%),2 scored 4(0.3%). 2nd action plan: Again, the on-call registrar reporting accuracy was within ’general target’ despite increased workload in 2014.Nevertheless,we recommend regular monitoring of on-call workload and registrar reporting accuracy to ensure a high standard for patient care and safety.
Background: Day case intervention reduces patient stay, can improve patient experience and reduce overall procedural cost. Provision of good patient experience an important measure of care. Standard, indicator and target: Standards were locally agreed, based on NICE standards (1) . 90% of patients should have been pre-assessed and 90% should receive written information in advance. We wished for mean satisfaction score of 4 out of 5. Methodology: Patient perception was assessed using a 13 part survey comprising binary questions and scaled responses (1-very poor to 5excellent). Data were collected over eight weeks. Results of 1st audit round: 53 patients were surveyed. 93% had preadmission consultation. 86.7% had received written information. Mean scores to other questions all met the agreed standard. 1st action plan: Improved systems for administering information leaflets. Results of 2nd round: 52 Patients were surveyed. 76.9% were pre-assessed by telephone or in person, fewer than 50% of patients received written information and 3.8% had no pre-admission information. Standards for satisfaction with the consent process and addressing consent questions were again met. Patients were highly satisfied with the availability of staff to answer questions and their reassurance post-procedure. Overall patient satisfaction was ’very good’ or better. 2nd action plan: Overall satisfaction was increased and met targets, but improved administration to ensure higher rates of pre-assessment and written information provision is required.
Unexpected findings e the importance of safety nets
Non-contrast CT KUB in acute renal colic
Authors: Sean J. McIlhone*, Bhavin Narayanan, Catriona Davies
Rawal, Jeremy
Lynch, Priya
Background: Timely communication of reports and alerting clinicans to critical/unexpected findings is essential for patient safety and in preventing serious untoward incidents. Following the RCR’s updated guidance on this issue (1), we audited the existing ’safety net’ procedures in our imaging department.
* * * *
Guarantor Guarantor Guarantor Guarantor
and and and and
correspondent: correspondent: correspondent: correspondent:
Victoria Chan Sean J. McIlhone Jessie Gil Matthew Pickford
Authors: Matthew Pickford*, George Yeung Background: We perceived a low diagnostic yield for CT KUB when balanced against radiation exposure in our younger patients, particularly young women. We therefore audited our detection rates for stones and other significant pathology. Standard, indicator and target: The highest and lowest detection rates identified by literature review for stones and other significant pathology
Methodology: A prospective audit was undertaken over a 6-week period across the department. Operators were unaware and were assumed to be cognisant of the 3 patient point ID check procedure from local training. Results of 1st audit round: A total of 154 procedures were audited from various imaging modalities and procedures were performed by different operators. Only 47/154 procedures had a full 3-point identification performed prior to commencement of examination. 1st action plan: Regular mandatory training of all staff and a mandatory 10 second time out pre procedure led by the operator was proposed. Results of 2nd round: A total of 178 procedures from various modalities and different level of operators were audited. 158/178 procedures had a full 3-point identification performed prior to commencement of examination. 2nd action plan: Although a positive improvement has been identified, further re-iteration is needed of the importance of the check prior to commencement of any radiological procedure. Audit of the proportion of CTs informally double reported at West Suffolk Hospital
S19
Grade given Correct Descriptor used DEXA recommendation Methodology: Reporting of vertebral fractures in our institution was evaluated for the following examinations performed on females over the age of 45 years, against the set standard. 406 thoracic and lumbar plain films 163 thorax/abdomen/pelvis CTs Results of 1st audit round: The vertebral fracture identification rate was 94.2% on plain film but only 13.5% on CT. However when fractures were identified, there was generally consistent use of the word ’fracture’ rather than ’wedging’ or ’compression’ in both modalities. 1st action plan: The enormous discrepancy in CT implies that radiologists are not routinely using sagittal reconstructions to evaluate the spine. The findings were presented in a local audit meeting to encourage the use of 3D reconstruction capability of CT to actively look for spinal fractures even on general CT scans. An audit of voice recognition errors in neuroradiology reports
Authors: Zosia Rodak*, E.R. Darrah Background: There is increasing recognition that double reporting, particularly of cross-sectional imaging is important for governance and the RCR has recognised its value for revalidation evidence. The literature and current guidance relates to “formal, blinded double reporting”, but a significant amount of more informal double reporting occurs in all radiology departments, as second opinions, MDT review etc. This audit is to quantify this and assess its potential value. Standard, indicator and target: Independent providers of radiology services double report 5-10% of scans as part of standard governance and the RCR includes double reporting in its toolkit for revalidation. Methodology: From 19/03/12 - 25/03/12 and 22/04/13 - 28/04/13, all radiologists at West Suffolk Hospital recorded when they double reported a CT, the reason for doing so, and if any change was noted. This was compared to the total number of CTs. Results of 1st audit round: 82 of 247 of the scans were double reported. 86% were reviewed in MDTs. 3.6% of scans were reported differently. 1st action plan: As a large proportion of scans are already being doublereported informally day-to-day, this should continue. The plan is to formalise this into a report for our governance meeting and to make it an annual process that can feed into ISAS outcomes. Results of 2nd round: 74 of 287 (25.7%) of the scans were double reported. 85% were reviewed in MDTs. 2.7% of scans were reported differently. 2nd action plan: To continue with this as an annual process to contribute to ISAS outcomes.
Authors: Mark Igra*, Stuart Currie, Jeremy Macmullen-Price, Daniel J. Warren, Ian Craven Background: Voice recognition (VR) use has increased since the 1980s. This increase has lead to more errors in reports.1,2 The Royal College of Radiologists states that wording of reports should be clear.3 Errors can have significant clinical consequences if not corrected promptly.4 Errors are more likely in noisy departments, those with high workloads or if English is not the reporter’s first language. Standard, indicator and target: Our aims were to determine the frequency and type of errors and whether errors affected report quality. 100% of reports should be free of errors affecting report interpretation. Image interpretation was not assessed. Methodology: 300 reports from 6 consultants (50 consecutive each) were analysed. Errors were categorized as wrong word substitutions, nonsense phrases, deletions, insertions or punctuation errors. Whether the error affected report interpretation was determined. Results of 1st audit round: Average words per report varied from 62 to 190, and average words per error was 929. Total number of individual errors per consultant varied from 2 to 37. 88% of reports were completely error free. 99.4% of reports were free of significant errors. 1st action plan: Data was presented to the department with advice on reducing grammatical errors. Re-audit will take place annually to ensure we are maintaining or improving standards. VR is an effective reporting tool but checking reports prior to verifying remains essential. Reminders on reporting software to check may help further reduce errors.
Standards of spinal fractures reports in plain radiograph and CT Audit of voice recognition system report accuracy Authors: Ayano Tachibana*, H’ssein Al-Chalabi, Clare Groves Authors: Ai-Lee Chang*, Dorothy Keane, Lance Cope Background: Osteoporotic vertebral fractures are associated with significant risk of further spinal and hip fractures, with subsequent increase in morbidity and mortality. Secondary prevention strategies are dependent on accurate reporting of spinal fractures. The International Osteoporosis Foundation recommends the use of clear, unambiguous terminology using the word ‘fracture’, and grading the fracture, based on the semi-quantitative system outlined by Genant and Wu (1993). Standard, indicator and target: Standard of 100% was set for criteria including:Fractures identified Term “Fracture” used if identified
* * * *
Guarantor Guarantor Guarantor Guarantor
and and and and
correspondent: correspondent: correspondent: correspondent:
Zosia Rodak Ayano Tachibana Mark Igra Ai-Lee Chang
Background: Audit to assess errors in voice recognition system (VRS) generated reports due to failure of software to correctly identify words and failure of user to spot error in text. Standard, indicator and target: VRS generated reports should be accurate and clear. Overall error rate <5%VRS reports. There should be no major errors where the report makes no sense, target 0%. Indicator:% of reports containing an error. Methodology: 50CR and 20CT consecutive reports for each VRS user Error classified as minor, moderate, major Clinical details and body of report reviewed VRS users: CT6 CR8.
S20
Abstract / Clinical Radiology 69 (2014) S11eS22
Results of 1st audit round: CT- avg 27%, median 15%, range 0-90% CR- avg 11%, median 8%, range 2-38% 1 CR report made no sense. Targets not met. Types of errors - similar sounding words, lack of punctuations, words cut off, extra words. 1st action plan: 1. Presented at departmental and hospital audit meeting 2. Recommend RIS that would support auto-transcription of ’clinical details’ 3. Feedback to reporter 4. Repeat audit in 6 months Results of 2nd round: March - April 2014. CTe avg 23%; median 5%; range 0-100% CReavg18%; median 8%; range 2e84% Targets not met. Reporter A continues to have high VR error rate. Other reporters have good improvement. Overall improvement in CT errors. CR errors have deteriorated. Errors are mainly of voice recognition and punctuation. Majority of errors are mild and in clinical details. 2nd action plan:1. Present at departmental audit meeting 2. Urgent recommendation of RIS that would support auto-transcription of ‘clinical details’ 3. Feedback to reporter
Standard, indicator and target: 100% of patients with unexpected findings should either: Have their report faxed to the referring clinician or Be added to a relevant MDT meeting for discussion. Methodology: Data Set: All emails regarding unexpected findings that were sent to radiology secretaries during a one month period (November 2013) were retrieved from the radiology information system. These emails may request for the report to be urgently faxed to the referring clinician OR request that the patient is added to the relevant MDT meeting. Hospital records were then analysed to ensure these patients had been followed up as appropriate. Results of 1st audit round: Of the email requests for a report to be faxed, 94% (50/53) had records confirming that a fax had been sent. Average time lag ¼ 3 days Of the email requests for MDT discussion, 90% (43/48) of patient records confirmed they were discussed at MDT. Average time lag ¼ 7 days All remaining patients were accounted for and followed up. 1st action plan: 1. Standardise the format of email requests and of the electronic record showing the request was actioned 2. Departmental meeting to suggest a more robust and failsafe method of communicating urgent findings to clinicians
Radiology registrar on-call reporting accuracy
Patient satisfaction with day case provision of interventional radiology procedures
Authors: Victoria Chan*, Heiko Parvizi, James Briggs
Peschl, Matthew
Smedley, Nassim Authors: Jessie Gil*, James H. Briggs, Brenda Shanahan, Mark Bratby, Raman Uberoi
Background: At our Trust, on- call registrars (ST3+) issue verified reports, subsequently checked by consultants within 24 hours. Clinical teams should be contacted when there are discrepancies to allow for changes in management. Standard, indicator and target: An RCR AuditLive template proposed a 2-5% error rate for on-call reporting (2% for significant errors,5% for all discrepancies affecting management). Methodology: All on-call registrar reports were compared with consultant addendums for February 2013.Discrepancies were categorised according to grading system, 1 ¼ minor error with no significance,2 ¼ minor error affecting long-term outcome, 3 ¼ major error affecting long-term outcome,4 ¼ major error affecting immediate outcome. Results of 1st audit round: Total of 646 reports,77(11.9%) had corrections:22 scored 1(3.4%),25 scored 2(3.9%),22 scored 3(3.4%),8 scored 4(1.2%). 1st action plan: This audit demonstrates that issuing on-call reports by registrars is safe with only a small proportion of acutely significant discrepancies.It also emphasises the value of Consultant review, with subsequent amendments impacting upon management. Results of 2nd round: Follow-up audit for February 2014 had total of 770 reports,66(8.6%) had corrections:34 scored 1(4.4%),17 scored 2(2.2%),13 scored 3(1.7%),2 scored 4(0.3%). 2nd action plan: Again, the on-call registrar reporting accuracy was within ’general target’ despite increased workload in 2014.Nevertheless,we recommend regular monitoring of on-call workload and registrar reporting accuracy to ensure a high standard for patient care and safety.
Background: Day case intervention reduces patient stay, can improve patient experience and reduce overall procedural cost. Provision of good patient experience an important measure of care. Standard, indicator and target: Standards were locally agreed, based on NICE standards (1) . 90% of patients should have been pre-assessed and 90% should receive written information in advance. We wished for mean satisfaction score of 4 out of 5. Methodology: Patient perception was assessed using a 13 part survey comprising binary questions and scaled responses (1-very poor to 5excellent). Data were collected over eight weeks. Results of 1st audit round: 53 patients were surveyed. 93% had preadmission consultation. 86.7% had received written information. Mean scores to other questions all met the agreed standard. 1st action plan: Improved systems for administering information leaflets. Results of 2nd round: 52 Patients were surveyed. 76.9% were pre-assessed by telephone or in person, fewer than 50% of patients received written information and 3.8% had no pre-admission information. Standards for satisfaction with the consent process and addressing consent questions were again met. Patients were highly satisfied with the availability of staff to answer questions and their reassurance post-procedure. Overall patient satisfaction was ’very good’ or better. 2nd action plan: Overall satisfaction was increased and met targets, but improved administration to ensure higher rates of pre-assessment and written information provision is required.
Unexpected findings e the importance of safety nets
Non-contrast CT KUB in acute renal colic
Authors: Sean J. McIlhone*, Bhavin Narayanan, Catriona Davies
Rawal, Jeremy
Lynch, Priya
Background: Timely communication of reports and alerting clinicans to critical/unexpected findings is essential for patient safety and in preventing serious untoward incidents. Following the RCR’s updated guidance on this issue (1), we audited the existing ’safety net’ procedures in our imaging department.
* * * *
Guarantor Guarantor Guarantor Guarantor
and and and and
correspondent: correspondent: correspondent: correspondent:
Victoria Chan Sean J. McIlhone Jessie Gil Matthew Pickford
Authors: Matthew Pickford*, George Yeung Background: We perceived a low diagnostic yield for CT KUB when balanced against radiation exposure in our younger patients, particularly young women. We therefore audited our detection rates for stones and other significant pathology. Standard, indicator and target: The highest and lowest detection rates identified by literature review for stones and other significant pathology