- Email: [email protected]
Audit of quality in a radiological department — A pilot study
ClinicalRadiology (1991) 44, 345-349
Audit of Quality in a Radiological DepartmentA Pilot Study P. G_ C O O K , I. W. J. B I R C H A L L and W. D. J E A N S
Department of Radiodiagnosis, Bristol Royal Infirmary, Bristol As part of the development of an auditing system for this department, a pilot study was designed to audit the quality of the product leaving the department. One hundred and six patient investigations were reviewed. For each investigation 26 items which were considered to reflect the quality of information on the request card, the films, the report and the condition of the film packets were assessed using a simple scoring system. Overall performances for various aspects of the departmental 'product', and for groups of staff within the department, were derived from these scores. Several problem areas were identified. Individual items were then scrutinized for recurring low scores and where scores fell outside a predetermined range steps were taken to improve standards. The scoring system developed is simple, easy and effective in use. It is considered that audit of quality is worthwhile arid should be extended to become a routine part of departmental management. Cook, P. G_, Birchall, I.W.J. & Jeans, W.D. ( 1991 ). Clinical Radiology 44, 345349. Audit of Quality in a Radiological Department - A Pilot Study
Auditing a department of radiology is inevitably a complex procedure and will cover m a n y aspects of departmental working. The document from the Royal College of Radiologists (1989) outlines the general aspects of auditing. T h e details of the auditing process will differ according to local circumstances and it is suggested that local arrangements are made for monitoring audit activities. This department has recently been engaged in setting up an auditing system based around a newly installed computer (Fujitsu 2500) and Radiological Patient Management System with diagnostic coding facilities (Radwise). In the development of the auditing system, an assessment of the quality of the 'product' from the department was considered to be essential. The films, film packets and reports leaving a radiology department can be looked upon as the 'product' of the department, the quality of which can be measured. This paper describes a pilot study designed to audit the quality of the product from the radiological department.
PATIENTS A N D M E T H O D S
The Scoring System
A scoring system was developed which scored each item assessed as - 1, 0 or + 1. If the item was assessed as completely inadequate or unhelpful and needing some research or time spent in correcting errors, or completing information, then it was scored as - 1. If the item was assessed as being inadequate or incomplete, but just sufficient and not necessarily requiring any further information, then it was scored as 0. I f the item was full, complete and relevant then it was scored as + 1. Twenty-six items were selected for assessment, grouped into three main areas and subdivided into areas identifiable as the responsibility of a particular professional group. Tables 1, 2 and 3 show the scoring system used in the assessment of the request card, the reports and the 'product' respectively. A total score for each of the three main areas of assessment was obtained by adding all the scores and calculating this as a percentage of the total possible score. This then provided a ~rating' of performance. A similar method was used to calculate a percentage score for each of the identifiable professional groups within the department. Items Assessed
Patients
The investigations of 10 patients per week were selected for assessment, one from each of the Korner groups, to a total of 106. The Korner Classification (Korner, 1982) groups radiological examinations according to the complexity of the examination for the purpose of information collection on the use of resources within the radiological department. The patients were chosen at r a n d o m from the request cards collected from all investigations in the department during the week. A delay of 4 weeks was allowed before assessment, to enable films to be returned to the film library_
Twenty-six items which were considered to influence the quality of the 'product' were assessed. These items were grouped according to various activities within the department and the activities of various professional groups within and outside the department.
The Request Card Ten items on the card, which were required to be completed by clinicians, were assessed for completeness of information. The list of items assessed is shown in Table 1 together with the scoring used_
The Radiology Report Correspondence to: Dr P. G. Cook, Department of Radiodiagnosis, Royal Cornwall Hospital (Treliske), Truro, Cornwall TR1 3LJ.
The report itself was assessed for typing mistakes, readability, relevance to the clinical information pro-
346
CLINICAL RADIOLOGY
Table 1 - Scoring system for request card
Code
Item
C1
Patient details
C2
C3
C4 C5 C6 C7 C8 C9
C10
Assessment
Complete Incomplete/sufficient Minimal, needs research Clinical information Full, relevant, legible Partial, not relevant Minimal, illegible or irrelevant to exam. Appropriate request Appropriate (with respect to Other invest, prob. better clinical info) Possibly appropriate Probably inappropriate Inappropriate Legibility Fully legible Partially legible Illegible/no sense Signature Signed by requesting Dr Unsigned or pp'd Bleep No/Ext No . Present Absent Dr's Name Clearly written None/illegible Transport Filled in Not filled in LMP Filled in and relevant Instructed to ignore Not applic, and not filled Not filled and applicable Previous investigations Filled completely Incomplete/irrelevant Not filled in
Table 3 - Scoring system for films, film bag and presentation of reports grouped as the 'product' of the department
Score 1 0 - I 1 0
Code
Item
Assessment
Score
Films F1
Quality
Excellent exp. and position Diagnostic Poor Present Absent Present Absent Completely Incompletely No Yes fully Incompletely No
l 0 - 1 1 -- 1 1 - 1 1 0 --1 1 0 -1
Excellent/new Mended Shabby/needs mend/repl. Present on front Absent
l 0 - 1 1 - 1
Present in good order Present + loose reports Absent In folder and in order In folder not in order Loose in bag Not in bag or folder Total no. None One >1
1 0 1 1 0 -- 1 - 1 1 0 --1
F2 -- 1 1 0 0 0 - 1 1 0 - 1 1 -- 1 1 - 1 1 - 1 1 -- 1 1 1 0 1 1 0 - 1
The left-hand column is the code for the item assessed, the second column the item assessed, the third column the assessments for each item and the right-hand column the score for each of the assessments. LMP, date of last menstrual period.
Marker; Side
F3
Radios.
F4
Numbered
F5
Indexed on bag
Bags B1
B2 Report folder RF1
State
Bar cocte
Folder
RF2
Exam. report
RF3
Loose reports
The columns are arranged as in Table 1. Table 2 - Scoring system for reports
Code
Item
Assessment
Score
R1
Typing mistakes
English words
1 0 --1 1 0 -I 1 0 - 1 1 l 0 - 1 l 1 0 - 1 1 l - 1
R2 R3
Readability
R4
Relevance to clin. info.
R5
Conclusion
R6
Coded
None 0-3 >3 Medical words None O3 >3 Plain English and to point Embellished/long winded Hedging/unclear Clearly unhelpful Directly relevant Probably relevant Not relevant Present No but short report No, probably needs one No, definitely needs one Yes, more than 1 code Yes, 1 code Not coded
m a r k e r s a n d r a d i o g r a p h e r ' s m a r k e r s . T h e i n d e x i n g was a s s e s s e d o n t w o i t e m s , l a b e l l i n g o r n u m b e r i n g o f t h e films a n d i n d e x i n g a c c u r a t e l y o n t h e film p a c k e t .
The Film Packet T h e film p a c k e t w a s a s s e s s e d b y its p h y s i c a l s t a t e a n d b y t h e p r e s e n c e o f a c o m p u t e r b a r c o d e o n t h e f r o n t o f the packet.
The Report Folder Inside the Fihn Packet T h e r e p o r t f o l d e r w a s a s s e s s e d b y its p r e s e n c e , its a p p e a r a n c e , t h e p r e s e n c e o f t h e r e l e v a n t r e p o r t in the f o l d e r a n d t h e n u m b e r o f l o o s e r e p o r t s i n s i d e t h e film packet. Table 3 shows the items assessed for each o f these entities and the scoring used.
The columns are arranged as in Table 1. RESULTS vided, the presence or absence of a conclusion where n e e d e d a n d t h e p r e s e n c e o f a d i a g n o s t i c c o d e . T a b l e 2 lists t h e i t e m s o f a s s e s s m e n t a n d t h e s c o r i n g used.
The Radiographic Image and Indexing The r a d i o g r a p h i c i m a g e was assessed on three items: t h e g e n e r a l q u a l i t y o f t h e i m a g e , t h e p r e s e n c e o f side
T h e i n v e s t i g a t i o n s o f 170 p a t i e n t s w e r e r e q u e s t e d f r o m t h e film l i b r a r y o v e r a 17 w e e k p e r i o d . O n e h u n d r e d a n d six p a t i e n t s ' films w e r e a v a i l a b l e at t h e t i m e o f r e q u e s t T h u s 106 o f 170 p a t i e n t s ' films w e r e r e t r i e v e d f o r the s t u d y , i n d i c a t i n g a 3 8 % film r e t r i e v a l f a i l u r e r a t e . O f 106 p a t i e n t i n v e s t i g a t i o n s e i g h t ( 5 . 7 % ) d i d n o t h a v e r e p o r t s in t h e film p a c k e t . Six o f t h e s e h a d r e p o r t s filed o n t h e c o m p u t e r s y s t e m w h i c h w e r e r e c o v e r e d f o r assess-
347
QUALITY IN A RADIOLOGICAL DEPARTMENT Table 4 - Table of scores for groups and items
Overall percentage
Item code
Scores by assessment item no.
- s (o/<>j
Card
*42.5
Report Typists
80.3 86.3
Radiologist
End product Radiographer
Clerical
77 ~4
*40.8 *55.7
*42.9
Typists
*15.7
C1 C2 C3 C4 C5 C6 C7 C8 C9 C10
2 (1.9) 4 (3.8) 0 (0) 2 (1.9) 2(1.9) *53 (50.0) *36 (34.0) *60 (56.6) "18 (81.8) "71 (67.0)
R1 R2
0 (%)
+ 1 ¢o/,j
16 (15.l) 13 (12_2) 5 (4.7) 17 (16.0),
14 (13 2)
88 (83.0) 89 (84.0) 101 (95.3) 87 (82.1) 104(98.1) 53 (50.0) 70 (66.0) 46 (43.4) 4 (18.2) 21 (19.8)
2 (1.9) 2 (1.9)
10 (9.4) 13 (12.3)
94 (88.7) 91 (85.8)
R3 R4 R5 R6
5 (4.7) 2 (1.9) 5 (4.7) *28 (26.4)
4 (3.8) 4 (3.8) 12 (11.3)
F1 F2 F3 F4 F5
5 (4.7) 11 (10.4) 15 (14.2) *36 (34.0) 4 (3.8)
B1
5 (4.7) "51 (48.1)
9 (8.5)
B2
92 (86.8) 55 (51.9)
RF1 RF2 RF3
*38 (35.8) *44 (41.5) *32 (30.2)
5 (4.7) 10 (9.4) 25 (23.6)
63 (59.4) 52 (49.1) 49 (46.2)
-
6 9 *63 11 2
(5.7) (8.5) (59.4) (10.4) (1.9)
97 100 89 78
(91.5) (94.3) (84.0) (73.6)
95 86 28 59 100
(89.6) (81.1) (26.4) (55.6) (94.3)
* Scores which fall outside the m i n i m u m standards. The left-hand side represents the overall percentage score for each group of assessment items and professionals in the department. The three right-hand columns show the n u m b e r (percentage) of investigations scoring - 1, 0 or 1 for each assessment item. See Tables 1, 2 and 3 for the item codes.
Previous investigation
'77774
~W/////////////////A
Indexed
LMP Transport Doctor's name Ext. bleep no. Signed
i ~////////////////A
Numbered
.............. ~
/H/////HN/I~
I
.............. ~
.............
H~/////H/11~////////4 Legibility Appropriate request ~"N'",,////////U/U//d E'd7 Clinical information ! P77 "H~H~H~///"HH//H///I Details I I i I 25 50 75 100(%) (%) 100 75 50 25 Poor score (0,-1) Good score (+1)
•-1
[] 0
Radiographer's marker
ment. T h e r e m a i n i n g two reports c o u l d n o t be f o u n d a n d were scored as - 1 for all r e p o r t scores. T h e results are presented in t a b u l a r f o r m in T a b l e 4, which shows all the d a t a obtained. T h e left-hand c o l u m n gives the overall percentage score for each o f the three m a i n assessment items, the card, the r e p o r t a n d the ' p r o d u c t ' . In this c o l u m n each p r o f e s s i o n a l g r o u p contrib u t i n g to the m a i n item is also given a score. T h e three r i g h t - h a n d c o l u m n s show the scoring for each assessment item as the n u m b e r (and percentage) o f investigations scoring - 1, 0 o r + 1.
U/////A
"////////////////////////A
Side m a r k e r
~///////////////////A
Film quality I
(%)
FA+I
Fig. 1 - T h e card score. The "good' scores ( + 1) are to the right of the centre and the poor scores (0, - 1) to the left to allow easy differentiation of the scores. A vertical line at 25% on the left of the chart represents the m i n i m u m standard for the scores of each individual item assessed.
g,/////////////
=I
I
I
I
I
100 75 50 ')5 0 25 50 75 100 (%) Poor score(0, 1) Good score (+1) •-1
[] 0
[] +1
Fig. 2 - The radiographer score. The arrangement is as in Fig. 1.
Figs 1-5 show the i n f o r m a t i o n in graphical form. The percentage o f investigations scoring -- 1, 0 or + 1 for each i n d i v i d u a l assessment item is shown a n d g r o u p e d a c c o r d ing to the staff g r o u p responsible_ ' P o o r ' scores ( - 1 a n d 0) are s h o w n to the left o f the zero line a n d ' g o o d ' scores to the right.
DISCUSSION All the items selected for assessment were considered to
348
CLINICAL RADIOLOGY
U///////////////////////////A Coded
U//////////////////////////////A
Conclusion
~///////////////////A
Relevance
U//////////////////////////////////A
Readability I
I
25 0 Poor score (0,-1) •
-1
I
I
i
25 50 75 Good score (+1)
(%) 50
[] 0
100 (%)
[ ] +1
Fig. 3 - T h e r a d i o l o g i s t score. T h e a r r a n g e m e n t is as in Fig. 1.
V_~/////////////////////////A
Typing (Medical)
/////////////////////////A
Typing (English) Loose reports Report in folder Folder in bag (%)
100
75 50 25 0 Poor score (0,-1)
I I 25 50 75 100 (%) Good score (+1)
[] 0
•-1
[] +1
Fig. 4 The secretarial score. The arrangement is as in Fig. 1.
Bar code on bag I
State of bag
(%)
I
100
I
I
I
I
I
I
I
75 50 25 0 25 50 75 lOg (%) Good score (+1) Poor score (0,-1)
• -1
[] O
[] +1
Fig. 5 - T h e clerical score. T h e a r r a n g e m e n t is as in Fig. 1.
reflect on the quality of the 'product' leaving the department. The filling in of the request card may not at first seem to be a useful assessment. The incomplete or inaccurate filling in of the card will inevitably lead to time lost by one or more departmental member acquiring and completing details. This is obviously an inefficient use of time and resources and an important part of audit. Furthermore, if the quality of information provided to the department is poor or inaccurate then the resulting investigation and report may be less than optimum, requiring repeat or different investigations. Other items assessed are more obviously and directly relevant. Minimum levels of standard were set to enable the quality of the product to be assessed and maintained. We felt that initially, arbitrary levels of 70% for the overall scores and a level not greater than 25% i'or - 1 or 0 scores for each individual item assessed should be set. Asterisks are used to mark the scores falling outside this range in Table 4 and lines are used on the bar charts, making identification easy. It can be seen from Table 4 and the graphic data that the request card and the 'product' do not score well overall. Looking further into the data reveals the areas where p o o r scoring is occurring. In particular, referring clinicians are poor at filling in bleep or extension numbers, printing their name next to their signature, selecting the appropriate method of transport and in filling in details of previous investigations_ Where relevant, the last menstrual period (LMP) section on the card was frequently not filled in. Radiologists are poor at adding diagnostic codes to the report. Radiographers do not perform well in marking their identity numbers on the films, or numbering the films to correspond with the index number on the film packet. The results of the secretarial/typist assessment in the report section are excellent, indicating accurate typing and spelling. In the assessment of the 'product'; the secretarial score shows poor performance in putting reports into folders and folders into film packets. Clerical assessment shows poor performance at placing bar codes on the front of film packets. In order to make more sense of the information provided by auditing in this way, data must be accumulated over m a n y months, and means and standard deviations applied to the scores. The regular assessment period should be selected for convenience and the results of this compared to the minimum standards and the long term means and standard deviations. Further detailed studies, based on these findings, have been initiated to determine the significance of the findings and to recommend changes in practice. Redesign of the request card and film packet is already underway. The current practice of typists being solely responsible for attaching reports to report folders is under reviewFurther studies are looking at request card filling by clinicians, film marking by radiographers, the number of unreported films leaving the departmenfand film retrieval rates from the film store. A repeat study at a later date will indicate the results of any changes in practice created by this study. A number of problems were apparent in this pilot study which are to be addressed before setting up the definitive study.
QUALITY
IN
A RADIOLOGICAL
++°I o
=ca 40 g t-
++++ol A
349
5 No attempt was made to assess the accuracy of the report, nor of the diagnostic code at the end of the report. It is important, however, that accuracy of reporting be monitored by some method and we feel a system of assessment should be included in future audit. 6 The minimum levels of standard have been set empirically and will be adjusted as experience and further data are obtained.
~ 60 g
0
DEPARTMENT
i
B
i
•
C
i
l D
ILltltlt E F Korner group
Sample
G
II
III
V
[ ] Departmental
Fig. 6 Comparisonof sample and departmental workload in terms of Korner group.
1 The source of requests in our sample did not closely reflect the source profile of the department as a whole. The proportions of out-patient, in-patient and GP referrals for our sample were 46%, 41% and 13% respectively, compared with the departmental workload of 40%, 27% and 13% respectively. No casualty films were included in our sample. 2 The distribution of Korner groups, shown in Fig. 6, demonstrates that although all Korner groups were covered, the distribution of investigations assessed did not reflect the workload of the department. It is intended that the computer be used to generate random samples of investigations which more closely reflect the departmental workload. 3 The failure to retrieve 38% of films requested does not compare favourably with a 3.6% failure rate for outpatient clinics in August 1989. This discrepancy may reflect the timing of the attempted retrieval at 1 month, when patients' films may still be with clinicians or have been requested for clinic appointments. 4 Objective assessment of a radiological report may be influenced by knowing who has written the report, leading to some bias. However, the bias is probably small.
It was estimated that the workload entailed in this study was approximately 20 h spent analysing the request card, image, film packet and report card, and approximately 2-3 h of clerical time to retrieve all the film packets. In this pilot study we have essentially been inward looking and only assessed those aspects which can be easily monitored within the department. We have not considered the patient and how they felt they were treated whilst in the department, nor have we assessed the response of the clinician to the report received. Inclusion of these aspects, together with waiting times within the department and delay times between the date of request and the date of reporting, will enable a fuller assessment of the service we are providing. It is planned to introduce these aspects into our audit system at a later date. The scoring system devised in this pilot study is easy to use and is expandable to include more items for assessment and finer tuning of the assessment. The data produced from the study were stored and manipulated easily on a computer_ Many data processing and graphics programs are readily available and should be chosen to suit the auditor and the system used. It is considered that audit of quality in a department of radiology is both possible and worthwhile and should be included as part of the routine departmental management process.
REFERENCES
Royal College of Radiologists (1989). Medical Audzt #7 Radiodiagnosis. Report of a Working Party of the Royal College of Radiologists, London. Korner, E (1982). The Department of Health and Social Security. First Report of the Steering Group on Health Services Information.
Chairman, Korner, E. HMSO Publications, London.
Audit of Quality in a Radiological DepartmentA Pilot Study P. G_ C O O K , I. W. J. B I R C H A L L and W. D. J E A N S
Department of Radiodiagnosis, Bristol Royal Infirmary, Bristol As part of the development of an auditing system for this department, a pilot study was designed to audit the quality of the product leaving the department. One hundred and six patient investigations were reviewed. For each investigation 26 items which were considered to reflect the quality of information on the request card, the films, the report and the condition of the film packets were assessed using a simple scoring system. Overall performances for various aspects of the departmental 'product', and for groups of staff within the department, were derived from these scores. Several problem areas were identified. Individual items were then scrutinized for recurring low scores and where scores fell outside a predetermined range steps were taken to improve standards. The scoring system developed is simple, easy and effective in use. It is considered that audit of quality is worthwhile arid should be extended to become a routine part of departmental management. Cook, P. G_, Birchall, I.W.J. & Jeans, W.D. ( 1991 ). Clinical Radiology 44, 345349. Audit of Quality in a Radiological Department - A Pilot Study
Auditing a department of radiology is inevitably a complex procedure and will cover m a n y aspects of departmental working. The document from the Royal College of Radiologists (1989) outlines the general aspects of auditing. T h e details of the auditing process will differ according to local circumstances and it is suggested that local arrangements are made for monitoring audit activities. This department has recently been engaged in setting up an auditing system based around a newly installed computer (Fujitsu 2500) and Radiological Patient Management System with diagnostic coding facilities (Radwise). In the development of the auditing system, an assessment of the quality of the 'product' from the department was considered to be essential. The films, film packets and reports leaving a radiology department can be looked upon as the 'product' of the department, the quality of which can be measured. This paper describes a pilot study designed to audit the quality of the product from the radiological department.
PATIENTS A N D M E T H O D S
The Scoring System
A scoring system was developed which scored each item assessed as - 1, 0 or + 1. If the item was assessed as completely inadequate or unhelpful and needing some research or time spent in correcting errors, or completing information, then it was scored as - 1. If the item was assessed as being inadequate or incomplete, but just sufficient and not necessarily requiring any further information, then it was scored as 0. I f the item was full, complete and relevant then it was scored as + 1. Twenty-six items were selected for assessment, grouped into three main areas and subdivided into areas identifiable as the responsibility of a particular professional group. Tables 1, 2 and 3 show the scoring system used in the assessment of the request card, the reports and the 'product' respectively. A total score for each of the three main areas of assessment was obtained by adding all the scores and calculating this as a percentage of the total possible score. This then provided a ~rating' of performance. A similar method was used to calculate a percentage score for each of the identifiable professional groups within the department. Items Assessed
Patients
The investigations of 10 patients per week were selected for assessment, one from each of the Korner groups, to a total of 106. The Korner Classification (Korner, 1982) groups radiological examinations according to the complexity of the examination for the purpose of information collection on the use of resources within the radiological department. The patients were chosen at r a n d o m from the request cards collected from all investigations in the department during the week. A delay of 4 weeks was allowed before assessment, to enable films to be returned to the film library_
Twenty-six items which were considered to influence the quality of the 'product' were assessed. These items were grouped according to various activities within the department and the activities of various professional groups within and outside the department.
The Request Card Ten items on the card, which were required to be completed by clinicians, were assessed for completeness of information. The list of items assessed is shown in Table 1 together with the scoring used_
The Radiology Report Correspondence to: Dr P. G. Cook, Department of Radiodiagnosis, Royal Cornwall Hospital (Treliske), Truro, Cornwall TR1 3LJ.
The report itself was assessed for typing mistakes, readability, relevance to the clinical information pro-
346
CLINICAL RADIOLOGY
Table 1 - Scoring system for request card
Code
Item
C1
Patient details
C2
C3
C4 C5 C6 C7 C8 C9
C10
Assessment
Complete Incomplete/sufficient Minimal, needs research Clinical information Full, relevant, legible Partial, not relevant Minimal, illegible or irrelevant to exam. Appropriate request Appropriate (with respect to Other invest, prob. better clinical info) Possibly appropriate Probably inappropriate Inappropriate Legibility Fully legible Partially legible Illegible/no sense Signature Signed by requesting Dr Unsigned or pp'd Bleep No/Ext No . Present Absent Dr's Name Clearly written None/illegible Transport Filled in Not filled in LMP Filled in and relevant Instructed to ignore Not applic, and not filled Not filled and applicable Previous investigations Filled completely Incomplete/irrelevant Not filled in
Table 3 - Scoring system for films, film bag and presentation of reports grouped as the 'product' of the department
Score 1 0 - I 1 0
Code
Item
Assessment
Score
Films F1
Quality
Excellent exp. and position Diagnostic Poor Present Absent Present Absent Completely Incompletely No Yes fully Incompletely No
l 0 - 1 1 -- 1 1 - 1 1 0 --1 1 0 -1
Excellent/new Mended Shabby/needs mend/repl. Present on front Absent
l 0 - 1 1 - 1
Present in good order Present + loose reports Absent In folder and in order In folder not in order Loose in bag Not in bag or folder Total no. None One >1
1 0 1 1 0 -- 1 - 1 1 0 --1
F2 -- 1 1 0 0 0 - 1 1 0 - 1 1 -- 1 1 - 1 1 - 1 1 -- 1 1 1 0 1 1 0 - 1
The left-hand column is the code for the item assessed, the second column the item assessed, the third column the assessments for each item and the right-hand column the score for each of the assessments. LMP, date of last menstrual period.
Marker; Side
F3
Radios.
F4
Numbered
F5
Indexed on bag
Bags B1
B2 Report folder RF1
State
Bar cocte
Folder
RF2
Exam. report
RF3
Loose reports
The columns are arranged as in Table 1. Table 2 - Scoring system for reports
Code
Item
Assessment
Score
R1
Typing mistakes
English words
1 0 --1 1 0 -I 1 0 - 1 1 l 0 - 1 l 1 0 - 1 1 l - 1
R2 R3
Readability
R4
Relevance to clin. info.
R5
Conclusion
R6
Coded
None 0-3 >3 Medical words None O3 >3 Plain English and to point Embellished/long winded Hedging/unclear Clearly unhelpful Directly relevant Probably relevant Not relevant Present No but short report No, probably needs one No, definitely needs one Yes, more than 1 code Yes, 1 code Not coded
m a r k e r s a n d r a d i o g r a p h e r ' s m a r k e r s . T h e i n d e x i n g was a s s e s s e d o n t w o i t e m s , l a b e l l i n g o r n u m b e r i n g o f t h e films a n d i n d e x i n g a c c u r a t e l y o n t h e film p a c k e t .
The Film Packet T h e film p a c k e t w a s a s s e s s e d b y its p h y s i c a l s t a t e a n d b y t h e p r e s e n c e o f a c o m p u t e r b a r c o d e o n t h e f r o n t o f the packet.
The Report Folder Inside the Fihn Packet T h e r e p o r t f o l d e r w a s a s s e s s e d b y its p r e s e n c e , its a p p e a r a n c e , t h e p r e s e n c e o f t h e r e l e v a n t r e p o r t in the f o l d e r a n d t h e n u m b e r o f l o o s e r e p o r t s i n s i d e t h e film packet. Table 3 shows the items assessed for each o f these entities and the scoring used.
The columns are arranged as in Table 1. RESULTS vided, the presence or absence of a conclusion where n e e d e d a n d t h e p r e s e n c e o f a d i a g n o s t i c c o d e . T a b l e 2 lists t h e i t e m s o f a s s e s s m e n t a n d t h e s c o r i n g used.
The Radiographic Image and Indexing The r a d i o g r a p h i c i m a g e was assessed on three items: t h e g e n e r a l q u a l i t y o f t h e i m a g e , t h e p r e s e n c e o f side
T h e i n v e s t i g a t i o n s o f 170 p a t i e n t s w e r e r e q u e s t e d f r o m t h e film l i b r a r y o v e r a 17 w e e k p e r i o d . O n e h u n d r e d a n d six p a t i e n t s ' films w e r e a v a i l a b l e at t h e t i m e o f r e q u e s t T h u s 106 o f 170 p a t i e n t s ' films w e r e r e t r i e v e d f o r the s t u d y , i n d i c a t i n g a 3 8 % film r e t r i e v a l f a i l u r e r a t e . O f 106 p a t i e n t i n v e s t i g a t i o n s e i g h t ( 5 . 7 % ) d i d n o t h a v e r e p o r t s in t h e film p a c k e t . Six o f t h e s e h a d r e p o r t s filed o n t h e c o m p u t e r s y s t e m w h i c h w e r e r e c o v e r e d f o r assess-
347
QUALITY IN A RADIOLOGICAL DEPARTMENT Table 4 - Table of scores for groups and items
Overall percentage
Item code
Scores by assessment item no.
- s (o/<>j
Card
*42.5
Report Typists
80.3 86.3
Radiologist
End product Radiographer
Clerical
77 ~4
*40.8 *55.7
*42.9
Typists
*15.7
C1 C2 C3 C4 C5 C6 C7 C8 C9 C10
2 (1.9) 4 (3.8) 0 (0) 2 (1.9) 2(1.9) *53 (50.0) *36 (34.0) *60 (56.6) "18 (81.8) "71 (67.0)
R1 R2
0 (%)
+ 1 ¢o/,j
16 (15.l) 13 (12_2) 5 (4.7) 17 (16.0),
14 (13 2)
88 (83.0) 89 (84.0) 101 (95.3) 87 (82.1) 104(98.1) 53 (50.0) 70 (66.0) 46 (43.4) 4 (18.2) 21 (19.8)
2 (1.9) 2 (1.9)
10 (9.4) 13 (12.3)
94 (88.7) 91 (85.8)
R3 R4 R5 R6
5 (4.7) 2 (1.9) 5 (4.7) *28 (26.4)
4 (3.8) 4 (3.8) 12 (11.3)
F1 F2 F3 F4 F5
5 (4.7) 11 (10.4) 15 (14.2) *36 (34.0) 4 (3.8)
B1
5 (4.7) "51 (48.1)
9 (8.5)
B2
92 (86.8) 55 (51.9)
RF1 RF2 RF3
*38 (35.8) *44 (41.5) *32 (30.2)
5 (4.7) 10 (9.4) 25 (23.6)
63 (59.4) 52 (49.1) 49 (46.2)
-
6 9 *63 11 2
(5.7) (8.5) (59.4) (10.4) (1.9)
97 100 89 78
(91.5) (94.3) (84.0) (73.6)
95 86 28 59 100
(89.6) (81.1) (26.4) (55.6) (94.3)
* Scores which fall outside the m i n i m u m standards. The left-hand side represents the overall percentage score for each group of assessment items and professionals in the department. The three right-hand columns show the n u m b e r (percentage) of investigations scoring - 1, 0 or 1 for each assessment item. See Tables 1, 2 and 3 for the item codes.
Previous investigation
'77774
~W/////////////////A
Indexed
LMP Transport Doctor's name Ext. bleep no. Signed
i ~////////////////A
Numbered
.............. ~
/H/////HN/I~
I
.............. ~
.............
H~/////H/11~////////4 Legibility Appropriate request ~"N'",,////////U/U//d E'd7 Clinical information ! P77 "H~H~H~///"HH//H///I Details I I i I 25 50 75 100(%) (%) 100 75 50 25 Poor score (0,-1) Good score (+1)
•-1
[] 0
Radiographer's marker
ment. T h e r e m a i n i n g two reports c o u l d n o t be f o u n d a n d were scored as - 1 for all r e p o r t scores. T h e results are presented in t a b u l a r f o r m in T a b l e 4, which shows all the d a t a obtained. T h e left-hand c o l u m n gives the overall percentage score for each o f the three m a i n assessment items, the card, the r e p o r t a n d the ' p r o d u c t ' . In this c o l u m n each p r o f e s s i o n a l g r o u p contrib u t i n g to the m a i n item is also given a score. T h e three r i g h t - h a n d c o l u m n s show the scoring for each assessment item as the n u m b e r (and percentage) o f investigations scoring - 1, 0 o r + 1.
U/////A
"////////////////////////A
Side m a r k e r
~///////////////////A
Film quality I
(%)
FA+I
Fig. 1 - T h e card score. The "good' scores ( + 1) are to the right of the centre and the poor scores (0, - 1) to the left to allow easy differentiation of the scores. A vertical line at 25% on the left of the chart represents the m i n i m u m standard for the scores of each individual item assessed.
g,/////////////
=I
I
I
I
I
100 75 50 ')5 0 25 50 75 100 (%) Poor score(0, 1) Good score (+1) •-1
[] 0
[] +1
Fig. 2 - The radiographer score. The arrangement is as in Fig. 1.
Figs 1-5 show the i n f o r m a t i o n in graphical form. The percentage o f investigations scoring -- 1, 0 or + 1 for each i n d i v i d u a l assessment item is shown a n d g r o u p e d a c c o r d ing to the staff g r o u p responsible_ ' P o o r ' scores ( - 1 a n d 0) are s h o w n to the left o f the zero line a n d ' g o o d ' scores to the right.
DISCUSSION All the items selected for assessment were considered to
348
CLINICAL RADIOLOGY
U///////////////////////////A Coded
U//////////////////////////////A
Conclusion
~///////////////////A
Relevance
U//////////////////////////////////A
Readability I
I
25 0 Poor score (0,-1) •
-1
I
I
i
25 50 75 Good score (+1)
(%) 50
[] 0
100 (%)
[ ] +1
Fig. 3 - T h e r a d i o l o g i s t score. T h e a r r a n g e m e n t is as in Fig. 1.
V_~/////////////////////////A
Typing (Medical)
/////////////////////////A
Typing (English) Loose reports Report in folder Folder in bag (%)
100
75 50 25 0 Poor score (0,-1)
I I 25 50 75 100 (%) Good score (+1)
[] 0
•-1
[] +1
Fig. 4 The secretarial score. The arrangement is as in Fig. 1.
Bar code on bag I
State of bag
(%)
I
100
I
I
I
I
I
I
I
75 50 25 0 25 50 75 lOg (%) Good score (+1) Poor score (0,-1)
• -1
[] O
[] +1
Fig. 5 - T h e clerical score. T h e a r r a n g e m e n t is as in Fig. 1.
reflect on the quality of the 'product' leaving the department. The filling in of the request card may not at first seem to be a useful assessment. The incomplete or inaccurate filling in of the card will inevitably lead to time lost by one or more departmental member acquiring and completing details. This is obviously an inefficient use of time and resources and an important part of audit. Furthermore, if the quality of information provided to the department is poor or inaccurate then the resulting investigation and report may be less than optimum, requiring repeat or different investigations. Other items assessed are more obviously and directly relevant. Minimum levels of standard were set to enable the quality of the product to be assessed and maintained. We felt that initially, arbitrary levels of 70% for the overall scores and a level not greater than 25% i'or - 1 or 0 scores for each individual item assessed should be set. Asterisks are used to mark the scores falling outside this range in Table 4 and lines are used on the bar charts, making identification easy. It can be seen from Table 4 and the graphic data that the request card and the 'product' do not score well overall. Looking further into the data reveals the areas where p o o r scoring is occurring. In particular, referring clinicians are poor at filling in bleep or extension numbers, printing their name next to their signature, selecting the appropriate method of transport and in filling in details of previous investigations_ Where relevant, the last menstrual period (LMP) section on the card was frequently not filled in. Radiologists are poor at adding diagnostic codes to the report. Radiographers do not perform well in marking their identity numbers on the films, or numbering the films to correspond with the index number on the film packet. The results of the secretarial/typist assessment in the report section are excellent, indicating accurate typing and spelling. In the assessment of the 'product'; the secretarial score shows poor performance in putting reports into folders and folders into film packets. Clerical assessment shows poor performance at placing bar codes on the front of film packets. In order to make more sense of the information provided by auditing in this way, data must be accumulated over m a n y months, and means and standard deviations applied to the scores. The regular assessment period should be selected for convenience and the results of this compared to the minimum standards and the long term means and standard deviations. Further detailed studies, based on these findings, have been initiated to determine the significance of the findings and to recommend changes in practice. Redesign of the request card and film packet is already underway. The current practice of typists being solely responsible for attaching reports to report folders is under reviewFurther studies are looking at request card filling by clinicians, film marking by radiographers, the number of unreported films leaving the departmenfand film retrieval rates from the film store. A repeat study at a later date will indicate the results of any changes in practice created by this study. A number of problems were apparent in this pilot study which are to be addressed before setting up the definitive study.
QUALITY
IN
A RADIOLOGICAL
++°I o
=ca 40 g t-
++++ol A
349
5 No attempt was made to assess the accuracy of the report, nor of the diagnostic code at the end of the report. It is important, however, that accuracy of reporting be monitored by some method and we feel a system of assessment should be included in future audit. 6 The minimum levels of standard have been set empirically and will be adjusted as experience and further data are obtained.
~ 60 g
0
DEPARTMENT
i
B
i
•
C
i
l D
ILltltlt E F Korner group
Sample
G
II
III
V
[ ] Departmental
Fig. 6 Comparisonof sample and departmental workload in terms of Korner group.
1 The source of requests in our sample did not closely reflect the source profile of the department as a whole. The proportions of out-patient, in-patient and GP referrals for our sample were 46%, 41% and 13% respectively, compared with the departmental workload of 40%, 27% and 13% respectively. No casualty films were included in our sample. 2 The distribution of Korner groups, shown in Fig. 6, demonstrates that although all Korner groups were covered, the distribution of investigations assessed did not reflect the workload of the department. It is intended that the computer be used to generate random samples of investigations which more closely reflect the departmental workload. 3 The failure to retrieve 38% of films requested does not compare favourably with a 3.6% failure rate for outpatient clinics in August 1989. This discrepancy may reflect the timing of the attempted retrieval at 1 month, when patients' films may still be with clinicians or have been requested for clinic appointments. 4 Objective assessment of a radiological report may be influenced by knowing who has written the report, leading to some bias. However, the bias is probably small.
It was estimated that the workload entailed in this study was approximately 20 h spent analysing the request card, image, film packet and report card, and approximately 2-3 h of clerical time to retrieve all the film packets. In this pilot study we have essentially been inward looking and only assessed those aspects which can be easily monitored within the department. We have not considered the patient and how they felt they were treated whilst in the department, nor have we assessed the response of the clinician to the report received. Inclusion of these aspects, together with waiting times within the department and delay times between the date of request and the date of reporting, will enable a fuller assessment of the service we are providing. It is planned to introduce these aspects into our audit system at a later date. The scoring system devised in this pilot study is easy to use and is expandable to include more items for assessment and finer tuning of the assessment. The data produced from the study were stored and manipulated easily on a computer_ Many data processing and graphics programs are readily available and should be chosen to suit the auditor and the system used. It is considered that audit of quality in a department of radiology is both possible and worthwhile and should be included as part of the routine departmental management process.
REFERENCES
Royal College of Radiologists (1989). Medical Audzt #7 Radiodiagnosis. Report of a Working Party of the Royal College of Radiologists, London. Korner, E (1982). The Department of Health and Social Security. First Report of the Steering Group on Health Services Information.
Chairman, Korner, E. HMSO Publications, London.