- Email: [email protected]
The Effect of Visual Hindsight Bias on Radiologist Perception
ARTICLE IN PRESS
Original Investigation
The Effect of Visual Hindsight Bias on Radiologist Perception Jacky Chen, BAppSc (Diagnostic Radiography)
Hons, Stephen Littlefair, PhD, Roger Bourne, PhD, Warren M. Reed, PhD
Rationale and Objectives: To measure the effect of visual hindsight bias on radiologists’ perception during chest radiograph pulmonary nodule detection. Materials and Methods: This was a prospective multi-observer study to assess the effect of hindsight bias on radiologists’ perception. Sixteen radiologists were asked to interpret 15 postero-anterior chest images containing a solitary lung nodule each consisting of 25 incremental levels of blur. Participants were requested initially to detect the nodule by reducing the blur of the images (foresight). They were then asked to increase the blur until the identified nodule was undetectable (hindsight). Participants then repeated the experiment, after being informed of the potential effects of hindsight bias and asked to counteract these effects. Participants were divided into two groups (experienced and less experienced) and the nodules were given different conspicuity ratings to determine the effect of expertise and task difficulty. Eye tracking technology was also utilised to capture visual search. Results: Wilcoxon analysis demonstrated significant differences between foresight and hindsight values of the radiologists (p = 0.02). However, after being informed of hindsight bias, these differences were no longer significant (p = 0.97). Friedman analysis also determined overall significance in the hindsight ratios between nodule conspicuities for both phases (phase 1: p = 0.02; phase 2: p = 0.02). There was no significance difference between the experienced and less experienced groups. Conclusion: This study demonstrated that radiologists exhibit hindsight bias but appeared to be able to compensate for this phenomenon once its effects were considered. Also, visual hindsight bias appears to be affected by task difficulty with a greater effect occurring with less conspicuous nodules. Key Words: Bias; Perception; Radiologists; Nodules; Malpractice. © 2019 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved. Abbreviations: CXR chest images, F1 foresight of experiment 1, F2 foresight of experiment 2, H1 hindsight of experiment 1, H2 hindsight of experiment 2, HR1 hindsight ratio of experiment 1, HR2 hindsight ratio of experiment 2, RANZCR Royal Australian and New Zealand College of Radiologists
INTRODUCTION indsight bias or the ‘knew-it-all-along effect’ (1), is a phenomenon that occurs when individuals, with knowledge of an outcome, perceive in retrospect, that an event was more likely to happen. This is where an observer’s cognitive processing fails to discount new information, which alters their perception and judgment, making them believe that the outcome would have predictably occurred in foresight (2,3). In regards to visual hindsight bias,
H
Acad Radiol 2019; &:1–8 From the Discipline of Medical Radiation Sciences, Faculty of Health Sciences, University of Sydney, Cumberland Campus, 75 East Street, Lidcombe, NSW 2141, Australia (J.C., R.B., W.M.R.); Discipline of Medical Imaging, Central Queensland University, Mackay, Queensland, Australia (S.L.); Medical Imaging Optimisation Perception Group, Discipline of Medical Radiation Sciences, Faculty of Health Sciences, University of Sydney, Lidcombe, New South Wales, Australia (R.B., W.M.R.). Received June 26, 2019; revised September 14, 2019; accepted September 25, 2019. Address correspondence to: J.C. e-mail: [email protected] © 2019 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.acra.2019.09.032
the analogy of ‘Where’s Waldo?’ (4) can be applied to explain how this phenomenon can also occur with images. This involves the problem observers experience in trying to forget or de-identify Waldo after it being initially difficult to localize him in the first instance (5). This can present issues in radiographic image interpretation where diagnostic information from a later examination can affect the interpretation of an earlier examination, possibly forming the basis for radiological medicolegal proceedings (6,7). Experts who are called upon to testify may be susceptible to the effects of hindsight bias (8,9). For example, this can be evident when an expert witness has prior knowledge of the patient’s imaging diagnosis and testifies that a tumor was visible on a previous radiograph that was initially reported as normal (10). Consequently, some cases that may not constitute malpractice, may be deemed to be such in the eyes of the justice system (11). Previous studies have demonstrated that health professionals can be subject to hindsight bias. A 1981 study found that physicians with outcome knowledge (hindsight bias) of a diagnosis would overestimate their ability to predict it in
1
ARTICLE IN PRESS CHEN ET AL
advance (12). The influence of outcome knowledge was also demonstrated in a screening program in 1983 that found that up to 90% of diagnosed lung carcinomas could be seen on retrospective review of chest radiographs that were initially interpreted as normal (13). These missed nodules were attributed to perceptual or technical errors, with studies demonstrating radiology error rates at around 30% (14,15). However, it can be argued that outcome knowledge may have directed radiologists to subsequently be able to detect these previously missed cancers (16). In a 2004 study (16), visual hindsight bias was investigated on university students using blurred images of celebrity faces that became increasingly clearer until recognised. After initial recognition the researchers incrementally blurred the images until the faces were no longer recognisable. The results demonstrated hindsight bias could occur for the visual perception of images of faces as participants consistently overestimated the blur level present when asked to recall at which point they continued to identify the celebrity. In this study, the more well-known celebrities exhibited less hindsight bias suggesting task difficulty may also play a role. These findings posed the question whether this phenomenon also occurs with radiologists and medical images. In the context of this research, hindsight bias is considered a perceptual phenomenon. In order to gauge these perceptual thresholds, it requires the use of established psychophysics methods such as the methods of limits (17). However, the use of this method can result in errors of anticipation and habituation, where participants respond incorrectly to the presented stimulus (18). Hence, similar to Harley et al. (16), this study will utilize the staircase method (19) to attempt to avoid those errors and to capture the perceptual thresholds in order to determine the hindsight bias readings. In spite of the lack of empirical data, the issue of hindsight bias can question the strength of an expert’s testimony in medicolegal litigation as they have received prior information on diagnosis before deciding that an abnormality should have been detected on a medical image (16,20,21). The perceived inability for ‘experts’ to place themselves in the position of the defendant and view evidence with “fresh eyes”, as if seeing the image for the first time, could represent this potential bias (22). Therefore, the purpose of this study was to prospectively measure the effect of visual hindsight bias upon radiologist perception when identifying pulmonary nodules.
Academic Radiology, Vol &, No &&, && 2019
with a minimum of 3 years and a maximum of 40 years board certification with RANZCR were included in the study. Participants were asked how many chest images (CXRs) they reported each year. Those who reported 3000 or more CXRs per year were considered ‘experienced’ (n = 7) and those who reported under 3000 CXRs per year were termed ‘less experienced’ (n = 9). A dataset of 15 de-identified adult postero-anterior CXRs (2048 £ 2048 matrix size, 0.175mm pixel size) was used throughout this study. Each image contained a solitary lung nodule and was randomly selected from an online database (23) via a random number generator. This online database contained 247 CXRs that can be used for medical imaging research studies (23). Each nodule was precategorised into different degrees of conspicuity from five (reasonably easily discernible) to one (extremely difficult) by 20 radiologists (23). Nodules of conspicuity 1 (Fig 1) and 2 (Fig 2) were excluded as the images needed to be challenging but not too difficult, given detection of the nodules contained in the images was necessary to achieve the experiment objective of obtaining hindsight readings. Therefore, the dataset used for this study consisted of five images from each of conspicuity 3, conspicuity 4 and conspicuity 5 (Fig 3). These nodules were selected to have been reasonably easily found by radiologists (23), rather than be mistaken for normal anatomy within a CXR. These images were processed and presented for reading in MATLAB (R2017a, MathWorks, Natick, Massachusetts) as follows. DICOM images were imported and converted to 1000 £ 1000 pixels using bicubic interpolation. This downsampling was performed to ensure the image display was not affected by the monitor resolution. Image degradation due to down-sampling to 1000 £ 1000 is not expected to affect the study results because: 1) all images contained nodules that could
MATERIALS AND METHODS This study investigated the impact of visual hindsight bias upon radiologists’ perception of nodules on adult chest images using sequential sharpening and blurring of images to identify and then de-identify chest nodules. Institutional ethics approval was granted (_2016/768_) and written informed consent was obtained from all participants for prospective data collection. Sixteen radiologists, at the 2017 Royal Australian and New Zealand College of Radiologists (RANZCR) conference 2
Figure 1. Sample of the nodules of conspicuity 1 not used in the study. The nodules have been labeled with a black arrow.
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019 EFFECT OF VISUAL HINDSIGHT BIAS ON RADIOLOGIST PERCEPTION
Figure 2. Sample of the nodules of conspicuity 2 not used in the study. The nodules have been labeled with a black arrow.
easily be located in the least blurred image; and 2) the reader decision about nodule visibility was always made based on strongly blurred images where the blurring level would make any image degradation effects due to the original down-sampling are assumed to be insignificant. Image blurring was performed using the ‘imfilter’ function and a Gaussian convolution filter of kernel size 35 £ 35. The sigma value of the kernel (created with ‘fspecial’) was incremented or decremented in steps of 0.8 or 1/0.8 from a minimum of 0.15 to a maximum of 40. Hence, our 15 cases each had 25 levels of blur increments creating a stack of images much like CT perceptual experiments (24). As these cases were presented twice, this equates to 750 individual images in total. Sample images at a range of blur levels (sigma values) are presented in figure 4. The images were displayed on a Viewsonic VG810b monitor (ViewSonic, Walnut, California) with a screen resolution of
1280 £ 1024 pixels using a graphics card (NVIDIA Quadro FX 560; Nvidia, Santa Clara, California) that exceeded the minimum recommendation of the American Association of Physicists in Medicine (25). Over the four days of the study, the monitor was calibrated to the Digital Imaging and Communications in Medicine gray-scale display function standard using Verilum software (Verilum; Image Smiths, Bethesda, Maryland) and luminance pod. Ambient lighting remained within 35 40 lux, as measured with a calibrated photometer (model 07 631; Nuclear Associates, Everett, Washington). An eye tracking system was also employed in the study to verify nodule location by the participants. This consisted of a single computer with a dual screen configuration. One screen was used by the researcher to ensure that the participant’s eye tracking data was optimal, while the other screen was used by the participant to view images. To accurately capture perception, each participant was monitored with a remote eye tracker (Tobii X50; Tobii Technology, Danderyd, Sweden). The inclusion and exclusion of the data was dependent on visual inspection of eye tracking recordings. All radiologists participated in a two phase experiment with all images being presented in a single reading session. Phase 2 was performed immediately after phase 1. Phase 1 Investigating Hindsight Bias: Each radiologist was asked to identify the nodule in the blurred CXRs that became increasingly clear in 25 increments under the control of the radiologist. The following instructions were read aloud to participants: ‘Each image will start very blurry and slowly become sharper on each mouse click. Click ‘nodule identified’ as soon as you recognise the nodule. After you have made your decision, the same image will become completely sharp. You will then have to increase the blur until the same identified nodule that you initially perceived is no longer visible. After selecting ‘nodule is no longer identified’, you will proceed to the next image. Each image contains one solitary lung nodule.’
Figure 3. Sample of the nodules of conspicuity 3, 4, and 5 used in the study. The nodules have been labeled with a black arrow.
3
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019
CHEN ET AL
Figure 4. Sample of a chest image (category 5) used in the experiment. Here, blur increments are shown with 40 being the maximum blur and 0.15 being the sharpest image.
Phase 2 Investigating Debiasing Hindsight: In this phase, radiologists were given the same image dataset and again were asked to indicate when a nodule was detected and undetected identically to phase 1. The images were presented in the same order, although participants were not informed of this. Prior to this second part, the following instructions were read aloud to participants: ‘This time I would like you to be aware of hindsight bias. Hindsight bias is when someone who knows the outcome of an event believes they would have predicted that outcome before it happened. Try to recall how blurry each image was when you first detected the nodule and when the image goes from clear to blurry, select ‘nodule is no longer identified’ at this point. You may have seen the nodule in a blurrier state than originally perceived previously. We believe this is caused by already knowing what the nodule looks like and where it is located. Please try to avoid this bias and be as accurate as you can.’ The data for each participant were exported into a Microsoft Excel spreadsheet, with each image having two values depending on the kernel sigma that the nodule was first seen for blurred to sharp (foresight-F) and last seen for sharp to blurred (hindsight-H). For phase 1, these were termed F1 and H1. For phase 2, these were termed F2 and H2; respectively. Hindsight ratios (HR) for each image were calculated by dividing the foresight by the hindsight values. An HR of less than one indicates hindsight bias (16). For instance, if a nodule was originally detected at a blur level of 13.11 but was de-identified at a blur level of 4
40, the HR would be 0.33 (13.11/40). These values were exported into GraphPadPrism version 7.00 for Windows (GraphPadPrism Software San Diego California USA) where a Shapiro-Wilk normality test was performed indicating that our data was positively skewed and hence was not normally distributed. Hence, the following Wilcoxon Matched Pairs comparisons were made:
F1 versus H1 F2 versus H2 F1 versus F2 H1 versus H2 HR1 versus HR2
Friedman tests and Wilcoxon Matched Pairs comparisons were also used to determine any significant differences between the values for the three levels of nodule conspicuity. Wilcoxon Matched Pairs and Mann-Whitney tests were utilized to determine if a radiologist experience affected hindsight bias. For all analysis a confidence interval of 95% was used, where p < 0.05 was an indicator of statistical significance. RESULTS The data acquired from the 16 participating radiologists were utilized for analysis. Data was only included provided the radiologist identified and de-identified the actual nodule through confirmation with eye tracking. The medians of the groups and p values for the comparisons of all
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019 EFFECT OF VISUAL HINDSIGHT BIAS ON RADIOLOGIST PERCEPTION
TABLE 1. Comparisons of Foresight, Hindsight and Hindsight Ratios of All Radiologists Wilcoxon Matched Pairs (n = 16)
p Value
Medians of Pairs
F1 vs H1 F2 vs H2 F1 vs F2 H1 vs H2 HR1 vs HR2
13.44 vs. 28.80 11.8 vs. 16.38 13.44 vs. 11.8 28.80 vs. 16.38 0.95 vs. 1
0.02 0.97 0.83 0.007 0.004
F1, foresight of phase 1; F2, foresight of phase 2; H1, hindsight of phase 1; H2, hindsight of phase 2; HR1, Hindsight ratio of phase 1; HR2, Hindsight ratio of phase 2.
radiologists (n = 16) are shown in table 1. For all analysis, the median was used. There was a significant difference between the foresight and hindsight values in the first phase (F1 = 13.44; H1 = 28.80, p = 0.02). The findings of the foresight and hindsight values in the second phase were not statistically different (F2 = 11.8; H2 = 16.38, p = 0.97). This was also the case between the foresight values of the two phases (F1 = 13.44; F2 = 11.8, p = 0.83), where the medians indicated no significant findings. However, there was significant reduction between the hindsight values of the two phases (H1 = 28.80; H1 = 16.38, p = 0.007). This translated into the significant reduction observed between the HR of the phases (HR 1 = 0.95, HR2 = 1.00, p = 0.004). Further analysis (Tables 2 and 3) focuses on the different nodule conspicuities to investigate whether hindsight bias is influenced by task difficulty. Table 2 indicates that Friedman analysis found an overall statistical difference in the HR between the conspicuities in both phases (Phase 1: p = 0.02, Phase 2: p = 0.017). TABLE 2. Summary of Friedman Comparison Between Nodule Conspicuity HR
Median hindsight ratios for phase 1 Interquartile range Friedman statistic
Conspicuity 3 (n = 16)
Conspicuity 4 (n = 16)
Conspicuity 5 (n = 16)
0.51
1
1
0.54
0.37
0
0.02 0.64
1.1
1
0.72
1.55
0.5
7.5 p value Median hindsight ratios for phase 2 Interquartile range Friedman statistic
8.2 p value 0.017
Table 3 demonstrates the results of Wilcoxon Matched Pairs analysis on the differing nodule conspicuities. Separate statistical comparisons established that these differences were demonstrated when comparing conspicuity 3 and 4 for both phases (Phase 1: p = 0.03, Phase 2: p = 0.01) and conspicuity 3 and 5 in the first phase (p = 0.03). When comparing conspicuities across the phases, statistical significance was only established for conspicuity 4 (p = 0.02). No other statistical differences were noted. Table 4 indicates that analysis found there was no statistical difference between ‘experienced’ and ‘less experienced’ radiologists. DISCUSSION This study enabled us to measure the impact of hindsight bias on radiologists interpreting pulmonary nodules. To the authors’ knowledge, this was the first reported study on visual hindsight bias using medical images and radiologists. We found evidence of visual hindsight bias in radiologists’ perception. When asked to indicate the level of blur where the identified nodule was no longer visible after initial detection, radiologists consistently de-identified the nodule in an image with an increased degree of blur than previously perceived when identified. However, once being informed of hindsight bias and its effects, they appeared able to compensate by reducing this bias. Also, visual hindsight bias appears to be affected by task difficulty with a greater effect occurring for more difficult images. Previous research (16) has demonstrated how the perception of images can be subject to hindsight bias. As a cognitive phenomenon, this appears true even for medical images and radiologists, as it was challenging for participants to ‘de-identify’ the visual features of each nodule after initial identification. Despite hindsight bias previously being considered very difficult to counteract (16), our radiologists appeared able to compensate for their hindsight bias once informed of its effects immediately prior to a reading session. Interestingly, the foresight values of both phases did not differ significantly. Hence, our radiologists did not see the same nodules any earlier in phase two, despite possibly knowing the nodules’ location. Therefore, this compensation may be attributed to the significant difference between the hindsight values of both phases rather than short term memory. Radiologists were able temper their personal predications and no longer ‘detect’ the identified nodules in a less blurry (sharper) image in phase two than phase one. Our findings are in broad agreement with early hindsight research by Hawkins and Hastie (2) who noted that newly acquired information is subconsciously integrated into a person’s knowledge of events that preceded the outcome. This is demonstrated in the form of visual hindsight bias for our radiologists. Two studies that investigated visual hindsight drew similar conclusions (13, 16). Muhm et al. (13) suggested that missed nodules became easier to visualize for radiologists once outcome knowledge was acquired. Whereas the findings of Harley et al. (16) demonstrated how observers systematically overestimated the degree of blur when recalling when celebrity faces 5
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019
CHEN ET AL
TABLE 3. Wilcoxon Matched Pairs comparisons of Median HR of Nodule Conspicuities Wilcoxon Matched Pairs Pair 1
Pair 2
Conspicuity 3 for phase 1 Conspicuity 3 for phase 1 Conspicuity 4 for phase 1 Conspicuity 3 for phase 2 Conspicuity 3 for phase 2 Conspicuity 4 for phase 2 Conspicuity 3 for phase 1 Conspicuity 4 for phase 1 Conspicuity 5 for phase 1
Conspicuity 4 for phase 1 Conspicuity 5 for phase 1 Conspicuity 5 for phase 1 Conspicuity 4 for phase 2 Conspicuity 5 for phase 2 Conspicuity 5 for phase 2 Conspicuity 3 for phase 2 Conspicuity 4 for phase 2 Conspicuity 5 for phase 2
TABLE 4. Analysis of the Experienced (n = 7) and Less Experienced (n = 9) Medians Comparisons
Medians of Pairs
p Value
Mann Whitney U test of experienced versus less experienced in phase 1 Mann Whitney U test of experienced versus less experienced in phase 2 Wilcoxon Matched Pairs test of experienced in phase 1 and 2 Wilcoxon Matched Pairs test of less experienced in phase 1 and 2
0.9 vs. 1
0.1
1 vs. 1
0.9
0.9 vs. 1
0.06
1 vs. 1
0.11
were first identified. Additionally, Harley et al. (16) and Fischhoff (26) attempted to reduce hindsight by informing participants about hindsight bias but found that this ‘debiasing’ was unsuccessful. Their results differed from our findings, which suggest that hindsight bias can be partly mitigated by radiologists. This could be attributed to the differing stimulus and methodologies utilized by researchers (16,26 28), and the constant exposure radiologists have in interpreting CXRs. Whilst facial recognition (16) and nodule identification are both visual tasks, facial recognition is about characterizing an object that has distinctive features, whereas there are many targets within a CXR that may resemble the appearance of a nodule. The outcome of phase 2 may also coincide with previous psychophysics research that has studied perceptual learning and its relationship with feedback (29 31), which indicated that there could be improvements in results due to cognitive instructions as demonstrated in our results. Our finding that hindsight bias is relative to task difficulty corresponded to the previous findings of Harley et al. (16), where celebrities that were harder to recognize exhibited more hindsight bias, making it more challenging for participants to de-identify the visual features of each celebrity face. In our study, nodules of conspicuity 3 were harder to de-identify once seen than the relatively easier image conspicuities of 4 and 5. No previous research was found in relation to radiology expertise and hindsight bias. Overall, our study demonstrated no evidence that hindsight bias was influenced by radiologist experience. 6
Medians of Pairs
p Value
0.51 vs. 1 0.51 vs. 1 1 vs. 1 0.64 vs. 1.1 0.64 vs. 1 1.1 vs 1 0.51 vs. 0.64 1 vs. 1.1 1 vs. 1
0.03 0.03 0.52 0.01 0.07 0.15 0.26 0.002 0.055
The implication of our research poses potential questions regarding the nature of errors in medical imaging. In the United States, studies in radiological malpractice litigation reveal that there is 50% likelihood that a radiologist will be a defendant by 60 years of age (32) with a missed lung cancer being the most common reason (33). Our study also questions if a proportion of the previously reported high percentage of missed cases visible on retrospective review of the total of radiology error rates (14,15) could be as a result of hindsight bias, and therefore, are not all perceptual or technical errors as previously assumed. The potential influence of hindsight bias may cause radiologists to fixate their attention towards the missed nodule with an overestimation of one’s judgment that overlooks other reasonable explanations (34). However, as shown by our research, there is the potential for radiologists’ hindsight bias to be mitigated, offering a possible solution. A potential reason that radiologists are able to compensate for hindsight bias is that it often needs to be factored directly into their clinical practice. Radiologists must consider all relevant studies when diagnosing patients. When interpreting new studies, they may run across older studies reported by a previous radiologist with apparent oversights. With the awareness of hindsight bias, a radiologist may then have to decide whether these missed findings are truly indicative of the other radiologist’s performance or could they also have been missed prospectively by the current reporting radiologist. Alternatively, when patients have had several studies in a short period, later reported examinations may need to be factored into an earlier unreported case with an awareness of hindsight bias. Whilst radiologists are aware of the issues surrounding hindsight bias, they may be unable to prevent its occurrence in medicolegal cases. In these legal settings there are many biases that may influence the expert witness (35). Currently, there is no proven way to mitigate the effects of hindsight bias. Hence, the practical advice that this paper offers is that expert witnesses could be instructed in the consequences of hindsight bias. Alternatively, these expert witnesses could be given multiple images, both normal and abnormal, with no knowledge of the prior history to diagnose rather than just seeing the image in question with often a diagnosis in deciding whether the defending radiologists should have seen the abnormality at the time. Perhaps within these malpractice lawsuits, there could be the inclusion of a neutral party
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019 EFFECT OF VISUAL HINDSIGHT BIAS ON RADIOLOGIST PERCEPTION
to caution against the pitfalls of hindsight bias. A potential limitation of our research is the lack of a washout period between the two phases, which poses questions on whether the second part was truly hindsight or memory effect (36). However, the radiologists did not detect the nodules any earlier in the second phase, which is inconsistent with the memory effect (36). Further limitations involve the lack of randomization or counterbalancing (37) of the image order, as the order was consistent for all participants. This was intended to ensure direct comparisons could be made between the two study parts. It was also noted that radiologists would not report blurred CXRs. However, the use of image blur provides 25 identical measurable increments that would not be obtainable with actual clinical images. Also as a laboratory experiment this would not be representative of the clinical environment. However, this study was not a diagnostic interpretation test but a visual perception task addressing the issue of visual hindsight in medical images that may have relevance to the clinical setting and the expert witness scenario. The participants were not being measured on their ability to find the nodule, but rather, if they could de-identify the nodule once seen. However, this is a preliminary study that focuses on visual hindsight bias for the first time in radiology. In summary, this study has provided evidence that radiologists were influenced by visual hindsight. However, radiologists appeared able to compensate for this bias when informed of its possible effects. Visual hindsight bias appears also to be affected by task difficulty with a greater bias occurring for more difficult images. Future research could investigate how radiologists consider their own biases when asked to predict how a naive peer (one who had not previously seen the images) would perform in the same experiment. The judgment of hindsight bias may differ between both self and for others. Further research could also examine the effects of hindsight bias on visual search through eye tracking, influence on different radiological specialities and radiological modalities, other methods of hindsight mitigation and a systematic review of medicolegal cases. ETHICS Institutional ethics approval was granted by the University of Sydney (_2016/768_) and written informed consent was obtained from all participants for prospective data collection. FUNDING The authors declare they have received no financial support in conjunction with the generation of their manuscript submission. ACKNOWLEDGMENTS The authors are grateful for all the radiologists who contributed their time to participate in this study. The authors would also like to acknowledge Dr Rob Heard for his advice in statistical analysis.
REFERENCES 1. Wood G. The knew-it-all-along effect. J Exp Psychol Hum Percept Perform 1978; 4:345–353. doi:10.1037/0096-1523.4.2.345. 2. Hawkins SA, Hastie R. Hindsight: biased judgments of past events after the outcomes are known. Psychol Bull 1990; 107:311–327. doi:10.1037/ 0033-2909.107.3.311. 3. Christensen-Szalanski JJJ, Willham CF. The hindsight bias: a meta-analysis. Organ Behav Hum Decis Process 1991; 48:147–168. doi:10.1016/ 0749-5978(91)90010-Q. 4. Handford M. Where’s Waldo? Boston, MA: Little, Brown and Company, 1987. 5. Hendrix RW. In defense of a missed lesion. Radiology 1995; 195:578–579. doi:10.1148/radiology.195.2.7724791. 6. Berlin L. Malpractice issues in radiology: perceptual errors. Am J Roentgenol 1996; 167:587–590. doi:10.2214/ajr.167.3.8751657. 7. Berlin L, Hendrix RW. Malpractice issues in radiology: perceptual errors and negligence. Am J Roentgenol 1998; 170:863–867. doi:10.2214/ ajr.170.4.9530024. 8. Berlin L. Malpractice issues in radiology: defending the “missed” radiographic diagnosis. Am J Roentgenol 2001; 176:317–322. doi:10.2214/ ajr.176.2.1760317. 9. Hugh TB, Tracy GD. Hindsight bias in medicolegal expert reports. Med J Aust 2002; 176. [cited 2018 June 5][277-8 pp.]. Available from: https:// www.mja.com.au/. 10. Berlin L. Hindsight bias. Am J Roentgenol 2000; 175:597–601. doi:10.2214/ajr.175.3.1750597. 11. Caldwell C, Seamone ER. Excusable neglect in malpractice suits against radiologists: a proposed jury instruction to recognize the human condition. Ann Health Law 2007; 16. [cited 2018 June 6][43-77 pp.]. Available from: http://lawecommons.luc.edu/annals/. 12. Arkes HR, Wortmann RL, Saville PD, et al. Hindsight bias among physicians weighing the likelihood of diagnoses. J Appl Psychol 1981; 66:252–254. doi:10.1037/0021-9010.66.2.252. 13. Muhm JR, Miller WE, Fontana RS, et al. Lung cancer detected during a screening program using four-month chest radiographs. Radiology 1983; 148:609–615. doi:10.1148/radiology.148.3.6308709. 14. Renfrew DL, Franken EA, Berbaum KS, et al. Error in radiology: classification and lessons in 182 cases presented at a problem case conference. Radiology 1992; 183:145–150. doi:10.1148/radiology.183.1.1549661. 15. Berlin L, Berlin JW. Malpractice and radiologists in Cook County, IL: trends in 20 years of litigation. Am J Roentgenol 1995; 165:781–788. doi:10.2214/ajr.165.4.7676967. 16. Harley EM, Carlsen KA, Loftus GR. The “saw-it-all-along” effect: demonstrations of visual hindsight bias. J Exp Psychol Learn Mem Cogn 2004; 30:960–968. doi:10.1037/0278-7393.30.5.960. 17. Fechner GT, Howes DH, Boring EG. Elements of psychophysics; translated by Helmut E. Adler; edited byDavis H. Howes, Edwin G. Boring. 1st ed: New York, London: Holt, Rinehart & Winston; 1966. 18. Salkind NJ. Encyclopedia of research design. 1st ed Thousand Oaks, California: SAGE Publications, Inc., 2010. 19. Cornsweet TN. The staircase-method in psychophysics. Am J Psychol 1962; 75:485–491. doi:10.2307/1419876. 20. Harley EM. Hindsight bias in legal decision making. Soc Cogn 2007; 25:48–63. doi:10.1521/soco.2007.25.1.48. 21. Hugh TB, Dekker SW. Hindsight bias and outcome bias in the social construction of medical negligence: a review. J Law Med 2009; 16. [cited 2018 June 5][846-57 pp.]. Available from: http://sites.thomsonreuters. com.au/journals/category/journal-of-law-and-medicine/. 22. Durand DJ, Robertson CT, Agarwal G, et al. Expert witness blinding strategies to mitigate bias in radiology malpractice cases: a comprehensive review of the literature. J Am Coll Radiol 2014; 11:868. doi:10.1016/ j.jacr.2014.05.001. 23. Shiraishi J, Katsuragawa S, Ikezoe J, et al. Development of a digital image database for chest radiographs with and without a lung nodule. Am J Roentgenol 2000; 174:71–74. doi:10.2214/ajr.174.1. 1740071. 24. Semelka RC, Ryan AF, Yonkers S, et al. Objective determination of standard of care: use of blind readings by external radiologists. Am J Roentgenol 2010; 195:429–431. doi:10.2214/AJR.09.3987. 25. Samei E, Badano A, Chakraborty D, et al. Assessment of display performance for medical imaging systems: executive summary of AAPM TG18 report. Med Phys 2005; 32:1205–1225. doi:10.1118/ 1.1861159.
7
ARTICLE IN PRESS CHEN ET AL
26. Fischhoff B. Hindsight is not equal to foresight: the effect of outcome knowledge on judgment under uncertainty. J Exp Psychol Hum Percept Perform 1975; 1:288–299. doi:10.1037/0096-1523.1.3.288. 27. Sanna LJ, Schwarz N. Debiasing the hindsight bias: the role of accessibility experiences and (mis)attributions. J Exp Soc Psychol 2003; 39:287–295. doi:10.1016/S0022-1031(02)00528-0. 28. Wu D-A, Shimojo S, Wang SW, et al. Shared visual attention reduces hindsight bias. Psychol Sci 2012; 23:1524–1533. 29. Petrov AA, Dosher BA, Lu Z-L. Comparable perceptual learning with and without feedback in non-stationary context: data and model. J Vis 2004; 4:306. doi:10.1167/4.8.306. 30. Seitz A, Watanabe T. A unified model for perceptual learning. Trends Cogn Sci 2005; 9:329–334. doi:10.1016/j.tics.2005.05.010. 31. Lengyel G, Fiser J. The relationship between initial threshold, learning, and generalization in perceptual learning. J Vis 2019; 19:1–17. doi:10.1167/19.4.28.
8
Academic Radiology, Vol &, No &&, && 2019
32. Baker SR, Whang JS, Luk L, et al. The demography of medical malpractice suits against radiologists. Radiology 2013; 266:539–547. doi:10.1148/radiol.12110971. 33. Baker SR, Patel RH, Yang L, et al. Malpractice suits in chest radiology: an evaluation of the histories of 8265 radiologists. J Thorac Imaging 2013; 28:388–391. doi:10.1097/RTI.0b013e3182a21be2. 34. Roese NJ, Vohs KD. Hindsight bias. Perspect Psychol Sci 2012; 7:411–426. doi:10.1177/1745691612454303. 35. Durand DJ, Robertson CT, Agarwal G, et al. Expert witness blinding strategies to mitigate bias in radiology malpractice cases: a comprehensive review of the literature. J American Coll Radio 2014; 11:868–873. doi:10.1016/j.jacr.2014.05.001. 36. Haygood TM, Qing Liu MA, Galvan EM, et al. Memory for previously viewed radiographs and the effect of prior knowledge of memory task. Acad Radiol 2013; 20:1598–1603. doi:10.1016/j.acra.2013.08.015. 37. Bernstein DM, Harley EM. Fluency misattribution and visual hindsight bias. Memory 2007; 15:548–560. doi:10.1080/09658210701390701.
Original Investigation
The Effect of Visual Hindsight Bias on Radiologist Perception Jacky Chen, BAppSc (Diagnostic Radiography)
Hons, Stephen Littlefair, PhD, Roger Bourne, PhD, Warren M. Reed, PhD
Rationale and Objectives: To measure the effect of visual hindsight bias on radiologists’ perception during chest radiograph pulmonary nodule detection. Materials and Methods: This was a prospective multi-observer study to assess the effect of hindsight bias on radiologists’ perception. Sixteen radiologists were asked to interpret 15 postero-anterior chest images containing a solitary lung nodule each consisting of 25 incremental levels of blur. Participants were requested initially to detect the nodule by reducing the blur of the images (foresight). They were then asked to increase the blur until the identified nodule was undetectable (hindsight). Participants then repeated the experiment, after being informed of the potential effects of hindsight bias and asked to counteract these effects. Participants were divided into two groups (experienced and less experienced) and the nodules were given different conspicuity ratings to determine the effect of expertise and task difficulty. Eye tracking technology was also utilised to capture visual search. Results: Wilcoxon analysis demonstrated significant differences between foresight and hindsight values of the radiologists (p = 0.02). However, after being informed of hindsight bias, these differences were no longer significant (p = 0.97). Friedman analysis also determined overall significance in the hindsight ratios between nodule conspicuities for both phases (phase 1: p = 0.02; phase 2: p = 0.02). There was no significance difference between the experienced and less experienced groups. Conclusion: This study demonstrated that radiologists exhibit hindsight bias but appeared to be able to compensate for this phenomenon once its effects were considered. Also, visual hindsight bias appears to be affected by task difficulty with a greater effect occurring with less conspicuous nodules. Key Words: Bias; Perception; Radiologists; Nodules; Malpractice. © 2019 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved. Abbreviations: CXR chest images, F1 foresight of experiment 1, F2 foresight of experiment 2, H1 hindsight of experiment 1, H2 hindsight of experiment 2, HR1 hindsight ratio of experiment 1, HR2 hindsight ratio of experiment 2, RANZCR Royal Australian and New Zealand College of Radiologists
INTRODUCTION indsight bias or the ‘knew-it-all-along effect’ (1), is a phenomenon that occurs when individuals, with knowledge of an outcome, perceive in retrospect, that an event was more likely to happen. This is where an observer’s cognitive processing fails to discount new information, which alters their perception and judgment, making them believe that the outcome would have predictably occurred in foresight (2,3). In regards to visual hindsight bias,
H
Acad Radiol 2019; &:1–8 From the Discipline of Medical Radiation Sciences, Faculty of Health Sciences, University of Sydney, Cumberland Campus, 75 East Street, Lidcombe, NSW 2141, Australia (J.C., R.B., W.M.R.); Discipline of Medical Imaging, Central Queensland University, Mackay, Queensland, Australia (S.L.); Medical Imaging Optimisation Perception Group, Discipline of Medical Radiation Sciences, Faculty of Health Sciences, University of Sydney, Lidcombe, New South Wales, Australia (R.B., W.M.R.). Received June 26, 2019; revised September 14, 2019; accepted September 25, 2019. Address correspondence to: J.C. e-mail: [email protected] © 2019 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.acra.2019.09.032
the analogy of ‘Where’s Waldo?’ (4) can be applied to explain how this phenomenon can also occur with images. This involves the problem observers experience in trying to forget or de-identify Waldo after it being initially difficult to localize him in the first instance (5). This can present issues in radiographic image interpretation where diagnostic information from a later examination can affect the interpretation of an earlier examination, possibly forming the basis for radiological medicolegal proceedings (6,7). Experts who are called upon to testify may be susceptible to the effects of hindsight bias (8,9). For example, this can be evident when an expert witness has prior knowledge of the patient’s imaging diagnosis and testifies that a tumor was visible on a previous radiograph that was initially reported as normal (10). Consequently, some cases that may not constitute malpractice, may be deemed to be such in the eyes of the justice system (11). Previous studies have demonstrated that health professionals can be subject to hindsight bias. A 1981 study found that physicians with outcome knowledge (hindsight bias) of a diagnosis would overestimate their ability to predict it in
1
ARTICLE IN PRESS CHEN ET AL
advance (12). The influence of outcome knowledge was also demonstrated in a screening program in 1983 that found that up to 90% of diagnosed lung carcinomas could be seen on retrospective review of chest radiographs that were initially interpreted as normal (13). These missed nodules were attributed to perceptual or technical errors, with studies demonstrating radiology error rates at around 30% (14,15). However, it can be argued that outcome knowledge may have directed radiologists to subsequently be able to detect these previously missed cancers (16). In a 2004 study (16), visual hindsight bias was investigated on university students using blurred images of celebrity faces that became increasingly clearer until recognised. After initial recognition the researchers incrementally blurred the images until the faces were no longer recognisable. The results demonstrated hindsight bias could occur for the visual perception of images of faces as participants consistently overestimated the blur level present when asked to recall at which point they continued to identify the celebrity. In this study, the more well-known celebrities exhibited less hindsight bias suggesting task difficulty may also play a role. These findings posed the question whether this phenomenon also occurs with radiologists and medical images. In the context of this research, hindsight bias is considered a perceptual phenomenon. In order to gauge these perceptual thresholds, it requires the use of established psychophysics methods such as the methods of limits (17). However, the use of this method can result in errors of anticipation and habituation, where participants respond incorrectly to the presented stimulus (18). Hence, similar to Harley et al. (16), this study will utilize the staircase method (19) to attempt to avoid those errors and to capture the perceptual thresholds in order to determine the hindsight bias readings. In spite of the lack of empirical data, the issue of hindsight bias can question the strength of an expert’s testimony in medicolegal litigation as they have received prior information on diagnosis before deciding that an abnormality should have been detected on a medical image (16,20,21). The perceived inability for ‘experts’ to place themselves in the position of the defendant and view evidence with “fresh eyes”, as if seeing the image for the first time, could represent this potential bias (22). Therefore, the purpose of this study was to prospectively measure the effect of visual hindsight bias upon radiologist perception when identifying pulmonary nodules.
Academic Radiology, Vol &, No &&, && 2019
with a minimum of 3 years and a maximum of 40 years board certification with RANZCR were included in the study. Participants were asked how many chest images (CXRs) they reported each year. Those who reported 3000 or more CXRs per year were considered ‘experienced’ (n = 7) and those who reported under 3000 CXRs per year were termed ‘less experienced’ (n = 9). A dataset of 15 de-identified adult postero-anterior CXRs (2048 £ 2048 matrix size, 0.175mm pixel size) was used throughout this study. Each image contained a solitary lung nodule and was randomly selected from an online database (23) via a random number generator. This online database contained 247 CXRs that can be used for medical imaging research studies (23). Each nodule was precategorised into different degrees of conspicuity from five (reasonably easily discernible) to one (extremely difficult) by 20 radiologists (23). Nodules of conspicuity 1 (Fig 1) and 2 (Fig 2) were excluded as the images needed to be challenging but not too difficult, given detection of the nodules contained in the images was necessary to achieve the experiment objective of obtaining hindsight readings. Therefore, the dataset used for this study consisted of five images from each of conspicuity 3, conspicuity 4 and conspicuity 5 (Fig 3). These nodules were selected to have been reasonably easily found by radiologists (23), rather than be mistaken for normal anatomy within a CXR. These images were processed and presented for reading in MATLAB (R2017a, MathWorks, Natick, Massachusetts) as follows. DICOM images were imported and converted to 1000 £ 1000 pixels using bicubic interpolation. This downsampling was performed to ensure the image display was not affected by the monitor resolution. Image degradation due to down-sampling to 1000 £ 1000 is not expected to affect the study results because: 1) all images contained nodules that could
MATERIALS AND METHODS This study investigated the impact of visual hindsight bias upon radiologists’ perception of nodules on adult chest images using sequential sharpening and blurring of images to identify and then de-identify chest nodules. Institutional ethics approval was granted (_2016/768_) and written informed consent was obtained from all participants for prospective data collection. Sixteen radiologists, at the 2017 Royal Australian and New Zealand College of Radiologists (RANZCR) conference 2
Figure 1. Sample of the nodules of conspicuity 1 not used in the study. The nodules have been labeled with a black arrow.
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019 EFFECT OF VISUAL HINDSIGHT BIAS ON RADIOLOGIST PERCEPTION
Figure 2. Sample of the nodules of conspicuity 2 not used in the study. The nodules have been labeled with a black arrow.
easily be located in the least blurred image; and 2) the reader decision about nodule visibility was always made based on strongly blurred images where the blurring level would make any image degradation effects due to the original down-sampling are assumed to be insignificant. Image blurring was performed using the ‘imfilter’ function and a Gaussian convolution filter of kernel size 35 £ 35. The sigma value of the kernel (created with ‘fspecial’) was incremented or decremented in steps of 0.8 or 1/0.8 from a minimum of 0.15 to a maximum of 40. Hence, our 15 cases each had 25 levels of blur increments creating a stack of images much like CT perceptual experiments (24). As these cases were presented twice, this equates to 750 individual images in total. Sample images at a range of blur levels (sigma values) are presented in figure 4. The images were displayed on a Viewsonic VG810b monitor (ViewSonic, Walnut, California) with a screen resolution of
1280 £ 1024 pixels using a graphics card (NVIDIA Quadro FX 560; Nvidia, Santa Clara, California) that exceeded the minimum recommendation of the American Association of Physicists in Medicine (25). Over the four days of the study, the monitor was calibrated to the Digital Imaging and Communications in Medicine gray-scale display function standard using Verilum software (Verilum; Image Smiths, Bethesda, Maryland) and luminance pod. Ambient lighting remained within 35 40 lux, as measured with a calibrated photometer (model 07 631; Nuclear Associates, Everett, Washington). An eye tracking system was also employed in the study to verify nodule location by the participants. This consisted of a single computer with a dual screen configuration. One screen was used by the researcher to ensure that the participant’s eye tracking data was optimal, while the other screen was used by the participant to view images. To accurately capture perception, each participant was monitored with a remote eye tracker (Tobii X50; Tobii Technology, Danderyd, Sweden). The inclusion and exclusion of the data was dependent on visual inspection of eye tracking recordings. All radiologists participated in a two phase experiment with all images being presented in a single reading session. Phase 2 was performed immediately after phase 1. Phase 1 Investigating Hindsight Bias: Each radiologist was asked to identify the nodule in the blurred CXRs that became increasingly clear in 25 increments under the control of the radiologist. The following instructions were read aloud to participants: ‘Each image will start very blurry and slowly become sharper on each mouse click. Click ‘nodule identified’ as soon as you recognise the nodule. After you have made your decision, the same image will become completely sharp. You will then have to increase the blur until the same identified nodule that you initially perceived is no longer visible. After selecting ‘nodule is no longer identified’, you will proceed to the next image. Each image contains one solitary lung nodule.’
Figure 3. Sample of the nodules of conspicuity 3, 4, and 5 used in the study. The nodules have been labeled with a black arrow.
3
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019
CHEN ET AL
Figure 4. Sample of a chest image (category 5) used in the experiment. Here, blur increments are shown with 40 being the maximum blur and 0.15 being the sharpest image.
Phase 2 Investigating Debiasing Hindsight: In this phase, radiologists were given the same image dataset and again were asked to indicate when a nodule was detected and undetected identically to phase 1. The images were presented in the same order, although participants were not informed of this. Prior to this second part, the following instructions were read aloud to participants: ‘This time I would like you to be aware of hindsight bias. Hindsight bias is when someone who knows the outcome of an event believes they would have predicted that outcome before it happened. Try to recall how blurry each image was when you first detected the nodule and when the image goes from clear to blurry, select ‘nodule is no longer identified’ at this point. You may have seen the nodule in a blurrier state than originally perceived previously. We believe this is caused by already knowing what the nodule looks like and where it is located. Please try to avoid this bias and be as accurate as you can.’ The data for each participant were exported into a Microsoft Excel spreadsheet, with each image having two values depending on the kernel sigma that the nodule was first seen for blurred to sharp (foresight-F) and last seen for sharp to blurred (hindsight-H). For phase 1, these were termed F1 and H1. For phase 2, these were termed F2 and H2; respectively. Hindsight ratios (HR) for each image were calculated by dividing the foresight by the hindsight values. An HR of less than one indicates hindsight bias (16). For instance, if a nodule was originally detected at a blur level of 13.11 but was de-identified at a blur level of 4
40, the HR would be 0.33 (13.11/40). These values were exported into GraphPadPrism version 7.00 for Windows (GraphPadPrism Software San Diego California USA) where a Shapiro-Wilk normality test was performed indicating that our data was positively skewed and hence was not normally distributed. Hence, the following Wilcoxon Matched Pairs comparisons were made:
F1 versus H1 F2 versus H2 F1 versus F2 H1 versus H2 HR1 versus HR2
Friedman tests and Wilcoxon Matched Pairs comparisons were also used to determine any significant differences between the values for the three levels of nodule conspicuity. Wilcoxon Matched Pairs and Mann-Whitney tests were utilized to determine if a radiologist experience affected hindsight bias. For all analysis a confidence interval of 95% was used, where p < 0.05 was an indicator of statistical significance. RESULTS The data acquired from the 16 participating radiologists were utilized for analysis. Data was only included provided the radiologist identified and de-identified the actual nodule through confirmation with eye tracking. The medians of the groups and p values for the comparisons of all
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019 EFFECT OF VISUAL HINDSIGHT BIAS ON RADIOLOGIST PERCEPTION
TABLE 1. Comparisons of Foresight, Hindsight and Hindsight Ratios of All Radiologists Wilcoxon Matched Pairs (n = 16)
p Value
Medians of Pairs
F1 vs H1 F2 vs H2 F1 vs F2 H1 vs H2 HR1 vs HR2
13.44 vs. 28.80 11.8 vs. 16.38 13.44 vs. 11.8 28.80 vs. 16.38 0.95 vs. 1
0.02 0.97 0.83 0.007 0.004
F1, foresight of phase 1; F2, foresight of phase 2; H1, hindsight of phase 1; H2, hindsight of phase 2; HR1, Hindsight ratio of phase 1; HR2, Hindsight ratio of phase 2.
radiologists (n = 16) are shown in table 1. For all analysis, the median was used. There was a significant difference between the foresight and hindsight values in the first phase (F1 = 13.44; H1 = 28.80, p = 0.02). The findings of the foresight and hindsight values in the second phase were not statistically different (F2 = 11.8; H2 = 16.38, p = 0.97). This was also the case between the foresight values of the two phases (F1 = 13.44; F2 = 11.8, p = 0.83), where the medians indicated no significant findings. However, there was significant reduction between the hindsight values of the two phases (H1 = 28.80; H1 = 16.38, p = 0.007). This translated into the significant reduction observed between the HR of the phases (HR 1 = 0.95, HR2 = 1.00, p = 0.004). Further analysis (Tables 2 and 3) focuses on the different nodule conspicuities to investigate whether hindsight bias is influenced by task difficulty. Table 2 indicates that Friedman analysis found an overall statistical difference in the HR between the conspicuities in both phases (Phase 1: p = 0.02, Phase 2: p = 0.017). TABLE 2. Summary of Friedman Comparison Between Nodule Conspicuity HR
Median hindsight ratios for phase 1 Interquartile range Friedman statistic
Conspicuity 3 (n = 16)
Conspicuity 4 (n = 16)
Conspicuity 5 (n = 16)
0.51
1
1
0.54
0.37
0
0.02 0.64
1.1
1
0.72
1.55
0.5
7.5 p value Median hindsight ratios for phase 2 Interquartile range Friedman statistic
8.2 p value 0.017
Table 3 demonstrates the results of Wilcoxon Matched Pairs analysis on the differing nodule conspicuities. Separate statistical comparisons established that these differences were demonstrated when comparing conspicuity 3 and 4 for both phases (Phase 1: p = 0.03, Phase 2: p = 0.01) and conspicuity 3 and 5 in the first phase (p = 0.03). When comparing conspicuities across the phases, statistical significance was only established for conspicuity 4 (p = 0.02). No other statistical differences were noted. Table 4 indicates that analysis found there was no statistical difference between ‘experienced’ and ‘less experienced’ radiologists. DISCUSSION This study enabled us to measure the impact of hindsight bias on radiologists interpreting pulmonary nodules. To the authors’ knowledge, this was the first reported study on visual hindsight bias using medical images and radiologists. We found evidence of visual hindsight bias in radiologists’ perception. When asked to indicate the level of blur where the identified nodule was no longer visible after initial detection, radiologists consistently de-identified the nodule in an image with an increased degree of blur than previously perceived when identified. However, once being informed of hindsight bias and its effects, they appeared able to compensate by reducing this bias. Also, visual hindsight bias appears to be affected by task difficulty with a greater effect occurring for more difficult images. Previous research (16) has demonstrated how the perception of images can be subject to hindsight bias. As a cognitive phenomenon, this appears true even for medical images and radiologists, as it was challenging for participants to ‘de-identify’ the visual features of each nodule after initial identification. Despite hindsight bias previously being considered very difficult to counteract (16), our radiologists appeared able to compensate for their hindsight bias once informed of its effects immediately prior to a reading session. Interestingly, the foresight values of both phases did not differ significantly. Hence, our radiologists did not see the same nodules any earlier in phase two, despite possibly knowing the nodules’ location. Therefore, this compensation may be attributed to the significant difference between the hindsight values of both phases rather than short term memory. Radiologists were able temper their personal predications and no longer ‘detect’ the identified nodules in a less blurry (sharper) image in phase two than phase one. Our findings are in broad agreement with early hindsight research by Hawkins and Hastie (2) who noted that newly acquired information is subconsciously integrated into a person’s knowledge of events that preceded the outcome. This is demonstrated in the form of visual hindsight bias for our radiologists. Two studies that investigated visual hindsight drew similar conclusions (13, 16). Muhm et al. (13) suggested that missed nodules became easier to visualize for radiologists once outcome knowledge was acquired. Whereas the findings of Harley et al. (16) demonstrated how observers systematically overestimated the degree of blur when recalling when celebrity faces 5
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019
CHEN ET AL
TABLE 3. Wilcoxon Matched Pairs comparisons of Median HR of Nodule Conspicuities Wilcoxon Matched Pairs Pair 1
Pair 2
Conspicuity 3 for phase 1 Conspicuity 3 for phase 1 Conspicuity 4 for phase 1 Conspicuity 3 for phase 2 Conspicuity 3 for phase 2 Conspicuity 4 for phase 2 Conspicuity 3 for phase 1 Conspicuity 4 for phase 1 Conspicuity 5 for phase 1
Conspicuity 4 for phase 1 Conspicuity 5 for phase 1 Conspicuity 5 for phase 1 Conspicuity 4 for phase 2 Conspicuity 5 for phase 2 Conspicuity 5 for phase 2 Conspicuity 3 for phase 2 Conspicuity 4 for phase 2 Conspicuity 5 for phase 2
TABLE 4. Analysis of the Experienced (n = 7) and Less Experienced (n = 9) Medians Comparisons
Medians of Pairs
p Value
Mann Whitney U test of experienced versus less experienced in phase 1 Mann Whitney U test of experienced versus less experienced in phase 2 Wilcoxon Matched Pairs test of experienced in phase 1 and 2 Wilcoxon Matched Pairs test of less experienced in phase 1 and 2
0.9 vs. 1
0.1
1 vs. 1
0.9
0.9 vs. 1
0.06
1 vs. 1
0.11
were first identified. Additionally, Harley et al. (16) and Fischhoff (26) attempted to reduce hindsight by informing participants about hindsight bias but found that this ‘debiasing’ was unsuccessful. Their results differed from our findings, which suggest that hindsight bias can be partly mitigated by radiologists. This could be attributed to the differing stimulus and methodologies utilized by researchers (16,26 28), and the constant exposure radiologists have in interpreting CXRs. Whilst facial recognition (16) and nodule identification are both visual tasks, facial recognition is about characterizing an object that has distinctive features, whereas there are many targets within a CXR that may resemble the appearance of a nodule. The outcome of phase 2 may also coincide with previous psychophysics research that has studied perceptual learning and its relationship with feedback (29 31), which indicated that there could be improvements in results due to cognitive instructions as demonstrated in our results. Our finding that hindsight bias is relative to task difficulty corresponded to the previous findings of Harley et al. (16), where celebrities that were harder to recognize exhibited more hindsight bias, making it more challenging for participants to de-identify the visual features of each celebrity face. In our study, nodules of conspicuity 3 were harder to de-identify once seen than the relatively easier image conspicuities of 4 and 5. No previous research was found in relation to radiology expertise and hindsight bias. Overall, our study demonstrated no evidence that hindsight bias was influenced by radiologist experience. 6
Medians of Pairs
p Value
0.51 vs. 1 0.51 vs. 1 1 vs. 1 0.64 vs. 1.1 0.64 vs. 1 1.1 vs 1 0.51 vs. 0.64 1 vs. 1.1 1 vs. 1
0.03 0.03 0.52 0.01 0.07 0.15 0.26 0.002 0.055
The implication of our research poses potential questions regarding the nature of errors in medical imaging. In the United States, studies in radiological malpractice litigation reveal that there is 50% likelihood that a radiologist will be a defendant by 60 years of age (32) with a missed lung cancer being the most common reason (33). Our study also questions if a proportion of the previously reported high percentage of missed cases visible on retrospective review of the total of radiology error rates (14,15) could be as a result of hindsight bias, and therefore, are not all perceptual or technical errors as previously assumed. The potential influence of hindsight bias may cause radiologists to fixate their attention towards the missed nodule with an overestimation of one’s judgment that overlooks other reasonable explanations (34). However, as shown by our research, there is the potential for radiologists’ hindsight bias to be mitigated, offering a possible solution. A potential reason that radiologists are able to compensate for hindsight bias is that it often needs to be factored directly into their clinical practice. Radiologists must consider all relevant studies when diagnosing patients. When interpreting new studies, they may run across older studies reported by a previous radiologist with apparent oversights. With the awareness of hindsight bias, a radiologist may then have to decide whether these missed findings are truly indicative of the other radiologist’s performance or could they also have been missed prospectively by the current reporting radiologist. Alternatively, when patients have had several studies in a short period, later reported examinations may need to be factored into an earlier unreported case with an awareness of hindsight bias. Whilst radiologists are aware of the issues surrounding hindsight bias, they may be unable to prevent its occurrence in medicolegal cases. In these legal settings there are many biases that may influence the expert witness (35). Currently, there is no proven way to mitigate the effects of hindsight bias. Hence, the practical advice that this paper offers is that expert witnesses could be instructed in the consequences of hindsight bias. Alternatively, these expert witnesses could be given multiple images, both normal and abnormal, with no knowledge of the prior history to diagnose rather than just seeing the image in question with often a diagnosis in deciding whether the defending radiologists should have seen the abnormality at the time. Perhaps within these malpractice lawsuits, there could be the inclusion of a neutral party
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019 EFFECT OF VISUAL HINDSIGHT BIAS ON RADIOLOGIST PERCEPTION
to caution against the pitfalls of hindsight bias. A potential limitation of our research is the lack of a washout period between the two phases, which poses questions on whether the second part was truly hindsight or memory effect (36). However, the radiologists did not detect the nodules any earlier in the second phase, which is inconsistent with the memory effect (36). Further limitations involve the lack of randomization or counterbalancing (37) of the image order, as the order was consistent for all participants. This was intended to ensure direct comparisons could be made between the two study parts. It was also noted that radiologists would not report blurred CXRs. However, the use of image blur provides 25 identical measurable increments that would not be obtainable with actual clinical images. Also as a laboratory experiment this would not be representative of the clinical environment. However, this study was not a diagnostic interpretation test but a visual perception task addressing the issue of visual hindsight in medical images that may have relevance to the clinical setting and the expert witness scenario. The participants were not being measured on their ability to find the nodule, but rather, if they could de-identify the nodule once seen. However, this is a preliminary study that focuses on visual hindsight bias for the first time in radiology. In summary, this study has provided evidence that radiologists were influenced by visual hindsight. However, radiologists appeared able to compensate for this bias when informed of its possible effects. Visual hindsight bias appears also to be affected by task difficulty with a greater bias occurring for more difficult images. Future research could investigate how radiologists consider their own biases when asked to predict how a naive peer (one who had not previously seen the images) would perform in the same experiment. The judgment of hindsight bias may differ between both self and for others. Further research could also examine the effects of hindsight bias on visual search through eye tracking, influence on different radiological specialities and radiological modalities, other methods of hindsight mitigation and a systematic review of medicolegal cases. ETHICS Institutional ethics approval was granted by the University of Sydney (_2016/768_) and written informed consent was obtained from all participants for prospective data collection. FUNDING The authors declare they have received no financial support in conjunction with the generation of their manuscript submission. ACKNOWLEDGMENTS The authors are grateful for all the radiologists who contributed their time to participate in this study. The authors would also like to acknowledge Dr Rob Heard for his advice in statistical analysis.
REFERENCES 1. Wood G. The knew-it-all-along effect. J Exp Psychol Hum Percept Perform 1978; 4:345–353. doi:10.1037/0096-1523.4.2.345. 2. Hawkins SA, Hastie R. Hindsight: biased judgments of past events after the outcomes are known. Psychol Bull 1990; 107:311–327. doi:10.1037/ 0033-2909.107.3.311. 3. Christensen-Szalanski JJJ, Willham CF. The hindsight bias: a meta-analysis. Organ Behav Hum Decis Process 1991; 48:147–168. doi:10.1016/ 0749-5978(91)90010-Q. 4. Handford M. Where’s Waldo? Boston, MA: Little, Brown and Company, 1987. 5. Hendrix RW. In defense of a missed lesion. Radiology 1995; 195:578–579. doi:10.1148/radiology.195.2.7724791. 6. Berlin L. Malpractice issues in radiology: perceptual errors. Am J Roentgenol 1996; 167:587–590. doi:10.2214/ajr.167.3.8751657. 7. Berlin L, Hendrix RW. Malpractice issues in radiology: perceptual errors and negligence. Am J Roentgenol 1998; 170:863–867. doi:10.2214/ ajr.170.4.9530024. 8. Berlin L. Malpractice issues in radiology: defending the “missed” radiographic diagnosis. Am J Roentgenol 2001; 176:317–322. doi:10.2214/ ajr.176.2.1760317. 9. Hugh TB, Tracy GD. Hindsight bias in medicolegal expert reports. Med J Aust 2002; 176. [cited 2018 June 5][277-8 pp.]. Available from: https:// www.mja.com.au/. 10. Berlin L. Hindsight bias. Am J Roentgenol 2000; 175:597–601. doi:10.2214/ajr.175.3.1750597. 11. Caldwell C, Seamone ER. Excusable neglect in malpractice suits against radiologists: a proposed jury instruction to recognize the human condition. Ann Health Law 2007; 16. [cited 2018 June 6][43-77 pp.]. Available from: http://lawecommons.luc.edu/annals/. 12. Arkes HR, Wortmann RL, Saville PD, et al. Hindsight bias among physicians weighing the likelihood of diagnoses. J Appl Psychol 1981; 66:252–254. doi:10.1037/0021-9010.66.2.252. 13. Muhm JR, Miller WE, Fontana RS, et al. Lung cancer detected during a screening program using four-month chest radiographs. Radiology 1983; 148:609–615. doi:10.1148/radiology.148.3.6308709. 14. Renfrew DL, Franken EA, Berbaum KS, et al. Error in radiology: classification and lessons in 182 cases presented at a problem case conference. Radiology 1992; 183:145–150. doi:10.1148/radiology.183.1.1549661. 15. Berlin L, Berlin JW. Malpractice and radiologists in Cook County, IL: trends in 20 years of litigation. Am J Roentgenol 1995; 165:781–788. doi:10.2214/ajr.165.4.7676967. 16. Harley EM, Carlsen KA, Loftus GR. The “saw-it-all-along” effect: demonstrations of visual hindsight bias. J Exp Psychol Learn Mem Cogn 2004; 30:960–968. doi:10.1037/0278-7393.30.5.960. 17. Fechner GT, Howes DH, Boring EG. Elements of psychophysics; translated by Helmut E. Adler; edited byDavis H. Howes, Edwin G. Boring. 1st ed: New York, London: Holt, Rinehart & Winston; 1966. 18. Salkind NJ. Encyclopedia of research design. 1st ed Thousand Oaks, California: SAGE Publications, Inc., 2010. 19. Cornsweet TN. The staircase-method in psychophysics. Am J Psychol 1962; 75:485–491. doi:10.2307/1419876. 20. Harley EM. Hindsight bias in legal decision making. Soc Cogn 2007; 25:48–63. doi:10.1521/soco.2007.25.1.48. 21. Hugh TB, Dekker SW. Hindsight bias and outcome bias in the social construction of medical negligence: a review. J Law Med 2009; 16. [cited 2018 June 5][846-57 pp.]. Available from: http://sites.thomsonreuters. com.au/journals/category/journal-of-law-and-medicine/. 22. Durand DJ, Robertson CT, Agarwal G, et al. Expert witness blinding strategies to mitigate bias in radiology malpractice cases: a comprehensive review of the literature. J Am Coll Radiol 2014; 11:868. doi:10.1016/ j.jacr.2014.05.001. 23. Shiraishi J, Katsuragawa S, Ikezoe J, et al. Development of a digital image database for chest radiographs with and without a lung nodule. Am J Roentgenol 2000; 174:71–74. doi:10.2214/ajr.174.1. 1740071. 24. Semelka RC, Ryan AF, Yonkers S, et al. Objective determination of standard of care: use of blind readings by external radiologists. Am J Roentgenol 2010; 195:429–431. doi:10.2214/AJR.09.3987. 25. Samei E, Badano A, Chakraborty D, et al. Assessment of display performance for medical imaging systems: executive summary of AAPM TG18 report. Med Phys 2005; 32:1205–1225. doi:10.1118/ 1.1861159.
7
ARTICLE IN PRESS CHEN ET AL
26. Fischhoff B. Hindsight is not equal to foresight: the effect of outcome knowledge on judgment under uncertainty. J Exp Psychol Hum Percept Perform 1975; 1:288–299. doi:10.1037/0096-1523.1.3.288. 27. Sanna LJ, Schwarz N. Debiasing the hindsight bias: the role of accessibility experiences and (mis)attributions. J Exp Soc Psychol 2003; 39:287–295. doi:10.1016/S0022-1031(02)00528-0. 28. Wu D-A, Shimojo S, Wang SW, et al. Shared visual attention reduces hindsight bias. Psychol Sci 2012; 23:1524–1533. 29. Petrov AA, Dosher BA, Lu Z-L. Comparable perceptual learning with and without feedback in non-stationary context: data and model. J Vis 2004; 4:306. doi:10.1167/4.8.306. 30. Seitz A, Watanabe T. A unified model for perceptual learning. Trends Cogn Sci 2005; 9:329–334. doi:10.1016/j.tics.2005.05.010. 31. Lengyel G, Fiser J. The relationship between initial threshold, learning, and generalization in perceptual learning. J Vis 2019; 19:1–17. doi:10.1167/19.4.28.
8
Academic Radiology, Vol &, No &&, && 2019
32. Baker SR, Whang JS, Luk L, et al. The demography of medical malpractice suits against radiologists. Radiology 2013; 266:539–547. doi:10.1148/radiol.12110971. 33. Baker SR, Patel RH, Yang L, et al. Malpractice suits in chest radiology: an evaluation of the histories of 8265 radiologists. J Thorac Imaging 2013; 28:388–391. doi:10.1097/RTI.0b013e3182a21be2. 34. Roese NJ, Vohs KD. Hindsight bias. Perspect Psychol Sci 2012; 7:411–426. doi:10.1177/1745691612454303. 35. Durand DJ, Robertson CT, Agarwal G, et al. Expert witness blinding strategies to mitigate bias in radiology malpractice cases: a comprehensive review of the literature. J American Coll Radio 2014; 11:868–873. doi:10.1016/j.jacr.2014.05.001. 36. Haygood TM, Qing Liu MA, Galvan EM, et al. Memory for previously viewed radiographs and the effect of prior knowledge of memory task. Acad Radiol 2013; 20:1598–1603. doi:10.1016/j.acra.2013.08.015. 37. Bernstein DM, Harley EM. Fluency misattribution and visual hindsight bias. Memory 2007; 15:548–560. doi:10.1080/09658210701390701.