- Email: [email protected]
Analysis of the treatment plan evaluation process in radiotherapy through eye tracking
ZEMEDI-10736; No. of Pages 8
ARTICLE IN PRESS
ORIGINAL PAPER
Analysis of the treatment plan evaluation process in radiotherapy through eye tracking A. Kyroudi 1 , K. Petersson 1 , M. Ozsahin 2 , J. Bourhis 2 , F. Bochud 1 , R. Moeckli 1,∗ 1 2
Institute of Radiation Physics (IRA), Lausanne University Hospital, Lausanne, Switzerland Department of Radiation Oncology, Lausanne University Hospital, Lausanne, Switzerland
Received 19 September 2017; accepted 21 November 2017
Abstract Background and purpose: Treatment plan evaluation is a clinical decision-making problem that involves visual search and analysis in a contextually rich environment, including delineated structures and isodose lines superposed on CT data. It is a two-step process that includes visual analysis and clinical reasoning. In this work, we used eye tracking methods to gain more knowledge about the treatment plan evaluation process in radiation therapy. Materials and methods: Dose distributions on a single transverse slice of ten prostate cancer treatment plans were presented to eight decision makers. Their eye movements and fixations were recorded with an EyeLink1000 remote eye-tracker. Total evaluation time, dwell time, number and duration of fixations on pre-segmented areas of interest were measured. Results: The main structures receiving more and longer fixations (PTV, rectum, bladder) correspond to the main trade-offs evaluated in a typical prostate plan. Radiation oncologists made more fixations on the main structures compared to the medical physicists. Radiation oncologists fixated longer on the rectum when visited for the first time, while medical physicists fixated longer on the bladder. Conclusion: Our results quantify differences in the visual evaluation patterns between radiation oncologists and medical physicists, which indicate differences in their decision making strategies.
∗ Corresponding
Analyse des Evaluierungsprozesses eines Behandlungsplans der Strahlentherapie anhand von Eye Tracking Zusammenfassung Hintergrund und Zweck: Die Evaluierung eines Behandlungsplans ist ein klinischer Entscheidungsprozess, welcher eine visuellen Suche und Analyse in einer komplizierten Umgebung beinhaltet, mit abgegrenzten Strukturen und Isodosislinien, welche CT-Daten überlagert sind. Es ist ein zweistufiger Prozess, der visuelle Analyse und klinische Erfahrung kombiniert. Für dieses Projekt haben wir Eye-Tracking-Methoden verwendet, um mehr Wissen über die Behandlung und deren Beurteilungsprozess in der Strahlentherapie zu gewinnen. Material und Methoden: Dosis-Verteilungen auf einer einzelnen transversalen Ebene von zehn ProstatakrebsBehandlungsplänen wurden acht Personen zur Auswertung präsentiert. Ihre Augenbewegungen und Fixierungen wurden mit einem EyeLink1000 Remote-Eye-Tracker aufgezeichnet. Die gesamte Evaluierungszeit, Verweilzeit, Anzahl und Dauer der Fokussierung an vorsegmentierten Bereichen von Interesse wurden gemessen. Resultate: Die Hauptstrukturen, welche mehrere und längere visuelle Fokussierungen erhalten (PTV, Rektum, Blase), entsprechen den wichtigsten Vorkommnissen, die in einem typischen Prostataplan evaluiert wurden.
author: Raphaël Moeckli, Institute of Radiation Physics, Rue du Grand-Pré 1, CH-1007 Lausanne, Switzerland. E-mail: [email protected] (R. Moeckli).
Z. Med. Phys. xxx (2017) xxx–xxx https://doi.org/10.1016/j.zemedi.2017.11.002 www.elsevier.com/locate/zemedi
ARTICLE IN PRESS 2
A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
Im Vergleich zu den Medizinphysikern haben die Strahlenonkologen mehr visuelle Fokussierungen an den Hauptstrukturen vorgenommen. Strahlenonkologen hatten eine visuelle längere Fokussierung auf das Rektum, während Medizinphysiker sich länger visuell auf die Blase konzentrieren. Schlussfolgerung: Unsere Ergebnisse quantifizieren Unterschiede in den visuellen Bewertungsmustern zwischen Strahlentherapeuten und Medizinphysikern, die auf Unterschiede in ihren Entscheidungsstrategien hinweisen. Keywords: Prostate, Decision making, Plan quality, Eye tracking
Introduction Treatment plan evaluation in radiation therapy is mainly a clinical decision-making process that involves visual perception and clinical reasoning [1]. The task is performed by radiation oncologists and medical physicists, who are often not fully consciously aware of how they observe and interpret clinical data or how they perform complex clinical assessments [2,3]. A large part of the treatment plan evaluation process involves a visual search and analysis of CT images, overlapped with dose distributions and structure delineations. Due to our limited working memory capacity and in order to avoid information overload in such complex visual scenes, observers selectively focus on the most informative parts of the image while ignoring irrelevant details [4]. This selection is guided by bottom-up salient visual cues, such as color, orientation, and size of the object, but also by the context of the visual scene [5,6]. In parallel, top-down bias signals, which depend on the intentions, expectations, and goals of the decision maker, also influence visual search. Shifts of attention to different parts of the image are accompanied by eye movements [7,8], which indicate the type of information searched and processed, along with the cognitive processes that take place during a task [9,10]. Eye movement behavior during visual search tasks is of great interest in cognitive science [11,12]. During interaction with an image, the human eyes make a series of fixations (aggregated gazepoints in which the eyes remain relatively stable and focused on a particular area), and saccades (rapid eye movements that occur from one fixation to another) [13,14]. The role of saccades is to bring the fovea, the high resolution central part of the retina, to another portion of the visual scene where interesting information may occur [15]. The location of the fixation that follows a saccade is indicative of the type of information available and its duration is indicative of how much information can be acquired.
Schlüsselwörter: Prostata, Entscheidung, Behandlungsplan qualität, Eye tracking
A common technique of recording spatial information, representative of where the eyes were pointed at a certain moment, is eye tracking [15,16]. Information resulting from eye tracking data analysis can provide insights into the type of information sought by the participant during an evaluation task [17]. Eye tracking has already been widely used for research in the medical domain, in order to identify visual behaviors through eye movements [18–21]. Particularly, in radiology it has been used to explore visual search patterns when searching for lung nodules on radiographs, breast lesions on mammographies, or fractures on radiographs [22–25]. Different search patterns between experts and novices have been demonstrated and visually guided learning has been shown to improve novice radiographers’ detection of pulmonary nodules in Xrays [26,27]. In an approach similar to what has been done in radiology since the 1970s, we used eye tracking techniques to enhance the understanding of how decision makers interact with the given visual information during treatment plan evaluation [28–31]. Analyzing eye movements enabled us to recognize the areas of the treatment plan on which the decision makers focus and also allowed us to identify different visual behaviors between radiation oncologists and medical physicists during the treatment plan evaluation process.
Materials and methods Four radiation oncologists (ROs) and four medical physicists (MPs) from the same radiotherapy department participated as decision makers (DMs) in the study. Although all of them were familiar with prostate treatment plans, their area of expertise and experience level varied (from none up to several prostate plan evaluations per week). They were asked to evaluate ten prostate cancer cases as follows: “Please evaluate the quality of the plan, based on the single transversal slice of the treatment plan, on a scale from 1 to 5, with 1 representing a totally unacceptable plan, 3 an acceptable plan, and 5 an
ARTICLE IN PRESS A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
3
Figure 1. An example of a transverse image slice of a treatment plan shown to decision makers during the eye tracking experiment.
excellent plan. Grades 2 and 4 represent intermediate assessments”. The task was framed in such a way as to engage the DMs into a natural process of decision-making concerning the plan quality. Each DM received the explanation of the experimental process and performed a brief training session with six plan evaluations, which were mimicking but not included in the real experiment. We used the Velocity software (Varian Medical Systems Inc., Palo Alto, CA, USA) to present a single transverse image slice of the plan on a medical color LCD monitor (RadiForce MX210, EIZO) with capture rate of 60 Hz, without any possible patient or treatment machine identifiers (Fig. 1). Each image was a CT scan with delineated anatomical structures (Clinical Target Volume (CTV), Planning Target Volume (PTV), rectum, bladder, and femoral heads), isodose lines corresponding to various dose levels received around the PTV and legends with explanatory information. Prostate plans were selected due to their common occurrence in radiotherapy and their anatomical simplicity. The ten plans were created with TomoTherapy (Accuray Inc, Sunnyvale, CA, USA) or Monaco (Elekta AB, Stockholm, Sweden) TPS, using the CT data of patients previously treated in our clinic. The plans of the lowest quality (plans 5, 6 and 8) did not respect dose constraints and requirements for the organs at risk (OARs) and the PTV, as defined by the EORTC [32,33]. Areas of interest (AOIs) were defined for each of the ten plans and involved the main structures of the patient’s geometry as they were delineated on the treatment plan, i.e. PTV, CTV, rectum, bladder, right and left femoral head, but it also included hotspots. The DMs’ eye movements were collected using an EyeLink1000 (SR Research Ltd., Mississauga, Ontario, Canada),
with a 500 Hz sampling rate. The remote option of the eye tracking system (without a fixed chinrest) was preferred in order to simulate routine clinical practice conditions as much as possible. The gaze position was determined through pupil and corneal reflection created by infrared illumination, with an average accuracy of 0.5◦ , corresponding to 6 mm resolution on the image for our setup. Fixations were formed by grouping x and y coordinates of the raw data, which were selected with a velocity-based algorithm that discriminates fixations and saccades based on an angular velocity threshold (30◦ /s) (EyeLink1000, SR Research Ltd., Mississauga, Ontario, Canada). The chosen threshold minimized the detection of short fixations (less than 100ms). Fixations belonging to two overlapping AOIs (e.g. PTV overlapping with bladder or rectum) were assigned to both regions. The Kolmogorov–Smirnov test was used to examine if the eye tracking results were normally distributed. Comparisons between the two groups of DMs (ROs and MPs) were examined with the use of the Mann–Whitney U test. The Pearson Chi Square (normally distributed data), or the Fisher’s exact test (non-normally distributed data) was used to examine differences in the plan evaluation results between the two groups of DMs. The significance level (α) chosen was 5% and the data analysis was carried out with the use of SPSS v.21.0.
Results ROs and MPs show some different trends in their evaluations, with only one RO evaluating one plan as totally unacceptable and only one MP evaluating one plan as excellent (Fig. 2). However, there was no significant difference
ARTICLE IN PRESS 4
A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
Figure 2. Treatment plan assessments of the ten plans made by the four radiation oncologists (ROs) and the four medical physicists (MPs).
regarding the total number of accepted or rejected plans among the two groups of DMs (Pearson Chi Square = 0.879, p = 0.348). The plan evaluated by a RO as being totally unacceptable (Grade 1) was a plan where the PTV was not covered by the 95% isodose, while for the MPs the plans considered as totally unacceptable were mainly plans with poor bladder sparing. The plans with good bladder sparing were often considered as very good (Grade 4) by the MPs, while the ROs considered most of these plans acceptable (Grade 3). The only plan considered as excellent (Grade 5) by a MP was a plan that respected all the constraints for the OAR and the PTV according to the EORTC (plan 9). On the contrary, for the ROs the seven plans considered as excellent were plans with good PTV coverage, regardless of some hotspots of above 105% of the prescription dose, and for which the rectum was dosimetrically well spared. In the majority of cases the same plans were accepted or rejected by the two groups, with ±1 plan difference among the two groups of DM (Table 1). Plans for which the bladder was not well spared (plans 6 and 8) were rejected by all MP. In plan 5, rejected by three ROs, the PTV volume was not fully covered by the 95% isodose (coldspot). RO2 systematically evaluated the plans higher than the other DMs. The kappa
index calculated for all the evaluations (accepted or rejected) and the two categories of DMs (ROs or MPs) is equal to 0.30, which is considered as being a “fair agreement” [34]. As expected, the structures mainly observed include the PTV, the bladder, and the rectum. Examination of the visual scanpaths and heatmaps demonstrated that large parts of the image were rarely or not at all fixated (Fig. 3). These parts included most of the CT image, apart from the delineated structures, and also the isodose lines. The number of fixations on the AOIs was significantly higher among the ROs compared to the MPs (p = 0.013) (Fig. 4a). The total number of fixations per structure shows that all structures did not receive equal number of fixations (Fig. 4b). The two groups of DMs made significantly different number of fixations on the CTV (p = 0.002), PTV (p = 0.005), rectum (p = 0.002) and femoral heads (p = 0.034). The number of fixations on the hotspots (p = 0.354) and bladder (p = 0.234) was not statistically significant between the two groups. There was no statistical difference on the total evaluation time before the decision was reached by the two groups of DMs (p = 0.116). However, differences were found on dwell times, i.e. sum of duration across all fixations in the same AOI. Among MPs similar dwell times were registered across the CTV, PTV, hot spots, bladder and rectum (p = 0.103), while the dwell time for these structures were dissimilar among ROs (p < 0.001) (Fig. 5). More precisely they spent statistically significant shorter dwell time on the femoral heads compared to the following structures: the PTV (p < 0.0001), the CTV (p = 0.005), and the bladder (p = 0.008). Additionally, the distributions of dwell times on the CTV, PTV and rectum were different for the two groups of DMs (p = 0.010, p = 0.026 and p = 0.001 respectively). In the majority of the cases (75 out of 80 plan evaluations, i.e. 10 plans × 8 DMs) the first structure that received a fixation was either the PTV (39 cases) or the bladder (46 cases). For the five remaining cases these structures were either the femoral heads or the rectum. There was no statistical difference between the two groups of DMs regarding the first
Table 1 Plan evaluations for MPs and ROs (with light gray are indicated the accepted plans and with dark gray the rejected plans). MP1
MP2
MP3
MP4
RO1
RO2
RO3
RO4
Plan 1
3
3
4
4
2
5
3
4
Plan 2
4
4
4
2
3
5
3
3
Plan 3
2
4
3
2
3
5
3
2
Plan 4
3
4
1
2
3
5
2
4
Plan 5
4
4
2
2
1
2
3
2
Plan 6
1
2
1
2
2
4
2
3
Plan 7
4
3
4
3
2
5
3
3
Plan 8
1
1
1
2
2
4
2
3
Plan 9
4
5
4
3
3
5
4
3
Plan 10
4
4
4
2
3
5
2
3
ARTICLE IN PRESS A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
5
Figure 3. Heatmaps of plan 6 evaluation from a RO (on the left) and a MP (on the right) superimposed on the 2D CT. The gradient scale on the right of each image represents the number of fixations (more fixations are represented with red and less fixations with green).
Figure 4. The number of fixations on the AOIs by the two groups of DMs (a: Global; b: Specific). Box plots show median values, 25th and 75th percentile, min and max values (whiskers). (* indicates a statistically significant difference between ROs and MPs.)
structure receiving a fixation. Among ROs the rectum was fixated longer when visited for the first time in 17 out of 40 cases, while in only 5 out of 40 cases among MPs. On the contrary, the bladder received the longest first fixation among ROs in 9 out of 40 cases while among MPs in 21 out of 40 cases.
Discussion The aim of this study was to use eye tracking to gain more knowledge about the treatment plan evaluation process in radiation therapy. We showed that eye movement information provides insights concerning the treatment plan evaluation and
ARTICLE IN PRESS 6
A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
Figure 5. Box plots of dwell time in the AOIs for the two groups of DMs (MPs and ROs). Box plots show median values, 25th and 75th percentile, min and max values (whiskers). (* indicates a statistically significant difference between ROs and MPs.)
decision making process and we showed differences between ROs and MPs in our department. Although there was no significant difference in the total number of plans accepted or rejected between the ROs and MPs, the evaluation process revealed slight differences in the perceived quality of the plans and therefore in differences in which plans were accepted or rejected. The ROs tended more easily to consider plans as excellent when the PTV was well covered and the rectum was well spared while MPs rarely considered a plan as excellent. The eye tracking data revealed that the ROs performed more fixations on the rectum and spent a longer time than MPs when fixating at the rectum. On the other hand, both ROs and MPs focused mainly on the bladder and the target. The plan evaluation results and the eye tracking data combined to reveal different evaluation strategies for the two groups of DMs, with the ROs evaluating specific and probably more limited number of parameters with clinical interest (e.g. tumor recurrence and rectum late complications), guided by top-down cues, and with the MPs checking fulfillment of dose constraints for all AOIs without prioritizing their clinical interest and being mainly guided by bottom-up visual cues. Structures receiving the first fixation and the duration of the first fixation in each structure were used as indicators of clinical judgments. In the majority of cases the first fixated structure was either the PTV or the bladder (including the PTV-bladder overlap region). This behavior indicates that
PTV coverage is the first criterion evaluated during treatment plan evaluation. Apart from being of clinical interest, these structures were relatively large, placed centrally in the image and were involved in one of the main trade-offs (PTV coverage vs. bladder sparing). Visually salient features, like these structures, are more likely to be visited first and fixated longer [35]. There is a difference between the two groups of DMs concerning the structure that is fixated longer compared to other structures when visited for the first time. For the ROs, the longest first fixation was more often at the rectum compared to the MPs, for whom it was more often on the bladder. Considering that fixation duration is correlated with the amount of cognitive activity, these results show that the two groups of DMs evaluate differently the doses to the rectum [36]. ROs collect more information in the observation phase of plan evaluation during their first visit to the rectum in order to make judgments regarding treatment complications, which affect the decision-making phase of plan evaluation. Although there was no significant difference in the total number of plans accepted or rejected by MPs and ROs, these were not always the same plans. In general, both groups made similar decisions regarding the acceptability of a plan for treatment, but our results suggest that they base these decisions on some different observational and evaluation criteria. These differences are probably due to the fact that ROs evaluate clinical implications of doses to the OAR such as the rectum, while
ARTICLE IN PRESS A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
MPs assure that the technical requirements of the plan and the recommended dose constraints to all OARs are satisfied. The main limitation of our study was that the DMs were asked to reach a decision based on a single transversal slice of the plan; in the clinical practice, a treatment plan consists of multiple CT slices covering the treated anatomy. However, even a single slice of the CT data provides useful information on what DMs observe on the image. Future work would include the whole series of CT slices involved in the treatment plan, in order to investigate the visual search in 3D data sets.
Conclusion In this work we applied eye tracking to investigate the plan evaluation process of prostate cancer treatment plans consisting of a single 2D CT image. With this method, we were able to quantify differences in evaluation behaviors between ROs and MPs in our department. This shows that eye tracking could be used to characterize the evaluation strategy of treatment plans for educational or for standardization purposes.
Acknowledgments The study was supported by a grant from the Swiss National Science Foundation (Project No. 320030 149489/1). The statistical analysis was performed by Dimakopoulos Georgios, Statistician MSc, Epirus Science and Technology Park, Campus of the University of Ioannina, Greece. The authors are grateful to Sam Hutton (SR Research Ltd., Mississauga, Canada) for his assistance with EyeLink1000. We would like to thank the radiation oncologists and medical physicists who participated in the study.
References [1] Krupinski EA. The role of perception in imaging: past and future. Semin Nucl Med 2011;41:392–400. [2] Kundel HL, Nodine CF, Krupinski EA. Computer-displayed eye position as a visual aid to pulmonary nodule interpretation. Investig Radiol 1990;25:890–6. [3] Bombari D, Mora B, Schaefer SC, Mast FW, Lehr HA. What was I thinking? Eye-tracking experiments underscore the bias that architecture exerts on nuclear grading in prostate cancer. PLoS ONE 2012;7:e38023. [4] Baddeley A. Working memory: looking back and looking forward. Nat Rev Neurosci 2003;4:829–39. [5] Connor CE, Egeth HE, Yantis S. Visual attention: bottom-up versus top-down. Curr Biol 2004;14:R850–2. [6] Chun MM. Contextual cueing of visual attention. Trends Cogn Sci 2000;4:170–8. [7] Hoffman J, Subramaniam B. The role of visual attention in saccadic eye movements. Percept Psychophys 1995;57:787–95. [8] Remington RW. Attention and saccadic eye movements. J Exp Psychol: Hum Percept Perform 1980;6:726–44. [9] Borji A, Itti L. Defending Yarbus: eye movements reveal observers’ task. J Vis 2014;14:29.
7
[10] Just MA, Carpenter PA. A theory of reading: from eye fixations to comprehension. Psychol Rev 1980;87:329–54. [11] Henderson JM. Human gaze control during real-world scene perception. Trends Cogn Sci 2003;7:498–504. [12] Viviani P. Eye movements in visual search: cognitive, perceptual and motor control aspects. Rev Oculomot Res 1990;4:353–93. [13] Buswell GT. How people look at pictures: a study of the psychology of perception in art. University of Chicago Press; 1935. [14] Latour PL. Visual threshold during eye movements. Vis Res 1962;2:261–2. [15] Hansen DW, Qiang J. In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans Pattern Anal Mach Intell 2010;32: 478–500. [16] Clark M. A two-dimensional Purkinje eye tracker. Behav Res Methods Instrum 1975;7:215–9. [17] Irwin DE. Fixation location and fixation duration as indices of cognitive processing. In: Ferreira JMHF, editor. The interface of language, vision, and action: eye movements and the visual world. New York, NY, US: Psychology Press; 2004. p. 105–33. [18] Krupinski EA, Tillack AA, Richter L, Henderson JT, Bhattacharyya AK, Scott KM, et al. Eye-movement study and human performance using telepathology virtual slides: implications for medical education and differences with experience. Hum Pathol 2006;37: 1543–56. [19] Bertram R, Helle L, Kaakinen JK, Svedström E. The effect of expertise on eye movement behaviour in medical image perception. PLOS ONE 2013;8:e66169. [20] Brunyé TT, Carney PA, Allison KH, Shapiro LG, Weaver DL, Elmore JG. Eye movements as an index of pathologist visual expertise: a pilot study. PLOS ONE 2014;9:e103447. [21] Matsumoto H, Terao Y, Yugeta A, Fukuda H, Emoto M, Furubayashi T, et al. Where do neurologists look when viewing brain CT images? An eye-tracking study involving stroke cases. PLoS ONE 2011;6: e28928. [22] Kundel HL, Nodine CF, Krupinski EA. Searching for lung nodules. Visual dwell indicates locations of false-positive and false-negative decisions. Investig Radiol 1989;24:472–8. [23] Krupinski EA. Visual scanning patterns of radiologists searching mammograms. Acad Radiol 1996;3:137–44. [24] Krupinski EA. Visual search of mammographic images: influence of lesion subtlety. Acad Radiol 2005;12:965–9. [25] Hu CH, Kundel HL, Nodine CF, Krupinski EA, Toto LC. Searching for bone fractures: a comparison with pulmonary nodule search. Acad Radiol 1994;1:25–32. [26] Litchfield D, Ball LJ, Donovan T, Manning DJ, Crawford T. Viewing another person’s eye movements improves identification of pulmonary nodules in chest X-ray inspection. J Exp Psychol: Appl 2010;16: 251–62. [27] Donovan T, Manning DJ, Crawford T. Performance changes in lung nodule detection following perceptual feedback of eye movements; 2008. p. 691703–9. [28] Kundel HL, Wright DJ. The influence of prior knowledge on visual search strategies during the viewing of chest radiographs. Radiology 1969;93:315–20. [29] Kundel HL, La Follette Jr PS. Visual search patterns and experience with radiological images. Radiology 1972;103:523–8. [30] Kundel HL, Nodine CF. Interpreting chest radiographs without visual search. Radiology 1975;116:527–32. [31] Kundel HL, Nodine CF, Carmody D. Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. Investig Radiol 1978;13:175–81. [32] Matzinger O, Duclos F, Bergh Avd, Carrie C, Villà S, Kitsios P, et al. Acute toxicity of curative radiotherapy for intermediate- and highrisk localised prostate cancer in the EORTC trial 22991. Eur J Cancer 2009;45:2825–34. [33] Matzinger O, Poortmans P, Giraud J-Y, Maingon P, Budiharto T, van den Bergh ACM, et al. Quality assurance in the 22991 EORTC ROG
ARTICLE IN PRESS 8
A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
trial in localized prostate cancer: dummy run and individual case review. Radiother Oncol 2009;90:285–90. [34] Kundel HL, Polansky M. Measurement of observer agreement. Radiology 2003;228:303–8.
[35] van der Laan LN, Hooge ITC, de Ridder DTD, Viergever MA, Smeets PAM. Do you like what you see? The role of first fixation and total fixation duration in consumer choice. Food Qual Prefer 2015;39:46–55. [36] Goldberg JH, Kotval XP. Computer interface evaluation using eye movements: methods and constructs. Int J Ind Ergon 1999;24:631–45.
Available online at www.sciencedirect.com
ScienceDirect
ARTICLE IN PRESS
ORIGINAL PAPER
Analysis of the treatment plan evaluation process in radiotherapy through eye tracking A. Kyroudi 1 , K. Petersson 1 , M. Ozsahin 2 , J. Bourhis 2 , F. Bochud 1 , R. Moeckli 1,∗ 1 2
Institute of Radiation Physics (IRA), Lausanne University Hospital, Lausanne, Switzerland Department of Radiation Oncology, Lausanne University Hospital, Lausanne, Switzerland
Received 19 September 2017; accepted 21 November 2017
Abstract Background and purpose: Treatment plan evaluation is a clinical decision-making problem that involves visual search and analysis in a contextually rich environment, including delineated structures and isodose lines superposed on CT data. It is a two-step process that includes visual analysis and clinical reasoning. In this work, we used eye tracking methods to gain more knowledge about the treatment plan evaluation process in radiation therapy. Materials and methods: Dose distributions on a single transverse slice of ten prostate cancer treatment plans were presented to eight decision makers. Their eye movements and fixations were recorded with an EyeLink1000 remote eye-tracker. Total evaluation time, dwell time, number and duration of fixations on pre-segmented areas of interest were measured. Results: The main structures receiving more and longer fixations (PTV, rectum, bladder) correspond to the main trade-offs evaluated in a typical prostate plan. Radiation oncologists made more fixations on the main structures compared to the medical physicists. Radiation oncologists fixated longer on the rectum when visited for the first time, while medical physicists fixated longer on the bladder. Conclusion: Our results quantify differences in the visual evaluation patterns between radiation oncologists and medical physicists, which indicate differences in their decision making strategies.
∗ Corresponding
Analyse des Evaluierungsprozesses eines Behandlungsplans der Strahlentherapie anhand von Eye Tracking Zusammenfassung Hintergrund und Zweck: Die Evaluierung eines Behandlungsplans ist ein klinischer Entscheidungsprozess, welcher eine visuellen Suche und Analyse in einer komplizierten Umgebung beinhaltet, mit abgegrenzten Strukturen und Isodosislinien, welche CT-Daten überlagert sind. Es ist ein zweistufiger Prozess, der visuelle Analyse und klinische Erfahrung kombiniert. Für dieses Projekt haben wir Eye-Tracking-Methoden verwendet, um mehr Wissen über die Behandlung und deren Beurteilungsprozess in der Strahlentherapie zu gewinnen. Material und Methoden: Dosis-Verteilungen auf einer einzelnen transversalen Ebene von zehn ProstatakrebsBehandlungsplänen wurden acht Personen zur Auswertung präsentiert. Ihre Augenbewegungen und Fixierungen wurden mit einem EyeLink1000 Remote-Eye-Tracker aufgezeichnet. Die gesamte Evaluierungszeit, Verweilzeit, Anzahl und Dauer der Fokussierung an vorsegmentierten Bereichen von Interesse wurden gemessen. Resultate: Die Hauptstrukturen, welche mehrere und längere visuelle Fokussierungen erhalten (PTV, Rektum, Blase), entsprechen den wichtigsten Vorkommnissen, die in einem typischen Prostataplan evaluiert wurden.
author: Raphaël Moeckli, Institute of Radiation Physics, Rue du Grand-Pré 1, CH-1007 Lausanne, Switzerland. E-mail: [email protected] (R. Moeckli).
Z. Med. Phys. xxx (2017) xxx–xxx https://doi.org/10.1016/j.zemedi.2017.11.002 www.elsevier.com/locate/zemedi
ARTICLE IN PRESS 2
A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
Im Vergleich zu den Medizinphysikern haben die Strahlenonkologen mehr visuelle Fokussierungen an den Hauptstrukturen vorgenommen. Strahlenonkologen hatten eine visuelle längere Fokussierung auf das Rektum, während Medizinphysiker sich länger visuell auf die Blase konzentrieren. Schlussfolgerung: Unsere Ergebnisse quantifizieren Unterschiede in den visuellen Bewertungsmustern zwischen Strahlentherapeuten und Medizinphysikern, die auf Unterschiede in ihren Entscheidungsstrategien hinweisen. Keywords: Prostate, Decision making, Plan quality, Eye tracking
Introduction Treatment plan evaluation in radiation therapy is mainly a clinical decision-making process that involves visual perception and clinical reasoning [1]. The task is performed by radiation oncologists and medical physicists, who are often not fully consciously aware of how they observe and interpret clinical data or how they perform complex clinical assessments [2,3]. A large part of the treatment plan evaluation process involves a visual search and analysis of CT images, overlapped with dose distributions and structure delineations. Due to our limited working memory capacity and in order to avoid information overload in such complex visual scenes, observers selectively focus on the most informative parts of the image while ignoring irrelevant details [4]. This selection is guided by bottom-up salient visual cues, such as color, orientation, and size of the object, but also by the context of the visual scene [5,6]. In parallel, top-down bias signals, which depend on the intentions, expectations, and goals of the decision maker, also influence visual search. Shifts of attention to different parts of the image are accompanied by eye movements [7,8], which indicate the type of information searched and processed, along with the cognitive processes that take place during a task [9,10]. Eye movement behavior during visual search tasks is of great interest in cognitive science [11,12]. During interaction with an image, the human eyes make a series of fixations (aggregated gazepoints in which the eyes remain relatively stable and focused on a particular area), and saccades (rapid eye movements that occur from one fixation to another) [13,14]. The role of saccades is to bring the fovea, the high resolution central part of the retina, to another portion of the visual scene where interesting information may occur [15]. The location of the fixation that follows a saccade is indicative of the type of information available and its duration is indicative of how much information can be acquired.
Schlüsselwörter: Prostata, Entscheidung, Behandlungsplan qualität, Eye tracking
A common technique of recording spatial information, representative of where the eyes were pointed at a certain moment, is eye tracking [15,16]. Information resulting from eye tracking data analysis can provide insights into the type of information sought by the participant during an evaluation task [17]. Eye tracking has already been widely used for research in the medical domain, in order to identify visual behaviors through eye movements [18–21]. Particularly, in radiology it has been used to explore visual search patterns when searching for lung nodules on radiographs, breast lesions on mammographies, or fractures on radiographs [22–25]. Different search patterns between experts and novices have been demonstrated and visually guided learning has been shown to improve novice radiographers’ detection of pulmonary nodules in Xrays [26,27]. In an approach similar to what has been done in radiology since the 1970s, we used eye tracking techniques to enhance the understanding of how decision makers interact with the given visual information during treatment plan evaluation [28–31]. Analyzing eye movements enabled us to recognize the areas of the treatment plan on which the decision makers focus and also allowed us to identify different visual behaviors between radiation oncologists and medical physicists during the treatment plan evaluation process.
Materials and methods Four radiation oncologists (ROs) and four medical physicists (MPs) from the same radiotherapy department participated as decision makers (DMs) in the study. Although all of them were familiar with prostate treatment plans, their area of expertise and experience level varied (from none up to several prostate plan evaluations per week). They were asked to evaluate ten prostate cancer cases as follows: “Please evaluate the quality of the plan, based on the single transversal slice of the treatment plan, on a scale from 1 to 5, with 1 representing a totally unacceptable plan, 3 an acceptable plan, and 5 an
ARTICLE IN PRESS A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
3
Figure 1. An example of a transverse image slice of a treatment plan shown to decision makers during the eye tracking experiment.
excellent plan. Grades 2 and 4 represent intermediate assessments”. The task was framed in such a way as to engage the DMs into a natural process of decision-making concerning the plan quality. Each DM received the explanation of the experimental process and performed a brief training session with six plan evaluations, which were mimicking but not included in the real experiment. We used the Velocity software (Varian Medical Systems Inc., Palo Alto, CA, USA) to present a single transverse image slice of the plan on a medical color LCD monitor (RadiForce MX210, EIZO) with capture rate of 60 Hz, without any possible patient or treatment machine identifiers (Fig. 1). Each image was a CT scan with delineated anatomical structures (Clinical Target Volume (CTV), Planning Target Volume (PTV), rectum, bladder, and femoral heads), isodose lines corresponding to various dose levels received around the PTV and legends with explanatory information. Prostate plans were selected due to their common occurrence in radiotherapy and their anatomical simplicity. The ten plans were created with TomoTherapy (Accuray Inc, Sunnyvale, CA, USA) or Monaco (Elekta AB, Stockholm, Sweden) TPS, using the CT data of patients previously treated in our clinic. The plans of the lowest quality (plans 5, 6 and 8) did not respect dose constraints and requirements for the organs at risk (OARs) and the PTV, as defined by the EORTC [32,33]. Areas of interest (AOIs) were defined for each of the ten plans and involved the main structures of the patient’s geometry as they were delineated on the treatment plan, i.e. PTV, CTV, rectum, bladder, right and left femoral head, but it also included hotspots. The DMs’ eye movements were collected using an EyeLink1000 (SR Research Ltd., Mississauga, Ontario, Canada),
with a 500 Hz sampling rate. The remote option of the eye tracking system (without a fixed chinrest) was preferred in order to simulate routine clinical practice conditions as much as possible. The gaze position was determined through pupil and corneal reflection created by infrared illumination, with an average accuracy of 0.5◦ , corresponding to 6 mm resolution on the image for our setup. Fixations were formed by grouping x and y coordinates of the raw data, which were selected with a velocity-based algorithm that discriminates fixations and saccades based on an angular velocity threshold (30◦ /s) (EyeLink1000, SR Research Ltd., Mississauga, Ontario, Canada). The chosen threshold minimized the detection of short fixations (less than 100ms). Fixations belonging to two overlapping AOIs (e.g. PTV overlapping with bladder or rectum) were assigned to both regions. The Kolmogorov–Smirnov test was used to examine if the eye tracking results were normally distributed. Comparisons between the two groups of DMs (ROs and MPs) were examined with the use of the Mann–Whitney U test. The Pearson Chi Square (normally distributed data), or the Fisher’s exact test (non-normally distributed data) was used to examine differences in the plan evaluation results between the two groups of DMs. The significance level (α) chosen was 5% and the data analysis was carried out with the use of SPSS v.21.0.
Results ROs and MPs show some different trends in their evaluations, with only one RO evaluating one plan as totally unacceptable and only one MP evaluating one plan as excellent (Fig. 2). However, there was no significant difference
ARTICLE IN PRESS 4
A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
Figure 2. Treatment plan assessments of the ten plans made by the four radiation oncologists (ROs) and the four medical physicists (MPs).
regarding the total number of accepted or rejected plans among the two groups of DMs (Pearson Chi Square = 0.879, p = 0.348). The plan evaluated by a RO as being totally unacceptable (Grade 1) was a plan where the PTV was not covered by the 95% isodose, while for the MPs the plans considered as totally unacceptable were mainly plans with poor bladder sparing. The plans with good bladder sparing were often considered as very good (Grade 4) by the MPs, while the ROs considered most of these plans acceptable (Grade 3). The only plan considered as excellent (Grade 5) by a MP was a plan that respected all the constraints for the OAR and the PTV according to the EORTC (plan 9). On the contrary, for the ROs the seven plans considered as excellent were plans with good PTV coverage, regardless of some hotspots of above 105% of the prescription dose, and for which the rectum was dosimetrically well spared. In the majority of cases the same plans were accepted or rejected by the two groups, with ±1 plan difference among the two groups of DM (Table 1). Plans for which the bladder was not well spared (plans 6 and 8) were rejected by all MP. In plan 5, rejected by three ROs, the PTV volume was not fully covered by the 95% isodose (coldspot). RO2 systematically evaluated the plans higher than the other DMs. The kappa
index calculated for all the evaluations (accepted or rejected) and the two categories of DMs (ROs or MPs) is equal to 0.30, which is considered as being a “fair agreement” [34]. As expected, the structures mainly observed include the PTV, the bladder, and the rectum. Examination of the visual scanpaths and heatmaps demonstrated that large parts of the image were rarely or not at all fixated (Fig. 3). These parts included most of the CT image, apart from the delineated structures, and also the isodose lines. The number of fixations on the AOIs was significantly higher among the ROs compared to the MPs (p = 0.013) (Fig. 4a). The total number of fixations per structure shows that all structures did not receive equal number of fixations (Fig. 4b). The two groups of DMs made significantly different number of fixations on the CTV (p = 0.002), PTV (p = 0.005), rectum (p = 0.002) and femoral heads (p = 0.034). The number of fixations on the hotspots (p = 0.354) and bladder (p = 0.234) was not statistically significant between the two groups. There was no statistical difference on the total evaluation time before the decision was reached by the two groups of DMs (p = 0.116). However, differences were found on dwell times, i.e. sum of duration across all fixations in the same AOI. Among MPs similar dwell times were registered across the CTV, PTV, hot spots, bladder and rectum (p = 0.103), while the dwell time for these structures were dissimilar among ROs (p < 0.001) (Fig. 5). More precisely they spent statistically significant shorter dwell time on the femoral heads compared to the following structures: the PTV (p < 0.0001), the CTV (p = 0.005), and the bladder (p = 0.008). Additionally, the distributions of dwell times on the CTV, PTV and rectum were different for the two groups of DMs (p = 0.010, p = 0.026 and p = 0.001 respectively). In the majority of the cases (75 out of 80 plan evaluations, i.e. 10 plans × 8 DMs) the first structure that received a fixation was either the PTV (39 cases) or the bladder (46 cases). For the five remaining cases these structures were either the femoral heads or the rectum. There was no statistical difference between the two groups of DMs regarding the first
Table 1 Plan evaluations for MPs and ROs (with light gray are indicated the accepted plans and with dark gray the rejected plans). MP1
MP2
MP3
MP4
RO1
RO2
RO3
RO4
Plan 1
3
3
4
4
2
5
3
4
Plan 2
4
4
4
2
3
5
3
3
Plan 3
2
4
3
2
3
5
3
2
Plan 4
3
4
1
2
3
5
2
4
Plan 5
4
4
2
2
1
2
3
2
Plan 6
1
2
1
2
2
4
2
3
Plan 7
4
3
4
3
2
5
3
3
Plan 8
1
1
1
2
2
4
2
3
Plan 9
4
5
4
3
3
5
4
3
Plan 10
4
4
4
2
3
5
2
3
ARTICLE IN PRESS A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
5
Figure 3. Heatmaps of plan 6 evaluation from a RO (on the left) and a MP (on the right) superimposed on the 2D CT. The gradient scale on the right of each image represents the number of fixations (more fixations are represented with red and less fixations with green).
Figure 4. The number of fixations on the AOIs by the two groups of DMs (a: Global; b: Specific). Box plots show median values, 25th and 75th percentile, min and max values (whiskers). (* indicates a statistically significant difference between ROs and MPs.)
structure receiving a fixation. Among ROs the rectum was fixated longer when visited for the first time in 17 out of 40 cases, while in only 5 out of 40 cases among MPs. On the contrary, the bladder received the longest first fixation among ROs in 9 out of 40 cases while among MPs in 21 out of 40 cases.
Discussion The aim of this study was to use eye tracking to gain more knowledge about the treatment plan evaluation process in radiation therapy. We showed that eye movement information provides insights concerning the treatment plan evaluation and
ARTICLE IN PRESS 6
A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
Figure 5. Box plots of dwell time in the AOIs for the two groups of DMs (MPs and ROs). Box plots show median values, 25th and 75th percentile, min and max values (whiskers). (* indicates a statistically significant difference between ROs and MPs.)
decision making process and we showed differences between ROs and MPs in our department. Although there was no significant difference in the total number of plans accepted or rejected between the ROs and MPs, the evaluation process revealed slight differences in the perceived quality of the plans and therefore in differences in which plans were accepted or rejected. The ROs tended more easily to consider plans as excellent when the PTV was well covered and the rectum was well spared while MPs rarely considered a plan as excellent. The eye tracking data revealed that the ROs performed more fixations on the rectum and spent a longer time than MPs when fixating at the rectum. On the other hand, both ROs and MPs focused mainly on the bladder and the target. The plan evaluation results and the eye tracking data combined to reveal different evaluation strategies for the two groups of DMs, with the ROs evaluating specific and probably more limited number of parameters with clinical interest (e.g. tumor recurrence and rectum late complications), guided by top-down cues, and with the MPs checking fulfillment of dose constraints for all AOIs without prioritizing their clinical interest and being mainly guided by bottom-up visual cues. Structures receiving the first fixation and the duration of the first fixation in each structure were used as indicators of clinical judgments. In the majority of cases the first fixated structure was either the PTV or the bladder (including the PTV-bladder overlap region). This behavior indicates that
PTV coverage is the first criterion evaluated during treatment plan evaluation. Apart from being of clinical interest, these structures were relatively large, placed centrally in the image and were involved in one of the main trade-offs (PTV coverage vs. bladder sparing). Visually salient features, like these structures, are more likely to be visited first and fixated longer [35]. There is a difference between the two groups of DMs concerning the structure that is fixated longer compared to other structures when visited for the first time. For the ROs, the longest first fixation was more often at the rectum compared to the MPs, for whom it was more often on the bladder. Considering that fixation duration is correlated with the amount of cognitive activity, these results show that the two groups of DMs evaluate differently the doses to the rectum [36]. ROs collect more information in the observation phase of plan evaluation during their first visit to the rectum in order to make judgments regarding treatment complications, which affect the decision-making phase of plan evaluation. Although there was no significant difference in the total number of plans accepted or rejected by MPs and ROs, these were not always the same plans. In general, both groups made similar decisions regarding the acceptability of a plan for treatment, but our results suggest that they base these decisions on some different observational and evaluation criteria. These differences are probably due to the fact that ROs evaluate clinical implications of doses to the OAR such as the rectum, while
ARTICLE IN PRESS A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
MPs assure that the technical requirements of the plan and the recommended dose constraints to all OARs are satisfied. The main limitation of our study was that the DMs were asked to reach a decision based on a single transversal slice of the plan; in the clinical practice, a treatment plan consists of multiple CT slices covering the treated anatomy. However, even a single slice of the CT data provides useful information on what DMs observe on the image. Future work would include the whole series of CT slices involved in the treatment plan, in order to investigate the visual search in 3D data sets.
Conclusion In this work we applied eye tracking to investigate the plan evaluation process of prostate cancer treatment plans consisting of a single 2D CT image. With this method, we were able to quantify differences in evaluation behaviors between ROs and MPs in our department. This shows that eye tracking could be used to characterize the evaluation strategy of treatment plans for educational or for standardization purposes.
Acknowledgments The study was supported by a grant from the Swiss National Science Foundation (Project No. 320030 149489/1). The statistical analysis was performed by Dimakopoulos Georgios, Statistician MSc, Epirus Science and Technology Park, Campus of the University of Ioannina, Greece. The authors are grateful to Sam Hutton (SR Research Ltd., Mississauga, Canada) for his assistance with EyeLink1000. We would like to thank the radiation oncologists and medical physicists who participated in the study.
References [1] Krupinski EA. The role of perception in imaging: past and future. Semin Nucl Med 2011;41:392–400. [2] Kundel HL, Nodine CF, Krupinski EA. Computer-displayed eye position as a visual aid to pulmonary nodule interpretation. Investig Radiol 1990;25:890–6. [3] Bombari D, Mora B, Schaefer SC, Mast FW, Lehr HA. What was I thinking? Eye-tracking experiments underscore the bias that architecture exerts on nuclear grading in prostate cancer. PLoS ONE 2012;7:e38023. [4] Baddeley A. Working memory: looking back and looking forward. Nat Rev Neurosci 2003;4:829–39. [5] Connor CE, Egeth HE, Yantis S. Visual attention: bottom-up versus top-down. Curr Biol 2004;14:R850–2. [6] Chun MM. Contextual cueing of visual attention. Trends Cogn Sci 2000;4:170–8. [7] Hoffman J, Subramaniam B. The role of visual attention in saccadic eye movements. Percept Psychophys 1995;57:787–95. [8] Remington RW. Attention and saccadic eye movements. J Exp Psychol: Hum Percept Perform 1980;6:726–44. [9] Borji A, Itti L. Defending Yarbus: eye movements reveal observers’ task. J Vis 2014;14:29.
7
[10] Just MA, Carpenter PA. A theory of reading: from eye fixations to comprehension. Psychol Rev 1980;87:329–54. [11] Henderson JM. Human gaze control during real-world scene perception. Trends Cogn Sci 2003;7:498–504. [12] Viviani P. Eye movements in visual search: cognitive, perceptual and motor control aspects. Rev Oculomot Res 1990;4:353–93. [13] Buswell GT. How people look at pictures: a study of the psychology of perception in art. University of Chicago Press; 1935. [14] Latour PL. Visual threshold during eye movements. Vis Res 1962;2:261–2. [15] Hansen DW, Qiang J. In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans Pattern Anal Mach Intell 2010;32: 478–500. [16] Clark M. A two-dimensional Purkinje eye tracker. Behav Res Methods Instrum 1975;7:215–9. [17] Irwin DE. Fixation location and fixation duration as indices of cognitive processing. In: Ferreira JMHF, editor. The interface of language, vision, and action: eye movements and the visual world. New York, NY, US: Psychology Press; 2004. p. 105–33. [18] Krupinski EA, Tillack AA, Richter L, Henderson JT, Bhattacharyya AK, Scott KM, et al. Eye-movement study and human performance using telepathology virtual slides: implications for medical education and differences with experience. Hum Pathol 2006;37: 1543–56. [19] Bertram R, Helle L, Kaakinen JK, Svedström E. The effect of expertise on eye movement behaviour in medical image perception. PLOS ONE 2013;8:e66169. [20] Brunyé TT, Carney PA, Allison KH, Shapiro LG, Weaver DL, Elmore JG. Eye movements as an index of pathologist visual expertise: a pilot study. PLOS ONE 2014;9:e103447. [21] Matsumoto H, Terao Y, Yugeta A, Fukuda H, Emoto M, Furubayashi T, et al. Where do neurologists look when viewing brain CT images? An eye-tracking study involving stroke cases. PLoS ONE 2011;6: e28928. [22] Kundel HL, Nodine CF, Krupinski EA. Searching for lung nodules. Visual dwell indicates locations of false-positive and false-negative decisions. Investig Radiol 1989;24:472–8. [23] Krupinski EA. Visual scanning patterns of radiologists searching mammograms. Acad Radiol 1996;3:137–44. [24] Krupinski EA. Visual search of mammographic images: influence of lesion subtlety. Acad Radiol 2005;12:965–9. [25] Hu CH, Kundel HL, Nodine CF, Krupinski EA, Toto LC. Searching for bone fractures: a comparison with pulmonary nodule search. Acad Radiol 1994;1:25–32. [26] Litchfield D, Ball LJ, Donovan T, Manning DJ, Crawford T. Viewing another person’s eye movements improves identification of pulmonary nodules in chest X-ray inspection. J Exp Psychol: Appl 2010;16: 251–62. [27] Donovan T, Manning DJ, Crawford T. Performance changes in lung nodule detection following perceptual feedback of eye movements; 2008. p. 691703–9. [28] Kundel HL, Wright DJ. The influence of prior knowledge on visual search strategies during the viewing of chest radiographs. Radiology 1969;93:315–20. [29] Kundel HL, La Follette Jr PS. Visual search patterns and experience with radiological images. Radiology 1972;103:523–8. [30] Kundel HL, Nodine CF. Interpreting chest radiographs without visual search. Radiology 1975;116:527–32. [31] Kundel HL, Nodine CF, Carmody D. Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. Investig Radiol 1978;13:175–81. [32] Matzinger O, Duclos F, Bergh Avd, Carrie C, Villà S, Kitsios P, et al. Acute toxicity of curative radiotherapy for intermediate- and highrisk localised prostate cancer in the EORTC trial 22991. Eur J Cancer 2009;45:2825–34. [33] Matzinger O, Poortmans P, Giraud J-Y, Maingon P, Budiharto T, van den Bergh ACM, et al. Quality assurance in the 22991 EORTC ROG
ARTICLE IN PRESS 8
A. Kyroudi et al. / Z. Med. Phys. xxx (2017) xxx–xxx
trial in localized prostate cancer: dummy run and individual case review. Radiother Oncol 2009;90:285–90. [34] Kundel HL, Polansky M. Measurement of observer agreement. Radiology 2003;228:303–8.
[35] van der Laan LN, Hooge ITC, de Ridder DTD, Viergever MA, Smeets PAM. Do you like what you see? The role of first fixation and total fixation duration in consumer choice. Food Qual Prefer 2015;39:46–55. [36] Goldberg JH, Kotval XP. Computer interface evaluation using eye movements: methods and constructs. Int J Ind Ergon 1999;24:631–45.
Available online at www.sciencedirect.com
ScienceDirect