- Email: [email protected]
Systematic review of educational interventions to improve contouring in radiotherapy
Radiotherapy and Oncology 144 (2020) 86–92
Contents lists available at ScienceDirect
Radiotherapy and Oncology journal homepage: www.thegreenjournal.com
Systematic Review
Systematic review of educational interventions to improve contouring in radiotherapy Jon Cacicedo a, Arturo Navarro-Martin b,⇑, Susana Gonzalez-Larragan c, Berardino De Bari f, Ahmed Salem d,e, Max Dahele g a Radiation Oncology Department, Cruces University Hospital, Osakidetza/Biocruces Health Research Institute/Department of Surgery, Radiology and Physical Medicine of the University of the Basque Country (UPV/EHU), Barakaldo; b Radiation Oncology Department, Hospital Duran i Reynals (ICO) Avda, Gran VIa de ´LHospitalet, Barcelona; c Department of Health Science Library, Cruces University Hospital, Osakidetza, Barakaldo, Spain; d Division of Cancer Sciences, University of Manchester; e Department of Clinical Oncology, The Christie Hospital NHS Trust, Manchester, United Kingdom; f Radiation Oncology Department, Centre Hospitalier Régional Universitaire Jean Minjoz, INSERM U1098 EFS/BFC, Besançon, France; g Department of Radiation Oncology, Cancer Center Amsterdam, Amsterdam UMC (VUmc location), the Netherlands
a r t i c l e
i n f o
Article history: Received 18 July 2019 Received in revised form 31 October 2019 Accepted 4 November 2019
Keywords: Radiotherapy contouring Educational Blended learning
a b s t r a c t Background and purpose: Contouring is a critical step in the radiotherapy process, but there is limited research on how to teach it and no consensus about the best method. We summarize the current evidence regarding improvement of contouring skills. Methods and materials: Comprehensive literature search of the Pubmed-MEDLINE database, EMBASE database and Cochrane Library to identify relevant studies (independently examined by two investigators) that included baseline contouring followed by a re-contouring assessment after an educational intervention. Results: 598 papers were identified. 16 studies met the inclusion criteria representing 370 participants (average number of participants per study of 23; range (4–141). Regarding the teaching methodology, 5/16 used onsite courses, 8/16 online courses, and 2/16 used blended learning. Study quality was heterogenous. There were only 3 randomized studies and only 3 analyzed the dosimetric impact of improving contouring homogeneity. Dice similarity coefficient was the most common evaluation metric (7/16), and in all these studies at least some contours improved significantly post-intervention. The time frame for evaluating the learning effect of the teaching intervention was almost exclusively short-time, with only one study evaluating the long-term utility of the educational program beyond 6 months. Conclusion: The literature on educational interventions designed to improve contouring performance is limited and heterogenous. Onsite, online and blended learning courses have all been shown to be helpful, however, sample sizes are small and impact assessment is almost exclusively short-term and typically does not take into account the effect on treatment planning. The most effective teaching methodology/format is unknown and impact on daily clinical practice is uncertain. Ó 2019 Elsevier B.V. All rights reserved. Radiotherapy and Oncology 144 (2019) 86–92
Radiotherapy (RT) is a vitally important treatment option for the management of cancer at many different anatomic locations. The delivery of safe and effective RT relies on accurate target and organ-at-risk (OAR) delineation. Currently, segmentation of these structures (gross target volume, GTV; clinical target volume, CTV; OARs) is a largely manual task, performed by radiation oncologists and radiation technologists. Contouring is time consuming [1] and
⇑ Corresponding author at: Radiation Oncology Department, Hospital Duran i Reynals (ICO) Avda, Gran VIa de L´Hospitalet, 199-203 08908 Hospitalet de Llobregat Barcelona, Spain. E-mail addresses: [email protected] (J. Cacicedo), [email protected] (A. Navarro-Martin), [email protected] (S. Gonzalez-Larragan), [email protected] (B. De Bari), ahmed. [email protected] (A. Salem), [email protected] (M. Dahele). https://doi.org/10.1016/j.radonc.2019.11.004 0167-8140/Ó 2019 Elsevier B.V. All rights reserved.
published data shows there is a high degree of inter-observer variability [2–9]. Because it is usually only performed once (unless the treatment is adapted), the quality of the initial contours can have significant effect on the therapeutic ratio, and several studies have linked contouring protocol deviations with decreased survival and increased toxicity [10–13]. Although it has been known about for a long time, sub-optimal contouring still remains a significant problem, even in the context of clinical trials [10,14,15]. In the phase 2 RTOG 0529 anal cancer study of dose-painted intensity-modulated RT for example, the GTV was inaccurately contoured in 21%, mesorectal lymph nodes in 55% and the small bowel in 60% [10]. While some variation in contouring is inevitable, there is a clear rationale for ensuring that this essential step in the radiotherapy process is performed well. This is true in routine clinical practice
J. Cacicedo et al. / Radiotherapy and Oncology 144 (2020) 86–92
and in clinical trials, where therapeutic ratios are being evaluated and contours may often be used to develop predictive models for tumor control and normal tissue complications. Good quality contouring is likely to become even more important as the uncertainty margins get smaller and on-line adaptive radiotherapy makes its way into daily practice (e.g. MR Linac). Surgery is another important oncological treatment, and training, a center’s experience and the quality of surgery have been shown to be prognostic factors [16]. In addition, a central review of pathology and feedback to surgeons can improve the quality of the surgical procedure [17]. Analogous with this, there are continuing efforts to improve the quality of the contouring process, one component of which is a reduction in variability. These include the integration of specific training programs [18–21], online and on-site contouring workshops based on development of software tools [22–26] or the use of shared delineation standardized guidelines [7,27,28] for different anatomic tumor locations. To date, however, there are scarce data on how to teach contouring and no consensus as to what is the best method and format. We undertook this review to summarize the current evidence on the different strategies used to teach and improve contouring skills. Material and methods We performed a comprehensive literature search of the Pubmed-MEDLINE, EMBASE database and Cochrane Library to identify relevant studies, published up to 30 September 2018. Search strategy A MEDLINE strategy was performed as follows using the following Medical Subject Heading (MeSH) terms: Problem Strategy: ((conformal radiotherapy[MeSH Terms]) AND ((contour*) OR volume*)) AND (((heterogeneity) OR interobserver variability[MeSH Terms]) OR intra observer variability [MeSH Terms]))). Measures Strategy: (((guideline[MeSH Terms]) OR workshop [MeSH Terms]) OR active learning[MeSH Terms]))) OR contouring atlas. After the MEDLINE strategy was finalized, it was adapted to the syntax and subject headings of the other databases. Additional relevant studies were identified through manual search/cross reference check. Study selection A professional librarian (SGL) helped to conduct the systematic search. All the titles and abstracts were independently examined by two investigators (AN and JC) to identify studies for full-text retrieval. Additional studies were included from references cited in the articles identified by the electronic search. Any discrepancy between author’s assessments was solved by mutual agreement. The inclusion criteria were: studies including baseline contouring followed by a re-contouring assessment after an educational intervention (e.g. a live or on-line course) and a clear methodology to evaluate the quality of the contouring (e.g. quantitative metrics and/or qualitative evaluation of the delineated volumes). Articles determining the inter-observer variability of expert/trainee delineation at any tumor location without a teaching assessment (i.e. no pre-/post-educational impact analysis) were excluded. Articles presenting a methodology without describing the specific contouring instructions and teaching intervention were also excluded. Only English language, published, full articles were included. This systematic review protocol was performed in accordance with Preferred Reporting Items for Systematic Reviews and Meta-
87
Analyses (PRISMA) guidelines (Fig. 1) [29]. In accordance with these guidelines, the present protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) [30] with the following REGISTRATION NUMBER: CRD42018105991. Results The literature search identified 598 papers, of which 16 [3,15,18–20,22,31–40] ultimately met the inclusion criteria (Fig. 1). The studies were focused on the following tumor sites/ indications: (1) head and neck (n = 6/16; Supplementary Table 1), (2) rectal (n = 4/16; Supplementary Table 2), (3) prostate (n = 2/16, Supplementary Table 3), and (4) gastric (n = 1/16), esophagus (n = 1/16), spinal stereotactic body radiation therapy (SBRT) (n = 1/16) and miscellaneous (n = 1/16) (Supplementary Table 4). Regarding the type of intervention used in the studies it can be summarized as follows: 5/16 were onsite courses, 8/16 were online courses, 2/16 used blended learning (a style of education in which students learn via electronic/online media, also including traditional face-to-face teaching) and 1/16 was a boot camp. The methodology of each study has been further described in Supplementary Tables 1–4. The studies were performed in the following countries: United States n = 4 [3,15,18,19], Canada n = 4 [20,31,38,39], Italy n = 2 [33,37], United Kingdom n = 2 [34,36] Turkey n = 1 [40], China n = 1 [32], Netherlands n = 1 [35], International n = 1 (onsite workshop with participants from 32 different countries) [22]. There was a total of 370 participants in all 16 articles with a range of 4 to 141/study. Only 3 out of 16 were randomized studies [15,18,31]. Three studies included dosimetric end-points [32–34], the results of which can be summarized as follows: (1) in Tao et al. [32], the teaching intervention significantly reduced variation in OAR dosimetry for some, but not all OARs; (2) in Lobefalo et al. [33], the mean V95% increased, in both conformal and intensitymodulated plans; (3) in Mitchel et al. [34], 2/18 plans were deemed unacceptable because of rectal dosimetry before the teaching intervention, increasing to 7/18 after teaching. This was due to a 33% increase in the CTV volume. The vast majority of studies looked at the short-term impact of the educational intervention - only one evaluated the impact beyond 6 months [35]. Considering the heterogeneity of the studies we assessed their quality using a validated tool (Tables 1 and 2). One of the most frequent deficits was the lack of sample size calculation to justify the number of participants needed to detect a significant improvement in contouring, and statistical methods were not consistently implemented to analyze if the pre-test and post-test improvements detected in contouring were statistically significant or not. In addition, quantitative analysis of interobserver variation in OAR contouring was evaluated using different indices (see Supplementary Tables 1–4) including Dice similarity coefficient (DSC), Jaccard index, volume coefficient of variation (CV), conformation number (CN), the Hausdorff distance (HD) and mean distance to agreement (MDA). One study performed a qualitative evaluation [19] defining different categories as adequate, adequate with minor variations, and inadequate with major variations. The most frequently used quantitative analysis was the DSC (7/16 studies), with results summarized in Supplementary Table 5. Discussion Accurate delineation of the target volume and OARs have long been recognized as critical steps in the radiotherapy process and
88
Contouring improvement
Fig. 1. PRISMA flow diagram.
numerous studies have demonstrated substantial interobserver variability in contouring among radiation providers [2,4,7,15,28]. We were able to identify 16 studies looking at the impact of various educational interventions on contouring performance. These studies showed that improvement in contouring quality could be achieved using several methods. Awan et al. [3] reported an improvement in head and neck lymph node and OAR contouring using an atlas combined with real-time visual feedback based on comparison of the participant’s contour with an ‘‘expert” contour derived from 6 individuals using Simultaneous Truth And Performance Level Estimation (STAPLE) methodology [41]. Gillespie et al/ [15] showed that an interactive web-based 3-dimensional contouring atlas (eContour) could improve target volume and parotid gland contouring. Mukesh et al. [36] showed that the COSTAR study contouring protocol atlas could improve head and neck target volume delineation. The study by Tao et al. [24] is slightly different in that the intervention was an atlas-based autosegmented contour that the subject then edits, leadin to reduced variation for multiple head and neck OARs. Fuller et al. [18] reported in a randomized trial that a visual atlas in addition to written instructions improved the quality of rectal cancer target volume contouring compared to reference expert contours [18]. The authors subsequently showed that access to the atlas in addition to written instructions resulted in increased tumor control probability (TCP) and decreased normal tissue complication probability (NTCP) for the small intestine [42]. Franco et al also
showed that the use of Anatom-eTM atlas system (http://anatome.com Anatom-e Information Systems Ltd., Houston, Texas) increased protocol adherence for rectal cancer CTV delineation [37]. Nijkamp et al. [35] reported that national consensus delineation guidelines, based on a digital atlas, reduced delineation variation in rectal cancer patients. It is noteworthy that in some of these studies substantial variation could still be seen after the intervention, despite statistically significant improvements in contouring. Online interactive resources such as eContour [15], Educase [22], Anatom-e [37], TaCTICS [3], ITK-SNAP [38] and ‘‘Big Brother” [18,35] computer software programs are now available and have been shown to improve contouring homogeneity and adherence ´ Souza et al. [39] implemented a to contouring-based guidelines. D seminar designed to offer a comprehensive overview of headand-neck anatomy and oncology, specifically aiming to develop competency in anatomic knowledge and ability to identify anatomic landmarks on cross sectional imaging relevant for target delineation. While theoretical knowledge increased, they observed suboptimal development of contouring competency (significant improvement in accuracy in only 3/20 structures) and suggested that the lack of hands-on contouring sessions was a problem [39]. Szumacher et al. [31] were unable to demonstrate a benefit from a combined interactive teaching and hands-on intervention, and speculated that differences in the baseline knowledge and/or academic ability in the study groups might have been a determin-
Table 1 Quality assessment tool for before-after (Pre-Post) studies with no control group. Author
Was the study question or objective clearly stated?
Were eligibility/ selection criteria for the study population prespecified and clearly described?
Were all eligible participants that met the prespecified entry criteria enrolled?
Was the sample size sufficiently large to provide confidence in the findings?
Was the test/ service/ intervention clearly described and delivered consistently across the study population?
Were the outcome measures prespecified, clearly defined, valid, reliable, and assessed consistently across all study participants?
Were the people assessing the outcomes blinded to the participants’ exposures/ interventions?
Was the loss to follow-up after baseline 20% or less? Were those lost to followup accounted for in the analysis?
Bekelman et al [19] Mukesh et al [36] Awan et al [3]
NA NA*
NA
D´Souza et al [39] Tao et al [32]
NA NA
Nijkamp et al [35] Lobefalo et al [33]
NA NA
NA
Franco et al [37] Mitchell et al [34]
NA
Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-topost changes?
Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (i.e., did they use an interrupted time-series design)?
If the intervention was conducted at a group level (e.g., a whole hospital, a community, etc.) did the statistical analysis take into account the use of individual-level data to determine effects at the group level?
J. Cacicedo et al. / Radiotherapy and Oncology 144 (2020) 86–92
Were the participants in the study representative of those who would be eligible for the test/service/ intervention in the general or clinical population of interest?
De Bari et al [22] Onal et al [40] Jaswal et al [38] Tai et al [20]
Reference: https://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort. Date accessed: 3/05/2019. * NA: not applicable
89
Contouring improvement
Szumacher et al [31] Fuller et al [18] Guillespie et al [15]
Reference: https://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort. Date accessed: 3/05/2019.
Was the treatment allocation concealed (so that assignments could not be predicted)? Was the method of randomization adequate (i.e., use of randomly generated assignment)? Was the study described as randomized, a randomized trial, a randomized clinical trial, or an RCT? Author
Table 2 Quality Assessment tool for controlled interventions.
Were study participants and providers blinded to treatment group assignment?
Were the people assessing the outcomes blinded to the participants’ group assignments?
Were the groups similar at baseline on important characteristics that could affect outcomes (e.g., demographics, risk factors, comorbid conditions)?
Was the overall drop-out rate from the study at endpoint 20% or lower of the number allocated to treatment?
Was the differential drop-out rate (between treatment groups) at endpoint 15 percentage points or lower?
Was there high adherence to the intervention protocols for each treatment group?
Were other interventions avoided or similar in the groups (e.g., similar background treatments)?
Were outcomes assessed using valid and reliable measures, implemented consistently across all study participants?
Did the authors report that the sample size was sufficiently large to be able to detect a difference in the main outcome between groups with at least 80% power?
Were outcomes reported or subgroups analyzed prespecified (i.e., identified before analyses were conducted)?
Were all randomized participants analyzed in the group to which they were originally assigned, i.e., did they use an intentionto-treat analysis?
90
ing factor; something to take into account if a comparative study is being designed. In general, there are several limitations to point out: the sample size is limited, and number of participants and quality are heterogeneous [16,23,31,13,14,30] (Tables 1 and 2). There is also a lack of studies in low- and middle-income countries. Typically only one or two clinical cases were contoured [3,18,19,22,31,37,39,40], and studies focused on one tumor site. All of these factors might affect the generalizability of the results. The definition of ‘‘gold standard” reference contours and the metrics for evaluating improvement in contouring varies [43,44] suggesting an opportunity for greater consensus. Although not specifically the focus of the educational interventions, there are few studies [32–34] demonstrating the dosimetric impact resulting from the improved contouring quality. Potential response bias also exists in some of the studies. For example, in Fuller et al. [18] only 8/26 (31%) institutes participated, representing a selected group. In Franco et al 10/14 centers participated [37], and in Nijkamp et al. [35] al, 17/24 invited participants contoured in the first round and 11/24 in the second round. Such bias might also affect other educational formats like onsite or online workshops, for example, it might be that registrants are inherently more motivated and more likely to demonstrate improvements to the intervention [18,22,37,38]. Such sources of bias need to be taken into consideration when generalizing results of studies. The vast majority of the studies have evaluated the short-term impact of the educational intervention, e.g. immediately [3,18,22,31,38,40] or within a few weeks [13,14,25,26,28,29,31]. The longer-term impact on contouring proficiency and performance is therefore uncertain. This has been built into a multicenter prospective study performed by the European Society for Radiotherapy and Oncology, with the International Atomic Energy Agency [45]. Finally, the controlled nature of the impact assessment means that a participant’s proficiency in daily clinical practice is uncertain. The heterogeneity and limitations in available studies mean that it is not possible to clearly define the optimum methodology or format [46] for educational interventions aimed at improving contouring. However, based on this systematic review and the authors’ experience, a program to improve contouring skills might consider the following items: Program duration: although short programs of, for example, 2 hours duration can enable participants to (at least transiently) improve specific contouring skills, the volume of information, number of anatomical sites, complexity and variety of cases that can be addressed will be limited. Sufficient time needs to be allocated for the intended program. Anatomical/theoretical knowledge: it seems not to be sufficient to provide only theoretical/didactic knowledge to participants and then to expect significant improvement in contouring. Actual contouring/re-contouring by participants, combined with discussion of the results, is suggested. General program design comments: discussion of baseline contouring prior to the program intervention(s); show and discuss ‘‘expert” contours (acknowledging that some may consider the use of the term ‘‘expert” subjective, as one look at the contouring variation between ‘‘experts” will often confirm); discussion of relevant anatomical and ‘‘best practice” considerations, including dosimetric importance of good/accurate contouring (which may vary depending on, for example, the clinical scenario and treatment planning/delivery techniques); recontouring after the program intervention; discussion of recontouring, improvements and residual errors; attention to course facilities, equipment, and staff/participant ratio (adequate acoustics, good visualization of materials, adequate workstations, good internet connections and network speed if
J. Cacicedo et al. / Radiotherapy and Oncology 144 (2020) 86–92
required, ±opportunity for participants to familiarize themselves with contouring program prior to the course, sufficient staff to circulate and address questions during practical exercises); consider whether participants will have long-term access to presentations and other course materials (including expert contours). Residual contouring errors: Although improvements in contouring can be expected from a well-designed and well-executed program, it is highly likely that, sometimes substantial, contouring errors will still be present in at least some of the participants. Depending on the purpose and goals of the program, more than one iteration, ±additional intervention targeted at those who are finding it hard to improve, may be considered. Post-course feedback: necessary to facilitate continuous improvement and avoid complacency. Post-course assessment of whether learning is embedded in clinical practice: certain programs may wish to attempt this, for example via a further contouring case, but in the authors’ experience to date, response rates can be (very) low. To conclude, the literature on educational interventions designed to improve contouring performance is limited and heterogenous. Onsite, online and blended learning courses have all been shown to be helpful, however, sample sizes are small and impact assessment is almost exclusively short-term and typically does not take into account the effect on treatment planning. The most effective teaching methodology/format is unknown and impact on daily clinical practice is uncertain. Conflict of interest statement MD declares research funding from Varian Medical Systems (Palo Alto, CA, USA). The rest of the author(s) indicated no potential conflicts of interest or funding for this research. Appendix A. Supplementary data Supplementary data to this article can be found online at https://doi.org/10.1016/j.radonc.2019.11.004. References [1] Andrianarison VA, Laouiti M, Fargier-Bochaton O, Dipasquale G, Wang X, Nguyen NP, et al. Contouring workload in adjuvant breast cancer radiotherapy. Cancer Radiother 2018;22:747–53. https://doi.org/10.1016/ j.canrad.2018.01.008. [2] Fiorino C, Reni M, Bolognesi A, Cattaneo GM, Calandrino R. Intra- and interobserver variability in contouring prostate and seminal vesicles: implications for conformal treatment planning. Radiother Oncol 1998;47:285–92. [3] Awan M, Kalpathy-Cramer J, Gunn GB, Beadle BM, Garden AS, Phan J, et al. Prospective assessment of an atlas-based intervention combined with realtime software feedback in contouring lymph node levels and organs-at-risk in the head and neck: Quantitative assessment of conformance to expert delineation. Pract Radiat Oncol 2013;3:186–93. https://doi.org/10.1016/j. prro.2012.11.002. [4] Hong TS, Tomé WA, Harari PM. Heterogeneity in head and neck IMRT target design and clinical practice. Radiother Oncol 2012;103:92–8. https://doi.org/ 10.1016/j.radonc.2012.02.010. [5] Loo SW, Martin WMC, Smith P, Cherian S, Roques TW. Interobserver variation in parotid gland delineation: a study of its impact on intensity-modulated radiotherapy solutions with a systematic review of the literature. Br J Radiol 2012;85:1070–7. https://doi.org/10.1259/bjr/32038456. [6] Holliday E, Fuller CD, Kalpathy-Cramer J, Gomez D, Rimner A, Li Y, et al. Quantitative assessment of target delineation variability for thymic cancers: Agreement evaluation of a prospective segmentation challenge. J Radiat Oncol 2016;5:55–61. https://doi.org/10.1007/s13566-015-0230-7. [7] Dimopoulos JCA, De Vos V, Berger D, Petric P, Dumas I, Kirisits C, et al. Interobserver comparison of target delineation for MRI-assisted cervical cancer brachytherapy: application of the GYN GEC-ESTRO recommendations. Radiother Oncol 2009;91:166–72. https://doi.org/10.1016/j. radonc.2008.10.023. [8] Yang J, Woodward WA, Reed VK, Strom EA, Perkins GH, Tereffe W, et al. Statistical modeling approach to quantitative analysis of interobserver
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
91
variability in breast contouring. Int J Radiat Oncol Biol Phys 2014;89:214–21. https://doi.org/10.1016/j.ijrobp.2014.01.010. Cloak K, Jameson MG, Paneghel A, Wiltshire K, Kneebone A, Pearse M, et al. Contour variation is a primary source of error when delivering post prostatectomy radiotherapy: Results of the Trans-Tasman Radiation Oncology Group 08.03 Radiotherapy Adjuvant Versus Early Salvage (RAVES) benchmarking exercise. J Med Imaging Radiat Oncol 2019;63:390–8. https:// doi.org/10.1111/1754-9485.12884. Kachnic LA, Winter K, Myerson RJ, Goodyear MD, Willins J, Esthappan J, et al. RTOG 0529: a phase 2 evaluation of dose-painted intensity modulated radiation therapy in combination with 5-fluorouracil and mitomycin-C for the reduction of acute morbidity in carcinoma of the anal canal. Int J Radiat Oncol Biol Phys 2013;86:27–33. https://doi.org/10.1016/j.ijrobp.2012.09.023. Peters LJ, O’Sullivan B, Giralt J, Fitzgerald TJ, Trotti A, Bernier J, et al. Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 02.02. J Clin Oncol 2010;28:2996–3001. https://doi.org/10.1200/JCO.2009.27.4498. Wuthrick EJ, Zhang Q, Machtay M, Rosenthal DI, Nguyen-Tan PF, Fortin A, et al. Institutional clinical trial accrual volume and survival of patients with head and neck cancer. J Clin Oncol 2015;33:156–64. https://doi.org/10.1200/ JCO.2014.56.5218. Belkacemi Y, Colson-Durand L, Fayolle-Campana M, Pons P, Rialland A, BastujiGarin S, et al. A Wake-Up Call for Routine Morbidity and Mortality Review Meeting Procedures as Part of a Quality Governance Programs in Radiation Therapy Departments: Results of the PROUST Survey. Pract Radiat Oncol 2019;9:108–14. https://doi.org/10.1016/j.prro.2018.09.004. Fairchild A, Weber DC, Bar-Deroma R, Gulyban A, Fenton PA, Stupp R, et al. Quality assurance in the EORTC 22033–26033/CE5 phase III randomized trial for low grade glioma: the digital individual case review. Radiother Oncol 2012;103:287–92. https://doi.org/10.1016/j.radonc.2012.04.002. Gillespie EF, Panjwani N, Golden DW, Gunther J, Chapman TR, Brower JV, et al. Multi-institutional Randomized Trial Testing the Utility of an Interactive Three-dimensional Contouring Atlas Among Radiation Oncology Residents. Int J Radiat Oncol Biol Phys 2017;98:547–54. https://doi.org/10.1016/j. ijrobp.2016.11.050. Bilimoria KY, Phillips JD, Rock CE, Hayman A, Prystowsky JB, Bentrem DJ. Effect of surgeon training, specialization, and experience on outcomes for cancer surgery: a systematic review of the literature. Ann Surg Oncol 2009;16:1799–808. https://doi.org/10.1245/s10434-009-0467-8. Bosch SL, Nagtegaal ID. The Importance of the Pathologist’s Role in Assessment of the Quality of the Mesorectum. Curr Colorectal Cancer Rep 2012;8:90–8. https://doi.org/10.1007/s11888-012-0124-7. Fuller CD, Nijkamp J, Duppen JC, Rasch CRN, Thomas CR, Wang SJ, et al. Prospective randomized double-blind pilot study of site-specific consensus atlas implementation for rectal cancer target volume delineation in the cooperative group setting. Int J Radiat Oncol Biol Phys 2011;79:481–9. https:// doi.org/10.1016/j.ijrobp.2009.11.012. Bekelman JE, Wolden S, Lee N. Head-and-neck target delineation among radiation oncology residents after a teaching intervention: a prospective, blinded pilot study. Int J Radiat Oncol Biol Phys 2009;73:416–23. https://doi. org/10.1016/j.ijrobp.2008.04.028. Tai P, Van Dyk J, Battista J, Yu E, Stitt L, Tonita J, et al. Improving the consistency in cervical esophageal target volume definition by special training. Int J Radiat Oncol Biol Phys 2002;53:766–74. Alfieri J, Portelance L, Souhami L, Steinert Y, McLeod P, Gallant F, et al. Development and impact evaluation of an e-learning radiation oncology module. Int J Radiat Oncol Biol Phys 2012;82:e573–80. https://doi.org/ 10.1016/j.ijrobp.2011.07.002. De Bari B, Dahele M, Palmu M, Kaylor S, Schiappacasse L, Guckenberger M, et al. Short interactive workshops reduce variability in contouring treatment volumes for spine stereotactic body radiation therapy: Experience with the ESTRO FALCON programme and EduCaseTM training tool. Radiother Oncol 2018;127:150–3. https://doi.org/10.1016/j.radonc.2017.10.038. Chao KSC, Bhide S, Chen H, Asper J, Bush S, Franklin G, et al. Reduce in variation and improve efficiency of target volume delineation by a computer-assisted system using a deformable image registration approach. Int J Radiat Oncol Biol Phys 2007;68:1512–21. https://doi.org/10.1016/j.ijrobp.2007.04.037. Ciernik IF, Huser M, Burger C, Davis JB, Szekely G. Automated functional image-guided radiation treatment planning for rectal cancer. Int J Radiat Oncol Biol Phys 2005;62:893–900. https://doi.org/10.1016/j.ijrobp.2004.12.089. Pasquier D, Lacornerie T, Vermandel M, Rousseau J, Lartigau E, Betrouni N. Automatic segmentation of pelvic structures from magnetic resonance images for prostate cancer radiotherapy. Int J Radiat Oncol Biol Phys 2007;68:592–600. https://doi.org/10.1016/j.ijrobp.2007.02.005. van Sörnsen de Koste JR, Senan S, Underberg RWM, Oei SS, Elshove D, Slotman BJ, et al. Use of CD-ROM-based tool for analyzing contouring variations in involved-field radiotherapy for Stage III NSCLC. Int J Radiat Oncol Biol Phys. 2005;63:334–9. https://doi.org/10.1016/j.ijrobp.2005.02.016. Grégoire V, Levendag P, Ang KK, Bernier J, Braaksma M, Budach V, et al. CTbased delineation of lymph node levels and related CTVs in the node-negative neck: DAHANCA, EORTC, GORTEC, NCIC, RTOG consensus guidelines. Radiother Oncol 2003;69:227–36. Poortmans P, Bossi A, Vandeputte K, Bosset M, Miralbell R, Maingon P, et al. Guidelines for target volume definition in post-operative radiotherapy for prostate cancer, on behalf of the EORTC Radiation Oncology Group. Radiother Oncol 2007;84:121–7. https://doi.org/10.1016/j.radonc.2007.07.017.
92
Contouring improvement
[29] Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:. https://doi.org/10.1136/bmj.b2535b2535. [30] Booth A, Clarke M, Dooley G, Ghersi D, Moher D, Petticrew M, et al. The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev 2012;1:2. https://doi.org/10.1186/2046-4053-1-2. [31] Szumacher E, Harnett N, Warner S, Kelly V, Danjoux C, Barker R, et al. Effectiveness of educational intervention on the congruence of prostate and rectal contouring as compared with a gold standard in three-dimensional radiotherapy for prostate. Int J Radiat Oncol Biol Phys 2010;76:379–85. https://doi.org/10.1016/j.ijrobp.2009.02.008. [32] Tao C-J, Yi J-L, Chen N-Y, Ren W, Cheng J, Tung S, et al. Multi-subject atlasbased auto-segmentation reduces interobserver variation and improves dosimetric parameter consistency for organs at risk in nasopharyngeal carcinoma: A multi-institution clinical study. Radiother Oncol 2015;115:407–11. https://doi.org/10.1016/j.radonc.2015.05.012. [33] Lobefalo F, Bignardi M, Reggiori G, Tozzi A, Tomatis S, Alongi F, et al. Dosimetric impact of inter-observer variability for 3D conformal radiotherapy and volumetric modulated arc therapy: the rectal tumor target definition case. Radiat Oncol 2013;8:176. https://doi.org/10.1186/1748-717X-8-176. [34] Mitchell DM, Perry L, Smith S, Elliott T, Wylie JP, Cowan RA, et al. Assessing the effect of a contouring protocol on postprostatectomy radiotherapy clinical target volumes and interphysician variation. Int J Radiat Oncol Biol Phys 2009;75:990–3. https://doi.org/10.1016/j.ijrobp.2008.12.042. [35] Nijkamp J, de Haas-Kock DFM, Beukema JC, Neelis KJ, Woutersen D, Ceha H, et al. Target volume delineation variation in radiotherapy for early stage rectal cancer in the Netherlands. Radiother Oncol 2012;102:14–21. https://doi.org/ 10.1016/j.radonc.2011.08.011. [36] Mukesh M, Benson R, Jena R, Hoole A, Roques T, Scrase C, et al. Interobserver variation in clinical target volume and organs at risk segmentation in postparotidectomy radiotherapy: can segmentation protocols help? Br J Radiol 2012;85:e530–6. https://doi.org/10.1259/bjr/66693547. [37] Franco P, Arcadipane F, Trino E, Gallio E, Martini S, Iorio GC, et al. Variability of clinical target volume delineation for rectal cancer patients planned for neoadjuvant radiotherapy with the aid of the platform Anatom-e. Clin Transl Radiat Oncol 2018;11:33–9. https://doi.org/10.1016/j.ctro.2018.06.002.
[38] Jaswal J, D’Souza L, Johnson M, Tay K, Fung K, Nichols A, et al. Evaluating the impact of a Canadian national anatomy and radiology contouring boot camp for radiation oncology residents. Int J Radiat Oncol Biol Phys 2015;91:701–7. https://doi.org/10.1016/j.ijrobp.2014.11.009. [39] D’Souza L, Jaswal J, Chan F, Johnson M, Tay KY, Fung Kevin, et al. Evaluating the impact of an integrated multidisciplinary head & neck competency-based anatomy & radiology teaching approach in radiation oncology: a prospective cohort study. BMC Med Educ 2014;14:124. https://doi.org/10.1186/14726920-14-124. [40] Onal C, Cengiz M, Guler OC, Dolek Y, Ozkok S. The role of delineation education programs for improving interobserver variability in target volume delineation in gastric cancer. Br J Radiol 2017;90:20160826. https://doi.org/10.1259/ bjr.20160826. [41] Warfield SK, Zou KH, Wells WM. Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging 2004;23:903–21. https://doi.org/10.1109/ TMI.2004.828354. [42] Mavroidis P, Giantsoudis D, Awan MJ, Nijkamp J, Rasch CRN, Duppen JC, et al. Consequences of anorectal cancer atlas implementation in the cooperative group setting: radiobiologic analysis of a prospective randomized in silico target delineation study. Radiother Oncol 2014;112:418–24. https://doi.org/ 10.1016/j.radonc.2014.05.011. [43] Rogelj P, Hudej R, Petric P. Distance deviation measure of contouring variability. Radiol Oncol 2013;47:86–96. https://doi.org/10.2478/raon-20130005. [44] Fotina I, Lütgendorf-Caucig C, Stock M, Pötter R, Georg D. Critical discussion of evaluation parameters for inter-observer variability in target definition for radiation therapy. Strahlenther Onkol 2012;188:160–7. https://doi.org/ 10.1007/s00066-011-0027-6. [45] Eriksen JG, Cacicedo J, Navarro-Martin A, Petit C, Atencio Rosselot MC, Lazaryan A, et al. PO-0864: Long-term impact on contouring skills can be achieved by online learning. An ESTRO-FALCON-IAEA study. Radiother. Oncol. 2018;127:S452–3. https://doi.org/10.1016/S0167-8140(18)31174-5. [46] Schwartzstein RM, Roberts DH. Saying Goodbye to Lectures in Medical School Paradigm Shift or Passing Fad? N Engl J Med 2017;377:605–7. https://doi.org/ 10.1056/NEJMp1706474.
Contents lists available at ScienceDirect
Radiotherapy and Oncology journal homepage: www.thegreenjournal.com
Systematic Review
Systematic review of educational interventions to improve contouring in radiotherapy Jon Cacicedo a, Arturo Navarro-Martin b,⇑, Susana Gonzalez-Larragan c, Berardino De Bari f, Ahmed Salem d,e, Max Dahele g a Radiation Oncology Department, Cruces University Hospital, Osakidetza/Biocruces Health Research Institute/Department of Surgery, Radiology and Physical Medicine of the University of the Basque Country (UPV/EHU), Barakaldo; b Radiation Oncology Department, Hospital Duran i Reynals (ICO) Avda, Gran VIa de ´LHospitalet, Barcelona; c Department of Health Science Library, Cruces University Hospital, Osakidetza, Barakaldo, Spain; d Division of Cancer Sciences, University of Manchester; e Department of Clinical Oncology, The Christie Hospital NHS Trust, Manchester, United Kingdom; f Radiation Oncology Department, Centre Hospitalier Régional Universitaire Jean Minjoz, INSERM U1098 EFS/BFC, Besançon, France; g Department of Radiation Oncology, Cancer Center Amsterdam, Amsterdam UMC (VUmc location), the Netherlands
a r t i c l e
i n f o
Article history: Received 18 July 2019 Received in revised form 31 October 2019 Accepted 4 November 2019
Keywords: Radiotherapy contouring Educational Blended learning
a b s t r a c t Background and purpose: Contouring is a critical step in the radiotherapy process, but there is limited research on how to teach it and no consensus about the best method. We summarize the current evidence regarding improvement of contouring skills. Methods and materials: Comprehensive literature search of the Pubmed-MEDLINE database, EMBASE database and Cochrane Library to identify relevant studies (independently examined by two investigators) that included baseline contouring followed by a re-contouring assessment after an educational intervention. Results: 598 papers were identified. 16 studies met the inclusion criteria representing 370 participants (average number of participants per study of 23; range (4–141). Regarding the teaching methodology, 5/16 used onsite courses, 8/16 online courses, and 2/16 used blended learning. Study quality was heterogenous. There were only 3 randomized studies and only 3 analyzed the dosimetric impact of improving contouring homogeneity. Dice similarity coefficient was the most common evaluation metric (7/16), and in all these studies at least some contours improved significantly post-intervention. The time frame for evaluating the learning effect of the teaching intervention was almost exclusively short-time, with only one study evaluating the long-term utility of the educational program beyond 6 months. Conclusion: The literature on educational interventions designed to improve contouring performance is limited and heterogenous. Onsite, online and blended learning courses have all been shown to be helpful, however, sample sizes are small and impact assessment is almost exclusively short-term and typically does not take into account the effect on treatment planning. The most effective teaching methodology/format is unknown and impact on daily clinical practice is uncertain. Ó 2019 Elsevier B.V. All rights reserved. Radiotherapy and Oncology 144 (2019) 86–92
Radiotherapy (RT) is a vitally important treatment option for the management of cancer at many different anatomic locations. The delivery of safe and effective RT relies on accurate target and organ-at-risk (OAR) delineation. Currently, segmentation of these structures (gross target volume, GTV; clinical target volume, CTV; OARs) is a largely manual task, performed by radiation oncologists and radiation technologists. Contouring is time consuming [1] and
⇑ Corresponding author at: Radiation Oncology Department, Hospital Duran i Reynals (ICO) Avda, Gran VIa de L´Hospitalet, 199-203 08908 Hospitalet de Llobregat Barcelona, Spain. E-mail addresses: [email protected] (J. Cacicedo), [email protected] (A. Navarro-Martin), [email protected] (S. Gonzalez-Larragan), [email protected] (B. De Bari), ahmed. [email protected] (A. Salem), [email protected] (M. Dahele). https://doi.org/10.1016/j.radonc.2019.11.004 0167-8140/Ó 2019 Elsevier B.V. All rights reserved.
published data shows there is a high degree of inter-observer variability [2–9]. Because it is usually only performed once (unless the treatment is adapted), the quality of the initial contours can have significant effect on the therapeutic ratio, and several studies have linked contouring protocol deviations with decreased survival and increased toxicity [10–13]. Although it has been known about for a long time, sub-optimal contouring still remains a significant problem, even in the context of clinical trials [10,14,15]. In the phase 2 RTOG 0529 anal cancer study of dose-painted intensity-modulated RT for example, the GTV was inaccurately contoured in 21%, mesorectal lymph nodes in 55% and the small bowel in 60% [10]. While some variation in contouring is inevitable, there is a clear rationale for ensuring that this essential step in the radiotherapy process is performed well. This is true in routine clinical practice
J. Cacicedo et al. / Radiotherapy and Oncology 144 (2020) 86–92
and in clinical trials, where therapeutic ratios are being evaluated and contours may often be used to develop predictive models for tumor control and normal tissue complications. Good quality contouring is likely to become even more important as the uncertainty margins get smaller and on-line adaptive radiotherapy makes its way into daily practice (e.g. MR Linac). Surgery is another important oncological treatment, and training, a center’s experience and the quality of surgery have been shown to be prognostic factors [16]. In addition, a central review of pathology and feedback to surgeons can improve the quality of the surgical procedure [17]. Analogous with this, there are continuing efforts to improve the quality of the contouring process, one component of which is a reduction in variability. These include the integration of specific training programs [18–21], online and on-site contouring workshops based on development of software tools [22–26] or the use of shared delineation standardized guidelines [7,27,28] for different anatomic tumor locations. To date, however, there are scarce data on how to teach contouring and no consensus as to what is the best method and format. We undertook this review to summarize the current evidence on the different strategies used to teach and improve contouring skills. Material and methods We performed a comprehensive literature search of the Pubmed-MEDLINE, EMBASE database and Cochrane Library to identify relevant studies, published up to 30 September 2018. Search strategy A MEDLINE strategy was performed as follows using the following Medical Subject Heading (MeSH) terms: Problem Strategy: ((conformal radiotherapy[MeSH Terms]) AND ((contour*) OR volume*)) AND (((heterogeneity) OR interobserver variability[MeSH Terms]) OR intra observer variability [MeSH Terms]))). Measures Strategy: (((guideline[MeSH Terms]) OR workshop [MeSH Terms]) OR active learning[MeSH Terms]))) OR contouring atlas. After the MEDLINE strategy was finalized, it was adapted to the syntax and subject headings of the other databases. Additional relevant studies were identified through manual search/cross reference check. Study selection A professional librarian (SGL) helped to conduct the systematic search. All the titles and abstracts were independently examined by two investigators (AN and JC) to identify studies for full-text retrieval. Additional studies were included from references cited in the articles identified by the electronic search. Any discrepancy between author’s assessments was solved by mutual agreement. The inclusion criteria were: studies including baseline contouring followed by a re-contouring assessment after an educational intervention (e.g. a live or on-line course) and a clear methodology to evaluate the quality of the contouring (e.g. quantitative metrics and/or qualitative evaluation of the delineated volumes). Articles determining the inter-observer variability of expert/trainee delineation at any tumor location without a teaching assessment (i.e. no pre-/post-educational impact analysis) were excluded. Articles presenting a methodology without describing the specific contouring instructions and teaching intervention were also excluded. Only English language, published, full articles were included. This systematic review protocol was performed in accordance with Preferred Reporting Items for Systematic Reviews and Meta-
87
Analyses (PRISMA) guidelines (Fig. 1) [29]. In accordance with these guidelines, the present protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) [30] with the following REGISTRATION NUMBER: CRD42018105991. Results The literature search identified 598 papers, of which 16 [3,15,18–20,22,31–40] ultimately met the inclusion criteria (Fig. 1). The studies were focused on the following tumor sites/ indications: (1) head and neck (n = 6/16; Supplementary Table 1), (2) rectal (n = 4/16; Supplementary Table 2), (3) prostate (n = 2/16, Supplementary Table 3), and (4) gastric (n = 1/16), esophagus (n = 1/16), spinal stereotactic body radiation therapy (SBRT) (n = 1/16) and miscellaneous (n = 1/16) (Supplementary Table 4). Regarding the type of intervention used in the studies it can be summarized as follows: 5/16 were onsite courses, 8/16 were online courses, 2/16 used blended learning (a style of education in which students learn via electronic/online media, also including traditional face-to-face teaching) and 1/16 was a boot camp. The methodology of each study has been further described in Supplementary Tables 1–4. The studies were performed in the following countries: United States n = 4 [3,15,18,19], Canada n = 4 [20,31,38,39], Italy n = 2 [33,37], United Kingdom n = 2 [34,36] Turkey n = 1 [40], China n = 1 [32], Netherlands n = 1 [35], International n = 1 (onsite workshop with participants from 32 different countries) [22]. There was a total of 370 participants in all 16 articles with a range of 4 to 141/study. Only 3 out of 16 were randomized studies [15,18,31]. Three studies included dosimetric end-points [32–34], the results of which can be summarized as follows: (1) in Tao et al. [32], the teaching intervention significantly reduced variation in OAR dosimetry for some, but not all OARs; (2) in Lobefalo et al. [33], the mean V95% increased, in both conformal and intensitymodulated plans; (3) in Mitchel et al. [34], 2/18 plans were deemed unacceptable because of rectal dosimetry before the teaching intervention, increasing to 7/18 after teaching. This was due to a 33% increase in the CTV volume. The vast majority of studies looked at the short-term impact of the educational intervention - only one evaluated the impact beyond 6 months [35]. Considering the heterogeneity of the studies we assessed their quality using a validated tool (Tables 1 and 2). One of the most frequent deficits was the lack of sample size calculation to justify the number of participants needed to detect a significant improvement in contouring, and statistical methods were not consistently implemented to analyze if the pre-test and post-test improvements detected in contouring were statistically significant or not. In addition, quantitative analysis of interobserver variation in OAR contouring was evaluated using different indices (see Supplementary Tables 1–4) including Dice similarity coefficient (DSC), Jaccard index, volume coefficient of variation (CV), conformation number (CN), the Hausdorff distance (HD) and mean distance to agreement (MDA). One study performed a qualitative evaluation [19] defining different categories as adequate, adequate with minor variations, and inadequate with major variations. The most frequently used quantitative analysis was the DSC (7/16 studies), with results summarized in Supplementary Table 5. Discussion Accurate delineation of the target volume and OARs have long been recognized as critical steps in the radiotherapy process and
88
Contouring improvement
Fig. 1. PRISMA flow diagram.
numerous studies have demonstrated substantial interobserver variability in contouring among radiation providers [2,4,7,15,28]. We were able to identify 16 studies looking at the impact of various educational interventions on contouring performance. These studies showed that improvement in contouring quality could be achieved using several methods. Awan et al. [3] reported an improvement in head and neck lymph node and OAR contouring using an atlas combined with real-time visual feedback based on comparison of the participant’s contour with an ‘‘expert” contour derived from 6 individuals using Simultaneous Truth And Performance Level Estimation (STAPLE) methodology [41]. Gillespie et al/ [15] showed that an interactive web-based 3-dimensional contouring atlas (eContour) could improve target volume and parotid gland contouring. Mukesh et al. [36] showed that the COSTAR study contouring protocol atlas could improve head and neck target volume delineation. The study by Tao et al. [24] is slightly different in that the intervention was an atlas-based autosegmented contour that the subject then edits, leadin to reduced variation for multiple head and neck OARs. Fuller et al. [18] reported in a randomized trial that a visual atlas in addition to written instructions improved the quality of rectal cancer target volume contouring compared to reference expert contours [18]. The authors subsequently showed that access to the atlas in addition to written instructions resulted in increased tumor control probability (TCP) and decreased normal tissue complication probability (NTCP) for the small intestine [42]. Franco et al also
showed that the use of Anatom-eTM atlas system (http://anatome.com Anatom-e Information Systems Ltd., Houston, Texas) increased protocol adherence for rectal cancer CTV delineation [37]. Nijkamp et al. [35] reported that national consensus delineation guidelines, based on a digital atlas, reduced delineation variation in rectal cancer patients. It is noteworthy that in some of these studies substantial variation could still be seen after the intervention, despite statistically significant improvements in contouring. Online interactive resources such as eContour [15], Educase [22], Anatom-e [37], TaCTICS [3], ITK-SNAP [38] and ‘‘Big Brother” [18,35] computer software programs are now available and have been shown to improve contouring homogeneity and adherence ´ Souza et al. [39] implemented a to contouring-based guidelines. D seminar designed to offer a comprehensive overview of headand-neck anatomy and oncology, specifically aiming to develop competency in anatomic knowledge and ability to identify anatomic landmarks on cross sectional imaging relevant for target delineation. While theoretical knowledge increased, they observed suboptimal development of contouring competency (significant improvement in accuracy in only 3/20 structures) and suggested that the lack of hands-on contouring sessions was a problem [39]. Szumacher et al. [31] were unable to demonstrate a benefit from a combined interactive teaching and hands-on intervention, and speculated that differences in the baseline knowledge and/or academic ability in the study groups might have been a determin-
Table 1 Quality assessment tool for before-after (Pre-Post) studies with no control group. Author
Was the study question or objective clearly stated?
Were eligibility/ selection criteria for the study population prespecified and clearly described?
Were all eligible participants that met the prespecified entry criteria enrolled?
Was the sample size sufficiently large to provide confidence in the findings?
Was the test/ service/ intervention clearly described and delivered consistently across the study population?
Were the outcome measures prespecified, clearly defined, valid, reliable, and assessed consistently across all study participants?
Were the people assessing the outcomes blinded to the participants’ exposures/ interventions?
Was the loss to follow-up after baseline 20% or less? Were those lost to followup accounted for in the analysis?
Bekelman et al [19] Mukesh et al [36] Awan et al [3]
NA NA*
NA
D´Souza et al [39] Tao et al [32]
NA NA
Nijkamp et al [35] Lobefalo et al [33]
NA NA
NA
Franco et al [37] Mitchell et al [34]
NA
Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-topost changes?
Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (i.e., did they use an interrupted time-series design)?
If the intervention was conducted at a group level (e.g., a whole hospital, a community, etc.) did the statistical analysis take into account the use of individual-level data to determine effects at the group level?
J. Cacicedo et al. / Radiotherapy and Oncology 144 (2020) 86–92
Were the participants in the study representative of those who would be eligible for the test/service/ intervention in the general or clinical population of interest?
De Bari et al [22] Onal et al [40] Jaswal et al [38] Tai et al [20]
Reference: https://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort. Date accessed: 3/05/2019. * NA: not applicable
89
Contouring improvement
Szumacher et al [31] Fuller et al [18] Guillespie et al [15]
Reference: https://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort. Date accessed: 3/05/2019.
Was the treatment allocation concealed (so that assignments could not be predicted)? Was the method of randomization adequate (i.e., use of randomly generated assignment)? Was the study described as randomized, a randomized trial, a randomized clinical trial, or an RCT? Author
Table 2 Quality Assessment tool for controlled interventions.
Were study participants and providers blinded to treatment group assignment?
Were the people assessing the outcomes blinded to the participants’ group assignments?
Were the groups similar at baseline on important characteristics that could affect outcomes (e.g., demographics, risk factors, comorbid conditions)?
Was the overall drop-out rate from the study at endpoint 20% or lower of the number allocated to treatment?
Was the differential drop-out rate (between treatment groups) at endpoint 15 percentage points or lower?
Was there high adherence to the intervention protocols for each treatment group?
Were other interventions avoided or similar in the groups (e.g., similar background treatments)?
Were outcomes assessed using valid and reliable measures, implemented consistently across all study participants?
Did the authors report that the sample size was sufficiently large to be able to detect a difference in the main outcome between groups with at least 80% power?
Were outcomes reported or subgroups analyzed prespecified (i.e., identified before analyses were conducted)?
Were all randomized participants analyzed in the group to which they were originally assigned, i.e., did they use an intentionto-treat analysis?
90
ing factor; something to take into account if a comparative study is being designed. In general, there are several limitations to point out: the sample size is limited, and number of participants and quality are heterogeneous [16,23,31,13,14,30] (Tables 1 and 2). There is also a lack of studies in low- and middle-income countries. Typically only one or two clinical cases were contoured [3,18,19,22,31,37,39,40], and studies focused on one tumor site. All of these factors might affect the generalizability of the results. The definition of ‘‘gold standard” reference contours and the metrics for evaluating improvement in contouring varies [43,44] suggesting an opportunity for greater consensus. Although not specifically the focus of the educational interventions, there are few studies [32–34] demonstrating the dosimetric impact resulting from the improved contouring quality. Potential response bias also exists in some of the studies. For example, in Fuller et al. [18] only 8/26 (31%) institutes participated, representing a selected group. In Franco et al 10/14 centers participated [37], and in Nijkamp et al. [35] al, 17/24 invited participants contoured in the first round and 11/24 in the second round. Such bias might also affect other educational formats like onsite or online workshops, for example, it might be that registrants are inherently more motivated and more likely to demonstrate improvements to the intervention [18,22,37,38]. Such sources of bias need to be taken into consideration when generalizing results of studies. The vast majority of the studies have evaluated the short-term impact of the educational intervention, e.g. immediately [3,18,22,31,38,40] or within a few weeks [13,14,25,26,28,29,31]. The longer-term impact on contouring proficiency and performance is therefore uncertain. This has been built into a multicenter prospective study performed by the European Society for Radiotherapy and Oncology, with the International Atomic Energy Agency [45]. Finally, the controlled nature of the impact assessment means that a participant’s proficiency in daily clinical practice is uncertain. The heterogeneity and limitations in available studies mean that it is not possible to clearly define the optimum methodology or format [46] for educational interventions aimed at improving contouring. However, based on this systematic review and the authors’ experience, a program to improve contouring skills might consider the following items: Program duration: although short programs of, for example, 2 hours duration can enable participants to (at least transiently) improve specific contouring skills, the volume of information, number of anatomical sites, complexity and variety of cases that can be addressed will be limited. Sufficient time needs to be allocated for the intended program. Anatomical/theoretical knowledge: it seems not to be sufficient to provide only theoretical/didactic knowledge to participants and then to expect significant improvement in contouring. Actual contouring/re-contouring by participants, combined with discussion of the results, is suggested. General program design comments: discussion of baseline contouring prior to the program intervention(s); show and discuss ‘‘expert” contours (acknowledging that some may consider the use of the term ‘‘expert” subjective, as one look at the contouring variation between ‘‘experts” will often confirm); discussion of relevant anatomical and ‘‘best practice” considerations, including dosimetric importance of good/accurate contouring (which may vary depending on, for example, the clinical scenario and treatment planning/delivery techniques); recontouring after the program intervention; discussion of recontouring, improvements and residual errors; attention to course facilities, equipment, and staff/participant ratio (adequate acoustics, good visualization of materials, adequate workstations, good internet connections and network speed if
J. Cacicedo et al. / Radiotherapy and Oncology 144 (2020) 86–92
required, ±opportunity for participants to familiarize themselves with contouring program prior to the course, sufficient staff to circulate and address questions during practical exercises); consider whether participants will have long-term access to presentations and other course materials (including expert contours). Residual contouring errors: Although improvements in contouring can be expected from a well-designed and well-executed program, it is highly likely that, sometimes substantial, contouring errors will still be present in at least some of the participants. Depending on the purpose and goals of the program, more than one iteration, ±additional intervention targeted at those who are finding it hard to improve, may be considered. Post-course feedback: necessary to facilitate continuous improvement and avoid complacency. Post-course assessment of whether learning is embedded in clinical practice: certain programs may wish to attempt this, for example via a further contouring case, but in the authors’ experience to date, response rates can be (very) low. To conclude, the literature on educational interventions designed to improve contouring performance is limited and heterogenous. Onsite, online and blended learning courses have all been shown to be helpful, however, sample sizes are small and impact assessment is almost exclusively short-term and typically does not take into account the effect on treatment planning. The most effective teaching methodology/format is unknown and impact on daily clinical practice is uncertain. Conflict of interest statement MD declares research funding from Varian Medical Systems (Palo Alto, CA, USA). The rest of the author(s) indicated no potential conflicts of interest or funding for this research. Appendix A. Supplementary data Supplementary data to this article can be found online at https://doi.org/10.1016/j.radonc.2019.11.004. References [1] Andrianarison VA, Laouiti M, Fargier-Bochaton O, Dipasquale G, Wang X, Nguyen NP, et al. Contouring workload in adjuvant breast cancer radiotherapy. Cancer Radiother 2018;22:747–53. https://doi.org/10.1016/ j.canrad.2018.01.008. [2] Fiorino C, Reni M, Bolognesi A, Cattaneo GM, Calandrino R. Intra- and interobserver variability in contouring prostate and seminal vesicles: implications for conformal treatment planning. Radiother Oncol 1998;47:285–92. [3] Awan M, Kalpathy-Cramer J, Gunn GB, Beadle BM, Garden AS, Phan J, et al. Prospective assessment of an atlas-based intervention combined with realtime software feedback in contouring lymph node levels and organs-at-risk in the head and neck: Quantitative assessment of conformance to expert delineation. Pract Radiat Oncol 2013;3:186–93. https://doi.org/10.1016/j. prro.2012.11.002. [4] Hong TS, Tomé WA, Harari PM. Heterogeneity in head and neck IMRT target design and clinical practice. Radiother Oncol 2012;103:92–8. https://doi.org/ 10.1016/j.radonc.2012.02.010. [5] Loo SW, Martin WMC, Smith P, Cherian S, Roques TW. Interobserver variation in parotid gland delineation: a study of its impact on intensity-modulated radiotherapy solutions with a systematic review of the literature. Br J Radiol 2012;85:1070–7. https://doi.org/10.1259/bjr/32038456. [6] Holliday E, Fuller CD, Kalpathy-Cramer J, Gomez D, Rimner A, Li Y, et al. Quantitative assessment of target delineation variability for thymic cancers: Agreement evaluation of a prospective segmentation challenge. J Radiat Oncol 2016;5:55–61. https://doi.org/10.1007/s13566-015-0230-7. [7] Dimopoulos JCA, De Vos V, Berger D, Petric P, Dumas I, Kirisits C, et al. Interobserver comparison of target delineation for MRI-assisted cervical cancer brachytherapy: application of the GYN GEC-ESTRO recommendations. Radiother Oncol 2009;91:166–72. https://doi.org/10.1016/j. radonc.2008.10.023. [8] Yang J, Woodward WA, Reed VK, Strom EA, Perkins GH, Tereffe W, et al. Statistical modeling approach to quantitative analysis of interobserver
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
91
variability in breast contouring. Int J Radiat Oncol Biol Phys 2014;89:214–21. https://doi.org/10.1016/j.ijrobp.2014.01.010. Cloak K, Jameson MG, Paneghel A, Wiltshire K, Kneebone A, Pearse M, et al. Contour variation is a primary source of error when delivering post prostatectomy radiotherapy: Results of the Trans-Tasman Radiation Oncology Group 08.03 Radiotherapy Adjuvant Versus Early Salvage (RAVES) benchmarking exercise. J Med Imaging Radiat Oncol 2019;63:390–8. https:// doi.org/10.1111/1754-9485.12884. Kachnic LA, Winter K, Myerson RJ, Goodyear MD, Willins J, Esthappan J, et al. RTOG 0529: a phase 2 evaluation of dose-painted intensity modulated radiation therapy in combination with 5-fluorouracil and mitomycin-C for the reduction of acute morbidity in carcinoma of the anal canal. Int J Radiat Oncol Biol Phys 2013;86:27–33. https://doi.org/10.1016/j.ijrobp.2012.09.023. Peters LJ, O’Sullivan B, Giralt J, Fitzgerald TJ, Trotti A, Bernier J, et al. Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 02.02. J Clin Oncol 2010;28:2996–3001. https://doi.org/10.1200/JCO.2009.27.4498. Wuthrick EJ, Zhang Q, Machtay M, Rosenthal DI, Nguyen-Tan PF, Fortin A, et al. Institutional clinical trial accrual volume and survival of patients with head and neck cancer. J Clin Oncol 2015;33:156–64. https://doi.org/10.1200/ JCO.2014.56.5218. Belkacemi Y, Colson-Durand L, Fayolle-Campana M, Pons P, Rialland A, BastujiGarin S, et al. A Wake-Up Call for Routine Morbidity and Mortality Review Meeting Procedures as Part of a Quality Governance Programs in Radiation Therapy Departments: Results of the PROUST Survey. Pract Radiat Oncol 2019;9:108–14. https://doi.org/10.1016/j.prro.2018.09.004. Fairchild A, Weber DC, Bar-Deroma R, Gulyban A, Fenton PA, Stupp R, et al. Quality assurance in the EORTC 22033–26033/CE5 phase III randomized trial for low grade glioma: the digital individual case review. Radiother Oncol 2012;103:287–92. https://doi.org/10.1016/j.radonc.2012.04.002. Gillespie EF, Panjwani N, Golden DW, Gunther J, Chapman TR, Brower JV, et al. Multi-institutional Randomized Trial Testing the Utility of an Interactive Three-dimensional Contouring Atlas Among Radiation Oncology Residents. Int J Radiat Oncol Biol Phys 2017;98:547–54. https://doi.org/10.1016/j. ijrobp.2016.11.050. Bilimoria KY, Phillips JD, Rock CE, Hayman A, Prystowsky JB, Bentrem DJ. Effect of surgeon training, specialization, and experience on outcomes for cancer surgery: a systematic review of the literature. Ann Surg Oncol 2009;16:1799–808. https://doi.org/10.1245/s10434-009-0467-8. Bosch SL, Nagtegaal ID. The Importance of the Pathologist’s Role in Assessment of the Quality of the Mesorectum. Curr Colorectal Cancer Rep 2012;8:90–8. https://doi.org/10.1007/s11888-012-0124-7. Fuller CD, Nijkamp J, Duppen JC, Rasch CRN, Thomas CR, Wang SJ, et al. Prospective randomized double-blind pilot study of site-specific consensus atlas implementation for rectal cancer target volume delineation in the cooperative group setting. Int J Radiat Oncol Biol Phys 2011;79:481–9. https:// doi.org/10.1016/j.ijrobp.2009.11.012. Bekelman JE, Wolden S, Lee N. Head-and-neck target delineation among radiation oncology residents after a teaching intervention: a prospective, blinded pilot study. Int J Radiat Oncol Biol Phys 2009;73:416–23. https://doi. org/10.1016/j.ijrobp.2008.04.028. Tai P, Van Dyk J, Battista J, Yu E, Stitt L, Tonita J, et al. Improving the consistency in cervical esophageal target volume definition by special training. Int J Radiat Oncol Biol Phys 2002;53:766–74. Alfieri J, Portelance L, Souhami L, Steinert Y, McLeod P, Gallant F, et al. Development and impact evaluation of an e-learning radiation oncology module. Int J Radiat Oncol Biol Phys 2012;82:e573–80. https://doi.org/ 10.1016/j.ijrobp.2011.07.002. De Bari B, Dahele M, Palmu M, Kaylor S, Schiappacasse L, Guckenberger M, et al. Short interactive workshops reduce variability in contouring treatment volumes for spine stereotactic body radiation therapy: Experience with the ESTRO FALCON programme and EduCaseTM training tool. Radiother Oncol 2018;127:150–3. https://doi.org/10.1016/j.radonc.2017.10.038. Chao KSC, Bhide S, Chen H, Asper J, Bush S, Franklin G, et al. Reduce in variation and improve efficiency of target volume delineation by a computer-assisted system using a deformable image registration approach. Int J Radiat Oncol Biol Phys 2007;68:1512–21. https://doi.org/10.1016/j.ijrobp.2007.04.037. Ciernik IF, Huser M, Burger C, Davis JB, Szekely G. Automated functional image-guided radiation treatment planning for rectal cancer. Int J Radiat Oncol Biol Phys 2005;62:893–900. https://doi.org/10.1016/j.ijrobp.2004.12.089. Pasquier D, Lacornerie T, Vermandel M, Rousseau J, Lartigau E, Betrouni N. Automatic segmentation of pelvic structures from magnetic resonance images for prostate cancer radiotherapy. Int J Radiat Oncol Biol Phys 2007;68:592–600. https://doi.org/10.1016/j.ijrobp.2007.02.005. van Sörnsen de Koste JR, Senan S, Underberg RWM, Oei SS, Elshove D, Slotman BJ, et al. Use of CD-ROM-based tool for analyzing contouring variations in involved-field radiotherapy for Stage III NSCLC. Int J Radiat Oncol Biol Phys. 2005;63:334–9. https://doi.org/10.1016/j.ijrobp.2005.02.016. Grégoire V, Levendag P, Ang KK, Bernier J, Braaksma M, Budach V, et al. CTbased delineation of lymph node levels and related CTVs in the node-negative neck: DAHANCA, EORTC, GORTEC, NCIC, RTOG consensus guidelines. Radiother Oncol 2003;69:227–36. Poortmans P, Bossi A, Vandeputte K, Bosset M, Miralbell R, Maingon P, et al. Guidelines for target volume definition in post-operative radiotherapy for prostate cancer, on behalf of the EORTC Radiation Oncology Group. Radiother Oncol 2007;84:121–7. https://doi.org/10.1016/j.radonc.2007.07.017.
92
Contouring improvement
[29] Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:. https://doi.org/10.1136/bmj.b2535b2535. [30] Booth A, Clarke M, Dooley G, Ghersi D, Moher D, Petticrew M, et al. The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev 2012;1:2. https://doi.org/10.1186/2046-4053-1-2. [31] Szumacher E, Harnett N, Warner S, Kelly V, Danjoux C, Barker R, et al. Effectiveness of educational intervention on the congruence of prostate and rectal contouring as compared with a gold standard in three-dimensional radiotherapy for prostate. Int J Radiat Oncol Biol Phys 2010;76:379–85. https://doi.org/10.1016/j.ijrobp.2009.02.008. [32] Tao C-J, Yi J-L, Chen N-Y, Ren W, Cheng J, Tung S, et al. Multi-subject atlasbased auto-segmentation reduces interobserver variation and improves dosimetric parameter consistency for organs at risk in nasopharyngeal carcinoma: A multi-institution clinical study. Radiother Oncol 2015;115:407–11. https://doi.org/10.1016/j.radonc.2015.05.012. [33] Lobefalo F, Bignardi M, Reggiori G, Tozzi A, Tomatis S, Alongi F, et al. Dosimetric impact of inter-observer variability for 3D conformal radiotherapy and volumetric modulated arc therapy: the rectal tumor target definition case. Radiat Oncol 2013;8:176. https://doi.org/10.1186/1748-717X-8-176. [34] Mitchell DM, Perry L, Smith S, Elliott T, Wylie JP, Cowan RA, et al. Assessing the effect of a contouring protocol on postprostatectomy radiotherapy clinical target volumes and interphysician variation. Int J Radiat Oncol Biol Phys 2009;75:990–3. https://doi.org/10.1016/j.ijrobp.2008.12.042. [35] Nijkamp J, de Haas-Kock DFM, Beukema JC, Neelis KJ, Woutersen D, Ceha H, et al. Target volume delineation variation in radiotherapy for early stage rectal cancer in the Netherlands. Radiother Oncol 2012;102:14–21. https://doi.org/ 10.1016/j.radonc.2011.08.011. [36] Mukesh M, Benson R, Jena R, Hoole A, Roques T, Scrase C, et al. Interobserver variation in clinical target volume and organs at risk segmentation in postparotidectomy radiotherapy: can segmentation protocols help? Br J Radiol 2012;85:e530–6. https://doi.org/10.1259/bjr/66693547. [37] Franco P, Arcadipane F, Trino E, Gallio E, Martini S, Iorio GC, et al. Variability of clinical target volume delineation for rectal cancer patients planned for neoadjuvant radiotherapy with the aid of the platform Anatom-e. Clin Transl Radiat Oncol 2018;11:33–9. https://doi.org/10.1016/j.ctro.2018.06.002.
[38] Jaswal J, D’Souza L, Johnson M, Tay K, Fung K, Nichols A, et al. Evaluating the impact of a Canadian national anatomy and radiology contouring boot camp for radiation oncology residents. Int J Radiat Oncol Biol Phys 2015;91:701–7. https://doi.org/10.1016/j.ijrobp.2014.11.009. [39] D’Souza L, Jaswal J, Chan F, Johnson M, Tay KY, Fung Kevin, et al. Evaluating the impact of an integrated multidisciplinary head & neck competency-based anatomy & radiology teaching approach in radiation oncology: a prospective cohort study. BMC Med Educ 2014;14:124. https://doi.org/10.1186/14726920-14-124. [40] Onal C, Cengiz M, Guler OC, Dolek Y, Ozkok S. The role of delineation education programs for improving interobserver variability in target volume delineation in gastric cancer. Br J Radiol 2017;90:20160826. https://doi.org/10.1259/ bjr.20160826. [41] Warfield SK, Zou KH, Wells WM. Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging 2004;23:903–21. https://doi.org/10.1109/ TMI.2004.828354. [42] Mavroidis P, Giantsoudis D, Awan MJ, Nijkamp J, Rasch CRN, Duppen JC, et al. Consequences of anorectal cancer atlas implementation in the cooperative group setting: radiobiologic analysis of a prospective randomized in silico target delineation study. Radiother Oncol 2014;112:418–24. https://doi.org/ 10.1016/j.radonc.2014.05.011. [43] Rogelj P, Hudej R, Petric P. Distance deviation measure of contouring variability. Radiol Oncol 2013;47:86–96. https://doi.org/10.2478/raon-20130005. [44] Fotina I, Lütgendorf-Caucig C, Stock M, Pötter R, Georg D. Critical discussion of evaluation parameters for inter-observer variability in target definition for radiation therapy. Strahlenther Onkol 2012;188:160–7. https://doi.org/ 10.1007/s00066-011-0027-6. [45] Eriksen JG, Cacicedo J, Navarro-Martin A, Petit C, Atencio Rosselot MC, Lazaryan A, et al. PO-0864: Long-term impact on contouring skills can be achieved by online learning. An ESTRO-FALCON-IAEA study. Radiother. Oncol. 2018;127:S452–3. https://doi.org/10.1016/S0167-8140(18)31174-5. [46] Schwartzstein RM, Roberts DH. Saying Goodbye to Lectures in Medical School Paradigm Shift or Passing Fad? N Engl J Med 2017;377:605–7. https://doi.org/ 10.1056/NEJMp1706474.