- Email: [email protected]
Cumulative sum failure analysis of the learning curve with endovascular abdominal aortic aneurysm repair
Cumulative sum failure analysis of the learning curve with endovascular abdominal aortic aneurysm repair Thomas L. Forbes, MD,a Guy DeRose, MD,a Stewart W. Kribs, MD,b and Kenneth A. Harris, MD,a London, Ontario, Canada Purpose: The purpose of this study was to evaluate the importance of experience and the learning curve with endovascular abdominal aortic aneurysm (AAA) repair. Methods: A retrospective analysis was performed of all elective endovascular AAA repairs attempted by an individual surgeon and radiologist over a 4-year period. The primary outcome variable was achievement and 30-day maintenance of initial clinical success as defined by the Society for Vascular Surgery/American Association of Vascular Surgery reporting standards. Following standard statistical analysis, the cumulative sum (CUSUM) method was used to analyze the learning curve, with a predetermined acceptable failure rate of 10% and calculated 80% alert and 95% alarm lines. Results: Ninety-six elective endovascular AAA repairs were attempted by this team between 1998 and 2002 (mean age 74 ⴞ 0.8 years; mean aneurysm diameter 5.98 ⴞ 0.8 cm). Initial clinical success was achieved and maintained in 85 of 96 patients (88.5%). Although results were acceptable throughout the study period, improved results with respect to the target failure rate (10%) were not achieved until 60 patients were treated. The learning or CUSUM curves did not differ for different device manufacturers, with improved results being achieved following 20 implantations of each device. The results did differ when comparing aortouniiliac grafts (n ⴝ 27) and bifurcated grafts (n ⴝ 64). Results with bifurcated grafts remained consistent throughout the study period, whereas with aortouniiliac grafts, results improved after only a few procedures in comparison with the target failure rate. Conclusion: Success rates with endovascular aneurysm repair will improve with an individual’s experience. The CUSUM method is a valuable tool in the evaluation of this learning curve, which has credentialing and training implications. Although acceptable results were obtained throughout the study period, this analysis indicates that 60 endovascular aneurysm repairs, or 20 with an individual device, are necessary before optimal rates of initial clinical success can be achieved. These results can be achieved more readily with aortouniiliac grafts than with bifurcated grafts. (J Vasc Surg 2004;39:102-8.)
Since its introduction in the early 1990s,1 vascular surgeons have found that endovascular aneurysm repair can be a technically demanding procedure requiring extensive preoperative planning and, for many practitioners, a new subset of skills. As with any new intervention or procedure, there is a learning curve to consider when adopting endovascular aneurysm repair. Although there are several wellestablished and widely accepted training and credentialing guidelines for such endovascular procedures as angiograms and angioplasties,2,3 there currently exists a distinct lack of similar recommendations encompassing endovascular aneurysm repair. The purpose of this study was to review the learning curve with respect to elective endovascular abdominal aortic aneurysm repair, using a unique analytical tool.
The contribution of different device manufacturers and designs was also examined. Such learning curve analyses can certainly have training and credentialing implications. Previous single-institution or physician-specific learning curve analyses employing standard statistical methods have been disadvantaged, given their retrospective nature and inability to fully appreciate the role of case volumes in clinical outcomes. This study uses the cumulative sum failure method (CUSUM), which is uniquely suited to learning-curve analysis. Specific advantages include the consideration of time and experience as a clinical variable and the ability of the method to be used in a prospective fashion.
From the Division of Vascular Surgerya and Department of Radiology,b London Health Sciences Centre, University of Western Ontario. Presented at the Twenty-fourth Annual Meeting of the Canadian Society for Vascular Surgery, Halifax, Nova Scotia, Oct 25-26, 2002. Competition of interest: G.D. and S.W.K. have served as paid consultants for Medtronic. Reprint requests: Thomas L. Forbes, MD, Division of Vascular Surgery, London Health Sciences Centre, University of Western Ontario, 375 South Street, N380, London ON, Canada N6A 4G5 (e-mail: [email protected]). 0741-5214/2004/$30.00 ⫹ 0 Copyright © 2004 by The Society for Vascular Surgery. doi:10.1016/S0741-5214(03)00922-4
The study group consisted of all patients over a 4-year period (1998-2002) who underwent elective endovascular abdominal aortic aneurysm repair by a team consisting of an individual vascular surgeon (G.D.) and interventional radiologist (S.W.K.) at a university-affiliated medical center. A retrospective review of our endovascular database by an independent reviewer was performed to obtain demographic and procedural information. The primary outcome variable was initial or 30-day clinical success as defined by the Ad Hoc Committee for Standardized Reporting Practices in Vascular Surgery of The Society for Vascular Sur-
102
METHODS
JOURNAL OF VASCULAR SURGERY Volume 39, Number 1
gery/American Association for Vascular Surgery.4 Data regarding clinical success were reported on an intent-totreat basis. Briefly, initial clinical success consisted of the following: successful deployment of the device at the intended location; absence of mortality, type I and III endoleak, graft infection, or thrombosis; absence of aneurysm expansion (diameter ⬎5 mm or volume ⬎5%), aneurysm rupture, or conversion to open repair; absence of graft migration or failure of device integrity; absence of type II endoleak with aneurysm expansion; and maintenance of above criteria for 30 days. CUSUM was then used to analyze the learning curve with endovascular AAA repair. This method recognizes the importance of time and experience in clinical practice and allows identification of improved or suboptimal performances before their recognition by standard statistical methods.5 The CUSUM calculation is defined as Sn ⫽ ⌺(Xi ⫺ Xo), where Xi ⫽ 0 for initial clinical success and Xi ⫽ 1 for initial clinical failure.5-7 After a review of the literature, Xo, or the “acceptable initial clinical failure rate,” was generously set at 10%. Individual CUSUM curves were then constructed for the entire cohort, for individual endograft manufacturers, and for individual endograft designs. Analysis of these curves involves identification of the point at which the graph adopts a generally downward slope. In this study, this point corresponded to the patient number, following which improved results with respect to the target success rate began to be observed. In addition, upper 80% alert and 95% alarm boundary lines and lower 80% reassurance lines were calculated by using a “target initial clinical failure rate” of 10% and an alternative failure rate of 20%, which signifies the value at which one wishes to be alerted if the failure rate rises to this level. CUSUM curves that cross these 80% and 95% lines correspond to statistical significance for the entire plotted cohort at the P ⬍ .20 and P ⬍ .05 levels, respectively. All summary data are represented by mean ⫾ SEM. The formulae used to construct these CUSUM curves are presented in the Appendix (online only). RESULTS Ninety-six patients underwent elective endovascular repair of an asymptomatic infrarenal abdominal aortic aneurysm during this 4-year time period (1998-2002). All cases were performed by a team consisting of an individual vascular surgeon (G.D.) and interventional radiologist (S.W.K.) and were reviewed for the purposes of this study by an independent observer. Data describing patient demographics and endograft manufacturers and designs are summarized in Table I. The majority of the study patients were male (84.4%), with a mean age of 74 ⫾ 0.8 years. The mean aneurysm diameter was 5.98 ⫾ 0.8 cm. All patients were assessed as being at high risk for standard open aneurysm repair because of their comorbid medical conditions. Preoperative contrast-enhanced computed tomography scans and aortograms were performed to confirm aneurysm suitability for endovascular repair. All endovascular repairs
Forbes et al 103
Table I. Patient demographics and endograft data Characteristic Mean age (years) Percentage male Mean aneurysm diameter (cm) Device manufacturer Device design
Data 74 ⫾ 0.8 84.4% 5.98 ⫾ 0.8 Talent (Medtronic) (n ⫽ 55) Vanguard (Boston Scientific) (n ⫽ 34) Zenith (Cook) (n ⫽ 7) Bifurcated (n ⫽ 64) Aortouniiliac (AUI) (n ⫽ 27) Tube/straight (n ⫽ 5)
Table II. Causes of failure
Initial clinical success rate Mortality Conversion to open repair Type I endoleak Limb thrombosis
Number (N ⫽ 96)
Percentage
85 4 3 2 2
88.5 4.2 3.1 2.1 2.1
were performed in the operating room with portable C-arm fluoroscopy. Three different device manufacturers supplied endografts for these patients. The Talent graft (MedtronicWorld Medical, Sunrise, Fla) was used in the majority of cases (n ⫽ 55, 57.3%), followed by the Vanguard (Boston Scientific, Natick, Mass; n ⫽ 34, 35.4%) and Zenith (Cook, Bjaeverskov, Denmark; n ⫽ 7, 7.3%) devices. Two thirds of the patients were treated with bifurcated devices (n ⫽ 64, 66.7%), whereas close to one third of patients were treated with an aortouniiliac (AUI) design and completed with a femoral crossover graft (n ⫽ 27, 28%). The distribution and frequency of cases in which AUI endografts were used did not change significantly throughout the study, although there was a trend toward more AUI grafts in our most recent experience. At the beginning of the study period, five patients were treated with a straight tube endograft (5.2%). These data are summarized in Table I. Initial or 30-day clinical success was achieved in 85 of 96 patients (88.5%). The causes for failure are listed in Table II. Frequency of failure did not differ between the various endograft manufacturers. Four patients died within 30 days of their endovascular repair as a result of cardiac complications, resulting in an early mortality rate of 4.2%. This was the most common cause of failure, followed by the need for conversion to open repair in three patients (3.1%), a persistent type I endoleak at 30 days in two patients (2.1%), and endograft limb thrombosis in two patients (2.1%). Conversion to an open procedure was required in one patient following avulsion of the external iliac artery with device delivery, whereas two further patients required conversion as a result of endograft maldeployment and resultant renal ischemia. Both patients who suffered from
104 Forbes et al
JOURNAL OF VASCULAR SURGERY January 2004
Fig 1. CUSUM: Initial clinical success.
Fig 2. CUSUM: Device manufacturer.
early limb thrombosis were treated initially with a bifurcated endograft and were revascularized with a femoral crossover graft and an axillofemoral bypass respectively. Two patients had a type I endoleak at 30 days that had not been evident on the completion aortogram. One endoleak subsequently sealed spontaneously, whereas the other patient was successfully treated with a proximal aortic extension. In all cases, an aortogram performed at the completion of the endovascular procedure and a contrastenhanced computed tomography scan and plain abdominal radiograph at 30 days were used to document any early endograft-related complications. CUSUM analysis was subsequently performed, and the results are summarized in Figs 1-3. These curves include upper 80% “alert” and 95% “alarm” lines and a lower 80% “reassurance” line constructed by using the formulae described in the Appendix (online only), with a target initial clinical failure rate (p0) of 10% and an alternative clinical failure rate (p1) of 20%. The alternative initial clinical failure rate represents the level at which one wishes to be alerted if the failure rate rises to this level. The upper boundary lines represent an unacceptably high failure rate, whereas the lower boundary line indicates improved results with respect to the targeted failure rate (10%). Endovascular cases are then plotted sequentially on the horizontal axis, producing CUSUM curves that recognize time and experience as variables. The slope of the resulting curve is important in analyzing this cumulative data. A flat or horizontal plot represents satisfactory results compared with the target failure rate. An upward-sloping curve represents deviation from acceptable failure rates toward an unacceptably high rate, whereas a downward-sloping curve depicts improved results with respect to the target failure rate. In Fig 1, all 96 patients who underwent attempted endovascular aneurysm repair are plotted sequentially. The
resulting CUSUM curve remains relatively horizontal and well within the acceptable range for the first 60 patients. There is no deviation to the upper “alert” and “alarm” boundary lines. Following 60 patients, the curve adopts a generally negative, downward slope, indicating that improved results compared with the 10% target failure rate are beginning to be achieved. Alternatively, although results were acceptable throughout the study period, improved results with respect to the target failure rate were not achieved until 60 patients were treated. Following the treatment of approximately 75 patients, the curve crosses and remains below the lower 80% “reassurance” line, indicating statistical significance for the entire cohort compared with the target success rate at the P ⬍ .20 level. In Fig 2 the role of different device manufacturers is shown, with a CUSUM curve plotted for both the Vanguard and Talent devices. The curves for these manufacturers do not differ. The curves pertaining to the Vanguard and Talent devices remain within the acceptable range for the first 20 patients, at which point the curves generally adopt a negative slope. Alternatively, optimal or improved results with respect to the target failure rate begin to be achieved following 20 endovascular repairs with either device. There is insufficient data to formulate a curve for the Zenith device, as it was deployed in only seven patients during this study. In Fig 3, the role of various device designs is considered. Aortoaortic tube devices were only used in five cases, so an adequate CUSUM analysis is impractical. However, when the curves describing our experience with AUI and bifurcated endografts are compared, there is a clear difference. Although results with bifurcated grafts remain consistent and acceptable throughout the study period, results with the AUI endografts improved after only several pro-
JOURNAL OF VASCULAR SURGERY Volume 39, Number 1
Fig 3. CUSUM: Device design.
cedures in comparison with the target failure rate. Fewer repairs with the AUI design were required to achieve optimal results compared with bifurcated grafts. DISCUSSION Although it seems reasonable that results with respect to any technical procedure will improve with experience, few studies have attempted to analyze the learning curve associated with endovascular aneurysm repair. Those that have been published have used a variety of statistical methods to examine this learning curve.8-11 These studies often fail to make recommendations that are applicable to an individual surgeon’s learning curve and experience. Lee and colleagues8 from Stanford described their experience with a single device (AneuRx, Medtronic AVE, Santa Rosa, Calif) by dividing their first 150 consecutive patients into two temporal groups of 75 cases each. This group used the achievement of technical success as the major outcome variable.12 A standard statistical analysis revealed no significant difference between the two groups with respect to technical success. However, there was a statistically significant decreased need for femoral artery reconstruction and fluoroscopy time in the second group. In addition, this analysis noted a trend toward less need for proximal extender cuffs in the second group of 75 patients. On the basis of their study, the Stanford group concluded that a learning curve did exist and that with more experience, decreased fluoroscopy times, more accurate deployment, and fewer access complications could be accomplished. By using a single device, this analysis does not allow one to comment on the role of different device manufacturers in the learning curve. As well, this form of analysis is not able to offer recommendations with respect to the
Forbes et al 105
volume of cases necessary to achieve optimal results with endovascular aneurysm repair. As an institutional experience, these results may not be applicable to an individual surgeon’s learning curve. Lobato and the Arizona Heart Institute group9 analyzed the learning curve by using a first-order differential equation to examine their 277 cases. As with the Stanford group, technical success was the main outcome variable.12 This study concluded that 55 endovascular aneurysm repairs were needed to achieve optimal results and that the time interval between procedures was also important, with one case every 10 days being the minimal frequency to achieve these success rates. This group used several different device manufacturers and designs and found that these did not influence the learning curve. As with the previous study, this institutional experience may not be transferable to an individual practitioner’s learning experience. The EUROSTAR collaborators10 analyzed the contribution of case volumes to results in their extensive database. Main outcomes were patient mortality, aneurysm rupture, and the need for secondary endovascular interventions. Specialist teams, using several device manufacturers, were ranked into quartiles on the basis of case volumes, with the lowest-experience quartile being the first 11 cases and the highest quartile being patient 92 or more. The study concluded that more experienced endovascular teams had lower mortality rates and fewer complications leading to secondary interventions. Aneurysm rupture did not change with experience. The authors suggested that 38 of their patients needed to be treated before a decrease in perioperative mortality and 92 patients may have needed treatment before a reduction in secondary interventions could be achieved. On the basis of these data, the authors recommended that the mentoring or proctoring process with endovascular aneurysm repair needs to be longer than is currently the case. As opposed to these retrospective analyses of institutional experiences, an individual surgeon’s learning curve would optimally be assessed as results are being achieved. Although applied retrospectively in the present study, the CUSUM failure technique is appropriate for prospective learning curve analyses. This would allow for the recognition of suboptimal outcomes earlier than their identification with standard post hoc statistical analytical tools. The CUSUM method was first described in 1950,13 and its value as an analytical tool for medical data was almost immediately realized.14 Essentially, CUSUM analysis repeatedly applies a sequential probability test to any data whose results can be simplified to either a “success” or “failure.” In this study, success is the achievement of initial clinical success, and failure is the lack of this achievement. A target or reference rate of performance is determined, and the slope of the resulting graph is analyzed. Suboptimal results compared with the target failure rate are represented by a positive slope, whereas a negative slope demonstrates improved results compared with the target rate.5,15,16 Despite its relative simplicity, the CUSUM method has been underused in the analysis of surgical data. This has
106 Forbes et al
changed somewhat recently with the publication of several reports in which various groups have described their use of CUSUM analysis in the setting of colonoscopy,17 abdominal ultrasound in trauma patients,18 the treatment of myocardial infarction,19 surgical training,20 and cardiac surgery.21 At our institution, Novick and colleagues have increased the awareness of this tool by reviewing the learning curve in a cardiac surgeon’s early years of practice6 as well as change to off-pump coronary artery bypass grafting.7,22 This experience prompted our group to use this analytical tool to review the learning curve of an individual surgeon with respect to repair of ruptured abdominal aortic aneurysms.23 In this study we set out to analyze our learning curve with endovascular aneurysm repair with this unique analytical tool. The target failure rate was obtained following a review of the literature, in which technical and clinical success rates with endovascular aneurysm repair ranged from 73.3% to 95%.24-32 For the purpose of this learning curve analysis, the target initial clinical success rate was set at 90%, or a target (or acceptable) failure rate of 10%. This measure of success was chosen for this analysis as it takes into account short-term follow-up along with initial procedural technical success. Initial clinical success, as defined by the Society for Vascular Surgery/American Association for Vascular Surgery, is a better marker of a surgeon’s experience and learning curve than is technical success, as it considers the often-intensive patient and device selection phase of endovascular procedures along with the actual technical performance of the procedure. Certainly several of the factors necessary for initial clinical success (the need for conversion to open repair, and type I endoleak) have been observed less frequently with increased experience.27,33 The period of preoperative planning often plays more of a role in achieving optimal results than does the actual performance of the endovascular repair. Following the initial perioperative period, reasons other than surgeon experience that are more specific to patient and graft dynamics may result in failure of the endovascular approach. Up to this point, few studies have investigated the role of surgical volumes with outcomes in vascular surgery.23,34,35 The CUSUM failure method is uniquely suited to analyze this relationship with respect to an individual’s experience. This study reviewed the learning curve of an endovascular team comprising a vascular surgeon and interventional radiologist. The 88.5% initial clinical success rate achieved by this team over a 4-year period compares favorably with published data.24-33,36-42 A sequential CUSUM plot of all endovascular repairs performed by this team (Fig 1) indicates that 60 endovascular aneurysm repairs are required to achieve a target initial clinical success rate of 90%. This case volume is irrespective of device design or manufacturer. Although similar to the results obtained by the institutional learning curve analysis performed at the Arizona Heart Institute,9 this study’s results are more applicable to what an individual may expect with respect to his or her learning experience.
JOURNAL OF VASCULAR SURGERY January 2004
Previous studies have failed to reveal a role for different device manufacturers in the evolution of results over time with endovascular aneurysm repair.9,10 However, as specific device manufacturers differ with respect to preoperative case planning, device design, measurements, and deployment, it is not unreasonable to expect a manufacturerspecific learning curve.26,32,39,42 As several procedural steps are common to all endovascular repairs, we recognize that the learning curve with one device influences results with other devices, with there being significant interaction in the acquisition of experience. Despite this interaction, this study attempted to compare the learning curves with respect to several different endograft manufacturers. A sequential CUSUM graph of aneurysm repairs performed with different device manufacturers (Fig 2) indicated that 20 repairs are required for a team, or individual, to obtain optimal results, defined in this study as an initial clinical success rate of 90%. This did not differ between the two devices that were predominantly used at our institution during this time period. Although we have not had extensive experience with other manufacturers, we see no reason why this manufacturer-specific learning curve should not be applicable to the use of other companies’ designs. During the relatively brief history of endovascular aneurysm repair, it has become apparent that endograft device design is a vital contributor to successful aneurysm exclusion.37 As in other institutions, several aortoaortic tube endografts were deployed at our center early in our endovascular experience. We have since abandoned the use of this design given the widespread concern regarding distal aortic attachment site complications and device migration.24 Since then, two thirds (66.7%) of the endovascular aneurysm repairs performed by this team have used bifurcated grafts, whereas AUI devices were deployed in approximately one third of cases (28%). A CUSUM plot of our experience with these different designs (Fig 3) confirms that satisfactory results, with respect to target initial clinical success rates, can be obtained with either design. However, vastly improved results compared with target rates are obtained following fewer repairs with AUI devices than with bifurcated endografts. This is in no small part the result of an increased risk of limb thrombosis with bifurcated grafts. Although this trend has not been universally observed, it is felt that the use of a bifurcated graft is one factor that may contribute to limb thrombosis, along with other factors such as the use of an unsupported limb, small limb diameter, and extension to the external iliac artery.25,40,41 In comparison, very few instances of endograft limb or femoral crossover graft thrombosis have been reported following the use of an AUI device. In this instance, published primary and secondary patency rates approach 98% and 100%, respectively.43 For these reasons, we have developed a distinct preference for the AUI design in our most recent endovascular experience and feel that it has wider applicability than the bifurcated design in patients with extensive aortoiliac aneurysmal disease. A limitation of this analysis is that, although it is useful in assessing an individual’s learning curve, it does not allow
JOURNAL OF VASCULAR SURGERY Volume 39, Number 1
comment on the time interval between cases. It would be logical to expect improved outcomes with shorter time intervals between interventions. Other forms of analyses, such as a first-order differential equation, have been used to evaluate this relationship.9 In addition, the length of the learning curve is inversely proportional to the predetermined target or acceptable failure rate. Shorter learning curves can be achieved by setting the acceptable failure rate at a higher level. Although this rate is somewhat arbitrarily set, it should be derived from previously published data, as in this study. Learning-curve analyses such as these have training and credentialing implications. Although training guidelines have been published,2,3,44 few have determined sufficient case volumes of endovascular aneurysm repair to obtain optimal results. This study reinforces the importance of the proctoring system during the preoperative planning phase as well as the technical application of this technology. Our group received industry-provided technical training followed by supervision by an experienced endovascular surgeon during the preoperative planning phase and intraoperative phase for the first half-dozen cases. Following this period, the various manufacturers provided on-site technical support and advice during the procedures, for varying durations. Some investigators have advocated the use of simulators in endovascular training.45 Although this may improve one’s technical endovascular skills, simulators alone are not sufficient to obtain optimal clinical results. These results can only be achieved with further experience in the preoperative planning and measurement phase of endovascular aneurysm repair, along with superior technical skills. CONCLUSIONS This study illustrates the value of the CUSUM technique in evaluating an individual’s learning curve. Although acceptable results were achieved throughout the study period with respect to endovascular aneurysm repair, this study’s results indicate that 60 elective endovascular repairs, or 20 with an individual device manufacturer, are required to obtain optimal results with respect to initial clinical success. In addition, these improved results can be achieved with fewer repairs with AUI devices than with bifurcated endografts. We thank Dr Richard Novick for his assistance and advice with respect to the CUSUM calculations and statistical analysis. REFERENCES 1. Parodi JC, Palmaz JC, Barone HD. Transfemoral intraluminal graft implantation for abdominal aortic aneurysms. Ann Vasc Surg 1991;5: 491-9. 2. Van Breda A, Becker GJ. Endovascular credentialing. J Vas Inverv Radiol 1995;5:90-2. 3. White RA, Hodgson KJ, Ahn SS, Hobson RW, Veith FJ. Endovascular interventions training and credentialing for vascular surgeons. J Vasc Surg 1999;29:177-86. 4. Chaikof EL, Blankensteijn JD, Harris PL, White GH, Zarins CK, Bernhard VM, et al, for the Ad Hoc Committee for Standardized
Forbes et al 107
5. 6. 7.
8.
9.
10.
11.
12.
13. 14. 15. 16. 17. 18.
19.
20.
21.
22.
23. 24.
25.
26.
Reporting Practices in Vascular Surgery of The Society of Vascular Surgery/American Association for Vascular Surgery. Reporting standards for endovascular aortic aneurysm repair. J Vasc Surg 2002;35: 1048-60. Williams SM, Parry BR, Schlup MMT. Quality control: an application of the CUSUM. BMJ 1992;304:1359-61. Novick RJ, Stitt LW. The learning curve of an academic cardiac surgeon: use of the CUSUM method. J Card Surg 1999;14:312-20. Novick RJ, Fox SA, Stitt LW, Swinamer SA, Lehnhardt KR, Rayman R, et al. Cumulative sum failure analysis of a policy change from on-pump to off-pump coronary artery bypass grafting. Ann Thorac Surg 2001; 72:S1016-21. Lee WA, Wolf YG, Hill BB, Cipriano P, Fogarty TJ, Zarins CK. The first 150 endovascular AAA repairs at a single institution: how steep is the learning curve? J Endovasc Ther 2002;9:269-76. Lobato AC, Rodriguez-Lopez J, Diethrich EB. Learning curve for endovascular abdominal aortic aneurysm repair: evaluation of a 277patient single-centre experience. J Endovasc Ther 2002;9:262-68. Laheij RJF, van Marrewijk CJ, Buth J, Harris PL. The influence of team experience on outcomes of endovascular stenting of abdominal aortic aneurysms. Eur J Vasc Endovasc Surg 2002;24:128-33. Zarins CK, Shaver DM, Arko FR, Schubart PJ, Lengle SJ, Dixon SM. Introduction of endovascular aneurysm repair into community practice: initial results with a new Food and Drug Administration-approved device. J Vasc Surg 2002;36:226-33. Ahn SS, Rutherford RB, Johnston KW, May J, Veith FJ, Baker JD, et al. for the Ad Hoc Committee for Standardized Reporting Practices in Vascular Surgery of The Society for Vascular Surgery/International Society for Cardiovascular Surgery. Reporting standards for infrarenal endovascular abdominal aortic aneurysm repair. J Vasc Surg 1997;25: 405-10. Hurst HE. Proc Am Soc Civil Eng 1950;76:11. Healy MJR. The disciplining of medical data. Br Med Bull 1968;24: 210-14. Chaput de Saintonge DM, Vere DW. Why don’t doctors use CUSUMs? Lancet 1974;1:120-1. Wohl H. The CUSUM plot: its utility in the analysis of clinical data. N Eng J Med 1977;296:1044-5. Parry BR, Williams SM. Competency and the colonoscopist: a learning curve. Aust N Z J Surg 1991;61:419-22. McCarter FD, Luchette FA, Molloy M, Hurst JM, Davis K, Johannigman JA, et al. Institutional and individual learning curves for focused abdominal ultrasound for trauma. Ann Surg 2000;231:689-700. Lawrance RA, Dorsch MF, Sapsford RJ, Mackintosh AF, Greenwood DC, Jackson BM, et al. Use of cumulative mortality data in patients with acute myocardial infarction for early detection of variation in clinical practice: observational study. BMJ 2001;323:324-7. Van Rij AM, McDonald JR, Pettigrew RA, Putterill MJ, Reddy CK, Wright JJ. Cusum as an aid to early assessment of the surgical trainee. Br J Surg 1995;82:1500-3. de Leval MR, Francois K, Bull C, Brawn W, Spiegelhalter D. Analysis of a cluster of surgical failures. Application to a series of neonatal arterial switch operations. J Thorac Cardiovasc Surg 1994;107:914-23. Novick RJ, Fox SA, Stitt LW, Kiaii BB, Swinamer SA, Rayman R, et al. Assessing the learning curve in off-pump coronary artery surgery via CUSUM failure analysis. Ann Thorac Surg 2002;73:S358-62. Forbes TL, DeRose G, Harris KA. A CUSUM analysis of ruptured abdominal aortic aneurysm repair. Ann Vasc Surg 2002;16:527-33. Faries PL, Briggs VL, Rhee JY, Burks JA, Gravereaux EC, Carroccio A, et al. Failure of endovascular aortoaortic tube grafts; a plea for preferential use of bifurcated grafts. J Vasc Surg 2002;35:868-73. Moore WS, Brewster DC, Bernhard VM, for the EVT/Guidant Investigators. Aorto-uni-iliac endograft for complex aortoiliac aneurysms compared with tube/bifurcation endografts: results of the EVT/ Guidant trials. J Vasc Surg 2001;33(2 Suppl):S11-20. Zarins CK, White RA, Schwarten D, Kinney E, Diethrich EB, Hodgson KJ, et al, for the investigators of the Medtronic AneuRx Multicenter Clinical Trial. AneuRx stent graft versus open surgical repair of abdominal aortic aneurysms: multicenter prospective clinical trial. J Vasc Surg 1999;29:292-308.
JOURNAL OF VASCULAR SURGERY January 2004
108 Forbes et al
27. Buth J, Laheij RJF, on behalf of the EUROSTAR Collaborators. Early complications and endoleaks after endovascular abdominal aortic aneurysm repair: report of a multicenter study. J Vasc Surg 2000;31:134-46. 28. Chuter TAM, Reilly LM, Faruqi RM, Kerlan RB, Sawhney R, Canto CJ, et al. Endovascular aneurysm repair in high-risk patients. J Vasc Surg 2000;31:122-33. 29. Liewald F, Scharrer-Pamler R, Go¨ rich J, Kapfer X, Seifarth H, Halter G, et al. Intraoperative, perioperative and late complications with endovascular therapy of aortic aneurysm. Eur J Vasc Endovasc Surg 2001;22: 251-6. 30. Walschot LHB, Laheij RJF, Verbeek ALM. Outcome after endovascular abdominal aortic aneurysm repair: a meta-analysis. J Endovasc Ther 2002;9:82-9. 31. Chaikof EL, Lin PH, Brinkman WT, Dodson TF, Weiss VJ, Lumsden AB, et al. Endovascular repair of abdominal aortic aneurysms: risk stratified outcomes. Ann Surg 2002;235:833-41. 32. Criado FJ, Wilson EP, Fairman RM, Abul-Khoudoud O, Wellons E. Update on the Talent aortic stent-graft: a preliminary report from United States phase I and II trials. J Vasc Surg 2001;33:S146-9. 33. Bo¨ ckler D, Probst T, Weber H, Raithel D. Surgical conversion after endovascular grafting for abdominal aortic aneurysms. J Endovasc Ther 2002;9:111-18. 34. Huber TS, Seeger JM. Dartmouth Atlas of Vascular Health Care review: impact of hospital volume, surgeon volume, and training on outcome. J Vasc Surg 2001;34:751-6. 35. Shackley P, Slack R, Booth A, Michaels J. Is there a positive volumeoutcome relationship in peripheral vascular surgery? Results of a systematic review. Eur J Vasc Endovasc Surg 2000;20:326-35. 36. Cuypers PWM, Laheij RJF, Buth J, & on behalf of the EUROSTAR Collaborators. Which factors increase the risk of conversion to open surgery following endovascular abdominal aortic aneurysm repair? Eur J Vasc Endovasc Surg 2000;20:183-9. 37. Resch T, Malina M, Lindblad B, Ivancev K. The impact of stent-graft
38.
39.
40.
41.
42.
43.
44. 45.
development on outcome of AAA repair—a 7-year experience. Eur J Vasc Endovasc Surg 2001;22:57-61. Ohki T, Veith FJ, Shaw P, Lipsitz E, Suggs WD, Wain RA, et al. Increasing incidence of midterm and long-term complications after endovascular graft repair of abdominal aortic aneurysms: a note of caution based on a 9-year experience. Ann Surg 2001;234:323-35. Abraham CZ, Chuter TAM, Reilly LM, Okuhn SP, Pethan LK, Kerlan RB, et al. Abdominal aortic aneurysm repair with the Zenith stent graft: short to midterm results. J Vasc Surg 2002;36:217-25. Carpenter JP, Neschis DG, Fairman RM, Barker CF, Golden MA, Velazquez OC, et al. Failure of endovascular abdominal aortic aneurysm graft limbs. J Vasc Surg 2001;33:296-303. Carroccio A, Faries PL, Morrissey NJ, Teodorescu V, Burks JA, Gravereaux EC, et al. Predicting iliac limb occlusions after bifurcated aortic stent grafting: anatomic and device-related causes. J Vasc Surg 2002; 36:679-84. Greenberg RK, Lawrence-Brown M, Bhandari G, Hartley D, Stelter W, Umscheid T, et al. An update of the Zenith endovascular graft for abdominal aortic aneurysms: initial implantation and mid-term follow-up data. J Vasc Surg 2001;33:S157-64. Rehring TF, Brewster DC, Cambria RP, Kaufman JA, Geller SC, Fan CM, et al. Utility and reliability of endovascular aortouniiliac with femorofemoral crossover graft for aortoiliac aneurysmal disease. J Vasc Surg 2000;31:1135-41. Kwolek CJ. Endovascular qualifications for practicing surgeons. Sem Vasc Surg 2002;15:174-7. Chong CK, Brennan J, How TV, Edwards R, Gilling-Smith GL, Harris PL. A prototype simulator for endovascular repair of abdominal aortic aneurysms. Eur J Vasc Endovasc Surg 1997;13:330-3.
Submitted May 8, 2003; accepted Jun 5, 2003.
Additional material for this article may be found online at www.mosby.com/jvs.
The JVS Ombudsman The ombudsman’s role is to act as an advocate for authors and represent their position to the editorial staff in relation to the process of manuscript submission, review, and publication. The ombudsman is not responsible for evaluating the content of a manuscript or determining whether the editors made the correct decision with regard to acceptance or rejection of the paper. If an author or other person has an unresolved complaint or question about the editorial process of the Journal, he or she should contact Dr James S. T. Yao (Northwestern University Medical School, Department of Surgery, 201 E. Huron Street, Suite 10-105, Chicago, IL 60611), who will review the matter.
Since its introduction in the early 1990s,1 vascular surgeons have found that endovascular aneurysm repair can be a technically demanding procedure requiring extensive preoperative planning and, for many practitioners, a new subset of skills. As with any new intervention or procedure, there is a learning curve to consider when adopting endovascular aneurysm repair. Although there are several wellestablished and widely accepted training and credentialing guidelines for such endovascular procedures as angiograms and angioplasties,2,3 there currently exists a distinct lack of similar recommendations encompassing endovascular aneurysm repair. The purpose of this study was to review the learning curve with respect to elective endovascular abdominal aortic aneurysm repair, using a unique analytical tool.
The contribution of different device manufacturers and designs was also examined. Such learning curve analyses can certainly have training and credentialing implications. Previous single-institution or physician-specific learning curve analyses employing standard statistical methods have been disadvantaged, given their retrospective nature and inability to fully appreciate the role of case volumes in clinical outcomes. This study uses the cumulative sum failure method (CUSUM), which is uniquely suited to learning-curve analysis. Specific advantages include the consideration of time and experience as a clinical variable and the ability of the method to be used in a prospective fashion.
From the Division of Vascular Surgerya and Department of Radiology,b London Health Sciences Centre, University of Western Ontario. Presented at the Twenty-fourth Annual Meeting of the Canadian Society for Vascular Surgery, Halifax, Nova Scotia, Oct 25-26, 2002. Competition of interest: G.D. and S.W.K. have served as paid consultants for Medtronic. Reprint requests: Thomas L. Forbes, MD, Division of Vascular Surgery, London Health Sciences Centre, University of Western Ontario, 375 South Street, N380, London ON, Canada N6A 4G5 (e-mail: [email protected]). 0741-5214/2004/$30.00 ⫹ 0 Copyright © 2004 by The Society for Vascular Surgery. doi:10.1016/S0741-5214(03)00922-4
The study group consisted of all patients over a 4-year period (1998-2002) who underwent elective endovascular abdominal aortic aneurysm repair by a team consisting of an individual vascular surgeon (G.D.) and interventional radiologist (S.W.K.) at a university-affiliated medical center. A retrospective review of our endovascular database by an independent reviewer was performed to obtain demographic and procedural information. The primary outcome variable was initial or 30-day clinical success as defined by the Ad Hoc Committee for Standardized Reporting Practices in Vascular Surgery of The Society for Vascular Sur-
102
METHODS
JOURNAL OF VASCULAR SURGERY Volume 39, Number 1
gery/American Association for Vascular Surgery.4 Data regarding clinical success were reported on an intent-totreat basis. Briefly, initial clinical success consisted of the following: successful deployment of the device at the intended location; absence of mortality, type I and III endoleak, graft infection, or thrombosis; absence of aneurysm expansion (diameter ⬎5 mm or volume ⬎5%), aneurysm rupture, or conversion to open repair; absence of graft migration or failure of device integrity; absence of type II endoleak with aneurysm expansion; and maintenance of above criteria for 30 days. CUSUM was then used to analyze the learning curve with endovascular AAA repair. This method recognizes the importance of time and experience in clinical practice and allows identification of improved or suboptimal performances before their recognition by standard statistical methods.5 The CUSUM calculation is defined as Sn ⫽ ⌺(Xi ⫺ Xo), where Xi ⫽ 0 for initial clinical success and Xi ⫽ 1 for initial clinical failure.5-7 After a review of the literature, Xo, or the “acceptable initial clinical failure rate,” was generously set at 10%. Individual CUSUM curves were then constructed for the entire cohort, for individual endograft manufacturers, and for individual endograft designs. Analysis of these curves involves identification of the point at which the graph adopts a generally downward slope. In this study, this point corresponded to the patient number, following which improved results with respect to the target success rate began to be observed. In addition, upper 80% alert and 95% alarm boundary lines and lower 80% reassurance lines were calculated by using a “target initial clinical failure rate” of 10% and an alternative failure rate of 20%, which signifies the value at which one wishes to be alerted if the failure rate rises to this level. CUSUM curves that cross these 80% and 95% lines correspond to statistical significance for the entire plotted cohort at the P ⬍ .20 and P ⬍ .05 levels, respectively. All summary data are represented by mean ⫾ SEM. The formulae used to construct these CUSUM curves are presented in the Appendix (online only). RESULTS Ninety-six patients underwent elective endovascular repair of an asymptomatic infrarenal abdominal aortic aneurysm during this 4-year time period (1998-2002). All cases were performed by a team consisting of an individual vascular surgeon (G.D.) and interventional radiologist (S.W.K.) and were reviewed for the purposes of this study by an independent observer. Data describing patient demographics and endograft manufacturers and designs are summarized in Table I. The majority of the study patients were male (84.4%), with a mean age of 74 ⫾ 0.8 years. The mean aneurysm diameter was 5.98 ⫾ 0.8 cm. All patients were assessed as being at high risk for standard open aneurysm repair because of their comorbid medical conditions. Preoperative contrast-enhanced computed tomography scans and aortograms were performed to confirm aneurysm suitability for endovascular repair. All endovascular repairs
Forbes et al 103
Table I. Patient demographics and endograft data Characteristic Mean age (years) Percentage male Mean aneurysm diameter (cm) Device manufacturer Device design
Data 74 ⫾ 0.8 84.4% 5.98 ⫾ 0.8 Talent (Medtronic) (n ⫽ 55) Vanguard (Boston Scientific) (n ⫽ 34) Zenith (Cook) (n ⫽ 7) Bifurcated (n ⫽ 64) Aortouniiliac (AUI) (n ⫽ 27) Tube/straight (n ⫽ 5)
Table II. Causes of failure
Initial clinical success rate Mortality Conversion to open repair Type I endoleak Limb thrombosis
Number (N ⫽ 96)
Percentage
85 4 3 2 2
88.5 4.2 3.1 2.1 2.1
were performed in the operating room with portable C-arm fluoroscopy. Three different device manufacturers supplied endografts for these patients. The Talent graft (MedtronicWorld Medical, Sunrise, Fla) was used in the majority of cases (n ⫽ 55, 57.3%), followed by the Vanguard (Boston Scientific, Natick, Mass; n ⫽ 34, 35.4%) and Zenith (Cook, Bjaeverskov, Denmark; n ⫽ 7, 7.3%) devices. Two thirds of the patients were treated with bifurcated devices (n ⫽ 64, 66.7%), whereas close to one third of patients were treated with an aortouniiliac (AUI) design and completed with a femoral crossover graft (n ⫽ 27, 28%). The distribution and frequency of cases in which AUI endografts were used did not change significantly throughout the study, although there was a trend toward more AUI grafts in our most recent experience. At the beginning of the study period, five patients were treated with a straight tube endograft (5.2%). These data are summarized in Table I. Initial or 30-day clinical success was achieved in 85 of 96 patients (88.5%). The causes for failure are listed in Table II. Frequency of failure did not differ between the various endograft manufacturers. Four patients died within 30 days of their endovascular repair as a result of cardiac complications, resulting in an early mortality rate of 4.2%. This was the most common cause of failure, followed by the need for conversion to open repair in three patients (3.1%), a persistent type I endoleak at 30 days in two patients (2.1%), and endograft limb thrombosis in two patients (2.1%). Conversion to an open procedure was required in one patient following avulsion of the external iliac artery with device delivery, whereas two further patients required conversion as a result of endograft maldeployment and resultant renal ischemia. Both patients who suffered from
104 Forbes et al
JOURNAL OF VASCULAR SURGERY January 2004
Fig 1. CUSUM: Initial clinical success.
Fig 2. CUSUM: Device manufacturer.
early limb thrombosis were treated initially with a bifurcated endograft and were revascularized with a femoral crossover graft and an axillofemoral bypass respectively. Two patients had a type I endoleak at 30 days that had not been evident on the completion aortogram. One endoleak subsequently sealed spontaneously, whereas the other patient was successfully treated with a proximal aortic extension. In all cases, an aortogram performed at the completion of the endovascular procedure and a contrastenhanced computed tomography scan and plain abdominal radiograph at 30 days were used to document any early endograft-related complications. CUSUM analysis was subsequently performed, and the results are summarized in Figs 1-3. These curves include upper 80% “alert” and 95% “alarm” lines and a lower 80% “reassurance” line constructed by using the formulae described in the Appendix (online only), with a target initial clinical failure rate (p0) of 10% and an alternative clinical failure rate (p1) of 20%. The alternative initial clinical failure rate represents the level at which one wishes to be alerted if the failure rate rises to this level. The upper boundary lines represent an unacceptably high failure rate, whereas the lower boundary line indicates improved results with respect to the targeted failure rate (10%). Endovascular cases are then plotted sequentially on the horizontal axis, producing CUSUM curves that recognize time and experience as variables. The slope of the resulting curve is important in analyzing this cumulative data. A flat or horizontal plot represents satisfactory results compared with the target failure rate. An upward-sloping curve represents deviation from acceptable failure rates toward an unacceptably high rate, whereas a downward-sloping curve depicts improved results with respect to the target failure rate. In Fig 1, all 96 patients who underwent attempted endovascular aneurysm repair are plotted sequentially. The
resulting CUSUM curve remains relatively horizontal and well within the acceptable range for the first 60 patients. There is no deviation to the upper “alert” and “alarm” boundary lines. Following 60 patients, the curve adopts a generally negative, downward slope, indicating that improved results compared with the 10% target failure rate are beginning to be achieved. Alternatively, although results were acceptable throughout the study period, improved results with respect to the target failure rate were not achieved until 60 patients were treated. Following the treatment of approximately 75 patients, the curve crosses and remains below the lower 80% “reassurance” line, indicating statistical significance for the entire cohort compared with the target success rate at the P ⬍ .20 level. In Fig 2 the role of different device manufacturers is shown, with a CUSUM curve plotted for both the Vanguard and Talent devices. The curves for these manufacturers do not differ. The curves pertaining to the Vanguard and Talent devices remain within the acceptable range for the first 20 patients, at which point the curves generally adopt a negative slope. Alternatively, optimal or improved results with respect to the target failure rate begin to be achieved following 20 endovascular repairs with either device. There is insufficient data to formulate a curve for the Zenith device, as it was deployed in only seven patients during this study. In Fig 3, the role of various device designs is considered. Aortoaortic tube devices were only used in five cases, so an adequate CUSUM analysis is impractical. However, when the curves describing our experience with AUI and bifurcated endografts are compared, there is a clear difference. Although results with bifurcated grafts remain consistent and acceptable throughout the study period, results with the AUI endografts improved after only several pro-
JOURNAL OF VASCULAR SURGERY Volume 39, Number 1
Fig 3. CUSUM: Device design.
cedures in comparison with the target failure rate. Fewer repairs with the AUI design were required to achieve optimal results compared with bifurcated grafts. DISCUSSION Although it seems reasonable that results with respect to any technical procedure will improve with experience, few studies have attempted to analyze the learning curve associated with endovascular aneurysm repair. Those that have been published have used a variety of statistical methods to examine this learning curve.8-11 These studies often fail to make recommendations that are applicable to an individual surgeon’s learning curve and experience. Lee and colleagues8 from Stanford described their experience with a single device (AneuRx, Medtronic AVE, Santa Rosa, Calif) by dividing their first 150 consecutive patients into two temporal groups of 75 cases each. This group used the achievement of technical success as the major outcome variable.12 A standard statistical analysis revealed no significant difference between the two groups with respect to technical success. However, there was a statistically significant decreased need for femoral artery reconstruction and fluoroscopy time in the second group. In addition, this analysis noted a trend toward less need for proximal extender cuffs in the second group of 75 patients. On the basis of their study, the Stanford group concluded that a learning curve did exist and that with more experience, decreased fluoroscopy times, more accurate deployment, and fewer access complications could be accomplished. By using a single device, this analysis does not allow one to comment on the role of different device manufacturers in the learning curve. As well, this form of analysis is not able to offer recommendations with respect to the
Forbes et al 105
volume of cases necessary to achieve optimal results with endovascular aneurysm repair. As an institutional experience, these results may not be applicable to an individual surgeon’s learning curve. Lobato and the Arizona Heart Institute group9 analyzed the learning curve by using a first-order differential equation to examine their 277 cases. As with the Stanford group, technical success was the main outcome variable.12 This study concluded that 55 endovascular aneurysm repairs were needed to achieve optimal results and that the time interval between procedures was also important, with one case every 10 days being the minimal frequency to achieve these success rates. This group used several different device manufacturers and designs and found that these did not influence the learning curve. As with the previous study, this institutional experience may not be transferable to an individual practitioner’s learning experience. The EUROSTAR collaborators10 analyzed the contribution of case volumes to results in their extensive database. Main outcomes were patient mortality, aneurysm rupture, and the need for secondary endovascular interventions. Specialist teams, using several device manufacturers, were ranked into quartiles on the basis of case volumes, with the lowest-experience quartile being the first 11 cases and the highest quartile being patient 92 or more. The study concluded that more experienced endovascular teams had lower mortality rates and fewer complications leading to secondary interventions. Aneurysm rupture did not change with experience. The authors suggested that 38 of their patients needed to be treated before a decrease in perioperative mortality and 92 patients may have needed treatment before a reduction in secondary interventions could be achieved. On the basis of these data, the authors recommended that the mentoring or proctoring process with endovascular aneurysm repair needs to be longer than is currently the case. As opposed to these retrospective analyses of institutional experiences, an individual surgeon’s learning curve would optimally be assessed as results are being achieved. Although applied retrospectively in the present study, the CUSUM failure technique is appropriate for prospective learning curve analyses. This would allow for the recognition of suboptimal outcomes earlier than their identification with standard post hoc statistical analytical tools. The CUSUM method was first described in 1950,13 and its value as an analytical tool for medical data was almost immediately realized.14 Essentially, CUSUM analysis repeatedly applies a sequential probability test to any data whose results can be simplified to either a “success” or “failure.” In this study, success is the achievement of initial clinical success, and failure is the lack of this achievement. A target or reference rate of performance is determined, and the slope of the resulting graph is analyzed. Suboptimal results compared with the target failure rate are represented by a positive slope, whereas a negative slope demonstrates improved results compared with the target rate.5,15,16 Despite its relative simplicity, the CUSUM method has been underused in the analysis of surgical data. This has
106 Forbes et al
changed somewhat recently with the publication of several reports in which various groups have described their use of CUSUM analysis in the setting of colonoscopy,17 abdominal ultrasound in trauma patients,18 the treatment of myocardial infarction,19 surgical training,20 and cardiac surgery.21 At our institution, Novick and colleagues have increased the awareness of this tool by reviewing the learning curve in a cardiac surgeon’s early years of practice6 as well as change to off-pump coronary artery bypass grafting.7,22 This experience prompted our group to use this analytical tool to review the learning curve of an individual surgeon with respect to repair of ruptured abdominal aortic aneurysms.23 In this study we set out to analyze our learning curve with endovascular aneurysm repair with this unique analytical tool. The target failure rate was obtained following a review of the literature, in which technical and clinical success rates with endovascular aneurysm repair ranged from 73.3% to 95%.24-32 For the purpose of this learning curve analysis, the target initial clinical success rate was set at 90%, or a target (or acceptable) failure rate of 10%. This measure of success was chosen for this analysis as it takes into account short-term follow-up along with initial procedural technical success. Initial clinical success, as defined by the Society for Vascular Surgery/American Association for Vascular Surgery, is a better marker of a surgeon’s experience and learning curve than is technical success, as it considers the often-intensive patient and device selection phase of endovascular procedures along with the actual technical performance of the procedure. Certainly several of the factors necessary for initial clinical success (the need for conversion to open repair, and type I endoleak) have been observed less frequently with increased experience.27,33 The period of preoperative planning often plays more of a role in achieving optimal results than does the actual performance of the endovascular repair. Following the initial perioperative period, reasons other than surgeon experience that are more specific to patient and graft dynamics may result in failure of the endovascular approach. Up to this point, few studies have investigated the role of surgical volumes with outcomes in vascular surgery.23,34,35 The CUSUM failure method is uniquely suited to analyze this relationship with respect to an individual’s experience. This study reviewed the learning curve of an endovascular team comprising a vascular surgeon and interventional radiologist. The 88.5% initial clinical success rate achieved by this team over a 4-year period compares favorably with published data.24-33,36-42 A sequential CUSUM plot of all endovascular repairs performed by this team (Fig 1) indicates that 60 endovascular aneurysm repairs are required to achieve a target initial clinical success rate of 90%. This case volume is irrespective of device design or manufacturer. Although similar to the results obtained by the institutional learning curve analysis performed at the Arizona Heart Institute,9 this study’s results are more applicable to what an individual may expect with respect to his or her learning experience.
JOURNAL OF VASCULAR SURGERY January 2004
Previous studies have failed to reveal a role for different device manufacturers in the evolution of results over time with endovascular aneurysm repair.9,10 However, as specific device manufacturers differ with respect to preoperative case planning, device design, measurements, and deployment, it is not unreasonable to expect a manufacturerspecific learning curve.26,32,39,42 As several procedural steps are common to all endovascular repairs, we recognize that the learning curve with one device influences results with other devices, with there being significant interaction in the acquisition of experience. Despite this interaction, this study attempted to compare the learning curves with respect to several different endograft manufacturers. A sequential CUSUM graph of aneurysm repairs performed with different device manufacturers (Fig 2) indicated that 20 repairs are required for a team, or individual, to obtain optimal results, defined in this study as an initial clinical success rate of 90%. This did not differ between the two devices that were predominantly used at our institution during this time period. Although we have not had extensive experience with other manufacturers, we see no reason why this manufacturer-specific learning curve should not be applicable to the use of other companies’ designs. During the relatively brief history of endovascular aneurysm repair, it has become apparent that endograft device design is a vital contributor to successful aneurysm exclusion.37 As in other institutions, several aortoaortic tube endografts were deployed at our center early in our endovascular experience. We have since abandoned the use of this design given the widespread concern regarding distal aortic attachment site complications and device migration.24 Since then, two thirds (66.7%) of the endovascular aneurysm repairs performed by this team have used bifurcated grafts, whereas AUI devices were deployed in approximately one third of cases (28%). A CUSUM plot of our experience with these different designs (Fig 3) confirms that satisfactory results, with respect to target initial clinical success rates, can be obtained with either design. However, vastly improved results compared with target rates are obtained following fewer repairs with AUI devices than with bifurcated endografts. This is in no small part the result of an increased risk of limb thrombosis with bifurcated grafts. Although this trend has not been universally observed, it is felt that the use of a bifurcated graft is one factor that may contribute to limb thrombosis, along with other factors such as the use of an unsupported limb, small limb diameter, and extension to the external iliac artery.25,40,41 In comparison, very few instances of endograft limb or femoral crossover graft thrombosis have been reported following the use of an AUI device. In this instance, published primary and secondary patency rates approach 98% and 100%, respectively.43 For these reasons, we have developed a distinct preference for the AUI design in our most recent endovascular experience and feel that it has wider applicability than the bifurcated design in patients with extensive aortoiliac aneurysmal disease. A limitation of this analysis is that, although it is useful in assessing an individual’s learning curve, it does not allow
JOURNAL OF VASCULAR SURGERY Volume 39, Number 1
comment on the time interval between cases. It would be logical to expect improved outcomes with shorter time intervals between interventions. Other forms of analyses, such as a first-order differential equation, have been used to evaluate this relationship.9 In addition, the length of the learning curve is inversely proportional to the predetermined target or acceptable failure rate. Shorter learning curves can be achieved by setting the acceptable failure rate at a higher level. Although this rate is somewhat arbitrarily set, it should be derived from previously published data, as in this study. Learning-curve analyses such as these have training and credentialing implications. Although training guidelines have been published,2,3,44 few have determined sufficient case volumes of endovascular aneurysm repair to obtain optimal results. This study reinforces the importance of the proctoring system during the preoperative planning phase as well as the technical application of this technology. Our group received industry-provided technical training followed by supervision by an experienced endovascular surgeon during the preoperative planning phase and intraoperative phase for the first half-dozen cases. Following this period, the various manufacturers provided on-site technical support and advice during the procedures, for varying durations. Some investigators have advocated the use of simulators in endovascular training.45 Although this may improve one’s technical endovascular skills, simulators alone are not sufficient to obtain optimal clinical results. These results can only be achieved with further experience in the preoperative planning and measurement phase of endovascular aneurysm repair, along with superior technical skills. CONCLUSIONS This study illustrates the value of the CUSUM technique in evaluating an individual’s learning curve. Although acceptable results were achieved throughout the study period with respect to endovascular aneurysm repair, this study’s results indicate that 60 elective endovascular repairs, or 20 with an individual device manufacturer, are required to obtain optimal results with respect to initial clinical success. In addition, these improved results can be achieved with fewer repairs with AUI devices than with bifurcated endografts. We thank Dr Richard Novick for his assistance and advice with respect to the CUSUM calculations and statistical analysis. REFERENCES 1. Parodi JC, Palmaz JC, Barone HD. Transfemoral intraluminal graft implantation for abdominal aortic aneurysms. Ann Vasc Surg 1991;5: 491-9. 2. Van Breda A, Becker GJ. Endovascular credentialing. J Vas Inverv Radiol 1995;5:90-2. 3. White RA, Hodgson KJ, Ahn SS, Hobson RW, Veith FJ. Endovascular interventions training and credentialing for vascular surgeons. J Vasc Surg 1999;29:177-86. 4. Chaikof EL, Blankensteijn JD, Harris PL, White GH, Zarins CK, Bernhard VM, et al, for the Ad Hoc Committee for Standardized
Forbes et al 107
5. 6. 7.
8.
9.
10.
11.
12.
13. 14. 15. 16. 17. 18.
19.
20.
21.
22.
23. 24.
25.
26.
Reporting Practices in Vascular Surgery of The Society of Vascular Surgery/American Association for Vascular Surgery. Reporting standards for endovascular aortic aneurysm repair. J Vasc Surg 2002;35: 1048-60. Williams SM, Parry BR, Schlup MMT. Quality control: an application of the CUSUM. BMJ 1992;304:1359-61. Novick RJ, Stitt LW. The learning curve of an academic cardiac surgeon: use of the CUSUM method. J Card Surg 1999;14:312-20. Novick RJ, Fox SA, Stitt LW, Swinamer SA, Lehnhardt KR, Rayman R, et al. Cumulative sum failure analysis of a policy change from on-pump to off-pump coronary artery bypass grafting. Ann Thorac Surg 2001; 72:S1016-21. Lee WA, Wolf YG, Hill BB, Cipriano P, Fogarty TJ, Zarins CK. The first 150 endovascular AAA repairs at a single institution: how steep is the learning curve? J Endovasc Ther 2002;9:269-76. Lobato AC, Rodriguez-Lopez J, Diethrich EB. Learning curve for endovascular abdominal aortic aneurysm repair: evaluation of a 277patient single-centre experience. J Endovasc Ther 2002;9:262-68. Laheij RJF, van Marrewijk CJ, Buth J, Harris PL. The influence of team experience on outcomes of endovascular stenting of abdominal aortic aneurysms. Eur J Vasc Endovasc Surg 2002;24:128-33. Zarins CK, Shaver DM, Arko FR, Schubart PJ, Lengle SJ, Dixon SM. Introduction of endovascular aneurysm repair into community practice: initial results with a new Food and Drug Administration-approved device. J Vasc Surg 2002;36:226-33. Ahn SS, Rutherford RB, Johnston KW, May J, Veith FJ, Baker JD, et al. for the Ad Hoc Committee for Standardized Reporting Practices in Vascular Surgery of The Society for Vascular Surgery/International Society for Cardiovascular Surgery. Reporting standards for infrarenal endovascular abdominal aortic aneurysm repair. J Vasc Surg 1997;25: 405-10. Hurst HE. Proc Am Soc Civil Eng 1950;76:11. Healy MJR. The disciplining of medical data. Br Med Bull 1968;24: 210-14. Chaput de Saintonge DM, Vere DW. Why don’t doctors use CUSUMs? Lancet 1974;1:120-1. Wohl H. The CUSUM plot: its utility in the analysis of clinical data. N Eng J Med 1977;296:1044-5. Parry BR, Williams SM. Competency and the colonoscopist: a learning curve. Aust N Z J Surg 1991;61:419-22. McCarter FD, Luchette FA, Molloy M, Hurst JM, Davis K, Johannigman JA, et al. Institutional and individual learning curves for focused abdominal ultrasound for trauma. Ann Surg 2000;231:689-700. Lawrance RA, Dorsch MF, Sapsford RJ, Mackintosh AF, Greenwood DC, Jackson BM, et al. Use of cumulative mortality data in patients with acute myocardial infarction for early detection of variation in clinical practice: observational study. BMJ 2001;323:324-7. Van Rij AM, McDonald JR, Pettigrew RA, Putterill MJ, Reddy CK, Wright JJ. Cusum as an aid to early assessment of the surgical trainee. Br J Surg 1995;82:1500-3. de Leval MR, Francois K, Bull C, Brawn W, Spiegelhalter D. Analysis of a cluster of surgical failures. Application to a series of neonatal arterial switch operations. J Thorac Cardiovasc Surg 1994;107:914-23. Novick RJ, Fox SA, Stitt LW, Kiaii BB, Swinamer SA, Rayman R, et al. Assessing the learning curve in off-pump coronary artery surgery via CUSUM failure analysis. Ann Thorac Surg 2002;73:S358-62. Forbes TL, DeRose G, Harris KA. A CUSUM analysis of ruptured abdominal aortic aneurysm repair. Ann Vasc Surg 2002;16:527-33. Faries PL, Briggs VL, Rhee JY, Burks JA, Gravereaux EC, Carroccio A, et al. Failure of endovascular aortoaortic tube grafts; a plea for preferential use of bifurcated grafts. J Vasc Surg 2002;35:868-73. Moore WS, Brewster DC, Bernhard VM, for the EVT/Guidant Investigators. Aorto-uni-iliac endograft for complex aortoiliac aneurysms compared with tube/bifurcation endografts: results of the EVT/ Guidant trials. J Vasc Surg 2001;33(2 Suppl):S11-20. Zarins CK, White RA, Schwarten D, Kinney E, Diethrich EB, Hodgson KJ, et al, for the investigators of the Medtronic AneuRx Multicenter Clinical Trial. AneuRx stent graft versus open surgical repair of abdominal aortic aneurysms: multicenter prospective clinical trial. J Vasc Surg 1999;29:292-308.
JOURNAL OF VASCULAR SURGERY January 2004
108 Forbes et al
27. Buth J, Laheij RJF, on behalf of the EUROSTAR Collaborators. Early complications and endoleaks after endovascular abdominal aortic aneurysm repair: report of a multicenter study. J Vasc Surg 2000;31:134-46. 28. Chuter TAM, Reilly LM, Faruqi RM, Kerlan RB, Sawhney R, Canto CJ, et al. Endovascular aneurysm repair in high-risk patients. J Vasc Surg 2000;31:122-33. 29. Liewald F, Scharrer-Pamler R, Go¨ rich J, Kapfer X, Seifarth H, Halter G, et al. Intraoperative, perioperative and late complications with endovascular therapy of aortic aneurysm. Eur J Vasc Endovasc Surg 2001;22: 251-6. 30. Walschot LHB, Laheij RJF, Verbeek ALM. Outcome after endovascular abdominal aortic aneurysm repair: a meta-analysis. J Endovasc Ther 2002;9:82-9. 31. Chaikof EL, Lin PH, Brinkman WT, Dodson TF, Weiss VJ, Lumsden AB, et al. Endovascular repair of abdominal aortic aneurysms: risk stratified outcomes. Ann Surg 2002;235:833-41. 32. Criado FJ, Wilson EP, Fairman RM, Abul-Khoudoud O, Wellons E. Update on the Talent aortic stent-graft: a preliminary report from United States phase I and II trials. J Vasc Surg 2001;33:S146-9. 33. Bo¨ ckler D, Probst T, Weber H, Raithel D. Surgical conversion after endovascular grafting for abdominal aortic aneurysms. J Endovasc Ther 2002;9:111-18. 34. Huber TS, Seeger JM. Dartmouth Atlas of Vascular Health Care review: impact of hospital volume, surgeon volume, and training on outcome. J Vasc Surg 2001;34:751-6. 35. Shackley P, Slack R, Booth A, Michaels J. Is there a positive volumeoutcome relationship in peripheral vascular surgery? Results of a systematic review. Eur J Vasc Endovasc Surg 2000;20:326-35. 36. Cuypers PWM, Laheij RJF, Buth J, & on behalf of the EUROSTAR Collaborators. Which factors increase the risk of conversion to open surgery following endovascular abdominal aortic aneurysm repair? Eur J Vasc Endovasc Surg 2000;20:183-9. 37. Resch T, Malina M, Lindblad B, Ivancev K. The impact of stent-graft
38.
39.
40.
41.
42.
43.
44. 45.
development on outcome of AAA repair—a 7-year experience. Eur J Vasc Endovasc Surg 2001;22:57-61. Ohki T, Veith FJ, Shaw P, Lipsitz E, Suggs WD, Wain RA, et al. Increasing incidence of midterm and long-term complications after endovascular graft repair of abdominal aortic aneurysms: a note of caution based on a 9-year experience. Ann Surg 2001;234:323-35. Abraham CZ, Chuter TAM, Reilly LM, Okuhn SP, Pethan LK, Kerlan RB, et al. Abdominal aortic aneurysm repair with the Zenith stent graft: short to midterm results. J Vasc Surg 2002;36:217-25. Carpenter JP, Neschis DG, Fairman RM, Barker CF, Golden MA, Velazquez OC, et al. Failure of endovascular abdominal aortic aneurysm graft limbs. J Vasc Surg 2001;33:296-303. Carroccio A, Faries PL, Morrissey NJ, Teodorescu V, Burks JA, Gravereaux EC, et al. Predicting iliac limb occlusions after bifurcated aortic stent grafting: anatomic and device-related causes. J Vasc Surg 2002; 36:679-84. Greenberg RK, Lawrence-Brown M, Bhandari G, Hartley D, Stelter W, Umscheid T, et al. An update of the Zenith endovascular graft for abdominal aortic aneurysms: initial implantation and mid-term follow-up data. J Vasc Surg 2001;33:S157-64. Rehring TF, Brewster DC, Cambria RP, Kaufman JA, Geller SC, Fan CM, et al. Utility and reliability of endovascular aortouniiliac with femorofemoral crossover graft for aortoiliac aneurysmal disease. J Vasc Surg 2000;31:1135-41. Kwolek CJ. Endovascular qualifications for practicing surgeons. Sem Vasc Surg 2002;15:174-7. Chong CK, Brennan J, How TV, Edwards R, Gilling-Smith GL, Harris PL. A prototype simulator for endovascular repair of abdominal aortic aneurysms. Eur J Vasc Endovasc Surg 1997;13:330-3.
Submitted May 8, 2003; accepted Jun 5, 2003.
Additional material for this article may be found online at www.mosby.com/jvs.
The JVS Ombudsman The ombudsman’s role is to act as an advocate for authors and represent their position to the editorial staff in relation to the process of manuscript submission, review, and publication. The ombudsman is not responsible for evaluating the content of a manuscript or determining whether the editors made the correct decision with regard to acceptance or rejection of the paper. If an author or other person has an unresolved complaint or question about the editorial process of the Journal, he or she should contact Dr James S. T. Yao (Northwestern University Medical School, Department of Surgery, 201 E. Huron Street, Suite 10-105, Chicago, IL 60611), who will review the matter.