- Email: [email protected]
An evidence-based checklist to assess neuropsychological outcomes of epilepsy surgery: How good is the evidence?
Epilepsy & Behavior 29 (2013) 443–448
Contents lists available at ScienceDirect
Epilepsy & Behavior journal homepage: www.elsevier.com/locate/yebeh
An evidence-based checklist to assess neuropsychological outcomes of epilepsy surgery: How good is the evidence? M. Hrabok a,b,c,d,⁎, J. Dykeman e, E.M.S. Sherman a,b,e,f, S. Wiebe a,b,g,h,i a
Alberta Health Services, Canada The Alberta Children's Hospital Research Institute, Canada Copeman Healthcare Centre, Edmonton, Canada d University of Alberta Hospital, Edmonton, Canada e Faculty of Medicine, University of Calgary, Canada f Copeman Healthcare Centre, Calgary, Canada g Department of Clinical Neurosciences, Faculty of Medicine, University of Calgary, Canada h Department of Community Health Sciences, Faculty of Medicine, University of Calgary, Canada i Faculty-wide Clinical Research Unit, Faculty of Medicine, University of Calgary, Canada b c
a r t i c l e
i n f o
Article history: Received 7 August 2013 Accepted 30 August 2013 Available online 11 October 2013 Keywords: Epilepsy Neuropsychology Evidence-based Surgery Outcome Methodology
a b s t r a c t Purpose: We aimed to assess the quality of evidence on neuropsychological outcomes after epilepsy surgery (ES). Accordingly, we created an evidence-based neuropsychology (EBNP) checklist to assess neuropsychological outcomes and applied this tool to studies from a systematic review. Methods: The EBNP checklist was created using clinical expert input, scale development methodology for item generation and reduction and inter-rater reliability, and critical appraisal guidelines for studies about treatment. The checklist was applied to articles obtained through a systematic review of resective ES neuropsychological outcomes. The proportion of studies fulfilling the quality criteria and the total quality score were used to assess the quality of the evidence. Results: An initial 45-item checklist was applied to 147 articles, with excellent inter-rater agreement (kappa = 0.80). The mean quality score was 23 (SD: 4, range: 12–33). There was substantial variability in the percentage of studies meeting the criteria for specific items (0–99%). The median proportion of papers fulfilling various quality criteria was 1.4% for items related to group comparisons, 37% for clinical applicability, 67% for patient description, 78% for outcome assessment, and 91% for interventions. Higher quality correlated with longitudinal design, reporting presurgical IQ, seizure frequency and antiepileptic drugs, and using validated measures of change in individual patients. The final EBNP checklist consisted of 19 items. Discussion: The EBNP checklist reliably identified quality strengths and threats to validity of neuropsychological outcome studies in ES. Studies would be most improved by the inclusion of random allocation to interventions or at minimum blinded outcome assessment, empirically based measures of reliable change and completeness of reporting of follow-up. © 2013 Published by Elsevier Inc.
1. Introduction Assessment of neuropsychological functioning is essential in epilepsy. Not only are cognitive deficits present in up to 75% of adults [1] and 25% of children [2] with new onset epilepsy, but deficits also relate to the etiology and type of epilepsy, interictal epileptiform activity, seizure frequency and severity, psychiatric comorbidities, and the effects of medical and surgical treatment [3]. The effectiveness of temporal lobe epilepsy surgery (ES) in achieving seizure remission has been
⁎ Corresponding author at: Brain Health Program, Copeman Healthcare Centre, 700-10216 124 Street, Edmonton AB T5N 4A3, Canada. E-mail addresses: [email protected] (M. Hrabok), [email protected] (J. Dykeman), [email protected] (E.M.S. Sherman), [email protected] (S. Wiebe). 1525-5050/$ – see front matter © 2013 Published by Elsevier Inc. http://dx.doi.org/10.1016/j.yebeh.2013.08.035
demonstrated in randomized controlled trials [4,5] and is supported by Grade “A” recommendations from the American Academy of Neurology [6]. However, assessment of cognitive outcomes in temporal lobe ES has not been subjected to the same rigor as seizure outcomes, despite data indicating that reliable declines in cognitive function occur in approximately 40% of patients after dominant temporal lobe resections [7]. Rating the quality of evidence has become standard practice in many clinical disciplines. Using principles of evidence-based medicine [8] (i.e., critically appraising the evidence for its validity and usefulness to assist in managing individual patients), authors of clinical practice guidelines rate the quality of individual studies to determine the strength of their statements and recommendations [9,10]. Scientific journals like Neurology require authors to explicitly grade their reports based on methodological quality (http://www.neurology.org/site/misc/auth2.xhtml).
444
M. Hrabok et al. / Epilepsy & Behavior 29 (2013) 443–448
Critical appraisal of the quality of the evidence regarding epilepsy treatment has been applied almost exclusively to assessing seizure outcomes following antiepileptic drug (AED) treatment [11–13] or ES [6]. The quality of evidence pertaining to cognitive outcomes following treatments for epilepsy has received little attention. In general, EBM principles have not been systematically or explicitly applied to research studies in clinical neuropsychology [14], despite application in other areas of clinical psychology [15]. Determining the quality of the evidence pertaining to neuropsychological outcomes after surgery is relevant to assist in lateralization and localization of the seizure focus and to help identify patients at risk for postoperative decline in cognition [16]. Postoperatively, neuropsychological evaluation is used to detect cognitive and psychological strengths and challenges, assist in treatment planning [17], and assess the impact of cognitive rehabilitation [18]. Our primary aims were to (1) assess the quality of the evidence regarding neuropsychological function after ES and (2) develop and apply a checklist to assess the quality and clinical usefulness of primary studies describing neuropsychological outcomes after ES. 2. Methods Development of the evidence-based neuropsychology (EBNP) checklist included a conceptual framework and item generation, tryouts and refinement to assess face validity and consistency, application of the items to relevant articles to determine frequency of endorsement, and item reduction to produce the final checklist. 2.1. Development sample — systematic review The articles used for checklist development and testing were taken from a previously published systematic review of neuropsychological outcomes after resective ES. Details of the exhaustive literature search are described previously [7]. PubMed, EmBase, PsycInfo, and the Cochrane databases were searched, and we included articles up to 2010 describing neuropsychological outcomes of ES. The references of a sample of studies were reviewed for additional relevant studies. Two reviewers independently applied study inclusion criteria and abstracted data for analyses. Neuropsychological outcomes were defined as IQ, memory, language, executive functioning, attention, and subjective cognitive changes. We restricted our sample to studies reporting original research that reported on a cognitive outcome in surgical patients. Only articles with original data were included. 2.2. Conceptual design and item generation The checklist was derived by incorporating elements from published, general critical appraisal checklists based on EBM principles [8,19,20] as well as methodological and clinical aspects uniquely important to ES and neuropsychology [14,16,17,21]. Each coauthor had the specific expertise necessary for conceptual design and item generation of the checklist [neuropsychology (EMS, MH), epileptology (SW), and clinical epidemiology (SW, JD)]. Our goal was not to create a tool to derive a quantitative quality score or a threshold for high- and low-quality studies. The goal was to generate a framework for critically appraising ES articles that described neuropsychological outcomes with regard to (a) scientific validity and (b) clinical usefulness in managing individual patients. 2.3. Tryouts, refinement, and application of the EBNP checklist Content validity of a comprehensive checklist was evaluated during two iterations of item tryout in a random sample of 10% of the articles. Two raters (MH/JD) independently applied the checklist items to the articles. Following each tryout, the checklist was reevaluated for item
clarity and consistency and revised by all authors. Two raters (MH/JD) independently applied the final checklist to all the articles included in the study. Discordant ratings were discussed, and consensus was reached for each article. The kappa statistic was used to assess interrater agreement. 2.4. Item reduction and final checklist To enhance practical application, we created an abbreviated checklist. Clinical and statistical approaches to item reduction were employed to derive the final EBNP checklist [22]. The clinical approach involved identifying themes across questions that related to each other or that provided information relevant to applicability of the study results to patient care. These themes included patient characteristics, comparison groups, randomization, sample descriptors, study design/outcomes, and clinical applicability. The statistical approach involved recursive partitioning and split sample validation to identify items that best predicted other items within clinically identified groups of questions. Analyses were completed using the rpart package in R version 2.14. The final checklist items were grouped into the four categories of the PICO system, a common heuristic in EBM which stands for (P) patients, (I) interventions, (C) comparison, and (O) outcomes. The PICO framework, widely used to ask focused research questions [23] and to identify gaps in clinical research [24], allows for a clinically relevant structure to enhance the checklist's ease of use. 2.5. Assessment of quality of studies We assessed areas of methodological strength and weakness in the literature by calculating the proportion of studies fulfilling each criterion in the checklist. To examine factors associated with methodological quality, a total quality score was obtained by unweighted addition of the number of checklist items fulfilled for each article. A higher score reflected higher quality. Linear regression analysis assessed the association between year of publication and journal and total quality score. We used the median score as a threshold for classifying studies as having higher or lower quality. Recursive partitioning was used to identify items associated with studies that had quality scores above or below the median. 3. Results Of 187 articles identified in the systematic review, 147 articles met the eligibility criteria and were used for checklist development. Of the excluded articles, 30 were not original research, six did not include surgical patients, and five did not report on cognitive outcomes. 3.1. Checklist development After two iterations of tryouts and refinement, a 45-item checklist (44 items with yes/no answers and 1 item with a numerical response for sample size) was applied to each of the 147 articles. Consensus was reached in 6615 ratings, with excellent inter-rater agreement (kappa = 0.80). The percentage of studies possessing each characteristic is shown in Table 1. 3.2. Quality of the evidence The total quality scores were normally distributed, with a mean of 23.2 (SD: 4.1), a median of 24 (interquartile range: 21 to 26), and a range of 12 to 33 (possible minimum: 0, maximum: 44). The journal of publication (n = 41) was not related to the total quality score (r = −0.03, p = 0.68). Although the date of publication was associated with total quality score, with increasing quality over time (r = 0.18, p = 0.026) (Fig. 1), the association was not significant after removing the two studies published before 1988. Longitudinal studies and those
M. Hrabok et al. / Epilepsy & Behavior 29 (2013) 443–448
90 95 81 79 67 17 27 27 73 16
Interventions (I) Total N having surgery Resection side Resection site Resection type Extent of resection Postsurgical AEDsb Patients treated equally
22 99 99 97 57 14 91
Comparisons (C) Treatment randomization Randomization list concealment Surgical comparison Nonsurgical comparison Sample size or power calculation a priori
1.4 0 73 23 0.68
Outcomes (O) Blinding of clinicians/assessors Independent blind outcome assessment Postsurgical seizure frequency N completing postsurgical neuropsychology assessment Reasons for loss to follow-up N70% pre/post completion Postsurgical deficits Pathology Pre- and postsurgical measurement Measure of change Standardized and valid neuropsychology instrument Quantitative measure of change N6 months follow-up
1.4 0 82 95 31 78 25 76 91 86 92 64 83
Clinical applicability Standard norm-referenced pre-/postsurgical scores Information relevant to counseling patients Validated measure of change Results were easily applicable at the individual level
48 73 26 23
Sample size
Mean (SD) 69 (56)
a b
IQ = intelligence quotient. AEDs = antiepileptic drugs.
reporting presurgical IQ and number of AEDs had a higher quality score by a mean of 10.3 items compared to studies that did not report these elements. Studies judged by the rater as being relevant for counseling patients had higher quality scores by a mean of 5.4 items. The most important factor associated with the study quality being above the median was whether presurgical IQ was reported. Sixty percent of studies reporting this were above the median compared to 7.1% of studies not reporting it. Additionally, if presurgical AEDs were reported, 100% of the studies were above the median. However, if presurgical AEDs, a quantitative measure of change, and the extent of resection were not reported, none of the studies was above the median. The complete
30 25 20
r=0.18, p=0.026
15
99 90 54 3.4 8.2
Linear Regression
10
% yes Patients (P) Sociodemographic Age Sex Education Ethnicity Employment Clinical Age at onset/duration N completing presurgical neuropsychology assessment Presurgical IQa Handedness/language dominance Baseline differences assessed Psychiatric comorbidity Consecutive patients Presurgical seizure frequency Eligibility criteria specified Presurgical AEDsb
Association Between Publication Year and Total Quality Score Total Quality Score (range: 11-30)
Table 1 Forty-five items applied to the studies and were used in item reduction. “% yes” represents the proportion of studies possessing each attribute.
445
1980
1990
2000
2010
Publication Year Fig. 1. Scatterplot of total quality score (dependent variable) and the publication year (independent variable). Fitted line is based on linear regression analysis, and the shaded bands represent 95% CI.
hierarchical relationship of the items related to the probability of a study quality being above the median is displayed in Fig. 2. 3.2.1. Patient characteristics A large number of studies provided adequate descriptions for age (99%), sex (90%), handedness or language dominance (79%), and presurgical IQ (81%). Over half of studies reported on education (54%) and extent of surgery (57%). Reporting on ethnicity (3%) and employment (8%) was rare. In terms of neurological variables, reporting on age at onset and duration of epilepsy (90%) and postsurgical seizure frequency (82%) was high, as was reporting on pathology (76%) and side (99%), site (99%), and type of surgery (97%). Few studies reported on pre- and postsurgical AED use (16% and 14%, respectively), presurgical seizure frequency (27%), psychiatric comorbidities (17%), or postsurgical neurological deficits (25%). 3.2.2. Allocation to treatment groups Random assignment to treatment groups was rare. Four studies (2.8%) used randomization, and only two of these (1.4%) included blinding. No studies described concealed randomization. 3.2.3. Control groups and baseline differences Four items related to comparisons and baseline differences. Only 23% of studies had a nonsurgical control group, and 73% had a surgical comparison group. Important baseline differences were assessed in two-thirds of the studies, and almost all studies described all patients as being treated equally throughout the study apart from the intervention of interest (91%) (Table 1). 3.2.4. Sample size and patient representativeness The sample size in the studies ranged from 6 to 290 (median: 53), and 25% of studies had b30 patients. The sample size or power needed to detect a predetermined effect size or to perform the required statistical comparisons was considered a priori only in one study (0.68%). Few studies reported including consecutive patients (27%), the number of patients who had and did not have surgery (22%), and the reasons for loss of follow-up (31%). On the other hand, most studies reported the number who completed presurgical (95%) and postsurgical (95%) neuropsychological assessment and whether at least 70% of patients achieved the intended follow-up (78%). 3.2.5. Outcome assessment Most studies described performing pre- and postsurgical assessment (91%), performing a follow-up of ≥6 months (83%), using standardized and validated neuropsychological instruments (92%), and using a measure of change (86%). However, change was assessed quantitatively less frequently (64%), a norm-referenced metric was used in less than half
446
M. Hrabok et al. / Epilepsy & Behavior 29 (2013) 443–448
Fig. 2. Hierarchical tree of the factors associated with the probability of studies having a total quality score above the median (good quality score). Percentages represent the proportion of the studies that are above the median and possess the characteristic pertaining to each branch. *Example: within the group of studies that described presurgical IQ (which is the strongest predictor of a good quality score), the probability of having a good quality score was zero if the study did not report presurgical AEDs, a quantitative measure of change, and the extent of surgical resection.
the studies (48%), and only 26% used a validated measure such as a reliable change index or a regression-based approach to assess the magnitude of change in individual patients. Only two studies (1.4%) used blinded assessment of neuropsychological outcomes. 3.2.6. Clinical applicability Two items captured the raters' overall impressions of the study's clinical applicability. The majority of the studies provided information relevant to counseling patients about neuropsychological outcomes of ES (73%), but few provided results or analyses that were easily applicable at the individual patient level (23%). 3.3. Item reduction and final checklist The group of items pertaining to patient characteristics represented the largest number of items and was further divided into subgroups characterized by sociodemographic variables and neurological and neurosurgical characteristics. The variability of endorsement within each item subgroup prohibited statistical assessment. Therefore, the items were retained as three composite subgroups rather than as individual items. Despite their rare endorsement, randomization and concealment are crucially important in minimizing selection bias. Therefore, both items were retained in the final checklist. Baseline assessment and equal treatment of groups were retained because of the importance of minimizing confounders between groups. The items relating to a comparison group were retained because of the importance of controls in obtaining valid net effects of interventions. Describing the reasons for loss to follow-up was retained because during training and testing validation, it was predictive of the total number having surgery and of preand postsurgical neuropsychological assessment. Description of patient accrual, whether the inclusion/exclusion criteria were specified and whether a sample size or power calculation was considered a priori, were retained because of their importance in characterizing the patient population and interpreting findings. Presenting pre- and postsurgical scores using a norm-referenced metric and quantitative change using a validated measure predicted the presence of the remaining items during training and testing. Therefore, these three items were retained in the final checklist. The two items related to the blinded assessment of outcomes were retained despite their rare use because they are crucial in minimizing outcome
assessment bias. The two items assessing applicability were retained in the final checklist because they reflected a primary aim of the checklist and constitute a core tenet of EBM (i.e., evaluation of clinical usefulness in managing individual patients). The final checklist consisted of 19 items grouped according to the PICO framework, with an additional category of clinical applicability (Table 2). The checklist can be accessed in an online interactive format at http://clinicalresearchunit.org/ebnp-checklist/. 4. Discussion Our first aim was to systematically appraise the quality and applicability of the evidence dealing with cognitive outcomes after ES. Most studies reported the number of patients who completed both presurgical and postsurgical assessment, but a substantial number failed to report the number of patients lost at each stage of follow-up and the reasons for the loss. Additionally, failure to include pertinent patient descriptive information (e.g., use of AEDs, ethnicity, employment, psychological comorbidities, and presurgical seizure frequency) suggests that the majority of papers do not provide enough information for clinicians to fully understand to whom the results can be applied. Recursive partitioning identified reporting presurgical IQ as the strongest predictor of study quality score above the median (Fig. 2). Presurgical IQ probably identifies studies that have a longitudinal design (as opposed to a cross-sectional design). In turn, longitudinal studies are more likely to satisfy a number of other quality criteria, including reporting of pre- and postsurgery variables and measures of change. Conceivably, reporting presurgical IQ might be a general indicator of researchers who pay closer attention to methodological quality. The final checklist (Table 2) contains questions that are composites of several items. Importantly, the average percentage of studies endorsing the component items encompasses items with high and low levels of endorsement. In assessing the quality of individual studies, it is particularly important that readers attend to component items that are less likely to be reported. For example, in question 2 (“were patients described clinically?”), the component items pertaining to reporting of presurgical seizure frequency, psychiatric comorbidity, and use of AEDs are reported in 16% to 27% of the studies, whereas those pertaining to language dominance, presurgical IQ, and age at onset are reported in 79% to 90% of the studies.
M. Hrabok et al. / Epilepsy & Behavior 29 (2013) 443–448
447
Table 2 Final 19-item EBNP checklist following item reduction, organized into PICO subgroups and clinical applicability. Average percentage of studies fulfilling the criteria Patients (P) 1. Were patients described sociodemographically (i.e., age, sex, education, ethnicity, employment)?a 2. Were patients described clinically (i.e., age at onset/duration of epilepsy, seizure frequency, AED use, handedness/language dominance, presurgical IQ, psychiatric comorbidity)?a 3. Were the inclusion/exclusion criteria specified? 4. Was patient selection specified (e.g., consecutive patients included)?
51b 52b 73 27
Interventions (I) 5. Aside from the surgical intervention, were patients in all groups treated equally? 6. Were patients described neurosurgically (i.e., side, site, type, and extent of resection; postsurgical neurological deficits; pathology)?a
91 68b
Comparisons (C) 7. Was there random assignment of treatment? 8. Was the randomization list concealed? 9. Was there a surgical and/or nonsurgical comparison group? 10. Were groups assessed for important baseline differences? 11. Was a sample size calculation or power calculation completed a priori?
1 0 48b 67 1
Outcomes (O) 12. Were clinicians and assessors blinded to groups under comparison? 13. Was there a blinded independent outcome assessor? 14. Were reasons for loss to follow-up described?a 15. Was a quantitative measure of change reported?a
3 0 59b 78b
Clinical applicability 16. Were pre- and postsurgical neuropsychological scores presented using a norm-referenced metric?a 17. Was the measure of reliable change empirically validated (e.g., reliable change index, standardized regression-based)?a 18. Was information relevant to counseling patients about neuropsychological outcomes reported in the study? 19. Were the results easily applied to individual patients?
48 59b 74 23
All items were retained based on clinical importance. a Item retention supported also by statistical methods. b Questions that are composites of several items from the original scale.
Our second aim was to use EBM and clinical principles to assemble a checklist that users of the literature and those creating systematic reviews may apply to assess and weight the quality of the evidence dealing with neuropsychological outcomes after ES. We acknowledge the accepted wisdom that concealed randomization, blinding of outcome assessment, and completeness of follow-up are the main methods to minimize bias [8] and that adequate description of patients, interventions, and results is the best way to assess whether the results are applicable to individual patients [8]. The final checklist encompasses all of these elements in addition to others that are of significant methodological and/or clinical importance. Recently, the GRADE system has been increasingly adopted as a framework for critical appraisal of the literature [25]. Because our aim was to provide users of the literature with a checklist to assess quality, rather than to generate strength of recommendations, our approach was similar to the JAMA Users' guide relating to therapy [19,20]. With some modification, this checklist could be applied to a range of clinical interventions in clinical populations where neuropsychological assessment plays a prominent role. Most studies used standardized, norm-referenced measures. However, few included measures of change that allow categorizing individual patients as changed or unchanged based on empirically based measures of reliable change. Randomization and blinding were rarely incorporated in study design. This finding is consistent with the dearth of randomized trials in ES [4,5] and the even rarer occurrence of randomized surgical trials with adequate assessment of cognitive outcomes. Overall, group characteristics were carefully attended to, including specification of inclusion/exclusion criteria, presurgical and postsurgical measurement for the same group of patients, consistent measures and follow-up times, and in many cases, assessment of preexisting group differences. Two themes emerged through the process of clinical and statistical reduction of items for the final checklist. These included the related concepts of pre- and postoperative assessment (i.e., longitudinal repeated measures) and the use of a validated empirical measure of
reliable change (i.e., reliable change index or standard regression-based norms). The ultimate goal of counseling patients regarding their potential outcomes is thwarted by providing solely postsurgical information at one time point or information in the form of group means. Similarly, with so many factors influencing patient outcomes, individualized and accurate measures of change are essential for research to be directly applicable to patient care. We show that EBM principles and individualized considerations relevant to neuropsychology can be validly and reliably applied to outcome research in clinical neuropsychology. We also identify important gaps and opportunities for improvement in cognitive outcome research in ES (Table 3). Many of the methodological weaknesses identified reflect the quality of studies of ES in general [6,26] and the fact that Table 3 Recommended areas of improvement for studies on neuropsychological outcome after ES. • Randomization or use of equivalent controls. Although randomized surgical trials are rare, the inclusion of adequate controls enables estimation of an unbiased net effect of the intervention. • Blinded or independent (masked) outcome assessment. Randomized ES trials have used blinded, independent adjudication for seizure outcomes [4,5]. This approach should be used in studies on cognitive outcomes in ES. • A priori consideration of power and sample size. It is important to specify the required sample size to detect a pre-defined effect size and to perform meaningful statistical comparisons. • Complete description of relevant patient characteristics. Socioeconomic status/ employment, pre-/postsurgical AEDs, psychological comorbidities, neurological deficits, and presurgical seizure frequency were infrequently reported. This information is routinely collected in clinical care and should be included because of its potential impact on cognitive assessment. • Greater validity and precision of measurement. Studies should include a valid and precise estimation of the magnitude of change within individual patients. Use of empirically based measures of reliable change (e.g., reliable change index or standardized regression-based estimates of change) is one of the most important means of improving the validity and applicability of studies. • Complete reporting of representativeness and follow-up. The sample size should be specified at surgery, presurgical assessment, and postsurgical assessment, including reasons for loss to follow-up.
448
M. Hrabok et al. / Epilepsy & Behavior 29 (2013) 443–448
cognitive assessment is frequently a secondary outcome in ES studies. However, our analysis reveals that even when cognitive assessment is a secondary outcome in nonrandomized studies focused mainly on seizure outcome, substantial improvements can be made in the quality of neuropsychological studies by careful attention to study design and the reporting of results. Author contributions Hrabok, M.: study concept and design, acquisition of data, and drafting of the manuscript. Dykeman, J.: study concept and design, acquisition of data, analysis and interpretation of data, and drafting of the manuscript. Sherman, E.M.S.: study concept and design and critical revision of the manuscript for important intellectual content. Wiebe, S.: study concept and design, analysis and interpretation of data, and drafting of the manuscript. Disclosures None of the authors has any conflict of interest to disclose. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines. References [1] Witt JA, Helmstaedter C. Should cognition be screened in new-onset epilepsies? A study in 247 untreated patients. J Neurol 2012;259:1727–31. [2] Berg AT, Langfitt JT, Testa FM, Levy SR, DiMario F, Westerveld M, et al. Global cognitive function in children with epilepsy: a community-based study. Epilepsia 2008;49:608–14. [3] Elger CE, Helmstaedter C, Kurthen M. Chronic epilepsy and cognition. Lancet Neurol 2004;3:663–72. [4] Engel Jr J, McDermott MP, Wiebe S, Langfitt JT, Stern JM, Dewar S, et al. Early surgical therapy for drug-resistant temporal lobe epilepsy: a randomized trial. JAMA 2012;307:922–30. [5] Wiebe S, Blume WT, Girvin JP, Eliasziw M. A randomized, controlled trial of surgery for temporal-lobe epilepsy. N Engl J Med 2001;345:311–8. [6] Engel Jr J, Wiebe S, French J, Sperling M, Williamson P, Spencer D, et al. Practice parameter: temporal lobe and localized neocortical resections for epilepsy: report of the Quality Standards Subcommittee of the American Academy of Neurology, in Association with the American Epilepsy Society and the American Association of Neurological Surgeons. Neurology 2003;60:538–47.
[7] Sherman EM, Wiebe S, Fay-McClymont TB, Tellez-Zenteno J, Metcalfe A, HernandezRonquillo L, et al. Neuropsychological outcomes after epilepsy surgery: systematic review and pooled estimates. Epilepsia 2011;52:857–69. [8] Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. Edinburgh: Churchill Livingstone; 2000. [9] Gronseth G, Woodroffe LM, Getchius TSD. Clinical practice guideline process manual. 2011 ed. St. Paul, Minnesota: American Academy of Neurology; 2011. p. 57. [10] (NICE) NIfHaCE. The guidelines manual. 2009 ed. London: National Institute for Health and Clinical Excellence; 2009. p. 147. [11] French JA, Kanner AM, Bautista J, Abou-Khalil B, Browne T, Harden CL, et al. Efficacy and tolerability of the new antiepileptic drugs I: treatment of new onset epilepsy: report of the Therapeutics and Technology Assessment Subcommittee and Quality Standards Subcommittee of the American Academy of Neurology and the American Epilepsy Society. Neurology 2004;62:1252–60. [12] French JA, Kanner AM, Bautista J, Abou-Khalil B, Browne T, Harden CL, et al. Efficacy and tolerability of the new antiepileptic drugs II: treatment of refractory epilepsy: report of the Therapeutics and Technology Assessment Subcommittee and Quality Standards Subcommittee of the American Academy of Neurology and the American Epilepsy Society. Neurology 2004;62:1261–73. [13] Glauser T, Ben-Menachem E, Bourgeois B, Cnaan A, Chadwick D, Guerreiro C, et al. ILAE treatment guidelines: evidence-based analysis of antiepileptic drug efficacy and effectiveness as initial monotherapy for epileptic seizures and syndromes. Epilepsia 2006;47:1094–120. [14] Chelune GJ. Evidence-based research and practice in clinical neuropsychology. Clin Neuropsychol 2010;24:454–67. [15] APA. Presidential Task Force on Evidence-Based Practice. Evidence-based practice in psychology. Am Psychol 2006;61:271–85. [16] Helmstaedter C. Neuropsychological aspects of epilepsy surgery. Epilepsy Behav 2004;5(Suppl. 1):S45–55. [17] Baxendale S. The impact of epilepsy surgery on cognition and behavior. Epilepsy Behav 2008;12:592–9. [18] Helmstaedter C, Loer B, Wohlfahrt R, Hammen A, Saar J, Steinhoff BJ, et al. The effects of cognitive rehabilitation on memory outcome after temporal lobe epilepsy surgery. Epilepsy Behav 2008;12:402–9. [19] Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. J Am Med Assoc 1993;270:2598–601. [20] Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. J Am Med Assoc 1994;271:59–63. [21] Strauss E, Sherman EMS, Spreen O. A compendium of neuropsychological tests. New York: Oxford University Press; 2006. [22] Marx RG, Bombardier C, Hogg-Johnson S, Wright JG. Clinimetric and psychometric strategies for development of a health measurement scale. J Clin Epidemiol 1999;52:105–11. [23] Wilton NK, Slim AM. Application of the principles of evidence-based medicine to patient care. South Med J 2012;105:136–43. [24] Robinson KA, Saldanha IJ, McKoy NA. Development of a framework to identify research gaps from systematic reviews. J Clin Epidemiol 2011;64:1325–30. [25] Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. Br Med J 2004;328:1490. [26] McIntosh AM, Wilson SJ, Berkovic SF. Seizure outcome after temporal lobectomy: current research practice and findings. Epilepsia 2001;42:1288–307.
Contents lists available at ScienceDirect
Epilepsy & Behavior journal homepage: www.elsevier.com/locate/yebeh
An evidence-based checklist to assess neuropsychological outcomes of epilepsy surgery: How good is the evidence? M. Hrabok a,b,c,d,⁎, J. Dykeman e, E.M.S. Sherman a,b,e,f, S. Wiebe a,b,g,h,i a
Alberta Health Services, Canada The Alberta Children's Hospital Research Institute, Canada Copeman Healthcare Centre, Edmonton, Canada d University of Alberta Hospital, Edmonton, Canada e Faculty of Medicine, University of Calgary, Canada f Copeman Healthcare Centre, Calgary, Canada g Department of Clinical Neurosciences, Faculty of Medicine, University of Calgary, Canada h Department of Community Health Sciences, Faculty of Medicine, University of Calgary, Canada i Faculty-wide Clinical Research Unit, Faculty of Medicine, University of Calgary, Canada b c
a r t i c l e
i n f o
Article history: Received 7 August 2013 Accepted 30 August 2013 Available online 11 October 2013 Keywords: Epilepsy Neuropsychology Evidence-based Surgery Outcome Methodology
a b s t r a c t Purpose: We aimed to assess the quality of evidence on neuropsychological outcomes after epilepsy surgery (ES). Accordingly, we created an evidence-based neuropsychology (EBNP) checklist to assess neuropsychological outcomes and applied this tool to studies from a systematic review. Methods: The EBNP checklist was created using clinical expert input, scale development methodology for item generation and reduction and inter-rater reliability, and critical appraisal guidelines for studies about treatment. The checklist was applied to articles obtained through a systematic review of resective ES neuropsychological outcomes. The proportion of studies fulfilling the quality criteria and the total quality score were used to assess the quality of the evidence. Results: An initial 45-item checklist was applied to 147 articles, with excellent inter-rater agreement (kappa = 0.80). The mean quality score was 23 (SD: 4, range: 12–33). There was substantial variability in the percentage of studies meeting the criteria for specific items (0–99%). The median proportion of papers fulfilling various quality criteria was 1.4% for items related to group comparisons, 37% for clinical applicability, 67% for patient description, 78% for outcome assessment, and 91% for interventions. Higher quality correlated with longitudinal design, reporting presurgical IQ, seizure frequency and antiepileptic drugs, and using validated measures of change in individual patients. The final EBNP checklist consisted of 19 items. Discussion: The EBNP checklist reliably identified quality strengths and threats to validity of neuropsychological outcome studies in ES. Studies would be most improved by the inclusion of random allocation to interventions or at minimum blinded outcome assessment, empirically based measures of reliable change and completeness of reporting of follow-up. © 2013 Published by Elsevier Inc.
1. Introduction Assessment of neuropsychological functioning is essential in epilepsy. Not only are cognitive deficits present in up to 75% of adults [1] and 25% of children [2] with new onset epilepsy, but deficits also relate to the etiology and type of epilepsy, interictal epileptiform activity, seizure frequency and severity, psychiatric comorbidities, and the effects of medical and surgical treatment [3]. The effectiveness of temporal lobe epilepsy surgery (ES) in achieving seizure remission has been
⁎ Corresponding author at: Brain Health Program, Copeman Healthcare Centre, 700-10216 124 Street, Edmonton AB T5N 4A3, Canada. E-mail addresses: [email protected] (M. Hrabok), [email protected] (J. Dykeman), [email protected] (E.M.S. Sherman), [email protected] (S. Wiebe). 1525-5050/$ – see front matter © 2013 Published by Elsevier Inc. http://dx.doi.org/10.1016/j.yebeh.2013.08.035
demonstrated in randomized controlled trials [4,5] and is supported by Grade “A” recommendations from the American Academy of Neurology [6]. However, assessment of cognitive outcomes in temporal lobe ES has not been subjected to the same rigor as seizure outcomes, despite data indicating that reliable declines in cognitive function occur in approximately 40% of patients after dominant temporal lobe resections [7]. Rating the quality of evidence has become standard practice in many clinical disciplines. Using principles of evidence-based medicine [8] (i.e., critically appraising the evidence for its validity and usefulness to assist in managing individual patients), authors of clinical practice guidelines rate the quality of individual studies to determine the strength of their statements and recommendations [9,10]. Scientific journals like Neurology require authors to explicitly grade their reports based on methodological quality (http://www.neurology.org/site/misc/auth2.xhtml).
444
M. Hrabok et al. / Epilepsy & Behavior 29 (2013) 443–448
Critical appraisal of the quality of the evidence regarding epilepsy treatment has been applied almost exclusively to assessing seizure outcomes following antiepileptic drug (AED) treatment [11–13] or ES [6]. The quality of evidence pertaining to cognitive outcomes following treatments for epilepsy has received little attention. In general, EBM principles have not been systematically or explicitly applied to research studies in clinical neuropsychology [14], despite application in other areas of clinical psychology [15]. Determining the quality of the evidence pertaining to neuropsychological outcomes after surgery is relevant to assist in lateralization and localization of the seizure focus and to help identify patients at risk for postoperative decline in cognition [16]. Postoperatively, neuropsychological evaluation is used to detect cognitive and psychological strengths and challenges, assist in treatment planning [17], and assess the impact of cognitive rehabilitation [18]. Our primary aims were to (1) assess the quality of the evidence regarding neuropsychological function after ES and (2) develop and apply a checklist to assess the quality and clinical usefulness of primary studies describing neuropsychological outcomes after ES. 2. Methods Development of the evidence-based neuropsychology (EBNP) checklist included a conceptual framework and item generation, tryouts and refinement to assess face validity and consistency, application of the items to relevant articles to determine frequency of endorsement, and item reduction to produce the final checklist. 2.1. Development sample — systematic review The articles used for checklist development and testing were taken from a previously published systematic review of neuropsychological outcomes after resective ES. Details of the exhaustive literature search are described previously [7]. PubMed, EmBase, PsycInfo, and the Cochrane databases were searched, and we included articles up to 2010 describing neuropsychological outcomes of ES. The references of a sample of studies were reviewed for additional relevant studies. Two reviewers independently applied study inclusion criteria and abstracted data for analyses. Neuropsychological outcomes were defined as IQ, memory, language, executive functioning, attention, and subjective cognitive changes. We restricted our sample to studies reporting original research that reported on a cognitive outcome in surgical patients. Only articles with original data were included. 2.2. Conceptual design and item generation The checklist was derived by incorporating elements from published, general critical appraisal checklists based on EBM principles [8,19,20] as well as methodological and clinical aspects uniquely important to ES and neuropsychology [14,16,17,21]. Each coauthor had the specific expertise necessary for conceptual design and item generation of the checklist [neuropsychology (EMS, MH), epileptology (SW), and clinical epidemiology (SW, JD)]. Our goal was not to create a tool to derive a quantitative quality score or a threshold for high- and low-quality studies. The goal was to generate a framework for critically appraising ES articles that described neuropsychological outcomes with regard to (a) scientific validity and (b) clinical usefulness in managing individual patients. 2.3. Tryouts, refinement, and application of the EBNP checklist Content validity of a comprehensive checklist was evaluated during two iterations of item tryout in a random sample of 10% of the articles. Two raters (MH/JD) independently applied the checklist items to the articles. Following each tryout, the checklist was reevaluated for item
clarity and consistency and revised by all authors. Two raters (MH/JD) independently applied the final checklist to all the articles included in the study. Discordant ratings were discussed, and consensus was reached for each article. The kappa statistic was used to assess interrater agreement. 2.4. Item reduction and final checklist To enhance practical application, we created an abbreviated checklist. Clinical and statistical approaches to item reduction were employed to derive the final EBNP checklist [22]. The clinical approach involved identifying themes across questions that related to each other or that provided information relevant to applicability of the study results to patient care. These themes included patient characteristics, comparison groups, randomization, sample descriptors, study design/outcomes, and clinical applicability. The statistical approach involved recursive partitioning and split sample validation to identify items that best predicted other items within clinically identified groups of questions. Analyses were completed using the rpart package in R version 2.14. The final checklist items were grouped into the four categories of the PICO system, a common heuristic in EBM which stands for (P) patients, (I) interventions, (C) comparison, and (O) outcomes. The PICO framework, widely used to ask focused research questions [23] and to identify gaps in clinical research [24], allows for a clinically relevant structure to enhance the checklist's ease of use. 2.5. Assessment of quality of studies We assessed areas of methodological strength and weakness in the literature by calculating the proportion of studies fulfilling each criterion in the checklist. To examine factors associated with methodological quality, a total quality score was obtained by unweighted addition of the number of checklist items fulfilled for each article. A higher score reflected higher quality. Linear regression analysis assessed the association between year of publication and journal and total quality score. We used the median score as a threshold for classifying studies as having higher or lower quality. Recursive partitioning was used to identify items associated with studies that had quality scores above or below the median. 3. Results Of 187 articles identified in the systematic review, 147 articles met the eligibility criteria and were used for checklist development. Of the excluded articles, 30 were not original research, six did not include surgical patients, and five did not report on cognitive outcomes. 3.1. Checklist development After two iterations of tryouts and refinement, a 45-item checklist (44 items with yes/no answers and 1 item with a numerical response for sample size) was applied to each of the 147 articles. Consensus was reached in 6615 ratings, with excellent inter-rater agreement (kappa = 0.80). The percentage of studies possessing each characteristic is shown in Table 1. 3.2. Quality of the evidence The total quality scores were normally distributed, with a mean of 23.2 (SD: 4.1), a median of 24 (interquartile range: 21 to 26), and a range of 12 to 33 (possible minimum: 0, maximum: 44). The journal of publication (n = 41) was not related to the total quality score (r = −0.03, p = 0.68). Although the date of publication was associated with total quality score, with increasing quality over time (r = 0.18, p = 0.026) (Fig. 1), the association was not significant after removing the two studies published before 1988. Longitudinal studies and those
M. Hrabok et al. / Epilepsy & Behavior 29 (2013) 443–448
90 95 81 79 67 17 27 27 73 16
Interventions (I) Total N having surgery Resection side Resection site Resection type Extent of resection Postsurgical AEDsb Patients treated equally
22 99 99 97 57 14 91
Comparisons (C) Treatment randomization Randomization list concealment Surgical comparison Nonsurgical comparison Sample size or power calculation a priori
1.4 0 73 23 0.68
Outcomes (O) Blinding of clinicians/assessors Independent blind outcome assessment Postsurgical seizure frequency N completing postsurgical neuropsychology assessment Reasons for loss to follow-up N70% pre/post completion Postsurgical deficits Pathology Pre- and postsurgical measurement Measure of change Standardized and valid neuropsychology instrument Quantitative measure of change N6 months follow-up
1.4 0 82 95 31 78 25 76 91 86 92 64 83
Clinical applicability Standard norm-referenced pre-/postsurgical scores Information relevant to counseling patients Validated measure of change Results were easily applicable at the individual level
48 73 26 23
Sample size
Mean (SD) 69 (56)
a b
IQ = intelligence quotient. AEDs = antiepileptic drugs.
reporting presurgical IQ and number of AEDs had a higher quality score by a mean of 10.3 items compared to studies that did not report these elements. Studies judged by the rater as being relevant for counseling patients had higher quality scores by a mean of 5.4 items. The most important factor associated with the study quality being above the median was whether presurgical IQ was reported. Sixty percent of studies reporting this were above the median compared to 7.1% of studies not reporting it. Additionally, if presurgical AEDs were reported, 100% of the studies were above the median. However, if presurgical AEDs, a quantitative measure of change, and the extent of resection were not reported, none of the studies was above the median. The complete
30 25 20
r=0.18, p=0.026
15
99 90 54 3.4 8.2
Linear Regression
10
% yes Patients (P) Sociodemographic Age Sex Education Ethnicity Employment Clinical Age at onset/duration N completing presurgical neuropsychology assessment Presurgical IQa Handedness/language dominance Baseline differences assessed Psychiatric comorbidity Consecutive patients Presurgical seizure frequency Eligibility criteria specified Presurgical AEDsb
Association Between Publication Year and Total Quality Score Total Quality Score (range: 11-30)
Table 1 Forty-five items applied to the studies and were used in item reduction. “% yes” represents the proportion of studies possessing each attribute.
445
1980
1990
2000
2010
Publication Year Fig. 1. Scatterplot of total quality score (dependent variable) and the publication year (independent variable). Fitted line is based on linear regression analysis, and the shaded bands represent 95% CI.
hierarchical relationship of the items related to the probability of a study quality being above the median is displayed in Fig. 2. 3.2.1. Patient characteristics A large number of studies provided adequate descriptions for age (99%), sex (90%), handedness or language dominance (79%), and presurgical IQ (81%). Over half of studies reported on education (54%) and extent of surgery (57%). Reporting on ethnicity (3%) and employment (8%) was rare. In terms of neurological variables, reporting on age at onset and duration of epilepsy (90%) and postsurgical seizure frequency (82%) was high, as was reporting on pathology (76%) and side (99%), site (99%), and type of surgery (97%). Few studies reported on pre- and postsurgical AED use (16% and 14%, respectively), presurgical seizure frequency (27%), psychiatric comorbidities (17%), or postsurgical neurological deficits (25%). 3.2.2. Allocation to treatment groups Random assignment to treatment groups was rare. Four studies (2.8%) used randomization, and only two of these (1.4%) included blinding. No studies described concealed randomization. 3.2.3. Control groups and baseline differences Four items related to comparisons and baseline differences. Only 23% of studies had a nonsurgical control group, and 73% had a surgical comparison group. Important baseline differences were assessed in two-thirds of the studies, and almost all studies described all patients as being treated equally throughout the study apart from the intervention of interest (91%) (Table 1). 3.2.4. Sample size and patient representativeness The sample size in the studies ranged from 6 to 290 (median: 53), and 25% of studies had b30 patients. The sample size or power needed to detect a predetermined effect size or to perform the required statistical comparisons was considered a priori only in one study (0.68%). Few studies reported including consecutive patients (27%), the number of patients who had and did not have surgery (22%), and the reasons for loss of follow-up (31%). On the other hand, most studies reported the number who completed presurgical (95%) and postsurgical (95%) neuropsychological assessment and whether at least 70% of patients achieved the intended follow-up (78%). 3.2.5. Outcome assessment Most studies described performing pre- and postsurgical assessment (91%), performing a follow-up of ≥6 months (83%), using standardized and validated neuropsychological instruments (92%), and using a measure of change (86%). However, change was assessed quantitatively less frequently (64%), a norm-referenced metric was used in less than half
446
M. Hrabok et al. / Epilepsy & Behavior 29 (2013) 443–448
Fig. 2. Hierarchical tree of the factors associated with the probability of studies having a total quality score above the median (good quality score). Percentages represent the proportion of the studies that are above the median and possess the characteristic pertaining to each branch. *Example: within the group of studies that described presurgical IQ (which is the strongest predictor of a good quality score), the probability of having a good quality score was zero if the study did not report presurgical AEDs, a quantitative measure of change, and the extent of surgical resection.
the studies (48%), and only 26% used a validated measure such as a reliable change index or a regression-based approach to assess the magnitude of change in individual patients. Only two studies (1.4%) used blinded assessment of neuropsychological outcomes. 3.2.6. Clinical applicability Two items captured the raters' overall impressions of the study's clinical applicability. The majority of the studies provided information relevant to counseling patients about neuropsychological outcomes of ES (73%), but few provided results or analyses that were easily applicable at the individual patient level (23%). 3.3. Item reduction and final checklist The group of items pertaining to patient characteristics represented the largest number of items and was further divided into subgroups characterized by sociodemographic variables and neurological and neurosurgical characteristics. The variability of endorsement within each item subgroup prohibited statistical assessment. Therefore, the items were retained as three composite subgroups rather than as individual items. Despite their rare endorsement, randomization and concealment are crucially important in minimizing selection bias. Therefore, both items were retained in the final checklist. Baseline assessment and equal treatment of groups were retained because of the importance of minimizing confounders between groups. The items relating to a comparison group were retained because of the importance of controls in obtaining valid net effects of interventions. Describing the reasons for loss to follow-up was retained because during training and testing validation, it was predictive of the total number having surgery and of preand postsurgical neuropsychological assessment. Description of patient accrual, whether the inclusion/exclusion criteria were specified and whether a sample size or power calculation was considered a priori, were retained because of their importance in characterizing the patient population and interpreting findings. Presenting pre- and postsurgical scores using a norm-referenced metric and quantitative change using a validated measure predicted the presence of the remaining items during training and testing. Therefore, these three items were retained in the final checklist. The two items related to the blinded assessment of outcomes were retained despite their rare use because they are crucial in minimizing outcome
assessment bias. The two items assessing applicability were retained in the final checklist because they reflected a primary aim of the checklist and constitute a core tenet of EBM (i.e., evaluation of clinical usefulness in managing individual patients). The final checklist consisted of 19 items grouped according to the PICO framework, with an additional category of clinical applicability (Table 2). The checklist can be accessed in an online interactive format at http://clinicalresearchunit.org/ebnp-checklist/. 4. Discussion Our first aim was to systematically appraise the quality and applicability of the evidence dealing with cognitive outcomes after ES. Most studies reported the number of patients who completed both presurgical and postsurgical assessment, but a substantial number failed to report the number of patients lost at each stage of follow-up and the reasons for the loss. Additionally, failure to include pertinent patient descriptive information (e.g., use of AEDs, ethnicity, employment, psychological comorbidities, and presurgical seizure frequency) suggests that the majority of papers do not provide enough information for clinicians to fully understand to whom the results can be applied. Recursive partitioning identified reporting presurgical IQ as the strongest predictor of study quality score above the median (Fig. 2). Presurgical IQ probably identifies studies that have a longitudinal design (as opposed to a cross-sectional design). In turn, longitudinal studies are more likely to satisfy a number of other quality criteria, including reporting of pre- and postsurgery variables and measures of change. Conceivably, reporting presurgical IQ might be a general indicator of researchers who pay closer attention to methodological quality. The final checklist (Table 2) contains questions that are composites of several items. Importantly, the average percentage of studies endorsing the component items encompasses items with high and low levels of endorsement. In assessing the quality of individual studies, it is particularly important that readers attend to component items that are less likely to be reported. For example, in question 2 (“were patients described clinically?”), the component items pertaining to reporting of presurgical seizure frequency, psychiatric comorbidity, and use of AEDs are reported in 16% to 27% of the studies, whereas those pertaining to language dominance, presurgical IQ, and age at onset are reported in 79% to 90% of the studies.
M. Hrabok et al. / Epilepsy & Behavior 29 (2013) 443–448
447
Table 2 Final 19-item EBNP checklist following item reduction, organized into PICO subgroups and clinical applicability. Average percentage of studies fulfilling the criteria Patients (P) 1. Were patients described sociodemographically (i.e., age, sex, education, ethnicity, employment)?a 2. Were patients described clinically (i.e., age at onset/duration of epilepsy, seizure frequency, AED use, handedness/language dominance, presurgical IQ, psychiatric comorbidity)?a 3. Were the inclusion/exclusion criteria specified? 4. Was patient selection specified (e.g., consecutive patients included)?
51b 52b 73 27
Interventions (I) 5. Aside from the surgical intervention, were patients in all groups treated equally? 6. Were patients described neurosurgically (i.e., side, site, type, and extent of resection; postsurgical neurological deficits; pathology)?a
91 68b
Comparisons (C) 7. Was there random assignment of treatment? 8. Was the randomization list concealed? 9. Was there a surgical and/or nonsurgical comparison group? 10. Were groups assessed for important baseline differences? 11. Was a sample size calculation or power calculation completed a priori?
1 0 48b 67 1
Outcomes (O) 12. Were clinicians and assessors blinded to groups under comparison? 13. Was there a blinded independent outcome assessor? 14. Were reasons for loss to follow-up described?a 15. Was a quantitative measure of change reported?a
3 0 59b 78b
Clinical applicability 16. Were pre- and postsurgical neuropsychological scores presented using a norm-referenced metric?a 17. Was the measure of reliable change empirically validated (e.g., reliable change index, standardized regression-based)?a 18. Was information relevant to counseling patients about neuropsychological outcomes reported in the study? 19. Were the results easily applied to individual patients?
48 59b 74 23
All items were retained based on clinical importance. a Item retention supported also by statistical methods. b Questions that are composites of several items from the original scale.
Our second aim was to use EBM and clinical principles to assemble a checklist that users of the literature and those creating systematic reviews may apply to assess and weight the quality of the evidence dealing with neuropsychological outcomes after ES. We acknowledge the accepted wisdom that concealed randomization, blinding of outcome assessment, and completeness of follow-up are the main methods to minimize bias [8] and that adequate description of patients, interventions, and results is the best way to assess whether the results are applicable to individual patients [8]. The final checklist encompasses all of these elements in addition to others that are of significant methodological and/or clinical importance. Recently, the GRADE system has been increasingly adopted as a framework for critical appraisal of the literature [25]. Because our aim was to provide users of the literature with a checklist to assess quality, rather than to generate strength of recommendations, our approach was similar to the JAMA Users' guide relating to therapy [19,20]. With some modification, this checklist could be applied to a range of clinical interventions in clinical populations where neuropsychological assessment plays a prominent role. Most studies used standardized, norm-referenced measures. However, few included measures of change that allow categorizing individual patients as changed or unchanged based on empirically based measures of reliable change. Randomization and blinding were rarely incorporated in study design. This finding is consistent with the dearth of randomized trials in ES [4,5] and the even rarer occurrence of randomized surgical trials with adequate assessment of cognitive outcomes. Overall, group characteristics were carefully attended to, including specification of inclusion/exclusion criteria, presurgical and postsurgical measurement for the same group of patients, consistent measures and follow-up times, and in many cases, assessment of preexisting group differences. Two themes emerged through the process of clinical and statistical reduction of items for the final checklist. These included the related concepts of pre- and postoperative assessment (i.e., longitudinal repeated measures) and the use of a validated empirical measure of
reliable change (i.e., reliable change index or standard regression-based norms). The ultimate goal of counseling patients regarding their potential outcomes is thwarted by providing solely postsurgical information at one time point or information in the form of group means. Similarly, with so many factors influencing patient outcomes, individualized and accurate measures of change are essential for research to be directly applicable to patient care. We show that EBM principles and individualized considerations relevant to neuropsychology can be validly and reliably applied to outcome research in clinical neuropsychology. We also identify important gaps and opportunities for improvement in cognitive outcome research in ES (Table 3). Many of the methodological weaknesses identified reflect the quality of studies of ES in general [6,26] and the fact that Table 3 Recommended areas of improvement for studies on neuropsychological outcome after ES. • Randomization or use of equivalent controls. Although randomized surgical trials are rare, the inclusion of adequate controls enables estimation of an unbiased net effect of the intervention. • Blinded or independent (masked) outcome assessment. Randomized ES trials have used blinded, independent adjudication for seizure outcomes [4,5]. This approach should be used in studies on cognitive outcomes in ES. • A priori consideration of power and sample size. It is important to specify the required sample size to detect a pre-defined effect size and to perform meaningful statistical comparisons. • Complete description of relevant patient characteristics. Socioeconomic status/ employment, pre-/postsurgical AEDs, psychological comorbidities, neurological deficits, and presurgical seizure frequency were infrequently reported. This information is routinely collected in clinical care and should be included because of its potential impact on cognitive assessment. • Greater validity and precision of measurement. Studies should include a valid and precise estimation of the magnitude of change within individual patients. Use of empirically based measures of reliable change (e.g., reliable change index or standardized regression-based estimates of change) is one of the most important means of improving the validity and applicability of studies. • Complete reporting of representativeness and follow-up. The sample size should be specified at surgery, presurgical assessment, and postsurgical assessment, including reasons for loss to follow-up.
448
M. Hrabok et al. / Epilepsy & Behavior 29 (2013) 443–448
cognitive assessment is frequently a secondary outcome in ES studies. However, our analysis reveals that even when cognitive assessment is a secondary outcome in nonrandomized studies focused mainly on seizure outcome, substantial improvements can be made in the quality of neuropsychological studies by careful attention to study design and the reporting of results. Author contributions Hrabok, M.: study concept and design, acquisition of data, and drafting of the manuscript. Dykeman, J.: study concept and design, acquisition of data, analysis and interpretation of data, and drafting of the manuscript. Sherman, E.M.S.: study concept and design and critical revision of the manuscript for important intellectual content. Wiebe, S.: study concept and design, analysis and interpretation of data, and drafting of the manuscript. Disclosures None of the authors has any conflict of interest to disclose. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines. References [1] Witt JA, Helmstaedter C. Should cognition be screened in new-onset epilepsies? A study in 247 untreated patients. J Neurol 2012;259:1727–31. [2] Berg AT, Langfitt JT, Testa FM, Levy SR, DiMario F, Westerveld M, et al. Global cognitive function in children with epilepsy: a community-based study. Epilepsia 2008;49:608–14. [3] Elger CE, Helmstaedter C, Kurthen M. Chronic epilepsy and cognition. Lancet Neurol 2004;3:663–72. [4] Engel Jr J, McDermott MP, Wiebe S, Langfitt JT, Stern JM, Dewar S, et al. Early surgical therapy for drug-resistant temporal lobe epilepsy: a randomized trial. JAMA 2012;307:922–30. [5] Wiebe S, Blume WT, Girvin JP, Eliasziw M. A randomized, controlled trial of surgery for temporal-lobe epilepsy. N Engl J Med 2001;345:311–8. [6] Engel Jr J, Wiebe S, French J, Sperling M, Williamson P, Spencer D, et al. Practice parameter: temporal lobe and localized neocortical resections for epilepsy: report of the Quality Standards Subcommittee of the American Academy of Neurology, in Association with the American Epilepsy Society and the American Association of Neurological Surgeons. Neurology 2003;60:538–47.
[7] Sherman EM, Wiebe S, Fay-McClymont TB, Tellez-Zenteno J, Metcalfe A, HernandezRonquillo L, et al. Neuropsychological outcomes after epilepsy surgery: systematic review and pooled estimates. Epilepsia 2011;52:857–69. [8] Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. Edinburgh: Churchill Livingstone; 2000. [9] Gronseth G, Woodroffe LM, Getchius TSD. Clinical practice guideline process manual. 2011 ed. St. Paul, Minnesota: American Academy of Neurology; 2011. p. 57. [10] (NICE) NIfHaCE. The guidelines manual. 2009 ed. London: National Institute for Health and Clinical Excellence; 2009. p. 147. [11] French JA, Kanner AM, Bautista J, Abou-Khalil B, Browne T, Harden CL, et al. Efficacy and tolerability of the new antiepileptic drugs I: treatment of new onset epilepsy: report of the Therapeutics and Technology Assessment Subcommittee and Quality Standards Subcommittee of the American Academy of Neurology and the American Epilepsy Society. Neurology 2004;62:1252–60. [12] French JA, Kanner AM, Bautista J, Abou-Khalil B, Browne T, Harden CL, et al. Efficacy and tolerability of the new antiepileptic drugs II: treatment of refractory epilepsy: report of the Therapeutics and Technology Assessment Subcommittee and Quality Standards Subcommittee of the American Academy of Neurology and the American Epilepsy Society. Neurology 2004;62:1261–73. [13] Glauser T, Ben-Menachem E, Bourgeois B, Cnaan A, Chadwick D, Guerreiro C, et al. ILAE treatment guidelines: evidence-based analysis of antiepileptic drug efficacy and effectiveness as initial monotherapy for epileptic seizures and syndromes. Epilepsia 2006;47:1094–120. [14] Chelune GJ. Evidence-based research and practice in clinical neuropsychology. Clin Neuropsychol 2010;24:454–67. [15] APA. Presidential Task Force on Evidence-Based Practice. Evidence-based practice in psychology. Am Psychol 2006;61:271–85. [16] Helmstaedter C. Neuropsychological aspects of epilepsy surgery. Epilepsy Behav 2004;5(Suppl. 1):S45–55. [17] Baxendale S. The impact of epilepsy surgery on cognition and behavior. Epilepsy Behav 2008;12:592–9. [18] Helmstaedter C, Loer B, Wohlfahrt R, Hammen A, Saar J, Steinhoff BJ, et al. The effects of cognitive rehabilitation on memory outcome after temporal lobe epilepsy surgery. Epilepsy Behav 2008;12:402–9. [19] Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. J Am Med Assoc 1993;270:2598–601. [20] Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. J Am Med Assoc 1994;271:59–63. [21] Strauss E, Sherman EMS, Spreen O. A compendium of neuropsychological tests. New York: Oxford University Press; 2006. [22] Marx RG, Bombardier C, Hogg-Johnson S, Wright JG. Clinimetric and psychometric strategies for development of a health measurement scale. J Clin Epidemiol 1999;52:105–11. [23] Wilton NK, Slim AM. Application of the principles of evidence-based medicine to patient care. South Med J 2012;105:136–43. [24] Robinson KA, Saldanha IJ, McKoy NA. Development of a framework to identify research gaps from systematic reviews. J Clin Epidemiol 2011;64:1325–30. [25] Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. Br Med J 2004;328:1490. [26] McIntosh AM, Wilson SJ, Berkovic SF. Seizure outcome after temporal lobectomy: current research practice and findings. Epilepsia 2001;42:1288–307.