- Email: [email protected]
Stability of predictors of outcome of surgical treatment for epilepsy
j Epilepsy 1990;3:29-35 0 1990 Demos Publications
Stability of Predictors of Outcome of Surgical Treatment for Epilepsy ‘Carl B. Dodrill,
2Gerald van Belle, and 3Robert J. Wilkus
Predictors of outcome of cortical resection surgery for epilepsy have differed markedly from report to report. A study of this problem was undertaken with 100 cases with 2-year follow-ups on whom 70 preoperative variables were obtained. Findings from predictive studies of differing sample sizeswere simulated by conducting serial analyseswith subgroups of 20,25,33, and 50 subjects. For each subgroup and for the total group of 100, subjects reporting at least a 75% reduction in seizure frequency in the second year following surgery (“successes’3 were statistically compared with those having a less favorable outcome (“failures”). Single predictors applied to groups of 20 or 25 subjects produced highly variable findings. Even single predictors from groups of 33 or 50 patients were replicated inconsistently. However, sets of multivariate predictors were more stable, especially in the larger groups. Such sets of variables in combination with larger sample sizes and conservative significance levels offer the best chance of producing results that will hold acrossstudies and over time. Key Words: Epilepsy surgeryEpilepsy-EEG-Neuropsychology-Statistics.
The investigation of variables that may be predictive of successful outcome of cortical resection surgery for epilepsy is important in view of the cost, the risk of morbidity, and the fact that not all patients profit from this surgery. At least 28 such variables have been studied by two or more investigators (l), but there is a great deal of inconsistency in the conclusions reached about the predictive value of most variables. This can be illustrated by a review of 26 studies, each of which reported on a minimum of 10 subjects and on one or more predictive variables (2-27). In these studies, 14 variables were addressed on five or more occasions. A listing of these variables is given in Table 1, along with the number of instances in which From the Departments of ‘Neurological Surgery, ‘Biostatistics,and 3Laboratory Medicine and Medicine, University of Washington School of Medicine, Seattle, WA, U.S.A. Address correspondence and reprint requests to Dr. C. B. Dodrill at Epilepsy Center (ZA-50), Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, U.S.A.
(a) the presence of the variable (or a larger numerical value of it) was associated with a positive surgical outcome, (b) there was no association found between the variable and surgical outcome, and (c) the number of instances in which the absence of the variable (or a smaller numerical value associated with it) was found in connection with a favorable surgical outcome. On only two of the 14 variables did the concurrence rate between the studies even reach 75% lobe(s)” and (“discharges, anterior-midtemporal “discharges mostly/solely on surgical side”). A further evaluation of the 26 investigations surveyed by Table 1 reveals two additional characteristics. First, the majority had relatively small sample sizes (often no more than 30 subjects). Second, in only two reports (1,14) was a multivariate approach attempted. In both of these, the multivariate approach proved to be superior to univariate prediction. In view of the above, the present study was directed toward assessing the stability and the validity of predictors by systematically studying the effects of group size on predictor reliability and also by evaluating the ] EPILEPSY, VOL. 3, NO. I, 2990
29
C. B. DODRILL
ET AL.
Table 1.
Numbers of studies of predictive variables investigated five times or more showing relationships with favorable outcome from surgery Relationship Variable
type/variable
General Age at operation Seizure history Age at onset of epilepsy Duration of epilepsy Etiology of epilepsy known Clinically localized to temporal lobe One seizure type only EEG Single focus Discharges, anterior-midtemporal lobe(s) Discharges mostly/solely on surgery side Ictal event recorded Long-term EEG/closed-circuit TV monitoring, Radiology/surgery Surgery on speech-related side Abnormal cerebral neuroradiological findings Abnormal cerebral neuroradiological findings
merits of multivariate dictors.
Positive
of any type/location in surgical area
as opposed to univariate pre-
Materials and Methods Subjects The subjects used in this study were identical to those described elsewhere (1). These 100 adults had cortical resection surgery for epilepsy at the University of Washington School of Medicine from September 11,1973, through July 28,1983. Preoperative data were available on each subject from a range of disciplines. There were 54 females and 46 males. At time of operation, they averaged 27 years of age and 12 years of education. Thirty-eight were employed full-time, 37 were employed part-time, and 25 were unemployed. The primary seizure diagnoses were complex partial in 85 cases, elementary partial in 9, and other types in 6. Age at onset of seizures averaged 10 years. Etiology was known in 55 cases. Surgery was on the left cerebral hemisphere in 42 cases, and in 45 it was on the same side as speech (intracarotid amobarbital testing showed speech only on the left in 86 cases, only on the right in 6 cases, and bilaterally in 8 cases). Surgery guided by cortical mapping and electrocorticography was performed solely on the temporal lobe in 180 cases, on a combination of the temporal lobe and other areas in 7 cases, and on areas other than the 30 1 EPILEPSY, VOL. 3, NO. 1, 2990
None
outcome Negative
1
4
2
4 5 5 4
5 5 3 2 2
1 3 0 0 0
4 6 10 3 4
2 0 3 2 2
0 0 0 1 0
2 1 1
3 4
0 0
4
0
0
any type
to favorable
various
temporal lobe in 13 cases. In the second postoperative year, 40 subjects had no attacks of any kind and 26 others had at least a 75% reduction in seizure frequency. These cases were considered surgical successes, whereas the 34 individuals with less favorable outcomes were considered failures. Procedure In addition to being evaluated as a single group of 100 cases (Group l), subjects were randomly divided into two groups of 50 (Groups 2A and 2B), three groups of 33 (Groups 3A, 3B, and 3C; one subject was discarded), four groups of 25 (Groups 4A, 48,4C, and 4D), and five groups of 20 (Groups 5A, 58,5C, 5D, and 5E). A total of 70 predictive variables were identified from five general categories. They were described in detail previously (1) and are summarized here: (a) general-four variables (age at operation; sex; history of psychiatric disorder or treatment; occupational status); (b) seizure history-seven variables (age at onset of seizures; duration; etiology; seizures clinically localized to the temporal lobe(s); number of seizure types; aura; precipitating factors); (c) radiology/surgey-threevariabl es (ab normal cerebral neuroradiological findings in any area; abnormal neuroradiological findings in the surgical area; surgery on speech-related hemisphere); (d) EEG-nine variables
PREDICTION
(single focus; discharges mostly from temporal lobe(s); discharges from anterior-midtemporal lobe(s); discharges only from side of operation; generalized discharges; rate of discharges in surgical area; diffuse nonepileptiform abnormalities; ictal event recorded; long-term EEG/closed-circuit TV monitoring); and (e) p~ychology/neuropsychology-47 variables [all subtests and IQ scores from the Wechsler Adult Intelligence Scale (WAIS); all 16 test measures from the Neuropsychological Battery for Epilepsy (9), plus the percent of the 16 scores that were outside normal limits; the Halstead Impairment Index and the Marching Test; standard scales of the Minnesota Multiphasic Personality Inventory (MMPI) and an index of overall profile elevation -average of scales l-4 and 6-O].
Analyses Two types of analyses were undertaken. The first was directed toward studying the effects of differing sample sizes on the consistency or stability of predictors of surgical outcome. For each group or subgroup, the scores of surgical successes were contrasted with those of surgical failures for ezch of the 70 predictive variables considered individually. Where the data were discrete (Yes or No, Present or Absent), the Chisquare statistic was utilized, and where the data were continuous, Student’s t statistic was applied. As is routinely done in studies of this type, the 0.05 probability level was used to determine statistical significance. The second type of statistical analysis was directed toward studying the potential value of using multivariate as compared to univariate predictors. For this analysis, only the eight predictive variables significant at the 0.01 level of confidence were used. They included four EEG variables (single focus; discharges mostly from the temporal lobe(s); discharges from the anterior-midtemporal lobe(s); discharges only from side of operation) and four psychology/neuropsychology variables (WAIS Digit Symbol; Marching Test preferred hand time; MMPI variables of Hypochondriasis and Hysteria). A patient was credited with one point for the presence of each EEG variable, and a summary EEG score across the four areas was obtained for each patient. Likewise, a composite psychology/neuropsychology score was obtained for each patient according to the number of favorable scores as previously determined (1) (WAIS Digit Symbol of 9 or greater or a raw score of 47 or greater on the WAIS-R Digit Symbol; time of 20 s or quicker to complete the Marching Test with the preferred hand; score of 75 or less on the MMPI Hypochondriasis
OF SURGICAL
OUTCOME
Scale; score of 80 or less on the Hysteria Scale). Fisher’s Exact Test was then applied to the two (outcome of surgery: success vs. failure) by two [EEG composite score of O-l vs. 2-4 or psychology/neuropsychology composite score of O-2 vs. 3-4 Or EEG + psychology/neuropsychology composite score of O-4 vs. 5-8; these dichotomous classifications were set by us previously (l)].
Results The initial analysis utilized all 100 subjects, and it compared each of the 70 variables across the surgical successes (n = 66) with the surgical failures (n = 34). In so doing, it was discovered that 17 variables differentiated these outcomes. None of these was in the general, seizure history, or radiology/surgery areas. However, four were EEG variables (single focus; discharges mostly from the temporal lobe(s); discharges from the anterior-midtemporal lobe(s); discharges only from side of operation) and 13 were psychological/neuropsychologicaI (WAIS Arithmetic, Digit Symbol, and Block Design; the neuropsychological measures of name writing speed, Seashore Rhythm Test, Seashore Tonal Memory Test, Trail Making Test Part B time, rating of constructional dyspraxia in paper and pencil drawings, Halstead Impairment Index, and Marching Test preferred hand time; the MMPI variables of Hypochondriasis Scale, Hysteria Scale, and Average Profile Elevation). In every instance, the intuitively more desirable circumstance (e.g., single focus, discharges from the anterior-midtemporal lobe(s), greater intelligence, less neuropsychological impairment, lower MMPI scores) was associated with a greater likelihood of surgical success. To determine the stability of variables with the randomly selected subgroups (2A, 2B, 3A, etc.), the same series of statistical analyses run with the total sample was accomplished with every subgroup. The extent to which statistically significant differences found with the total group were replicated in the subgroups of various sizes is presented in Table 2. In interpreting this table and Table 3, recall that originally the numbers of variables evaluated were as follows: general, 4; seizure history, 7; radiology/surgery, 3; EEG, 9; psychology/neuropsychology, 47. Since there were no statistically significant predictors within the general, seizure history, and radiology/surgery areas for the total sample (n = loo), replication could not be attempted in these areas. For EEG, there were four variables that did produce statistically signiEcant findings with the total sample, and each of these four variables was examined for the two groups of 50 subJ EPILEPSY, VOL. 3, NO. 1, 2990
31
C. B. DODRILL
Table 2.
ET AL.
Proportions of statistically significant (p < 0.05) relationships found in subgroups for those predictors that were significant for the total sample . Subgroup Number of statistically significant predictors for total sample (n = 100)
Type of variable General Seizure history Radiology/surgery EEG
13
The number of predictors is the number of variables demonstrating measure (relief from seizures) when all 100 subjects were used.
jects (total of eight statistical comparisons), for the three groups of 33 subjects (12 comparisons), for the four groups of 25 subjects (16 comparisons), and for the five groups of 20 subjects (20 comparisons). In the two groups of 50 subjects, five of the eight (5/8 = 0.62) statistical comparisons resulted in the same statistically significant findings noted with the total sample. For n = 33,5 of 12 (5/12 = 0.42) comparisons resulted in similar findings. The other values in the table were obtained in the same fashion. As can be seen, the smaller the sample size, the less the likelihood that results similar to those with the total group (n = 100) will be found. Also, the psychology/neuropsychology variables held up less well than the EEG variables, apparently because 9 of the 13 were statistically significant at a lower level (0.05) than all four EEG variables (0.01). The degree to which false-positive findings were observed in the subgroups was evaluated by deter-
33
0.62 (5/8) 0.31 (E/26)
0 0 0 4
Psychology/neuropsychology
Table 3.
50
a statistically
0.42 (5/12) 0.28 (11/39)
significant
relationship
0.31 (6/16) 0.12 (6/52)
0.20 (4/20) 0.10 (7/65)
with the outcome
the proportions of statistically significant associations in the subgroups that had not been found in the total group. The results of these analyses are presented in Table 3. The numbers of predictors on which the proportions are based are those that did not achieve statistical significance with the total sample (n = 100). For example, all four general predictors were nonsignificant with the total sample. Each of these was considered for each of the two groups of 50, so that there were eight comparisons. Of these, not a single one achieved significance (O/8 = 0.00). For n = 33, however, the four general variables were considered on each of three occasions (once for each group), and in one instance a significant difference was found (l/12 = 0.08). Overall, the smaller the group size, the greater the likelihood of findings that were spurious. EEG represents an exception to this, probably for the reason already mentioned. Attention was then turned to analyses in which
Number of predictors not statistically significant in the total sample (n = 100)
General
4
50
33
0.00
0.08 (l/12) 0.00 (O/21) 0.22 (219) 0.00 (O/15) 0.06 (6/102)
WV history
7
Radiology/surgery
3
EEG
5
0.00 (O/14) 0.00
(O/6) Psychology/neuropsychology of predictors
34 is the number
of variables
thatfailed to demonstrate
outcome measure (relief from seizures) when all 100 subjects were used. 32
20
Proportions of statistically significant (p < 0.05) relationships found in subgroups for those predictors that were not significant for the total sample
Type of variable
The number
25
mining
Subgroup
Seizure
size
J EPILEPSY, VOL. 3, NO. 1, 2990
0.00 (O/10) 0.09 (6/68)
a statistically
significant
size 25 0.06 (l/16) 0.07 (2/28) 0.00 (O/12) 0.05 (l/20) 0.11 (5/136) relationship
20 0.05 (l/20) 0.03 (l/35) 0.13 (2/15) 0.00 (O/25) 0.08 (19/170) with the
PREDICTION
Table 4.
OF SURGICAL OUTCOME
Statistical significance with the application of summa y predictors to the total group and to subgroups using Fisher‘s Exact Test Type of summary predictor
Group 1 2A 28 3A 3B 3c
4A 4B 4c 4D
5A 5B 5c 5D 5E
Group size 100 50 50 33 33 33 25 25 25 25 20 20 20 20 20
EEG
Psychology/ neuropsychology
EEG + psychology/ neuropsychology
0.001 NS
0.001 0.05
0.01 NS 0.05 NS 0.05 NS NS NS NS NS NS NS 0.05
0.001 NS 0.01
0.01 NS 0.01 NS 0.05 NS NS NS
0.01 NS NS NS NS NS 0.01
multiple variables were summed in composite scores for EEG (number of predictive variables in a favorable range), for psychology/neuropsychology (number of predictive variables in a favorable range), and for EEG + psychology/neuropsychology taken together. The results are summarized in Table 4 for the total sample (Group 1) and for every subgroup. Whereas statistical significance is consistently apparent for the first analysis using all 100 subjects, it starts to become intermittent as soon as groups of 50 are considered, and it becomes less and less frequent as the group sizes diminish. A study of Table 4 will reveal that statistical significance was found in 67% (4/6) of the comparisons for n = 50, in 44% (4/9) for n = 33, in 25% (3/12) for n = 25, and in 27% (4/15) for n = 20. Furthermore, it should be noted that multiple regression failed to result in rates of improved stability over the multivariate approach just described. Thus, the detailed multiple regression equations and their outcomes will not be presented.
Discussion Individual predictors of seizure relief from cortical resection surgery for epilepsy are weak and they tend to be unstable. As sample sizes decrease, it becomes less and less likely that a variable will be identified as predictive of seizure relief. This is true even for variables that are predictive when the total sample is considered. Even with EEG variables, only 62% of those demonstrating statistically significant findings with
0.001 NS NS 0.01 NS NS NS NS 0.05 0.01
the total sample did so with our samples of 50 subjects. When groups of 20 subjects were considered, only one EEG predictor in five was identified as having been known to be valid with all subjects. It is clear that sample size is related to stability of outcome. Table 3 reveals that spurious findings also appear in the subgroups. With statistical tests performed at the p < 0.05 level, one would expect 24 spurious findings from the 476 tests run on the variables that did not demonstrate significance with the total sample. In actuality, 36 were observed (7.56%). This slightly increased error rate may be in part due to intercorrelation among predictive variables. In any such evaluation of small groups, errors include both (a) not finding variables known to be of value as demonstrated by a larger sample and (b) finding variables that are known not to be of value. Whereas the relevance of sample size to study outcome is demonstrated both here and in statistical simulation studies (28,29), it has also been noted that the potency of each variable (its strength as a predictor) is also directly relevant to study outcome (28). The validity of this less well-appreciated fact is demonstrated in the present study. In the last line of Table 2, it was shown that the proportion of statistical findings replicated with the subgroups for the psychology/ neuropsychology area was less than for the EEG area. It has also been noted that whereas all four EEG variables were statistically related to the outcome measure at the p < 0.01 level, for the psychology/ neuropsychology variables, 9 of the 13 were statisticalJ EPILEPSY, VOL. 3, NO. 2, 2990
33
C. B. DODRIU
ET AL.
ly related at the 0.05 level and four at the 0.01 level. Had only these latter variables been considered, the four proportions in the last line of Table 2 would have been at higher and more acceptable levels (0.50,0.33,0.19, and 0.15). Furthermore, had the same 0.01 level been applied to Table 3, the 34 spurious findings in the psychology/neuropsychology area would have been reduced to four, and the values in the last line of that table would have been 0.01, 0.02, 0.00, and 0.00. Again, this is a much more acceptable error rate. Thus, the strength of the predictive relationship originally demonstrated is important, along with the number of variables used as predictors. Insisting on using a more conservative level of statistical confidence (e.g., p < 0.01) would do much to help provide stable predictors in future studies. This investigation had the great advantage of having the subsamples as part of the overall sample. A lesser degree of correspondence between groups would be expected when samples are entirely independent, such as when they are from different centers. It is possible that predictor stability would have been greater had the study been restricted to patients with temporal (or extratemporal) resections alone, but sample size limitations made this impossible for the extratemporal group, and perusal of the data did not provide support for this hypothesis. It is also possible that division of cases by those who were totally seizure free (successes) versus all others (failures) might have resulted in increased stability of predictors, but no evidence for this was found. Multivariate analyses as summarized in Table 4 did appear to result in slightly improved stability of prediction over single variables. However, when the sample size dropped to less than 50, prediction was still very inconsistent. Furthermore, as was noted earlier, multiple regression did not result in increased predictability beyond that obtained by simply adding up the number of variables that fell in the desirable range. These findings are similar to those from our previous work (l), and while multivariate procedures are of some value, it is not clear to us that any such method would result in greatly enhanced predictability. The reason, of course, is that the single variables on which they are based have limited stability. This agrees with the inconsistent results noted in the published literature (Table l), and since the majority of these studies employed fewer than 50 subjects, at least one reason is more clearly evident for the variable findings previously reported. The predictive variables used in this study and in most other studies were at best of only moderate potency, and this clearly contributes to their limited stability. Under these circumstances, increasing the 34
J EPILEPSY, VOL. 3, NO. 1, 2990
sample size to 300 or more cases has been recommended (28). Because of the limitations of any single surgical center, it may be necessary for several centers to participate in joint data gathering efforts to perform truly definitive studies. In general, the following recommendations are offered for future investigations: (a) study an absolute minimum of 50 subjects with 100 or more cases being highly desirable, when the purpose is to identify predictive variables; (b) utilize a more conservative rather than a less conservative level of statistical significance to help avoid false-positives; and, (c) employ composite scores instead of single predictors whenever possible to enhance stability and generalizability of findings. Acknowledgment: This investigation was supported by grants NS 17277 and NS 17111 awarded by the National Institute of Neurological Disorders and Stroke, PHS/DHHS.
References 1. Dodrill CB, Wilkus RJ, Ojemann GA, Ward AA, Wyler AR, Van Belle G, Tamas L. Multidisciplinary prediction of seizure relief from cortical resection surgery. Ann Neural 1986;20:2-12. 2. Bailey P, Gibbs FA. The surgical treatment of psychomotor epilepsy. JAMA 1951;145:365-9. 3. Bengzon ARA, Rasmussen T, Gloor P, Dassault J, Stephens M. Prognostic factors in the surgical treatment of temporal lobe epileptics. Neurology 1968;18:717-31. 4. Bergen D, Morrel F, Bleck TP, Whisler WW. Predictors of success in surgical treatment of intractable epilepsy. Epilepsia 1984;25:665. 5. Bhatia R, Kollevold T. A follow-up study of 91 patients operated on for focal epilepsy. Epilepsia 1976;17:616. 6. Bidzinski J. Proba oceny wynikow operacyjnego leczenia pedaczki skroniowej w zaleznosci od etiologii, charakreru napadow i poziomu umystowego. Neural Neurochir PO1 1971;21:427-30. 7. Bloom D, Jasper H, Rasmussen T. Surgical therapy in
patients with temporal lobe seizures and bilateral EEG abnormality. Epilepsia 1959/1960;1:351-5. 8. Delgado-Escueta AV, Walsh GO. The selection process for surgery of intractable complex partial seizures: surface EEG and depth electrography. In: Ward AA Jr, Penry JK, Purpura D, eds. Epilepsy. New York: Raven Press, 1983:295-326. 9. Dodrill CB. A neuropsychological battery for epilepsy. Epilepsia 1978;19:611-23. 10. Falconer MA, Serafetinides EA. A follow-up study of surgery and temporal lobe epilepsy. J Neural Neurosurg Psychiatry 1963;26:154-65. 11. Fenyes I, Zoltan I, Fenyes G. Temporal epilepsies with deep-seated epileptogenic foci. Arch Neural 1961;4: 559-71. 12. Green JR, Scheetz DG. Surgery of epileptogenic lesions of the temporal lobe. Arch Neural 1964;10:13548.
PREDlC77ON 13. Jensen I. Temporal lobe epilepsy: etiological factors and surgical results. Actu Neurol Stand 1976;53:10318. 14. Lieb JP, Engle J Jr, Gevins A, Crandall PH. Surface and deep EEG correlates of surgical outcome in temporal lobe epilepsy. Epilepsia 1981;22:515-38. 15. Lieb JP, Rausch R, Engle J Jr, Brown WJ, Crandall PH. Changes in intelligence following temporal lobectomy: relationships to EEG activity, seizure relief, and pathology. Epilepsia 1982;23:1-13. 16. Morris AA. Temporal lobectomywith removal of uncus, hippocampus and amygdala. Arch Nero1 Psychiatry 1956;76:479-96. 17. Paillas JE. Aspects cliniques de l’epilepsie temporale. In: Baldwin M, Bailey P, eds. Temporal lobe epilepsy. Springfield, IL: Charles C Thomas, 1958:411-39. 18. Rapport RL, Ojemann GA, Wyler AR, Ward AA Jr. Surgical management of epilepsy. West J Med 1977;127: 185-9. 19. Rasmussen TB. Surgical treatment of patients with complex partial seizures. In: Penry JK, Daly DD, eds. Complex partial seizures and their treatment. New York: Raven Press, 1975:415-42. (Advances in neurology; vol 11.) 20. Rausch R, Crandall PH. Psychological status related to surgical control of temporal lobe seizures. Epilepsiu 1982;23:191-202. 21. Shefer DG, Belyaev YI, Bein VN, Boreiko VB. Remote results subsequent to surgical treatment of temporal
22. 23. 24.
25
26. 27.
28. 29.
OF SURGICAL
OuTCOh4E
epilepsy through partial resection of the temporal lobe. Vopr Nejrokhir 1970;34:17-24. Stepien L, Bidzinski J, Mazurowski W. The results of surgical treatment of temporal lobe epilepsy. Pal Med 1 1969;8:1184-90. Taylor DC, Falconer MA. Clinical, socio-economic, and psychological changes after temporal lobectomy for epilepsy. Br J Psychiutry 1968;114:1247-61. Wannamaker BB, Matthews CG. Prognostic implications of neuropsychological test performance for surgical treatment of epilepsy. J Nerv Merit Dis 1976;163: 29-34. Wieser HG, Yasargil G. Selective amygdalohippocampectomy: follow-up study of 103 patients. In: Wolf P, Dam M, Janz D, Dreifuss FE, eds. Adunnces in epileptology, ~0116. New York: Raven Press, 1987~331-5. Wyler AR, Walker G, Richey ET, Hermann BP. Chronic subdural strip electrode recordings for difficult epileptic problems. J Epilepsy 1988;1:71-8. Wyllie E, Luders H, Morris HI-I, Lesser RP, Dinner DS, Hahn J, Estes ML, Rothner AD, Erenberg MD, Cruse R, Friedman D. Clinical outcome after complete or partial cortical resection for intractable epilepsy. Neurology 1987;37:1634-41. Flack VF, Chang PC. Frequency of selection noise variables in subset regression analysis: a simulation study. Am Statistician 1987;41:84-6. Guadagnoli E, Velicer WF. Relation of sample size to the stability of component patterns. Psychol Bull 1988; 103:265-75.
1 EPILEPSY, VOL. 3, NO. 1, 2990
35
Stability of Predictors of Outcome of Surgical Treatment for Epilepsy ‘Carl B. Dodrill,
2Gerald van Belle, and 3Robert J. Wilkus
Predictors of outcome of cortical resection surgery for epilepsy have differed markedly from report to report. A study of this problem was undertaken with 100 cases with 2-year follow-ups on whom 70 preoperative variables were obtained. Findings from predictive studies of differing sample sizeswere simulated by conducting serial analyseswith subgroups of 20,25,33, and 50 subjects. For each subgroup and for the total group of 100, subjects reporting at least a 75% reduction in seizure frequency in the second year following surgery (“successes’3 were statistically compared with those having a less favorable outcome (“failures”). Single predictors applied to groups of 20 or 25 subjects produced highly variable findings. Even single predictors from groups of 33 or 50 patients were replicated inconsistently. However, sets of multivariate predictors were more stable, especially in the larger groups. Such sets of variables in combination with larger sample sizes and conservative significance levels offer the best chance of producing results that will hold acrossstudies and over time. Key Words: Epilepsy surgeryEpilepsy-EEG-Neuropsychology-Statistics.
The investigation of variables that may be predictive of successful outcome of cortical resection surgery for epilepsy is important in view of the cost, the risk of morbidity, and the fact that not all patients profit from this surgery. At least 28 such variables have been studied by two or more investigators (l), but there is a great deal of inconsistency in the conclusions reached about the predictive value of most variables. This can be illustrated by a review of 26 studies, each of which reported on a minimum of 10 subjects and on one or more predictive variables (2-27). In these studies, 14 variables were addressed on five or more occasions. A listing of these variables is given in Table 1, along with the number of instances in which From the Departments of ‘Neurological Surgery, ‘Biostatistics,and 3Laboratory Medicine and Medicine, University of Washington School of Medicine, Seattle, WA, U.S.A. Address correspondence and reprint requests to Dr. C. B. Dodrill at Epilepsy Center (ZA-50), Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, U.S.A.
(a) the presence of the variable (or a larger numerical value of it) was associated with a positive surgical outcome, (b) there was no association found between the variable and surgical outcome, and (c) the number of instances in which the absence of the variable (or a smaller numerical value associated with it) was found in connection with a favorable surgical outcome. On only two of the 14 variables did the concurrence rate between the studies even reach 75% lobe(s)” and (“discharges, anterior-midtemporal “discharges mostly/solely on surgical side”). A further evaluation of the 26 investigations surveyed by Table 1 reveals two additional characteristics. First, the majority had relatively small sample sizes (often no more than 30 subjects). Second, in only two reports (1,14) was a multivariate approach attempted. In both of these, the multivariate approach proved to be superior to univariate prediction. In view of the above, the present study was directed toward assessing the stability and the validity of predictors by systematically studying the effects of group size on predictor reliability and also by evaluating the ] EPILEPSY, VOL. 3, NO. I, 2990
29
C. B. DODRILL
ET AL.
Table 1.
Numbers of studies of predictive variables investigated five times or more showing relationships with favorable outcome from surgery Relationship Variable
type/variable
General Age at operation Seizure history Age at onset of epilepsy Duration of epilepsy Etiology of epilepsy known Clinically localized to temporal lobe One seizure type only EEG Single focus Discharges, anterior-midtemporal lobe(s) Discharges mostly/solely on surgery side Ictal event recorded Long-term EEG/closed-circuit TV monitoring, Radiology/surgery Surgery on speech-related side Abnormal cerebral neuroradiological findings Abnormal cerebral neuroradiological findings
merits of multivariate dictors.
Positive
of any type/location in surgical area
as opposed to univariate pre-
Materials and Methods Subjects The subjects used in this study were identical to those described elsewhere (1). These 100 adults had cortical resection surgery for epilepsy at the University of Washington School of Medicine from September 11,1973, through July 28,1983. Preoperative data were available on each subject from a range of disciplines. There were 54 females and 46 males. At time of operation, they averaged 27 years of age and 12 years of education. Thirty-eight were employed full-time, 37 were employed part-time, and 25 were unemployed. The primary seizure diagnoses were complex partial in 85 cases, elementary partial in 9, and other types in 6. Age at onset of seizures averaged 10 years. Etiology was known in 55 cases. Surgery was on the left cerebral hemisphere in 42 cases, and in 45 it was on the same side as speech (intracarotid amobarbital testing showed speech only on the left in 86 cases, only on the right in 6 cases, and bilaterally in 8 cases). Surgery guided by cortical mapping and electrocorticography was performed solely on the temporal lobe in 180 cases, on a combination of the temporal lobe and other areas in 7 cases, and on areas other than the 30 1 EPILEPSY, VOL. 3, NO. 1, 2990
None
outcome Negative
1
4
2
4 5 5 4
5 5 3 2 2
1 3 0 0 0
4 6 10 3 4
2 0 3 2 2
0 0 0 1 0
2 1 1
3 4
0 0
4
0
0
any type
to favorable
various
temporal lobe in 13 cases. In the second postoperative year, 40 subjects had no attacks of any kind and 26 others had at least a 75% reduction in seizure frequency. These cases were considered surgical successes, whereas the 34 individuals with less favorable outcomes were considered failures. Procedure In addition to being evaluated as a single group of 100 cases (Group l), subjects were randomly divided into two groups of 50 (Groups 2A and 2B), three groups of 33 (Groups 3A, 3B, and 3C; one subject was discarded), four groups of 25 (Groups 4A, 48,4C, and 4D), and five groups of 20 (Groups 5A, 58,5C, 5D, and 5E). A total of 70 predictive variables were identified from five general categories. They were described in detail previously (1) and are summarized here: (a) general-four variables (age at operation; sex; history of psychiatric disorder or treatment; occupational status); (b) seizure history-seven variables (age at onset of seizures; duration; etiology; seizures clinically localized to the temporal lobe(s); number of seizure types; aura; precipitating factors); (c) radiology/surgey-threevariabl es (ab normal cerebral neuroradiological findings in any area; abnormal neuroradiological findings in the surgical area; surgery on speech-related hemisphere); (d) EEG-nine variables
PREDICTION
(single focus; discharges mostly from temporal lobe(s); discharges from anterior-midtemporal lobe(s); discharges only from side of operation; generalized discharges; rate of discharges in surgical area; diffuse nonepileptiform abnormalities; ictal event recorded; long-term EEG/closed-circuit TV monitoring); and (e) p~ychology/neuropsychology-47 variables [all subtests and IQ scores from the Wechsler Adult Intelligence Scale (WAIS); all 16 test measures from the Neuropsychological Battery for Epilepsy (9), plus the percent of the 16 scores that were outside normal limits; the Halstead Impairment Index and the Marching Test; standard scales of the Minnesota Multiphasic Personality Inventory (MMPI) and an index of overall profile elevation -average of scales l-4 and 6-O].
Analyses Two types of analyses were undertaken. The first was directed toward studying the effects of differing sample sizes on the consistency or stability of predictors of surgical outcome. For each group or subgroup, the scores of surgical successes were contrasted with those of surgical failures for ezch of the 70 predictive variables considered individually. Where the data were discrete (Yes or No, Present or Absent), the Chisquare statistic was utilized, and where the data were continuous, Student’s t statistic was applied. As is routinely done in studies of this type, the 0.05 probability level was used to determine statistical significance. The second type of statistical analysis was directed toward studying the potential value of using multivariate as compared to univariate predictors. For this analysis, only the eight predictive variables significant at the 0.01 level of confidence were used. They included four EEG variables (single focus; discharges mostly from the temporal lobe(s); discharges from the anterior-midtemporal lobe(s); discharges only from side of operation) and four psychology/neuropsychology variables (WAIS Digit Symbol; Marching Test preferred hand time; MMPI variables of Hypochondriasis and Hysteria). A patient was credited with one point for the presence of each EEG variable, and a summary EEG score across the four areas was obtained for each patient. Likewise, a composite psychology/neuropsychology score was obtained for each patient according to the number of favorable scores as previously determined (1) (WAIS Digit Symbol of 9 or greater or a raw score of 47 or greater on the WAIS-R Digit Symbol; time of 20 s or quicker to complete the Marching Test with the preferred hand; score of 75 or less on the MMPI Hypochondriasis
OF SURGICAL
OUTCOME
Scale; score of 80 or less on the Hysteria Scale). Fisher’s Exact Test was then applied to the two (outcome of surgery: success vs. failure) by two [EEG composite score of O-l vs. 2-4 or psychology/neuropsychology composite score of O-2 vs. 3-4 Or EEG + psychology/neuropsychology composite score of O-4 vs. 5-8; these dichotomous classifications were set by us previously (l)].
Results The initial analysis utilized all 100 subjects, and it compared each of the 70 variables across the surgical successes (n = 66) with the surgical failures (n = 34). In so doing, it was discovered that 17 variables differentiated these outcomes. None of these was in the general, seizure history, or radiology/surgery areas. However, four were EEG variables (single focus; discharges mostly from the temporal lobe(s); discharges from the anterior-midtemporal lobe(s); discharges only from side of operation) and 13 were psychological/neuropsychologicaI (WAIS Arithmetic, Digit Symbol, and Block Design; the neuropsychological measures of name writing speed, Seashore Rhythm Test, Seashore Tonal Memory Test, Trail Making Test Part B time, rating of constructional dyspraxia in paper and pencil drawings, Halstead Impairment Index, and Marching Test preferred hand time; the MMPI variables of Hypochondriasis Scale, Hysteria Scale, and Average Profile Elevation). In every instance, the intuitively more desirable circumstance (e.g., single focus, discharges from the anterior-midtemporal lobe(s), greater intelligence, less neuropsychological impairment, lower MMPI scores) was associated with a greater likelihood of surgical success. To determine the stability of variables with the randomly selected subgroups (2A, 2B, 3A, etc.), the same series of statistical analyses run with the total sample was accomplished with every subgroup. The extent to which statistically significant differences found with the total group were replicated in the subgroups of various sizes is presented in Table 2. In interpreting this table and Table 3, recall that originally the numbers of variables evaluated were as follows: general, 4; seizure history, 7; radiology/surgery, 3; EEG, 9; psychology/neuropsychology, 47. Since there were no statistically significant predictors within the general, seizure history, and radiology/surgery areas for the total sample (n = loo), replication could not be attempted in these areas. For EEG, there were four variables that did produce statistically signiEcant findings with the total sample, and each of these four variables was examined for the two groups of 50 subJ EPILEPSY, VOL. 3, NO. 1, 2990
31
C. B. DODRILL
Table 2.
ET AL.
Proportions of statistically significant (p < 0.05) relationships found in subgroups for those predictors that were significant for the total sample . Subgroup Number of statistically significant predictors for total sample (n = 100)
Type of variable General Seizure history Radiology/surgery EEG
13
The number of predictors is the number of variables demonstrating measure (relief from seizures) when all 100 subjects were used.
jects (total of eight statistical comparisons), for the three groups of 33 subjects (12 comparisons), for the four groups of 25 subjects (16 comparisons), and for the five groups of 20 subjects (20 comparisons). In the two groups of 50 subjects, five of the eight (5/8 = 0.62) statistical comparisons resulted in the same statistically significant findings noted with the total sample. For n = 33,5 of 12 (5/12 = 0.42) comparisons resulted in similar findings. The other values in the table were obtained in the same fashion. As can be seen, the smaller the sample size, the less the likelihood that results similar to those with the total group (n = 100) will be found. Also, the psychology/neuropsychology variables held up less well than the EEG variables, apparently because 9 of the 13 were statistically significant at a lower level (0.05) than all four EEG variables (0.01). The degree to which false-positive findings were observed in the subgroups was evaluated by deter-
33
0.62 (5/8) 0.31 (E/26)
0 0 0 4
Psychology/neuropsychology
Table 3.
50
a statistically
0.42 (5/12) 0.28 (11/39)
significant
relationship
0.31 (6/16) 0.12 (6/52)
0.20 (4/20) 0.10 (7/65)
with the outcome
the proportions of statistically significant associations in the subgroups that had not been found in the total group. The results of these analyses are presented in Table 3. The numbers of predictors on which the proportions are based are those that did not achieve statistical significance with the total sample (n = 100). For example, all four general predictors were nonsignificant with the total sample. Each of these was considered for each of the two groups of 50, so that there were eight comparisons. Of these, not a single one achieved significance (O/8 = 0.00). For n = 33, however, the four general variables were considered on each of three occasions (once for each group), and in one instance a significant difference was found (l/12 = 0.08). Overall, the smaller the group size, the greater the likelihood of findings that were spurious. EEG represents an exception to this, probably for the reason already mentioned. Attention was then turned to analyses in which
Number of predictors not statistically significant in the total sample (n = 100)
General
4
50
33
0.00
0.08 (l/12) 0.00 (O/21) 0.22 (219) 0.00 (O/15) 0.06 (6/102)
WV history
7
Radiology/surgery
3
EEG
5
0.00 (O/14) 0.00
(O/6) Psychology/neuropsychology of predictors
34 is the number
of variables
thatfailed to demonstrate
outcome measure (relief from seizures) when all 100 subjects were used. 32
20
Proportions of statistically significant (p < 0.05) relationships found in subgroups for those predictors that were not significant for the total sample
Type of variable
The number
25
mining
Subgroup
Seizure
size
J EPILEPSY, VOL. 3, NO. 1, 2990
0.00 (O/10) 0.09 (6/68)
a statistically
significant
size 25 0.06 (l/16) 0.07 (2/28) 0.00 (O/12) 0.05 (l/20) 0.11 (5/136) relationship
20 0.05 (l/20) 0.03 (l/35) 0.13 (2/15) 0.00 (O/25) 0.08 (19/170) with the
PREDICTION
Table 4.
OF SURGICAL OUTCOME
Statistical significance with the application of summa y predictors to the total group and to subgroups using Fisher‘s Exact Test Type of summary predictor
Group 1 2A 28 3A 3B 3c
4A 4B 4c 4D
5A 5B 5c 5D 5E
Group size 100 50 50 33 33 33 25 25 25 25 20 20 20 20 20
EEG
Psychology/ neuropsychology
EEG + psychology/ neuropsychology
0.001 NS
0.001 0.05
0.01 NS 0.05 NS 0.05 NS NS NS NS NS NS NS 0.05
0.001 NS 0.01
0.01 NS 0.01 NS 0.05 NS NS NS
0.01 NS NS NS NS NS 0.01
multiple variables were summed in composite scores for EEG (number of predictive variables in a favorable range), for psychology/neuropsychology (number of predictive variables in a favorable range), and for EEG + psychology/neuropsychology taken together. The results are summarized in Table 4 for the total sample (Group 1) and for every subgroup. Whereas statistical significance is consistently apparent for the first analysis using all 100 subjects, it starts to become intermittent as soon as groups of 50 are considered, and it becomes less and less frequent as the group sizes diminish. A study of Table 4 will reveal that statistical significance was found in 67% (4/6) of the comparisons for n = 50, in 44% (4/9) for n = 33, in 25% (3/12) for n = 25, and in 27% (4/15) for n = 20. Furthermore, it should be noted that multiple regression failed to result in rates of improved stability over the multivariate approach just described. Thus, the detailed multiple regression equations and their outcomes will not be presented.
Discussion Individual predictors of seizure relief from cortical resection surgery for epilepsy are weak and they tend to be unstable. As sample sizes decrease, it becomes less and less likely that a variable will be identified as predictive of seizure relief. This is true even for variables that are predictive when the total sample is considered. Even with EEG variables, only 62% of those demonstrating statistically significant findings with
0.001 NS NS 0.01 NS NS NS NS 0.05 0.01
the total sample did so with our samples of 50 subjects. When groups of 20 subjects were considered, only one EEG predictor in five was identified as having been known to be valid with all subjects. It is clear that sample size is related to stability of outcome. Table 3 reveals that spurious findings also appear in the subgroups. With statistical tests performed at the p < 0.05 level, one would expect 24 spurious findings from the 476 tests run on the variables that did not demonstrate significance with the total sample. In actuality, 36 were observed (7.56%). This slightly increased error rate may be in part due to intercorrelation among predictive variables. In any such evaluation of small groups, errors include both (a) not finding variables known to be of value as demonstrated by a larger sample and (b) finding variables that are known not to be of value. Whereas the relevance of sample size to study outcome is demonstrated both here and in statistical simulation studies (28,29), it has also been noted that the potency of each variable (its strength as a predictor) is also directly relevant to study outcome (28). The validity of this less well-appreciated fact is demonstrated in the present study. In the last line of Table 2, it was shown that the proportion of statistical findings replicated with the subgroups for the psychology/ neuropsychology area was less than for the EEG area. It has also been noted that whereas all four EEG variables were statistically related to the outcome measure at the p < 0.01 level, for the psychology/ neuropsychology variables, 9 of the 13 were statisticalJ EPILEPSY, VOL. 3, NO. 2, 2990
33
C. B. DODRIU
ET AL.
ly related at the 0.05 level and four at the 0.01 level. Had only these latter variables been considered, the four proportions in the last line of Table 2 would have been at higher and more acceptable levels (0.50,0.33,0.19, and 0.15). Furthermore, had the same 0.01 level been applied to Table 3, the 34 spurious findings in the psychology/neuropsychology area would have been reduced to four, and the values in the last line of that table would have been 0.01, 0.02, 0.00, and 0.00. Again, this is a much more acceptable error rate. Thus, the strength of the predictive relationship originally demonstrated is important, along with the number of variables used as predictors. Insisting on using a more conservative level of statistical confidence (e.g., p < 0.01) would do much to help provide stable predictors in future studies. This investigation had the great advantage of having the subsamples as part of the overall sample. A lesser degree of correspondence between groups would be expected when samples are entirely independent, such as when they are from different centers. It is possible that predictor stability would have been greater had the study been restricted to patients with temporal (or extratemporal) resections alone, but sample size limitations made this impossible for the extratemporal group, and perusal of the data did not provide support for this hypothesis. It is also possible that division of cases by those who were totally seizure free (successes) versus all others (failures) might have resulted in increased stability of predictors, but no evidence for this was found. Multivariate analyses as summarized in Table 4 did appear to result in slightly improved stability of prediction over single variables. However, when the sample size dropped to less than 50, prediction was still very inconsistent. Furthermore, as was noted earlier, multiple regression did not result in increased predictability beyond that obtained by simply adding up the number of variables that fell in the desirable range. These findings are similar to those from our previous work (l), and while multivariate procedures are of some value, it is not clear to us that any such method would result in greatly enhanced predictability. The reason, of course, is that the single variables on which they are based have limited stability. This agrees with the inconsistent results noted in the published literature (Table l), and since the majority of these studies employed fewer than 50 subjects, at least one reason is more clearly evident for the variable findings previously reported. The predictive variables used in this study and in most other studies were at best of only moderate potency, and this clearly contributes to their limited stability. Under these circumstances, increasing the 34
J EPILEPSY, VOL. 3, NO. 1, 2990
sample size to 300 or more cases has been recommended (28). Because of the limitations of any single surgical center, it may be necessary for several centers to participate in joint data gathering efforts to perform truly definitive studies. In general, the following recommendations are offered for future investigations: (a) study an absolute minimum of 50 subjects with 100 or more cases being highly desirable, when the purpose is to identify predictive variables; (b) utilize a more conservative rather than a less conservative level of statistical significance to help avoid false-positives; and, (c) employ composite scores instead of single predictors whenever possible to enhance stability and generalizability of findings. Acknowledgment: This investigation was supported by grants NS 17277 and NS 17111 awarded by the National Institute of Neurological Disorders and Stroke, PHS/DHHS.
References 1. Dodrill CB, Wilkus RJ, Ojemann GA, Ward AA, Wyler AR, Van Belle G, Tamas L. Multidisciplinary prediction of seizure relief from cortical resection surgery. Ann Neural 1986;20:2-12. 2. Bailey P, Gibbs FA. The surgical treatment of psychomotor epilepsy. JAMA 1951;145:365-9. 3. Bengzon ARA, Rasmussen T, Gloor P, Dassault J, Stephens M. Prognostic factors in the surgical treatment of temporal lobe epileptics. Neurology 1968;18:717-31. 4. Bergen D, Morrel F, Bleck TP, Whisler WW. Predictors of success in surgical treatment of intractable epilepsy. Epilepsia 1984;25:665. 5. Bhatia R, Kollevold T. A follow-up study of 91 patients operated on for focal epilepsy. Epilepsia 1976;17:616. 6. Bidzinski J. Proba oceny wynikow operacyjnego leczenia pedaczki skroniowej w zaleznosci od etiologii, charakreru napadow i poziomu umystowego. Neural Neurochir PO1 1971;21:427-30. 7. Bloom D, Jasper H, Rasmussen T. Surgical therapy in
patients with temporal lobe seizures and bilateral EEG abnormality. Epilepsia 1959/1960;1:351-5. 8. Delgado-Escueta AV, Walsh GO. The selection process for surgery of intractable complex partial seizures: surface EEG and depth electrography. In: Ward AA Jr, Penry JK, Purpura D, eds. Epilepsy. New York: Raven Press, 1983:295-326. 9. Dodrill CB. A neuropsychological battery for epilepsy. Epilepsia 1978;19:611-23. 10. Falconer MA, Serafetinides EA. A follow-up study of surgery and temporal lobe epilepsy. J Neural Neurosurg Psychiatry 1963;26:154-65. 11. Fenyes I, Zoltan I, Fenyes G. Temporal epilepsies with deep-seated epileptogenic foci. Arch Neural 1961;4: 559-71. 12. Green JR, Scheetz DG. Surgery of epileptogenic lesions of the temporal lobe. Arch Neural 1964;10:13548.
PREDlC77ON 13. Jensen I. Temporal lobe epilepsy: etiological factors and surgical results. Actu Neurol Stand 1976;53:10318. 14. Lieb JP, Engle J Jr, Gevins A, Crandall PH. Surface and deep EEG correlates of surgical outcome in temporal lobe epilepsy. Epilepsia 1981;22:515-38. 15. Lieb JP, Rausch R, Engle J Jr, Brown WJ, Crandall PH. Changes in intelligence following temporal lobectomy: relationships to EEG activity, seizure relief, and pathology. Epilepsia 1982;23:1-13. 16. Morris AA. Temporal lobectomywith removal of uncus, hippocampus and amygdala. Arch Nero1 Psychiatry 1956;76:479-96. 17. Paillas JE. Aspects cliniques de l’epilepsie temporale. In: Baldwin M, Bailey P, eds. Temporal lobe epilepsy. Springfield, IL: Charles C Thomas, 1958:411-39. 18. Rapport RL, Ojemann GA, Wyler AR, Ward AA Jr. Surgical management of epilepsy. West J Med 1977;127: 185-9. 19. Rasmussen TB. Surgical treatment of patients with complex partial seizures. In: Penry JK, Daly DD, eds. Complex partial seizures and their treatment. New York: Raven Press, 1975:415-42. (Advances in neurology; vol 11.) 20. Rausch R, Crandall PH. Psychological status related to surgical control of temporal lobe seizures. Epilepsiu 1982;23:191-202. 21. Shefer DG, Belyaev YI, Bein VN, Boreiko VB. Remote results subsequent to surgical treatment of temporal
22. 23. 24.
25
26. 27.
28. 29.
OF SURGICAL
OuTCOh4E
epilepsy through partial resection of the temporal lobe. Vopr Nejrokhir 1970;34:17-24. Stepien L, Bidzinski J, Mazurowski W. The results of surgical treatment of temporal lobe epilepsy. Pal Med 1 1969;8:1184-90. Taylor DC, Falconer MA. Clinical, socio-economic, and psychological changes after temporal lobectomy for epilepsy. Br J Psychiutry 1968;114:1247-61. Wannamaker BB, Matthews CG. Prognostic implications of neuropsychological test performance for surgical treatment of epilepsy. J Nerv Merit Dis 1976;163: 29-34. Wieser HG, Yasargil G. Selective amygdalohippocampectomy: follow-up study of 103 patients. In: Wolf P, Dam M, Janz D, Dreifuss FE, eds. Adunnces in epileptology, ~0116. New York: Raven Press, 1987~331-5. Wyler AR, Walker G, Richey ET, Hermann BP. Chronic subdural strip electrode recordings for difficult epileptic problems. J Epilepsy 1988;1:71-8. Wyllie E, Luders H, Morris HI-I, Lesser RP, Dinner DS, Hahn J, Estes ML, Rothner AD, Erenberg MD, Cruse R, Friedman D. Clinical outcome after complete or partial cortical resection for intractable epilepsy. Neurology 1987;37:1634-41. Flack VF, Chang PC. Frequency of selection noise variables in subset regression analysis: a simulation study. Am Statistician 1987;41:84-6. Guadagnoli E, Velicer WF. Relation of sample size to the stability of component patterns. Psychol Bull 1988; 103:265-75.
1 EPILEPSY, VOL. 3, NO. 1, 2990
35