- Email: [email protected]
Reliability and Validity of Culturally Adapted Executive Function Performance Test for Koreans with Stroke
ARTICLE IN PRESS
Reliability and Validity of Culturally Adapted Executive Function Performance Test for Koreans with Stroke Hee Kim,
OT, PhD,*
Yu-Na Lee, OT, MS,† Eun-Mi Jo, Eun-Young Lee, OT, BHSc§
OT, MS,‡
and
Background: Executive Function Performance Test was unusable in Asia because of cultural differences, although evaluating the effect that executive function has on real life is essential to people with stroke. The aims of this study were to report the development and standardization of culturally appropriate Executive Function Performance Test for Koreans (EFPT-K) and to verify its reliability and validity in the stroke population. Methods: EFPT-K was developed by going through the process of translation, back-translation, and an expert committee’s conference on cultural adaptation. Inter-rater reliability was examined and 34 people with stroke were recruited to test the internal consistency and criterion validity of EFPT-K. To verify the criterion validity, EFPT-K scores were compared with those of the Trail-Making Test, the Korean Color–Word Stroop Test, the Digit Span Test, and the Assessment of Motor and Process Skills (AMPS). Results: Interrater reliability and internal consistency of the total score of EFPT-K supported high levels of reliability. In the criterion validity of EFPT-K, all neuropsychological tests but digits backward showed a correlation with the total score of EFPT-K, and AMPS components of initiation, notice/response, and termination showed a moderate correlation with the EFPT-K score. Conclusions: EFPT-K is a useful tool to evaluate the executive functioning of patients with stroke in real-life tasks and could be used as a sample in other Asian cultures where thorough evaluation of executive function in the performance of daily life is needed. Key Words: Executive function—performance—criterion validity—cultural adaptation—stroke. © 2017 National Stroke Association. Published by Elsevier Inc. All rights reserved.
Introduction Executive function disorder affects 19%-75% of patients with stroke according to how it is defined and assessed in each literature.1,2 As a higher cognitive function, From the *Department of Occupational Therapy, Konyang University, Daejeon, Republic of Korea; †Department of Occupational Therapy, Kyungbok University, Pocheon, Republic of Korea; ‡Department of Occupational Therapy, Sowoon Psychiatry Clinic, Seongnam, Republic of Korea; and §Department of Occupational Therapy, Hansol Convalescent Hospital, Jeonju, Republic of Korea. Received October 25, 2016; revision received November 29, 2016; accepted December 19, 2016. Address correspondence to Yu-Na Lee, OT, MS, Department of Occupational Therapy, Kyungbok University, 154, Sinpyeong-ro, Sinbuk-myeon, Pocheon-si, Gyeonggi-do, Republic of Korea. E-mail: [email protected]. 1052-3057/$ - see front matter © 2017 National Stroke Association. Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jstrokecerebrovasdis.2016.12.013
executive function controls and regulates lower ones, and integrates purposeful and goal-directed behavior processes.3,4 Executive function disorder lasts chronically even after 6 months from the onset of a stroke,5 diminishes the effects of rehabilitation and functional recovery,2 and limits returning to the workplace and social participation.6,7 Standardized executive function tests are essential to screen executive dysfunctions after stroke, to plan rehabilitation interventions considering the dysfunctions, and to evaluate the effect of cognitive rehabilitation in real life.1 Until now, paper-based neuropsychological tests such as the Stroop Color–Word Interference Test and the Trail-Making Test (TMT) were primarily used to evaluate executive functions. Besides, performance-based executive function tests such as the Allen Cognitive Levels Screening Battery (ACLS), Assessment of Motor and Process Skills (AMPS), Kitchen Task Assessment, and Behavioral Assessment of the Dysexecutive Syndrome (BADS) are used.
Journal of Stroke and Cerebrovascular Diseases, Vol. ■■, No. ■■ (■■), 2016: pp ■■–■■
1
ARTICLE IN PRESS H. KIM ET AL.
2
Nevertheless, paper-based neuropsychological tests cannot predict the performance of executive functions in naturalistic environment or real life.8 In particular, many neuropsychological tests are not sensitive to multitasking and have limitations of low ecological validity, which means the inability to evaluate the effect that executive dysfunctions have on daily activities.3,4,9,10 Among the performance-based tests, ACLS and AMPS are the most difficult to use when clearly evaluating the evaluator’s capability of assistance because these tests do not reflect the amount of help on the score when the evaluator provides gradual assistance, and Kitchen Task Assessment requires performance and assessment of only 1 task, which limits assessment of diverse aspects of the executive function.4 BADS is not usable in most clinical settings not only because preparing and setting up for complex 6 tasks are difficult but also because the ecological validity is low and could only be used in a certain cultural context.11,12 The Executive Function Performance Test (EFPT) was developed to overcome such limitations.3,4 EFPT is an evaluation tool that tests the degree of executive function deficits and one’s ability to independently function based on one’s performance on 4 daily tasks: cooking oatmeal, using a telephone, taking medications, and paying bills.4,13 Several studies support the use of original EFPT by verifying the reliability and validity when the test is performed on people with acute and chronic stroke and schizophrenia.4,14,15 Although EFPT was tested with reliability and validity, and consistently used in Western culture such as the United States, Israel, Sweden from the 2000s,4,14,15 original EFPT was not usable in Asia due to the cultural differences in instrumental activities of daily living (ADLs) that demand executive functions and how they perform the tasks. Using the 4 tasks from the original EFPT is impracticable as these tasks are culturally unsuitable, not used in Korean society, or used differently. Thus, not only translations but also replacing the tasks with equivalent levels of culturally fit tasks are necessary, and testing the reliability and validity is crucial to apply EFPT in Korea. The purpose of the present study was to report the development and standardization process of the culturally appropriate Executive Function Performance Test for Koreans (EFPT-K) and to verify its inter-rater reliability, internal consistency, and criterion validity.
Methods Translation and Cultural Adaptation Developing the EFPT-K was processed with the following 4 steps (Fig 1). The first step was to translate the original EFPT in Korean and back-translate it into English to examine the appropriateness of EFPT-K. After getting approved by the developer of the original EFPT, an occupational therapist (first author) bilingual in Korean and English translated the manual and evaluation form of the original EFPT
Translation and back-translation of EFPT
Cultural adaptation of EFPT tasks by expert committee
Survey on content validity and utility of EFPT-K targeting occupational therapists
Reliability and validity of EFPT-K to people with stroke Figure 1. Development process of EFPT-K. Abbreviation: EFPT-K, Executive Function Performance Test for Koreans.
obtained from the Cognitive Rehabilitation Research Group’s Website (http://crrg.wustl.edu/outcome_assessment.html) into Korean (V1). A bilingual undergraduate student majoring in occupational therapy (BT1) and a bilingual professor who majored in clinical psychology (BT2) each back-translated the V1 into English. Both back-translators were not familiar with EFPT, were blinded to the original one, and worked independently on back-translation. By comparing the two back-translations and original EFPT, the authors verified the clinical components and appropriateness of terminology. Discrepancies were discussed to find agreements to complete the EFPT translated in Korean (V2). Second, an expert committee was formed consisting of the translator, 1 occupational therapy professor expert in cognitive rehabilitation, and 3 occupational therapists with more than 5 years of clinical experience in cognitive rehabilitation. After discussing the cultural appropriateness of EFPT, the committee selected unsuitable tasks and processed to replace the tasks as shown in Table 1. Necessary materials, a manual, and an evaluation form were modified for the adapted tasks to make the prefinal version of EFPT-K.
Table 1. Tasks replaced by culturally appropriate ones in EFPT-K EFPT tasks
EFPT-K tasks
Making instant oatmeal Ordering groceries by phone Taking prescribed medications Paying bills by check
Making instant rice porridge Ordering delivery food by phone Taking prescribed vitamins Paying bills by transferring from bank account
Abbreviations: EFPT, Executive Function Performance Test; EFPT-K, Executive Function Performance Test for Koreans.
ARTICLE IN PRESS EXECUTIVE FUNCTION PERFORMANCE TEST FOR KOREANS (EFPT-K) WITH STROKE
Third, we examined the content validity and utility of the EFPT-K by conducting a survey targeting a professional group. The professional group consisted of 29 occupational therapy professors and occupational therapists with more than 3 years of clinical experience in the cognitive rehabilitation of stroke patients. The results of the survey are written in another article of the authors in Korean. After the survey, the final version of EFPT-K was completed. In the last step, the result of which was recorded in the present paper, EFPT-K was tested on people with stroke to verify its reliability and validity.
Study Design and Participants The study design was cross-sectional, performancebased, and psychometric. Study participants were 34 people who experienced a stroke and were recruited from the cities of Wonju and Jeonju in the Republic of Korea during October 2012 and February 2013. To recruit the participants, hospitals and welfare centers of the regions were contacted, and the purpose of the research was explained and approved by these hospitals and welfare centers. The purpose of the research and the rights of the participants were explained to each participant, and all participants signed informed consent forms. Inclusion criteria were (1) a diagnosis of stroke by a physician, (2) more than 6 months after stroke onset and at least 20 years of age, (3) a Mini-Mental State Examination— Korean (MMSE-K) score of 24 or higher, and (4) a Korean Modified Barthel Index (K-MBI) score of 62 or higher. Exclusion criteria were history of mental retardation before the stroke onset and other comorbid neurological or mental illness that can affect executive functioning. Among 34 participants, 24 were male and 10 were female, and their average age was 54.41. Twenty of the participants were inpatients of hospitals with rehabilitation setting and the other 14 were outpatients. Further characteristics of the study participants are presented in Table 2.
Measurements EFPT-K The EFPT, an instrument to evaluate one’s amount of executive dysfunctions and ability to perform independence, is composed of essential tasks to maintain self and independent living such as simple cooking, using a telephone, medication management, and paying bills.13 While a participant performs each task, an examiner scores 5 components of executive function, which are initiation, organization, sequencing, judgment and safety, and completion. For each component, the examiner scores the highest level of cueing the participant needs from 6 levels of the cueing system (0 = no cue required, 1 = verbal guidance,
3
Table 2. Demographic characteristics (N = 34) Characteristic Sex Male Female Patient type Inpatient Outpatient
Age (years) Education (years) Duration from onset (months) MMSE-K score K-MBI score
Participants
Percentage
24 10
70.6 29.4
20 14
58.8 41.2
Mean (SD)
Range
54.41 (12.22) 10.82 (3.46) 49.71 (57.01)
20-76 0-16 7-276
27.21 (2.04) 87.41 (11.43)
24-30 62-100
Abbreviations: K-MBI, Korean Modified Barthel Index; MMSE-K, Mini-Mental State Examination—Korean; SD, standard deviation.
2 = gestured guidance, 3 = direct verbal assistance, 4 = physical assistance, and 5 = therapist does the component instead of the participant). The total score of EFPT ranges from 0 to 100, indicating that a higher score requires more cueing and more severe executive function deficits. We used the EFPT-K in which we replaced the 4 tasks of the original EFPT by culturally adapted ones (Table 1). Korean Color–Word Stroop Test (K-CWST) K-CWST, an instrument composed of words and colors– words, is a neuropsychological test that evaluates the speed of processing information, selective attention, cognitive flexibility, and executive function.16 Participants are asked to correctly say the written words or colors as fast as possible in 45 seconds.17 The number of correctly named words or colors is the score indicating that, to have better executive function, one should achieve a higher score. TMT TMT, a neuropsychological test to evaluate the speed of mental exercise, complex attention, and executive function, is composed of A and B. TMT-A asks the participants to draw a line between numerals in order from 1 to 13, and TMT-B asks the participants to draw lines, alternatively connecting numbers and letters in order, such as 1-A, 2-B, and 3-C. The total times to perform a task and to commit mistakes are scored.18,19 People with higher TMT score need more time to solve the problem, which means they are in a lower executive function level. Digit Span Test Digit Span Test evaluates attention and working memory, and is composed of digit span forward and backward.20,21
ARTICLE IN PRESS H. KIM ET AL.
4
After the examiner verbally presents digits, participants are asked to repeat the digits in the presented sequence in the forward test and in the reverse sequence in the backward test. A higher score indicates better cognitive abilities. AMPS AMPS is an assessment tool to examine one’s motor and process skills when performing ADLs.22 The examinee selects 2 or 3 tasks that are meaningful and performed in real life among 83 tasks, and then 16 motor skills and 20 process skills are rated from 1 (deficit) to 4 (competent) points each. The raw scores are tapped in the AMPS program to be converted into logit scores, and changes in occupational performances of more than .5 logits are considered to be clinically and statistically meaningful ones.22 The test–retest reliabilities of conducting 2 tasks are r = .91 and r = .85, whereas those of performing a task are r = .81 and r = .71. In the present study, we used AMPS to compare the subscores of the specific performance skills such as initiation, organization, sequencing, notice/ response, and termination with that of executive function components of EFPT and analyze their correlation.
occupational therapists independently. Later, the rate of concordance on the observation, grading, and analysis result were examined. To verify the criterion validity of the final version of the EFPT-K, EFPT-K scores were compared with those of the TMT, K-CWST, and Digit Span Test, which are paperbased executive function tests, and AMPS, which is a performance-based test. All 34 participants went through the TMT, K-CWST, Digit Span Test, and AMPS to confirm the correlation between their scores in TMT, K-CWST, Digit Span Test, and AMPS and the EFPT-K scores.
Statistical Analysis All statistical analyses in the present study were conducted using the SPSS for Windows (Version 18.0; SPSS Inc., Chicago, IL). The demographic characteristics of the participants were computed using descriptive statistics. Intraclass correlation coefficients (ICCs) were computed to determine the inter-rater reliability and the Cronbach alpha coefficients were computed to determine internal consistency. Spearman’s rank correlation was conducted to determine criterion validity because EFPT adopts the Likert scale, which is considered ordinal.
MMSE-K MMSE-K is a dementia screening test altered to fit the Korean social context and tests cognitive abilities such as orientation, memory, attention and calculation, language function, understanding, and judgment by interviewing. The total score ranges from 0 to 30, and a score of 24 or higher is interpreted as normal, 20~23 as having mild cognitive disorder, 10~19 as having moderate cognitive disorder, and 9 or lower as having serious cognitive disorder.23 In the present study, MMSE-K was used as a criterion for including participants. K-MBI K-MBI, an evaluation tool to score chronic stroke patients’ independence in basic daily living, evaluates 10 basic ADLs with a 5-point scale and the total score ranges from 0 to 100. The basic ADLs are self-care, bathing, feeding, toileting, stair climbing, dressing, bladder and bowel control, walking (or wheelchair use), and transferring to a chair or a bed. A lower score means more dependence. The test–retest reliability is .89 and the interrater reliability is .95.24 K-MBI was utilized to set an inclusion criterion of the recruitment for the present study.
Process of Testing Reliability and Validity To test the reliability of the final version of EFPT-K, 3 occupational therapists who were well-informed of the methods, process of observation, and analysis result of EFPT partook to score 5 people with stroke. Participants’ performances were videotaped and later scored by 3
Results The averages and ranges of the participants’ scores in the EFPT-K, the neuropsychological test, and the AMPS are shown in Table 3. The mean of the EFPT-K total score was 20.38 (11.67) and managing medication scored the lowest, which was 3.56 (3.26) on average, and paying bills scored the highest, which was 7.35 (3.71) on average among EFPT-K tasks. Among the executive function components of EFPT-K, initiation was the lowest, which scored .0 (.0), and sequencing was the highest, with a mean of 9.00 (4.13). Subscores of AMPS, which were similar to that of EFPT-K, were from 2.12 to 2.88 on average.
Reliability The ICC was .87 for the total score, indicating high levels of inter-rater reliability among the 3 occupational therapists. To check the overall consistency among components, we confirmed the internal consistency and the Cronbach alpha coefficient was .77 for the total score supporting high levels of correlation. Subtest Cronbach alpha coefficients were .61 for cooking, .56 for using the telephone, .53 for managing medication, and .63 for paying bills. Pearson correlation coefficients between each executive function components and the total score were as follows: organization, r = .76**; sequencing, r = .91**; safety and judgment, r = .91**; and completion, r = .62**. The correlation between initiation and the total score was not calculated because all of the data of initiation were 0, which indicates that no cue was required.
ARTICLE IN PRESS EXECUTIVE FUNCTION PERFORMANCE TEST FOR KOREANS (EFPT-K) WITH STROKE
Table 3. Performance on EFPT-K, neuropsychological tests, and AMPS (N = 34) Tests
Mean (SD)
Range
EFPT-K total score EFPT-K tasks Cooking Telephone Medication Paying bills EFPT-K EF components Initiation Organization Sequencing Safety and judgment Completion Neuropsychological tests Stroop C Stroop CW Trails A Trails B Digits forward Digits backward AMPS component Initiation Organization Sequencing Notice/response Termination
21.38 (11.67)
7-47
6.47 (3.43) 4.00 (3.38) 3.56 (3.26) 7.35 (3.71)
1-15 0-11 0-12 2-16
0 (0) 5.09 (3.09) 9.00 (4.13) 6.06 (4.38) 1.24 (2.46)
0 0-16 1-18 0-15 0-11
30.62 (17.01) 63.88 (28.96) 71.06 (58.01) 176.35 (94.12) 7.62 (4.02) 5.21 (3.66)
3-74 2-112 19-360 56-300 0-22 0-20
2.88 (1.15) 2.59 (.93) 2.12 (.88) 2.94 (.95) 2.94 (.95)
1-4 1-4 1-4 2-4 2-4
Abbreviations: AMPS, Assessment of Motor and Process Skills; EF, Executive Function; EFPT-K, Executive Function Performance Test for Koreans; Stroop C, Stroop color test; Stroop CW, Stroop color–word test; SD, standard deviation.
5
moderate correlations with the Stroop color test (rho = −.43*) and the Stroop color–word test (rho = −.41*). The EFPT-K total score had a significantly moderate correlation with trails A (rho = .65**) and trails B (rho = .68**). The EFPT-K total score and the Digit Span Test showed significant low to moderate correlations of −.34* in the digits forward score but did not show a statistically significant correlation with the digits backward score. All of the EFPT-K tasks and trails A and B were significantly correlated from .44** to .76** and each EFPT-K component was significantly correlated with trails A and B, which were from .42* to .69**. Because all participants scored 0 in initiation subscores, the scores could not be analyzed. Correlations between EFPT-K and AMPS subscores are presented in Table 5. Compared with the EFPT-K total score, AMPS notice/response and AMPS termination scores suggested significant moderate correlations of −.45** and −.53**, respectively. All EFPT-K tasks but the EFPT-K telephone task were significantly low to moderately correlated with AMPS notice/response (rho = −.37*~−.45**), and all EFPT-K tasks were low to moderately correlated with the AMPS terminates score (rho = −.40*~−.50**). Among the correlations between executive function components of EFPT-K and AMPS components, AMPS notice/response was moderately correlated with EFPT-K safety and judgment (rho = −.58**) and AMPS termination was low to moderately correlated with EFPT-K sequencing and safety and judgment (rho = −.40*~−.52**). EFPT-K initiation was excluded from the analysis result because all of the participants scored 0 in this specific subscore.
Discussion
Criterion Validity We compared the scores of the EFPT-K and neuropsychological tests to examine the criterion validity in Table 4. The total score of EFPT-K showed significantly low to
In the present study, we developed a culturally appropriate EFPT-K and standardized it by applying it to people with stroke and verified inter-rater reliability, internal
Table 4. Spearman’s rank correlation between EFPT-K total score and neuropsychological and function tests (N = 34) Variables
Stroop C
Stroop CW
Trails A
Trails B
Digits forward
Digits backward
EFPT-K total score EFPT-K tasks Cooking Telephone Medication Paying bills EFPT-K EF components Organization Sequencing Safety and judgment Completion
−.43*
−.41*
.65**
.68**
−.34*
−.25
−.42* −.23 −.42* −.48**
−.35* −.25 −.41* −.46**
.46** .46** .51** .73**
.44** .56** .61** .76**
−.43** −.21 −.19 −.33
−.25 −.01 −.26 −.34*
−.39* −.47** −.36* −.38*
−.37* −.42* −.32 −.47**
.69** .65** .51** .42*
.63** .64** .58** .55**
−.32 −.31 −.20 −.40*
−.27 −.29 −.10 −.35*
Abbreviations: EF, Executive Function; EFPT-K, Executive Function Performance Test for Koreans; Stroop C, Stroop color test; Stroop CW, Stroop color–word test. *P < .05; **P < .01.
ARTICLE IN PRESS H. KIM ET AL.
6
Table 5. Spearman’s rank correlation between EFPT-K scores and AMPS component scores (N = 34)
Variables EFPT-K total score EFPT-K tasks Cooking Telephone Medication Paying bills EFPT-K EF components Organization Sequencing Safety and judgment Completion
AMPS, Initiation
AMPS, Organization
AMPS, Sequencing
AMPS, Notice/ response
AMPS, Termination
−.31
−.31
−.20
−.45**
−.53**
−.33 −.19 −.26 −.28
−.09 .00 −.02 −.03
−.03 −.17 −.18 −.28
−.43* −.30 −.45** −.37*
−.42* −.43* −.40* −.50**
−.30 −.26 −.39* .01
−.00 −.21 .04 .16
−.31 −.13 −.08 −.20
−.25 −.39* −.58** −.04
−.35* −.40* −.52** −.32
Abbreviations: AMPS, Assessment of Motor and Process Skills; EF, Executive Function; EFPT-K, Executive Function Performance Test for Koreans. *P < .05; **P < .01.
consistency, and criterion validity of EFPT-K. EFPT-K was a reliable and valid tool to evaluate executive functions of stroke patients. This result accords with that of preceding research of EFPT on people with multiple sclerosis, schizophrenia, or mild to moderate stroke.4,14,15,25 Translating not only the language but also the process to verify the translated EFPT is necessary to introduce EFPT in Korea.26 Thus, back-translation was conducted to investigate clinical components and appropriateness of terminology, and discordances were discussed by authors to reconcile them. Performance methods of EFPT tasks differ greatly depending on one’s culture, and modifying and verifying the tasks so that one’s performance is not affected are crucial.15 Hence, we created an expert committee to discuss the cultural difference and verified the appropriateness of the translation of tasks, procedure, and result interpretation of EFPT-K. The expert committee was a professional group composed of occupational therapists and the goal of the discussion was to minimize the cultural differences and analytical errors. When 5 people with stroke were tested by 3 different therapists, the inter-rater reliability of the EFPT-K total score had an ICC of .87, which indicated a high level of consistency. This result was slightly lower than that of a previous study, which had an ICC of .91, where 3 therapists assessed 5 healthy controls and 5 people with stroke, but was still high enough to carry on the research.4 Also the internal consistency of the items within EFPT-K was .77 for the total score, which is considered high internal consistency and above minimal qualifications for the all tasks (α = .53 ~ .63). In the prior study, the internal consistencies were all higher than that of our study including the total score (α = .94).4 However, when the calculated alpha is over .90, the test is considered an overinspection because several items on the test overlap each other to excessively increase the alpha value, making the alpha
value between .70 and .90 more desirable than excessively higher ones.30 In the correlation between the total score and executive function components, sequencing and safety and judgment showed a higher correlation, and organization and completion showed a lower correlation than that of a previous study, but all of them showed moderate to high correlations.4 Confirming the criterion validity, EFPT-K showed significant correlations with all neuropsychological tests that are known to evaluate executive functions with the exception of digits backward could support EFPT-K as a tool to evaluate executive functions. Although activities of EFPT-K and the original EFPT are not identical but similar that fits the context, we compared our results on EFPT-K with that of the original EFPT. A previous study suggested that EFPT has a significant correlation with other neuropsychological tests but Trail Making Test (TMT)-A and digit forward, indicating that these 2 tests were not correlated because they are far from testing executive functions.4 TMT-A is known to assess processing speed and TMT-B is known to test cognitive flexibility, but the performance time of TMT-A and TMT-B is correlated with each other.27 Also Tamez et al refuted the general opinion of previous studies suggesting TMT-B and digit backward are more sensitive to executive dysfunction than TMT-A and digit forward and claimed TMT-A is correlated with the severity of stroke as much as TMT-B.28 Certain items of the process skills of AMPS were correlated with EFPT-K scores. Especially, notice/response and termination of AMPS were correlated with the total score and most tasks and components of EFPT-K; initiation of AMPS was correlated with safety and judgment of EFPT-K. In the study of Robinson and Fisher, the total score of process skills of AMPS was moderately correlated with cognitive function assessment tools, which include test abstract thinking.29 Also, Cederfeldt et al iden-
ARTICLE IN PRESS EXECUTIVE FUNCTION PERFORMANCE TEST FOR KOREANS (EFPT-K) WITH STROKE
tified the correlation between the total score of AMPS process skills and EFPT.14 Our study additionally confirmed that the EFPT-K is related to the ability to initiate, recognize danger, and complete tasks among executive functions. One of the limitations of the present study is that only a small number of people participated and the validity was only analyzed with a group of stroke patients without a healthy control. Another is that all participants with stroke scored 0 on the initiation component, which indicates that no help was needed with initiating a task, so we were not able to calculate the correlation of the initiation unlike previous studies.4,14 Scoring 0 point on the initiation component means the participant was able to independently approach the table where all equipment and materials were set up when the therapist asks to start.13 In previous studies with stroke patients, the initiation components were scored from .83 to 3.43 on average, the lowest of all executive function components, thus the least help was needed for initiation for patients with stroke among all executive function components.4,14 Yet, the reason why participants in our study showed flawless results for the initiation component could be the fact that the average age of our participants was 54.41, which is 10-18 years younger than those of 2 previous studies. Most of our participants were in the chronic phase of recovery whose average time since the onset was 4 years; and although there were more inpatients than outpatients, most participants had a lot of experience living in the community because of Korea’s distinct medical situations of most people experiencing a stroke constantly, with frequent hospitalization and discharge. Later studies should test people with diverse diagnoses that cause executive dysfunctions such as dementia, brain injury, or psychiatric disorders using EFPT-K. Also people with stroke should be divided into a younger group and an elderly group; chronic and acute phase; and community dwellings and inpatients to compare their EFPT-K results, and the necessity to include the initiation component when testing people with stroke should be discussed.
Conclusions The present study established a foundation to use a performance-based executive function test, EFPT, in Asian culture where usage was delayed because of the cultural gap. Although our study included only people with stroke among many other neurological disorders accompanying executive dysfunctions and only a small number of subjects were included, we hope to be a preliminary study to apply the Executive Function Performance Test for Koreans to people with executive dysfunctions due to other diverse diagnoses and suggest an example of modification to practitioners in other Asian cultures who wish to adapt an executive function test.
7
References 1. Poulin V, Korner-Bitensky N, Dawson DR, et al. Efficacy of executive function interventions after stroke: a systematic review. Top Stroke Rehabil 2012;19:158-171. 2. Zinn S, Bosworth HB, Hoenig HM, et al. Executive function deficits in acute stroke. Arch Phys Med Rehabil 2007;88:173-180. 3. Alvarez JA, Emory E. Executive function and the frontal lobes: a meta-analytic review. Neuropsychol Rev 2006;16:17-42. 4. Baum CM, Connor LT, Morrison T, et al. Reliability, validity, and clinical utility of the Executive Function Performance Test: a measure of executive function in a sample of people with stroke. Am J Occup Ther 2008;62:446455. 5. Rasquin S, Lodder J, Ponds RW, et al. Cognitive functioning after stroke: a one-year follow-up study. Dement Geriatr Cogn Disord 2004;18:138-144. 6. McDowd JM, Filion DL, Pohl PS, et al. Attentional abilities and functional outcomes following stroke. J Gerontol B Psychol Sci Soc Sci 2003;58:P45-P53. 7. Ownsworth T, Shum D. Relationship between executive functions and productivity outcomes following stroke. Disabil Rehabil 2008;30:531-540. 8. Dawson DR, Anderson ND, Burgess P, et al. Further development of the Multiple Errands Test: standardized scoring, reliability, and ecological validity for the Baycrest version. Arch Phys Med Rehabil 2009;90:S41-S51. 9. Raphael-Greenfield E. Assessing executive and community functioning among homeless persons with substance use disorders using the executive function performance test. Occup Ther Int 2012;19:135-143. 10. Wolf TJ, Stift S, Connor LT, et al. Feasibility of using the EFPT to detect executive function deficits at the acute stage of stroke. Work 2010;36:405-412. 11. Bennett PC, Ong B, Ponsford J. Assessment of executive dysfunction following traumatic brain injury: comparison of the BADS with other clinical neuropsychological measures. J Int Neuropsychol Soc 2005;11:606-613. 12. Wood RL, Liossi C. The ecological validity of executive tests in a severely brain injured sample. Arch Clin Neuropsychol 2006;21:429-437. 13. Baum CM, Morrison T, Hahn M, et al. Test manual: executive function performance test. St. Louis (MO): Washington University, 2003. 14. Cederfeldt M, Widell Y, Andersson EE, et al. Concurrent validity of the Executive Function Performance Test in people with mild stroke. Br J Occup Ther 2011;74:443-449. 15. Katz N, Tadmor I, Felzen B, et al. Validity of the Executive Function Performance Test in individuals with schizophrenia. OTJR (Thorofare N J) 2007;27:44-51. 16. Ahn HJ, Chin J, Park A, et al. Seoul Neuropsychological Screening Battery-dementia version (SNSB-D): a useful tool for assessing and monitoring cognitive impairments in dementia patients. J Korean Med Sci 2010;25:1071-1076. 17. Seo EH, Lee DY, Kim SG, et al. Normative study of the Stroop Color and Word Test in an educationally diverse elderly population. Int J Geriatr Psychiatry 2008;23:10201027. 18. Reitan RM. Trail making test results for normal and braindamaged children. Percept Mot Skills 1971;33:575-581. 19. Stuss DT, Bisschop SM, Alexander MP, et al. The Trail Making Test: a study in focal lesion patients. Psychol Assess 2001;13:230. 20. Ryan JJ, Lopez SJ, Paolo AM. Digit span performance of persons 75-96 years of age: base rates and associations
ARTICLE IN PRESS H. KIM ET AL.
8
21.
22. 23.
24.
25.
with selected demographic variables. Psychol Assess 1996;8:324. GrÉGoire J, Van der Linden M. Effect of age on forward and backward digit spans. Aging Neuropsychol Cogn 1997;4:140-149. Fisher AG. Assessment of motor and process skills. 5th ed. Fort Collins (CO): Three Star Press, 2003. Kang Y, Na DL, Hahn S. A validity study on the Korean Mini-Mental State Examination (K-MMSE) in dementia patients. J Korean Neurol Assoc 1997;15:300-308. Shah S, Vanclay F, Cooper B. Improving the sensitivity of the Barthel Index for stroke rehabilitation. Clin Epidemiol 1989;42:703-709. Goverover Y, Kalmar J, Gaudino-Goering E, et al. The relation between subjective and objective measures of everyday life activities in persons with multiple sclerosis. Arch Phys Med Rehabil 2005;86:2303-2308.
26. Mannion AF, Boneschi M, Teli M, et al. Reliability and validity of the cross-culturally adapted Italian version of the Core Outcome Measures Index. Eur Spine J 2012;21:737749. 27. Kopp B, Rösser N, Tabeling S, et al. Errors on the trail making test are associated with right hemispheric frontal lobe damage in stroke patients. Behav Neurol 2015;2015: 309235. 28. Tamez E, Myerson J, Morris L, et al. Assessing executive abilities following acute stroke with the trail making test and digit span. Behav Neurol 2011;24:177-185. 29. Robinson SE, Fisher AG. A study to examine the relationship of the Assessment of Motor and Process Skills (AMPS) to other tests of cognition and function. Br J Occup Ther 1996;59:260-263. 30. Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ 2011;2:53.
Reliability and Validity of Culturally Adapted Executive Function Performance Test for Koreans with Stroke Hee Kim,
OT, PhD,*
Yu-Na Lee, OT, MS,† Eun-Mi Jo, Eun-Young Lee, OT, BHSc§
OT, MS,‡
and
Background: Executive Function Performance Test was unusable in Asia because of cultural differences, although evaluating the effect that executive function has on real life is essential to people with stroke. The aims of this study were to report the development and standardization of culturally appropriate Executive Function Performance Test for Koreans (EFPT-K) and to verify its reliability and validity in the stroke population. Methods: EFPT-K was developed by going through the process of translation, back-translation, and an expert committee’s conference on cultural adaptation. Inter-rater reliability was examined and 34 people with stroke were recruited to test the internal consistency and criterion validity of EFPT-K. To verify the criterion validity, EFPT-K scores were compared with those of the Trail-Making Test, the Korean Color–Word Stroop Test, the Digit Span Test, and the Assessment of Motor and Process Skills (AMPS). Results: Interrater reliability and internal consistency of the total score of EFPT-K supported high levels of reliability. In the criterion validity of EFPT-K, all neuropsychological tests but digits backward showed a correlation with the total score of EFPT-K, and AMPS components of initiation, notice/response, and termination showed a moderate correlation with the EFPT-K score. Conclusions: EFPT-K is a useful tool to evaluate the executive functioning of patients with stroke in real-life tasks and could be used as a sample in other Asian cultures where thorough evaluation of executive function in the performance of daily life is needed. Key Words: Executive function—performance—criterion validity—cultural adaptation—stroke. © 2017 National Stroke Association. Published by Elsevier Inc. All rights reserved.
Introduction Executive function disorder affects 19%-75% of patients with stroke according to how it is defined and assessed in each literature.1,2 As a higher cognitive function, From the *Department of Occupational Therapy, Konyang University, Daejeon, Republic of Korea; †Department of Occupational Therapy, Kyungbok University, Pocheon, Republic of Korea; ‡Department of Occupational Therapy, Sowoon Psychiatry Clinic, Seongnam, Republic of Korea; and §Department of Occupational Therapy, Hansol Convalescent Hospital, Jeonju, Republic of Korea. Received October 25, 2016; revision received November 29, 2016; accepted December 19, 2016. Address correspondence to Yu-Na Lee, OT, MS, Department of Occupational Therapy, Kyungbok University, 154, Sinpyeong-ro, Sinbuk-myeon, Pocheon-si, Gyeonggi-do, Republic of Korea. E-mail: [email protected]. 1052-3057/$ - see front matter © 2017 National Stroke Association. Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jstrokecerebrovasdis.2016.12.013
executive function controls and regulates lower ones, and integrates purposeful and goal-directed behavior processes.3,4 Executive function disorder lasts chronically even after 6 months from the onset of a stroke,5 diminishes the effects of rehabilitation and functional recovery,2 and limits returning to the workplace and social participation.6,7 Standardized executive function tests are essential to screen executive dysfunctions after stroke, to plan rehabilitation interventions considering the dysfunctions, and to evaluate the effect of cognitive rehabilitation in real life.1 Until now, paper-based neuropsychological tests such as the Stroop Color–Word Interference Test and the Trail-Making Test (TMT) were primarily used to evaluate executive functions. Besides, performance-based executive function tests such as the Allen Cognitive Levels Screening Battery (ACLS), Assessment of Motor and Process Skills (AMPS), Kitchen Task Assessment, and Behavioral Assessment of the Dysexecutive Syndrome (BADS) are used.
Journal of Stroke and Cerebrovascular Diseases, Vol. ■■, No. ■■ (■■), 2016: pp ■■–■■
1
ARTICLE IN PRESS H. KIM ET AL.
2
Nevertheless, paper-based neuropsychological tests cannot predict the performance of executive functions in naturalistic environment or real life.8 In particular, many neuropsychological tests are not sensitive to multitasking and have limitations of low ecological validity, which means the inability to evaluate the effect that executive dysfunctions have on daily activities.3,4,9,10 Among the performance-based tests, ACLS and AMPS are the most difficult to use when clearly evaluating the evaluator’s capability of assistance because these tests do not reflect the amount of help on the score when the evaluator provides gradual assistance, and Kitchen Task Assessment requires performance and assessment of only 1 task, which limits assessment of diverse aspects of the executive function.4 BADS is not usable in most clinical settings not only because preparing and setting up for complex 6 tasks are difficult but also because the ecological validity is low and could only be used in a certain cultural context.11,12 The Executive Function Performance Test (EFPT) was developed to overcome such limitations.3,4 EFPT is an evaluation tool that tests the degree of executive function deficits and one’s ability to independently function based on one’s performance on 4 daily tasks: cooking oatmeal, using a telephone, taking medications, and paying bills.4,13 Several studies support the use of original EFPT by verifying the reliability and validity when the test is performed on people with acute and chronic stroke and schizophrenia.4,14,15 Although EFPT was tested with reliability and validity, and consistently used in Western culture such as the United States, Israel, Sweden from the 2000s,4,14,15 original EFPT was not usable in Asia due to the cultural differences in instrumental activities of daily living (ADLs) that demand executive functions and how they perform the tasks. Using the 4 tasks from the original EFPT is impracticable as these tasks are culturally unsuitable, not used in Korean society, or used differently. Thus, not only translations but also replacing the tasks with equivalent levels of culturally fit tasks are necessary, and testing the reliability and validity is crucial to apply EFPT in Korea. The purpose of the present study was to report the development and standardization process of the culturally appropriate Executive Function Performance Test for Koreans (EFPT-K) and to verify its inter-rater reliability, internal consistency, and criterion validity.
Methods Translation and Cultural Adaptation Developing the EFPT-K was processed with the following 4 steps (Fig 1). The first step was to translate the original EFPT in Korean and back-translate it into English to examine the appropriateness of EFPT-K. After getting approved by the developer of the original EFPT, an occupational therapist (first author) bilingual in Korean and English translated the manual and evaluation form of the original EFPT
Translation and back-translation of EFPT
Cultural adaptation of EFPT tasks by expert committee
Survey on content validity and utility of EFPT-K targeting occupational therapists
Reliability and validity of EFPT-K to people with stroke Figure 1. Development process of EFPT-K. Abbreviation: EFPT-K, Executive Function Performance Test for Koreans.
obtained from the Cognitive Rehabilitation Research Group’s Website (http://crrg.wustl.edu/outcome_assessment.html) into Korean (V1). A bilingual undergraduate student majoring in occupational therapy (BT1) and a bilingual professor who majored in clinical psychology (BT2) each back-translated the V1 into English. Both back-translators were not familiar with EFPT, were blinded to the original one, and worked independently on back-translation. By comparing the two back-translations and original EFPT, the authors verified the clinical components and appropriateness of terminology. Discrepancies were discussed to find agreements to complete the EFPT translated in Korean (V2). Second, an expert committee was formed consisting of the translator, 1 occupational therapy professor expert in cognitive rehabilitation, and 3 occupational therapists with more than 5 years of clinical experience in cognitive rehabilitation. After discussing the cultural appropriateness of EFPT, the committee selected unsuitable tasks and processed to replace the tasks as shown in Table 1. Necessary materials, a manual, and an evaluation form were modified for the adapted tasks to make the prefinal version of EFPT-K.
Table 1. Tasks replaced by culturally appropriate ones in EFPT-K EFPT tasks
EFPT-K tasks
Making instant oatmeal Ordering groceries by phone Taking prescribed medications Paying bills by check
Making instant rice porridge Ordering delivery food by phone Taking prescribed vitamins Paying bills by transferring from bank account
Abbreviations: EFPT, Executive Function Performance Test; EFPT-K, Executive Function Performance Test for Koreans.
ARTICLE IN PRESS EXECUTIVE FUNCTION PERFORMANCE TEST FOR KOREANS (EFPT-K) WITH STROKE
Third, we examined the content validity and utility of the EFPT-K by conducting a survey targeting a professional group. The professional group consisted of 29 occupational therapy professors and occupational therapists with more than 3 years of clinical experience in the cognitive rehabilitation of stroke patients. The results of the survey are written in another article of the authors in Korean. After the survey, the final version of EFPT-K was completed. In the last step, the result of which was recorded in the present paper, EFPT-K was tested on people with stroke to verify its reliability and validity.
Study Design and Participants The study design was cross-sectional, performancebased, and psychometric. Study participants were 34 people who experienced a stroke and were recruited from the cities of Wonju and Jeonju in the Republic of Korea during October 2012 and February 2013. To recruit the participants, hospitals and welfare centers of the regions were contacted, and the purpose of the research was explained and approved by these hospitals and welfare centers. The purpose of the research and the rights of the participants were explained to each participant, and all participants signed informed consent forms. Inclusion criteria were (1) a diagnosis of stroke by a physician, (2) more than 6 months after stroke onset and at least 20 years of age, (3) a Mini-Mental State Examination— Korean (MMSE-K) score of 24 or higher, and (4) a Korean Modified Barthel Index (K-MBI) score of 62 or higher. Exclusion criteria were history of mental retardation before the stroke onset and other comorbid neurological or mental illness that can affect executive functioning. Among 34 participants, 24 were male and 10 were female, and their average age was 54.41. Twenty of the participants were inpatients of hospitals with rehabilitation setting and the other 14 were outpatients. Further characteristics of the study participants are presented in Table 2.
Measurements EFPT-K The EFPT, an instrument to evaluate one’s amount of executive dysfunctions and ability to perform independence, is composed of essential tasks to maintain self and independent living such as simple cooking, using a telephone, medication management, and paying bills.13 While a participant performs each task, an examiner scores 5 components of executive function, which are initiation, organization, sequencing, judgment and safety, and completion. For each component, the examiner scores the highest level of cueing the participant needs from 6 levels of the cueing system (0 = no cue required, 1 = verbal guidance,
3
Table 2. Demographic characteristics (N = 34) Characteristic Sex Male Female Patient type Inpatient Outpatient
Age (years) Education (years) Duration from onset (months) MMSE-K score K-MBI score
Participants
Percentage
24 10
70.6 29.4
20 14
58.8 41.2
Mean (SD)
Range
54.41 (12.22) 10.82 (3.46) 49.71 (57.01)
20-76 0-16 7-276
27.21 (2.04) 87.41 (11.43)
24-30 62-100
Abbreviations: K-MBI, Korean Modified Barthel Index; MMSE-K, Mini-Mental State Examination—Korean; SD, standard deviation.
2 = gestured guidance, 3 = direct verbal assistance, 4 = physical assistance, and 5 = therapist does the component instead of the participant). The total score of EFPT ranges from 0 to 100, indicating that a higher score requires more cueing and more severe executive function deficits. We used the EFPT-K in which we replaced the 4 tasks of the original EFPT by culturally adapted ones (Table 1). Korean Color–Word Stroop Test (K-CWST) K-CWST, an instrument composed of words and colors– words, is a neuropsychological test that evaluates the speed of processing information, selective attention, cognitive flexibility, and executive function.16 Participants are asked to correctly say the written words or colors as fast as possible in 45 seconds.17 The number of correctly named words or colors is the score indicating that, to have better executive function, one should achieve a higher score. TMT TMT, a neuropsychological test to evaluate the speed of mental exercise, complex attention, and executive function, is composed of A and B. TMT-A asks the participants to draw a line between numerals in order from 1 to 13, and TMT-B asks the participants to draw lines, alternatively connecting numbers and letters in order, such as 1-A, 2-B, and 3-C. The total times to perform a task and to commit mistakes are scored.18,19 People with higher TMT score need more time to solve the problem, which means they are in a lower executive function level. Digit Span Test Digit Span Test evaluates attention and working memory, and is composed of digit span forward and backward.20,21
ARTICLE IN PRESS H. KIM ET AL.
4
After the examiner verbally presents digits, participants are asked to repeat the digits in the presented sequence in the forward test and in the reverse sequence in the backward test. A higher score indicates better cognitive abilities. AMPS AMPS is an assessment tool to examine one’s motor and process skills when performing ADLs.22 The examinee selects 2 or 3 tasks that are meaningful and performed in real life among 83 tasks, and then 16 motor skills and 20 process skills are rated from 1 (deficit) to 4 (competent) points each. The raw scores are tapped in the AMPS program to be converted into logit scores, and changes in occupational performances of more than .5 logits are considered to be clinically and statistically meaningful ones.22 The test–retest reliabilities of conducting 2 tasks are r = .91 and r = .85, whereas those of performing a task are r = .81 and r = .71. In the present study, we used AMPS to compare the subscores of the specific performance skills such as initiation, organization, sequencing, notice/ response, and termination with that of executive function components of EFPT and analyze their correlation.
occupational therapists independently. Later, the rate of concordance on the observation, grading, and analysis result were examined. To verify the criterion validity of the final version of the EFPT-K, EFPT-K scores were compared with those of the TMT, K-CWST, and Digit Span Test, which are paperbased executive function tests, and AMPS, which is a performance-based test. All 34 participants went through the TMT, K-CWST, Digit Span Test, and AMPS to confirm the correlation between their scores in TMT, K-CWST, Digit Span Test, and AMPS and the EFPT-K scores.
Statistical Analysis All statistical analyses in the present study were conducted using the SPSS for Windows (Version 18.0; SPSS Inc., Chicago, IL). The demographic characteristics of the participants were computed using descriptive statistics. Intraclass correlation coefficients (ICCs) were computed to determine the inter-rater reliability and the Cronbach alpha coefficients were computed to determine internal consistency. Spearman’s rank correlation was conducted to determine criterion validity because EFPT adopts the Likert scale, which is considered ordinal.
MMSE-K MMSE-K is a dementia screening test altered to fit the Korean social context and tests cognitive abilities such as orientation, memory, attention and calculation, language function, understanding, and judgment by interviewing. The total score ranges from 0 to 30, and a score of 24 or higher is interpreted as normal, 20~23 as having mild cognitive disorder, 10~19 as having moderate cognitive disorder, and 9 or lower as having serious cognitive disorder.23 In the present study, MMSE-K was used as a criterion for including participants. K-MBI K-MBI, an evaluation tool to score chronic stroke patients’ independence in basic daily living, evaluates 10 basic ADLs with a 5-point scale and the total score ranges from 0 to 100. The basic ADLs are self-care, bathing, feeding, toileting, stair climbing, dressing, bladder and bowel control, walking (or wheelchair use), and transferring to a chair or a bed. A lower score means more dependence. The test–retest reliability is .89 and the interrater reliability is .95.24 K-MBI was utilized to set an inclusion criterion of the recruitment for the present study.
Process of Testing Reliability and Validity To test the reliability of the final version of EFPT-K, 3 occupational therapists who were well-informed of the methods, process of observation, and analysis result of EFPT partook to score 5 people with stroke. Participants’ performances were videotaped and later scored by 3
Results The averages and ranges of the participants’ scores in the EFPT-K, the neuropsychological test, and the AMPS are shown in Table 3. The mean of the EFPT-K total score was 20.38 (11.67) and managing medication scored the lowest, which was 3.56 (3.26) on average, and paying bills scored the highest, which was 7.35 (3.71) on average among EFPT-K tasks. Among the executive function components of EFPT-K, initiation was the lowest, which scored .0 (.0), and sequencing was the highest, with a mean of 9.00 (4.13). Subscores of AMPS, which were similar to that of EFPT-K, were from 2.12 to 2.88 on average.
Reliability The ICC was .87 for the total score, indicating high levels of inter-rater reliability among the 3 occupational therapists. To check the overall consistency among components, we confirmed the internal consistency and the Cronbach alpha coefficient was .77 for the total score supporting high levels of correlation. Subtest Cronbach alpha coefficients were .61 for cooking, .56 for using the telephone, .53 for managing medication, and .63 for paying bills. Pearson correlation coefficients between each executive function components and the total score were as follows: organization, r = .76**; sequencing, r = .91**; safety and judgment, r = .91**; and completion, r = .62**. The correlation between initiation and the total score was not calculated because all of the data of initiation were 0, which indicates that no cue was required.
ARTICLE IN PRESS EXECUTIVE FUNCTION PERFORMANCE TEST FOR KOREANS (EFPT-K) WITH STROKE
Table 3. Performance on EFPT-K, neuropsychological tests, and AMPS (N = 34) Tests
Mean (SD)
Range
EFPT-K total score EFPT-K tasks Cooking Telephone Medication Paying bills EFPT-K EF components Initiation Organization Sequencing Safety and judgment Completion Neuropsychological tests Stroop C Stroop CW Trails A Trails B Digits forward Digits backward AMPS component Initiation Organization Sequencing Notice/response Termination
21.38 (11.67)
7-47
6.47 (3.43) 4.00 (3.38) 3.56 (3.26) 7.35 (3.71)
1-15 0-11 0-12 2-16
0 (0) 5.09 (3.09) 9.00 (4.13) 6.06 (4.38) 1.24 (2.46)
0 0-16 1-18 0-15 0-11
30.62 (17.01) 63.88 (28.96) 71.06 (58.01) 176.35 (94.12) 7.62 (4.02) 5.21 (3.66)
3-74 2-112 19-360 56-300 0-22 0-20
2.88 (1.15) 2.59 (.93) 2.12 (.88) 2.94 (.95) 2.94 (.95)
1-4 1-4 1-4 2-4 2-4
Abbreviations: AMPS, Assessment of Motor and Process Skills; EF, Executive Function; EFPT-K, Executive Function Performance Test for Koreans; Stroop C, Stroop color test; Stroop CW, Stroop color–word test; SD, standard deviation.
5
moderate correlations with the Stroop color test (rho = −.43*) and the Stroop color–word test (rho = −.41*). The EFPT-K total score had a significantly moderate correlation with trails A (rho = .65**) and trails B (rho = .68**). The EFPT-K total score and the Digit Span Test showed significant low to moderate correlations of −.34* in the digits forward score but did not show a statistically significant correlation with the digits backward score. All of the EFPT-K tasks and trails A and B were significantly correlated from .44** to .76** and each EFPT-K component was significantly correlated with trails A and B, which were from .42* to .69**. Because all participants scored 0 in initiation subscores, the scores could not be analyzed. Correlations between EFPT-K and AMPS subscores are presented in Table 5. Compared with the EFPT-K total score, AMPS notice/response and AMPS termination scores suggested significant moderate correlations of −.45** and −.53**, respectively. All EFPT-K tasks but the EFPT-K telephone task were significantly low to moderately correlated with AMPS notice/response (rho = −.37*~−.45**), and all EFPT-K tasks were low to moderately correlated with the AMPS terminates score (rho = −.40*~−.50**). Among the correlations between executive function components of EFPT-K and AMPS components, AMPS notice/response was moderately correlated with EFPT-K safety and judgment (rho = −.58**) and AMPS termination was low to moderately correlated with EFPT-K sequencing and safety and judgment (rho = −.40*~−.52**). EFPT-K initiation was excluded from the analysis result because all of the participants scored 0 in this specific subscore.
Discussion
Criterion Validity We compared the scores of the EFPT-K and neuropsychological tests to examine the criterion validity in Table 4. The total score of EFPT-K showed significantly low to
In the present study, we developed a culturally appropriate EFPT-K and standardized it by applying it to people with stroke and verified inter-rater reliability, internal
Table 4. Spearman’s rank correlation between EFPT-K total score and neuropsychological and function tests (N = 34) Variables
Stroop C
Stroop CW
Trails A
Trails B
Digits forward
Digits backward
EFPT-K total score EFPT-K tasks Cooking Telephone Medication Paying bills EFPT-K EF components Organization Sequencing Safety and judgment Completion
−.43*
−.41*
.65**
.68**
−.34*
−.25
−.42* −.23 −.42* −.48**
−.35* −.25 −.41* −.46**
.46** .46** .51** .73**
.44** .56** .61** .76**
−.43** −.21 −.19 −.33
−.25 −.01 −.26 −.34*
−.39* −.47** −.36* −.38*
−.37* −.42* −.32 −.47**
.69** .65** .51** .42*
.63** .64** .58** .55**
−.32 −.31 −.20 −.40*
−.27 −.29 −.10 −.35*
Abbreviations: EF, Executive Function; EFPT-K, Executive Function Performance Test for Koreans; Stroop C, Stroop color test; Stroop CW, Stroop color–word test. *P < .05; **P < .01.
ARTICLE IN PRESS H. KIM ET AL.
6
Table 5. Spearman’s rank correlation between EFPT-K scores and AMPS component scores (N = 34)
Variables EFPT-K total score EFPT-K tasks Cooking Telephone Medication Paying bills EFPT-K EF components Organization Sequencing Safety and judgment Completion
AMPS, Initiation
AMPS, Organization
AMPS, Sequencing
AMPS, Notice/ response
AMPS, Termination
−.31
−.31
−.20
−.45**
−.53**
−.33 −.19 −.26 −.28
−.09 .00 −.02 −.03
−.03 −.17 −.18 −.28
−.43* −.30 −.45** −.37*
−.42* −.43* −.40* −.50**
−.30 −.26 −.39* .01
−.00 −.21 .04 .16
−.31 −.13 −.08 −.20
−.25 −.39* −.58** −.04
−.35* −.40* −.52** −.32
Abbreviations: AMPS, Assessment of Motor and Process Skills; EF, Executive Function; EFPT-K, Executive Function Performance Test for Koreans. *P < .05; **P < .01.
consistency, and criterion validity of EFPT-K. EFPT-K was a reliable and valid tool to evaluate executive functions of stroke patients. This result accords with that of preceding research of EFPT on people with multiple sclerosis, schizophrenia, or mild to moderate stroke.4,14,15,25 Translating not only the language but also the process to verify the translated EFPT is necessary to introduce EFPT in Korea.26 Thus, back-translation was conducted to investigate clinical components and appropriateness of terminology, and discordances were discussed by authors to reconcile them. Performance methods of EFPT tasks differ greatly depending on one’s culture, and modifying and verifying the tasks so that one’s performance is not affected are crucial.15 Hence, we created an expert committee to discuss the cultural difference and verified the appropriateness of the translation of tasks, procedure, and result interpretation of EFPT-K. The expert committee was a professional group composed of occupational therapists and the goal of the discussion was to minimize the cultural differences and analytical errors. When 5 people with stroke were tested by 3 different therapists, the inter-rater reliability of the EFPT-K total score had an ICC of .87, which indicated a high level of consistency. This result was slightly lower than that of a previous study, which had an ICC of .91, where 3 therapists assessed 5 healthy controls and 5 people with stroke, but was still high enough to carry on the research.4 Also the internal consistency of the items within EFPT-K was .77 for the total score, which is considered high internal consistency and above minimal qualifications for the all tasks (α = .53 ~ .63). In the prior study, the internal consistencies were all higher than that of our study including the total score (α = .94).4 However, when the calculated alpha is over .90, the test is considered an overinspection because several items on the test overlap each other to excessively increase the alpha value, making the alpha
value between .70 and .90 more desirable than excessively higher ones.30 In the correlation between the total score and executive function components, sequencing and safety and judgment showed a higher correlation, and organization and completion showed a lower correlation than that of a previous study, but all of them showed moderate to high correlations.4 Confirming the criterion validity, EFPT-K showed significant correlations with all neuropsychological tests that are known to evaluate executive functions with the exception of digits backward could support EFPT-K as a tool to evaluate executive functions. Although activities of EFPT-K and the original EFPT are not identical but similar that fits the context, we compared our results on EFPT-K with that of the original EFPT. A previous study suggested that EFPT has a significant correlation with other neuropsychological tests but Trail Making Test (TMT)-A and digit forward, indicating that these 2 tests were not correlated because they are far from testing executive functions.4 TMT-A is known to assess processing speed and TMT-B is known to test cognitive flexibility, but the performance time of TMT-A and TMT-B is correlated with each other.27 Also Tamez et al refuted the general opinion of previous studies suggesting TMT-B and digit backward are more sensitive to executive dysfunction than TMT-A and digit forward and claimed TMT-A is correlated with the severity of stroke as much as TMT-B.28 Certain items of the process skills of AMPS were correlated with EFPT-K scores. Especially, notice/response and termination of AMPS were correlated with the total score and most tasks and components of EFPT-K; initiation of AMPS was correlated with safety and judgment of EFPT-K. In the study of Robinson and Fisher, the total score of process skills of AMPS was moderately correlated with cognitive function assessment tools, which include test abstract thinking.29 Also, Cederfeldt et al iden-
ARTICLE IN PRESS EXECUTIVE FUNCTION PERFORMANCE TEST FOR KOREANS (EFPT-K) WITH STROKE
tified the correlation between the total score of AMPS process skills and EFPT.14 Our study additionally confirmed that the EFPT-K is related to the ability to initiate, recognize danger, and complete tasks among executive functions. One of the limitations of the present study is that only a small number of people participated and the validity was only analyzed with a group of stroke patients without a healthy control. Another is that all participants with stroke scored 0 on the initiation component, which indicates that no help was needed with initiating a task, so we were not able to calculate the correlation of the initiation unlike previous studies.4,14 Scoring 0 point on the initiation component means the participant was able to independently approach the table where all equipment and materials were set up when the therapist asks to start.13 In previous studies with stroke patients, the initiation components were scored from .83 to 3.43 on average, the lowest of all executive function components, thus the least help was needed for initiation for patients with stroke among all executive function components.4,14 Yet, the reason why participants in our study showed flawless results for the initiation component could be the fact that the average age of our participants was 54.41, which is 10-18 years younger than those of 2 previous studies. Most of our participants were in the chronic phase of recovery whose average time since the onset was 4 years; and although there were more inpatients than outpatients, most participants had a lot of experience living in the community because of Korea’s distinct medical situations of most people experiencing a stroke constantly, with frequent hospitalization and discharge. Later studies should test people with diverse diagnoses that cause executive dysfunctions such as dementia, brain injury, or psychiatric disorders using EFPT-K. Also people with stroke should be divided into a younger group and an elderly group; chronic and acute phase; and community dwellings and inpatients to compare their EFPT-K results, and the necessity to include the initiation component when testing people with stroke should be discussed.
Conclusions The present study established a foundation to use a performance-based executive function test, EFPT, in Asian culture where usage was delayed because of the cultural gap. Although our study included only people with stroke among many other neurological disorders accompanying executive dysfunctions and only a small number of subjects were included, we hope to be a preliminary study to apply the Executive Function Performance Test for Koreans to people with executive dysfunctions due to other diverse diagnoses and suggest an example of modification to practitioners in other Asian cultures who wish to adapt an executive function test.
7
References 1. Poulin V, Korner-Bitensky N, Dawson DR, et al. Efficacy of executive function interventions after stroke: a systematic review. Top Stroke Rehabil 2012;19:158-171. 2. Zinn S, Bosworth HB, Hoenig HM, et al. Executive function deficits in acute stroke. Arch Phys Med Rehabil 2007;88:173-180. 3. Alvarez JA, Emory E. Executive function and the frontal lobes: a meta-analytic review. Neuropsychol Rev 2006;16:17-42. 4. Baum CM, Connor LT, Morrison T, et al. Reliability, validity, and clinical utility of the Executive Function Performance Test: a measure of executive function in a sample of people with stroke. Am J Occup Ther 2008;62:446455. 5. Rasquin S, Lodder J, Ponds RW, et al. Cognitive functioning after stroke: a one-year follow-up study. Dement Geriatr Cogn Disord 2004;18:138-144. 6. McDowd JM, Filion DL, Pohl PS, et al. Attentional abilities and functional outcomes following stroke. J Gerontol B Psychol Sci Soc Sci 2003;58:P45-P53. 7. Ownsworth T, Shum D. Relationship between executive functions and productivity outcomes following stroke. Disabil Rehabil 2008;30:531-540. 8. Dawson DR, Anderson ND, Burgess P, et al. Further development of the Multiple Errands Test: standardized scoring, reliability, and ecological validity for the Baycrest version. Arch Phys Med Rehabil 2009;90:S41-S51. 9. Raphael-Greenfield E. Assessing executive and community functioning among homeless persons with substance use disorders using the executive function performance test. Occup Ther Int 2012;19:135-143. 10. Wolf TJ, Stift S, Connor LT, et al. Feasibility of using the EFPT to detect executive function deficits at the acute stage of stroke. Work 2010;36:405-412. 11. Bennett PC, Ong B, Ponsford J. Assessment of executive dysfunction following traumatic brain injury: comparison of the BADS with other clinical neuropsychological measures. J Int Neuropsychol Soc 2005;11:606-613. 12. Wood RL, Liossi C. The ecological validity of executive tests in a severely brain injured sample. Arch Clin Neuropsychol 2006;21:429-437. 13. Baum CM, Morrison T, Hahn M, et al. Test manual: executive function performance test. St. Louis (MO): Washington University, 2003. 14. Cederfeldt M, Widell Y, Andersson EE, et al. Concurrent validity of the Executive Function Performance Test in people with mild stroke. Br J Occup Ther 2011;74:443-449. 15. Katz N, Tadmor I, Felzen B, et al. Validity of the Executive Function Performance Test in individuals with schizophrenia. OTJR (Thorofare N J) 2007;27:44-51. 16. Ahn HJ, Chin J, Park A, et al. Seoul Neuropsychological Screening Battery-dementia version (SNSB-D): a useful tool for assessing and monitoring cognitive impairments in dementia patients. J Korean Med Sci 2010;25:1071-1076. 17. Seo EH, Lee DY, Kim SG, et al. Normative study of the Stroop Color and Word Test in an educationally diverse elderly population. Int J Geriatr Psychiatry 2008;23:10201027. 18. Reitan RM. Trail making test results for normal and braindamaged children. Percept Mot Skills 1971;33:575-581. 19. Stuss DT, Bisschop SM, Alexander MP, et al. The Trail Making Test: a study in focal lesion patients. Psychol Assess 2001;13:230. 20. Ryan JJ, Lopez SJ, Paolo AM. Digit span performance of persons 75-96 years of age: base rates and associations
ARTICLE IN PRESS H. KIM ET AL.
8
21.
22. 23.
24.
25.
with selected demographic variables. Psychol Assess 1996;8:324. GrÉGoire J, Van der Linden M. Effect of age on forward and backward digit spans. Aging Neuropsychol Cogn 1997;4:140-149. Fisher AG. Assessment of motor and process skills. 5th ed. Fort Collins (CO): Three Star Press, 2003. Kang Y, Na DL, Hahn S. A validity study on the Korean Mini-Mental State Examination (K-MMSE) in dementia patients. J Korean Neurol Assoc 1997;15:300-308. Shah S, Vanclay F, Cooper B. Improving the sensitivity of the Barthel Index for stroke rehabilitation. Clin Epidemiol 1989;42:703-709. Goverover Y, Kalmar J, Gaudino-Goering E, et al. The relation between subjective and objective measures of everyday life activities in persons with multiple sclerosis. Arch Phys Med Rehabil 2005;86:2303-2308.
26. Mannion AF, Boneschi M, Teli M, et al. Reliability and validity of the cross-culturally adapted Italian version of the Core Outcome Measures Index. Eur Spine J 2012;21:737749. 27. Kopp B, Rösser N, Tabeling S, et al. Errors on the trail making test are associated with right hemispheric frontal lobe damage in stroke patients. Behav Neurol 2015;2015: 309235. 28. Tamez E, Myerson J, Morris L, et al. Assessing executive abilities following acute stroke with the trail making test and digit span. Behav Neurol 2011;24:177-185. 29. Robinson SE, Fisher AG. A study to examine the relationship of the Assessment of Motor and Process Skills (AMPS) to other tests of cognition and function. Br J Occup Ther 1996;59:260-263. 30. Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ 2011;2:53.