- Email: [email protected]
Smart watch, smarter EDSS: Improving disability assessment in multiple sclerosis clinical practice
Accepted Manuscript Smart watch, smarter EDSS: Improving disability assessment in multiple sclerosis clinical practice
Gloria Dalla Costa, Marta Radaelli, Simona Maida, Francesca Sangalli, Bruno Colombo, Lucia Moiola, Giancarlo Comi, Vittorio Martinelli PII: DOI: Reference:
S0022-510X(17)34425-8 doi:10.1016/j.jns.2017.10.043 JNS 15638
To appear in:
Journal of the Neurological Sciences
Received date: Revised date: Accepted date:
18 April 2017 19 October 2017 26 October 2017
Please cite this article as: Gloria Dalla Costa, Marta Radaelli, Simona Maida, Francesca Sangalli, Bruno Colombo, Lucia Moiola, Giancarlo Comi, Vittorio Martinelli , Smart watch, smarter EDSS: Improving disability assessment in multiple sclerosis clinical practice. The address for the corresponding author was captured as affiliation for all authors. Please check if appropriate. Jns(2017), doi:10.1016/j.jns.2017.10.043
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT Dalla Costa G. Smart watch, smarter EDSS: Improving disability assessment in multiple sclerosis clinical practice
Gloria Dalla Costa 1* , Marta Radaelli 1* , Simona Maida 1 , Francesca Sangalli 1 , Bruno Colombo 1 , Lucia Moiola 1 , Giancarlo Comi 1 Vittorio Martinelli 1 *
Department of Neurology, San Raffaele Hospital, Milan, Italy. These authors equally contributed to this study
SC RI P
Short title: Smart watch, smarter disability assessment in MS practice
T
1
AC
CE
PT
ED
MA
NU
Corresponding author: Vittorio Martinelli, Department of Neurology, San Raffaele Hospital, via Olgettina 48, 20132 Milan, Italy. Phone: 0039 02 26433836 Email: [email protected]
1
ACCEPTED MANUSCRIPT Dalla Costa G.
2
ABSTRACT
Background: Patients’ walking ability is critical for assessing the EDSS, the disability scale commonly used in MS clinical practice. Such assessment is usually based on patients’ estimates or on
T
the measures the neurologists observe during periodic visits
SC RI P
Objectives and Methods: We evaluated the agreement between patients’ and neurologists’ estimates of maximum walking ability and patients’ mean maximum walking ability measured in their daily life through a GPS smartwatch, and assessed limitations of the current methods. Results: Seventy-three patients with a median walking ability of 500 meters (IQR 400-800) were enrolled in the study. The agreement between patients’ estimates and GPS measurements was mod-
NU
est (ICC 0.29, 95% CIs 0.06-0.49) and was influenced by course of the disease, patients’ mood and inaccuracy at estimating long distances. A better reliability was found between neurologists’ and GPS measures (ICC 0.68, 95% CIs 0.53-0.78),, but the variability increased for longer distances
MA
and was influenced by patients’ depressive symptoms, fatigue and course of the disease. Conclusions: This study showed a poor agreement between patients' and neurologists' estimates of
ED
maximum walking ability and patients' mean maximum walking ability measured in their daily life through a GPS smartwatch, with many factors affecting patient’s and neurologists’ estimates of the EDSS. The use of remote measurement technologies may provide a better understanding of the im-
CE
PT
pact of MS in a patient’s life.
AC
Keywords: multiple sclerosis, EDSS, walking distance, GPS tracking technologies, disability, remote monitoring technologies
ACCEPTED MANUSCRIPT Dalla Costa G.
3
1. INTRODUCTION
The assessment of patients’ walking ability is critical for the Kurtzke’s expanded disability status scale score (EDSS), the disability scale commonly used in multiple sclerosis (MS) clinical practice [1]. The scale ranges from 0 (no clinical sign and no disability) to 10 (death caused by the disease),
T
and a 1.0 step change is important, although the difference between grades 5.5, 5.0, 4.5, and 4.0 is
SC RI P
the ability to walk 100, 200, 300, or at least 500 meters respectively. Current common practice measurement for walking activity in MS patients typically involves patient-reported walking ability, with considerable intra and inter-patient variability in accuracy. In fact, walking ability may be influenced by ecological factors and the subject’s emotional and mood characteristics. Moreover, other clinical parameters such as the stride-time variability have been suggested to be relevant in clini-
NU
cal practice [2]. Therefore, a one-time assessment by the treating neurologist during periodic visits may not be representative of the actual severity of the disease. The last decade has seen an explo-
MA
sion in the capability of monitoring different health parameters via sensors in smartphones or wearable devices, which may overcome these limitations by accurately representing the disease status through continuous assessments and increasing quality in clinical decision making and engagement
ED
with patients.
The aim of this study was to evaluate the agreement between patients’ and neurologists’ estimates
PT
of walking ability and patients’ mean walking ability measured in their daily life through a GPS
2. METHODS
CE
smartwatch, and assess limitations of the current methods.
AC
This was a prospective study involving MS patients with restricted ambulation (with or without assistance, EDSS 4.0-6.5), recruited at the MS Center of San Raffaele Hospital, Milan, Italy. Patients estimated their maximum walking distance (PWD) and, consequently, an EDSS score was calculated. Patients’ walking ability was then assessed by a certified EDSS rating physician (GDC, MR), by directly measuring the maximum walking distance (NWD) patients covered walking in an aisle on a surface without obstacles. Then patients were trained to use a GPS smartwatch (Garmin Forerunner 230) for four weeks, to obtain daily objectively measured values of maximum walking distance, and from the available measures the mean maximum walking distance (MWD) was calculated. The Mini Mental State Examination (MMSE), the Modified Fatigue Impact Scale (MFIS), the Beck Depression Inventory (BDI) and the Short Form-36 (SF-36) questionnaires were used as dummy variables (cutoff: median value) to measure cognitive status, fatigue, depression and the general health
ACCEPTED MANUSCRIPT Dalla Costa G.
4
status of the patients, respectively. Patient’s feedback on their overall experience in the study and with the use of the GPS was collected at the end of the study. The intraclass correlation coefficient (ICC) using a two-way mixed effects model [3] was used to calculate agreement between measurements by the neurologists, the patients and the GPS smartwatch. Values for ICC(2,1) measures were interpreted using the guidelines of Portney and Watkins [4]. Sample size calculations were based on optimal and minimal levels of reliability, and a sample size of 72 was sufficient to detect a true reliability of 0.75 from a null value of 0.50, with 90% pow-
T
er using ICC(2,1) [5]. Bland and Altman plot analysis [6] was applied to visually assess individual
SC RI P
agreement between methods. The 95% limits of individual agreement between two methods were calculated as the mean difference between the two methods ± 1.96 standard deviation. Multiple regression analysis was performed to determine the factors influencing the agreement between the methods. This study was approved by the Ethical Committee of San Raffaele Hospital.
NU
3. RESULTS
Seventy-three patients were enrolled in the study (mean age 47.6 years, 50.7% females). Half were
MA
evaluated in the morning by the treating neurologist, the others in the afternoon. Fifty-three patients (72.6%) had a relapsing remitting course of the disease and the others had a progressive phase. Their main clinical and demographic characteristics are shown in Table 1.
ED
The median MWD measured by the smartwatch was 500 meters (IQR 400-800, range 100-1300), corresponding to an EDSS of 4.0. A correspondence between PWD and MWD was found in only
PT
15% of patients. Overall 79.5% patients underestimated their walking ability and 5.4% overestimated it. This corresponded to a correction of the self-reported EDSS scores of >0.5 point in 84.9% of
CE
patients. Therefore, a low ICC between the two methods was observed (0.29, 95% CIs 0.06-0.49). In multiple regression analysis, older patients with a longer disease history, or a progressive form of disease, or who were on second-line therapies, with a high SF36 score tended to overestimate their
AC
ability, while patients with depressive symptoms or with the best MWD underestimated it (Table 2). A better reliability was found between NWD and MWD (ICC 0.68, 95% CIs 0.53-0.78), but the agreement was significantly dependent upon the form of disease and type of ongoing DMD, BDI MFIS and SF36 scores and MWD (Table 2). The plot analysis confirmed these findings (Supplementary data). Overall, most of the patients (93%) gave positive feedback on the study and reported an increased motivation to walk for the whole period they were using the smartwatch.
4. DISCUSSION
ACCEPTED MANUSCRIPT Dalla Costa G.
5
The evaluation of patients’ walking ability is critical for assessment of Kurtzke’s EDSS score, the disability scale commonly used in MS clinical practice. This is particularly true for EDSS scores from 4.0 to 6.0, where the difference between grades 5.5, 5.0, 4.5, and 4.0 is the ability to walk 100, 200, 300, or at least 500 meters respectively. Such assessment is usually based on patients’ selfreported walking ability or on the measures the neurologists observe during periodic visits. The present study evaluated the agreement between patients’ and neurologists’ estimates and the measurements of a GPS smartwatch.
T
A poor agreement between patients’ estimates and GPS objective measurements was observed. Par-
SC RI P
ticularly, depressive symptoms led the patients to underestimate their walking abilities probably because of their negative outlook on their capacities. Furthermore, these patients may not be challenged in their daily life routines and may not be doing enough physical activity, which may lead to faster disability worsening. Also, patients whose objective maximum walking distances was longer underestimated their capacities, and these results are in line with a previous study showing that ar-
NU
tillery observers estimating target distances showed wide variability [7], and in another study, patients as well as doctors were inaccurate at estimating particularly long distances [8].
MA
The agreement between the evaluating neurologist and the GPS measures was better than the patients’ estimates, as the neurologists’ estimates were based on the observed distance the patients were able to walk during the visit to the neurologist. Nevertheless, this was greatly influenced by
ED
the disease status, patients’ mood, and patients’ fatigue during the clinical examination. This is in line with previous findings that showed that the day-to-day variability in walking ability of MS pa-
PT
tients was much higher than for controls [9-10]. This study has some limitations First of all, it involved patients with a moderate disability. Second-
CE
ly patients wearing the smartwatch could have been more motivated to walk than usually. These may prevent the generalization of the study findings to the MS population. Nevertheless, the study
AC
has important clinical implications as it supports the role of remote measurement technologies (RMT) as an innovation which could, in the foreseeable future, provide real-time information on the patient’s walking ability in the patient’s home environment, increasing quality and accuracy in clinical decision making whilst ensuring feasible and appropriate levels of engagement with patients [11]. Finally, overall patient experience was positive and most MS patients reported that the device was useful in increasing motivation to walk for the whole period they were using the smartwatch. Further prospective multi-centric studies are needed to validate our results and optimize future use of RMT, bridging the gap between technological innovation and clinical utility. ACKNOWLEDGEMENTS
ACCEPTED MANUSCRIPT Dalla Costa G.
6
We would like to thank all our patients for their enthusiastic participation, and Acesm Onlus for supporting this study. FUNDING
This research received no specific grant from any funding agency in the public, commercial, or notfor-profit sectors.
T
CONFLICT OF INTEREST STATEMENT
SC RI P
G. Dalla Costa, M Radaelli, S Maida, F Sangalli, B Colombo report no discosures. L. Moiola has received honoraria for speaking in scientific meetings and for advisory board from Biogen TEVA, Genzyme and Merck. G. Comi has received personal compensation for consulting services and/or speaking activities from Novartis, Teva, Sanofi, Genzyme, Merck, Biogen, Excemed, Serono Symposia International Foundation, Roche, Almirall, Receptos, Celgene, Forward Pharma. V. Martinelli
NU
has received honoraria for activities with Biogen, Merck, Bayer, TEVA, Novartis and Sanofi as a
MA
speaker. REFERENCES
1. Kurtzke JF. Rating neurologic impairment in multiple sclerosis: an expanded disability sta-
PT
ED
tus scale (EDSS). Neurology.;33(11):1444-52. 2. Allali G, Laidet M, Hermann FR, Armand S, Elsworth-Edelsten C, Assal F and Lalive PH. Gait variability in multiple sclerosis: a better falls predictor than EDSS in patients with low disability. J Neural Transm 2016 Apr;123(4):447-50. 3. British Standards Institution. Precision of test methods 1: Guide for the determination and
AC
CE
reproducibility for a standard test method (BS 5497, part 1). London: BSI, 1979. 4. Portney LG, Watkins MP. Foundations of clinical research: Applications to practice. Upper Saddle River, NJ: Pearson Prentice Hall, 2009. 5. Zou GY. Sample size formulas for estimating intraclass correlation coefficients with precision and assurance. Statistics in Medicine, 31(29), 3972–3981, 2012 6. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1:307–310, 1986. 7. Fine BJ, Kobrick JL. Individual differences in distance estimation: Comparison of judgments in the field with those from projected slides of the same scenes. Percept Motor Skills:57: 3–14, 1983. 8. Sharrack B, Hughes RAC. Reliability of distance estimation by doctors and patients: Cross sectional study. BMJ: 315, 1997. 9. Albrecht H, Wötzel C, Erasmus LP, Kleinpeter M, König N, Pöllmann W. Day-to-day variability of maximum walking distance in MS patients can mislead to relevant changes in the
ACCEPTED MANUSCRIPT Dalla Costa G. Expanded Disability Status Scale (EDSS): Average walking speed is a more constant parameter. MultScler:7:105–109, 2001. 10. Morris ME, Cantwell C, Vowels L, Dodd K. Changes in gait and fatigue from morning to afternoon in people with multiple sclerosis. J NeurolNeurosurgPsychiatry:72:361–365, 2002.
AC
CE
PT
ED
MA
NU
SC RI P
T
11. Dalla Costa G, Maida S, Sorrentino P, Braunstein M, Comi G, Martinelli V. Opinion & special articles: professionalism in neurology. Maintaining patient rapport in a world of EMR. Neurology. 2014;83(2):e12–5.
7
ACCEPTED MANUSCRIPT Dalla Costa G.
Table 1. Clinical and Demographic Characteristics of the Seventy-three Patients Enrolled in the Study
Gender Females, No. (%) Males, No. (%)
37 (50.7) 36 (49.3)
Disease duration, mean (SD)
14.4 ± 7.3
Type of disease Relapsing-remitting, No. (%) Progressive form, No. (%)
53 (72.6) 20 (27.4)
Type of DMD First-line therapy, No. (%) Second-line therapy, No. (%)
61 (83.6) 12 (16.4)
Baseline assessment time Morning, No. (%) Afternoon, No. (%)
MA
MMSE score, mean (SD)
27.2 ± 2.1
ED
41 (56.2) 32 (43.8)
SF-36 score, mean (SD)
AC
CE
BDI score, mean (SD)
36.1 ± 15.5
PT
MFIS score, mean (SD)
SC RI P
47.6 ± 7.9
NU
Age, mean (SD)
T
Characteristic
95.2 ± 11.3 13.0 ± 8.3
8
ACCEPTED MANUSCRIPT Dalla Costa G.
9
Table 2. Predictors (independent variables) of Agreement between Patients’, Neurologists’ and GPS Smartwatch Estimates of Walking Ability (dependent variables of Model A and B respectively) M ultivariate analysis
Coefficient
95% CIs
p
Coefficient
95% CIs
p
M ale sex
11.3
-122.2 - 144.9
0.87
-
-
-
Age at baseline
-9.4
-18.8 - 0.02
0.07
4.9
0.7 - 9.1
0.02
Baseline assessment in the afternoon
22.5
-5.7 - 54.2
0.32
-
-
-
Disease duration
-3.4
-12.8 - 5.9
0.47
T
Univariate analysis
3.2 - 11.9
< 0.01
Progressive disease
-39.5
-188.9 - 109.9
0.60
80.1
9.6 - 150.5
0.03
Second-line therapy
242.9
72.2 - 413.6
<0.01
206.1
127.8 - 284.4
< 0.01
BDI score > 13
-9.5
-17.0 - -1.9
0.01
-130.0.4
-187.4 - -72.6
< 0.01
M FIS score > 36
21.6
-111-9 - 155.0
0.75
-
-
-
SF36 score > 95
-0.2
-6.9 - 6.6
0.96
169.6
113.5 - 225.8
< 0.01
GPS mean maximum walking ability
-0.8
-1.0 - -0.7
<0.01
-0.8
-0.9 - -0.7
< 0.01
Patient vs GPS Smartwatch (M odel A)
SC RI P
-0.07
-110.5 - 110.4
1.0
-
-
-
Age at baseline
-9.7
-17.3 - -2.0
0.01
-
-
-
Baseline assessment in the afternoon
91.7
-17.9 - 201.2
0.10
-
-
-
-8.9
-16.3 - -1.4
0.02
-
-
-
41.7
-81.8 - 165.1
0.50
121.6
32.2 - 132.9
< 0.01
172.8
29.6 - 316.1
0.02
123.8
39.4 - 156.7
0.01
-178.2
.280.3 - -76.1
<0.01
-128.9
- 132.0 - -35.5
< 0.01
-93.1
-201.3 - 15.1
0.09
-70.6
-129.2 - 8.1
< 0.01
71.3
-37.8 - 180.5
0.20
129.2
49.4 - 100.9
< 0.01
-0.5
-0.7 - -0.4
<0.01
-0.5
-0.6 - -0.35
< 0.01
PT
Progressive disease
M FIS score > 36
AC
SF36 score > 95
CE
Second-line therapy BDI score > 13
ED
M ale sex
Disease duration
GPS mean maximum walking ability 1
NU
MA
Neurologist vs GPS Smartwatch (M odel B)
7.5
Univariate and multivariate regression analysis was performed to determine the factors influencing the difference between patients’ and neurologists’ estimates of walking ability and patients’ mean walking ability measured in their daily life through a GPS (outcomes of model A and B respectively). Backward selection technique based on the Akaike information criterion was used to select the variables to be included in the final models (Multivariate model A Adjusted R2 0.87 and multivariate model B Adjusted R2 0.67; intraclass correlation coefficient between patients’ estimates and GPS 0.29, intraclass correlation coefficient between neurologists’ estimates and GPS 0.68)
ACCEPTED MANUSCRIPT Dalla Costa G.
10
AC
CE
PT
ED
MA
NU
SC RI P
T
Highlights: The evaluation of patients’ walking distance is critical for disability assessment in MS; This assessment is usually based on patients’or neurologists’ estimate during periodic visit; A poor agreement between these estimates and an objective GPS estimate was observed; Remote monitoring technologies are better at representing the disease status in daily life;
Gloria Dalla Costa, Marta Radaelli, Simona Maida, Francesca Sangalli, Bruno Colombo, Lucia Moiola, Giancarlo Comi, Vittorio Martinelli PII: DOI: Reference:
S0022-510X(17)34425-8 doi:10.1016/j.jns.2017.10.043 JNS 15638
To appear in:
Journal of the Neurological Sciences
Received date: Revised date: Accepted date:
18 April 2017 19 October 2017 26 October 2017
Please cite this article as: Gloria Dalla Costa, Marta Radaelli, Simona Maida, Francesca Sangalli, Bruno Colombo, Lucia Moiola, Giancarlo Comi, Vittorio Martinelli , Smart watch, smarter EDSS: Improving disability assessment in multiple sclerosis clinical practice. The address for the corresponding author was captured as affiliation for all authors. Please check if appropriate. Jns(2017), doi:10.1016/j.jns.2017.10.043
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT Dalla Costa G. Smart watch, smarter EDSS: Improving disability assessment in multiple sclerosis clinical practice
Gloria Dalla Costa 1* , Marta Radaelli 1* , Simona Maida 1 , Francesca Sangalli 1 , Bruno Colombo 1 , Lucia Moiola 1 , Giancarlo Comi 1 Vittorio Martinelli 1 *
Department of Neurology, San Raffaele Hospital, Milan, Italy. These authors equally contributed to this study
SC RI P
Short title: Smart watch, smarter disability assessment in MS practice
T
1
AC
CE
PT
ED
MA
NU
Corresponding author: Vittorio Martinelli, Department of Neurology, San Raffaele Hospital, via Olgettina 48, 20132 Milan, Italy. Phone: 0039 02 26433836 Email: [email protected]
1
ACCEPTED MANUSCRIPT Dalla Costa G.
2
ABSTRACT
Background: Patients’ walking ability is critical for assessing the EDSS, the disability scale commonly used in MS clinical practice. Such assessment is usually based on patients’ estimates or on
T
the measures the neurologists observe during periodic visits
SC RI P
Objectives and Methods: We evaluated the agreement between patients’ and neurologists’ estimates of maximum walking ability and patients’ mean maximum walking ability measured in their daily life through a GPS smartwatch, and assessed limitations of the current methods. Results: Seventy-three patients with a median walking ability of 500 meters (IQR 400-800) were enrolled in the study. The agreement between patients’ estimates and GPS measurements was mod-
NU
est (ICC 0.29, 95% CIs 0.06-0.49) and was influenced by course of the disease, patients’ mood and inaccuracy at estimating long distances. A better reliability was found between neurologists’ and GPS measures (ICC 0.68, 95% CIs 0.53-0.78),, but the variability increased for longer distances
MA
and was influenced by patients’ depressive symptoms, fatigue and course of the disease. Conclusions: This study showed a poor agreement between patients' and neurologists' estimates of
ED
maximum walking ability and patients' mean maximum walking ability measured in their daily life through a GPS smartwatch, with many factors affecting patient’s and neurologists’ estimates of the EDSS. The use of remote measurement technologies may provide a better understanding of the im-
CE
PT
pact of MS in a patient’s life.
AC
Keywords: multiple sclerosis, EDSS, walking distance, GPS tracking technologies, disability, remote monitoring technologies
ACCEPTED MANUSCRIPT Dalla Costa G.
3
1. INTRODUCTION
The assessment of patients’ walking ability is critical for the Kurtzke’s expanded disability status scale score (EDSS), the disability scale commonly used in multiple sclerosis (MS) clinical practice [1]. The scale ranges from 0 (no clinical sign and no disability) to 10 (death caused by the disease),
T
and a 1.0 step change is important, although the difference between grades 5.5, 5.0, 4.5, and 4.0 is
SC RI P
the ability to walk 100, 200, 300, or at least 500 meters respectively. Current common practice measurement for walking activity in MS patients typically involves patient-reported walking ability, with considerable intra and inter-patient variability in accuracy. In fact, walking ability may be influenced by ecological factors and the subject’s emotional and mood characteristics. Moreover, other clinical parameters such as the stride-time variability have been suggested to be relevant in clini-
NU
cal practice [2]. Therefore, a one-time assessment by the treating neurologist during periodic visits may not be representative of the actual severity of the disease. The last decade has seen an explo-
MA
sion in the capability of monitoring different health parameters via sensors in smartphones or wearable devices, which may overcome these limitations by accurately representing the disease status through continuous assessments and increasing quality in clinical decision making and engagement
ED
with patients.
The aim of this study was to evaluate the agreement between patients’ and neurologists’ estimates
PT
of walking ability and patients’ mean walking ability measured in their daily life through a GPS
2. METHODS
CE
smartwatch, and assess limitations of the current methods.
AC
This was a prospective study involving MS patients with restricted ambulation (with or without assistance, EDSS 4.0-6.5), recruited at the MS Center of San Raffaele Hospital, Milan, Italy. Patients estimated their maximum walking distance (PWD) and, consequently, an EDSS score was calculated. Patients’ walking ability was then assessed by a certified EDSS rating physician (GDC, MR), by directly measuring the maximum walking distance (NWD) patients covered walking in an aisle on a surface without obstacles. Then patients were trained to use a GPS smartwatch (Garmin Forerunner 230) for four weeks, to obtain daily objectively measured values of maximum walking distance, and from the available measures the mean maximum walking distance (MWD) was calculated. The Mini Mental State Examination (MMSE), the Modified Fatigue Impact Scale (MFIS), the Beck Depression Inventory (BDI) and the Short Form-36 (SF-36) questionnaires were used as dummy variables (cutoff: median value) to measure cognitive status, fatigue, depression and the general health
ACCEPTED MANUSCRIPT Dalla Costa G.
4
status of the patients, respectively. Patient’s feedback on their overall experience in the study and with the use of the GPS was collected at the end of the study. The intraclass correlation coefficient (ICC) using a two-way mixed effects model [3] was used to calculate agreement between measurements by the neurologists, the patients and the GPS smartwatch. Values for ICC(2,1) measures were interpreted using the guidelines of Portney and Watkins [4]. Sample size calculations were based on optimal and minimal levels of reliability, and a sample size of 72 was sufficient to detect a true reliability of 0.75 from a null value of 0.50, with 90% pow-
T
er using ICC(2,1) [5]. Bland and Altman plot analysis [6] was applied to visually assess individual
SC RI P
agreement between methods. The 95% limits of individual agreement between two methods were calculated as the mean difference between the two methods ± 1.96 standard deviation. Multiple regression analysis was performed to determine the factors influencing the agreement between the methods. This study was approved by the Ethical Committee of San Raffaele Hospital.
NU
3. RESULTS
Seventy-three patients were enrolled in the study (mean age 47.6 years, 50.7% females). Half were
MA
evaluated in the morning by the treating neurologist, the others in the afternoon. Fifty-three patients (72.6%) had a relapsing remitting course of the disease and the others had a progressive phase. Their main clinical and demographic characteristics are shown in Table 1.
ED
The median MWD measured by the smartwatch was 500 meters (IQR 400-800, range 100-1300), corresponding to an EDSS of 4.0. A correspondence between PWD and MWD was found in only
PT
15% of patients. Overall 79.5% patients underestimated their walking ability and 5.4% overestimated it. This corresponded to a correction of the self-reported EDSS scores of >0.5 point in 84.9% of
CE
patients. Therefore, a low ICC between the two methods was observed (0.29, 95% CIs 0.06-0.49). In multiple regression analysis, older patients with a longer disease history, or a progressive form of disease, or who were on second-line therapies, with a high SF36 score tended to overestimate their
AC
ability, while patients with depressive symptoms or with the best MWD underestimated it (Table 2). A better reliability was found between NWD and MWD (ICC 0.68, 95% CIs 0.53-0.78), but the agreement was significantly dependent upon the form of disease and type of ongoing DMD, BDI MFIS and SF36 scores and MWD (Table 2). The plot analysis confirmed these findings (Supplementary data). Overall, most of the patients (93%) gave positive feedback on the study and reported an increased motivation to walk for the whole period they were using the smartwatch.
4. DISCUSSION
ACCEPTED MANUSCRIPT Dalla Costa G.
5
The evaluation of patients’ walking ability is critical for assessment of Kurtzke’s EDSS score, the disability scale commonly used in MS clinical practice. This is particularly true for EDSS scores from 4.0 to 6.0, where the difference between grades 5.5, 5.0, 4.5, and 4.0 is the ability to walk 100, 200, 300, or at least 500 meters respectively. Such assessment is usually based on patients’ selfreported walking ability or on the measures the neurologists observe during periodic visits. The present study evaluated the agreement between patients’ and neurologists’ estimates and the measurements of a GPS smartwatch.
T
A poor agreement between patients’ estimates and GPS objective measurements was observed. Par-
SC RI P
ticularly, depressive symptoms led the patients to underestimate their walking abilities probably because of their negative outlook on their capacities. Furthermore, these patients may not be challenged in their daily life routines and may not be doing enough physical activity, which may lead to faster disability worsening. Also, patients whose objective maximum walking distances was longer underestimated their capacities, and these results are in line with a previous study showing that ar-
NU
tillery observers estimating target distances showed wide variability [7], and in another study, patients as well as doctors were inaccurate at estimating particularly long distances [8].
MA
The agreement between the evaluating neurologist and the GPS measures was better than the patients’ estimates, as the neurologists’ estimates were based on the observed distance the patients were able to walk during the visit to the neurologist. Nevertheless, this was greatly influenced by
ED
the disease status, patients’ mood, and patients’ fatigue during the clinical examination. This is in line with previous findings that showed that the day-to-day variability in walking ability of MS pa-
PT
tients was much higher than for controls [9-10]. This study has some limitations First of all, it involved patients with a moderate disability. Second-
CE
ly patients wearing the smartwatch could have been more motivated to walk than usually. These may prevent the generalization of the study findings to the MS population. Nevertheless, the study
AC
has important clinical implications as it supports the role of remote measurement technologies (RMT) as an innovation which could, in the foreseeable future, provide real-time information on the patient’s walking ability in the patient’s home environment, increasing quality and accuracy in clinical decision making whilst ensuring feasible and appropriate levels of engagement with patients [11]. Finally, overall patient experience was positive and most MS patients reported that the device was useful in increasing motivation to walk for the whole period they were using the smartwatch. Further prospective multi-centric studies are needed to validate our results and optimize future use of RMT, bridging the gap between technological innovation and clinical utility. ACKNOWLEDGEMENTS
ACCEPTED MANUSCRIPT Dalla Costa G.
6
We would like to thank all our patients for their enthusiastic participation, and Acesm Onlus for supporting this study. FUNDING
This research received no specific grant from any funding agency in the public, commercial, or notfor-profit sectors.
T
CONFLICT OF INTEREST STATEMENT
SC RI P
G. Dalla Costa, M Radaelli, S Maida, F Sangalli, B Colombo report no discosures. L. Moiola has received honoraria for speaking in scientific meetings and for advisory board from Biogen TEVA, Genzyme and Merck. G. Comi has received personal compensation for consulting services and/or speaking activities from Novartis, Teva, Sanofi, Genzyme, Merck, Biogen, Excemed, Serono Symposia International Foundation, Roche, Almirall, Receptos, Celgene, Forward Pharma. V. Martinelli
NU
has received honoraria for activities with Biogen, Merck, Bayer, TEVA, Novartis and Sanofi as a
MA
speaker. REFERENCES
1. Kurtzke JF. Rating neurologic impairment in multiple sclerosis: an expanded disability sta-
PT
ED
tus scale (EDSS). Neurology.;33(11):1444-52. 2. Allali G, Laidet M, Hermann FR, Armand S, Elsworth-Edelsten C, Assal F and Lalive PH. Gait variability in multiple sclerosis: a better falls predictor than EDSS in patients with low disability. J Neural Transm 2016 Apr;123(4):447-50. 3. British Standards Institution. Precision of test methods 1: Guide for the determination and
AC
CE
reproducibility for a standard test method (BS 5497, part 1). London: BSI, 1979. 4. Portney LG, Watkins MP. Foundations of clinical research: Applications to practice. Upper Saddle River, NJ: Pearson Prentice Hall, 2009. 5. Zou GY. Sample size formulas for estimating intraclass correlation coefficients with precision and assurance. Statistics in Medicine, 31(29), 3972–3981, 2012 6. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1:307–310, 1986. 7. Fine BJ, Kobrick JL. Individual differences in distance estimation: Comparison of judgments in the field with those from projected slides of the same scenes. Percept Motor Skills:57: 3–14, 1983. 8. Sharrack B, Hughes RAC. Reliability of distance estimation by doctors and patients: Cross sectional study. BMJ: 315, 1997. 9. Albrecht H, Wötzel C, Erasmus LP, Kleinpeter M, König N, Pöllmann W. Day-to-day variability of maximum walking distance in MS patients can mislead to relevant changes in the
ACCEPTED MANUSCRIPT Dalla Costa G. Expanded Disability Status Scale (EDSS): Average walking speed is a more constant parameter. MultScler:7:105–109, 2001. 10. Morris ME, Cantwell C, Vowels L, Dodd K. Changes in gait and fatigue from morning to afternoon in people with multiple sclerosis. J NeurolNeurosurgPsychiatry:72:361–365, 2002.
AC
CE
PT
ED
MA
NU
SC RI P
T
11. Dalla Costa G, Maida S, Sorrentino P, Braunstein M, Comi G, Martinelli V. Opinion & special articles: professionalism in neurology. Maintaining patient rapport in a world of EMR. Neurology. 2014;83(2):e12–5.
7
ACCEPTED MANUSCRIPT Dalla Costa G.
Table 1. Clinical and Demographic Characteristics of the Seventy-three Patients Enrolled in the Study
Gender Females, No. (%) Males, No. (%)
37 (50.7) 36 (49.3)
Disease duration, mean (SD)
14.4 ± 7.3
Type of disease Relapsing-remitting, No. (%) Progressive form, No. (%)
53 (72.6) 20 (27.4)
Type of DMD First-line therapy, No. (%) Second-line therapy, No. (%)
61 (83.6) 12 (16.4)
Baseline assessment time Morning, No. (%) Afternoon, No. (%)
MA
MMSE score, mean (SD)
27.2 ± 2.1
ED
41 (56.2) 32 (43.8)
SF-36 score, mean (SD)
AC
CE
BDI score, mean (SD)
36.1 ± 15.5
PT
MFIS score, mean (SD)
SC RI P
47.6 ± 7.9
NU
Age, mean (SD)
T
Characteristic
95.2 ± 11.3 13.0 ± 8.3
8
ACCEPTED MANUSCRIPT Dalla Costa G.
9
Table 2. Predictors (independent variables) of Agreement between Patients’, Neurologists’ and GPS Smartwatch Estimates of Walking Ability (dependent variables of Model A and B respectively) M ultivariate analysis
Coefficient
95% CIs
p
Coefficient
95% CIs
p
M ale sex
11.3
-122.2 - 144.9
0.87
-
-
-
Age at baseline
-9.4
-18.8 - 0.02
0.07
4.9
0.7 - 9.1
0.02
Baseline assessment in the afternoon
22.5
-5.7 - 54.2
0.32
-
-
-
Disease duration
-3.4
-12.8 - 5.9
0.47
T
Univariate analysis
3.2 - 11.9
< 0.01
Progressive disease
-39.5
-188.9 - 109.9
0.60
80.1
9.6 - 150.5
0.03
Second-line therapy
242.9
72.2 - 413.6
<0.01
206.1
127.8 - 284.4
< 0.01
BDI score > 13
-9.5
-17.0 - -1.9
0.01
-130.0.4
-187.4 - -72.6
< 0.01
M FIS score > 36
21.6
-111-9 - 155.0
0.75
-
-
-
SF36 score > 95
-0.2
-6.9 - 6.6
0.96
169.6
113.5 - 225.8
< 0.01
GPS mean maximum walking ability
-0.8
-1.0 - -0.7
<0.01
-0.8
-0.9 - -0.7
< 0.01
Patient vs GPS Smartwatch (M odel A)
SC RI P
-0.07
-110.5 - 110.4
1.0
-
-
-
Age at baseline
-9.7
-17.3 - -2.0
0.01
-
-
-
Baseline assessment in the afternoon
91.7
-17.9 - 201.2
0.10
-
-
-
-8.9
-16.3 - -1.4
0.02
-
-
-
41.7
-81.8 - 165.1
0.50
121.6
32.2 - 132.9
< 0.01
172.8
29.6 - 316.1
0.02
123.8
39.4 - 156.7
0.01
-178.2
.280.3 - -76.1
<0.01
-128.9
- 132.0 - -35.5
< 0.01
-93.1
-201.3 - 15.1
0.09
-70.6
-129.2 - 8.1
< 0.01
71.3
-37.8 - 180.5
0.20
129.2
49.4 - 100.9
< 0.01
-0.5
-0.7 - -0.4
<0.01
-0.5
-0.6 - -0.35
< 0.01
PT
Progressive disease
M FIS score > 36
AC
SF36 score > 95
CE
Second-line therapy BDI score > 13
ED
M ale sex
Disease duration
GPS mean maximum walking ability 1
NU
MA
Neurologist vs GPS Smartwatch (M odel B)
7.5
Univariate and multivariate regression analysis was performed to determine the factors influencing the difference between patients’ and neurologists’ estimates of walking ability and patients’ mean walking ability measured in their daily life through a GPS (outcomes of model A and B respectively). Backward selection technique based on the Akaike information criterion was used to select the variables to be included in the final models (Multivariate model A Adjusted R2 0.87 and multivariate model B Adjusted R2 0.67; intraclass correlation coefficient between patients’ estimates and GPS 0.29, intraclass correlation coefficient between neurologists’ estimates and GPS 0.68)
ACCEPTED MANUSCRIPT Dalla Costa G.
10
AC
CE
PT
ED
MA
NU
SC RI P
T
Highlights: The evaluation of patients’ walking distance is critical for disability assessment in MS; This assessment is usually based on patients’or neurologists’ estimate during periodic visit; A poor agreement between these estimates and an objective GPS estimate was observed; Remote monitoring technologies are better at representing the disease status in daily life;