- Email: [email protected]
Drawing Assist System Considering Nonperiodic Involuntary Movements
12th IFAC Symposium on Analysis, Design, and Evaluation of Human-Machine Systems August 11-15, 2013. Las Vegas, NV, USA
Drawing Assist System Considering Nonperiodic Involuntary Movements T. Nakao ∗ R. Sakamoto ∗∗ K. Yano ∗ ∗
the Department of Mechanical Engineering| Mie University, 1577 Kurimamachiya-cho, Tsu City, Mie 514-8507, Japan (e-mail: [email protected]). ∗∗ Mie University Hospital, 2-174 Edobasi, Tsu City, Mie 514-8507, Japan Abstract: Creative activities such as painting and music are a source of satisfaction and fulfillment for many people with disabilities, but some individuals with disabilities cannot satisfactorily enjoy such activities due to their involuntary movements. We developed a drawing assistance system for people with nonperiodic involuntary movements that has a position correction filter which adaptively varies the attenuation in the involuntary movements in real time, and error-drawing prevention with force control. Here we confirmed that the system enables a user to draw or paint based on the user’s own senses and motor control, even when experiencing involuntary movement. Keywords: Human-machine interface, Adaptive digital filters, Force control, Real-time systems, Position control, Velocity feedback, Signal analysis 1. INTRODUCTION Many people with physical disabilities enjoy creative activities as a means of self-expression and motivation in life. Painting is one such activity that the physically disabled can perform using residual functions. Whether the act of painting can be satisfactorily enjoyed depends in part on the degree of the individual’s disability. Insofar as involuntary movements affect body control, precise fine-motor movements become difficult to perform, limiting the ability to perform desired tasks such as painting and drawing. To address these problems for people with involuntary movements who draw using computer graphics, studies have been conducted on systems that filter the effects of involuntary movements from the input signal of a device so that the onscreen pointer does not show any unintended movement. Levine and Schappert (1945) developed the Assistive Mouse Adapter where the cut-off frequency of the linear low-pass filter is tuned with a dial. Some methods that cut- off a frequency or frequency band are favorable for periodic involuntary movements that have a characteristic frequency, such as the tremor in Parkinson’s disease. Several other studies have been conducted using the moving average method for nonperiodic involuntary movements, such as those seen in athetosis. Morimoto et al. (2004) proposed the averaging switching methods based on the moving average method as solutions for attenuating the effects of sudden shaking. Shiraishi et al. (2010) proposed a linesmoothing method that applies the additional moving average method to a distance between an input termination point and a termination point corrected by another moving average method. These methods are favorable for users with partial paralysis, who can move and click a mouse on a table. For users with severe involuntary movement and for those who have difficulty in keeping the input device on the same plane, ⋆ This study was supported in part by city area grants from the southern Gifu innovation cluster program.
978-3-902823-41-0/2013 © IFAC
260
the irregularity and variability of the strength of their involuntary movements should be considered. Our research group has proposed a method that changes attenuation based on the strength of involuntary movements by using the magnitude of velocity from a three-dimensional input device (Aoyama et al. (2011)). However, a method based on the moving average method needs to be designed with a moving average period that is long enough to determine the attenuation strength, which worsens operability and limits what the user may draw. In addition, erroneous drawing (i.e., unintentional drawing) occurs. This results significant involuntary movements and bad effects on expression in manipulation with involuntary movements. Against these backgrounds, a drawing assist system that attenuates nonperiodic involuntary movements and exhibits superior operability is desired. The purpose of the present study was to develop a drawing assist system that enables free-form drawing and painting, even in the presence of involuntary movements. Here, a drawing was considered an ensemble of line and dot expressions. We propose a position correction filter for attenuating the effects of involuntary movements in real time, plus error-drawing prevention with force control that suppresses excessive manipulation and reduces error drawing for users with nonperiodic complex involuntary movements, on the basis of movement analyses. 2. CHARACTERISTICS OF INVOLUNTARY MOVEMENTS Involuntary movements can be classified as being with or without regularity. A representative involuntary movement with regularity is a tremor whose characteristic frequency with respect to type and impetus has been ascertained. A representative involuntary movement with regularity is athetosis (Chino and Ando (2005); Kaji (2006)). In individuals with athetosis, muscle tone fluctuates with voluntary movement and mental distress, which makes it difficult for an involuntary-movement patient to achieve continuous postural maintenance. These exces10.3182/20130811-5-US-2037.00021
IFAC HMS 2013 August 11-15, 2013. Las Vegas, USA
3. ANALYSIS OF INVOLUNTARY MOVEMENTS WITH THE DRAWING ASSISTANCE SYSTEM
When manipulated in the front-back direction (yP ), the pointer moves in up-down direction (yD ). The up-down direction (zP ) of the stylus is used to judge ‘draw start’ and ‘draw end’ made by the user. By the use of difference between a current coordinate and another past coordinate on z axis, a judgment regarding the ‘draw start’ (i.e., the beginning of a line being made) is made when the user swings down the stylus, such as when putting the brush closer to the canvas as expressed in (1). A judgment regarding the ‘draw end’ (i.e., the end of the line drawn) is made when the user swings the stylus up, such as when removing the brush from the canvas as expressed in (2). The raster graphics editor used in the present study was ArtRage, version 3.0 (e frontier, Tokyo). ∆zP = zP (t) − zP (t − 0.5) ≤ −0.05 [m]
(1)
∆zP = zP (t) − zP (t − 0.19) ≥ 0.03
(2)
Input device
ΣD xD User
Input device
zP
P
50 40 30 20 Draw start 10 0 -10 -20 -30 -40 Draw end -50 -80 -60 -40 -20 0 20 40 60 80 -3
y position [m]
Display
y position [m]
×10-3
yD
x position [m] (a)
yP
[m]
In an initial test of the system, the user (an individual with athetoid cerebral palsy) traced the design of an ogee (a shape similar to an “S”) under the experimental setup shown in Fig. 1. A square on the right of the display was the start point, and a circle on the left side of the display was the end point or goal. Fig. 2 shows the experimental results. The characteristics of the involuntary movements mentioned in section 2 were observed in the user. The user encountered difficulty in controlling the direction and the travel distance of the pointer and the drawing start and end within the designated part of the task. The involuntary movements of athetosis include both weak and strong movements. In the case of strong involuntary movements, the user almost completely loses control of his or her voluntary q movement, too. In analyses using synthesized ve2 + v 2 + v 2 as an analysis element, the users’ locity Vs = vpx py pz strong involuntary movements occurred most frequently at velocities greater than 0.20 [m/s] (Nakao et al. (2011)).
×10
×10-3
×10-3
50 40 Draw end 30 20 10 0 -10 -20 -30 -40 -50 -80 -60 -40 -20 0
Draw start
20 40 60 80 -3
x position [m] (b)
×10
y position [m]
sive movements do not stop occurring when voluntary movements are attempted (Gomi (2005)). Nakai (1984) reported that in their analysis of power grip movement and extension and flexion of the wrist in patients with cerebral palsy, it was found that strong hypertonic behavior in extensor muscle groups, instantaneous movements, surges of power, and difficulty of power control are all characteristics of athetosis. Sugawara and Takahama (1979) found that in their motor functional assessment of upper limb by electromyography, non-smooth functioning of the upper limbs of individuals with cerebral palsy was the result of an imbalance between flexor and extensor muscle groups. Yamada et al. (2010) reported that when individuals with athetosis operated a computer by using a joystick or switch, their involuntary movements made it difficult to manipulate the system. The imbalance between agonist, synergistic and antagonist muscles is characteristic of athetosis. In effect on synergistic and antagonist muscles, Adiadochokinesis includes overruns due to the delaying of muscle responses, and rebound in an adverse direction due to the extension of muscle contraction time and repetitive movement (Kaji (2006)). In particular, the involuntary movements of individuals with athetoid cerebral palsy and muscle tension occur suddenly and are considerably heightened by exertion. These involuntary movements manifest themselves when a person with the condition attempts to maintain a posture or sustain a certain movement; in addition, the muscle tone of individuals with athetoid cerebral palsy increases during postural maintenance and decreases while at rest (Chino and Ando (2005)). The involuntary movements of athetosis cannot be predicted, and they are irregular in direction and include twisting and reflexing. In addition, there are not only wrist-, elbow- and shoulder joint-centered involuntary movements but also body trunk destabilizing. Thus, the involuntary movements of athetosis are three-dimensional, complex movements (Aoyama et al. (2011)).
50 40 30 20 Draw end 10 0 -10 -20 -30 -40 -50-80 -60 -40 -20 0
Draw start
20 40 60 80 -3
x position [m] (c)
×10
Fig. 2. Tracing results: Raw data on the x-y plane.
xP
4. POSITION CORRECTION CONTROL OF ADAPTIVE INVOLUNTARY BEHAVIOR ATTENUATION FILTER
Fig. 1. Experimental setup of the drawing assistance system. Our drawing assistance system consists of an input device that the user manipulates directly, a control device that runs the program, and an output device that displays the control results. In the present study, we also used a compact haptic interface, the Phantom Omni (SensAble, Inc., Wilmington, MA) shown in Fig. 1. We considered that users with involuntary movements have difficulty in keeping the device on plane and keeping to grasp the device, ease of re-grasping, and possibility of exerting resistance force in the selection process of input device. In our system, the point of rotation at the center of the gimbal is used as the input point P (xP , yP , zP ) for drawing. When the stylus on the gimbal is manipulated in the right-left direction (xP ), a pointer on display moves in the same direction (xD ). 261
We designed an adaptive involuntary behavior attenuation filter (AIBAF) that both prevents the pointer from overrunning the target even when the user has lost motor control due to an involuntary movement and attenuates the effect of involuntary movements adaptively with changes in synthesized velocity. The coordinates of the corrected point at sample i, G(i) = (xG (i), yG (i), zG (i)), as shown in (3), is updated by using a coordinate of the corrected point before a sample G(i − 1), the current measurement of the coordinate of input point P (i), and an attenuation weight coefficient w(i). The range of w(i) is 0 < w(i) ≤ 1. The attenuation weight coefficient w(i), given by (4), decreases with the increase in synthesized velocity Vs (i) corresponding to the state of involuntary movement. This results in high attenuation.
IFAC HMS 2013 August 11-15, 2013. Las Vegas, USA
Vs (i) vth
w(i) = αul β(i) Dth
(3)
Uncorrected data AIBAF
-3
100 ×10
(4)
αul , β(i) and Dth are between 0 and 1. αul is an upper limit coefficient which designs attenuation at weak involuntary movements and low velocity of manipulation. β(i) is a coefficient to increase attenuation at the up- and down-swings for drawing judgments. Dth is a transition attenuation that controls the value of w(i) when Vs (i) equals the transition velocity vth . Based on the results of our analyses, we found that αul =0.0030, vth =0.20[m/s] and Dth =0.11. We conducted tests of the system assisting a user’s line expression with the AIBAF as the user traced the outline of a displayed tulip template. Without the assistance as shown in Fig.3a, the user had much difficulty tracing the tulip and lost enthusiasm within 90 seconds of starting to trace. With the assistance (Fig.3b), the user was able to draw a picture that was very much like the tulip.
80
y position [m]
G(i) = (1 − w(i))G(i − 1) + w(i)P (i)
60
Draw start
40 20 0
Draw end
-20 -40 -60 -80 -100
-140-120-100 -80 -60 -40 -20
0 20 40 60 80 100 120 140 -3 ×10
x position [m]
Fig. 4. Results of raw data and AIBAF-corrected trajectory made by tracing the left-side curve of a tulip. manipulation, and the system enabled a user with involuntary movements to move the pointer in intentional directions and create line expressions.
(b)Result with assist of AIBAF
Fig. 3. Uncorrected tracing result and AIBAF-corrected tracing result. We further examined the attenuation of the effect of involuntary movements in the tulip tracing test. The comparison of differences between the AIBAF-corrected and uncorrected position on the x-y plane in tracing the left-side curve of the tulip petal is shown in Fig.4. The drawing was started at the upper end of the template, with continuous tracing from the leftmost corner to the end at the lower part of the petal. The movements are shown as the uncorrected trajectory, ranging quite widely with frequent veering from the tulip template. In contrast, the AIBAF-corrected pointer trajectory changes with the curve of the template. The relation between synthesized velocity Vs (i), drawing judgment velocity VD (i) and attenuation weight coefficient w(i) of these data is shown in Fig.5. Additionally Vs (i), VD (i), the uncorrected travel distance, and the AIBAF-corrected travel distance of these data are shown in 6. For the first 8.0 s from the start of the drawing, the user was tracing from the corner part on the upper left to the curve part, and the user’s manipulation including an excessive down-swing was observed. This appears as high Vs (i) and VD (i) levels in the graphs. In this regard, this is intentional manipulation although the Vs (i) value is high, and not a strong involuntary movement as it seemed by observation. The weight and the corrected travel distance decreased with the increase in velocity. For the latter period of drawing, the user was tracing the curve part with weak involuntary movements. The Vs (i) changed around 0.20 [m/s], and w(i) changed in the range of 0.50×10−3 to 2.5×10−3 . According to the above results, the AIBAF-implemented drawing assistance system was able to change the attenuation adapted to the change of velocity corresponding to the state of involuntary movements and 262
Draw end VS VD
245
250
255
260
255
260
Time [s] -3
×10
Weight
(a)Result without assist
Velocity [m/s]
Draw start 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0
3.0 2.5 2.0 1.5 1.0 0.5 0 245
250 Time [s]
Fig. 5. Velocities and attenuation weight changes during the tracing of the left-side curve of the tulip. 5. ERRONEOUS DRAWING PREVENTION WITH RESISTANCE FORCE CONTROL 5.1 Erroneous Drawing in Line-dot Complex Expression A number of erroneous drawings occurred in all user tests (where ”erroneous drawing” is defined as drawing in which the draw start and/or draw end were unintentionally executed). Most of the erroneous drawings were caused by involuntary movements. The occurrence of many erroneous drawings can lead to lengthening of creation, fatigue, tension and significant involuntary movements. Thus, the prevention and reduction of erroneous drawings are desirable. Users with involuntary movements often intentionally undo erroneous drawings. With our drawing assistance system, undos are accomplished using an undo button on the screen. A line or dot is thus undone by an undo operation. When erroneous drawings occurred in the tulip
IFAC HMS 2013 August 11-15, 2013. Las Vegas, USA
Velocity [m/s]
Draw start 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0
by a resistance force controller based on mass, a damper and a spring system with respect to the input point coordinate P (i), the resistance force on each axis F(i) = (Fx (i), Fy (i), Fz (i)) as shown in (5)™ (7).
Draw end VS VD
˙ + KPK (q(i)) ) F(i) = Efc (i)( M¨q(i) + Dq(i) 245
250
255
260
Time [s]
PK (i) =
-3
Deviation [m]
×10 30
Uncorrected
xds (k) − xp (i) yds (k) − yp (i) zds (k) − dz − zp (i)
Efc (i) = diag( Ef cxy (i), Ef cxy (i), Ef cz (i) )
20
(6) (7)
T
10 0
245
250
255
260
Time [s] -3
×10 0.4 Deviation [m]
(5)
Corrected
0.3 0.2 0.1 0
245
250
255
260
Time [s]
Fig. 6. Velocities and travel distance changes during the tracing of the left-side curve of the tulip. tracing test described above, the total number of drawings was 113, and that of the undos was 78, or 69.0% of the total number. We found a tendency for most of the erroneous drawings to be undone, and for all user tests conducted, the proportion of undos to the total drawings was approx. 70.0%. In the user tests, we observed that the causes of erroneous drawings fell into four basic categories: grasping the stylus, moving the pointer, weakness, and unintentional dot expression at the start of a line expression. Here we will focus on the latter category, since it has a direct impact on expression and is the only unintentional draw-start. This type of erroneous drawing is hereinafter called “dotted-error drawing.” In the tests examining the users’ drawings from the starting part of an ogee and tracing the ogee and other test cases, we found that the range of time required between the judgment of the draw start and the draw end in dotted-error drawings is 0.15 to 0.40 s. Dotted-error drawing is suspected to be due to an imbalance among agonist, synergistic and antagonist muscles, as the characteristics of involuntary movements described in Chapter 2 appeared in the present users’ manipulation. In particular, we observed that the down-swing action at the draw start was often excessive and rebounded into an up-swing. We suspect that this is caused by the imbalance between synergistic muscles and agonist muscles, making the user lose motor control. Additionally, in the cases of dotted-error drawing with a non-excessive down-swing, the reaction time of the antagonist muscles to stop the down-swing movement is extended. The up-swings may have been caused by an overrun of antagonist muscles and resulted in an unintentional draw-end judgment. 5.2 Resistance Force Control Suppressing Excess Movements We propose a method that suppresses excessive down-swings and unintentional up-swings at the draw start by controlling the exerted force of a compact haptic interface as an input device to prevent dotted-error drawing. The exerted force is controlled 263
where, q(i) = ( xp (i), yp (i), zp (i) ) is the variable vector. Elastic, viscosity and inertia coefficients on each axis are respectively stored in K=diag( Kx , Ky , Kz )|D=diag( Dx , Dy , Dz )| M=diag( 0, 0, Mz ). Positional functions are stored in PK . A coordinate Pds (k) = (xds (k), yds (k), zds (k)) on an input coordinate system at the moment of the kth draw-start judgment was defined as the draw-start coordinate. A draw-start coordinate is held until the draw-end judgment and updated at every draw-start judgment. dz corresponds to the natural length of the elastic element. The resistance force Fz (i) is exerted against down-swings and up-swings, and Fx (i) and Fy (i) are exerted to suppress operating movements leaving from Pds (k) along the right-and-left and back-and-forth directions. Additionally, to prevent excessive strain from disturbing intentional line and dot expression, the period during the exertion of the resistance force was limited from the draw start. Ef cxy (i) and Ef cz (i) shown in (7) effect time functions determining the limited time for each axis. Ef cxy (i) and Ef cz (i) were 1.5 and 0.60 s, respectively. 5.3 User Test of Drawing Assistance System Integrating the Resistance Force Controller User tests of ogee tracing and dotting were carried out to determine the utility of the resistance force controller for reducing or eliminating dotted-error drawing. In ogee tracing tests, we assessed whether the user could draw a line without any dots at the draw start. A set of this test included three successful traces of the ogee. In dotting tests, we assessed whether the user could intentionally express dots in the case with resistance force. We instructed the user to draw a dot in a circle, and draw dots in five circle in order of preference. A set of this test included successful dotting of all circles. Three ogee tracing tests and three dotting tests, each with and without resistance force were carried out for a total of 12 sets. The parameters of the resistance force controller were set at Kx =Ky =40| Kz =20, Dx =Dy =Dz =4.0| Mz =0.10 and dz =-0.15[m]. In the experimental results for the ogee tracing, the final drawn results on the raster graphics editor without and with resistance force are shown in Fig.7 and Fig.8, respectively. Not all of the drawings remain in these figures, because the undo function was used during the tests. The number of dotted-error drawings in panel (A) of Fig.7 is 22 times; that in (B) is 10 times and that in panel (C) is three. In contrast, the number of dottederror drawings in all tests that used the resistance force was 0. Corresponding to the second trace in Fig.8(B), the changes of exerted resistance force and position on the z-axis are shown in Fig.9. We can see that Fz is exerted for 0.60 s from the draw start, and Fx and Fy are exerted for 1.5 s from the draw start. In the change of Fz , the direction of Fz is positive, opposite to the dropping of the z-axis coordinate at the draw start. A brake
IFAC HMS 2013 August 11-15, 2013. Las Vegas, USA
force against the down-swing is then exerted. Additionally, the direction of Fz is negative, opposite to the rising of the z-axis coordinate at 175.6 s and 176.0 s. The brake force against the early phase of the up-swing is then exerted. (B)
(A)
(C)
Fig. 10. Drawing results: Task = pointing within five circles without force control. (B)
(A)
(C)
Fig. 7. Drawing results: The ogee tracing task without force control. (B)
(A)
(C)
Fig. 11. Drawing results: Task = pointing within five circles with force control. Draw start (B)
(C)
Fig. 8. Drawing results: The task of ogee tracing with force control.
Draw start 3 2 1 0 -1 -2 -3 175
tEFCxy
tEFCz
Position [m]
Force [N] Position [m]
143.5 -3
z force 175.5
176 176.5 Time [s]
177
-3
100
5 4 3 2 1 0 -1 -2 -3 -4 -5
Draw End
×10
tEFCz
tEFCxy
144.5
145
144.5
145
Force [N]
(A)
200 150 100 50 0 -50
×10
144 Time [s]
z position
143.5
144 Time [s]
z position 50
Fig. 12. Change of exerted force of pointing on the No. 4 circle in Fig.11C
0 -50 175
175.5
176 176.5 Time [s]
6. CREATING PAINTINGS UNDER INVOLUNTARY MOVEMENTS
177
Fig. 9. Change of exerted force and z-axis position with force control in Fig.8B In the dotting experiment, the final drawn results on the raster graphics editor without and with resistance force are shown in Fig.10 and Fig.11, respectively. In all six cases, the user achieved dotting, draw start and draw end inside the circle frame, although there were a few failed drawings. Corresponding to the dotting in the left lower circle in Fig.11C, the changes of exerted resistance force and position on the z-axis are shown in Fig.12. The resistance force was exerted as designed, as was the case for the ogee tracing. An up-swing and a draw-end judgment between 0.60 s and 1.5 s from the draw start were performed. According to the above results, our drawing assistance system and integrated AIBAF with resistance force controller did exert resistance force to prevent dotted-error drawing; the system also assisted line expression and did not disturb intentional dot expression. 264
We carried out user tests that the user actually created paintings in user’s own sense with drawing assistance system integrated AIBAF with resistance force controller. The subjects of the paintings chosen by the user were an orange and a four-leaf clover. For the creation of the painting of an orange, we devised a plan in which the user practiced the sequence of creation and how to use the painting materials and techniques by operating a numeric keypad like a mouse that is user’s normal environment, so that the user have the creating sequence and image in advance. In the creation of the four-leaf clover, all plans were designed under the user’s initiative. The following four prevention measures against erroneous drawings were used. 1. Adding a condition VD (i) ≥ 0.30[m/s] to the draw-start judgment to prevent erroneous drawings due to the moving pointer and weakness. 2. Gravity compensation to prevent erroneous drawings due to the operating portion’s own weight. 3. Adding a limited range −45 ≤ ψgbl ≤ 45 [deg] to the draw-start judgment to prevent unintentional draw starts at the stylus’s unfitted orientation. where ψgbl is the angle of the fourth joint of the haptic device. ψgbl =0[deg] when the stylus
IFAC HMS 2013 August 11-15, 2013. Las Vegas, USA
is in parallel with the arm portion. 4. A resistance force controller to prevent dotted error drawing. The parameters in these user tests were Kx =Ky =50| Kz =-10, Dx =Dy =3.0| Dz =4.0| Mz =0.10, dz =-0.15[m]. A completed painting of an orange is shown in Fig.13. The total number of drawings was 1,224 and there were 263 undoings (21.5%). The total time required to create the painting was approx. 4 h. Four types of materials (oil paints, watercolors and two types of palette knife) were used in the painting’s creation. Curves were expressed with over-glazing using the oil paint, and the atmospheric effect was achieved with a palette knife. Thus, mainly curve expressions were created; the hull part of the orange was created in dotted expression. With our system, the user was able to create an orange still-life painting that resembles impressionist art. A completed fourleaf clover painting is shown in Fig.14. The total number of drawings was 798, and there were 129 undoings (16.2%). This percentage of undos was reduced by approx. 50% compared to the system used without the integrated erroneous drawingprevention control. The total creation time for this painting was approx. 2 h. The clover’s leaves and patterns were expressed mainly in curves with the oil paint and palette knife. The shadow parts were expressed in dot aggregate. Finally user could create a painting of clover like illustration. We believe that our proposed system will improve user’s experience because with this system, an individual can use several types of materials and an undo function, and can change the colors on user’s own. In light of the results presented above, we contend that the drawing assistance system with an integrated AIBAF and resistance force controller can prevent error drawings and reduce undos, enabling the creation of paintings of a user’s own design and choosing.
Fig. 13. Orange painted by a user with the drawing assistance system. 7. CONCLUSION We found that our drawing assistance system with an integrated adaptive involuntary behavior attenuation filter that changes attenuation adapted to the users’ involuntary movements and manipulation in real time plus the error-drawing prevention control with a resistance force controller functioned well. Our system can be used by individuals with athetoid cerebral palsy or muscle tension based on involuntary movements. With this system, the number of necessary undos was reduced, and users with nonperiodic complex involuntary movement who have great 265
Fig. 14. Four-leaf clover painted by a user with the drawing assistance system. difficulty painting without assistance were able to make linedot ensemble paintings according to their own design choices. We believe that our method has the capability of integration with devices manipulating actual materials instead of display, and application to handwriting assistance for the users. REFERENCES Aoyama, H., Nakao, T., Miyagawa, N., Kubota, N., Horihata, S., and Yano, K. (2011). Development of drawing assist system for patients with cerebral palsy of the tension athetosis type. Proceedings of IEEE ICRA2011 International Conference on Robotics and Automation, 4664–4669. Chino, N. and Ando, N. (2005). Rehabilitation of Neurogenic Disorders. KANEHARA and CO., Tokyo. Gomi, S. (2005). Rehabilitation Medicine Seminar. Ishiyaku Publishers, Inc., Tokyo. Kaji, R. (2006). Diacrisis and Treatment of Involuntary Monement. SHINDAN-TO-CHIRYOSHA CO., Tokyo. Levine, J.L. and Schappert, M.A. (1945). A mouse adapter for people with hand tremor. IBM SYSTEMS JOURNAL, 623– 624. Morimoto, D., Nawate, M., Watanabe, T., Fukuma, S., and Honda, S. (2004). A painting tool with blurring compensation for people having involuntary hand motion. Technical Report of The Institute of Erectronics, Information and Communication Engineers, 59–64. Nakai, S. (1984). Power grip movement and extension-flexion movement of wrist in cerebral palsied children. Japanese Journal of Special Education, 7–15. Nakao, T., Matsui, H., Yano, K., Kubota, N., Miyagawa, N., and Horihata, S. (2011). Drawing assist system for reducing effects of involuntary movements. Journal of Biomechanical Science and Engineering, 6, 5, 362–377. Shiraishi, M., Kawase, H., and Shinya, M. (2010). A compensation method for hand-drawn illustration by moving average. The Institute of Image Information and Television Engineers Technical Report, 34, 18, 23–26. Sugawara, M. and Takahama, M. (1979). Movement analysis of upper limb of individuals with cerabral palsy. Sogo Rihabiriteshon, 7, 3, 193–200. Yamada, S., Tanioka, T., Okazaki, Y., Watanabe, K., and Kondo, H. (2010). A development of a chin-controllable mouse system for a quadriplegic person by cerebral palsy. IEICE TRANSACTIONS on Information and Systems, J93-D, 10, 2268–2280.
Drawing Assist System Considering Nonperiodic Involuntary Movements T. Nakao ∗ R. Sakamoto ∗∗ K. Yano ∗ ∗
the Department of Mechanical Engineering| Mie University, 1577 Kurimamachiya-cho, Tsu City, Mie 514-8507, Japan (e-mail: [email protected]). ∗∗ Mie University Hospital, 2-174 Edobasi, Tsu City, Mie 514-8507, Japan Abstract: Creative activities such as painting and music are a source of satisfaction and fulfillment for many people with disabilities, but some individuals with disabilities cannot satisfactorily enjoy such activities due to their involuntary movements. We developed a drawing assistance system for people with nonperiodic involuntary movements that has a position correction filter which adaptively varies the attenuation in the involuntary movements in real time, and error-drawing prevention with force control. Here we confirmed that the system enables a user to draw or paint based on the user’s own senses and motor control, even when experiencing involuntary movement. Keywords: Human-machine interface, Adaptive digital filters, Force control, Real-time systems, Position control, Velocity feedback, Signal analysis 1. INTRODUCTION Many people with physical disabilities enjoy creative activities as a means of self-expression and motivation in life. Painting is one such activity that the physically disabled can perform using residual functions. Whether the act of painting can be satisfactorily enjoyed depends in part on the degree of the individual’s disability. Insofar as involuntary movements affect body control, precise fine-motor movements become difficult to perform, limiting the ability to perform desired tasks such as painting and drawing. To address these problems for people with involuntary movements who draw using computer graphics, studies have been conducted on systems that filter the effects of involuntary movements from the input signal of a device so that the onscreen pointer does not show any unintended movement. Levine and Schappert (1945) developed the Assistive Mouse Adapter where the cut-off frequency of the linear low-pass filter is tuned with a dial. Some methods that cut- off a frequency or frequency band are favorable for periodic involuntary movements that have a characteristic frequency, such as the tremor in Parkinson’s disease. Several other studies have been conducted using the moving average method for nonperiodic involuntary movements, such as those seen in athetosis. Morimoto et al. (2004) proposed the averaging switching methods based on the moving average method as solutions for attenuating the effects of sudden shaking. Shiraishi et al. (2010) proposed a linesmoothing method that applies the additional moving average method to a distance between an input termination point and a termination point corrected by another moving average method. These methods are favorable for users with partial paralysis, who can move and click a mouse on a table. For users with severe involuntary movement and for those who have difficulty in keeping the input device on the same plane, ⋆ This study was supported in part by city area grants from the southern Gifu innovation cluster program.
978-3-902823-41-0/2013 © IFAC
260
the irregularity and variability of the strength of their involuntary movements should be considered. Our research group has proposed a method that changes attenuation based on the strength of involuntary movements by using the magnitude of velocity from a three-dimensional input device (Aoyama et al. (2011)). However, a method based on the moving average method needs to be designed with a moving average period that is long enough to determine the attenuation strength, which worsens operability and limits what the user may draw. In addition, erroneous drawing (i.e., unintentional drawing) occurs. This results significant involuntary movements and bad effects on expression in manipulation with involuntary movements. Against these backgrounds, a drawing assist system that attenuates nonperiodic involuntary movements and exhibits superior operability is desired. The purpose of the present study was to develop a drawing assist system that enables free-form drawing and painting, even in the presence of involuntary movements. Here, a drawing was considered an ensemble of line and dot expressions. We propose a position correction filter for attenuating the effects of involuntary movements in real time, plus error-drawing prevention with force control that suppresses excessive manipulation and reduces error drawing for users with nonperiodic complex involuntary movements, on the basis of movement analyses. 2. CHARACTERISTICS OF INVOLUNTARY MOVEMENTS Involuntary movements can be classified as being with or without regularity. A representative involuntary movement with regularity is a tremor whose characteristic frequency with respect to type and impetus has been ascertained. A representative involuntary movement with regularity is athetosis (Chino and Ando (2005); Kaji (2006)). In individuals with athetosis, muscle tone fluctuates with voluntary movement and mental distress, which makes it difficult for an involuntary-movement patient to achieve continuous postural maintenance. These exces10.3182/20130811-5-US-2037.00021
IFAC HMS 2013 August 11-15, 2013. Las Vegas, USA
3. ANALYSIS OF INVOLUNTARY MOVEMENTS WITH THE DRAWING ASSISTANCE SYSTEM
When manipulated in the front-back direction (yP ), the pointer moves in up-down direction (yD ). The up-down direction (zP ) of the stylus is used to judge ‘draw start’ and ‘draw end’ made by the user. By the use of difference between a current coordinate and another past coordinate on z axis, a judgment regarding the ‘draw start’ (i.e., the beginning of a line being made) is made when the user swings down the stylus, such as when putting the brush closer to the canvas as expressed in (1). A judgment regarding the ‘draw end’ (i.e., the end of the line drawn) is made when the user swings the stylus up, such as when removing the brush from the canvas as expressed in (2). The raster graphics editor used in the present study was ArtRage, version 3.0 (e frontier, Tokyo). ∆zP = zP (t) − zP (t − 0.5) ≤ −0.05 [m]
(1)
∆zP = zP (t) − zP (t − 0.19) ≥ 0.03
(2)
Input device
ΣD xD User
Input device
zP
P
50 40 30 20 Draw start 10 0 -10 -20 -30 -40 Draw end -50 -80 -60 -40 -20 0 20 40 60 80 -3
y position [m]
Display
y position [m]
×10-3
yD
x position [m] (a)
yP
[m]
In an initial test of the system, the user (an individual with athetoid cerebral palsy) traced the design of an ogee (a shape similar to an “S”) under the experimental setup shown in Fig. 1. A square on the right of the display was the start point, and a circle on the left side of the display was the end point or goal. Fig. 2 shows the experimental results. The characteristics of the involuntary movements mentioned in section 2 were observed in the user. The user encountered difficulty in controlling the direction and the travel distance of the pointer and the drawing start and end within the designated part of the task. The involuntary movements of athetosis include both weak and strong movements. In the case of strong involuntary movements, the user almost completely loses control of his or her voluntary q movement, too. In analyses using synthesized ve2 + v 2 + v 2 as an analysis element, the users’ locity Vs = vpx py pz strong involuntary movements occurred most frequently at velocities greater than 0.20 [m/s] (Nakao et al. (2011)).
×10
×10-3
×10-3
50 40 Draw end 30 20 10 0 -10 -20 -30 -40 -50 -80 -60 -40 -20 0
Draw start
20 40 60 80 -3
x position [m] (b)
×10
y position [m]
sive movements do not stop occurring when voluntary movements are attempted (Gomi (2005)). Nakai (1984) reported that in their analysis of power grip movement and extension and flexion of the wrist in patients with cerebral palsy, it was found that strong hypertonic behavior in extensor muscle groups, instantaneous movements, surges of power, and difficulty of power control are all characteristics of athetosis. Sugawara and Takahama (1979) found that in their motor functional assessment of upper limb by electromyography, non-smooth functioning of the upper limbs of individuals with cerebral palsy was the result of an imbalance between flexor and extensor muscle groups. Yamada et al. (2010) reported that when individuals with athetosis operated a computer by using a joystick or switch, their involuntary movements made it difficult to manipulate the system. The imbalance between agonist, synergistic and antagonist muscles is characteristic of athetosis. In effect on synergistic and antagonist muscles, Adiadochokinesis includes overruns due to the delaying of muscle responses, and rebound in an adverse direction due to the extension of muscle contraction time and repetitive movement (Kaji (2006)). In particular, the involuntary movements of individuals with athetoid cerebral palsy and muscle tension occur suddenly and are considerably heightened by exertion. These involuntary movements manifest themselves when a person with the condition attempts to maintain a posture or sustain a certain movement; in addition, the muscle tone of individuals with athetoid cerebral palsy increases during postural maintenance and decreases while at rest (Chino and Ando (2005)). The involuntary movements of athetosis cannot be predicted, and they are irregular in direction and include twisting and reflexing. In addition, there are not only wrist-, elbow- and shoulder joint-centered involuntary movements but also body trunk destabilizing. Thus, the involuntary movements of athetosis are three-dimensional, complex movements (Aoyama et al. (2011)).
50 40 30 20 Draw end 10 0 -10 -20 -30 -40 -50-80 -60 -40 -20 0
Draw start
20 40 60 80 -3
x position [m] (c)
×10
Fig. 2. Tracing results: Raw data on the x-y plane.
xP
4. POSITION CORRECTION CONTROL OF ADAPTIVE INVOLUNTARY BEHAVIOR ATTENUATION FILTER
Fig. 1. Experimental setup of the drawing assistance system. Our drawing assistance system consists of an input device that the user manipulates directly, a control device that runs the program, and an output device that displays the control results. In the present study, we also used a compact haptic interface, the Phantom Omni (SensAble, Inc., Wilmington, MA) shown in Fig. 1. We considered that users with involuntary movements have difficulty in keeping the device on plane and keeping to grasp the device, ease of re-grasping, and possibility of exerting resistance force in the selection process of input device. In our system, the point of rotation at the center of the gimbal is used as the input point P (xP , yP , zP ) for drawing. When the stylus on the gimbal is manipulated in the right-left direction (xP ), a pointer on display moves in the same direction (xD ). 261
We designed an adaptive involuntary behavior attenuation filter (AIBAF) that both prevents the pointer from overrunning the target even when the user has lost motor control due to an involuntary movement and attenuates the effect of involuntary movements adaptively with changes in synthesized velocity. The coordinates of the corrected point at sample i, G(i) = (xG (i), yG (i), zG (i)), as shown in (3), is updated by using a coordinate of the corrected point before a sample G(i − 1), the current measurement of the coordinate of input point P (i), and an attenuation weight coefficient w(i). The range of w(i) is 0 < w(i) ≤ 1. The attenuation weight coefficient w(i), given by (4), decreases with the increase in synthesized velocity Vs (i) corresponding to the state of involuntary movement. This results in high attenuation.
IFAC HMS 2013 August 11-15, 2013. Las Vegas, USA
Vs (i) vth
w(i) = αul β(i) Dth
(3)
Uncorrected data AIBAF
-3
100 ×10
(4)
αul , β(i) and Dth are between 0 and 1. αul is an upper limit coefficient which designs attenuation at weak involuntary movements and low velocity of manipulation. β(i) is a coefficient to increase attenuation at the up- and down-swings for drawing judgments. Dth is a transition attenuation that controls the value of w(i) when Vs (i) equals the transition velocity vth . Based on the results of our analyses, we found that αul =0.0030, vth =0.20[m/s] and Dth =0.11. We conducted tests of the system assisting a user’s line expression with the AIBAF as the user traced the outline of a displayed tulip template. Without the assistance as shown in Fig.3a, the user had much difficulty tracing the tulip and lost enthusiasm within 90 seconds of starting to trace. With the assistance (Fig.3b), the user was able to draw a picture that was very much like the tulip.
80
y position [m]
G(i) = (1 − w(i))G(i − 1) + w(i)P (i)
60
Draw start
40 20 0
Draw end
-20 -40 -60 -80 -100
-140-120-100 -80 -60 -40 -20
0 20 40 60 80 100 120 140 -3 ×10
x position [m]
Fig. 4. Results of raw data and AIBAF-corrected trajectory made by tracing the left-side curve of a tulip. manipulation, and the system enabled a user with involuntary movements to move the pointer in intentional directions and create line expressions.
(b)Result with assist of AIBAF
Fig. 3. Uncorrected tracing result and AIBAF-corrected tracing result. We further examined the attenuation of the effect of involuntary movements in the tulip tracing test. The comparison of differences between the AIBAF-corrected and uncorrected position on the x-y plane in tracing the left-side curve of the tulip petal is shown in Fig.4. The drawing was started at the upper end of the template, with continuous tracing from the leftmost corner to the end at the lower part of the petal. The movements are shown as the uncorrected trajectory, ranging quite widely with frequent veering from the tulip template. In contrast, the AIBAF-corrected pointer trajectory changes with the curve of the template. The relation between synthesized velocity Vs (i), drawing judgment velocity VD (i) and attenuation weight coefficient w(i) of these data is shown in Fig.5. Additionally Vs (i), VD (i), the uncorrected travel distance, and the AIBAF-corrected travel distance of these data are shown in 6. For the first 8.0 s from the start of the drawing, the user was tracing from the corner part on the upper left to the curve part, and the user’s manipulation including an excessive down-swing was observed. This appears as high Vs (i) and VD (i) levels in the graphs. In this regard, this is intentional manipulation although the Vs (i) value is high, and not a strong involuntary movement as it seemed by observation. The weight and the corrected travel distance decreased with the increase in velocity. For the latter period of drawing, the user was tracing the curve part with weak involuntary movements. The Vs (i) changed around 0.20 [m/s], and w(i) changed in the range of 0.50×10−3 to 2.5×10−3 . According to the above results, the AIBAF-implemented drawing assistance system was able to change the attenuation adapted to the change of velocity corresponding to the state of involuntary movements and 262
Draw end VS VD
245
250
255
260
255
260
Time [s] -3
×10
Weight
(a)Result without assist
Velocity [m/s]
Draw start 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0
3.0 2.5 2.0 1.5 1.0 0.5 0 245
250 Time [s]
Fig. 5. Velocities and attenuation weight changes during the tracing of the left-side curve of the tulip. 5. ERRONEOUS DRAWING PREVENTION WITH RESISTANCE FORCE CONTROL 5.1 Erroneous Drawing in Line-dot Complex Expression A number of erroneous drawings occurred in all user tests (where ”erroneous drawing” is defined as drawing in which the draw start and/or draw end were unintentionally executed). Most of the erroneous drawings were caused by involuntary movements. The occurrence of many erroneous drawings can lead to lengthening of creation, fatigue, tension and significant involuntary movements. Thus, the prevention and reduction of erroneous drawings are desirable. Users with involuntary movements often intentionally undo erroneous drawings. With our drawing assistance system, undos are accomplished using an undo button on the screen. A line or dot is thus undone by an undo operation. When erroneous drawings occurred in the tulip
IFAC HMS 2013 August 11-15, 2013. Las Vegas, USA
Velocity [m/s]
Draw start 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0
by a resistance force controller based on mass, a damper and a spring system with respect to the input point coordinate P (i), the resistance force on each axis F(i) = (Fx (i), Fy (i), Fz (i)) as shown in (5)™ (7).
Draw end VS VD
˙ + KPK (q(i)) ) F(i) = Efc (i)( M¨q(i) + Dq(i) 245
250
255
260
Time [s]
PK (i) =
-3
Deviation [m]
×10 30
Uncorrected
xds (k) − xp (i) yds (k) − yp (i) zds (k) − dz − zp (i)
Efc (i) = diag( Ef cxy (i), Ef cxy (i), Ef cz (i) )
20
(6) (7)
T
10 0
245
250
255
260
Time [s] -3
×10 0.4 Deviation [m]
(5)
Corrected
0.3 0.2 0.1 0
245
250
255
260
Time [s]
Fig. 6. Velocities and travel distance changes during the tracing of the left-side curve of the tulip. tracing test described above, the total number of drawings was 113, and that of the undos was 78, or 69.0% of the total number. We found a tendency for most of the erroneous drawings to be undone, and for all user tests conducted, the proportion of undos to the total drawings was approx. 70.0%. In the user tests, we observed that the causes of erroneous drawings fell into four basic categories: grasping the stylus, moving the pointer, weakness, and unintentional dot expression at the start of a line expression. Here we will focus on the latter category, since it has a direct impact on expression and is the only unintentional draw-start. This type of erroneous drawing is hereinafter called “dotted-error drawing.” In the tests examining the users’ drawings from the starting part of an ogee and tracing the ogee and other test cases, we found that the range of time required between the judgment of the draw start and the draw end in dotted-error drawings is 0.15 to 0.40 s. Dotted-error drawing is suspected to be due to an imbalance among agonist, synergistic and antagonist muscles, as the characteristics of involuntary movements described in Chapter 2 appeared in the present users’ manipulation. In particular, we observed that the down-swing action at the draw start was often excessive and rebounded into an up-swing. We suspect that this is caused by the imbalance between synergistic muscles and agonist muscles, making the user lose motor control. Additionally, in the cases of dotted-error drawing with a non-excessive down-swing, the reaction time of the antagonist muscles to stop the down-swing movement is extended. The up-swings may have been caused by an overrun of antagonist muscles and resulted in an unintentional draw-end judgment. 5.2 Resistance Force Control Suppressing Excess Movements We propose a method that suppresses excessive down-swings and unintentional up-swings at the draw start by controlling the exerted force of a compact haptic interface as an input device to prevent dotted-error drawing. The exerted force is controlled 263
where, q(i) = ( xp (i), yp (i), zp (i) ) is the variable vector. Elastic, viscosity and inertia coefficients on each axis are respectively stored in K=diag( Kx , Ky , Kz )|D=diag( Dx , Dy , Dz )| M=diag( 0, 0, Mz ). Positional functions are stored in PK . A coordinate Pds (k) = (xds (k), yds (k), zds (k)) on an input coordinate system at the moment of the kth draw-start judgment was defined as the draw-start coordinate. A draw-start coordinate is held until the draw-end judgment and updated at every draw-start judgment. dz corresponds to the natural length of the elastic element. The resistance force Fz (i) is exerted against down-swings and up-swings, and Fx (i) and Fy (i) are exerted to suppress operating movements leaving from Pds (k) along the right-and-left and back-and-forth directions. Additionally, to prevent excessive strain from disturbing intentional line and dot expression, the period during the exertion of the resistance force was limited from the draw start. Ef cxy (i) and Ef cz (i) shown in (7) effect time functions determining the limited time for each axis. Ef cxy (i) and Ef cz (i) were 1.5 and 0.60 s, respectively. 5.3 User Test of Drawing Assistance System Integrating the Resistance Force Controller User tests of ogee tracing and dotting were carried out to determine the utility of the resistance force controller for reducing or eliminating dotted-error drawing. In ogee tracing tests, we assessed whether the user could draw a line without any dots at the draw start. A set of this test included three successful traces of the ogee. In dotting tests, we assessed whether the user could intentionally express dots in the case with resistance force. We instructed the user to draw a dot in a circle, and draw dots in five circle in order of preference. A set of this test included successful dotting of all circles. Three ogee tracing tests and three dotting tests, each with and without resistance force were carried out for a total of 12 sets. The parameters of the resistance force controller were set at Kx =Ky =40| Kz =20, Dx =Dy =Dz =4.0| Mz =0.10 and dz =-0.15[m]. In the experimental results for the ogee tracing, the final drawn results on the raster graphics editor without and with resistance force are shown in Fig.7 and Fig.8, respectively. Not all of the drawings remain in these figures, because the undo function was used during the tests. The number of dotted-error drawings in panel (A) of Fig.7 is 22 times; that in (B) is 10 times and that in panel (C) is three. In contrast, the number of dottederror drawings in all tests that used the resistance force was 0. Corresponding to the second trace in Fig.8(B), the changes of exerted resistance force and position on the z-axis are shown in Fig.9. We can see that Fz is exerted for 0.60 s from the draw start, and Fx and Fy are exerted for 1.5 s from the draw start. In the change of Fz , the direction of Fz is positive, opposite to the dropping of the z-axis coordinate at the draw start. A brake
IFAC HMS 2013 August 11-15, 2013. Las Vegas, USA
force against the down-swing is then exerted. Additionally, the direction of Fz is negative, opposite to the rising of the z-axis coordinate at 175.6 s and 176.0 s. The brake force against the early phase of the up-swing is then exerted. (B)
(A)
(C)
Fig. 10. Drawing results: Task = pointing within five circles without force control. (B)
(A)
(C)
Fig. 7. Drawing results: The ogee tracing task without force control. (B)
(A)
(C)
Fig. 11. Drawing results: Task = pointing within five circles with force control. Draw start (B)
(C)
Fig. 8. Drawing results: The task of ogee tracing with force control.
Draw start 3 2 1 0 -1 -2 -3 175
tEFCxy
tEFCz
Position [m]
Force [N] Position [m]
143.5 -3
z force 175.5
176 176.5 Time [s]
177
-3
100
5 4 3 2 1 0 -1 -2 -3 -4 -5
Draw End
×10
tEFCz
tEFCxy
144.5
145
144.5
145
Force [N]
(A)
200 150 100 50 0 -50
×10
144 Time [s]
z position
143.5
144 Time [s]
z position 50
Fig. 12. Change of exerted force of pointing on the No. 4 circle in Fig.11C
0 -50 175
175.5
176 176.5 Time [s]
6. CREATING PAINTINGS UNDER INVOLUNTARY MOVEMENTS
177
Fig. 9. Change of exerted force and z-axis position with force control in Fig.8B In the dotting experiment, the final drawn results on the raster graphics editor without and with resistance force are shown in Fig.10 and Fig.11, respectively. In all six cases, the user achieved dotting, draw start and draw end inside the circle frame, although there were a few failed drawings. Corresponding to the dotting in the left lower circle in Fig.11C, the changes of exerted resistance force and position on the z-axis are shown in Fig.12. The resistance force was exerted as designed, as was the case for the ogee tracing. An up-swing and a draw-end judgment between 0.60 s and 1.5 s from the draw start were performed. According to the above results, our drawing assistance system and integrated AIBAF with resistance force controller did exert resistance force to prevent dotted-error drawing; the system also assisted line expression and did not disturb intentional dot expression. 264
We carried out user tests that the user actually created paintings in user’s own sense with drawing assistance system integrated AIBAF with resistance force controller. The subjects of the paintings chosen by the user were an orange and a four-leaf clover. For the creation of the painting of an orange, we devised a plan in which the user practiced the sequence of creation and how to use the painting materials and techniques by operating a numeric keypad like a mouse that is user’s normal environment, so that the user have the creating sequence and image in advance. In the creation of the four-leaf clover, all plans were designed under the user’s initiative. The following four prevention measures against erroneous drawings were used. 1. Adding a condition VD (i) ≥ 0.30[m/s] to the draw-start judgment to prevent erroneous drawings due to the moving pointer and weakness. 2. Gravity compensation to prevent erroneous drawings due to the operating portion’s own weight. 3. Adding a limited range −45 ≤ ψgbl ≤ 45 [deg] to the draw-start judgment to prevent unintentional draw starts at the stylus’s unfitted orientation. where ψgbl is the angle of the fourth joint of the haptic device. ψgbl =0[deg] when the stylus
IFAC HMS 2013 August 11-15, 2013. Las Vegas, USA
is in parallel with the arm portion. 4. A resistance force controller to prevent dotted error drawing. The parameters in these user tests were Kx =Ky =50| Kz =-10, Dx =Dy =3.0| Dz =4.0| Mz =0.10, dz =-0.15[m]. A completed painting of an orange is shown in Fig.13. The total number of drawings was 1,224 and there were 263 undoings (21.5%). The total time required to create the painting was approx. 4 h. Four types of materials (oil paints, watercolors and two types of palette knife) were used in the painting’s creation. Curves were expressed with over-glazing using the oil paint, and the atmospheric effect was achieved with a palette knife. Thus, mainly curve expressions were created; the hull part of the orange was created in dotted expression. With our system, the user was able to create an orange still-life painting that resembles impressionist art. A completed fourleaf clover painting is shown in Fig.14. The total number of drawings was 798, and there were 129 undoings (16.2%). This percentage of undos was reduced by approx. 50% compared to the system used without the integrated erroneous drawingprevention control. The total creation time for this painting was approx. 2 h. The clover’s leaves and patterns were expressed mainly in curves with the oil paint and palette knife. The shadow parts were expressed in dot aggregate. Finally user could create a painting of clover like illustration. We believe that our proposed system will improve user’s experience because with this system, an individual can use several types of materials and an undo function, and can change the colors on user’s own. In light of the results presented above, we contend that the drawing assistance system with an integrated AIBAF and resistance force controller can prevent error drawings and reduce undos, enabling the creation of paintings of a user’s own design and choosing.
Fig. 13. Orange painted by a user with the drawing assistance system. 7. CONCLUSION We found that our drawing assistance system with an integrated adaptive involuntary behavior attenuation filter that changes attenuation adapted to the users’ involuntary movements and manipulation in real time plus the error-drawing prevention control with a resistance force controller functioned well. Our system can be used by individuals with athetoid cerebral palsy or muscle tension based on involuntary movements. With this system, the number of necessary undos was reduced, and users with nonperiodic complex involuntary movement who have great 265
Fig. 14. Four-leaf clover painted by a user with the drawing assistance system. difficulty painting without assistance were able to make linedot ensemble paintings according to their own design choices. We believe that our method has the capability of integration with devices manipulating actual materials instead of display, and application to handwriting assistance for the users. REFERENCES Aoyama, H., Nakao, T., Miyagawa, N., Kubota, N., Horihata, S., and Yano, K. (2011). Development of drawing assist system for patients with cerebral palsy of the tension athetosis type. Proceedings of IEEE ICRA2011 International Conference on Robotics and Automation, 4664–4669. Chino, N. and Ando, N. (2005). Rehabilitation of Neurogenic Disorders. KANEHARA and CO., Tokyo. Gomi, S. (2005). Rehabilitation Medicine Seminar. Ishiyaku Publishers, Inc., Tokyo. Kaji, R. (2006). Diacrisis and Treatment of Involuntary Monement. SHINDAN-TO-CHIRYOSHA CO., Tokyo. Levine, J.L. and Schappert, M.A. (1945). A mouse adapter for people with hand tremor. IBM SYSTEMS JOURNAL, 623– 624. Morimoto, D., Nawate, M., Watanabe, T., Fukuma, S., and Honda, S. (2004). A painting tool with blurring compensation for people having involuntary hand motion. Technical Report of The Institute of Erectronics, Information and Communication Engineers, 59–64. Nakai, S. (1984). Power grip movement and extension-flexion movement of wrist in cerebral palsied children. Japanese Journal of Special Education, 7–15. Nakao, T., Matsui, H., Yano, K., Kubota, N., Miyagawa, N., and Horihata, S. (2011). Drawing assist system for reducing effects of involuntary movements. Journal of Biomechanical Science and Engineering, 6, 5, 362–377. Shiraishi, M., Kawase, H., and Shinya, M. (2010). A compensation method for hand-drawn illustration by moving average. The Institute of Image Information and Television Engineers Technical Report, 34, 18, 23–26. Sugawara, M. and Takahama, M. (1979). Movement analysis of upper limb of individuals with cerabral palsy. Sogo Rihabiriteshon, 7, 3, 193–200. Yamada, S., Tanioka, T., Okazaki, Y., Watanabe, K., and Kondo, H. (2010). A development of a chin-controllable mouse system for a quadriplegic person by cerebral palsy. IEICE TRANSACTIONS on Information and Systems, J93-D, 10, 2268–2280.