- Email: [email protected]
An estimation of driver's drowsiness level using interval of steering adjustment for lane keeping
197
Technical notes /JSAE Review 16 (1995) 185-199
Technical Notes
An estimation of driver's drowsiness level using interval of steering adjustment for lane keeping Junichi Fukuda, Eisaku Akutsu, Keiji Aoki Future Project Division, No. 1, Toyota Motor Co., Ltd., Mishuku 1200, Susono-shi, Shizuoka, 410-11 Japan
Received21 September1994
1. Introduction Automobile safety requirements have greatly intensified in recent years. The prevention of accidents caused by driver's drowsiness in particular is crucial, as such drowsiness has a high risk of causing serious accidents. The authors selected a steering sensor as the sensor to detect drowsiness, for development of a proper system for the prevention of accidents caused by drowsiness, taking account of the following requirements. Namely, a type that does not come into contact with the driver, high reliability of sensor signals and a low cost of the entire system. Moreover, particular attention was paid to the relationship between drowsiness and the fluctuation of steering adjustment intervals. Specifically, characteristic features of steering adjustment were defined, the steering adjustment points were identified and intervals for individual steering adjustments were extracted. Further, drowsiness judgment levels were set according to the difference in vehicle speed and individual drivers, and a proper algorithm was constructed accordingly. This algorithm was evaluated on a test course and favorable data were obtained, which is also reported in this paper.
2. Steering adjustment driver model Figure 1 shows the driver-vehicle model. The vehicle lateral displacement, yaw angle, vehicle speed, the course of the vehicle, etc. are assumed on the driver's input side, while the steering angle is used on the output side in this model. The driver recognizes the vehicle lateral displacement, yaw angle, vehicle speed, the course ahead of vehicle, etc. according to the visual information, and judges whether he or she should make a proper steering adjustment or further
recognitions again. This is referred to as the judgment system. Hence the following two phenomena will be affected by the drowsiness of the driver: - sampling period to proceed from the judgment system to further recognitions; and - judgment for the necessity to proceed from the judgment system to a steering adjustment. If drowsiness occurs, therefore the interval of steering adjustment will be prolonged. In this regard, the authors propose a simple method capable of extracting the interval of steering adjustment by means of the steering angle alone, and estimating the level of drowsiness according to the change in the interval.
3. Steering adjustment recognition method The steering waveform contains the reaction force from the road surface, disturbances from the environment, road curvature etc. in addition to the component of steering adjustment. Therefore, the authors have used a waveform recognition method in order to extract the steering adjustment component alone.
DRIVER
VEHICLE
~ter~ ~Y
¢
Fig. 1. Driver-vehiclemodelfor steeringadjustment.
0389-4304/95/$09.50 © 1995 Societyof AutomotiveEngineersof Japan, Inc. and ElsevierScienceB.V. All rights reserved SSDI 0389-4304(94)00070-0
JSAE9532263
198
Technical notes /JSAE Review 16 (1995) 185-199
'••&"
(n-k)
8 (n)
# (n+k)
768
,
(ms)
Pa~em B
. [ Time (sec)
PatternC
Pattern D
Fig. 2. Steering adjustment extraction pattern.
The following was found from the on-vehicle measurement data: - minimum value of steering adjustment angle: 0.5 (deg.), - minimum value of steering adjustment interval: 0.5 (s). Hence the pattern shown in Fig. 2 is devised as the waveform recognition pattern capable of extracting the steering adjustment component alone as much as possible. Specifically, the pattern A is expressed as follows, assuming that one sample point of steering angle is /5 (n): tS(n) > 8 ( n -
k),
6 ( n ) - a, X k / 6 > 6 ( n + k), (k=l
to 6),
constant a , = 0 . 5 .
n, which meets all of the above, is recognized as in steering adjustment points. The resolution of the steering angle sensor actually used in this study is 0.1 deg, and the sampling period is 128 ms.
4. Drowsiness
estimation
method
4.1. Application of drowsiness estimation algorithm to vehicle speed variation Evaluation results of steering adjustment interval variation caused by the vehicle speed variation are shown in Fig. 3. Only one example is shown here, but the same tendency is found over the entire data in terms of inclination caused by the vehicle speed variation, though the absolute values of steering adjustment intervals differ from driver to driver. Another characteristic feature is that the amplitude of
variation in steering adjustment interval becomes greater as the vehicle speed becomes lower. Therefore, the following two features are applied to the drowsiness estimation algorithm against the vehicle speed variation: (1) In order not to impair the real time performance of the drowsiness judgment system, the curve shown in Fig. 3 is normalized, and the steering adjustment interval is converted into the interval at the vehicle speed of 80 k m / h regardless of the actual vehicle speed at that time. (2) The drowsiness judgment threshold level is made variable according to the vehicle speed at a given time, since the amplitude of the steering adjustment interval variation in the driver's normal state is apt to change according to the vehicle speed. (Coefficient of correction against the fluctuation of steering adjustment interval in the normal state according to the variation of vehicle speed: Gv(V)).
4.2. Application of drowsiness estimation algorithm to drivers' individual differences When considering the application of the algorithm to individual differences among drivers, it is necessary to take account of the following two points: (1) It is necessary for the system to learn the steering adjustment intervals of individual drivers according to the vehicle speeds, and to make the learning time shorter as the vehicle speed becomes higher where the fluctuation in the interval becomes smaller. It is also necessary to add a feature so that the steering adjustment data that could not be learned within a given period of time can be estimated by the conversion coefficient determined with the normalization of the curve shown in Fig. 3. (2) It is necessary to set the drowsiness judgment threshold level according to the values of steering adjustment intervals learned by the system, since the amplitude of change from the normal state to the state of drowsiness has strong correlationship with the values of the steering adjustment intervals inherent to each driver in the normal state. (Coefficient of drowsiness judgment according to the absolute values of steering adjustment intervals of each driver: GI(Dr).)
4.3. Drowsiness judgment method 2.4
D
By summarizing what has been described in Sections 4.1 and 4.2, it may be said that the actual drowsiness can be judged by
2.2
m
De>Gt(Dr) X Gv(V)
-.,.,
O O
XD
r
where 1.2
40
60
80
100
120
Vehicle Speed (kin/h)
Fig. 3. Vehicle speed-steering adjustment interval characteristics.
D(i) = steering adjustment interval, = mean value of learned steering adjustment intervals in the normal state,
Dr
Technical notes/JSAE Review 16 (1995) 185-199
De
= mean value of most recent steering adjustment intervals,
Table 1 Coincidence of the estimated light drowsiness levels with a-wave and driver's declaration
E7 , o ( i ) Dr-
n
(n = steering adjustment period within period to learn normal value),
EiLm pD( i) oeP (m = most recent interval, m > n, p = number of intervals averaged out). The value of p is set at 40 in this analysis. The steering angle where both the initial and final steering angles for adjustment are within the tolerance of _ 10° is assumed as the estimatable steering angle. For reference,, the rate of time meeting this steering angle is approximately 99% on highways.
5. Results of analysis and evaluation In order to confirm the accuracy of the proposed algorithm, ten drivers were evaluated for an hour while driving at almost 80 k m / h . The drivers declared their estimated drowsiness: Alertness, Light drowsiness or Drowsiness. Alertness means that they are wide awake, Light Drowsiness that they feel sleepy and they don't want to drive very much and Drowsiness that they feel extremely sleepy and they can hardly drive. At the same time, the a-wave rate, which is one of the physiological signals, was measured. Figure 4 and Table 1 show the evaluation result of one driver. Figure 4 shows the evaluation data of drowsiness judgment algorithm. Table 1 shows the quantitative evaluation data, in terms of percentage of time, in which the estimated values of drowsiness judged by the drowsiness judgement levels are compared with the median value of a-wave rate and the value of drowsiness declared by each driver.
~ 2.0 ~ 1 1 ~ ~ _
~
~
Drowsiness threshold
(1.88sec) 4--LightDrowsiness threshold (1.55ses)
199
Correct Error Estimated level: drowsiness Estimated level: alertness
a-Wave
Driver's declaration
76%
88%
15%
7%
9%
5%
As found from Fig. 4, the trend of the estimated values of drowsiness agrees very well with those of a-wave rate and the values declared by the drivers. The setting of drowsiness judgment levels also appear to be correct. According to the quantitative evaluation data shown in Table 1, the rate of correct answers is 76 to 88%, which is also a favorable result. Similarly, favorable results are obtained from the evaluation of other drivers. Ten drivers were subjected to the sensory evaluation test using this technique, with warnings given over two stages. The results agreed relatively well with most of the drivers' actual feelings of drowsiness. The method to estimate the drowsiness from the steering adjustment inter"val is effective, according to the data described in the foregoing.
6. Conclusions The authors have succeeded in accurate extraction of the steering adjustment points defined as the driver's component of steering adjustment, by means of pattern recognition. It was verified through experiments that accurate estimation of drowsiness can be done by adding appropriate parameters such as driver's individual differences and vehicle speed to the steering adjustment interval. It was confirmed that drowsiness judgement data agree relatively well with driver's feelings. Future objectives are to enhance the drowsiness judgment method and to increase the accuracy on ordinary roads.
4.-- LNht Drowsiness Alertness
Reference o Wave Rate when human closed his eyes and rested
,o
~o
~o Time
4'o
(min)
~o
Fig. 4. Estimation of drowsiness level.
[1] H. Iguchi, A. Kozato: EEG variation at long-term monotonous work, Jpn. J. Ergonomics, Vol. 29, 380-381, 1993. [2] H. Watanabe, Y. Koike, N. Sakurai, A. Takahashi, H. Iguchi: An analysis of EEG changes during mental working, Jpn. J. EEG EMG, Vol. 19 No. 3, 253-263, 1991.
Technical notes /JSAE Review 16 (1995) 185-199
Technical Notes
An estimation of driver's drowsiness level using interval of steering adjustment for lane keeping Junichi Fukuda, Eisaku Akutsu, Keiji Aoki Future Project Division, No. 1, Toyota Motor Co., Ltd., Mishuku 1200, Susono-shi, Shizuoka, 410-11 Japan
Received21 September1994
1. Introduction Automobile safety requirements have greatly intensified in recent years. The prevention of accidents caused by driver's drowsiness in particular is crucial, as such drowsiness has a high risk of causing serious accidents. The authors selected a steering sensor as the sensor to detect drowsiness, for development of a proper system for the prevention of accidents caused by drowsiness, taking account of the following requirements. Namely, a type that does not come into contact with the driver, high reliability of sensor signals and a low cost of the entire system. Moreover, particular attention was paid to the relationship between drowsiness and the fluctuation of steering adjustment intervals. Specifically, characteristic features of steering adjustment were defined, the steering adjustment points were identified and intervals for individual steering adjustments were extracted. Further, drowsiness judgment levels were set according to the difference in vehicle speed and individual drivers, and a proper algorithm was constructed accordingly. This algorithm was evaluated on a test course and favorable data were obtained, which is also reported in this paper.
2. Steering adjustment driver model Figure 1 shows the driver-vehicle model. The vehicle lateral displacement, yaw angle, vehicle speed, the course of the vehicle, etc. are assumed on the driver's input side, while the steering angle is used on the output side in this model. The driver recognizes the vehicle lateral displacement, yaw angle, vehicle speed, the course ahead of vehicle, etc. according to the visual information, and judges whether he or she should make a proper steering adjustment or further
recognitions again. This is referred to as the judgment system. Hence the following two phenomena will be affected by the drowsiness of the driver: - sampling period to proceed from the judgment system to further recognitions; and - judgment for the necessity to proceed from the judgment system to a steering adjustment. If drowsiness occurs, therefore the interval of steering adjustment will be prolonged. In this regard, the authors propose a simple method capable of extracting the interval of steering adjustment by means of the steering angle alone, and estimating the level of drowsiness according to the change in the interval.
3. Steering adjustment recognition method The steering waveform contains the reaction force from the road surface, disturbances from the environment, road curvature etc. in addition to the component of steering adjustment. Therefore, the authors have used a waveform recognition method in order to extract the steering adjustment component alone.
DRIVER
VEHICLE
~ter~ ~Y
¢
Fig. 1. Driver-vehiclemodelfor steeringadjustment.
0389-4304/95/$09.50 © 1995 Societyof AutomotiveEngineersof Japan, Inc. and ElsevierScienceB.V. All rights reserved SSDI 0389-4304(94)00070-0
JSAE9532263
198
Technical notes /JSAE Review 16 (1995) 185-199
'••&"
(n-k)
8 (n)
# (n+k)
768
,
(ms)
Pa~em B
. [ Time (sec)
PatternC
Pattern D
Fig. 2. Steering adjustment extraction pattern.
The following was found from the on-vehicle measurement data: - minimum value of steering adjustment angle: 0.5 (deg.), - minimum value of steering adjustment interval: 0.5 (s). Hence the pattern shown in Fig. 2 is devised as the waveform recognition pattern capable of extracting the steering adjustment component alone as much as possible. Specifically, the pattern A is expressed as follows, assuming that one sample point of steering angle is /5 (n): tS(n) > 8 ( n -
k),
6 ( n ) - a, X k / 6 > 6 ( n + k), (k=l
to 6),
constant a , = 0 . 5 .
n, which meets all of the above, is recognized as in steering adjustment points. The resolution of the steering angle sensor actually used in this study is 0.1 deg, and the sampling period is 128 ms.
4. Drowsiness
estimation
method
4.1. Application of drowsiness estimation algorithm to vehicle speed variation Evaluation results of steering adjustment interval variation caused by the vehicle speed variation are shown in Fig. 3. Only one example is shown here, but the same tendency is found over the entire data in terms of inclination caused by the vehicle speed variation, though the absolute values of steering adjustment intervals differ from driver to driver. Another characteristic feature is that the amplitude of
variation in steering adjustment interval becomes greater as the vehicle speed becomes lower. Therefore, the following two features are applied to the drowsiness estimation algorithm against the vehicle speed variation: (1) In order not to impair the real time performance of the drowsiness judgment system, the curve shown in Fig. 3 is normalized, and the steering adjustment interval is converted into the interval at the vehicle speed of 80 k m / h regardless of the actual vehicle speed at that time. (2) The drowsiness judgment threshold level is made variable according to the vehicle speed at a given time, since the amplitude of the steering adjustment interval variation in the driver's normal state is apt to change according to the vehicle speed. (Coefficient of correction against the fluctuation of steering adjustment interval in the normal state according to the variation of vehicle speed: Gv(V)).
4.2. Application of drowsiness estimation algorithm to drivers' individual differences When considering the application of the algorithm to individual differences among drivers, it is necessary to take account of the following two points: (1) It is necessary for the system to learn the steering adjustment intervals of individual drivers according to the vehicle speeds, and to make the learning time shorter as the vehicle speed becomes higher where the fluctuation in the interval becomes smaller. It is also necessary to add a feature so that the steering adjustment data that could not be learned within a given period of time can be estimated by the conversion coefficient determined with the normalization of the curve shown in Fig. 3. (2) It is necessary to set the drowsiness judgment threshold level according to the values of steering adjustment intervals learned by the system, since the amplitude of change from the normal state to the state of drowsiness has strong correlationship with the values of the steering adjustment intervals inherent to each driver in the normal state. (Coefficient of drowsiness judgment according to the absolute values of steering adjustment intervals of each driver: GI(Dr).)
4.3. Drowsiness judgment method 2.4
D
By summarizing what has been described in Sections 4.1 and 4.2, it may be said that the actual drowsiness can be judged by
2.2
m
De>Gt(Dr) X Gv(V)
-.,.,
O O
XD
r
where 1.2
40
60
80
100
120
Vehicle Speed (kin/h)
Fig. 3. Vehicle speed-steering adjustment interval characteristics.
D(i) = steering adjustment interval, = mean value of learned steering adjustment intervals in the normal state,
Dr
Technical notes/JSAE Review 16 (1995) 185-199
De
= mean value of most recent steering adjustment intervals,
Table 1 Coincidence of the estimated light drowsiness levels with a-wave and driver's declaration
E7 , o ( i ) Dr-
n
(n = steering adjustment period within period to learn normal value),
EiLm pD( i) oeP (m = most recent interval, m > n, p = number of intervals averaged out). The value of p is set at 40 in this analysis. The steering angle where both the initial and final steering angles for adjustment are within the tolerance of _ 10° is assumed as the estimatable steering angle. For reference,, the rate of time meeting this steering angle is approximately 99% on highways.
5. Results of analysis and evaluation In order to confirm the accuracy of the proposed algorithm, ten drivers were evaluated for an hour while driving at almost 80 k m / h . The drivers declared their estimated drowsiness: Alertness, Light drowsiness or Drowsiness. Alertness means that they are wide awake, Light Drowsiness that they feel sleepy and they don't want to drive very much and Drowsiness that they feel extremely sleepy and they can hardly drive. At the same time, the a-wave rate, which is one of the physiological signals, was measured. Figure 4 and Table 1 show the evaluation result of one driver. Figure 4 shows the evaluation data of drowsiness judgment algorithm. Table 1 shows the quantitative evaluation data, in terms of percentage of time, in which the estimated values of drowsiness judged by the drowsiness judgement levels are compared with the median value of a-wave rate and the value of drowsiness declared by each driver.
~ 2.0 ~ 1 1 ~ ~ _
~
~
Drowsiness threshold
(1.88sec) 4--LightDrowsiness threshold (1.55ses)
199
Correct Error Estimated level: drowsiness Estimated level: alertness
a-Wave
Driver's declaration
76%
88%
15%
7%
9%
5%
As found from Fig. 4, the trend of the estimated values of drowsiness agrees very well with those of a-wave rate and the values declared by the drivers. The setting of drowsiness judgment levels also appear to be correct. According to the quantitative evaluation data shown in Table 1, the rate of correct answers is 76 to 88%, which is also a favorable result. Similarly, favorable results are obtained from the evaluation of other drivers. Ten drivers were subjected to the sensory evaluation test using this technique, with warnings given over two stages. The results agreed relatively well with most of the drivers' actual feelings of drowsiness. The method to estimate the drowsiness from the steering adjustment inter"val is effective, according to the data described in the foregoing.
6. Conclusions The authors have succeeded in accurate extraction of the steering adjustment points defined as the driver's component of steering adjustment, by means of pattern recognition. It was verified through experiments that accurate estimation of drowsiness can be done by adding appropriate parameters such as driver's individual differences and vehicle speed to the steering adjustment interval. It was confirmed that drowsiness judgement data agree relatively well with driver's feelings. Future objectives are to enhance the drowsiness judgment method and to increase the accuracy on ordinary roads.
4.-- LNht Drowsiness Alertness
Reference o Wave Rate when human closed his eyes and rested
,o
~o
~o Time
4'o
(min)
~o
Fig. 4. Estimation of drowsiness level.
[1] H. Iguchi, A. Kozato: EEG variation at long-term monotonous work, Jpn. J. Ergonomics, Vol. 29, 380-381, 1993. [2] H. Watanabe, Y. Koike, N. Sakurai, A. Takahashi, H. Iguchi: An analysis of EEG changes during mental working, Jpn. J. EEG EMG, Vol. 19 No. 3, 253-263, 1991.