- Email: [email protected]
A neural network approach to target classification for active safety system using microwave radar
Expert Systems with Applications 37 (2010) 2340–2346
Contents lists available at ScienceDirect
Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa
A neural network approach to target classification for active safety system using microwave radar Seongkeun Park a, Jae Pil Hwang a, Euntai Kim a,*, Heejin Lee b, Ho Gi Jung c a
School of Electrical and Electronic Engineering, Yonsei University, C613, Sinchon-dong, Seodaemun-gu, Seoul 120-749, Republic of Korea Department of Information and Control Engineering, Hankyong National University, Republic of Korea c Mando Central Research Center, Gyeonggi-do 449-901, Republic of Korea b
a r t i c l e
i n f o
Keywords: Target classification Multilayer perceptron neural network Active safety system Belief update Probabilistic classification
a b s t r a c t As a sensor in the active safety system of vehicles, the microwave radar (MWR) would be a good choice for the localization of the nearby targets but could be a bad choice for their classification or identification. In this paper, a target classification system using a 24 GHz microwave radar sensor is proposed for the active safety system. The basic idea of this paper is that the pedestrians and the vehicles have different reflection characteristics for a microwave. A multilayer perceptron (MLP) neural network is employed to classify the targets and the probabilistic fusion is conduct over time to improve the classification accuracy. Some experiments are performed to show the validity of the proposed system. Ó 2009 Elsevier Ltd. All rights reserved.
1. Introduction Safety has been a hot issue in recent vehicular technology and a tremendous research has been conducted towards the direction. The researches concerning the safety of the passengers and drivers in the vehicles have produced two paradigms: passive safety system and active safety system. The passive safety system purposes to minimize the damage after car accident and an air bag and safety belt belong to this class (Chan, 2007; Watanabe, Umezawa, & Abe, 1994). The passive system does not aim at reducing the possibility of the car accidents. On the contrary, active safety system purposes to prevent the car accidents before they occur and it is now receiving much attention within vehicular community. Mainly, the active safety system recognizes the surrounding environment around its own car and alerts the car driver about the nearby possible dangers. The road sign recognition system (Nguwi & Kouzani, 2006; Yoon, Lee, Kim, & Park, 2008), and the blind spot warning system (Krips, Velten, Kummert, & Teuner, 2004; US Patent 6859148, 2005), the adaptive cruise control (ACC) system (Bifulco, Simonelli, & Di Pace, 2008; Ioannou & Stefanovic, 2005; Wang, Zhang, & Bubb, 2007) and pedestrian protection systems (PPS) (Gandhi & Trivedi, 2007) certainly belong to the active safety system. In the active safety system, the key technology is the understanding of the surrounding objects, that is, detection, tracking and identification of the nearby objects. For the purpose, several * Corresponding author. E-mail address: [email protected] (E. Kim). 0957-4174/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2009.07.070
sensors are used and CCD cameras and range finders are the most common ones. The CCD camera returns rich information about the nearby target objects and provides relatively easy method for target recognition. However, it is difficult to measure the range to the nearby targets from the car. On the contrary, range finders such as a laser scanner or microwave radar easily measure the location of nearby objects and are robust to the variation of the weather or time. But the range finders have difficulty in recognizing the target objects (Fuerstenberg & Dietmayer, 2004; Fuerstenberg & Willhoeft, 2001; Maclachlan & Mertz, 2006). In this paper, we develop a new microwave radar-based target classification system for an active safety system. We do not use a CCD camera but use only a 24 GHz microwave radar to classify the nearby objects. Thus, with a single radar, we can fulfill both target tracking and target recognition simultaneously at no extra cost. The basic idea of the target recognition by a microwave radar is that the pedestrians and the vehicles have different reflection characteristics for a microwave. Based on the idea, we build a classifier using a multilayer perceptron neural network (MLP). Further, we present a probabilistic fusion method to make a classification decision based on not only the current measurement but also all the past measurements. The remaining of this paper is organized as follows: in Section 2, the microwave radar sensor and the experimental setup are explained. In Section 3, the problem is formulated and a neural network classifier is designed for target classification. In Section 4, the neural network classifier results are fused over time to improve the classification result. Finally, some conclusions are drawn in Section 5.
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
2341
signal power and many features of a target is summarized in Radar Cross Section (RCS) equations (Knott, Shaeffer, & Tuley, 1993). Even though the relations expressed in equations are available, we do not use them directly since we cannot know the shape, size, or reflectivity of the nearby targets. Instead, we gather some samples from the radar and train the classifiers to predict the objects.
3. Pedestrian and vehicle classification by multilayer perceptron neural network 3.1. Multilayer perceptron neural network classifier
Fig. 1. Configuration of active safety system.
2. 24 GHz microwave radar sensor In this paper, a 24 GHz microwave radar (MWR) named MASRAU0025 is employed to sense and classify targets around the vehicle. The radar is manufactured by M/A-COM and is composed of a 24 GHz radar sensor and microprocessor unit (Manual of MA-COM MASRAU0025, 2005). Fig. 1 shows the configuration of our experimental setup for target classification. As in Fig. 1, the 24 GHz microwave sensor MASRAU0025 is mounted on the car and is connected to a computer through CAN (Control Area Network). In the system, we use a laptop computer equipped with P-IV and the program is run on Window XP. Fig. 2 shows the experimental setup in which two microwave radars are mounted on the vehicle to be connected to a computer. The basic idea of this paper is to use the power of the microwave signal that is emitted from and returns to the radar, reflected from the targets, and classify the nearby objects. It means how much power of the microwave is attenuated in decibel when it is reflected from the target. The received power is very sensitive and varies depending on fascia material, paint, dirt and mud buildup, foam material or other mechanical obstructions in the sensor field of view. The relationship between the received
Neural networks imitate the human brains and acquire knowledge about the hidden relationships between input and output directly from samples (Duda, Hart, & Stork, 2001; Hagan, Demuth, & Beale, 1995). Fig. 3 shows a three layer MLP with input, hidden, and output layers. Suppose we are given data set X ¼ fx1 ; x2 ; . . . ; xN g and each data T xn ¼ xn1 ; xn2 ; . . . ; xnm 2 Rm belongs to one of C classes xl ðl ¼ 1; 2; . . . CÞ, where C is the number of classes, xk s are disjointed and X ¼ x1 [ x2 [ . . . [ xC . Then the output of neural networks shown in Fig. 3 is represented by
T NN NN g NN ðxn Þ ¼ g NN 1 ðxn Þ; g 2 ðxn Þ; . . . g C ðxn Þ ! ! nh m X X NN oh hi n wlj f2 wji xi þ bjo þ blo ; g l ðxn Þ ¼ f1 j¼1
ð1Þ ð2Þ
i¼1
oh where whi ji and wlj are weight vector between ith input node and jth hidden node, and between jth hidden node and lth output node, respectively. nh is the number of hidden node, f ðÞ is activation function. We train the MLP such that the output node associated with the true class returns ‘‘1” and all remaining nodes return ‘‘0”. Thus, we select the target values by
tðxn Þ ¼ ½t1 ; t 2 ; . . . tC T 1 if xn 2 xl tl ðxn Þ ¼ 0 otherwise
ð3Þ ð4Þ
As in Duda et al. (2001), we train the MLP such that the following function is minimized:
Fig. 2. Experimental setup in which 24 GHz microwave radar is mounted.
2342
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
dxn þ ¼
C X
Z
pðxn ; xl Þdxn
pðxl Þ
Z
2 g NN l ðxn Þ pðxn jxl Þ pðxn Þdxn
l¼1
þ
Z
pðxl jxn Þpðxi–l jxn Þpðxn Þdxn
ð6Þ
The second term of (6) does not involve W and thus we obtain
g NN l ðxn Þ pðxl jxn Þ
That is, when the MLP is configured and trained as in (3) through (6), it returns the posterior probability of each class. From the posteriors, we determine the class by
Fig. 3. Multilayer perceptron with three layers.
JðWÞ ¼
N X
C X
g NN l ðxn Þ t l ðxn Þ
2
l ¼ arg max g NN l ðxn Þ arg max pðxl jxn Þ: l
n¼1 l¼1
¼
C X
X 2 X NN 2 g NN þ g l ðxn Þ 0 l ðxn Þ 1 xn 2xl
l¼1
ð8Þ
3.2. Target classification using microwave radar
xn Rxl
C X 2 N N l Nl 1 X NN 1 g l ðxn Þ 1 þ N N Nl N N l x 2 x l¼1 n l ! X 2 g NN l ðxn Þ 0
ð5Þ
xn Rxl
where W is weight matrix and N l is the number of data which belongs class xl . As N approaches infinity, we apply Bayes rule, then
Z C X NN 2 1 pðxl Þ g l ðxn Þ 1 pðxn jxl Þdxn JðWÞ ~JðWÞ ¼ N!1 N l¼1 Z NN 2 þ pðxi–l Þ g l ðxn Þ 0 pðxn jxi–l Þdxn lim
Z Z C X NN 2 pðxl Þ g l ðxn Þ pðxn Þdxn 2 g NN l ðxn Þpðxn ; xl Þ l¼1
l
!
¼N
ð7Þ
Instead of using Radar Cross Section (RCS) equations (Knott et al., 1993), we collect the training data from the vehicles and pedestrians and build an MLP for classification since the received power from the microwave depends on many factors of the target. We change the location of the target from 0m to 30 ms with an interval of 1 m, from 30° to +30° with the interval of 15° and measure the received power of the microwave for pedestrians and vehicles. Fig. 4 shows the received power in case of zero bearing from both vehicles and pedestrians. It can be observed that the received power is noisy and varies with respect to the distance between radar and objects. As it can be seen in the figure, there is a tendency that the vehicles have larger received power than the pedestrians at the same distance and this is the key feature that we exploit in this paper. We collect 1853 vehicle samples and 2202 pedestrian samples and train an MLP as shown in Fig. 5.
Fig. 4. Received signal power (Zero bearing).
2343
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
(PÞ; X1:t ¼ fx1 ; x2 ; . . . xt g denotes an accumulated sequence of relative distance, bearing and received powers from time 1 up to t. We name the new quantity as belief, and it is the accumulated posterior probability of x ¼ l (l can be taken as a vehicle (V) or a pedestrian (P)) conditioned on all current and past measurements. This belief measure is motivated by Thrun, Bugard, and Fox (2005). As in Thrun et al. (2005), we rewrite the belief of vehicle class into
belt ðx ¼ VÞ ¼ pðx ¼ VjX1:t Þ ¼
pðxt jx ¼ V; X1:t1 Þpðx ¼ VjX1:t1 Þ pðxt jX1:t1 Þ
Since the current measurement xt is independent of X1:t1 conditioned on x, we can rewrite (11) into
Fig. 5. MLP classifier.
belt ðx ¼ VÞ ¼
Table 1 Correct classification rate (CCR) of 10 runs with sevenfold cross-validation using the MLP classifier.
Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Set 7 Overall
Mean
Variance
0.7814 0.8245 0.8474 0.7759 0.8465 0.7562 0.7693 0.8002
0.0142 0.0034 0.0023 0.0175 0.0018 0.0195 0.0176 0.0110
pðxt jX1:t1 Þ
ð12Þ
pðxt jx ¼ VÞ ¼
pðx ¼ Vjxt Þpðxt Þ pðx ¼ VÞ
ð13Þ
into (12) yields
belt ðx ¼ VÞ ¼
pðx ¼ Vjxt Þpðxt Þ pðx ¼ VjX1:t1 Þ pðx ¼ VÞ pðxt jX1:t1 Þ
ð14Þ
In the same way, we compute the belief at time t for x ¼ P
We use a three MLP with 25 nodes in the hidden layer. We conduct sevenfold cross-validation and repeat it 10 times to test the statistical reliability. Table 1 shows the result of 10 runs of sevenfold cross-validation. The microwave radar is noisy in nature, as shown in Fig. 4, and often returns outliers. The overall CCR (correct classification rate) reaches 0.8002, as shown in Table 1. The results are not bad but not satisfactory, either. In the next section, we improve the classification rate by considering the problem in a probabilistic framework.
pðx ¼ Pjxt Þpðxt Þ pðx ¼ PjX1:t1 Þ pðx ¼ PÞ pðxt jX1:t1 Þ
ð15Þ
Then, dividing (14) by (15) and canceling some intractable terms yields
belt ðx ¼ VÞ pðx ¼ Vjxt Þ pðx ¼ VjX1:t1 Þ pðx ¼ PÞ ¼ belt ðx ¼ PÞ pðx ¼ Pjxt Þ pðx ¼ PjX1:t1 Þ pðx ¼ VÞ
ð16Þ
By plugging the outputs of the MLP t t g NN l ðx Þ pðxl jx Þ
ð17Þ
into (16), we obtain the following sequential equation for the beliefs:
4. Probabilistic target classification The target classification based on MLP sometimes makes a wrong decision due to outliers produced by the microwave radar. To solve this problem, we utilize not only the current measurement but also all the previous measurements in a probabilistic framework to make a decision about the class of the target. Here, the measurement means the features used in the MLP classification: distance, bearing and received signal power. More specifically, in Section 3, we used the MLP
ð9Þ
in making a decision at time t and it actually was the posterior probability based on the current measurement (Duda et al., 2001). In this section, we define an accumulated posterior probability at time t
belt ðxÞ ¼ pðxjX1:t Þ
pðxt jx ¼ VÞpðx ¼ VjX1:t1 Þ
Plugging the Bayes rule,
belt ðx ¼ PÞ ¼
t t g NN l ðx Þ pðxl jx Þ
ð11Þ
ð10Þ
based on not only the current measurement xt but also all the previous measurements X1:t1 , where x is a binary variable denoting whether the target is a vehicle (VÞ or a pedestrian
belt ðx ¼ VÞ pðx ¼ Vjxt Þ pðx ¼ VjX1:t1 Þ 1 pðx ¼ VÞ ¼ t 1 belt ðx ¼ VÞ 1 pðx ¼ Vjx Þ 1 pðx ¼ VjX1:t1 Þ pðx ¼ VÞ ¼
t g NN belt1 ðx ¼ VÞ 1 bel0 ðx ¼ VÞ x¼V ðx Þ ¼ qtV NN 1 g x¼V ðxt Þ 1 belt1 ðx ¼ VÞ bel0 ðx ¼ VÞ
ð18Þ Thus, from the previous belief belt1 ðx ¼ VÞ (or belt1 ðx ¼ PÞÞ and the current output of the MLP, we can update the current belief and we do not have to store all the measurement X1:t1 to compute the belief. Once qtV is obtained, belt ðx ¼ lÞ can be recovered by
qtV 1 þ qtV qtP belt ðx ¼ PÞ ¼ 1 þ qtP
belt ðx ¼ VÞ ¼
ð19Þ ð20Þ
Then, we can make a classification based on belief by
x ¼ arg max½belt ðx ¼ lÞ ¼ arg max½pðx ¼ ljX1:t Þ l¼P;V
l¼P;V
ð21Þ
2344
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
Fig. 6. The proposed probabilistic classification strategy for the active safety system.
The probabilistic classification strategy is summarized in Fig. 6. In summary, the MLP makes soft decision on the target based on the current radar outputs and the soft decision of the MLP is fused over time in the Bayesian probabilistic framework as shown in Fig. 6. 5. Experimental results In this section, we conduct some experiments to show the effectiveness of the proposed method. In the probabilistic classification, we set both bel0 ðx ¼ VÞ and bel0 ðx ¼ PÞ to 0.5 since we have no prior information about the target. We use the same MLP trained from 1853 vehicle samples and 2202 pedestrian samples in the previous section. Fig. 7 shows our experimental setup in which
both the vehicle and the pedestrian move at almost the same speed. MLP makes a soft decision and it is fused over time when the targets are moving. Fig. 8a and b shows specific examples of probabilistic classification result based on the MLP for a vehicle and a pedestrian, respectively. As can be seen in the figures, the MLP often makes wrong decisions because the microwave radar is noisy but the probabilistic fusion compensates misbehavior of the MLP successfully by taking into account the previous measurements in the probabilistic framework. As in Section 3, we conduct sevenfold cross-validation and repeat it 10 times to test the statistical reliability. Table 2 shows the result of the probabilistic classification for sevenfold cross-validation. The overall CCR reaches 0.9278 and is improved from the MLP by almost 0.1. 6. Conclusion In this paper, we have developed a target classification for active safety system. We implemented it using only using a microwave radar without the help of any CCD camera. Since we used only a 24 GHz radar, we can fulfill both target tracking and recognition simultaneously at no extra cost. Our system was built and its validity was demonstrated through a real world experimentation. The contribution of this paper is threefold: (1) Target classification system for an active safety system is implemented with only microwave radar and a power reflected from the target is used as key feature. (2) An MLP neural network is trained to classify the target based on the radar outputs. (3) The classification performance is greatly improved by combining the classification results over time in the probabilistic framework.
Fig. 7. A pedestrian and a vehicle moving at the same speed.
However, more works still should be done such that our system would be more stabilized and be installed in commercial vehicles.
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
2345
Fig. 8. Comparison of MLP classifier and probabilistic classifier.
Acknowledgements
References
This work is supported by Mando Co., Active Protection Pedestrian System Project.
Bifulco, G. N., Simonelli, F., & Di Pace, R. (2008). Experiments toward an human-like Adaptive Cruise Control. In Intelligent vehicles symposium (pp. 919–924).
2346
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
Table 2 CCR of 10 runs of sevenfold cross-validation using probabilistic classifier.
Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Set 7 Overall
Mean
Variance
0.8729 1.0000 0.9159 0.8836 0.9086 0.9407 0.9726 0.9278
0.0453 0 0.0316 0.0289 0.0177 0.0122 0.0058 0.0200
Chan, C. Y. (2007). Trends in crash detection and occupant restraint technology. IEEE Proceedings, 95(2), 388–396. Duda, R., Hart, P., & Stork, D. (2001). Pattern classification. Addison Wesley. Fuerstenberg, K. & Dietmayer, K. (2004). Object tracking and classification for multiple active safety and comfort applications using a multilayer laser scanner. In IEEE intelligence vehicle symposium (pp. 802–807). Parma, Italy. Fuerstenberg, K., & Willhoeft, V. (2001). Object tracking and classification using laserscanners – Pedestrian recognition in urban environment. In Intelligence transportation systems conference (pp. 451–453). Oakland, USA. Gandhi, T., & Trivedi, M. M. (2007). Pedestrian protection systems: Issues, survey, and challenges. IEEE Tranactions on Intelligence Transportation Systems, 8(3), 413–430.
Hagan, M. T., Demuth, H. B., & Beale, H. (1995). Neural network design. USA: PWS Publishing Company. Ioannou, P. A., & Stefanovic, M. (2005). Evaluation of ACC vehicles in mixed traffic: Lane change effects and sensitivity analysis. IEEE Transactions on Intelligence Transportation Systems, 6(1), 79–89. Knott, E. F., Shaeffer, J. F., & Tuley, M. T. (1993). Radar cross section. Boston: Artech House. Krips, M., Velten, J., Kummert, A., & Teuner, A. (2004). AdTM tracking for blind spot collision avoidance. In Intelligent vehicles symposium, 2004 IEEE 14–17 (pp. 544– 548). Maclachlan, R., & Mertz, C. (2006). Tracking of moving objects from a moving vehicle using a scanning laser rangefinder. In Intelligence transportation systems conference (pp. 301–306), Toronto, Canada. Manual of MA-COM MASRAU0025, 2005. Nguwi, Y.-Y., & Kouzani, A. Z. (2006). A study on automatic recognition of road signs. In 2006 IEEE conference on cybernetics and intelligent systems (pp. 1–6). Thrun, S., Bugard, W., & Fox, D. (2005). Probabilistic robotics. Cambridge, MA: MIT Press. US Patent 6859148. (2005). Blind spot warning system for an automotive vehicle. US Patent Issued on February 22, 2005. Wang, W., Zhang, W., & Bubb, H. (2007). Car-following safety algorithms based on adaptive cruise control strategies. In Intelligent systems and informatics, 2007. SISY 2007. Fifth international symposium (pp.135–140). Watanabe, K., Umezawa, Y., & Abe, K. (1994). Advanced passive safety system via prediction and sensor fusion. In Vehicle navigation and information systems conference (pp. 435–440). Yoon, C., Lee, H., Kim, E., & Park, M. (2008). Real-time road sign detection using fuzzy-boosting. IEICE Transactions on Fundamentals of Electronics Communications and Computer Sciences, E91-A(11), 3346–3355.
Contents lists available at ScienceDirect
Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa
A neural network approach to target classification for active safety system using microwave radar Seongkeun Park a, Jae Pil Hwang a, Euntai Kim a,*, Heejin Lee b, Ho Gi Jung c a
School of Electrical and Electronic Engineering, Yonsei University, C613, Sinchon-dong, Seodaemun-gu, Seoul 120-749, Republic of Korea Department of Information and Control Engineering, Hankyong National University, Republic of Korea c Mando Central Research Center, Gyeonggi-do 449-901, Republic of Korea b
a r t i c l e
i n f o
Keywords: Target classification Multilayer perceptron neural network Active safety system Belief update Probabilistic classification
a b s t r a c t As a sensor in the active safety system of vehicles, the microwave radar (MWR) would be a good choice for the localization of the nearby targets but could be a bad choice for their classification or identification. In this paper, a target classification system using a 24 GHz microwave radar sensor is proposed for the active safety system. The basic idea of this paper is that the pedestrians and the vehicles have different reflection characteristics for a microwave. A multilayer perceptron (MLP) neural network is employed to classify the targets and the probabilistic fusion is conduct over time to improve the classification accuracy. Some experiments are performed to show the validity of the proposed system. Ó 2009 Elsevier Ltd. All rights reserved.
1. Introduction Safety has been a hot issue in recent vehicular technology and a tremendous research has been conducted towards the direction. The researches concerning the safety of the passengers and drivers in the vehicles have produced two paradigms: passive safety system and active safety system. The passive safety system purposes to minimize the damage after car accident and an air bag and safety belt belong to this class (Chan, 2007; Watanabe, Umezawa, & Abe, 1994). The passive system does not aim at reducing the possibility of the car accidents. On the contrary, active safety system purposes to prevent the car accidents before they occur and it is now receiving much attention within vehicular community. Mainly, the active safety system recognizes the surrounding environment around its own car and alerts the car driver about the nearby possible dangers. The road sign recognition system (Nguwi & Kouzani, 2006; Yoon, Lee, Kim, & Park, 2008), and the blind spot warning system (Krips, Velten, Kummert, & Teuner, 2004; US Patent 6859148, 2005), the adaptive cruise control (ACC) system (Bifulco, Simonelli, & Di Pace, 2008; Ioannou & Stefanovic, 2005; Wang, Zhang, & Bubb, 2007) and pedestrian protection systems (PPS) (Gandhi & Trivedi, 2007) certainly belong to the active safety system. In the active safety system, the key technology is the understanding of the surrounding objects, that is, detection, tracking and identification of the nearby objects. For the purpose, several * Corresponding author. E-mail address: [email protected] (E. Kim). 0957-4174/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2009.07.070
sensors are used and CCD cameras and range finders are the most common ones. The CCD camera returns rich information about the nearby target objects and provides relatively easy method for target recognition. However, it is difficult to measure the range to the nearby targets from the car. On the contrary, range finders such as a laser scanner or microwave radar easily measure the location of nearby objects and are robust to the variation of the weather or time. But the range finders have difficulty in recognizing the target objects (Fuerstenberg & Dietmayer, 2004; Fuerstenberg & Willhoeft, 2001; Maclachlan & Mertz, 2006). In this paper, we develop a new microwave radar-based target classification system for an active safety system. We do not use a CCD camera but use only a 24 GHz microwave radar to classify the nearby objects. Thus, with a single radar, we can fulfill both target tracking and target recognition simultaneously at no extra cost. The basic idea of the target recognition by a microwave radar is that the pedestrians and the vehicles have different reflection characteristics for a microwave. Based on the idea, we build a classifier using a multilayer perceptron neural network (MLP). Further, we present a probabilistic fusion method to make a classification decision based on not only the current measurement but also all the past measurements. The remaining of this paper is organized as follows: in Section 2, the microwave radar sensor and the experimental setup are explained. In Section 3, the problem is formulated and a neural network classifier is designed for target classification. In Section 4, the neural network classifier results are fused over time to improve the classification result. Finally, some conclusions are drawn in Section 5.
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
2341
signal power and many features of a target is summarized in Radar Cross Section (RCS) equations (Knott, Shaeffer, & Tuley, 1993). Even though the relations expressed in equations are available, we do not use them directly since we cannot know the shape, size, or reflectivity of the nearby targets. Instead, we gather some samples from the radar and train the classifiers to predict the objects.
3. Pedestrian and vehicle classification by multilayer perceptron neural network 3.1. Multilayer perceptron neural network classifier
Fig. 1. Configuration of active safety system.
2. 24 GHz microwave radar sensor In this paper, a 24 GHz microwave radar (MWR) named MASRAU0025 is employed to sense and classify targets around the vehicle. The radar is manufactured by M/A-COM and is composed of a 24 GHz radar sensor and microprocessor unit (Manual of MA-COM MASRAU0025, 2005). Fig. 1 shows the configuration of our experimental setup for target classification. As in Fig. 1, the 24 GHz microwave sensor MASRAU0025 is mounted on the car and is connected to a computer through CAN (Control Area Network). In the system, we use a laptop computer equipped with P-IV and the program is run on Window XP. Fig. 2 shows the experimental setup in which two microwave radars are mounted on the vehicle to be connected to a computer. The basic idea of this paper is to use the power of the microwave signal that is emitted from and returns to the radar, reflected from the targets, and classify the nearby objects. It means how much power of the microwave is attenuated in decibel when it is reflected from the target. The received power is very sensitive and varies depending on fascia material, paint, dirt and mud buildup, foam material or other mechanical obstructions in the sensor field of view. The relationship between the received
Neural networks imitate the human brains and acquire knowledge about the hidden relationships between input and output directly from samples (Duda, Hart, & Stork, 2001; Hagan, Demuth, & Beale, 1995). Fig. 3 shows a three layer MLP with input, hidden, and output layers. Suppose we are given data set X ¼ fx1 ; x2 ; . . . ; xN g and each data T xn ¼ xn1 ; xn2 ; . . . ; xnm 2 Rm belongs to one of C classes xl ðl ¼ 1; 2; . . . CÞ, where C is the number of classes, xk s are disjointed and X ¼ x1 [ x2 [ . . . [ xC . Then the output of neural networks shown in Fig. 3 is represented by
T NN NN g NN ðxn Þ ¼ g NN 1 ðxn Þ; g 2 ðxn Þ; . . . g C ðxn Þ ! ! nh m X X NN oh hi n wlj f2 wji xi þ bjo þ blo ; g l ðxn Þ ¼ f1 j¼1
ð1Þ ð2Þ
i¼1
oh where whi ji and wlj are weight vector between ith input node and jth hidden node, and between jth hidden node and lth output node, respectively. nh is the number of hidden node, f ðÞ is activation function. We train the MLP such that the output node associated with the true class returns ‘‘1” and all remaining nodes return ‘‘0”. Thus, we select the target values by
tðxn Þ ¼ ½t1 ; t 2 ; . . . tC T 1 if xn 2 xl tl ðxn Þ ¼ 0 otherwise
ð3Þ ð4Þ
As in Duda et al. (2001), we train the MLP such that the following function is minimized:
Fig. 2. Experimental setup in which 24 GHz microwave radar is mounted.
2342
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
dxn þ ¼
C X
Z
pðxn ; xl Þdxn
pðxl Þ
Z
2 g NN l ðxn Þ pðxn jxl Þ pðxn Þdxn
l¼1
þ
Z
pðxl jxn Þpðxi–l jxn Þpðxn Þdxn
ð6Þ
The second term of (6) does not involve W and thus we obtain
g NN l ðxn Þ pðxl jxn Þ
That is, when the MLP is configured and trained as in (3) through (6), it returns the posterior probability of each class. From the posteriors, we determine the class by
Fig. 3. Multilayer perceptron with three layers.
JðWÞ ¼
N X
C X
g NN l ðxn Þ t l ðxn Þ
2
l ¼ arg max g NN l ðxn Þ arg max pðxl jxn Þ: l
n¼1 l¼1
¼
C X
X 2 X NN 2 g NN þ g l ðxn Þ 0 l ðxn Þ 1 xn 2xl
l¼1
ð8Þ
3.2. Target classification using microwave radar
xn Rxl
C X 2 N N l Nl 1 X NN 1 g l ðxn Þ 1 þ N N Nl N N l x 2 x l¼1 n l ! X 2 g NN l ðxn Þ 0
ð5Þ
xn Rxl
where W is weight matrix and N l is the number of data which belongs class xl . As N approaches infinity, we apply Bayes rule, then
Z C X NN 2 1 pðxl Þ g l ðxn Þ 1 pðxn jxl Þdxn JðWÞ ~JðWÞ ¼ N!1 N l¼1 Z NN 2 þ pðxi–l Þ g l ðxn Þ 0 pðxn jxi–l Þdxn lim
Z Z C X NN 2 pðxl Þ g l ðxn Þ pðxn Þdxn 2 g NN l ðxn Þpðxn ; xl Þ l¼1
l
!
¼N
ð7Þ
Instead of using Radar Cross Section (RCS) equations (Knott et al., 1993), we collect the training data from the vehicles and pedestrians and build an MLP for classification since the received power from the microwave depends on many factors of the target. We change the location of the target from 0m to 30 ms with an interval of 1 m, from 30° to +30° with the interval of 15° and measure the received power of the microwave for pedestrians and vehicles. Fig. 4 shows the received power in case of zero bearing from both vehicles and pedestrians. It can be observed that the received power is noisy and varies with respect to the distance between radar and objects. As it can be seen in the figure, there is a tendency that the vehicles have larger received power than the pedestrians at the same distance and this is the key feature that we exploit in this paper. We collect 1853 vehicle samples and 2202 pedestrian samples and train an MLP as shown in Fig. 5.
Fig. 4. Received signal power (Zero bearing).
2343
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
(PÞ; X1:t ¼ fx1 ; x2 ; . . . xt g denotes an accumulated sequence of relative distance, bearing and received powers from time 1 up to t. We name the new quantity as belief, and it is the accumulated posterior probability of x ¼ l (l can be taken as a vehicle (V) or a pedestrian (P)) conditioned on all current and past measurements. This belief measure is motivated by Thrun, Bugard, and Fox (2005). As in Thrun et al. (2005), we rewrite the belief of vehicle class into
belt ðx ¼ VÞ ¼ pðx ¼ VjX1:t Þ ¼
pðxt jx ¼ V; X1:t1 Þpðx ¼ VjX1:t1 Þ pðxt jX1:t1 Þ
Since the current measurement xt is independent of X1:t1 conditioned on x, we can rewrite (11) into
Fig. 5. MLP classifier.
belt ðx ¼ VÞ ¼
Table 1 Correct classification rate (CCR) of 10 runs with sevenfold cross-validation using the MLP classifier.
Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Set 7 Overall
Mean
Variance
0.7814 0.8245 0.8474 0.7759 0.8465 0.7562 0.7693 0.8002
0.0142 0.0034 0.0023 0.0175 0.0018 0.0195 0.0176 0.0110
pðxt jX1:t1 Þ
ð12Þ
pðxt jx ¼ VÞ ¼
pðx ¼ Vjxt Þpðxt Þ pðx ¼ VÞ
ð13Þ
into (12) yields
belt ðx ¼ VÞ ¼
pðx ¼ Vjxt Þpðxt Þ pðx ¼ VjX1:t1 Þ pðx ¼ VÞ pðxt jX1:t1 Þ
ð14Þ
In the same way, we compute the belief at time t for x ¼ P
We use a three MLP with 25 nodes in the hidden layer. We conduct sevenfold cross-validation and repeat it 10 times to test the statistical reliability. Table 1 shows the result of 10 runs of sevenfold cross-validation. The microwave radar is noisy in nature, as shown in Fig. 4, and often returns outliers. The overall CCR (correct classification rate) reaches 0.8002, as shown in Table 1. The results are not bad but not satisfactory, either. In the next section, we improve the classification rate by considering the problem in a probabilistic framework.
pðx ¼ Pjxt Þpðxt Þ pðx ¼ PjX1:t1 Þ pðx ¼ PÞ pðxt jX1:t1 Þ
ð15Þ
Then, dividing (14) by (15) and canceling some intractable terms yields
belt ðx ¼ VÞ pðx ¼ Vjxt Þ pðx ¼ VjX1:t1 Þ pðx ¼ PÞ ¼ belt ðx ¼ PÞ pðx ¼ Pjxt Þ pðx ¼ PjX1:t1 Þ pðx ¼ VÞ
ð16Þ
By plugging the outputs of the MLP t t g NN l ðx Þ pðxl jx Þ
ð17Þ
into (16), we obtain the following sequential equation for the beliefs:
4. Probabilistic target classification The target classification based on MLP sometimes makes a wrong decision due to outliers produced by the microwave radar. To solve this problem, we utilize not only the current measurement but also all the previous measurements in a probabilistic framework to make a decision about the class of the target. Here, the measurement means the features used in the MLP classification: distance, bearing and received signal power. More specifically, in Section 3, we used the MLP
ð9Þ
in making a decision at time t and it actually was the posterior probability based on the current measurement (Duda et al., 2001). In this section, we define an accumulated posterior probability at time t
belt ðxÞ ¼ pðxjX1:t Þ
pðxt jx ¼ VÞpðx ¼ VjX1:t1 Þ
Plugging the Bayes rule,
belt ðx ¼ PÞ ¼
t t g NN l ðx Þ pðxl jx Þ
ð11Þ
ð10Þ
based on not only the current measurement xt but also all the previous measurements X1:t1 , where x is a binary variable denoting whether the target is a vehicle (VÞ or a pedestrian
belt ðx ¼ VÞ pðx ¼ Vjxt Þ pðx ¼ VjX1:t1 Þ 1 pðx ¼ VÞ ¼ t 1 belt ðx ¼ VÞ 1 pðx ¼ Vjx Þ 1 pðx ¼ VjX1:t1 Þ pðx ¼ VÞ ¼
t g NN belt1 ðx ¼ VÞ 1 bel0 ðx ¼ VÞ x¼V ðx Þ ¼ qtV NN 1 g x¼V ðxt Þ 1 belt1 ðx ¼ VÞ bel0 ðx ¼ VÞ
ð18Þ Thus, from the previous belief belt1 ðx ¼ VÞ (or belt1 ðx ¼ PÞÞ and the current output of the MLP, we can update the current belief and we do not have to store all the measurement X1:t1 to compute the belief. Once qtV is obtained, belt ðx ¼ lÞ can be recovered by
qtV 1 þ qtV qtP belt ðx ¼ PÞ ¼ 1 þ qtP
belt ðx ¼ VÞ ¼
ð19Þ ð20Þ
Then, we can make a classification based on belief by
x ¼ arg max½belt ðx ¼ lÞ ¼ arg max½pðx ¼ ljX1:t Þ l¼P;V
l¼P;V
ð21Þ
2344
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
Fig. 6. The proposed probabilistic classification strategy for the active safety system.
The probabilistic classification strategy is summarized in Fig. 6. In summary, the MLP makes soft decision on the target based on the current radar outputs and the soft decision of the MLP is fused over time in the Bayesian probabilistic framework as shown in Fig. 6. 5. Experimental results In this section, we conduct some experiments to show the effectiveness of the proposed method. In the probabilistic classification, we set both bel0 ðx ¼ VÞ and bel0 ðx ¼ PÞ to 0.5 since we have no prior information about the target. We use the same MLP trained from 1853 vehicle samples and 2202 pedestrian samples in the previous section. Fig. 7 shows our experimental setup in which
both the vehicle and the pedestrian move at almost the same speed. MLP makes a soft decision and it is fused over time when the targets are moving. Fig. 8a and b shows specific examples of probabilistic classification result based on the MLP for a vehicle and a pedestrian, respectively. As can be seen in the figures, the MLP often makes wrong decisions because the microwave radar is noisy but the probabilistic fusion compensates misbehavior of the MLP successfully by taking into account the previous measurements in the probabilistic framework. As in Section 3, we conduct sevenfold cross-validation and repeat it 10 times to test the statistical reliability. Table 2 shows the result of the probabilistic classification for sevenfold cross-validation. The overall CCR reaches 0.9278 and is improved from the MLP by almost 0.1. 6. Conclusion In this paper, we have developed a target classification for active safety system. We implemented it using only using a microwave radar without the help of any CCD camera. Since we used only a 24 GHz radar, we can fulfill both target tracking and recognition simultaneously at no extra cost. Our system was built and its validity was demonstrated through a real world experimentation. The contribution of this paper is threefold: (1) Target classification system for an active safety system is implemented with only microwave radar and a power reflected from the target is used as key feature. (2) An MLP neural network is trained to classify the target based on the radar outputs. (3) The classification performance is greatly improved by combining the classification results over time in the probabilistic framework.
Fig. 7. A pedestrian and a vehicle moving at the same speed.
However, more works still should be done such that our system would be more stabilized and be installed in commercial vehicles.
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
2345
Fig. 8. Comparison of MLP classifier and probabilistic classifier.
Acknowledgements
References
This work is supported by Mando Co., Active Protection Pedestrian System Project.
Bifulco, G. N., Simonelli, F., & Di Pace, R. (2008). Experiments toward an human-like Adaptive Cruise Control. In Intelligent vehicles symposium (pp. 919–924).
2346
S. Park et al. / Expert Systems with Applications 37 (2010) 2340–2346
Table 2 CCR of 10 runs of sevenfold cross-validation using probabilistic classifier.
Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Set 7 Overall
Mean
Variance
0.8729 1.0000 0.9159 0.8836 0.9086 0.9407 0.9726 0.9278
0.0453 0 0.0316 0.0289 0.0177 0.0122 0.0058 0.0200
Chan, C. Y. (2007). Trends in crash detection and occupant restraint technology. IEEE Proceedings, 95(2), 388–396. Duda, R., Hart, P., & Stork, D. (2001). Pattern classification. Addison Wesley. Fuerstenberg, K. & Dietmayer, K. (2004). Object tracking and classification for multiple active safety and comfort applications using a multilayer laser scanner. In IEEE intelligence vehicle symposium (pp. 802–807). Parma, Italy. Fuerstenberg, K., & Willhoeft, V. (2001). Object tracking and classification using laserscanners – Pedestrian recognition in urban environment. In Intelligence transportation systems conference (pp. 451–453). Oakland, USA. Gandhi, T., & Trivedi, M. M. (2007). Pedestrian protection systems: Issues, survey, and challenges. IEEE Tranactions on Intelligence Transportation Systems, 8(3), 413–430.
Hagan, M. T., Demuth, H. B., & Beale, H. (1995). Neural network design. USA: PWS Publishing Company. Ioannou, P. A., & Stefanovic, M. (2005). Evaluation of ACC vehicles in mixed traffic: Lane change effects and sensitivity analysis. IEEE Transactions on Intelligence Transportation Systems, 6(1), 79–89. Knott, E. F., Shaeffer, J. F., & Tuley, M. T. (1993). Radar cross section. Boston: Artech House. Krips, M., Velten, J., Kummert, A., & Teuner, A. (2004). AdTM tracking for blind spot collision avoidance. In Intelligent vehicles symposium, 2004 IEEE 14–17 (pp. 544– 548). Maclachlan, R., & Mertz, C. (2006). Tracking of moving objects from a moving vehicle using a scanning laser rangefinder. In Intelligence transportation systems conference (pp. 301–306), Toronto, Canada. Manual of MA-COM MASRAU0025, 2005. Nguwi, Y.-Y., & Kouzani, A. Z. (2006). A study on automatic recognition of road signs. In 2006 IEEE conference on cybernetics and intelligent systems (pp. 1–6). Thrun, S., Bugard, W., & Fox, D. (2005). Probabilistic robotics. Cambridge, MA: MIT Press. US Patent 6859148. (2005). Blind spot warning system for an automotive vehicle. US Patent Issued on February 22, 2005. Wang, W., Zhang, W., & Bubb, H. (2007). Car-following safety algorithms based on adaptive cruise control strategies. In Intelligent systems and informatics, 2007. SISY 2007. Fifth international symposium (pp.135–140). Watanabe, K., Umezawa, Y., & Abe, K. (1994). Advanced passive safety system via prediction and sensor fusion. In Vehicle navigation and information systems conference (pp. 435–440). Yoon, C., Lee, H., Kim, E., & Park, M. (2008). Real-time road sign detection using fuzzy-boosting. IEICE Transactions on Fundamentals of Electronics Communications and Computer Sciences, E91-A(11), 3346–3355.