- Email: [email protected]
An application of brain-inspired computing architecture to time series prediction tasks
International Congress Series 1269 (2004) 145 – 148
www.ics-elsevier.com
An application of brain-inspired computing architecture to time series prediction tasks Hiroshi Wakuya * Department of Electrical and Electronic Engineering, Faculty of Science and Engineering, Saga University, Honjo-machi, Saga 840-8502, Japan
Abstract. In general, a biological nervous system consists of two major parts, i.e., a motor control subsystem and a sensory reception subsystem, and they work together cooperatively as direct-inverse transformation. Paying attention to this bi-directional computing manner, it is essential to perform various kinds of complex activities. Also, it must be important in most application-oriented tasks. In order to confirm its effectiveness, a bi-directional computing architecture inspired from abovementioned scheme is constructed and applied to time series prediction tasks. At that time, one subsystem in the bi-directional model is assigned to ordinary future prediction (present ! future), while the other is to past prediction (present ! past) newly introduced in this study. As a result of computer simulations, it is clear that the proposed bi-directional computing architecture shows better performance than the conventional uni-directional one. D 2004 Elsevier B.V. All rights reserved. Keywords: Bi-directional computing architecture; Time series prediction; Future prediction; Past prediction; Information integration
1. Introduction A biological nervous system is a highly sophisticated computing architecture developed through evolution and individual training. It roughly divides into two major parts, the motor control and sensory reception subsystems, which work together cooperatively as direct-inverse transformation. Paying attention to this bi-directional computing manner, it is essential to perform various kinds of complex activities in our body. Also, its analogy must be important in most application-oriented tasks for better performance than the conventional engineering-based approach. 2. Basic concept Fig. 1(a) shows an outline of signal flow in the biological nervous system. With reference to this schematic diagram, a bi-directional neural network model [1] is constructed as a sensorimotor coordinated model for temporal generation and recognition. * Tel.: +81-952-28-8636; fax: +81-952-28-8865. E-mail address: [email protected] (H. Wakuya). 0531-5131/ D 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.ics.2004.05.134
146
H. Wakuya / International Congress Series 1269 (2004) 145–148
Fig. 1. Outline of signal flow in the bi-directional computing architecture.
It is a primitive model around the utterance and auditory systems in the brain, then linguistic processing abilities related to both speaking and hearing are investigated from the viewpoint of temporal signal processing. By the way, in order to evaluate the effectiveness of the proposed bi-directional computing architecture, time series prediction is adopted in this study. It is a simple task to estimate a certain future value based on the present and past history, and its signal transformation is trained by one subsystem in the bi-directional model. In contrast, an additional transformation, whose task is to estimate a certain past value based on the present and past history, is introduced as an inverse transformation, and it is trained by the other subsystem [2]. Following the signal transformation to be trained, each subsystem is referred to as a future/past prediction subsystem hereafter, and its overview is summarized in Fig. 1(b). Then, an improvement of prediction accuracy is expected with the help of future –past information integration. 3. Bi-directional computing architecture for time series prediction Fig. 2 shows an above-mentioned bi-directional model consisting of two mutually connected subsystems [1]. Each of them is a standard recurrent neural network and assigned to the future/past prediction subsystem. The future prediction subsystem, for
Fig. 2. A bi-directional neural network model for time series prediction.
H. Wakuya / International Congress Series 1269 (2004) 145–148
147
example, consists of four layers, whose output signals are denoted as y[0], y[1] y[2], y[3], respectively, and its signal transmission equations are defined as follows: ½0
yi ¼ ½yin i ; X X ½F ½F X ½P ½P ½1 ½1 ½0 yi ¼ f1 wij yj þ wij sj þ wij sj ; j ½F
s
dsj dt
½2 yi
j
½F
j
½1
þ s j ¼ yj ;
¼ f2
X
! ½2 ½1 wij yj
;
j
½yout i ¼
½3 yi
¼ f3
X
! ½3 ½2 wij yj
;
j
f1 ðxÞ ¼ f2 ðxÞ ¼
1 ; 1 þ expðxÞ
f3 ðxÞ ¼ x; where yin and yout are input and output signals of the future prediction subsystem, respectively, and s is a time constant of the dynamic neurons with first-order decay property. Signal transmission equations for the past prediction subsystem are defined similarly as those for the future prediction subsystem. Each subsystem is trained alternatively and can develop temporal signal transformation adequately by minimizing the error between the actual and desired signals. One of the great interests in this study is the advantage of the proposed bi-directional technique against the conventional uni-directional one, therefore the future and past prediction subsystems are separated and trained independently to ascertain its inherent signal processing ability. Also, the future prediction system, which has the same number of hidden neurons as the bi-directional model, is prepared because the network size is proportional to the potential ability. 4. Computer simulations: method and results Several data sets have been tried so far [2 – 4]. The laser intensity time record ‘‘Data Set A’’ is one example, and its result is summarized in Table 1. Each figure in it shows the error averaged over 10 trials and the training success rate. In order to clarify the advantage Table 1 Results of computer simulations
Future prediction system Past prediction system Rate
Bi-directional model [1 – 9 – 9 – 1] 2
Uni-directional model [1 – 9 – 9 – 1]
Uni-directional model [1 – 18 – 18 – 1]
Uni-directional model [1 – 9 – 9 – 1]
1.933 0.247 2/10
4.189 – 0/10
11.423 – 0/10
– 2.128 7/10
The description [1 – x – x – 1] indicates each model’s structure.
148
H. Wakuya / International Congress Series 1269 (2004) 145–148
of the bi-directional computing architecture, an index for improvement quality (IIQ) is defined as ½e IIQ ¼ bidirectional model ; ½eunidirectional model where e is the error at the output layer of the future prediction system. Positive bidirectionalization effect is for IIQ < 1, while negative effect is for IIQ z 1. With this index, the objective bi-directional effect is estimated as follows: 1:933 1:933 c0:461 < 1; IIQf 2 ¼ c0:169 < 1: IIQf 1 ¼ 4:189 11:423 Though the advanced results are not shown here for the sake of brevity, the coupling effect between the future and past prediction subsystems is confirmed on trainability, generalization, and prediction quality. 5. Discussion In this study, a simple intuition inspired from the biological nervous system leads to an improvement of the model’s performance. In spite of further considerations, its detailed mechanism has not been made clear yet. One clue to overcome the difficulties might be in the traditional studies covering various kinds of fields. Following the classification of signal transformation in control engineering, for example, future prediction corresponds literally to prediction and past prediction corresponds to smoothing, respectively. Therefore, rich knowledge around this area might help to discover and understand the mechanism of bi-directional computing. 6. Conclusions In this paper, the brain-inspired computing architecture is applied to time series prediction tasks. As a result, it is found experientially that the proposed bi-directional method is effective not only in the biological system but also in the application-oriented tasks. Acknowledgements This work was partially supported by a Grant-in-Aid for Young Scientists (B) No. 14780297 from the Ministry of Education, Culture, Sports, Science and Technology of Japan. References [1] H. Wakuya, R. Futami, N. Hoshimiya, A bi-directional neural network model for generation and recognition of temporal patterns, IEICE Trans. Inf. Syst. (Jpn. Ed.) J77-D-II (1) (1994) 236 – 243. [2] H. Wakuya, J.M. Zurada, Bi-directional computing architecture for time series prediction, Neural Netw. 14 (9) (2001) 1307 – 1321. [3] H. Wakuya, K. Shida, Time series prediction by a neural network model based on bi-directional computation style: A study on generalization performance with the computer generated time series ‘‘Data Set D’’, Syst. Comput. Jpn. 34 (10) (2003) 64 – 75. [4] H. Wakuya, K. Shida, Time series prediction with a neural network model based on bi-directional computation style: An analytical study and its estimation on acquired signal transformation, Electr. Eng. Jpn. 145 (3) (2003) 50 – 60.
www.ics-elsevier.com
An application of brain-inspired computing architecture to time series prediction tasks Hiroshi Wakuya * Department of Electrical and Electronic Engineering, Faculty of Science and Engineering, Saga University, Honjo-machi, Saga 840-8502, Japan
Abstract. In general, a biological nervous system consists of two major parts, i.e., a motor control subsystem and a sensory reception subsystem, and they work together cooperatively as direct-inverse transformation. Paying attention to this bi-directional computing manner, it is essential to perform various kinds of complex activities. Also, it must be important in most application-oriented tasks. In order to confirm its effectiveness, a bi-directional computing architecture inspired from abovementioned scheme is constructed and applied to time series prediction tasks. At that time, one subsystem in the bi-directional model is assigned to ordinary future prediction (present ! future), while the other is to past prediction (present ! past) newly introduced in this study. As a result of computer simulations, it is clear that the proposed bi-directional computing architecture shows better performance than the conventional uni-directional one. D 2004 Elsevier B.V. All rights reserved. Keywords: Bi-directional computing architecture; Time series prediction; Future prediction; Past prediction; Information integration
1. Introduction A biological nervous system is a highly sophisticated computing architecture developed through evolution and individual training. It roughly divides into two major parts, the motor control and sensory reception subsystems, which work together cooperatively as direct-inverse transformation. Paying attention to this bi-directional computing manner, it is essential to perform various kinds of complex activities in our body. Also, its analogy must be important in most application-oriented tasks for better performance than the conventional engineering-based approach. 2. Basic concept Fig. 1(a) shows an outline of signal flow in the biological nervous system. With reference to this schematic diagram, a bi-directional neural network model [1] is constructed as a sensorimotor coordinated model for temporal generation and recognition. * Tel.: +81-952-28-8636; fax: +81-952-28-8865. E-mail address: [email protected] (H. Wakuya). 0531-5131/ D 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.ics.2004.05.134
146
H. Wakuya / International Congress Series 1269 (2004) 145–148
Fig. 1. Outline of signal flow in the bi-directional computing architecture.
It is a primitive model around the utterance and auditory systems in the brain, then linguistic processing abilities related to both speaking and hearing are investigated from the viewpoint of temporal signal processing. By the way, in order to evaluate the effectiveness of the proposed bi-directional computing architecture, time series prediction is adopted in this study. It is a simple task to estimate a certain future value based on the present and past history, and its signal transformation is trained by one subsystem in the bi-directional model. In contrast, an additional transformation, whose task is to estimate a certain past value based on the present and past history, is introduced as an inverse transformation, and it is trained by the other subsystem [2]. Following the signal transformation to be trained, each subsystem is referred to as a future/past prediction subsystem hereafter, and its overview is summarized in Fig. 1(b). Then, an improvement of prediction accuracy is expected with the help of future –past information integration. 3. Bi-directional computing architecture for time series prediction Fig. 2 shows an above-mentioned bi-directional model consisting of two mutually connected subsystems [1]. Each of them is a standard recurrent neural network and assigned to the future/past prediction subsystem. The future prediction subsystem, for
Fig. 2. A bi-directional neural network model for time series prediction.
H. Wakuya / International Congress Series 1269 (2004) 145–148
147
example, consists of four layers, whose output signals are denoted as y[0], y[1] y[2], y[3], respectively, and its signal transmission equations are defined as follows: ½0
yi ¼ ½yin i ; X X ½F ½F X ½P ½P ½1 ½1 ½0 yi ¼ f1 wij yj þ wij sj þ wij sj ; j ½F
s
dsj dt
½2 yi
j
½F
j
½1
þ s j ¼ yj ;
¼ f2
X
! ½2 ½1 wij yj
;
j
½yout i ¼
½3 yi
¼ f3
X
! ½3 ½2 wij yj
;
j
f1 ðxÞ ¼ f2 ðxÞ ¼
1 ; 1 þ expðxÞ
f3 ðxÞ ¼ x; where yin and yout are input and output signals of the future prediction subsystem, respectively, and s is a time constant of the dynamic neurons with first-order decay property. Signal transmission equations for the past prediction subsystem are defined similarly as those for the future prediction subsystem. Each subsystem is trained alternatively and can develop temporal signal transformation adequately by minimizing the error between the actual and desired signals. One of the great interests in this study is the advantage of the proposed bi-directional technique against the conventional uni-directional one, therefore the future and past prediction subsystems are separated and trained independently to ascertain its inherent signal processing ability. Also, the future prediction system, which has the same number of hidden neurons as the bi-directional model, is prepared because the network size is proportional to the potential ability. 4. Computer simulations: method and results Several data sets have been tried so far [2 – 4]. The laser intensity time record ‘‘Data Set A’’ is one example, and its result is summarized in Table 1. Each figure in it shows the error averaged over 10 trials and the training success rate. In order to clarify the advantage Table 1 Results of computer simulations
Future prediction system Past prediction system Rate
Bi-directional model [1 – 9 – 9 – 1] 2
Uni-directional model [1 – 9 – 9 – 1]
Uni-directional model [1 – 18 – 18 – 1]
Uni-directional model [1 – 9 – 9 – 1]
1.933 0.247 2/10
4.189 – 0/10
11.423 – 0/10
– 2.128 7/10
The description [1 – x – x – 1] indicates each model’s structure.
148
H. Wakuya / International Congress Series 1269 (2004) 145–148
of the bi-directional computing architecture, an index for improvement quality (IIQ) is defined as ½e IIQ ¼ bidirectional model ; ½eunidirectional model where e is the error at the output layer of the future prediction system. Positive bidirectionalization effect is for IIQ < 1, while negative effect is for IIQ z 1. With this index, the objective bi-directional effect is estimated as follows: 1:933 1:933 c0:461 < 1; IIQf 2 ¼ c0:169 < 1: IIQf 1 ¼ 4:189 11:423 Though the advanced results are not shown here for the sake of brevity, the coupling effect between the future and past prediction subsystems is confirmed on trainability, generalization, and prediction quality. 5. Discussion In this study, a simple intuition inspired from the biological nervous system leads to an improvement of the model’s performance. In spite of further considerations, its detailed mechanism has not been made clear yet. One clue to overcome the difficulties might be in the traditional studies covering various kinds of fields. Following the classification of signal transformation in control engineering, for example, future prediction corresponds literally to prediction and past prediction corresponds to smoothing, respectively. Therefore, rich knowledge around this area might help to discover and understand the mechanism of bi-directional computing. 6. Conclusions In this paper, the brain-inspired computing architecture is applied to time series prediction tasks. As a result, it is found experientially that the proposed bi-directional method is effective not only in the biological system but also in the application-oriented tasks. Acknowledgements This work was partially supported by a Grant-in-Aid for Young Scientists (B) No. 14780297 from the Ministry of Education, Culture, Sports, Science and Technology of Japan. References [1] H. Wakuya, R. Futami, N. Hoshimiya, A bi-directional neural network model for generation and recognition of temporal patterns, IEICE Trans. Inf. Syst. (Jpn. Ed.) J77-D-II (1) (1994) 236 – 243. [2] H. Wakuya, J.M. Zurada, Bi-directional computing architecture for time series prediction, Neural Netw. 14 (9) (2001) 1307 – 1321. [3] H. Wakuya, K. Shida, Time series prediction by a neural network model based on bi-directional computation style: A study on generalization performance with the computer generated time series ‘‘Data Set D’’, Syst. Comput. Jpn. 34 (10) (2003) 64 – 75. [4] H. Wakuya, K. Shida, Time series prediction with a neural network model based on bi-directional computation style: An analytical study and its estimation on acquired signal transformation, Electr. Eng. Jpn. 145 (3) (2003) 50 – 60.