- Email: [email protected]
Feature Extraction of NWP Data for Wind Power Forecasting Using 3D-Convolutional Neural Networks
ScienceDirect ScienceDirect Energy Procedia 00 (2018) 000–000
Available online at www.sciencedirect.com
Availableonline onlineatatwww.sciencedirect.com www.sciencedirect.com Available Energy Procedia 00 (2018) 000–000
ScienceDirect ScienceDirect
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
Energy (2018) 000–000 350–358 EnergyProcedia Procedia155 00 (2017) www.elsevier.com/locate/procedia 12th International Renewable Energy Storage Conference, IRES 2018
International Renewable Energy Storage Conference, IRES 2018 Feature12th Extraction of NWP Data for Wind Power Forecasting Using 3D-Convolutional Neural Networks Feature Extraction of NWP Data for Wind Power Forecasting The 15th International Symposium on District Heating and Cooling Using 3D-Convolutional Networks a a Kazutoshi Higashiyama *, Yu FujimotoNeural , Yasuhiro Hayashia Assessing theWaseda feasibility using thea,169-8555, heat demand-outdoor a of University, 3-4-1 Shinjuku, Tokyo, Japan Kazutoshi Higashiyama *,Okubo Yu Fujimoto Yasuhiro Hayashia temperature function for a long-term district heat demand forecast Waseda University, 3-4-1 Okubo Shinjuku, Tokyo, 169-8555, Japan a
a
Abstract
I. Andrića,b,c*, A. Pinaa, P. Ferrãoa, J. Fournierb., B. Lacarrièrec, O. Le Correc
Abstract Wind power is one of the most attractive forms of electricity from the viewpoints of cost efficiency and environmental protection. a IN+ Center for Innovation, Technology andaPolicy Research Instituto Rovisco Paisforecasting 1, 1049-001will Lisbon, However, the instability of wind power has serious impact- on a gridSuperior system.Técnico, ReliableAv.wind power helpPortugal to utilize b Veolia Recherche & Innovation, Avenue Daniel, Limay, Wind power is one of the most attractive forms of electricity from theDreyfous viewpoints of 78520 cost and environmental storage systems and backup generators effectively for 291 mitigating the instability. Thisefficiency paperFrance proposes a feature protection. extraction c Systèmes Énergétiques et Environnement 4 rue Alfred Nantes, France However, instability wind power has a serious impact on -a IMT grid system. Reliable windKastler, power44300 forecasting will help to utilize procedure the forDépartement numericalofweather prediction (NWP) data based on Atlantique, the three-dimensional convolutional neural networks (3Dstorage and backup generators for extract mitigating the instability. This paper proposes a feature extraction CNNs). systems An advantage of 3D-CNNs is toeffectively automatically the spatio-temporal features from NWP data focusing on the targeted wind farm. Feature extraction based(NWP) on 3D-CNNs wasonapplied to real-world datasets; the results significant procedure for numerical weather prediction data based the three-dimensional convolutional neuralshow networks (3Dperformance in comparison to severalisbenchmark approaches, andthe also show that the features proposedfrom extraction based on CNNs). An advantage of 3D-CNNs to automatically extract spatio-temporal NWP scheme data focusing on 3Dthe Abstract targetedachieves wind farm. Feature extraction on 3D-CNNs was applied to real-world datasets; CNNs to derive intrinsic featuresbased for prediction of wind power generation from NWP data. the results show significant performance in comparison to several benchmark approaches, and also show that the proposed extraction scheme based on 3DDistrict heating networks are commonly addressed in the literature as one of the most effective solutions for decreasing the CNNs achieves to derive intrinsic features for prediction of wind power generation from NWP data. greenhouse gas emissions from the building sector. These systems require high investments which are returned through the heat © 2018 The Authors. Published by Elsevier Ltd. sales. Due to the changed climate conditions renovation policies, heat demand in the future could decrease, © 2018 The Authors. by Elsevier Ltd. and building This is an open accessPublished article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) prolonging the access investment return period. Selection andAuthors. peer-review underbyresponsibility © 2018 The Published Elsevier Ltd.of the scientific committee of the 12th International Renewable Energy Storage Selection peer-review under responsibility of the scientific of the –12th International Renewable The mainand scope of this paper is to assess the feasibility of usingcommittee the heat demand outdoor temperature functionEnergy for heatStorage demand This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Conference. Conference. forecast. The district of Alvalade, located in Lisbon (Portugal), was used as a case study. The district is consisted of 665 Selection and peer-review under responsibility of the scientific committee of the 12th International Renewable Energy Storage buildings that vary in both construction period and typology. Three weather scenarios (low, medium, high) and three district Keywords: wind power forecasting; feature extraction; convolutional neural networks; deep learning; numerical weather prediction Conference. renovation scenarios were developed (shallow, intermediate, deep). To estimate the error, obtained heat demand values were compared with results from a dynamic heat demand model, previously developed and validated by the authors. Keywords: wind power forecasting; feature extraction; convolutional neural networks; deep learning; numerical weather prediction results showed that when only weather change is considered, the margin of error could be acceptable for some applications 1.The Introduction (the error in annual demand was lower than 20% for all weather scenarios considered). However, after introducing renovation the error value increased up to 59.5% (depending on the weather and renovation scenarios combination considered). 1.scenarios, Introduction Wind as coefficient an alternative to fossil fuels, has made substantial contribution to adecade, more sustainable society. It The valueenergy, of slope increased on average within thea range of 3.8% up to 8% per that corresponds to the has been introduced worldwide a way to reduceduring greenhouse gas season emission. According to combination the Global Wind Energy decrease in the number of heatingas hours of 22-139h the heating (depending on the of weather and Wind energy, as an alternative fossil made intercept a substantial contribution to a more sustainable society. It renovation scenarios considered). Ontothe otherfuels, hand,has function increased for 7.8-12.7% per decade (depending on the has been introduced worldwide as a way to reduce greenhouse gas emission. According to the Global Wind Energy coupled scenarios). The values suggested could be used to modify the function parameters for the scenarios considered, and improve the accuracy of heat demand estimations. © 2017 The Authors. Published by Elsevier Ltd. * Corresponding author. Tel.: +81-3-5286-8122; fax: +81-3-5286-8035. Peer-review under responsibility of the Scientific Committee of The 15th International Symposium on District Heating and E-mail address: [email protected] Cooling. * Corresponding author. Tel.: +81-3-5286-8122; fax: +81-3-5286-8035.
1876-6102 © 2018 The Authors. Published by Elsevier Ltd. E-mail address: [email protected] Keywords: Heat demand; Forecast; Climate change This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Selection and peer-review under responsibility of the scientific 1876-6102 © 2018 The Authors. Published by Elsevier Ltd. committee of the 12th International Renewable Energy Storage Conference. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Selection and peer-review under responsibility of the scientific committee of the 12th International Renewable Energy Storage Conference.
1876-6102 © 2017 The Authors. Published by Elsevier Ltd. 1876-6102 © 2018 The Authors. Published by Elsevier Ltd. Peer-review under responsibility of the Scientific Committee of The 15th International Symposium on District Heating and Cooling. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Selection and peer-review under responsibility of the scientific committee of the 12th International Renewable Energy Storage Conference. 10.1016/j.egypro.2018.11.043
2
Kazutoshi Higashiyama et al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018) 000–000
351
Council [1], the cumulative capacity of wind power reached 486 GW across the global market in 2016. Wind power is expected to significantly expand leading to an overall zero emission power system. One of the greatest challenges in large-scale penetration of wind power is to control its unstable output. This is a serious problem for grid operators since supply and demand power have to be balanced. In particular, rapid output changes may result in frequency deviation of a grid system. One of the most effective solutions to this problem is to introduce reliable wind power forecasting (WPF). WPF helps to optimize the operation schedule of storage systems and backup generators for mitigating the instabilities. Several WPF approaches have been proposed [2]. Among them, numerical weather predictions (NWP) models have been widely used [3, 4] . NWP models calculate meteorological variables, such as wind speeds, for each gridded point by solving complex simultaneous equations relating to the atmosphere. Machine learning approaches utilizing the NWP results have also been actively discussed [5, 6] so as to improve the forecasting accuracy; this approach achieves to model the relationship between the meteorological variables obtained by the NWP model and the observed power generation output. A major obstacle is to handle the high dimensionality of NWP data; i.e., machine learning approaches may not perform appropriately in the case that the volume of input data is substantially large. This is so-called the “curse of dimensionality” [7]. To deal with the problem, several studies explored a feature extraction scheme for NWP data. Davò et al. [8] used principal components analysis (PCA) to obtain the dimensionally reduced NWP dataset. The dataset was applied to the analog ensemble technique and neural networks as input for deterministic and probabilistic WPF. Andrade et al. [9] studied feature engineering techniques using NWP grid domain knowledge. In the previous study, we have focused on the feature extraction scheme based on two-dimensional convolutional neural networks (2D-CNNs) [10]. Our feature extraction scheme has achieved to obtain low-dimensional features from a large region of spatial NWP data corresponding to the output of targeted wind farms. In this paper, we explore the feature extraction scheme based on three-dimensional convolutional neural networks (3D-CNNs). The 3D-CNNs can handle not only spatial dimensions but also a temporal dimension. Therefore, the 3D-CNNs are expected to capture the dynamics of wind contained in NWP data by focusing on time slices in a certain period. This paper is organized as follows: section 2 describes the datasets and a procedure for WPF, section 3 describes the proposed feature extraction architecture, section 4 explains the configuration of numerical experiments, section 5 discusses the evaluation results, and section 6 concludes the paper. 2. Wind power forecasting framework 2.1. Wind farm The wind power dataset used in this paper is collected from an onshore wind farm located in the Tohoku region, which is the north-eastern area of Japan. The wind power output was measured every 10 seconds and collected from April 2015 to July 2017. The dataset was converted into per unit system. 2.2. Numerical Weather predictions (NWP) data The NWP data were provided by the Numerical Weather Forecasting and Analysis System (NuWFAS) [11]. The NuWFAS is based on the Weather Research and Forecasting (WRF) [12], which is the mesoscale meteorological model. A plurality of frames included in the NWP data continue along a time axis. The temporal resolution is 30 minutes; the data are provided by using boundary condition given at 00:00 UTC and cover the time horizons from 30 minutes to 72 hours ahead. The NWP data cover the Tohoku area, and contain forecasted meteorological variables on a 5-km horizontal grid. Although the NWP data contain several kinds of meteorological variables, the zonal and meridional components of wind speed vectors at 60 meters above sea level are employed in this study. Fig. 1 describes the grid of the NWP data. The rectangular grid includes 50 × 50 = 2,500 points around the targeted wind power plant.
Kazutoshi Higashiyama et al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018) 000–000
352
3
Fig. 1. Wind speed on the NWP grid used in this study. Each dot contains the forecasted meteorological variables. The star mark in the center indicates the targeted wind farm location.
2.3. Wind Power Forecasting (WPF) Framework The WPF framework in this study is shown in Fig. 2. The WPF framework is composed of two phases, the feature extraction phase and the power forecasting phase. In the feature extraction phase, the NWP data are converted to spatio-temporal feature maps. The aim is to reduce the data dimensionality and automatically derive the important features from the wide area NWP data. In the power forecasting phase, wind power predictors generate the wind power output. We assumed the two types of power forecasting phase for comparison: one is the case that uses only the low-dimensional features obtained by the feature extraction phase, the other is the case that uses the features and historical wind power data. Historical wind power data are known to be essential to improve the WPF accuracy, especially in the short-term horizons [10]. The historical wind power output data within the last 12 hours are utilized as inputs of the power predictors in case (ii). The wind power predictors are individually constructed corresponding to the targeted time horizons and provide half-hourly wind power output with horizons up to 48 hours every 30 minutes. We employ gradient boosting trees regression (GBT) [13], which is an ensemble learning algorithm that builds a set of decision trees, as our wind power predictors. The advantages of GBTs are nonparametric models and low computational cost for prediction. In general, GBTs require careful hyper parameters tuning, thus we use [9] as reference. 3. Feature extraction In this section, we explain in detail the feature extraction architecture based on 3D-CNNs. CNNs [14] have greatly been implemented for various tasks in the past few years thanks to improved deep learning techniques [15]. CNNs have an advantage to extract nonlinear intrinsic features from input. 2D-CNNs have been popularly applied to image datasets and achieved remarkable results on several visual recognition tasks. In our recent work, 2D-CNNs have also been applied to one frame of the sequence of NWP data [10]; meanwhile, this study focuses on 3D-CNNs considering spatial and temporal dimensions. The idea is inspired by the feature extraction for video datasets discussed in [16]. They have suggested that 3D-CNNs can extract more intrinsic features for a time series dataset like videos. In this paper, 3D-CNNs are applied to time-series NWP data. 3.1. 3D convolution CNNs are characterized by a convolution operation. Fig. 3 explains the difference between 2D convolution and 3D convolution. A convolution operation is performed between input data and a kernel obtained by training. 2D convolution is defined as:
4
Kazutoshi Higashiyama et al. / Energy Procedia 155000–000 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018)
353
Fig. 2. Wind power forecasting framework.
Fig. 3. Comparison of 2D convolution and 3D convolution: (a)2D convolution; (b) 3D convolution. In the 2D convolution, rectangular kernels slide on the spatio input. In the 3D convolution, cubic kernels slide on the spatio-temporal input. ;78 678
(4)
(4)
𝑢𝑢)* = 𝜙𝜙 ,- - - 𝑥𝑥)/0,*/1 𝑤𝑤01 + 𝑏𝑏)* > (1) 4 09: 19:
(4)
where 𝑥𝑥)/0,*/1 is the input value at the position (𝑖𝑖 + 𝑝𝑝, 𝑗𝑗 + 𝑞𝑞), 𝑤𝑤01 is the weight at relative position (𝑝𝑝, 𝑞𝑞) of the 𝑚𝑚(4) th 2D kernel ℝ;×6 , 𝑏𝑏)* is bias, and 𝜙𝜙(∙) is an activation function. For our implementation, a rectified linear unit (ReLU) [17] is adopted to an activation function. On the other hand, 3D convolution is defined as: ;78 678 I78
(4)
(4)
𝑢𝑢)*G = 𝜙𝜙 ,- - - - 𝑥𝑥)/0,*/1,G/H 𝑤𝑤01H + 𝑏𝑏)*G > (2) 4 09: 19: H9:
where 𝑤𝑤01H is the weight at position (𝑝𝑝, 𝑞𝑞, 𝑟𝑟) of the 𝑚𝑚-th 3D kernel ℝ;×6×I . Here, 𝑅𝑅 represents the size of a temporal dimension. 3D convolution is operated spatio-temporally by adopting the 3D kernels while 2D convolution is operated only spatially by adopting 2D kernels. For our setup, 3D-CNNs are expected to be suitable on NWP data in order to capture temporal wind dynamics in NWP data. 3.2. 3D-CNN architecture We designed 3D-CNNs including the 3D convolution scheme for WPF. The feature extraction phase based on
Kazutoshi Higashiyama al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. et / Energy Procedia 00 (2018) 000–000
354
5
Fig. 4. Feature extraction phase based on 3D-CNN.
the 3D-CNNs is shown in Fig. 4. Our 3D-CNNs model is composed of four blocks, followed by one fully connected layer. Each block includes two convolution layers, a batch normalization layer [18], and a max pooling layer [19]. For simplicity, only the first block employs 3D convolution layers whereas the rest of blocks employs 2D convolution layers. We applied zero-padding, the process of placing zeroes at surrounding matrix, for 2D convolution layers. The fully connected layer includes one output unit to forecast wind power generation. We focus on three frames of NWP data corresponding to time slices {𝑡𝑡 − 1, 𝑡𝑡, 𝑡𝑡 + 1} as inputs to derive features for forecasting wind power at target time 𝑡𝑡. The 3D-CNNs model was learned by adaptive subgradient methods (AdaGrad) [20]. After training the 3D-CNNs, information obtained by the Block 4 of the 3D-CNNs shown is regarded as the spatio-temporal feature maps. The feature extractor generates eight feature maps of ℝP×P. The feature maps can be considered as low-dimensional information having a high correlation with the targeted wind power generation. The same scheme is applied to 2D-CNNs, which are used as a comparative method in section 4. 4. Numerical experiment We conducted the numerical experiments to verify the performance of the proposed 3D-CNNs. In order to understand the feature maps obtained by the 3D-CNNs carefully, we assumed the two cases in the power forecasting phase: (i) the case that uses only the feature maps, and (ii) the case that uses the feature maps and the observed wind power data simultaneously. 4.1. Dataset As shown in Table 1, the dataset was split into the training dataset and the validation dataset. The training dataset is composed of 19 months from April 2015 to October 2016 whereas the validation dataset is composed of 9 months from November 2016 to July 2017. Table 1. Training and validation periods. Prediction target Frequency of prediction Training period Validation period
Single site wind power forecasting Every 30 minutes with horizons up to 48 hours April 2015 – October 2016 (19 months) November 2016 – July 2017 (9 months)
6
Kazutoshi Higashiyama et al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018) 000–000
355
4.2. Benchmark approaches The following benchmark approaches are used for comparison. 1) Naive approaches In naive approaches, a feature extraction phase is omitted, that is, NWP data are directly utilized for WPF. Besides, two cases about input NWP data are considered. One uses only the spatio features, namely one frame of NWP data. The other uses the spatio-temporal features, namely three frames of NWP data. 2) Principal components analysis (PCA) Principal components analysis (PCA) [21] is a widely used linear dimensionality reduction algorithm [22]. The objective of PCA is to approximate the original data with a small number of synthetic variables that are called principal components. It is important to select the number of the principal components; hence, we changed the cumulative explained variance in 5% increments and adopted the 95% cumulative explained variance, which represents 22 principal components, with the relatively good accuracy. Note that PCA discovers only internal patterns of original data without considering wind power generation. 3) 2D convolutional neural networks (2D-CNNs) The 2D-CNNs in this study are the basically same architecture except for the first block [10]. In the 2D-CNNs, the two convolution layers in the first block employs 2D convolutions. Accordingly, only one frame of NWP at the targeted time is used as input. 4.3. Metrics To evaluate the performance of each approach, two metrics: the absolute error (MAE) and the root mean squared error (RMSE), were employed. The metrics are defined as: Y
1 𝑀𝑀𝑀𝑀𝑀𝑀 = -U𝑃𝑃W(𝑛𝑛) − 𝑃𝑃(𝑛𝑛)U (3) 𝑁𝑁 Z9:
Y
` 1 𝑅𝑅𝑀𝑀𝑀𝑀𝑀𝑀 = ]- ^𝑃𝑃W(𝑛𝑛) − 𝑃𝑃(𝑛𝑛)_ (4) 𝑁𝑁 Z9:
where 𝑁𝑁 is the number of samples in a validation dataset, 𝑃𝑃W is the prediction value and 𝑃𝑃 is the actual observation value. 5. Results Figures 5 (a) and (b) show the MAE and RMSE of the case (i), respectively. In naive approaches, there is no remarkable difference between using the spatio features and the spatio-temporal features; thus, it is invalid to simply extend the temporal dimension. In 3D-CNNs, it can be seen that the proposed 3D-CNNs improve the accuracy of WPF, especially after 20 hours horizons. The results show that the 3D-CNNs can extract the important features from spatio-temporal NWP data. Similarly, Figs. 6 (a) and (b) show the MAE and RMSE of the case (ii). Compared to the case (i), all approaches in the case (ii) have better accuracy in short horizons due to the observed wind power data. After 20 hours horizons, 2D-CNNs and 3D-CNNs have better accuracy than other comparative approaches. On the other hand, before 20 hours horizons, 3D-CNNs have the best accuracy in the approaches in this experiment.
356
Kazutoshi Higashiyama al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. et / Energy Procedia 00 (2018) 000–000
7
Fig. 5. Results of the case (i): (a) MAE; (b) RMSE.
Fig. 6. Results of the case (ii): (a) MAE; (b) RMSE.
To understand the function of the proposed 3D-CNNs, we analyse the feature maps obtained by the 3D-CNNs by t-distributed stochastic neighbour embedding (t-SNE) [23]. t-SNE is a nonlinear dimensionality reduction algorithm that is well suited for the visualization of a high-dimensional dataset. Here, t-SNE converts the feature maps into a two-dimensional space, so that similar features derived from NWP data are embedded closely and dissimilar features are embedded far away. Figure 7 visualizes the embedded feature maps obtained by 2D-CNNs and 3DCNNs, respectively. In 2D-CNNs, the feature maps are roughly grouped into four clusters; however, each cluster consists of similar feature maps from the viewpoint of NWP data does not contain similar wind power generation values. In contrast, in 3D-CNNs, the embedded maps show a clear characteristic corresponding to the amount of wind power generation; that is, the cases of high wind power generation tend to be placed in the upper right, and the cases of low wind power generation tend to be placed in the lower left. The results indicate that 3D-CNNs achieves to reduce the dimensionality of NWP data while deriving rather intrinsic features from NWP data in the context of wind power forecasting. 6. Conclusion In this study, we developed the feature extraction procedure based on 3D-CNNs to extract the spatio-temporal features from NWP data. Results of numerical experiments showed that the feature maps obtained by the proposed 3D-CNNs lead to improve the accuracy of WPF. Moreover, we visualized the feature maps via t-SNE. Results showed that 3D-CNNs achieve to extract more important features from NWP data in the context of wind power forecasting. This study presented that 3D-CNNs can be effective for feature extraction of NWP data. This study can be a key technique toward the wind ramp forecasting [24, 25, 26]. In future, we hope to apply our feature extraction approach to wind power ramp forecasting.
8
Kazutoshi Higashiyama et al. / Energy Procedia 155000–000 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018)
357
Fig. 7. Two-dimensional feature maps using t-SNE: (a) 2D-CNNs; (b)3D-CNNs. Similar features derived from NWP data are embedded closely. The color of each point represents the amount of wind power generation at the targeted time.
Acknowledgements This work was based on results obtained from R&D Project on Grid Integration of Variable Renewable Energy Mitigation Technologies on Output Fluctuations of Renewable Energy Generations in Power Grid commissioned by the New Energy and Industrial Technology Development Organization (NEDO). References [1] Global Wind Energy Council. "GWEC Global Wind Report 2016." [Online] Available: http://gwec.net/publications/global-wind-report-2/global-wind-report-2016/ [2] S. S. Soman, H. Zareipour, O. Malik, and P. Mandal. "A review of wind power and wind speed forecasting methods with different time horizons." North American Power Symposium (2010): 1-8. [3] M. Khalid and A. V. Savkin. "A Method for Short-Term Wind Power Prediction with Multiple Observation Points." IEEE Transactions on Power Systems 27(2) (2012): 579-586. [4] N. Chen, Z. Qian, I. T. Nabney, and X. Meng. "Wind Power Forecasts Using Gaussian Processes and Numerical Weather Prediction." IEEE Transactions on Power Systems 29 (2) (2014): 656-665. [5] T. G. Barbounis, J. B. Theocharis, M. C. Alexiadis, and P. S. Dokopoulos. "Long-term wind speed and power forecasting using local recurrent neural network models." IEEE Transactions on Energy Conversion 21(1) (2016): 273-284. [6] M. B. Ozkan and P. Karagoz. "A Novel Wind Power Forecast Model: Statistical Hybrid Wind Power Forecast Technique (SHWIP)." IEEE Transactions on Industrial Informatics 11 (2) (2015): 375-387. [7] R.E. Bellman. "Adaptive Control Processes." Princeton University Press (1961): 94. [8] F. Davò, S. Alessandrini, S. Sperati, L. D. Monache, D. Airoldi, and M.T. Vespucci. "Post-processing techniques and principal component analysis for regional wind power and solar irradiance forecasting." Solar Energy 134: 327-338. [9] J. R. Andrade and R. J. Bessa. "Improving Renewable Energy Forecasting with a Grid of Numerical Weather Predictions." IEEE Transactions on Sustainable Energy 8 (4) (2017): 1571-1580. [10] K. Higashiyama, Y. Fujimoto, and Y. Hayashi. "Feature Extraction of Numerical Weather Prediction Results Toward Reliable Wind Power Prediction." IEEE PES Innovative Smart Grid Technologies Europe (2017). [11] A. Hashimoto, H. Hirakuchi, H. Tamura, Y. Hattori, and S. Matsunashi. "A 53-year reproduction run over Japan using a regional climate model." Technical Report N13004. Central Research Institute of Electric Power Industry. (2013). [12] W. C. Skamarock, J. B. Klemp, J. Dudhia, D. O. Gill, D. M. Barker, M. G. Duda, X. Huang, W. Wang, and J. G. Powers. "A description of the advanced research WRF version 3." NCAR TECHNICAL NOTE. National Center for Atmospheric Research. (2008). [13] J. H. Friedman. "Greedy function approximation: a gradient boosting machine." Annals of statistics 29 (5) (2001). 1189-1232. [14] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based learning applied to document recognition." Proceedings of the IEEE 86 (11) (1998): 2278-2324. [15] Y. LeCun, Y. Bengio, and G. Hinton. "Deep learning." nature 521 (7533) (2015): 436-444.
358
Kazutoshi Higashiyama et al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018) 000–000
9
[16] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. "Learning Spatiotemporal Features with 3D Convolutional Networks." IEEE International Conference on Computer Vision (ICCV) (2015): 4489-4497. [17] N. Vinod and G. E. Hinton. "Rectified linear units improve restricted boltzmann machines." Proceedings of the 27th international conference on machine learning (2010): 807-814. [18] I. Sergey and C. Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." Proceedings of the 32nd International Conference on Machine Learning (2015): 448-456. [19] Y. Boureau, J. Ponce, and Y. LeCun. "A theoretical analysis of feature pooling in visual recognition" Proceedings of the 27th international conference on machine learning (2010): 111-118. [20] D. John, E. Hazan, and Y. Singer. "Adaptive subgradient methods for online learning and stochastic optimization." Journal of Machine Learning Research 12 (2011): 2121-2159. [21] M. E. Tipping and C. M. Bishop. "Probabilistic principal component analysis." Journal of the Royal Statistical Society: Series B (Statistical Methodology) 61 (3) (1999): 611-622. [22] Z. Wang, W. Wang, and B. Wang. "Regional wind power forecasting model with NWP grid data optimized." Frontiers in Energy 11 (2) (2017): 175-183. [23] Maaten, Laurens van der, and Geoffrey Hinton. "Visualizing data using t-SNE." Journal of Machine Learning Research 9 (2008): 25792605. [24] T. Suga, N. Hayasaki, and K. Ogimoto. "Project Overview of “Grid Integration of Variable Renewable Energy : Mitigation Technologies on Output Fluctuations of Renewable Energy Generations in Power Grid” R&D Item (1): Enhanced wind power forecast and control technology." Grand Renewable Energy 2018 International Conference and Exhibition (2018). [25] Y. Fujimoto, Y. Takahashi, and Y. Hayashi. "Alerting to Rare Large- Scale Ramp Events in Wind Power Generation." IEEE Transactions on Sustainable Energy (99) (2018) (in press). [Online] https://ieeexplore.ieee.org/document/8331103/ [26] Y. Takahashi, Y. Fujimoto, and Y. Hayashi. "Forecast of Infrequent Wind Power Ramps Based on Data Sampling Strategy," Energy Procedia 135 (2017): 496-503.
Available online at www.sciencedirect.com
Availableonline onlineatatwww.sciencedirect.com www.sciencedirect.com Available Energy Procedia 00 (2018) 000–000
ScienceDirect ScienceDirect
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
Energy (2018) 000–000 350–358 EnergyProcedia Procedia155 00 (2017) www.elsevier.com/locate/procedia 12th International Renewable Energy Storage Conference, IRES 2018
International Renewable Energy Storage Conference, IRES 2018 Feature12th Extraction of NWP Data for Wind Power Forecasting Using 3D-Convolutional Neural Networks Feature Extraction of NWP Data for Wind Power Forecasting The 15th International Symposium on District Heating and Cooling Using 3D-Convolutional Networks a a Kazutoshi Higashiyama *, Yu FujimotoNeural , Yasuhiro Hayashia Assessing theWaseda feasibility using thea,169-8555, heat demand-outdoor a of University, 3-4-1 Shinjuku, Tokyo, Japan Kazutoshi Higashiyama *,Okubo Yu Fujimoto Yasuhiro Hayashia temperature function for a long-term district heat demand forecast Waseda University, 3-4-1 Okubo Shinjuku, Tokyo, 169-8555, Japan a
a
Abstract
I. Andrića,b,c*, A. Pinaa, P. Ferrãoa, J. Fournierb., B. Lacarrièrec, O. Le Correc
Abstract Wind power is one of the most attractive forms of electricity from the viewpoints of cost efficiency and environmental protection. a IN+ Center for Innovation, Technology andaPolicy Research Instituto Rovisco Paisforecasting 1, 1049-001will Lisbon, However, the instability of wind power has serious impact- on a gridSuperior system.Técnico, ReliableAv.wind power helpPortugal to utilize b Veolia Recherche & Innovation, Avenue Daniel, Limay, Wind power is one of the most attractive forms of electricity from theDreyfous viewpoints of 78520 cost and environmental storage systems and backup generators effectively for 291 mitigating the instability. Thisefficiency paperFrance proposes a feature protection. extraction c Systèmes Énergétiques et Environnement 4 rue Alfred Nantes, France However, instability wind power has a serious impact on -a IMT grid system. Reliable windKastler, power44300 forecasting will help to utilize procedure the forDépartement numericalofweather prediction (NWP) data based on Atlantique, the three-dimensional convolutional neural networks (3Dstorage and backup generators for extract mitigating the instability. This paper proposes a feature extraction CNNs). systems An advantage of 3D-CNNs is toeffectively automatically the spatio-temporal features from NWP data focusing on the targeted wind farm. Feature extraction based(NWP) on 3D-CNNs wasonapplied to real-world datasets; the results significant procedure for numerical weather prediction data based the three-dimensional convolutional neuralshow networks (3Dperformance in comparison to severalisbenchmark approaches, andthe also show that the features proposedfrom extraction based on CNNs). An advantage of 3D-CNNs to automatically extract spatio-temporal NWP scheme data focusing on 3Dthe Abstract targetedachieves wind farm. Feature extraction on 3D-CNNs was applied to real-world datasets; CNNs to derive intrinsic featuresbased for prediction of wind power generation from NWP data. the results show significant performance in comparison to several benchmark approaches, and also show that the proposed extraction scheme based on 3DDistrict heating networks are commonly addressed in the literature as one of the most effective solutions for decreasing the CNNs achieves to derive intrinsic features for prediction of wind power generation from NWP data. greenhouse gas emissions from the building sector. These systems require high investments which are returned through the heat © 2018 The Authors. Published by Elsevier Ltd. sales. Due to the changed climate conditions renovation policies, heat demand in the future could decrease, © 2018 The Authors. by Elsevier Ltd. and building This is an open accessPublished article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) prolonging the access investment return period. Selection andAuthors. peer-review underbyresponsibility © 2018 The Published Elsevier Ltd.of the scientific committee of the 12th International Renewable Energy Storage Selection peer-review under responsibility of the scientific of the –12th International Renewable The mainand scope of this paper is to assess the feasibility of usingcommittee the heat demand outdoor temperature functionEnergy for heatStorage demand This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Conference. Conference. forecast. The district of Alvalade, located in Lisbon (Portugal), was used as a case study. The district is consisted of 665 Selection and peer-review under responsibility of the scientific committee of the 12th International Renewable Energy Storage buildings that vary in both construction period and typology. Three weather scenarios (low, medium, high) and three district Keywords: wind power forecasting; feature extraction; convolutional neural networks; deep learning; numerical weather prediction Conference. renovation scenarios were developed (shallow, intermediate, deep). To estimate the error, obtained heat demand values were compared with results from a dynamic heat demand model, previously developed and validated by the authors. Keywords: wind power forecasting; feature extraction; convolutional neural networks; deep learning; numerical weather prediction results showed that when only weather change is considered, the margin of error could be acceptable for some applications 1.The Introduction (the error in annual demand was lower than 20% for all weather scenarios considered). However, after introducing renovation the error value increased up to 59.5% (depending on the weather and renovation scenarios combination considered). 1.scenarios, Introduction Wind as coefficient an alternative to fossil fuels, has made substantial contribution to adecade, more sustainable society. It The valueenergy, of slope increased on average within thea range of 3.8% up to 8% per that corresponds to the has been introduced worldwide a way to reduceduring greenhouse gas season emission. According to combination the Global Wind Energy decrease in the number of heatingas hours of 22-139h the heating (depending on the of weather and Wind energy, as an alternative fossil made intercept a substantial contribution to a more sustainable society. It renovation scenarios considered). Ontothe otherfuels, hand,has function increased for 7.8-12.7% per decade (depending on the has been introduced worldwide as a way to reduce greenhouse gas emission. According to the Global Wind Energy coupled scenarios). The values suggested could be used to modify the function parameters for the scenarios considered, and improve the accuracy of heat demand estimations. © 2017 The Authors. Published by Elsevier Ltd. * Corresponding author. Tel.: +81-3-5286-8122; fax: +81-3-5286-8035. Peer-review under responsibility of the Scientific Committee of The 15th International Symposium on District Heating and E-mail address: [email protected] Cooling. * Corresponding author. Tel.: +81-3-5286-8122; fax: +81-3-5286-8035.
1876-6102 © 2018 The Authors. Published by Elsevier Ltd. E-mail address: [email protected] Keywords: Heat demand; Forecast; Climate change This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Selection and peer-review under responsibility of the scientific 1876-6102 © 2018 The Authors. Published by Elsevier Ltd. committee of the 12th International Renewable Energy Storage Conference. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Selection and peer-review under responsibility of the scientific committee of the 12th International Renewable Energy Storage Conference.
1876-6102 © 2017 The Authors. Published by Elsevier Ltd. 1876-6102 © 2018 The Authors. Published by Elsevier Ltd. Peer-review under responsibility of the Scientific Committee of The 15th International Symposium on District Heating and Cooling. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Selection and peer-review under responsibility of the scientific committee of the 12th International Renewable Energy Storage Conference. 10.1016/j.egypro.2018.11.043
2
Kazutoshi Higashiyama et al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018) 000–000
351
Council [1], the cumulative capacity of wind power reached 486 GW across the global market in 2016. Wind power is expected to significantly expand leading to an overall zero emission power system. One of the greatest challenges in large-scale penetration of wind power is to control its unstable output. This is a serious problem for grid operators since supply and demand power have to be balanced. In particular, rapid output changes may result in frequency deviation of a grid system. One of the most effective solutions to this problem is to introduce reliable wind power forecasting (WPF). WPF helps to optimize the operation schedule of storage systems and backup generators for mitigating the instabilities. Several WPF approaches have been proposed [2]. Among them, numerical weather predictions (NWP) models have been widely used [3, 4] . NWP models calculate meteorological variables, such as wind speeds, for each gridded point by solving complex simultaneous equations relating to the atmosphere. Machine learning approaches utilizing the NWP results have also been actively discussed [5, 6] so as to improve the forecasting accuracy; this approach achieves to model the relationship between the meteorological variables obtained by the NWP model and the observed power generation output. A major obstacle is to handle the high dimensionality of NWP data; i.e., machine learning approaches may not perform appropriately in the case that the volume of input data is substantially large. This is so-called the “curse of dimensionality” [7]. To deal with the problem, several studies explored a feature extraction scheme for NWP data. Davò et al. [8] used principal components analysis (PCA) to obtain the dimensionally reduced NWP dataset. The dataset was applied to the analog ensemble technique and neural networks as input for deterministic and probabilistic WPF. Andrade et al. [9] studied feature engineering techniques using NWP grid domain knowledge. In the previous study, we have focused on the feature extraction scheme based on two-dimensional convolutional neural networks (2D-CNNs) [10]. Our feature extraction scheme has achieved to obtain low-dimensional features from a large region of spatial NWP data corresponding to the output of targeted wind farms. In this paper, we explore the feature extraction scheme based on three-dimensional convolutional neural networks (3D-CNNs). The 3D-CNNs can handle not only spatial dimensions but also a temporal dimension. Therefore, the 3D-CNNs are expected to capture the dynamics of wind contained in NWP data by focusing on time slices in a certain period. This paper is organized as follows: section 2 describes the datasets and a procedure for WPF, section 3 describes the proposed feature extraction architecture, section 4 explains the configuration of numerical experiments, section 5 discusses the evaluation results, and section 6 concludes the paper. 2. Wind power forecasting framework 2.1. Wind farm The wind power dataset used in this paper is collected from an onshore wind farm located in the Tohoku region, which is the north-eastern area of Japan. The wind power output was measured every 10 seconds and collected from April 2015 to July 2017. The dataset was converted into per unit system. 2.2. Numerical Weather predictions (NWP) data The NWP data were provided by the Numerical Weather Forecasting and Analysis System (NuWFAS) [11]. The NuWFAS is based on the Weather Research and Forecasting (WRF) [12], which is the mesoscale meteorological model. A plurality of frames included in the NWP data continue along a time axis. The temporal resolution is 30 minutes; the data are provided by using boundary condition given at 00:00 UTC and cover the time horizons from 30 minutes to 72 hours ahead. The NWP data cover the Tohoku area, and contain forecasted meteorological variables on a 5-km horizontal grid. Although the NWP data contain several kinds of meteorological variables, the zonal and meridional components of wind speed vectors at 60 meters above sea level are employed in this study. Fig. 1 describes the grid of the NWP data. The rectangular grid includes 50 × 50 = 2,500 points around the targeted wind power plant.
Kazutoshi Higashiyama et al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018) 000–000
352
3
Fig. 1. Wind speed on the NWP grid used in this study. Each dot contains the forecasted meteorological variables. The star mark in the center indicates the targeted wind farm location.
2.3. Wind Power Forecasting (WPF) Framework The WPF framework in this study is shown in Fig. 2. The WPF framework is composed of two phases, the feature extraction phase and the power forecasting phase. In the feature extraction phase, the NWP data are converted to spatio-temporal feature maps. The aim is to reduce the data dimensionality and automatically derive the important features from the wide area NWP data. In the power forecasting phase, wind power predictors generate the wind power output. We assumed the two types of power forecasting phase for comparison: one is the case that uses only the low-dimensional features obtained by the feature extraction phase, the other is the case that uses the features and historical wind power data. Historical wind power data are known to be essential to improve the WPF accuracy, especially in the short-term horizons [10]. The historical wind power output data within the last 12 hours are utilized as inputs of the power predictors in case (ii). The wind power predictors are individually constructed corresponding to the targeted time horizons and provide half-hourly wind power output with horizons up to 48 hours every 30 minutes. We employ gradient boosting trees regression (GBT) [13], which is an ensemble learning algorithm that builds a set of decision trees, as our wind power predictors. The advantages of GBTs are nonparametric models and low computational cost for prediction. In general, GBTs require careful hyper parameters tuning, thus we use [9] as reference. 3. Feature extraction In this section, we explain in detail the feature extraction architecture based on 3D-CNNs. CNNs [14] have greatly been implemented for various tasks in the past few years thanks to improved deep learning techniques [15]. CNNs have an advantage to extract nonlinear intrinsic features from input. 2D-CNNs have been popularly applied to image datasets and achieved remarkable results on several visual recognition tasks. In our recent work, 2D-CNNs have also been applied to one frame of the sequence of NWP data [10]; meanwhile, this study focuses on 3D-CNNs considering spatial and temporal dimensions. The idea is inspired by the feature extraction for video datasets discussed in [16]. They have suggested that 3D-CNNs can extract more intrinsic features for a time series dataset like videos. In this paper, 3D-CNNs are applied to time-series NWP data. 3.1. 3D convolution CNNs are characterized by a convolution operation. Fig. 3 explains the difference between 2D convolution and 3D convolution. A convolution operation is performed between input data and a kernel obtained by training. 2D convolution is defined as:
4
Kazutoshi Higashiyama et al. / Energy Procedia 155000–000 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018)
353
Fig. 2. Wind power forecasting framework.
Fig. 3. Comparison of 2D convolution and 3D convolution: (a)2D convolution; (b) 3D convolution. In the 2D convolution, rectangular kernels slide on the spatio input. In the 3D convolution, cubic kernels slide on the spatio-temporal input. ;78 678
(4)
(4)
𝑢𝑢)* = 𝜙𝜙 ,- - - 𝑥𝑥)/0,*/1 𝑤𝑤01 + 𝑏𝑏)* > (1) 4 09: 19:
(4)
where 𝑥𝑥)/0,*/1 is the input value at the position (𝑖𝑖 + 𝑝𝑝, 𝑗𝑗 + 𝑞𝑞), 𝑤𝑤01 is the weight at relative position (𝑝𝑝, 𝑞𝑞) of the 𝑚𝑚(4) th 2D kernel ℝ;×6 , 𝑏𝑏)* is bias, and 𝜙𝜙(∙) is an activation function. For our implementation, a rectified linear unit (ReLU) [17] is adopted to an activation function. On the other hand, 3D convolution is defined as: ;78 678 I78
(4)
(4)
𝑢𝑢)*G = 𝜙𝜙 ,- - - - 𝑥𝑥)/0,*/1,G/H 𝑤𝑤01H + 𝑏𝑏)*G > (2) 4 09: 19: H9:
where 𝑤𝑤01H is the weight at position (𝑝𝑝, 𝑞𝑞, 𝑟𝑟) of the 𝑚𝑚-th 3D kernel ℝ;×6×I . Here, 𝑅𝑅 represents the size of a temporal dimension. 3D convolution is operated spatio-temporally by adopting the 3D kernels while 2D convolution is operated only spatially by adopting 2D kernels. For our setup, 3D-CNNs are expected to be suitable on NWP data in order to capture temporal wind dynamics in NWP data. 3.2. 3D-CNN architecture We designed 3D-CNNs including the 3D convolution scheme for WPF. The feature extraction phase based on
Kazutoshi Higashiyama al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. et / Energy Procedia 00 (2018) 000–000
354
5
Fig. 4. Feature extraction phase based on 3D-CNN.
the 3D-CNNs is shown in Fig. 4. Our 3D-CNNs model is composed of four blocks, followed by one fully connected layer. Each block includes two convolution layers, a batch normalization layer [18], and a max pooling layer [19]. For simplicity, only the first block employs 3D convolution layers whereas the rest of blocks employs 2D convolution layers. We applied zero-padding, the process of placing zeroes at surrounding matrix, for 2D convolution layers. The fully connected layer includes one output unit to forecast wind power generation. We focus on three frames of NWP data corresponding to time slices {𝑡𝑡 − 1, 𝑡𝑡, 𝑡𝑡 + 1} as inputs to derive features for forecasting wind power at target time 𝑡𝑡. The 3D-CNNs model was learned by adaptive subgradient methods (AdaGrad) [20]. After training the 3D-CNNs, information obtained by the Block 4 of the 3D-CNNs shown is regarded as the spatio-temporal feature maps. The feature extractor generates eight feature maps of ℝP×P. The feature maps can be considered as low-dimensional information having a high correlation with the targeted wind power generation. The same scheme is applied to 2D-CNNs, which are used as a comparative method in section 4. 4. Numerical experiment We conducted the numerical experiments to verify the performance of the proposed 3D-CNNs. In order to understand the feature maps obtained by the 3D-CNNs carefully, we assumed the two cases in the power forecasting phase: (i) the case that uses only the feature maps, and (ii) the case that uses the feature maps and the observed wind power data simultaneously. 4.1. Dataset As shown in Table 1, the dataset was split into the training dataset and the validation dataset. The training dataset is composed of 19 months from April 2015 to October 2016 whereas the validation dataset is composed of 9 months from November 2016 to July 2017. Table 1. Training and validation periods. Prediction target Frequency of prediction Training period Validation period
Single site wind power forecasting Every 30 minutes with horizons up to 48 hours April 2015 – October 2016 (19 months) November 2016 – July 2017 (9 months)
6
Kazutoshi Higashiyama et al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018) 000–000
355
4.2. Benchmark approaches The following benchmark approaches are used for comparison. 1) Naive approaches In naive approaches, a feature extraction phase is omitted, that is, NWP data are directly utilized for WPF. Besides, two cases about input NWP data are considered. One uses only the spatio features, namely one frame of NWP data. The other uses the spatio-temporal features, namely three frames of NWP data. 2) Principal components analysis (PCA) Principal components analysis (PCA) [21] is a widely used linear dimensionality reduction algorithm [22]. The objective of PCA is to approximate the original data with a small number of synthetic variables that are called principal components. It is important to select the number of the principal components; hence, we changed the cumulative explained variance in 5% increments and adopted the 95% cumulative explained variance, which represents 22 principal components, with the relatively good accuracy. Note that PCA discovers only internal patterns of original data without considering wind power generation. 3) 2D convolutional neural networks (2D-CNNs) The 2D-CNNs in this study are the basically same architecture except for the first block [10]. In the 2D-CNNs, the two convolution layers in the first block employs 2D convolutions. Accordingly, only one frame of NWP at the targeted time is used as input. 4.3. Metrics To evaluate the performance of each approach, two metrics: the absolute error (MAE) and the root mean squared error (RMSE), were employed. The metrics are defined as: Y
1 𝑀𝑀𝑀𝑀𝑀𝑀 = -U𝑃𝑃W(𝑛𝑛) − 𝑃𝑃(𝑛𝑛)U (3) 𝑁𝑁 Z9:
Y
` 1 𝑅𝑅𝑀𝑀𝑀𝑀𝑀𝑀 = ]- ^𝑃𝑃W(𝑛𝑛) − 𝑃𝑃(𝑛𝑛)_ (4) 𝑁𝑁 Z9:
where 𝑁𝑁 is the number of samples in a validation dataset, 𝑃𝑃W is the prediction value and 𝑃𝑃 is the actual observation value. 5. Results Figures 5 (a) and (b) show the MAE and RMSE of the case (i), respectively. In naive approaches, there is no remarkable difference between using the spatio features and the spatio-temporal features; thus, it is invalid to simply extend the temporal dimension. In 3D-CNNs, it can be seen that the proposed 3D-CNNs improve the accuracy of WPF, especially after 20 hours horizons. The results show that the 3D-CNNs can extract the important features from spatio-temporal NWP data. Similarly, Figs. 6 (a) and (b) show the MAE and RMSE of the case (ii). Compared to the case (i), all approaches in the case (ii) have better accuracy in short horizons due to the observed wind power data. After 20 hours horizons, 2D-CNNs and 3D-CNNs have better accuracy than other comparative approaches. On the other hand, before 20 hours horizons, 3D-CNNs have the best accuracy in the approaches in this experiment.
356
Kazutoshi Higashiyama al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. et / Energy Procedia 00 (2018) 000–000
7
Fig. 5. Results of the case (i): (a) MAE; (b) RMSE.
Fig. 6. Results of the case (ii): (a) MAE; (b) RMSE.
To understand the function of the proposed 3D-CNNs, we analyse the feature maps obtained by the 3D-CNNs by t-distributed stochastic neighbour embedding (t-SNE) [23]. t-SNE is a nonlinear dimensionality reduction algorithm that is well suited for the visualization of a high-dimensional dataset. Here, t-SNE converts the feature maps into a two-dimensional space, so that similar features derived from NWP data are embedded closely and dissimilar features are embedded far away. Figure 7 visualizes the embedded feature maps obtained by 2D-CNNs and 3DCNNs, respectively. In 2D-CNNs, the feature maps are roughly grouped into four clusters; however, each cluster consists of similar feature maps from the viewpoint of NWP data does not contain similar wind power generation values. In contrast, in 3D-CNNs, the embedded maps show a clear characteristic corresponding to the amount of wind power generation; that is, the cases of high wind power generation tend to be placed in the upper right, and the cases of low wind power generation tend to be placed in the lower left. The results indicate that 3D-CNNs achieves to reduce the dimensionality of NWP data while deriving rather intrinsic features from NWP data in the context of wind power forecasting. 6. Conclusion In this study, we developed the feature extraction procedure based on 3D-CNNs to extract the spatio-temporal features from NWP data. Results of numerical experiments showed that the feature maps obtained by the proposed 3D-CNNs lead to improve the accuracy of WPF. Moreover, we visualized the feature maps via t-SNE. Results showed that 3D-CNNs achieve to extract more important features from NWP data in the context of wind power forecasting. This study presented that 3D-CNNs can be effective for feature extraction of NWP data. This study can be a key technique toward the wind ramp forecasting [24, 25, 26]. In future, we hope to apply our feature extraction approach to wind power ramp forecasting.
8
Kazutoshi Higashiyama et al. / Energy Procedia 155000–000 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018)
357
Fig. 7. Two-dimensional feature maps using t-SNE: (a) 2D-CNNs; (b)3D-CNNs. Similar features derived from NWP data are embedded closely. The color of each point represents the amount of wind power generation at the targeted time.
Acknowledgements This work was based on results obtained from R&D Project on Grid Integration of Variable Renewable Energy Mitigation Technologies on Output Fluctuations of Renewable Energy Generations in Power Grid commissioned by the New Energy and Industrial Technology Development Organization (NEDO). References [1] Global Wind Energy Council. "GWEC Global Wind Report 2016." [Online] Available: http://gwec.net/publications/global-wind-report-2/global-wind-report-2016/ [2] S. S. Soman, H. Zareipour, O. Malik, and P. Mandal. "A review of wind power and wind speed forecasting methods with different time horizons." North American Power Symposium (2010): 1-8. [3] M. Khalid and A. V. Savkin. "A Method for Short-Term Wind Power Prediction with Multiple Observation Points." IEEE Transactions on Power Systems 27(2) (2012): 579-586. [4] N. Chen, Z. Qian, I. T. Nabney, and X. Meng. "Wind Power Forecasts Using Gaussian Processes and Numerical Weather Prediction." IEEE Transactions on Power Systems 29 (2) (2014): 656-665. [5] T. G. Barbounis, J. B. Theocharis, M. C. Alexiadis, and P. S. Dokopoulos. "Long-term wind speed and power forecasting using local recurrent neural network models." IEEE Transactions on Energy Conversion 21(1) (2016): 273-284. [6] M. B. Ozkan and P. Karagoz. "A Novel Wind Power Forecast Model: Statistical Hybrid Wind Power Forecast Technique (SHWIP)." IEEE Transactions on Industrial Informatics 11 (2) (2015): 375-387. [7] R.E. Bellman. "Adaptive Control Processes." Princeton University Press (1961): 94. [8] F. Davò, S. Alessandrini, S. Sperati, L. D. Monache, D. Airoldi, and M.T. Vespucci. "Post-processing techniques and principal component analysis for regional wind power and solar irradiance forecasting." Solar Energy 134: 327-338. [9] J. R. Andrade and R. J. Bessa. "Improving Renewable Energy Forecasting with a Grid of Numerical Weather Predictions." IEEE Transactions on Sustainable Energy 8 (4) (2017): 1571-1580. [10] K. Higashiyama, Y. Fujimoto, and Y. Hayashi. "Feature Extraction of Numerical Weather Prediction Results Toward Reliable Wind Power Prediction." IEEE PES Innovative Smart Grid Technologies Europe (2017). [11] A. Hashimoto, H. Hirakuchi, H. Tamura, Y. Hattori, and S. Matsunashi. "A 53-year reproduction run over Japan using a regional climate model." Technical Report N13004. Central Research Institute of Electric Power Industry. (2013). [12] W. C. Skamarock, J. B. Klemp, J. Dudhia, D. O. Gill, D. M. Barker, M. G. Duda, X. Huang, W. Wang, and J. G. Powers. "A description of the advanced research WRF version 3." NCAR TECHNICAL NOTE. National Center for Atmospheric Research. (2008). [13] J. H. Friedman. "Greedy function approximation: a gradient boosting machine." Annals of statistics 29 (5) (2001). 1189-1232. [14] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based learning applied to document recognition." Proceedings of the IEEE 86 (11) (1998): 2278-2324. [15] Y. LeCun, Y. Bengio, and G. Hinton. "Deep learning." nature 521 (7533) (2015): 436-444.
358
Kazutoshi Higashiyama et al. / Energy Procedia 155 (2018) 350–358 Kazutoshi Higashiyama et al. / Energy Procedia 00 (2018) 000–000
9
[16] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. "Learning Spatiotemporal Features with 3D Convolutional Networks." IEEE International Conference on Computer Vision (ICCV) (2015): 4489-4497. [17] N. Vinod and G. E. Hinton. "Rectified linear units improve restricted boltzmann machines." Proceedings of the 27th international conference on machine learning (2010): 807-814. [18] I. Sergey and C. Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." Proceedings of the 32nd International Conference on Machine Learning (2015): 448-456. [19] Y. Boureau, J. Ponce, and Y. LeCun. "A theoretical analysis of feature pooling in visual recognition" Proceedings of the 27th international conference on machine learning (2010): 111-118. [20] D. John, E. Hazan, and Y. Singer. "Adaptive subgradient methods for online learning and stochastic optimization." Journal of Machine Learning Research 12 (2011): 2121-2159. [21] M. E. Tipping and C. M. Bishop. "Probabilistic principal component analysis." Journal of the Royal Statistical Society: Series B (Statistical Methodology) 61 (3) (1999): 611-622. [22] Z. Wang, W. Wang, and B. Wang. "Regional wind power forecasting model with NWP grid data optimized." Frontiers in Energy 11 (2) (2017): 175-183. [23] Maaten, Laurens van der, and Geoffrey Hinton. "Visualizing data using t-SNE." Journal of Machine Learning Research 9 (2008): 25792605. [24] T. Suga, N. Hayasaki, and K. Ogimoto. "Project Overview of “Grid Integration of Variable Renewable Energy : Mitigation Technologies on Output Fluctuations of Renewable Energy Generations in Power Grid” R&D Item (1): Enhanced wind power forecast and control technology." Grand Renewable Energy 2018 International Conference and Exhibition (2018). [25] Y. Fujimoto, Y. Takahashi, and Y. Hayashi. "Alerting to Rare Large- Scale Ramp Events in Wind Power Generation." IEEE Transactions on Sustainable Energy (99) (2018) (in press). [Online] https://ieeexplore.ieee.org/document/8331103/ [26] Y. Takahashi, Y. Fujimoto, and Y. Hayashi. "Forecast of Infrequent Wind Power Ramps Based on Data Sampling Strategy," Energy Procedia 135 (2017): 496-503.