- Email: [email protected]
Circuit reliability prediction based on deep autoencoder network
ARTICLE IN PRESS
JID: NEUCOM
[m5G;September 10, 2019;23:55]
Neurocomputing xxx (xxxx) xxx
Contents lists available at ScienceDirect
Neurocomputing journal homepage: www.elsevier.com/locate/neucom
Circuit reliability prediction based on deep autoencoder network Jie Xiao a, Weifeng Ma a, Jungang Lou b,∗, Jianhui Jiang c, Yujiao Huang a, Zhanhui Shi a, Qing Shen b, Xuhua Yang a a
College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China College of Information Science, Huzhou University, Huzhou, China c Software Engineering Department, Tongji University, Shanghai, China b
a r t i c l e
i n f o
Article history: Received 1 March 2019 Revised 14 June 2019 Accepted 7 July 2019 Available online xxx Communicated by Dr. Ma Jiayi Keywords: Logic circuit Input vector Feature set Deep auto encoder network Reliability prediction
a b s t r a c t As semiconductor feature size continues to decrease and the density of integration continues to increase, highly reliable circuit design is experiencing many challenges, including reliability evaluation, which is one of the most important steps in circuit design. However, faced with the very large scale of integrated circuits at present, traditional simulation-based methods are slightly inadequate in terms of computational complexity and do not apply to the circuits at the concept stage. To solve this problem, this paper presents a new prediction method for circuit reliability based on deep auto encoder networks. Firstly, we analyze and extract the main features associated with circuit reliability. Next, we construct an efficient method for data collection by combining the characteristics of the feature set with the requirements of deep auto encoder networks. Then, we build a deep auto encoder network model oriented to circuit reliability prediction in a supervised learning manner. Simulation results on 74-series circuits and ISCAS85 benchmark circuits show that although the accuracy of the proposed method is slightly lower than that of both the Monte Carlo (MC) method and the fast probabilistic transfer matrix (F-PTM) model, its timespace consumption is approximately constant on different circuits, and it is 102,458,469 times faster than the MC method, and approximately 4,383 times faster than the F-PTM model. Furthermore, the proposed method could be used to predict circuit reliability at the conceptual stage, and it is a very efficient approximation method that could greatly reduce the power consumption of the calculation. © 2019 Published by Elsevier B.V.
1. Introduction With the extensive application of integrated circuit products in various fields, such as aviation, medical, and industry, high reliability is a requirement [1]. However, with the continued reduction in semiconductor feature size and the increase in integration density, uncertainty failure, such as intermittent and transient faults, has a more pronounced effect on circuit reliability [2]. Therefore, improving the reliability of circuits is a very important task that is directly related to the safety of people’s lives and property. Past analyses have found that reliability evaluation is one of the most important bases for improving circuit reliability [3]. Faced with the current very-large-scale integrated circuits, we need a quick and efficient evaluation method to measure their reliability at the early stages of circuit design so as to make timely decisions, thereby shortening the product development cycle to improve market
∗
Corresponding author. E-mail addresses: [email protected] (J. Xiao), [email protected] (J. Lou), [email protected] (J. Jiang).
competitiveness. Circuit reliability refers to the probability that the circuit can correctly perform the specified tasks in a certain environment [2]. At present, the evaluation approaches for circuit reliability mainly include the measurement-based, the simulation-based and the analytical approach. There are two basic schemes in the measurement-based approach: sequential truncation tests, and timed or fixed-number truncation tests. Although the test results are very practical, the schemes usually require a large number of samples and have a large lag in the quality evaluation. The simulation-based approach models a fault randomly and injects the fault into the various gates by a given rule at a certain abstract level, and then counts the number of failures of the circuit to obtain its reliability. The typical model is the Monte Carlo (MC) method, which can be applied to the various abstract levels of circuits to achieve high accuracy but has time-consuming. For large-scale integrated circuits, the MC method is more commonly used as a verification method. The analytical approach is commonly implemented at a specified abstract level to evaluate the reliability of circuits in a multi-fault environment. The typical methods include the probabilistic transfer matrix model (PTM)
https://doi.org/10.1016/j.neucom.2019.07.100 0925-2312/© 2019 Published by Elsevier B.V.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM 2
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
[4], probabilistic gate model (PGM) [5], Bayesian networks method [6], and stochastic calculation method [7]. They tend to achieve high accuracy but often have large time-space consumption, and require that the object to be analyzed is an existing entity. In summary, the approaches mentioned above are not applicable to the circuits at the conception stage. However, with the acceleration of new technology applications and product revisions, it is necessary to have new technologies to fast evaluate the reliability of circuits at the conception stage to boost the speed of circuit structure design. Noticing that the deep learning model is a method for discovering distributed representations of data by simulating the neural centers of the human brain, and combining the features at a low level to form more abstract representations at a high level [8]. The technology of deep learning can achieve object classification or result prediction by combining classifiers or predictors, and shows approximately constant time-space consumption this meets the new requirements put forward by the existing evaluation methods of circuit reliability. Therefore, this technology is chosen in this study to quickly evaluate the reliability of integrated circuits at the conception stage. According to the label attributes of the training data, the deep learning model can be classified into supervised learning and unsupervised learning. In general, the former is more accurate than the latter because its input data is tagged, while the latter does not have tagged data [9]. Considering that the F-PTM model [10] can provide the reliability of the applied circuits to label the input features of the circuits, a deep learning network with supervised learning is chosen in this study to implement the follow-up research. At present, the research on deep learning, such as convolution neural networks (CNNs) [11], the deep belief networks (DBNs) [12] and deep autoencoder networks (DANs) [13], mainly focuses on the fields of machine vision, speech recognition, natural language processing, and traffic prediction, etc. Some studies have pointed out that the CNN mainly focuses on unstructured data, while the DBN and DAN perform well for both the structured and the unstructured data, especially the structured data [14,15]. Further studies have found that a DBN is more suitable for processing continuous data, while a DAN performs better when dealing with discrete data [16]. Analysis found that the reliability of a circuit is mainly depended on its topological structure, the failure probability of the basic gates, and its application environment (see Section 4.1 for the details). Obviously, the dataset constructed by the feature set has structured and discrete characteristics. Therefore, this paper chooses the DAN model to conduct research on the reliability prediction for circuit structures. The gates in this paper are assumed to be subjected to von-Neumann faults which consists of transient bit-flip faults caused by signal noise, although other stack at fault models can also be used in this paper. However, the structure of the DAN model is influenced by many factors, such as the characteristics, scale, and quality of the dataset, which directly determine the prediction accuracy of the model. Therefore, it is necessary to preprocess the dataset according to the characteristics of the DAN model and the dataset, and to configure the structure reasonably to obtain acceptable predictions, which requires a wealth of expertise and superb skills. Therefore, there is still a lot of work to do in using the DAN model to predict circuit reliability and achieving acceptable accuracy. Firstly, we select the features associated with circuit reliability to determine the feature set, and we construct an efficient method for data collection for the feature set. Next, we preprocess the dataset to meet the requirements of the DAN model. Then, we train and test the constructed DAN model with the constructed training set and the testing set to find a reasonable structure for the DAN model that meets the accuracy requirements. Finally, we predict the reliability of the circuits according to their applied input features.
The rest of the paper is organized as follows. The preparatory work related to this study is introduced in Section 2, Section 3 describes the extraction of labeled data by the F-PTM model, followed by circuit feature analysis and extraction in Section 4. The prediction method for circuit reliability based on the DAN model is presented in Section 5. The experimental results are presented in Section 6, followed by the conclusions in Section 7. 2. Preparatory work To successfully conduct this study, the preparatory work involved is the description of the principles and construction method for the DAN model. Since the feature dataset in this paper has labeled attributes, and supervised learning is superior to unsupervised learning in terms of classification and prediction accuracy, we choose the DAN model with supervised learning to predict circuit reliability. The structure and the construction of the DAN model are presented in Figs. 1 and 2, respectively. It can be seen from Fig. 1 that the DAN is a neural network model with multiple hidden layers that consists of multiple autoencoders (AEs), and each AE is a learning network with only one input layer and one hidden layer, where the output of the AE at the bottom is used as the input to the AE at the top. Based on the representation in the previous layer, each layer in the DAN model learns multiple layer-based representations from the raw data, so as to learn more essential features [17]. It can be seen from Fig. 2 that the construction of the DAN model mainly includes the following parts. (a) Data collection and preprocessing. Firstly, we collect the feature data according to the characteristics of the DAN model. Next, we preprocess the feature dataset to meet the requirements of the DAN model. Then, we divide the dataset into training set and testing set by the previously determined proportions. (b) Construct a DAN-based prediction model. Firstly, we construct a feature learning network based on AEs. Next, we initialize the weight W and the bias vector b for the deep feedforward neural network by the feature learning network with layer-wise training. Then, we fine-tune the DAN model with supervised learning with the labeled training dataset. (c) Model testing and selection. We test and evaluate the DAN model with the test dataset and the error evaluation function, respectively, to determine whether the constructed model meets the accuracy requirements, so as to select an efficient prediction model. 2.1. AE The role of the AE is to recover the raw data from the abstracted data as much as possible, so as to extract the distributed feature representation from the raw input data. The AE is a special three-layer back propagation (BP) neural network consisting of an input layer, an encoder, and a decoder, as shown in Fig. 3, where V=(v1 ,v2 , .,vin ) is the feature dataset, vi (i = 1, 2, …, in) is the ith input feature dimension vector, and xi is the ith dimension input signal for the input layer. In the AE, w(1) and b(1) are the shared weight matrix and the bias vector between the input layer and the hidden layer, respectively, and w(2) and b(2) are the shared weight matrix and bias vector between the hidden layer and the output layer, respectively. The "+1" node is the bias node, and hw,b (x) is the output signal. To extract the essential features from the input data, the trained steps of the AE can be summarized simply, as follows. Firstly, we perform encoding transformation on the input signal x using Eqs. (1) and (2) to obtain another represent ation h of x. Next, we decode h using Eqs. (3) and (4) to obtain the output signal hw,b (x). Then, we update the shared weight matrix w and the bias vector
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
3
Fig. 1. The structure of the DAN model.
Fig. 2. Construct an AE-based layer-wise feature-learning network.
h = a (2 ) = f z (2 ) ,
(2)
z(3 ) = w(2 ) a(2 ) + b(2 ) ,
(3)
hw,b (x ) = a(3 ) − f z(3 ) , J AE(w,b) =
(4)
Lw,b a(3 ) , x ,
(5)
x∈V
f (z ) =
1 , 1 + exp (−z )
(6)
2.2. Small batch gradient descent method Fig. 3. Typical AE structure.
b in the AE using the cost function presented in Eq. (5) and the BP algorithm. We repeat the above steps until we find the w and b that minimize Eq. (6), (hw,b (x)≈x), where z(2) is the input to the hidden layer; a(2) is the encoding result in the AE (denoted as the output of hidden layer); a(3) is the input to the output layer, which is the decoding result in AE; f is the sigmoid function (nonlinear activation function) presented in Eq. (6); and L is the reconstruction error function, which is represented by the mean square error function in this study.
z(2) = w(1) x + b(1) ,
(1)
To optimize the parameter θ ={w,b}, the solution is usually achieved by the gradient descent method in error back propagation. However, for a large training dataset, the learning process is slow because the gradient vector J(θ ) is calculated once for each training instance. Therefore, the mini-batch gradient descent (MBGD) is introduced in this study. Unlike the gradient descent algorithm (where all the training data is used to update the parameters in each training), MBGD decomposes one training into T iterations, and each iteration is performed by randomly selecting N samples (mini batch) from the training dataset to calculate the gradient of the loss function, so as to update the parameters. To boost the convergence, the latest updated items are added to the
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM 4
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
Fig. 4. Training flowchart of the DAN model.
updating parameters in the momentum algorithm, so that the parameters can be updated in time, where the e-th training can be expressed by Eqs. (7)–(9). Jm (θ ) represents the gradient vector in the loss function, and is calculated by using the m-th batch of training data, m∈{1,2,…,T}; S is the number of samples in the m-th batch of the dataset; θ m (e) is the parameter calculated in the m-th iteration in the e-th training; Gm (e) is the update parameter calculated in the first m iterations in the e-th training, G0 (0) = 0; γ is the momentum super-parameter (its value is usually less than or equal to 0.9), and η is the learning rate.
θm+1 (e ) = θm (e ) − Gm (e ),
(7)
Gm (e ) = γ Gm−1 (e ) + η∇ Jm (θ (e ) ),
(8)
∇ Jm (θ ) =
S
ji (θ ),
(9)
i=1
2.3. Model training The training of the DAN model is described in Fig. 4, and its main steps are as follows: (a) Construct k AEs, and initialize their relevant parameters in a uniformly distributed manner within the interval β shown in Eq. (10), and initialize h0 = V. (b) Initialize the input signal of the i th AE (denoted as AE_i) with hi -1 (i∈{1, 2, ., k}), train AE_i and implement Eqs. (1)–(9). Then encode hi -1 with the trained AE_i to obtain the output signal hi to the hidden layer. (c) If i and Eqs. (1)–(5), and Eqs. (7)–(9), as well as Eq. (11). Where n( j ) is the number of nodes in the jth layer in AE, j∈{1, 2}; yˆ is the predicted value of the model, y is the corresponding label value, and Ym is the labeled data corresponding to the mth batch of the training data. Eq. (10) is used to initialize the elements of W using a random matrix with column mean 0, which leads to have small W and helps to prevent over-fitting in the training of the DAN model. Eq. (11) is a cost function and is used to estimate
the difference between the predicted value and the label value in each training.
β= −
6 , n( j ) + n( j+1)
JDAN (w, b) =
6 , n( j ) + n( j+1)
Lw,b yˆ, y ,
(10)
(11)
y∈Ym
3. F-PTM model This section introduces the F-PTM model for the collection of the labeled data corresponding to the feature sets. For any applied input vectors, the F-PTM model [10] can provide results to exactly reflect the reliability R of the corresponding circuit features. Therefore, the results calculated by the F-PTM model are used as the labels for the feature data. The calculation processes are presented in Algorithm 1, and a simple example is shown in Fig. 5. The list of completed nodes (denoted as LC) is the linked list where the input information for any nodes can be obtained from the output information of their predecessor nodes. According to the analysis of Ref. [10], the time-space consumption of the F-PTM model increases linearly with the increase of the number of circuit gates. 4. The feature set associated with circuit reliability The feature dataset is the raw data of the deep learning model, which has a critical impact on the final model. To obtain the dataset, we select the proper granularity to represent the circuit reliability as the first step, see Section 4.1 for details. Then we construct an efficient algorithm to extract the data corresponding to the selected feature set from the circuit net list, as described in Section 3.2. 4.1. Select the feature set associated with circuit reliability In reality, circuit reliability is affected by many factors, including circuit structure and process technology as well as the application environment, as shown in Fig. 6. Circuit structure mainly refers to the basic gates and their interconnection. The process technology is commonly measured by the functional and parametric faults of the devices, and the faults are usually quantified by the failure probability of the basic gates [18]. The application environment mainly refers to all possible circuit input signals, which are usually expressed by the input vectors [19].
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
5
Algorithm 1 Circuit reliability calculation for the applied input vectors. Input: Circuit net list Output: Circuit reliability 1. Parse circuit net list and initialize relevant parameters. 1.1) Read the net list, extract the information of the basic gates, build a LC for the circuit, and identify all the primary inputs and outputs of the circuit; then perform i = 1; 1.2) With the type, the failure probability p, and the input number ns of the basic gate, build its probability transfer matrix PMtype-ns and its ideal transfer matrix IMtype-ns ; 1.3) Extract the input signals of the circuit primary inputs, and construct the corresponding primary input probability distribution pipt, where pipt = [1, 0] when the input signal is 0; otherwise, pipt = [0, 1]. 2. Calculate the fault output probability distribution fopti and the ideal output probability distribution iopti for the i th node in LC. 2.1) Extract the type, the fault probability pi and the input number mi for the i th node in LC, and generate its probability transfer matrix PMtype-nsi and its ideal transfer matrix IMtype-nsi , then perform j = 1; 2.2) Extract the j-th input of gi . If it is the primary input of the circuit, assign pipt to the fault input probability distribution fiptij and the ideal input probability distribution iiptij ; otherwise, extract fiptij and iiptij from the fault output probability distribution foptq and the ideal output probability distribution ioptq of the q-th node (1 ≤ q < i) in LC; 2.3) If j = m, then go to step 2.4); else perform j = j + 1, and go to step 2.2); 2.4) Calculate fipti and iipti by fipti =fipti 1 fipti 2 … fiptimi , iipti =iipti 1 iipti 2 … iiptimi , where is the operation of tensor product; 2.5) Calculate fopti and iopti by fopti =fipti × PMtype-nsi iopti =iipti × IMtype-nsi ; 2.6) If the output of gi is the primary output of the circuit, then go to step 2.7); otherwise, go to step 2.8); 2.7) Calculate the circuit reliability r for the applied input vector by r=sum(fopti . × iopti ), where sum represents the summation operation; 2.8) If reach the end of LC, then go to step 3; otherwise, perform i = i + 1 and go to step 2.1). 3. Output r.
Fig. 5. A simple example for Algorithm 1 (p = 0.05).
Fig. 6. Feature set associated with circuit reliability.
4.1.1. Feature set associated with circuit structure Some studies [20] have found that the constituent units of the circuits, such as primary input number (PIs), circuit tier number
(Ts), circuit gate number (Gs), and the numbers of basic gates with different types, including the AND gate number (As), NOT gate number (Ns), OR gate number (Os), NAND gate number (NAs), and NOR gate number (NOs), are all the important members of the circuit structure. Further analysis finds that other than the above members, the XOR gate number (XOs), XNOR gate number (XNOs) and buff number (Bs) are also the important members of the circuit structure. In addition, the wires among the basic gates are also the important members. Analysis has found that the circuits with the same members mentioned above tend to have different reliabilities due to their different connections, such as their fan-outs [18,21]. Therefore, it is necessary to append the fan-out attribute as a structure member, which is quantified by the fan-out source number (FOs) and the fan-out number (Fs). Fig. 7 is the feature set associated with circuit structure in this study.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM 6
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
Fig. 7. Feature set associated with circuit structure.
4.1.2. Application environment As mentioned above, the application environment is usually measured by circuit input vectors, which are encoded with a binary string. The length of the binary string is equal to the number of primary inputs, 0 and 1 indicate the low-level input and the high-level input, respectively. According to the analysis presented in Section 4.1, we know that the input vectors represented by binary strings with unequal lengths are different from the attribute of other circuit features, such as As, Os and Ns. To unify their attributes, the input vectors are transferred to decimal representations for characterization in this study. However, there is no appropriate data type to represent the decimal data converted from an overly long binary string. To solve this problem, a partitioning strategy is given, the details are shown as follows. Firstly, we determine the dividing width according to the size of the applied data type. Secondly, we divide the applied input vector represented by an overly long binary string into several segments. Thirdly, each segment is mapped into a decimal representation using Eq. (12). Finally, we merge the results obtained in step 3 to construct a new input feature A using Eq. (13). Although this strategy will destroy the integrity of the input vectors, as long as the number of segments is small enough, the destructiveness can be reduced to be small enough, and the complexity of the DAN model can also be reduced. A relevant experiment is shown in Section 6.
g(bi ) =
k
ai j × 2 j ,
(12)
j=0
A = (g(b1 ), g(b2 ), · · ·, g(bn )),
(13)
Where n is the number of segments obtained by dividing a binary string, n = ceil(nb/blen) and ceil() is a ceiling function; nb and blen are the lengths of the binary string and a segment, respectively. bi is the ith segment, and bi = aik …ai 1 ai 0, i∈{1, 2, …, n}, k <= blen, aij ∈ {0, 1}, j∈{0, 1, …, k}; g(bi ) is a function that is used to map bi into a decimal numeral. Further analysis found that the DAN model requires the input feature with a deterministic dimension. However, the lengths of the input vectors for different circuits tend to be different, so that the lengths of corresponding features obtained by Eq. (13) are often different, which cannot meet the requirement of the DAN model for the input feature set. To deal with this problem, a unitive feature  for A is proposed in Eq. (14) to concretize the size of
the input feature set (Max_n).
⎧ (g(b1 ), g(b2 ), . . . , g(bn )), ⎞ n = Max_n ⎪ ⎨⎛ Max_n−n Aˆ = T (A ) = ⎝ , g(b1 ), g(b2 ), . . . , g(bn ), −1, . . . , −1⎠, n < Max_n ⎪ ⎩ (14) Where ‘−1’ means that the corresponding bits do not exist. For example, assuming that the input vectors for circuitscc1 and cc2 are ‘1010 0 010’ and ‘001101001100’, respectively, and blen = 3. According to Eqs. (12)–(14), Max_n is assigned with 4, and Âs for circuitscc1 and cc2 are expressed as (5, 0, 2, −1) and (1, 5, 1, 4), respectively. 4.1.3. Fault probability of basic gates With the introduction of new materials and new technologies as well as increased circuit integration, the effect of process technology on circuit reliability has become more significant [1]. The fault probability of basic gates PF can be measured by parameters, such as activation factor, electric field strength and defect size, which reflects the effect of the above-mentioned factors on circuit reliability. Therefore, PF is used to measure the level of process technology in this paper. For current technologies, Ref. [22] and other experiments have pointed out that PF is very small and can be set to [0, 0.1]. For convenience, PF is set as follows: PF = {10d | d = −1, −2,…, −10}. In summary, so far the feature set for circuit reliability is determined and expressed as follows: V = [PIs, Ts, Gs, As, NAs, Os, NOs, Ns, XOs, XNOs, Bs, FOs, Fs, Â, PF], where the size of V is Max_n + 14. The labeled data (circuit reliability R) corresponding to V can be accurately calculated by the F-PTM method. 4.2. Data collection There are two steps taken to create a dataset for circuit reliability. Firstly, we produce circuit sub-net list, due to that the feature set presented in Section 3.1 is not enough to identify a circuit with multiple primary outputs. Dividing the circuit so that each subcircuit contains only one primary output is an effective solution, and this process is presented in Algorithm 2. Secondly, we build the dataset corresponding to Vby combining  with PF, and calculate R corresponding to V by the F-PTM method. This process is presented in Algorithm 3. Since Algorithms 2 and 3 are only used to create the dataset and are not involved in the prediction of the DAN model, by convention, their time-space consumptions are not analyzed in this study. Through Algorithm 2, a circuit sub-netlist with only one primary output can be produced. Then, according to the requirements
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
7
Algorithm 2 Circuit sub-netlist produce algorithm. Input: circuit netlist (circuit.isc) Output: Circuit sub-netlist 1. Parse circuit and initialize relevant parameters. 1.1) Perform i = 1, j = 1; 1.2) Traverse netlist, if reach its end, then go to step 1.5); else extract its i th record; 1.3) If the input for the record is labeled, then put it into the end of LC, or if it is the primary output of the circuit, then put it into the list of output nodes (denoted as LO) together with its position in the LC; otherwise, put it into the list of incompleted nodes (denoted as LI); 1.4) Perform i = i + 1 and go to step 1.2); 1.5) Traverse LI. If all nodes are labeled, go to step 2; otherwise, extract the j-th record from LI and its input from LC; 1.6) If the record is labeled, then put it into the end of LC and label its input. If it is the primary output of the circuit, put it into LO together with its position in LC; 1.7) Perform j = j + 1 and go to step 1.5). 2. Produce circuit sub-netlist. 2.1) Perform i = 1 and j = 1; 2.2) Traverse LO, if reach its end, go to step 3; 2.3) Extract the i th node gi from LO and label gi with ni; 2.4) Traverse LC from gi up to circuit primary inputs with the width-first recessive search algorithm and label the corresponding nodes on the sensitized path with ni; 2.5) Create an empty sub-netlist file named circuit_ni.isc, extract the nodes labeled with ni in LC and write them into the end of the circuit_ni.isc; 2.6) Perform i = i + 1 and go to step 2.2). 3. End of the algorithm.
Algorithm 3 Reliability-oriented data collection method. Input: Circuit sub-netlist Output: Dataset [V, R] 1. Parse sub-netlist and initialize the relevant parameters. 1.1) Perform step 1 in Algorithm 2; 1.2) Layer circuit and identify its layer number. 2. Collect the features for circuit topology structure and build the corresponding dataset TCV. 3. Createthe feature dataset V and its corresponding labeldata R. 3.1) Initialize the vector of fault probability ofgates VPF and the number of circuit input vectors num, initialize c = 0 and j = 1; 3.2) Traverse VPF, if reach its end, then go to step 3.8); else extract the j-th data pfj from VPF; 3.3) If c
of the DAN model and using the obtained sub-netlists, the feature dataset corresponding to V and R can be created by Algorithm 3. 5. Experimental setup According to the analysis mentioned-above, we know that the following three works need to be performed to obtain a trained DAN model for the prediction of circuit reliability. Firstly, the data collected by Algorithm 3 needs to be preprocessed to relieve its size effect caused by the inconsistencies in data gradients, the details are shown in Section 5.1. Secondly, the error evaluation function should be given to determine whether the training of the DAN model meets the stop criteria, the function is presented in Section 5.2. Thirdly, A grid-based search method is proposed to find a reasonable network structure for the DAN model, the details are shown in Section 5.3.
the data with a large gradient, the adverse effect of the size effect on the dataset cannot be treated well only by the normalization. Other methods are required to preprocess the dataset to reduce its gradient, and the method adopted is determined by the distribution of the dimension data itself. For example, the logarithm function presented in Eq. (16) can be used to preprocess the dimension data that follows an exponential distribution, where y is the converted value of x; xmax and xmin are the maximum and the minimum values of sample x, respectively; ymax and ymin are the maximum and the minimum values of y in the mapping interval, respectively.
y = (y max −y min ) ×
x − x min + y min, x max −x min
y = log10 x,
(15) (16)
5.2. Error evaluation function 5.1. Data preprocessing Our analysis found that the data corresponding to each feature in V tends to have different dimensions. To relieve their size effect, the raw data often needs to be standardized or normalized before being input into the DAN model. Noting that the dataset corresponding to V is discrete and its value range is fixed, so the linear transformation function (also called deviation standardization) presented in Eq. (15) is used to normalize the dataset in this study, which maps the dataset to [0,1] or [−1,1] to speed up the gradient descent and accelerate the convergence of the model. However, for
In general, the error evaluation functions are used to evaluate the prediction accuracy of the constructed model to judge whether the model meets the requirements, such as root mean squared error (RMSE) [23] and mean absolute percentage error (MAPE) [23], which are shown in Eqs. (17) and (18). Note that RMSE can be well used to evaluate the prediction accuracy of the DAN model, but cannot measure the extent to which it is unbiased, while MAPE can solve the problem of unbiased testing of the prediction model well. Therefore, they are both used to evaluate the prediction error of the DAN model constructed in this study, where yi is obtained by
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM 8
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
Algorithm 4 Selection method for the structure of the DAN model. Input: Training dataset Output: The optimal DAN model 1. Initialize the relevant parameters, including the maximum number of hidden layers ly, the maximum number of hidden layer nodes nd, the maximum training number nt, the increment sl, the constant ln and the counter l = 1. 2. If l ≤ly, then initialize i = 1 and go to step 3; else go to step 10. 3. If 2i+ln ≤ nd, then go to step 4; else go to step 9. 4. Initialize the number of hidden layer nodes hnd=2i + ln and the training number t = ne. 5. Construct and train AE with layer-wise learning. 5.1) Initialize j = 1; 5.2) If t ≤ nt, then initialize label-free training data h0 ; else go to step 8; 5.3) If j ≤ l, then construct the j-th AE with hnd hidden layer nodes (denoted as AE_l_i_t_j); else go to step 6; 5.4) Perform t trainings on AE_l_i_t_j by data hj -1 ; 5.5) Encode hj -1 with the encoder of AE_l_i_t_j, and record the result as hj ; 5.6) Perform j = j + 1, then go to step 5.3). 6. Construct and train the DAN model. 6.1) Construct a deep neural network DAN_l_i_t with l hidden layers and hnd nodes in each hidden layer; 6.2) Initialize j = 1; 6.3) If j ≤ l, assign the shared weight matrixes and offset vectors in the input layer and the hidden layer of AE_l_i_t_j to the shared weight matrixes and bias vectors in the j-th layer and the (j + 1)-th layer of DAN_l_i_t, respectively; else go to step 6.5); 6.4) Perform j = j + 1, then go to step 6.3); 6.5) Perform t trainings on DAN_l_i_t in supervised manner using the labeled training dataset. 7. Perform t = t + sl, then go to step 5. 8. Perform i = i + 1, then go to step 3. 9. Perform l = l + 1, then go to step 2. 10. Test the constructed DAN model using the test dataset and calculate its corresponding RMSE and MAPE. 11. Extract the DAN model with the smallest error. 12. End the algorithm.
the method presented in Algorithm 1, yˆi is the predicted value of yi by the DAN model, and N is the sample size of the test dataset.
RMSE =
N yi − yˆi2 i=1
N
,
(17)
N 1 yi − yˆi MAPE = yi , N
(18)
i=1
5.3. The structure of the DAN model As mentioned above, the structure of the DAN model mainly contains three parameters which are the number of input layer nodes, the number of hidden layers and the number of hidden layer nodes. Different combinations of the three parameters in the DAN model tend to result in different prediction accuracies for the same feature dataset [24]. Noting that the value ranges of the parameters are determined by the size of the feature set which has been presented in Section 4.1, thus we can find an optimal network structure for the DAN model in the given ranges using the search algorithms. The details are discussed as follows. Except for the offset nodes, each node in the input layer in the DAN model only receives one dimension data and each dimension data corresponds only to one node in the input layer [23]. Therefore, the number of the nodes in the input layer for the DAN model is assigned to Max_n + 15 when the size of the feature set is Max_n + 14. At present, except for the search algorithms, there is no effective way to determine the optimal numbers of hidden layers and hidden layer nodes of the model for a given feature dataset to meet the prediction accuracy of the model. However, the solution domain consisting of all possible combinations of the number of hidden layers and the number of hidden layer nodes tends to have a very large scale, thus resulting in time-consuming to find the optimal solution by the exhaustive search algorithms. Thus, it is necessary to improve the search algorithm, so as to speed up the solution. In view of the lack of mathematical foundation to support the accurate configuration of the number of hidden layers and
the number of hidden layer nodes in the deep learning networks [25], according to the characteristics of the provided dataset and referring to the approaches presented in Ref. [24,25], a grid-based method is proposed to configure the structure of the DAN model in the sufficiently large domains. Although the proposed method may not find the best configuration, it can provide a valid configuration for the DAN model, because the range of the number of hidden layers is large enough to effectively support the simple value-taking method of the number of hidden layer nodes [24]. The details of the construction of the DAN model are as follows. Firstly, we initialize the number of hidden layers l(∈{1, 2, …, ly}), and the number of hidden layer nodes 2i (i∈{1, 2, …, md}) as well as the maximum number of training epochs nt. Then, we perform Algorithm 4 on the above-mentioned value ranges based on the training method presented in Section 2.1 to find the optimal numbers of hidden layers and hidden layer nodes, as well as the optimal training number. The flowchart is presented in Fig. 8. Similarly, the time-space complexity of Algorithm 4 is not analyzed in this paper, due to the fact that Algorithm 4 is not involved in the prediction of the DAN model.
6. Simulation results To verify the effectiveness of the model proposed in this paper, some simulations are performed on the 74-series and ISCAS 85 benchmark circuits [5,7,10] using a personal computer with a 1.9 GHz processor and 4 GB RAM. Firstly, we collect data using Algorithm 3 and preprocess the data with the method presented in Section 5.1 to relieve the size effect. Secondly, according to the characteristics of the applied dataset, we construct an appropriate structure for the DAN model using the method presented in Algorithm 4. The results are shown in Tables 2 and 3. Thirdly, we perform experiments using the MC method, the F-PTM model presented in Algorithm 1 and the method presented in Ref. [26] to test the effectiveness of the DAN-based prediction model for circuit reliability. The results are shown in Tables 4 and 5. In addition, the effects of some main parameters on the prediction accuracy of the constructed model are also analyzed. The results are shown in Figs. 13 and 14.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
9
Fig. 8. Selection scheme for the structure of the DAN model. Table 1 Preprocessing methods for the raw feature dataset. Input /Output
Parameters
Preprocessing
PIs Ts Gs [As, Nas, Os, Nos, Ns, Xos, XNOs, Bs] FOs Fs PF Â R
primary input number layer number gate number The numbers of basic gates of different types fanout source number fanout number fault probability of basic gate input vector circuit reliability
map map map map map map map map map
6.1. Data collection and preprocessing Firstly, we parse 20 circuits selected from the 74-series and ISCAS85 benchmark circuits [5,7,10] using Algorithm 2 to generate 508 circuit sub-netlists, where each sub-netlist contains only one primary output, PIs∈[1, 194], Gs∈[1, 2327], Ts∈[2, 174]. Secondly, we extract 60,800 records corresponding to from the sub-netlists using Algorithm 3 when blen = 10, Max_n = 20, and PF∈[10−1 , 10−10 ], and the primary input vectors are randomly generated. Finally, we preprocess the collected data by the method presented in Section 5.1 to relieve the size effect. With results are shown in Table 1.Where usage of mapping to [−1, 1] is beneficial to the uniform distribution of the relatively concentrated data used in this study. 6.2. Prediction model construction Due to the fact that there is no effective method to quickly give a reasonable structure for the DAN model [23,24], according to the
to to to to to to to to to
[−1, 1] by Eq. (15) [−1, 1] by Eq. (15) [–1, 1] by Eq. (15) [−1,1] by Eq. (15) [−1,1] by Eq. (15) [−1,1] by Eq. (15) [–1, 1] by Eqs. (15) and (16) [–1, 1] by Eq. (15) [–1, 1] by Eq. (15)
characteristics of the dataset, we determine its structure using the experimental method presented in Algorithm 4 in this paper. The steps are as follows. Firstly, we extract 50,800 training data and 10,0 0 0 test data from the 60,800 data given in Section 5.1 using insight from expertise research and the method presented in Ref. [10], then we initialize ly = 7, ln = 3, nd = 6, nt = 10 0 0, and sl = 100. Secondly, we determine the appropriate numbers of hidden layers and hidden layer nodes ofthe DAN model using Algorithm 4, the result is shown in Fig. 9. Finally, according to the results shown in Fig. 9, we extract the optimal numbers of the weights and iterations, as well as the training time for the model with the smallest RMSE when the numbers of hidden layers and hidden layer nodes are different. The results are shown in Tables 2 and 3. The main parameters in the model are the learning rate η = 0.05, momentum super parameter γ = 0.5, batch size = 100; NumNodes-Hlayer is the number of hidden layer nodes, and Num-Hlayer is the number of hidden layers. Fig. 9 shows that the DAN model has the smallest RMSE when the number of hidden layer nodes is 128 for all hidden layers for
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM 10
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
Fig. 9. Relationship between the number of hidden layer nodes and RMSE in different numbers of hidden layers.
Table 2 Test error and training time for the DAN model when NumNodes-Hlayer is 128. Num-Hlayer
MAPE
RMSE
Epochs
weight number
Time (s)
1 2 3 4 5 6 7
0.0300 0.0289 0.0283 0.0283 0.0278 0.0279 0.0280
0.0509 0.0496 0.0490 0.0488 0.0482 0.0484 0.0485
1000 1000 900 900 1000 1000 1000
4480 20,864 37,248 53,632 70,016 86,400 102,784
744 1886 3045 4436 5505 6391 7720
the dataset provided in this paper. When NumNodes-Hlayer is 128, the RMSE firstly decreases with the increase of Num-Hlayer, then increases with the increase of Num-Hlayer when Num-Hlayer is greater than 5. In short, the DAN model has the smallest RMSE when Num-Hlayer and NumNodes-Hlayer are 5 and 128, respectively. This means that the DAN model with 5 hidden layers and 128 hidden layer nodes can best explain the dataset in this study. MAPE is not described in detail in this paper because MAPE and RMSE are similar. Tables 2 and 3 show that the epochs reach or approach to 10 0 0 when the RMSE is smallest for the different Num-Hlayer and NumNodes-Hlayer. In addition, the training time and complexity (denoted as weight number) both increase with the increase of Num-Hlayer and NumNodes-Hlayer in the DAN model, and the model has the smallest RMSE and MAPE when Num-Hlayer and NumNodes-Hlayer are 5 and 128, respectively. Therefore, from this point forward, Num-Hlayer, NumNodes-Hlayer, and epochs are set to 5, 128, and 10 0 0 to perform the experiments, respectively. 6.3. Accuracy To further verify the effectiveness of the proposed model, the following experiments are performed on the 74-series and ISCAS85
benchmark circuits. Firstly, we compare the results obtained by the F-PTM model and the DAN model presented in Section 6.2 on 100 samples extracted from the test set. The results are shown in Fig. 10. We further compare them with the results obtained by the MC method and the method presented in Ref. [26] to test the prediction accuracy of the constructed DAN model. The results are shown in Table 4. Secondly, we compare the time-space consumptions obtained by the four methods mentioned above. The results are shown in Table 5. Finally, we analyzed the effect of some main features (such as Ts and Gs) on RMSE and MAPE for the dataset with an imbalanced distribution. The results are shown in Figs. 11 and 12. For ease of calculation, and without loss of generality, all circuit primary inputs are valued at 0 and 1, respectively, which are expressed as inputs = 0 and inputs = 1, respectively; pf is assigned with 0.01 which is extracted from [10−1 , 10−10 ] in a random way, and the number of experimental samples for the MC method is 10 0,0 0 0 [10]; the result for each primary output for the MC, FPTM and Ref. [26] methods is extracted from the calculation results for the entire circuit unless otherwise stated. Notice that although the MC method has some accuracy loss due to the fault injection adopted in a pseudo-random way, it can be used to evaluate the reliability of circuits at different abstract levels, and its results are widely recognized by academia and industry [27]. Therefore, its results are also used as reference values in this paper. Fig. 10 shows that the results obtained by the F-PTM model are quite close to that by the DAN model on the test data selected at random. The results in Tables 4 and 5 show that the accuracy of the proposed method is lower than, but quite close to, that of the MC method and the F-PTM method. Its accuracy is higher than, but close to, that of Ref. [26] method. The results can be proven by its RMSEs of 5.026% and 4.986%. However, the predicted values of this method still incur small error on individual circuits. For example, the maximum relative error is 17.4046% for C1908-2888. In
Table 3 Test error and training time for the DAN model when Num-Hlayer is 5. Num Nodes-Hlayer
MAPE
RMSE
Epochs
Weight number
Time (s)
16 32 64 128 256 512
0.0309 0.0291 0.0283 0.0278 0.0283 0.0284
0.0501 0.0493 0.0489 0.0482 0.0493 0.0491
1000 1000 1000 1000 900 1000
1584 5216 18,624 70,016 271,104 1,066,496
1152 2018 2933 5505 13,116 54,075
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
11
Fig. 10. Distribution of the predicted values obtained by the DAN model for data extracted from the test dataset. Table 4 Comparison of the results obtained by the different methods (pf = 0.01). Circuits
74148-78 74182-63 74283-97 741811-201 c499-753 c1355-1324 c1908-2888 c3540-1713 c6288-1581 c7552-10839 MAPE(%) RMSE(%)
MC
F-PTM
Ref. [26] method
The proposed method
Inputs = 0
Inputs = 1
Inputs = 0
Inputs = 1
Inputs = 0
Inputs = 1
Inputs = 0
Inputs = 1
0.9713 0.9513 0.9892 0.6155 0.9779 0.9546 0.7233 0.9330 0.9051 0.8546 – –
0.8974 0.9899 0.9418 0.5659 0.9703 0.953 0.6889 0.9795 0.9259 0.8563 – –
0.9716 0.9514 0.9900 0.6157 0.9788 0.9546 0.7239 0.9334 0.9060 0.8553 0.554 0.058
0.8988 0.9900 0.9419 0.5659 0.9712 0.9529 0.6890 0.9801 0.9268 0.8575 0.593 0.074
0.97 0.95 0.99 0.71 0.98 0.95 0.57 0.93 0.89 0.75 5.722 6.606
0.89 0.99 0.94 0.93 0.97 0.94 0.68 0.98 0.92 0.80 5.064 11.67
0.9613 0.9699 0.9622 0.6670 0.9569 0.8840 0.6744 0.9738 0.9066 0.9639 4.771 5.026
0.9554 0.9660 0.9378 0.6182 0.9283 0.9440 0.8088 0.9734 0.9056 0.8951 4.857 4.986
Table 5 Comparison of the time-space consumptions obtained by the different methods (pf = 0.01, PIi = 1). Circuits
74148_78 74182_63 74283_97 741811_201 c499_753 c1355_1324 c1908_2888 c3540_1713 c6288_1581 c7552_10,839 AVG
MC
Ref. [26] method
This proposed method
Time (s)
Memory (MB)
Time (s)
F-PTM Memory (MB)
Time (s)
Memory (MB)
Time (s)
Memory (MB)
5472 3420 13,428 15,766 119,490 888,717 1503,187 2754,009 4656,292 6372,099 1633,188
0.1030 0.0893 0.1321 0.2580 0.6344 1.7260 2.4137 4.4985 8.0071 9.6042 2.75
0.108 0.102 0.164 0.3318 2.329 18.550 37.284 113.241 377.587 149.011 69.87
0.1040 0.0913 0.1342 0.2640 0.6429 1.750 2.448 4.562 8.120 9.739 2.79
0.131 0.095 0.147 0.762 2.233 15.203 30.096 105.318 375.506 465.511 99.50
0.1030 0.0893 0.1320 0.2434 0.6343 1.726 2.298 4.498 8.007 9.604 2.73
0.01593 0.01587 0.01613 0.01598 0.01573 0.01618 0.01585 0.01596 0.01584 0.01596 0.01594
2.5151 2.5151 2.5151 2.5151 2.5151 2.5151 2.5151 2.5151 2.5151 2.5151 2.5151
addition, the accuracy is often different for the same circuit with different inputs, such as for C499-753. The time and space consumptions of this method are approximately constant, and are basically unaffected by the change in circuit scale, while that of the MC method, the F-PTM model and Ref. [26] method are affected by the circuit size, and increase with the increase in circuit scale, especially for the MC method. The proposed method, on average, is approximately 102,458,469 times faster than the MC method, and approximately 4383 times faster than the F-PTM model, and approximately 6242 times faster than Ref. [26] method.
Moreover, comparing with no-mapping operations, the experiments have shown that the mapping operations presented in Table 1 can not only boost the convergence speed, but also can improve the predicted accuracy, because the size effect existing in the dataset is relieved by the mapping operations. For example, the MAPE and the RMSE are 11.51% and 12.84%, respectively, when nomapping operations are performed. This is mainly because the MC method, the F-PTM model and Ref. [26] method calculate circuit reliability by the strategy of structure simulation, while the proposed method predicts circuit
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM 12
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
Fig. 11. Effect of Gs on RMSE and MAPE in imbalanced distribution.
Fig. 12. Effect of Ts on RMSE and MAPE in imbalanced distribution.
reliability by the method of feature learning. Ref. [26] method causes a loss of accuracy on some circuits due to the repeated consideration on some gates when using some applied input vectors. With this method, it is often difficult to achieve full learning for small-sized features, which lead to the emergence of outliers, resulting in some loss in prediction accuracy. This is a major problem to be solved in the future. The computational consumption of the proposed method is basically unaffected by circuit inputs, due to the fact that it is the equivalent of a mathematical analytical expression. The computational consumptions of the MC method, the F-PTM model and Ref. [26] method increase with the increase in circuit size, especially for the MC method, because they implement their evaluation by the simulation of circuit structure. The MC model requires large-scale simulations to ensure its accuracy due to the adopted fault injection with a pseudo-random way. Figs. 11 and 12 show that the effect of the dataset with imbalanced distributions on RMSE and MAPE tends to be different, and the effect of the dataset corresponding to the same feature with imbalanced distribution on RMSE and MAPE also tends to be different. The feature samples with a high ratio in the dataset tend to have small RMSE and MAPE. This is mainly because the DAN model extracts more information from the feature samples with a higher ratio in the dataset, resulting in better performance in processing those feature samples. This is to say that the problem for the samples with an imbalanced distribution must be addressed to further improve the prediction accuracy of the DAN model, which is beyond the scope of this paper. We will analyze it in future.
6.4. Relevantfactors As mentioned in Sections 4 and 5, the completeness of the feature set and the size of the feature set associated with the input vector are all important factors that influence the prediction accuracy of the DAN model. The experimental results are shown in Figs. 13 and 14. Fig. 13 shows the increment changes in RMSE and MAPE for each hidden layer in the DAN model before and after removing the 5 features of XOs, XNOs, Bs, FOs, and Fs. These changes are equal to the differences between the indexes after the feature removal and the indexes before the feature removal. Fig. 14 shows the increment changes in RMSE and MAPE for each hidden layer in the DAN model when the division widths of the input vector are 1, 5, and 10, respectively. Scheme A, Scheme B, and Scheme C are the differences among the indexes corresponding to the division widths of 5 and 10, 1 and 5, 1 and 10, respectively. Figs. 13 and 14 show that both RMSE and MAPE for each hidden layer in the DAN model undergo a certain increase when the features of XOs, XNOs, Bs, Fos, and Fs are removed, or the division width of the input vector is reduced. That is to say, the mentioned-above operations result in a loss of accuracy in the model. Therefore, a more complete feature set and a larger division width are needed to improve the accuracy of the DAN model constructed in this study. In addition, other factors, such as learning rate and momentum factor, also tend to cause a loss of accuracy in the DAN model. This is the research that we will carry out in the future.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
13
Fig. 13. Changes in RMSE and MAPE for the feature set of size 34 to 29.
Fig. 14. Changes in RMSE and MAPE for Scheme A, B and C, corresponding to  equal to 194, 39 and 20, respectively.
7. Conclusion In view of the computational complexity and hysteresis encountered in simulation-based methods for circuit reliability assessment, this paper proposes a new prediction method for circuit reliability based on a DAN model. Firstly, according to the analysis of the factors associated with circuit reliability, a feature set to characterize circuit reliability is determined. Secondly, on the basis of the characteristics of both the circuit netlists and the DAN model, an effective collection and preprocessing method for the dataset corresponding to the feature set is proposed to meet the requirements of the DAN model. Thirdly, according to the scale and characteristics of the dataset, a construction method for the DAN model based on grid search technology is proposed to determine its reasonable structure. Simulation results on benchmark circuits show that the trained DAN model could be used to achieve a fast assessment for circuit reliability at the conception stage. The accuracy of this model closely approached to that of the simulation-based methods, and its time-space consumption is basically unaffected by circuit size and is approximately constant. However, some issues uncounted in the experiments need to be solved. For example, both the imbalanced distribution of the feature dataset and the representation of the features associated with the input vectors tend to damage the accuracy of the DAN model. We leave these problems for future research.
Declaration of Competing Interest None. Acknowledgement Financial support for the study was provided by the National Natural Science Foundation of China (Nos. 61502422, 61772199, 61802123, 61503338 and 61432017), the Natural Science Foundation of Zhejiang Province (Nos. LY18F020028 and LY18F030023). References [1] A.S. Oates, K.P. Cheung, Reliability of nanoelectronic devices, Nanoelectron. Mater. Devices Appl. 1 (4) (Apr. 2017) 317–330. [2] Y. Chien-Chih, Probabilistic analysis for modeling and simulating digital circuits, Department of Electrical Engineering, The University of Michigan, Michigan, USA, 2012. [3] S. Krishnaswamy, I.L. Markov, J.P. Hayes, Design, Analysis and Test of Logic Circuits Under Uncertainty, Springer Science & Business Media, 2012. [4] S. Krishnaswamy, G.F. Viamontes, I.L. Markov, et al., Probabilistic transfer matrices in symbolic reliability analysis of logic circuits, ACM Trans. Des. Autom. Electron. Syst. 13 (1) (Jan. 2008) 1–8. [5] J. Han, H. Chen, E. Boykin, et al., Reliability evaluation of logic circuits using probabilistic gate models, Microelectron. Reliab. 51 (2) (Feb. 2011) 468–476. [6] T. Rejimon, K. Lingasubramanian, S. Bhanja, Probabilistic error modeling for nano-domain logic circuits, IEEE Trans. Very Large Scale Integr. VLSI Syst. 17 (1) (Jan. 2009) 55–65.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM 14
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
[7] J. Han, C. Hao, J.H. Liang, et al., A stochastic computational approach for accurate and efficient reliability evaluation, IEEE Trans. Comput. 63 (6) (Jun. 2014) 1336–1350. [8] Y. Lecun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7) (May 2015) 436–444. [9] J. Schmidhuber, Deep learning in neural networks, Neural Netw 61 (1) (Jan. 2015) 85–117. [10] J. Xiao, J. Lou, J. Jiang, A fast and effective sensitivity calculation method for circuit input vectors, IEEE Trans. Reliab. (2019), doi:10.1109/TR.2019.2897455. [11] G. Trigeorgis, F. Ringeval, R. Brueckner, et al., Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network, in: Proceedings of the IEEE International Conference on Acoustic Speech and Signal Processing, May 2016, pp. 5200–5204. [12] G.E. Hinton, S. Osindero, W. Teh, A fast learning algorithm for deep belief nets, Neural Comput. 18 (7) (Jul. 2006) 1527–1554. [13] G.E. Hinton, R.R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science 313 (28) (Jul. 2006) 504–507. [14] G.E. Hinton, Learning multiple layers of representation, Trends Cogn. Sci. 11 (10) (Oct. 2007) 428–434. [15] S. Haykin, B. Kosko, Gradient based learning applied to document recognition, Proc. IEEE 86 (11) (Nov. 1998) 306–351. [16] J. Gehring, Y. Miao, F. Metze, et al., Extracting deep bottleneck features using stacked auto-encoders, in: Proceedings of the IEEE International Conference on Acoustic Speech and Signal Processing, Oct. 2013, pp. 3377–3381. [17] Y. Bengio, Learning deep architectures for ai, Found. Trends Mach. Learn. 2 (1) (Jan. 2009) 1–27. [18] J. Xiao, W. Lee, J.H. Jiang, et al., Circuit reliability estimation based on an iterative PTM model with hybrid coding, Microelectron. J. 52 (4) (Jun. 2016) 117–123. [19] J. Xiao, W. Lee, J.H. Jiang, et al., Sensitivity evaluation of input vectors with masking effects in digital circuits, Chin. J. Comput. 41 (10) (Oct. 2018) 2282–2294. [20] J. Xiao, J.H. Jiang, J.R. Liang, Transistor-level oriented calculation of reliability for generalized gates based on ptm, Sci. China Inf. Sci. 44 (10) (Oct. 2014) 1226–1238. [21] J. Xiao, J.H. Jiang, X.X. Li, et al., A novel trust evaluation method for logic circuits in IoT applications based on the E-PTM model, IEEE Access. 6 (7) (Jul. 2018) 35683–35696. [22] Y. Du, S. Chen, A novel layout-based single event transient injection approach to evaluate the soft error rate of large combinational circuits in complimentary metal-oxide-semiconductor bulk technology, IEEE Trans. Reliab. 65 (1) (Mar. 2016) 248–255. [23] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, Posts & Telecom Press, 2017. [24] H. Wenhao, S. Guojie, H. Haikun, et al., Deep architecture for traffic flow prediction:deep belief networks with multitask learning, IEEE Trans. Intell. Transp. Syst. 15 (5) (Oct. 2014) 2191–2201. [25] Y. Bengio, Deep Learning, MIT Press, 2016. [26] W. Ibrahim, M. Shousha, J.W. Chinneck, Accurate and efficient estimation of logic circuits reliability bounds, IEEE Trans. Comput. 64 (5) (May 2015) 1217–1229. [27] R.Y. Rubinstein, D.P. Kroese, Simulation and the Monte Carlo Method, Wiley, 2016. Jie Xiao received his Ph.D. degree in computer system architecture from Tongji University, Shanghai, China, in 2013. He is currently working with the department of computer science and technology, Zhejiang University of Technology, Hangzhou, China. His-current research interests include reliability evaluation and fault-tolerant design, deep learning and combinatorial optimizationcomputation. He also serves as a consultant and technical adviser for a research institute in electronic information fields. He is a member of CCF.
Weifeng Ma received his M.E. degree in college of computer science and technology from Zhejiang University of Technology, Hangzhou, China, in 2015. He is now a graduate at Zhejiang University of Technology. His-current research interests include reliability evaluation and deep learning.
Jungang Lou received the B.S. degree in Mathematics from Zhejiang Normal University, China, in 2003, and the M.S. degree in computational mathematics and the Ph.D. degree in computer science and technology from Tongji University, Shanghai, China, in 2006 and 2010, respectively. He is currently a Professor with the School of Information Engineering, Huzhou University, Huzhou, China. He also holds a postdoctoral position at the Institute of Cyber-Systems and Control, School of Control Science and Engineering, Zhejiang University, Zhejiang, China. He was a Visiting Scholar with the department of Computer Science at The University of Texas at San Antonio between Nov. 2017 and May 2018. His-current research interests include dependable computing, neural network optimization, reliability engineering, computer system performanceevaluation, and time series prediction. He has published over 60 papers in refereed international journals including Journal of Network and Computer Applications, IEEE Transactions on Neural Networks and Learning Systems, IEEE Transactions on Cybnetics and so on. Jianhui Jiang received his B.E., M.E. and Ph.D. degrees in traffic information engineering and control from Shanghai Tiedao University (in April 20 0 0, it was merged to Tongji University) in 1985, 1988, and 1999, respectively. From 1994–20 0 0, he was an Associate Professor of Computer Science and Technology at Shanghai Tiedao University. Since 20 0 0, he has been a full professor of Computer Science and Technology at Tongji University. From 2007– 2011, he was chair of the Department of Computer Science and Technology at Tongji University. Since 2011, he is Associate Dean of the School of Software Engineering at Tongji University. He is Vice Director of Technical Committee on Fault-tolerant Computing, Chinese Computer Federation (CCF). He has served on several program committees of national or international symposiums or workshops including IEEE Pacific Rim International Symposium on Dependable Computing, IEEE Asian Test Symposium, IEEE Workshop on RTL and High-Level Testing. He has coauthored two books and published more than 200 technical papers. His-current research interests include dependable systems and net-works, software reliability engineering, VLSI/SoC testing and fault-tolerance. His-research support is provided by the Ministry of Science and Technology of PRC, the National Natural Science Foundation of China, the Science and Technology Committee of Shanghai, the Science and Technology Department of Jiangsu, Huawei, and Shanghai Shentong Metro Group Company. He is a senior member of CCF. Yujiao Huang received the B.S. degree in information and computer science, the M.S. degree in computational mathematics and the Ph.D. degree in control theory and control engineering from Northeastern University, Shenyang, China, in 2008, 2010 and 2014, respectively. She is now a lecturer at Zhejiang University of Technology. Her research interests are in areas of artificial neural networks, stability theory, and dynamical systems.
Zhanhui Shi do his postgraduate work at the department of computer science and technology, Zhejiang University of Technology, Hangzhou, China. His-current research interests include reliability evaluation and fault-tolerant design, deep learning and combinatorial optimizationcomputation.
Qing Shen received her M.E. degree in Computer Application Technology in July 2007 from North University of China. She is currently an Associate Professor with the School of Information Engineering of Huzhou University. Her current research interests including artificial intelligence, pattern recognition, kernel learning, software testing, reliability evaluation.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx Xu-Hua Yang received his B.E. degree in Automation from China University of Petroleum, Dongying, China, in 1993; and received his MSdegree and Ph.D. degree in control science and engineering from Zhejiang University, Hangzhou, China, in 2001 and 2004, respectively.
15
He is currently a Professor with Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China. His-current research interests include artificial intelligent, complex network system, intelligent transportation sytem, link prediction, and deep learning.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM
[m5G;September 10, 2019;23:55]
Neurocomputing xxx (xxxx) xxx
Contents lists available at ScienceDirect
Neurocomputing journal homepage: www.elsevier.com/locate/neucom
Circuit reliability prediction based on deep autoencoder network Jie Xiao a, Weifeng Ma a, Jungang Lou b,∗, Jianhui Jiang c, Yujiao Huang a, Zhanhui Shi a, Qing Shen b, Xuhua Yang a a
College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China College of Information Science, Huzhou University, Huzhou, China c Software Engineering Department, Tongji University, Shanghai, China b
a r t i c l e
i n f o
Article history: Received 1 March 2019 Revised 14 June 2019 Accepted 7 July 2019 Available online xxx Communicated by Dr. Ma Jiayi Keywords: Logic circuit Input vector Feature set Deep auto encoder network Reliability prediction
a b s t r a c t As semiconductor feature size continues to decrease and the density of integration continues to increase, highly reliable circuit design is experiencing many challenges, including reliability evaluation, which is one of the most important steps in circuit design. However, faced with the very large scale of integrated circuits at present, traditional simulation-based methods are slightly inadequate in terms of computational complexity and do not apply to the circuits at the concept stage. To solve this problem, this paper presents a new prediction method for circuit reliability based on deep auto encoder networks. Firstly, we analyze and extract the main features associated with circuit reliability. Next, we construct an efficient method for data collection by combining the characteristics of the feature set with the requirements of deep auto encoder networks. Then, we build a deep auto encoder network model oriented to circuit reliability prediction in a supervised learning manner. Simulation results on 74-series circuits and ISCAS85 benchmark circuits show that although the accuracy of the proposed method is slightly lower than that of both the Monte Carlo (MC) method and the fast probabilistic transfer matrix (F-PTM) model, its timespace consumption is approximately constant on different circuits, and it is 102,458,469 times faster than the MC method, and approximately 4,383 times faster than the F-PTM model. Furthermore, the proposed method could be used to predict circuit reliability at the conceptual stage, and it is a very efficient approximation method that could greatly reduce the power consumption of the calculation. © 2019 Published by Elsevier B.V.
1. Introduction With the extensive application of integrated circuit products in various fields, such as aviation, medical, and industry, high reliability is a requirement [1]. However, with the continued reduction in semiconductor feature size and the increase in integration density, uncertainty failure, such as intermittent and transient faults, has a more pronounced effect on circuit reliability [2]. Therefore, improving the reliability of circuits is a very important task that is directly related to the safety of people’s lives and property. Past analyses have found that reliability evaluation is one of the most important bases for improving circuit reliability [3]. Faced with the current very-large-scale integrated circuits, we need a quick and efficient evaluation method to measure their reliability at the early stages of circuit design so as to make timely decisions, thereby shortening the product development cycle to improve market
∗
Corresponding author. E-mail addresses: [email protected] (J. Xiao), [email protected] (J. Lou), [email protected] (J. Jiang).
competitiveness. Circuit reliability refers to the probability that the circuit can correctly perform the specified tasks in a certain environment [2]. At present, the evaluation approaches for circuit reliability mainly include the measurement-based, the simulation-based and the analytical approach. There are two basic schemes in the measurement-based approach: sequential truncation tests, and timed or fixed-number truncation tests. Although the test results are very practical, the schemes usually require a large number of samples and have a large lag in the quality evaluation. The simulation-based approach models a fault randomly and injects the fault into the various gates by a given rule at a certain abstract level, and then counts the number of failures of the circuit to obtain its reliability. The typical model is the Monte Carlo (MC) method, which can be applied to the various abstract levels of circuits to achieve high accuracy but has time-consuming. For large-scale integrated circuits, the MC method is more commonly used as a verification method. The analytical approach is commonly implemented at a specified abstract level to evaluate the reliability of circuits in a multi-fault environment. The typical methods include the probabilistic transfer matrix model (PTM)
https://doi.org/10.1016/j.neucom.2019.07.100 0925-2312/© 2019 Published by Elsevier B.V.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM 2
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
[4], probabilistic gate model (PGM) [5], Bayesian networks method [6], and stochastic calculation method [7]. They tend to achieve high accuracy but often have large time-space consumption, and require that the object to be analyzed is an existing entity. In summary, the approaches mentioned above are not applicable to the circuits at the conception stage. However, with the acceleration of new technology applications and product revisions, it is necessary to have new technologies to fast evaluate the reliability of circuits at the conception stage to boost the speed of circuit structure design. Noticing that the deep learning model is a method for discovering distributed representations of data by simulating the neural centers of the human brain, and combining the features at a low level to form more abstract representations at a high level [8]. The technology of deep learning can achieve object classification or result prediction by combining classifiers or predictors, and shows approximately constant time-space consumption this meets the new requirements put forward by the existing evaluation methods of circuit reliability. Therefore, this technology is chosen in this study to quickly evaluate the reliability of integrated circuits at the conception stage. According to the label attributes of the training data, the deep learning model can be classified into supervised learning and unsupervised learning. In general, the former is more accurate than the latter because its input data is tagged, while the latter does not have tagged data [9]. Considering that the F-PTM model [10] can provide the reliability of the applied circuits to label the input features of the circuits, a deep learning network with supervised learning is chosen in this study to implement the follow-up research. At present, the research on deep learning, such as convolution neural networks (CNNs) [11], the deep belief networks (DBNs) [12] and deep autoencoder networks (DANs) [13], mainly focuses on the fields of machine vision, speech recognition, natural language processing, and traffic prediction, etc. Some studies have pointed out that the CNN mainly focuses on unstructured data, while the DBN and DAN perform well for both the structured and the unstructured data, especially the structured data [14,15]. Further studies have found that a DBN is more suitable for processing continuous data, while a DAN performs better when dealing with discrete data [16]. Analysis found that the reliability of a circuit is mainly depended on its topological structure, the failure probability of the basic gates, and its application environment (see Section 4.1 for the details). Obviously, the dataset constructed by the feature set has structured and discrete characteristics. Therefore, this paper chooses the DAN model to conduct research on the reliability prediction for circuit structures. The gates in this paper are assumed to be subjected to von-Neumann faults which consists of transient bit-flip faults caused by signal noise, although other stack at fault models can also be used in this paper. However, the structure of the DAN model is influenced by many factors, such as the characteristics, scale, and quality of the dataset, which directly determine the prediction accuracy of the model. Therefore, it is necessary to preprocess the dataset according to the characteristics of the DAN model and the dataset, and to configure the structure reasonably to obtain acceptable predictions, which requires a wealth of expertise and superb skills. Therefore, there is still a lot of work to do in using the DAN model to predict circuit reliability and achieving acceptable accuracy. Firstly, we select the features associated with circuit reliability to determine the feature set, and we construct an efficient method for data collection for the feature set. Next, we preprocess the dataset to meet the requirements of the DAN model. Then, we train and test the constructed DAN model with the constructed training set and the testing set to find a reasonable structure for the DAN model that meets the accuracy requirements. Finally, we predict the reliability of the circuits according to their applied input features.
The rest of the paper is organized as follows. The preparatory work related to this study is introduced in Section 2, Section 3 describes the extraction of labeled data by the F-PTM model, followed by circuit feature analysis and extraction in Section 4. The prediction method for circuit reliability based on the DAN model is presented in Section 5. The experimental results are presented in Section 6, followed by the conclusions in Section 7. 2. Preparatory work To successfully conduct this study, the preparatory work involved is the description of the principles and construction method for the DAN model. Since the feature dataset in this paper has labeled attributes, and supervised learning is superior to unsupervised learning in terms of classification and prediction accuracy, we choose the DAN model with supervised learning to predict circuit reliability. The structure and the construction of the DAN model are presented in Figs. 1 and 2, respectively. It can be seen from Fig. 1 that the DAN is a neural network model with multiple hidden layers that consists of multiple autoencoders (AEs), and each AE is a learning network with only one input layer and one hidden layer, where the output of the AE at the bottom is used as the input to the AE at the top. Based on the representation in the previous layer, each layer in the DAN model learns multiple layer-based representations from the raw data, so as to learn more essential features [17]. It can be seen from Fig. 2 that the construction of the DAN model mainly includes the following parts. (a) Data collection and preprocessing. Firstly, we collect the feature data according to the characteristics of the DAN model. Next, we preprocess the feature dataset to meet the requirements of the DAN model. Then, we divide the dataset into training set and testing set by the previously determined proportions. (b) Construct a DAN-based prediction model. Firstly, we construct a feature learning network based on AEs. Next, we initialize the weight W and the bias vector b for the deep feedforward neural network by the feature learning network with layer-wise training. Then, we fine-tune the DAN model with supervised learning with the labeled training dataset. (c) Model testing and selection. We test and evaluate the DAN model with the test dataset and the error evaluation function, respectively, to determine whether the constructed model meets the accuracy requirements, so as to select an efficient prediction model. 2.1. AE The role of the AE is to recover the raw data from the abstracted data as much as possible, so as to extract the distributed feature representation from the raw input data. The AE is a special three-layer back propagation (BP) neural network consisting of an input layer, an encoder, and a decoder, as shown in Fig. 3, where V=(v1 ,v2 , .,vin ) is the feature dataset, vi (i = 1, 2, …, in) is the ith input feature dimension vector, and xi is the ith dimension input signal for the input layer. In the AE, w(1) and b(1) are the shared weight matrix and the bias vector between the input layer and the hidden layer, respectively, and w(2) and b(2) are the shared weight matrix and bias vector between the hidden layer and the output layer, respectively. The "+1" node is the bias node, and hw,b (x) is the output signal. To extract the essential features from the input data, the trained steps of the AE can be summarized simply, as follows. Firstly, we perform encoding transformation on the input signal x using Eqs. (1) and (2) to obtain another represent ation h of x. Next, we decode h using Eqs. (3) and (4) to obtain the output signal hw,b (x). Then, we update the shared weight matrix w and the bias vector
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
3
Fig. 1. The structure of the DAN model.
Fig. 2. Construct an AE-based layer-wise feature-learning network.
h = a (2 ) = f z (2 ) ,
(2)
z(3 ) = w(2 ) a(2 ) + b(2 ) ,
(3)
hw,b (x ) = a(3 ) − f z(3 ) , J AE(w,b) =
(4)
Lw,b a(3 ) , x ,
(5)
x∈V
f (z ) =
1 , 1 + exp (−z )
(6)
2.2. Small batch gradient descent method Fig. 3. Typical AE structure.
b in the AE using the cost function presented in Eq. (5) and the BP algorithm. We repeat the above steps until we find the w and b that minimize Eq. (6), (hw,b (x)≈x), where z(2) is the input to the hidden layer; a(2) is the encoding result in the AE (denoted as the output of hidden layer); a(3) is the input to the output layer, which is the decoding result in AE; f is the sigmoid function (nonlinear activation function) presented in Eq. (6); and L is the reconstruction error function, which is represented by the mean square error function in this study.
z(2) = w(1) x + b(1) ,
(1)
To optimize the parameter θ ={w,b}, the solution is usually achieved by the gradient descent method in error back propagation. However, for a large training dataset, the learning process is slow because the gradient vector J(θ ) is calculated once for each training instance. Therefore, the mini-batch gradient descent (MBGD) is introduced in this study. Unlike the gradient descent algorithm (where all the training data is used to update the parameters in each training), MBGD decomposes one training into T iterations, and each iteration is performed by randomly selecting N samples (mini batch) from the training dataset to calculate the gradient of the loss function, so as to update the parameters. To boost the convergence, the latest updated items are added to the
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM 4
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
Fig. 4. Training flowchart of the DAN model.
updating parameters in the momentum algorithm, so that the parameters can be updated in time, where the e-th training can be expressed by Eqs. (7)–(9). Jm (θ ) represents the gradient vector in the loss function, and is calculated by using the m-th batch of training data, m∈{1,2,…,T}; S is the number of samples in the m-th batch of the dataset; θ m (e) is the parameter calculated in the m-th iteration in the e-th training; Gm (e) is the update parameter calculated in the first m iterations in the e-th training, G0 (0) = 0; γ is the momentum super-parameter (its value is usually less than or equal to 0.9), and η is the learning rate.
θm+1 (e ) = θm (e ) − Gm (e ),
(7)
Gm (e ) = γ Gm−1 (e ) + η∇ Jm (θ (e ) ),
(8)
∇ Jm (θ ) =
S
ji (θ ),
(9)
i=1
2.3. Model training The training of the DAN model is described in Fig. 4, and its main steps are as follows: (a) Construct k AEs, and initialize their relevant parameters in a uniformly distributed manner within the interval β shown in Eq. (10), and initialize h0 = V. (b) Initialize the input signal of the i th AE (denoted as AE_i) with hi -1 (i∈{1, 2, ., k}), train AE_i and implement Eqs. (1)–(9). Then encode hi -1 with the trained AE_i to obtain the output signal hi to the hidden layer. (c) If i
the difference between the predicted value and the label value in each training.
β= −
6 , n( j ) + n( j+1)
JDAN (w, b) =
6 , n( j ) + n( j+1)
Lw,b yˆ, y ,
(10)
(11)
y∈Ym
3. F-PTM model This section introduces the F-PTM model for the collection of the labeled data corresponding to the feature sets. For any applied input vectors, the F-PTM model [10] can provide results to exactly reflect the reliability R of the corresponding circuit features. Therefore, the results calculated by the F-PTM model are used as the labels for the feature data. The calculation processes are presented in Algorithm 1, and a simple example is shown in Fig. 5. The list of completed nodes (denoted as LC) is the linked list where the input information for any nodes can be obtained from the output information of their predecessor nodes. According to the analysis of Ref. [10], the time-space consumption of the F-PTM model increases linearly with the increase of the number of circuit gates. 4. The feature set associated with circuit reliability The feature dataset is the raw data of the deep learning model, which has a critical impact on the final model. To obtain the dataset, we select the proper granularity to represent the circuit reliability as the first step, see Section 4.1 for details. Then we construct an efficient algorithm to extract the data corresponding to the selected feature set from the circuit net list, as described in Section 3.2. 4.1. Select the feature set associated with circuit reliability In reality, circuit reliability is affected by many factors, including circuit structure and process technology as well as the application environment, as shown in Fig. 6. Circuit structure mainly refers to the basic gates and their interconnection. The process technology is commonly measured by the functional and parametric faults of the devices, and the faults are usually quantified by the failure probability of the basic gates [18]. The application environment mainly refers to all possible circuit input signals, which are usually expressed by the input vectors [19].
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
5
Algorithm 1 Circuit reliability calculation for the applied input vectors. Input: Circuit net list Output: Circuit reliability 1. Parse circuit net list and initialize relevant parameters. 1.1) Read the net list, extract the information of the basic gates, build a LC for the circuit, and identify all the primary inputs and outputs of the circuit; then perform i = 1; 1.2) With the type, the failure probability p, and the input number ns of the basic gate, build its probability transfer matrix PMtype-ns and its ideal transfer matrix IMtype-ns ; 1.3) Extract the input signals of the circuit primary inputs, and construct the corresponding primary input probability distribution pipt, where pipt = [1, 0] when the input signal is 0; otherwise, pipt = [0, 1]. 2. Calculate the fault output probability distribution fopti and the ideal output probability distribution iopti for the i th node in LC. 2.1) Extract the type, the fault probability pi and the input number mi for the i th node in LC, and generate its probability transfer matrix PMtype-nsi and its ideal transfer matrix IMtype-nsi , then perform j = 1; 2.2) Extract the j-th input of gi . If it is the primary input of the circuit, assign pipt to the fault input probability distribution fiptij and the ideal input probability distribution iiptij ; otherwise, extract fiptij and iiptij from the fault output probability distribution foptq and the ideal output probability distribution ioptq of the q-th node (1 ≤ q < i) in LC; 2.3) If j = m, then go to step 2.4); else perform j = j + 1, and go to step 2.2); 2.4) Calculate fipti and iipti by fipti =fipti 1 fipti 2 … fiptimi , iipti =iipti 1 iipti 2 … iiptimi , where is the operation of tensor product; 2.5) Calculate fopti and iopti by fopti =fipti × PMtype-nsi iopti =iipti × IMtype-nsi ; 2.6) If the output of gi is the primary output of the circuit, then go to step 2.7); otherwise, go to step 2.8); 2.7) Calculate the circuit reliability r for the applied input vector by r=sum(fopti . × iopti ), where sum represents the summation operation; 2.8) If reach the end of LC, then go to step 3; otherwise, perform i = i + 1 and go to step 2.1). 3. Output r.
Fig. 5. A simple example for Algorithm 1 (p = 0.05).
Fig. 6. Feature set associated with circuit reliability.
4.1.1. Feature set associated with circuit structure Some studies [20] have found that the constituent units of the circuits, such as primary input number (PIs), circuit tier number
(Ts), circuit gate number (Gs), and the numbers of basic gates with different types, including the AND gate number (As), NOT gate number (Ns), OR gate number (Os), NAND gate number (NAs), and NOR gate number (NOs), are all the important members of the circuit structure. Further analysis finds that other than the above members, the XOR gate number (XOs), XNOR gate number (XNOs) and buff number (Bs) are also the important members of the circuit structure. In addition, the wires among the basic gates are also the important members. Analysis has found that the circuits with the same members mentioned above tend to have different reliabilities due to their different connections, such as their fan-outs [18,21]. Therefore, it is necessary to append the fan-out attribute as a structure member, which is quantified by the fan-out source number (FOs) and the fan-out number (Fs). Fig. 7 is the feature set associated with circuit structure in this study.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM 6
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
Fig. 7. Feature set associated with circuit structure.
4.1.2. Application environment As mentioned above, the application environment is usually measured by circuit input vectors, which are encoded with a binary string. The length of the binary string is equal to the number of primary inputs, 0 and 1 indicate the low-level input and the high-level input, respectively. According to the analysis presented in Section 4.1, we know that the input vectors represented by binary strings with unequal lengths are different from the attribute of other circuit features, such as As, Os and Ns. To unify their attributes, the input vectors are transferred to decimal representations for characterization in this study. However, there is no appropriate data type to represent the decimal data converted from an overly long binary string. To solve this problem, a partitioning strategy is given, the details are shown as follows. Firstly, we determine the dividing width according to the size of the applied data type. Secondly, we divide the applied input vector represented by an overly long binary string into several segments. Thirdly, each segment is mapped into a decimal representation using Eq. (12). Finally, we merge the results obtained in step 3 to construct a new input feature A using Eq. (13). Although this strategy will destroy the integrity of the input vectors, as long as the number of segments is small enough, the destructiveness can be reduced to be small enough, and the complexity of the DAN model can also be reduced. A relevant experiment is shown in Section 6.
g(bi ) =
k
ai j × 2 j ,
(12)
j=0
A = (g(b1 ), g(b2 ), · · ·, g(bn )),
(13)
Where n is the number of segments obtained by dividing a binary string, n = ceil(nb/blen) and ceil() is a ceiling function; nb and blen are the lengths of the binary string and a segment, respectively. bi is the ith segment, and bi = aik …ai 1 ai 0, i∈{1, 2, …, n}, k <= blen, aij ∈ {0, 1}, j∈{0, 1, …, k}; g(bi ) is a function that is used to map bi into a decimal numeral. Further analysis found that the DAN model requires the input feature with a deterministic dimension. However, the lengths of the input vectors for different circuits tend to be different, so that the lengths of corresponding features obtained by Eq. (13) are often different, which cannot meet the requirement of the DAN model for the input feature set. To deal with this problem, a unitive feature  for A is proposed in Eq. (14) to concretize the size of
the input feature set (Max_n).
⎧ (g(b1 ), g(b2 ), . . . , g(bn )), ⎞ n = Max_n ⎪ ⎨⎛ Max_n−n Aˆ = T (A ) = ⎝ , g(b1 ), g(b2 ), . . . , g(bn ), −1, . . . , −1⎠, n < Max_n ⎪ ⎩ (14) Where ‘−1’ means that the corresponding bits do not exist. For example, assuming that the input vectors for circuitscc1 and cc2 are ‘1010 0 010’ and ‘001101001100’, respectively, and blen = 3. According to Eqs. (12)–(14), Max_n is assigned with 4, and Âs for circuitscc1 and cc2 are expressed as (5, 0, 2, −1) and (1, 5, 1, 4), respectively. 4.1.3. Fault probability of basic gates With the introduction of new materials and new technologies as well as increased circuit integration, the effect of process technology on circuit reliability has become more significant [1]. The fault probability of basic gates PF can be measured by parameters, such as activation factor, electric field strength and defect size, which reflects the effect of the above-mentioned factors on circuit reliability. Therefore, PF is used to measure the level of process technology in this paper. For current technologies, Ref. [22] and other experiments have pointed out that PF is very small and can be set to [0, 0.1]. For convenience, PF is set as follows: PF = {10d | d = −1, −2,…, −10}. In summary, so far the feature set for circuit reliability is determined and expressed as follows: V = [PIs, Ts, Gs, As, NAs, Os, NOs, Ns, XOs, XNOs, Bs, FOs, Fs, Â, PF], where the size of V is Max_n + 14. The labeled data (circuit reliability R) corresponding to V can be accurately calculated by the F-PTM method. 4.2. Data collection There are two steps taken to create a dataset for circuit reliability. Firstly, we produce circuit sub-net list, due to that the feature set presented in Section 3.1 is not enough to identify a circuit with multiple primary outputs. Dividing the circuit so that each subcircuit contains only one primary output is an effective solution, and this process is presented in Algorithm 2. Secondly, we build the dataset corresponding to Vby combining  with PF, and calculate R corresponding to V by the F-PTM method. This process is presented in Algorithm 3. Since Algorithms 2 and 3 are only used to create the dataset and are not involved in the prediction of the DAN model, by convention, their time-space consumptions are not analyzed in this study. Through Algorithm 2, a circuit sub-netlist with only one primary output can be produced. Then, according to the requirements
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
7
Algorithm 2 Circuit sub-netlist produce algorithm. Input: circuit netlist (circuit.isc) Output: Circuit sub-netlist 1. Parse circuit and initialize relevant parameters. 1.1) Perform i = 1, j = 1; 1.2) Traverse netlist, if reach its end, then go to step 1.5); else extract its i th record; 1.3) If the input for the record is labeled, then put it into the end of LC, or if it is the primary output of the circuit, then put it into the list of output nodes (denoted as LO) together with its position in the LC; otherwise, put it into the list of incompleted nodes (denoted as LI); 1.4) Perform i = i + 1 and go to step 1.2); 1.5) Traverse LI. If all nodes are labeled, go to step 2; otherwise, extract the j-th record from LI and its input from LC; 1.6) If the record is labeled, then put it into the end of LC and label its input. If it is the primary output of the circuit, put it into LO together with its position in LC; 1.7) Perform j = j + 1 and go to step 1.5). 2. Produce circuit sub-netlist. 2.1) Perform i = 1 and j = 1; 2.2) Traverse LO, if reach its end, go to step 3; 2.3) Extract the i th node gi from LO and label gi with ni; 2.4) Traverse LC from gi up to circuit primary inputs with the width-first recessive search algorithm and label the corresponding nodes on the sensitized path with ni; 2.5) Create an empty sub-netlist file named circuit_ni.isc, extract the nodes labeled with ni in LC and write them into the end of the circuit_ni.isc; 2.6) Perform i = i + 1 and go to step 2.2). 3. End of the algorithm.
Algorithm 3 Reliability-oriented data collection method. Input: Circuit sub-netlist Output: Dataset [V, R] 1. Parse sub-netlist and initialize the relevant parameters. 1.1) Perform step 1 in Algorithm 2; 1.2) Layer circuit and identify its layer number. 2. Collect the features for circuit topology structure and build the corresponding dataset TCV. 3. Createthe feature dataset V and its corresponding labeldata R. 3.1) Initialize the vector of fault probability ofgates VPF and the number of circuit input vectors num, initialize c = 0 and j = 1; 3.2) Traverse VPF, if reach its end, then go to step 3.8); else extract the j-th data pfj from VPF; 3.3) If c
of the DAN model and using the obtained sub-netlists, the feature dataset corresponding to V and R can be created by Algorithm 3. 5. Experimental setup According to the analysis mentioned-above, we know that the following three works need to be performed to obtain a trained DAN model for the prediction of circuit reliability. Firstly, the data collected by Algorithm 3 needs to be preprocessed to relieve its size effect caused by the inconsistencies in data gradients, the details are shown in Section 5.1. Secondly, the error evaluation function should be given to determine whether the training of the DAN model meets the stop criteria, the function is presented in Section 5.2. Thirdly, A grid-based search method is proposed to find a reasonable network structure for the DAN model, the details are shown in Section 5.3.
the data with a large gradient, the adverse effect of the size effect on the dataset cannot be treated well only by the normalization. Other methods are required to preprocess the dataset to reduce its gradient, and the method adopted is determined by the distribution of the dimension data itself. For example, the logarithm function presented in Eq. (16) can be used to preprocess the dimension data that follows an exponential distribution, where y is the converted value of x; xmax and xmin are the maximum and the minimum values of sample x, respectively; ymax and ymin are the maximum and the minimum values of y in the mapping interval, respectively.
y = (y max −y min ) ×
x − x min + y min, x max −x min
y = log10 x,
(15) (16)
5.2. Error evaluation function 5.1. Data preprocessing Our analysis found that the data corresponding to each feature in V tends to have different dimensions. To relieve their size effect, the raw data often needs to be standardized or normalized before being input into the DAN model. Noting that the dataset corresponding to V is discrete and its value range is fixed, so the linear transformation function (also called deviation standardization) presented in Eq. (15) is used to normalize the dataset in this study, which maps the dataset to [0,1] or [−1,1] to speed up the gradient descent and accelerate the convergence of the model. However, for
In general, the error evaluation functions are used to evaluate the prediction accuracy of the constructed model to judge whether the model meets the requirements, such as root mean squared error (RMSE) [23] and mean absolute percentage error (MAPE) [23], which are shown in Eqs. (17) and (18). Note that RMSE can be well used to evaluate the prediction accuracy of the DAN model, but cannot measure the extent to which it is unbiased, while MAPE can solve the problem of unbiased testing of the prediction model well. Therefore, they are both used to evaluate the prediction error of the DAN model constructed in this study, where yi is obtained by
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM 8
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
Algorithm 4 Selection method for the structure of the DAN model. Input: Training dataset Output: The optimal DAN model 1. Initialize the relevant parameters, including the maximum number of hidden layers ly, the maximum number of hidden layer nodes nd, the maximum training number nt, the increment sl, the constant ln and the counter l = 1. 2. If l ≤ly, then initialize i = 1 and go to step 3; else go to step 10. 3. If 2i+ln ≤ nd, then go to step 4; else go to step 9. 4. Initialize the number of hidden layer nodes hnd=2i + ln and the training number t = ne. 5. Construct and train AE with layer-wise learning. 5.1) Initialize j = 1; 5.2) If t ≤ nt, then initialize label-free training data h0 ; else go to step 8; 5.3) If j ≤ l, then construct the j-th AE with hnd hidden layer nodes (denoted as AE_l_i_t_j); else go to step 6; 5.4) Perform t trainings on AE_l_i_t_j by data hj -1 ; 5.5) Encode hj -1 with the encoder of AE_l_i_t_j, and record the result as hj ; 5.6) Perform j = j + 1, then go to step 5.3). 6. Construct and train the DAN model. 6.1) Construct a deep neural network DAN_l_i_t with l hidden layers and hnd nodes in each hidden layer; 6.2) Initialize j = 1; 6.3) If j ≤ l, assign the shared weight matrixes and offset vectors in the input layer and the hidden layer of AE_l_i_t_j to the shared weight matrixes and bias vectors in the j-th layer and the (j + 1)-th layer of DAN_l_i_t, respectively; else go to step 6.5); 6.4) Perform j = j + 1, then go to step 6.3); 6.5) Perform t trainings on DAN_l_i_t in supervised manner using the labeled training dataset. 7. Perform t = t + sl, then go to step 5. 8. Perform i = i + 1, then go to step 3. 9. Perform l = l + 1, then go to step 2. 10. Test the constructed DAN model using the test dataset and calculate its corresponding RMSE and MAPE. 11. Extract the DAN model with the smallest error. 12. End the algorithm.
the method presented in Algorithm 1, yˆi is the predicted value of yi by the DAN model, and N is the sample size of the test dataset.
RMSE =
N yi − yˆi2 i=1
N
,
(17)
N 1 yi − yˆi MAPE = yi , N
(18)
i=1
5.3. The structure of the DAN model As mentioned above, the structure of the DAN model mainly contains three parameters which are the number of input layer nodes, the number of hidden layers and the number of hidden layer nodes. Different combinations of the three parameters in the DAN model tend to result in different prediction accuracies for the same feature dataset [24]. Noting that the value ranges of the parameters are determined by the size of the feature set which has been presented in Section 4.1, thus we can find an optimal network structure for the DAN model in the given ranges using the search algorithms. The details are discussed as follows. Except for the offset nodes, each node in the input layer in the DAN model only receives one dimension data and each dimension data corresponds only to one node in the input layer [23]. Therefore, the number of the nodes in the input layer for the DAN model is assigned to Max_n + 15 when the size of the feature set is Max_n + 14. At present, except for the search algorithms, there is no effective way to determine the optimal numbers of hidden layers and hidden layer nodes of the model for a given feature dataset to meet the prediction accuracy of the model. However, the solution domain consisting of all possible combinations of the number of hidden layers and the number of hidden layer nodes tends to have a very large scale, thus resulting in time-consuming to find the optimal solution by the exhaustive search algorithms. Thus, it is necessary to improve the search algorithm, so as to speed up the solution. In view of the lack of mathematical foundation to support the accurate configuration of the number of hidden layers and
the number of hidden layer nodes in the deep learning networks [25], according to the characteristics of the provided dataset and referring to the approaches presented in Ref. [24,25], a grid-based method is proposed to configure the structure of the DAN model in the sufficiently large domains. Although the proposed method may not find the best configuration, it can provide a valid configuration for the DAN model, because the range of the number of hidden layers is large enough to effectively support the simple value-taking method of the number of hidden layer nodes [24]. The details of the construction of the DAN model are as follows. Firstly, we initialize the number of hidden layers l(∈{1, 2, …, ly}), and the number of hidden layer nodes 2i (i∈{1, 2, …, md}) as well as the maximum number of training epochs nt. Then, we perform Algorithm 4 on the above-mentioned value ranges based on the training method presented in Section 2.1 to find the optimal numbers of hidden layers and hidden layer nodes, as well as the optimal training number. The flowchart is presented in Fig. 8. Similarly, the time-space complexity of Algorithm 4 is not analyzed in this paper, due to the fact that Algorithm 4 is not involved in the prediction of the DAN model.
6. Simulation results To verify the effectiveness of the model proposed in this paper, some simulations are performed on the 74-series and ISCAS 85 benchmark circuits [5,7,10] using a personal computer with a 1.9 GHz processor and 4 GB RAM. Firstly, we collect data using Algorithm 3 and preprocess the data with the method presented in Section 5.1 to relieve the size effect. Secondly, according to the characteristics of the applied dataset, we construct an appropriate structure for the DAN model using the method presented in Algorithm 4. The results are shown in Tables 2 and 3. Thirdly, we perform experiments using the MC method, the F-PTM model presented in Algorithm 1 and the method presented in Ref. [26] to test the effectiveness of the DAN-based prediction model for circuit reliability. The results are shown in Tables 4 and 5. In addition, the effects of some main parameters on the prediction accuracy of the constructed model are also analyzed. The results are shown in Figs. 13 and 14.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
9
Fig. 8. Selection scheme for the structure of the DAN model. Table 1 Preprocessing methods for the raw feature dataset. Input /Output
Parameters
Preprocessing
PIs Ts Gs [As, Nas, Os, Nos, Ns, Xos, XNOs, Bs] FOs Fs PF Â R
primary input number layer number gate number The numbers of basic gates of different types fanout source number fanout number fault probability of basic gate input vector circuit reliability
map map map map map map map map map
6.1. Data collection and preprocessing Firstly, we parse 20 circuits selected from the 74-series and ISCAS85 benchmark circuits [5,7,10] using Algorithm 2 to generate 508 circuit sub-netlists, where each sub-netlist contains only one primary output, PIs∈[1, 194], Gs∈[1, 2327], Ts∈[2, 174]. Secondly, we extract 60,800 records corresponding to
to to to to to to to to to
[−1, 1] by Eq. (15) [−1, 1] by Eq. (15) [–1, 1] by Eq. (15) [−1,1] by Eq. (15) [−1,1] by Eq. (15) [−1,1] by Eq. (15) [–1, 1] by Eqs. (15) and (16) [–1, 1] by Eq. (15) [–1, 1] by Eq. (15)
characteristics of the dataset, we determine its structure using the experimental method presented in Algorithm 4 in this paper. The steps are as follows. Firstly, we extract 50,800 training data and 10,0 0 0 test data from the 60,800 data given in Section 5.1 using insight from expertise research and the method presented in Ref. [10], then we initialize ly = 7, ln = 3, nd = 6, nt = 10 0 0, and sl = 100. Secondly, we determine the appropriate numbers of hidden layers and hidden layer nodes ofthe DAN model using Algorithm 4, the result is shown in Fig. 9. Finally, according to the results shown in Fig. 9, we extract the optimal numbers of the weights and iterations, as well as the training time for the model with the smallest RMSE when the numbers of hidden layers and hidden layer nodes are different. The results are shown in Tables 2 and 3. The main parameters in the model are the learning rate η = 0.05, momentum super parameter γ = 0.5, batch size = 100; NumNodes-Hlayer is the number of hidden layer nodes, and Num-Hlayer is the number of hidden layers. Fig. 9 shows that the DAN model has the smallest RMSE when the number of hidden layer nodes is 128 for all hidden layers for
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM 10
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
Fig. 9. Relationship between the number of hidden layer nodes and RMSE in different numbers of hidden layers.
Table 2 Test error and training time for the DAN model when NumNodes-Hlayer is 128. Num-Hlayer
MAPE
RMSE
Epochs
weight number
Time (s)
1 2 3 4 5 6 7
0.0300 0.0289 0.0283 0.0283 0.0278 0.0279 0.0280
0.0509 0.0496 0.0490 0.0488 0.0482 0.0484 0.0485
1000 1000 900 900 1000 1000 1000
4480 20,864 37,248 53,632 70,016 86,400 102,784
744 1886 3045 4436 5505 6391 7720
the dataset provided in this paper. When NumNodes-Hlayer is 128, the RMSE firstly decreases with the increase of Num-Hlayer, then increases with the increase of Num-Hlayer when Num-Hlayer is greater than 5. In short, the DAN model has the smallest RMSE when Num-Hlayer and NumNodes-Hlayer are 5 and 128, respectively. This means that the DAN model with 5 hidden layers and 128 hidden layer nodes can best explain the dataset in this study. MAPE is not described in detail in this paper because MAPE and RMSE are similar. Tables 2 and 3 show that the epochs reach or approach to 10 0 0 when the RMSE is smallest for the different Num-Hlayer and NumNodes-Hlayer. In addition, the training time and complexity (denoted as weight number) both increase with the increase of Num-Hlayer and NumNodes-Hlayer in the DAN model, and the model has the smallest RMSE and MAPE when Num-Hlayer and NumNodes-Hlayer are 5 and 128, respectively. Therefore, from this point forward, Num-Hlayer, NumNodes-Hlayer, and epochs are set to 5, 128, and 10 0 0 to perform the experiments, respectively. 6.3. Accuracy To further verify the effectiveness of the proposed model, the following experiments are performed on the 74-series and ISCAS85
benchmark circuits. Firstly, we compare the results obtained by the F-PTM model and the DAN model presented in Section 6.2 on 100 samples extracted from the test set. The results are shown in Fig. 10. We further compare them with the results obtained by the MC method and the method presented in Ref. [26] to test the prediction accuracy of the constructed DAN model. The results are shown in Table 4. Secondly, we compare the time-space consumptions obtained by the four methods mentioned above. The results are shown in Table 5. Finally, we analyzed the effect of some main features (such as Ts and Gs) on RMSE and MAPE for the dataset with an imbalanced distribution. The results are shown in Figs. 11 and 12. For ease of calculation, and without loss of generality, all circuit primary inputs are valued at 0 and 1, respectively, which are expressed as inputs = 0 and inputs = 1, respectively; pf is assigned with 0.01 which is extracted from [10−1 , 10−10 ] in a random way, and the number of experimental samples for the MC method is 10 0,0 0 0 [10]; the result for each primary output for the MC, FPTM and Ref. [26] methods is extracted from the calculation results for the entire circuit unless otherwise stated. Notice that although the MC method has some accuracy loss due to the fault injection adopted in a pseudo-random way, it can be used to evaluate the reliability of circuits at different abstract levels, and its results are widely recognized by academia and industry [27]. Therefore, its results are also used as reference values in this paper. Fig. 10 shows that the results obtained by the F-PTM model are quite close to that by the DAN model on the test data selected at random. The results in Tables 4 and 5 show that the accuracy of the proposed method is lower than, but quite close to, that of the MC method and the F-PTM method. Its accuracy is higher than, but close to, that of Ref. [26] method. The results can be proven by its RMSEs of 5.026% and 4.986%. However, the predicted values of this method still incur small error on individual circuits. For example, the maximum relative error is 17.4046% for C1908-2888. In
Table 3 Test error and training time for the DAN model when Num-Hlayer is 5. Num Nodes-Hlayer
MAPE
RMSE
Epochs
Weight number
Time (s)
16 32 64 128 256 512
0.0309 0.0291 0.0283 0.0278 0.0283 0.0284
0.0501 0.0493 0.0489 0.0482 0.0493 0.0491
1000 1000 1000 1000 900 1000
1584 5216 18,624 70,016 271,104 1,066,496
1152 2018 2933 5505 13,116 54,075
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
ARTICLE IN PRESS
JID: NEUCOM
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
11
Fig. 10. Distribution of the predicted values obtained by the DAN model for data extracted from the test dataset. Table 4 Comparison of the results obtained by the different methods (pf = 0.01). Circuits
74148-78 74182-63 74283-97 741811-201 c499-753 c1355-1324 c1908-2888 c3540-1713 c6288-1581 c7552-10839 MAPE(%) RMSE(%)
MC
F-PTM
Ref. [26] method
The proposed method
Inputs = 0
Inputs = 1
Inputs = 0
Inputs = 1
Inputs = 0
Inputs = 1
Inputs = 0
Inputs = 1
0.9713 0.9513 0.9892 0.6155 0.9779 0.9546 0.7233 0.9330 0.9051 0.8546 – –
0.8974 0.9899 0.9418 0.5659 0.9703 0.953 0.6889 0.9795 0.9259 0.8563 – –
0.9716 0.9514 0.9900 0.6157 0.9788 0.9546 0.7239 0.9334 0.9060 0.8553 0.554 0.058
0.8988 0.9900 0.9419 0.5659 0.9712 0.9529 0.6890 0.9801 0.9268 0.8575 0.593 0.074
0.97 0.95 0.99 0.71 0.98 0.95 0.57 0.93 0.89 0.75 5.722 6.606
0.89 0.99 0.94 0.93 0.97 0.94 0.68 0.98 0.92 0.80 5.064 11.67
0.9613 0.9699 0.9622 0.6670 0.9569 0.8840 0.6744 0.9738 0.9066 0.9639 4.771 5.026
0.9554 0.9660 0.9378 0.6182 0.9283 0.9440 0.8088 0.9734 0.9056 0.8951 4.857 4.986
Table 5 Comparison of the time-space consumptions obtained by the different methods (pf = 0.01, PIi = 1). Circuits
74148_78 74182_63 74283_97 741811_201 c499_753 c1355_1324 c1908_2888 c3540_1713 c6288_1581 c7552_10,839 AVG
MC
Ref. [26] method
This proposed method
Time (s)
Memory (MB)
Time (s)
F-PTM Memory (MB)
Time (s)
Memory (MB)
Time (s)
Memory (MB)
5472 3420 13,428 15,766 119,490 888,717 1503,187 2754,009 4656,292 6372,099 1633,188
0.1030 0.0893 0.1321 0.2580 0.6344 1.7260 2.4137 4.4985 8.0071 9.6042 2.75
0.108 0.102 0.164 0.3318 2.329 18.550 37.284 113.241 377.587 149.011 69.87
0.1040 0.0913 0.1342 0.2640 0.6429 1.750 2.448 4.562 8.120 9.739 2.79
0.131 0.095 0.147 0.762 2.233 15.203 30.096 105.318 375.506 465.511 99.50
0.1030 0.0893 0.1320 0.2434 0.6343 1.726 2.298 4.498 8.007 9.604 2.73
0.01593 0.01587 0.01613 0.01598 0.01573 0.01618 0.01585 0.01596 0.01584 0.01596 0.01594
2.5151 2.5151 2.5151 2.5151 2.5151 2.5151 2.5151 2.5151 2.5151 2.5151 2.5151
addition, the accuracy is often different for the same circuit with different inputs, such as for C499-753. The time and space consumptions of this method are approximately constant, and are basically unaffected by the change in circuit scale, while that of the MC method, the F-PTM model and Ref. [26] method are affected by the circuit size, and increase with the increase in circuit scale, especially for the MC method. The proposed method, on average, is approximately 102,458,469 times faster than the MC method, and approximately 4383 times faster than the F-PTM model, and approximately 6242 times faster than Ref. [26] method.
Moreover, comparing with no-mapping operations, the experiments have shown that the mapping operations presented in Table 1 can not only boost the convergence speed, but also can improve the predicted accuracy, because the size effect existing in the dataset is relieved by the mapping operations. For example, the MAPE and the RMSE are 11.51% and 12.84%, respectively, when nomapping operations are performed. This is mainly because the MC method, the F-PTM model and Ref. [26] method calculate circuit reliability by the strategy of structure simulation, while the proposed method predicts circuit
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM 12
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
Fig. 11. Effect of Gs on RMSE and MAPE in imbalanced distribution.
Fig. 12. Effect of Ts on RMSE and MAPE in imbalanced distribution.
reliability by the method of feature learning. Ref. [26] method causes a loss of accuracy on some circuits due to the repeated consideration on some gates when using some applied input vectors. With this method, it is often difficult to achieve full learning for small-sized features, which lead to the emergence of outliers, resulting in some loss in prediction accuracy. This is a major problem to be solved in the future. The computational consumption of the proposed method is basically unaffected by circuit inputs, due to the fact that it is the equivalent of a mathematical analytical expression. The computational consumptions of the MC method, the F-PTM model and Ref. [26] method increase with the increase in circuit size, especially for the MC method, because they implement their evaluation by the simulation of circuit structure. The MC model requires large-scale simulations to ensure its accuracy due to the adopted fault injection with a pseudo-random way. Figs. 11 and 12 show that the effect of the dataset with imbalanced distributions on RMSE and MAPE tends to be different, and the effect of the dataset corresponding to the same feature with imbalanced distribution on RMSE and MAPE also tends to be different. The feature samples with a high ratio in the dataset tend to have small RMSE and MAPE. This is mainly because the DAN model extracts more information from the feature samples with a higher ratio in the dataset, resulting in better performance in processing those feature samples. This is to say that the problem for the samples with an imbalanced distribution must be addressed to further improve the prediction accuracy of the DAN model, which is beyond the scope of this paper. We will analyze it in future.
6.4. Relevantfactors As mentioned in Sections 4 and 5, the completeness of the feature set and the size of the feature set associated with the input vector are all important factors that influence the prediction accuracy of the DAN model. The experimental results are shown in Figs. 13 and 14. Fig. 13 shows the increment changes in RMSE and MAPE for each hidden layer in the DAN model before and after removing the 5 features of XOs, XNOs, Bs, FOs, and Fs. These changes are equal to the differences between the indexes after the feature removal and the indexes before the feature removal. Fig. 14 shows the increment changes in RMSE and MAPE for each hidden layer in the DAN model when the division widths of the input vector are 1, 5, and 10, respectively. Scheme A, Scheme B, and Scheme C are the differences among the indexes corresponding to the division widths of 5 and 10, 1 and 5, 1 and 10, respectively. Figs. 13 and 14 show that both RMSE and MAPE for each hidden layer in the DAN model undergo a certain increase when the features of XOs, XNOs, Bs, Fos, and Fs are removed, or the division width of the input vector is reduced. That is to say, the mentioned-above operations result in a loss of accuracy in the model. Therefore, a more complete feature set and a larger division width are needed to improve the accuracy of the DAN model constructed in this study. In addition, other factors, such as learning rate and momentum factor, also tend to cause a loss of accuracy in the DAN model. This is the research that we will carry out in the future.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
13
Fig. 13. Changes in RMSE and MAPE for the feature set of size 34 to 29.
Fig. 14. Changes in RMSE and MAPE for Scheme A, B and C, corresponding to  equal to 194, 39 and 20, respectively.
7. Conclusion In view of the computational complexity and hysteresis encountered in simulation-based methods for circuit reliability assessment, this paper proposes a new prediction method for circuit reliability based on a DAN model. Firstly, according to the analysis of the factors associated with circuit reliability, a feature set to characterize circuit reliability is determined. Secondly, on the basis of the characteristics of both the circuit netlists and the DAN model, an effective collection and preprocessing method for the dataset corresponding to the feature set is proposed to meet the requirements of the DAN model. Thirdly, according to the scale and characteristics of the dataset, a construction method for the DAN model based on grid search technology is proposed to determine its reasonable structure. Simulation results on benchmark circuits show that the trained DAN model could be used to achieve a fast assessment for circuit reliability at the conception stage. The accuracy of this model closely approached to that of the simulation-based methods, and its time-space consumption is basically unaffected by circuit size and is approximately constant. However, some issues uncounted in the experiments need to be solved. For example, both the imbalanced distribution of the feature dataset and the representation of the features associated with the input vectors tend to damage the accuracy of the DAN model. We leave these problems for future research.
Declaration of Competing Interest None. Acknowledgement Financial support for the study was provided by the National Natural Science Foundation of China (Nos. 61502422, 61772199, 61802123, 61503338 and 61432017), the Natural Science Foundation of Zhejiang Province (Nos. LY18F020028 and LY18F030023). References [1] A.S. Oates, K.P. Cheung, Reliability of nanoelectronic devices, Nanoelectron. Mater. Devices Appl. 1 (4) (Apr. 2017) 317–330. [2] Y. Chien-Chih, Probabilistic analysis for modeling and simulating digital circuits, Department of Electrical Engineering, The University of Michigan, Michigan, USA, 2012. [3] S. Krishnaswamy, I.L. Markov, J.P. Hayes, Design, Analysis and Test of Logic Circuits Under Uncertainty, Springer Science & Business Media, 2012. [4] S. Krishnaswamy, G.F. Viamontes, I.L. Markov, et al., Probabilistic transfer matrices in symbolic reliability analysis of logic circuits, ACM Trans. Des. Autom. Electron. Syst. 13 (1) (Jan. 2008) 1–8. [5] J. Han, H. Chen, E. Boykin, et al., Reliability evaluation of logic circuits using probabilistic gate models, Microelectron. Reliab. 51 (2) (Feb. 2011) 468–476. [6] T. Rejimon, K. Lingasubramanian, S. Bhanja, Probabilistic error modeling for nano-domain logic circuits, IEEE Trans. Very Large Scale Integr. VLSI Syst. 17 (1) (Jan. 2009) 55–65.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM 14
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx
[7] J. Han, C. Hao, J.H. Liang, et al., A stochastic computational approach for accurate and efficient reliability evaluation, IEEE Trans. Comput. 63 (6) (Jun. 2014) 1336–1350. [8] Y. Lecun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7) (May 2015) 436–444. [9] J. Schmidhuber, Deep learning in neural networks, Neural Netw 61 (1) (Jan. 2015) 85–117. [10] J. Xiao, J. Lou, J. Jiang, A fast and effective sensitivity calculation method for circuit input vectors, IEEE Trans. Reliab. (2019), doi:10.1109/TR.2019.2897455. [11] G. Trigeorgis, F. Ringeval, R. Brueckner, et al., Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network, in: Proceedings of the IEEE International Conference on Acoustic Speech and Signal Processing, May 2016, pp. 5200–5204. [12] G.E. Hinton, S. Osindero, W. Teh, A fast learning algorithm for deep belief nets, Neural Comput. 18 (7) (Jul. 2006) 1527–1554. [13] G.E. Hinton, R.R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science 313 (28) (Jul. 2006) 504–507. [14] G.E. Hinton, Learning multiple layers of representation, Trends Cogn. Sci. 11 (10) (Oct. 2007) 428–434. [15] S. Haykin, B. Kosko, Gradient based learning applied to document recognition, Proc. IEEE 86 (11) (Nov. 1998) 306–351. [16] J. Gehring, Y. Miao, F. Metze, et al., Extracting deep bottleneck features using stacked auto-encoders, in: Proceedings of the IEEE International Conference on Acoustic Speech and Signal Processing, Oct. 2013, pp. 3377–3381. [17] Y. Bengio, Learning deep architectures for ai, Found. Trends Mach. Learn. 2 (1) (Jan. 2009) 1–27. [18] J. Xiao, W. Lee, J.H. Jiang, et al., Circuit reliability estimation based on an iterative PTM model with hybrid coding, Microelectron. J. 52 (4) (Jun. 2016) 117–123. [19] J. Xiao, W. Lee, J.H. Jiang, et al., Sensitivity evaluation of input vectors with masking effects in digital circuits, Chin. J. Comput. 41 (10) (Oct. 2018) 2282–2294. [20] J. Xiao, J.H. Jiang, J.R. Liang, Transistor-level oriented calculation of reliability for generalized gates based on ptm, Sci. China Inf. Sci. 44 (10) (Oct. 2014) 1226–1238. [21] J. Xiao, J.H. Jiang, X.X. Li, et al., A novel trust evaluation method for logic circuits in IoT applications based on the E-PTM model, IEEE Access. 6 (7) (Jul. 2018) 35683–35696. [22] Y. Du, S. Chen, A novel layout-based single event transient injection approach to evaluate the soft error rate of large combinational circuits in complimentary metal-oxide-semiconductor bulk technology, IEEE Trans. Reliab. 65 (1) (Mar. 2016) 248–255. [23] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, Posts & Telecom Press, 2017. [24] H. Wenhao, S. Guojie, H. Haikun, et al., Deep architecture for traffic flow prediction:deep belief networks with multitask learning, IEEE Trans. Intell. Transp. Syst. 15 (5) (Oct. 2014) 2191–2201. [25] Y. Bengio, Deep Learning, MIT Press, 2016. [26] W. Ibrahim, M. Shousha, J.W. Chinneck, Accurate and efficient estimation of logic circuits reliability bounds, IEEE Trans. Comput. 64 (5) (May 2015) 1217–1229. [27] R.Y. Rubinstein, D.P. Kroese, Simulation and the Monte Carlo Method, Wiley, 2016. Jie Xiao received his Ph.D. degree in computer system architecture from Tongji University, Shanghai, China, in 2013. He is currently working with the department of computer science and technology, Zhejiang University of Technology, Hangzhou, China. His-current research interests include reliability evaluation and fault-tolerant design, deep learning and combinatorial optimizationcomputation. He also serves as a consultant and technical adviser for a research institute in electronic information fields. He is a member of CCF.
Weifeng Ma received his M.E. degree in college of computer science and technology from Zhejiang University of Technology, Hangzhou, China, in 2015. He is now a graduate at Zhejiang University of Technology. His-current research interests include reliability evaluation and deep learning.
Jungang Lou received the B.S. degree in Mathematics from Zhejiang Normal University, China, in 2003, and the M.S. degree in computational mathematics and the Ph.D. degree in computer science and technology from Tongji University, Shanghai, China, in 2006 and 2010, respectively. He is currently a Professor with the School of Information Engineering, Huzhou University, Huzhou, China. He also holds a postdoctoral position at the Institute of Cyber-Systems and Control, School of Control Science and Engineering, Zhejiang University, Zhejiang, China. He was a Visiting Scholar with the department of Computer Science at The University of Texas at San Antonio between Nov. 2017 and May 2018. His-current research interests include dependable computing, neural network optimization, reliability engineering, computer system performanceevaluation, and time series prediction. He has published over 60 papers in refereed international journals including Journal of Network and Computer Applications, IEEE Transactions on Neural Networks and Learning Systems, IEEE Transactions on Cybnetics and so on. Jianhui Jiang received his B.E., M.E. and Ph.D. degrees in traffic information engineering and control from Shanghai Tiedao University (in April 20 0 0, it was merged to Tongji University) in 1985, 1988, and 1999, respectively. From 1994–20 0 0, he was an Associate Professor of Computer Science and Technology at Shanghai Tiedao University. Since 20 0 0, he has been a full professor of Computer Science and Technology at Tongji University. From 2007– 2011, he was chair of the Department of Computer Science and Technology at Tongji University. Since 2011, he is Associate Dean of the School of Software Engineering at Tongji University. He is Vice Director of Technical Committee on Fault-tolerant Computing, Chinese Computer Federation (CCF). He has served on several program committees of national or international symposiums or workshops including IEEE Pacific Rim International Symposium on Dependable Computing, IEEE Asian Test Symposium, IEEE Workshop on RTL and High-Level Testing. He has coauthored two books and published more than 200 technical papers. His-current research interests include dependable systems and net-works, software reliability engineering, VLSI/SoC testing and fault-tolerance. His-research support is provided by the Ministry of Science and Technology of PRC, the National Natural Science Foundation of China, the Science and Technology Committee of Shanghai, the Science and Technology Department of Jiangsu, Huawei, and Shanghai Shentong Metro Group Company. He is a senior member of CCF. Yujiao Huang received the B.S. degree in information and computer science, the M.S. degree in computational mathematics and the Ph.D. degree in control theory and control engineering from Northeastern University, Shenyang, China, in 2008, 2010 and 2014, respectively. She is now a lecturer at Zhejiang University of Technology. Her research interests are in areas of artificial neural networks, stability theory, and dynamical systems.
Zhanhui Shi do his postgraduate work at the department of computer science and technology, Zhejiang University of Technology, Hangzhou, China. His-current research interests include reliability evaluation and fault-tolerant design, deep learning and combinatorial optimizationcomputation.
Qing Shen received her M.E. degree in Computer Application Technology in July 2007 from North University of China. She is currently an Associate Professor with the School of Information Engineering of Huzhou University. Her current research interests including artificial intelligence, pattern recognition, kernel learning, software testing, reliability evaluation.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100
JID: NEUCOM
ARTICLE IN PRESS
[m5G;September 10, 2019;23:55]
J. Xiao, W. Ma and J. Lou et al. / Neurocomputing xxx (xxxx) xxx Xu-Hua Yang received his B.E. degree in Automation from China University of Petroleum, Dongying, China, in 1993; and received his MSdegree and Ph.D. degree in control science and engineering from Zhejiang University, Hangzhou, China, in 2001 and 2004, respectively.
15
He is currently a Professor with Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China. His-current research interests include artificial intelligent, complex network system, intelligent transportation sytem, link prediction, and deep learning.
Please cite this article as: J. Xiao, W. Ma and J. Lou et al., Circuit reliability prediction based on deep autoencoder network, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.07.100