- Email: [email protected]
Extended Naive Bayes classifier for mixed data
Available online at www.sciencedirect.com
Expert Systems with Applications Expert Systems with Applications 35 (2008) 1080–1083 www.elsevier.com/locate/eswa
Extended Naive Bayes classifier for mixed data Chung-Chian Hsu a, Yan-Ping Huang
a,b,*
, Keng-Wei Chang
a
a
b
Department of Information Management, National Yunlin University of Science and Technology, 123, Section 3, University Road, Douliu Yunlin 640, Taiwan, ROC Department of Information Management, Chin Min Institute of Technology, 110, Hsueh-Fu Road, Tou-Fen, Miao-Li 351, Taiwan, ROC
Abstract Naive Bayes induction algorithm is very popular in classification field. Traditional method for dealing with numeric data is to discrete numeric attributes data into symbols. The difference of distinct discredited criteria has significant effect on performance. Moreover, several researches had recently employed the normal distribution to handle numeric data, but using only one value to estimate the population easily leads to the incorrect estimation. Therefore, the research for classification of mixed data using Naive Bayes classifiers is not very successful. In this paper, we propose a classification method, Extended Naive Bayes (ENB), which is capable for handling mixed data. The experimental results have demonstrated the efficiency of our algorithm in comparison with other classification algorithms ex. CART, DT and MLP’s. Ó 2007 Elsevier Ltd. All rights reserved. Keywords: Naive Bayes classifier; Classification; Mixed data
1. Introduction Naive Bayes classifiers are very robust to discriminate irrelevant attributes, and to classify evidence from many attributes to make the final prediction. Naive Bayes classifiers are generally easy to understand and the induction of these classifiers is extremely fast, requiring only a signal pass through the data. However, the algorithm is limited to categorical or discrete data. In other words, the classification of mixed data, which includes categorical and numeric data, is inapplicable. Traditional method for dealing with numeric data is to discrete numeric attributes data into symbols. However, the difference of distinct discredited criteria has significant effect on performance. Moreover, several researches has recently employed the normal distribution to handle
numeric data, but using only one value to estimate the population easily leads to the incorrect estimation. Hence, the research for classification of mixed data using Naive Bayes classifiers is not very successful. In this paper, we propose a classification method, ENB which is capable of handling mixed data. For categorical data, we utilize the original approach in the Naive Bayes algorithm to calculate the probabilities of categorical values. For continuous data, we adopt the statistical theory, in which we not only take the average into account but also consider the variance of numeric values. For an unknown input pattern, the product of the probabilities and the P-values are calculated and then the class which results in the maximum product is designated as the target class to which the input pattern belongs. 2. Naive Bayesian classifier
* Corresponding author. Address: Department of Information Management, National Yunlin University of Science and Technology, 123, Sec. 3, University Road, Douliu, Yunlin 640, Taiwan, ROC. Tel.: +886 37627153; fax: +886 97605684. E-mail addresses: [email protected] (C.-C. Hsu), sunny@ chinmin.edu.tw (Y.-P. Huang), [email protected] (K.-W. Chang).
0957-4174/$ - see front matter Ó 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2007.08.031
Bayesian networks have been successfully applied to a great number of classification problems. There has been a surge of interests in learning Bayesian networks from data. The goal is to induce a network that best captures the dependencies among the variables for the given data. A
C.-C. Hsu et al. / Expert Systems with Applications 35 (2008) 1080–1083
Naive Bayesian classifier assumes conditional independence among all attributes given the class variable. It learns from training data the conditional probability of each attribute given its class label (Duda & Hart, 1973; Langley, Iba, & Thompson, 1992). A simple Bayesian network classifier, which in practice often performs surprisingly well, is the Naive Bayesian classifier. This classifier basically learns the class-conditional probabilities P(Xi = xijC = cl) of each attribute Xi given the class label cl. A new test case X1 = x1, X2 = x2, . . ., Xn = xn is then classified by using Bayesian rule to compute the posterior probability of each class cl given the vector of observed attribute values according to the following equation: P ðC ¼ cl j X 1 ¼ x1 ; X 2 ¼ x2 ; . . . ; X n ¼ xn Þ ¼
P ðC ¼ cl ÞP ðX 1 ¼ x1 ; X 2 ¼ x2 ; . . . ; X n ¼ xn j C ¼ cl Þ P ðX 1 ¼ x1 ; X 2 ¼ x2 ; . . . ; X n Þ ð1Þ
The simplifying assumption behind the Naive Bayesian classifier is that the attributes are conditionally independent, given the class label according to the following equation: P ðX 1 ¼ x1 ; X 2 ¼ x2 ; . . . ; X n ¼ xn j C ¼ cl Þ n Y ¼ P ðX i ¼ xi j C ¼ cl Þ
ð2Þ
i¼1
This assumption simplifies the estimation of the classconditional probabilities from the training data. Notice that one does not estimate the denominator in Eq. (1) since it is independent of the class. Instead, one normalizes the nominator term to 1 over all classes. Bayesian network models are widely used for discriminative prediction tasks such as classification. In recent years, it has been recognized, both theoretically and experimentally, that in many situations it is better to use a matching ‘discriminative’ or ‘supervised’ learning algorithm such as conditional likelihood maximization (Friedman, Geiger, & Goldszmidt, 1997; Greiner, Grove, & Schuurmans, 1997; Kontkanen, Myllymaki, & Tirri, 2001; Ng & Jordan, 2001). Naive Bayesian classifiers have been proven successful in many domains, despite the simplicity of the model and the restrictiveness of the independent assumptions it made. Naive Bayesian algorithm handles only categorical data, but could not reasonably express the probability between two numeric values and preserve the structure of numeric values. Extended Naive Bayesian algorithm is used in data mining as a simple and effective classification algorithm. The ENB algorithm properly handles the mixed data. 3. Extended Naive Bayesian classification algorithm The ENB algorithm has been widely used in data mining as a simple and effective classification algorithm. It handles
1081
the mixed data. For a categorical attribute, the conditional probability that an instance belongs to a certain class c given that the instance has an attribute value A = a, P(C = c j A = a) is given by the following equation: P ðC ¼ cjA ¼ aÞ ¼
P ðC ¼ c \ A ¼ aÞ nac ¼ P ðA ¼ aÞ na
ð3Þ
where nac is the number of instances in the training set which has the class value c and an attribute value of a, while na is the number of instances which simply has an attribute value of a. Due to horizontal partitioning of data, each party has partial information about every attribute. Each party can locally compute the local count of instances. The global count is given by the sum of the local counts. For a numeric attribute, the necessary parameters are the mean l and variance r2 for all different classes. Again, the necessary information is split between the parties. In order to compute the mean, each party needs to sum the attribute values of the appropriate instances having the same class value. These local sums are added together and divided by the total number of instances having the same class to get the mean for that class value. The ENB clustering algorithm usually has the following steps Step 1. A training dataset requires xi parties, Ck class values and w attribute values. If there occur numeric data attributes, calculate the mean value and the variance value. However, the time is counted if categorical data attributes occur. Step 2. It calculates the probability of an instance having the class and the attribute value. If there occur numeric data attributes, the probability of attribute value xi in class Ck determines the probability according to the following equation: pðxi jC k Þ ¼ pðC k Þ 0
atti Y i¼1
pðwi;t jC k Þ
attj Y j¼iþ1
2 1
B C jX j X 0j j ðlj l0j Þ rffiffiffiffiffiffiffiffiffiffiffiffiffi pB mC @z P A: ^2j ^02 r r j þ 0 nj n j
ð4Þ If there occur categorical data attributes, the probability of attribute value xi domain value wi,t in class Ck determines the probability such that P(Cijx) > P(Cjjx), x is in class Ci; else x is in class Cj. The Bayesian approach to classify the new instance is to assign the most probable target value, P(Cijx), given the attribute values {w1, w2, . . ., wn} that describe the instance according the following equation: P ðC i jxÞ ¼
P ðC i \ xÞ P ðxjC i ÞP ðC i Þ ¼ P ðxÞ P ðxÞ
ð5Þ
1082
C.-C. Hsu et al. / Expert Systems with Applications 35 (2008) 1080–1083
Table 1 The ARR score in five datasets, six kinds of classification algorithm Algorithm
Credit approval
German credit
Hepatitis
Horse
Breast cancer
ARR
ENB
6.9245 Duda and Hart (1973) (6.2083, 8.1053)
3.0322 Duda and Hart (1973) (2.6022, 3.8551)
11.9333 Duda and Hart (1973) (7.0000, 27.0000)
5.2185 Friedman et al. (1997) (3.9286, 6.6667)
25.0150 Langley et al. (1992) 25.0150 (22.1000, 32.0000)
0.7667
MLP’s
5.8885 Greiner et al. (1997) (5.0702, 6.8636)
2.6528 Greiner et al. (1997) (2.2843, 2.8953)
5.5871 Greiner et al. (1997) (4.0909, 10.2000)
3.9193 Greiner et al. (1997) (2.6316, 5.2727)
29.2250 Duda and Hart (1973) (20.0000, 45.2000)
0.4000
CART
5.4120 Ng and Jordan (2001) (4.1642, 7.2381)
2.5758 Ng and Jordan (2001) (2.2212, 2.9412)
6.9730 Friedman et al. (1997) (3.3077, 10.2000)
26.4563 Duda and Hart (1973) (3.3125, 68.0000)
14.0205 Ng and Jordan (2001) (7.8846, 20.0000)
0.3867
DT
6.2979 Friedman et al. (1997) (4.7667, 7.8718)
2.5758 Ng and Jordan (2001) (2.2212, 2.9412)
8.1958 Langley et al. (1992) (4.0909, 17.6667)
21.1898 Langley et al. (1992) (5.2727, 33.5000)
14.8546 Greiner et al. (1997) (8.2400, 22.1000)
0.3567
The ENB classifier makes the simplification assumption that the attribute values are conditionally independent given the target value according the following equation: P ðxjC i Þ ¼
n Y
P ðxk jC i Þ:
ð6Þ
k¼1
The categorical values have been normalized before calculating the probability of each class. It determines the normalized according to the following equation: P 1 þ xi 2Ck N ðwi;t ; xi ÞpðC k jxi Þ P ðwi;t jC k Þ ¼ P PjV j jV j þ xi 2Ck t¼1 N ðwi;t ; xi ÞpðC k jxi Þ ð7Þ where jVj is the total domain value in the attribute value xi. Step 3. All parties calculate the probability of each class, according to the following equation: pi ¼
m Y
p½j
ð8Þ
j¼1
Step 4. It selects the maximal of the probability of each class according to the following equation: classCount
max ðp½iÞ i¼1
ð9Þ
The real mixed datasets Australian Credit Approval, German Credit Data, Hepatitis, Horse Colic and Breast Cancer from the UCI repository (Merz & Murphy, 1996). Australian Credit Approval has 690 records of 14 attributes, including eight categorical attributes and six numerical attributes. German Credit Data has 1000 records of 20 attributes, including thirteen categorical attributes and six numerical attributes. Hepatitis has 155 records of 19 attributes, including thirteen categorical attributes and six numerical attributes. Horse Colic has 368 records of 27 attributes, including twenty categorical attributes and seven numerical attributes. Breast Cancer has 699 records of 9 numerical attributes This study uses the average reciprocal rank (ARR) (Quinlan, 1986; Voorhees & Tice, 2000) as our evaluation metric. ARR is defined as shown in Eq. (10). Assume that the system only retrieves n relevant items and they are ranked as r1, r2, . . ., rn. n 1 X 1 ð10Þ ARR ¼ N i¼1 ri The ARR score for classification using the accuracy rate are shown in Table 1. The ARR score for classification using an average amount l, minimize value and maximize value. The accuracy rate is shown in Table 1. The higher the ARR values, the better the classification result. ENB has the highest ARR score. The results clearly show that the ENB algorithm has good classification quality. 5. Conclusions and future work
4. Experiments and results The algorithm is developed by using Java 2.0, Access database and SPSS Clementine 7.2. In the experiments, it presents the results of the ENB algorithm with mixed data and compares with other classification algorithms, ex. Decision Tree (DT) (Quinlan, 1993), Classification And Regression Tree (CART) (Breiman, Friedman, Olshen, & Stone, 1984), and Multiplayer Perceptions (MLP’s) (Simon, 1999).
The aim of this paper which demonstrates the technology for building classification algorithm from examples is fairly robust. This paper also proposes an efficient ENB algorithm for classification. Meanwhile, it achieves ARR value reappearing nearly 76% and it also endeavors to improve the accuracy in the classification. The further study can apply this algorithm in other time series databases, like financial databases.
C.-C. Hsu et al. / Expert Systems with Applications 35 (2008) 1080–1083
References Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and Regression Trees. The Wadsworth and Brooks. Duda, R. O., & Hart, P. E. (1973). Pattern Classification and Scene Analysis. John Wiley & Sons. Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian network classifiers. Machine Learning, 29(2), 131–163. Greiner, R., Grove, A., & Schuurmans, D. (1997). Learning Bayesian nets that perform well. In Proceedings of the thirteenth annual conference on uncertainty in artificial intelligence (pp. 198–207). San Francisco. Kontkanen, P., Myllymaki, P., & Tirri, H. (2001). Classifier learning with supervised marginal likelihood. In Proceedings of the seventeenth conference on uncertainty in artificial intelligence (pp. 277–284). San Francisco.
1083
Langley, P., Iba, W., & Thompson, K. (1992). An analysis of Bayesian classifiers. In Proceedings of the international conference on artificial intelligence. Merz, C.J., & Murphy, P. (1996). UCI repository of ML databases, http:// www.ics.uci.edu/~mlearn/MLRepository.html. Ng, A., & Jordan, M. (2001). On discriminative vs. generative classifiers: A comparison of logistic regression and Naive Bayes. Advances in neural information processing systems, 14, 605–610. Quinlan, J. R. (1986). Induction of decision tree. Machine Learning., 1, 81–106. Quinlan, J. R. (1993). C4.5: Programs for machine learning. San Mateo. CA: Morgan Kaufman. Simon, H. (1999). Neural Networks: A Comprehensive Foundation. Voorhees E.M., & Tice, D.M. (2000). The TREC-8 Question Answering Track Report. The Eighth Text Retrieval Conference (TREC-8).
Expert Systems with Applications Expert Systems with Applications 35 (2008) 1080–1083 www.elsevier.com/locate/eswa
Extended Naive Bayes classifier for mixed data Chung-Chian Hsu a, Yan-Ping Huang
a,b,*
, Keng-Wei Chang
a
a
b
Department of Information Management, National Yunlin University of Science and Technology, 123, Section 3, University Road, Douliu Yunlin 640, Taiwan, ROC Department of Information Management, Chin Min Institute of Technology, 110, Hsueh-Fu Road, Tou-Fen, Miao-Li 351, Taiwan, ROC
Abstract Naive Bayes induction algorithm is very popular in classification field. Traditional method for dealing with numeric data is to discrete numeric attributes data into symbols. The difference of distinct discredited criteria has significant effect on performance. Moreover, several researches had recently employed the normal distribution to handle numeric data, but using only one value to estimate the population easily leads to the incorrect estimation. Therefore, the research for classification of mixed data using Naive Bayes classifiers is not very successful. In this paper, we propose a classification method, Extended Naive Bayes (ENB), which is capable for handling mixed data. The experimental results have demonstrated the efficiency of our algorithm in comparison with other classification algorithms ex. CART, DT and MLP’s. Ó 2007 Elsevier Ltd. All rights reserved. Keywords: Naive Bayes classifier; Classification; Mixed data
1. Introduction Naive Bayes classifiers are very robust to discriminate irrelevant attributes, and to classify evidence from many attributes to make the final prediction. Naive Bayes classifiers are generally easy to understand and the induction of these classifiers is extremely fast, requiring only a signal pass through the data. However, the algorithm is limited to categorical or discrete data. In other words, the classification of mixed data, which includes categorical and numeric data, is inapplicable. Traditional method for dealing with numeric data is to discrete numeric attributes data into symbols. However, the difference of distinct discredited criteria has significant effect on performance. Moreover, several researches has recently employed the normal distribution to handle
numeric data, but using only one value to estimate the population easily leads to the incorrect estimation. Hence, the research for classification of mixed data using Naive Bayes classifiers is not very successful. In this paper, we propose a classification method, ENB which is capable of handling mixed data. For categorical data, we utilize the original approach in the Naive Bayes algorithm to calculate the probabilities of categorical values. For continuous data, we adopt the statistical theory, in which we not only take the average into account but also consider the variance of numeric values. For an unknown input pattern, the product of the probabilities and the P-values are calculated and then the class which results in the maximum product is designated as the target class to which the input pattern belongs. 2. Naive Bayesian classifier
* Corresponding author. Address: Department of Information Management, National Yunlin University of Science and Technology, 123, Sec. 3, University Road, Douliu, Yunlin 640, Taiwan, ROC. Tel.: +886 37627153; fax: +886 97605684. E-mail addresses: [email protected] (C.-C. Hsu), sunny@ chinmin.edu.tw (Y.-P. Huang), [email protected] (K.-W. Chang).
0957-4174/$ - see front matter Ó 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2007.08.031
Bayesian networks have been successfully applied to a great number of classification problems. There has been a surge of interests in learning Bayesian networks from data. The goal is to induce a network that best captures the dependencies among the variables for the given data. A
C.-C. Hsu et al. / Expert Systems with Applications 35 (2008) 1080–1083
Naive Bayesian classifier assumes conditional independence among all attributes given the class variable. It learns from training data the conditional probability of each attribute given its class label (Duda & Hart, 1973; Langley, Iba, & Thompson, 1992). A simple Bayesian network classifier, which in practice often performs surprisingly well, is the Naive Bayesian classifier. This classifier basically learns the class-conditional probabilities P(Xi = xijC = cl) of each attribute Xi given the class label cl. A new test case X1 = x1, X2 = x2, . . ., Xn = xn is then classified by using Bayesian rule to compute the posterior probability of each class cl given the vector of observed attribute values according to the following equation: P ðC ¼ cl j X 1 ¼ x1 ; X 2 ¼ x2 ; . . . ; X n ¼ xn Þ ¼
P ðC ¼ cl ÞP ðX 1 ¼ x1 ; X 2 ¼ x2 ; . . . ; X n ¼ xn j C ¼ cl Þ P ðX 1 ¼ x1 ; X 2 ¼ x2 ; . . . ; X n Þ ð1Þ
The simplifying assumption behind the Naive Bayesian classifier is that the attributes are conditionally independent, given the class label according to the following equation: P ðX 1 ¼ x1 ; X 2 ¼ x2 ; . . . ; X n ¼ xn j C ¼ cl Þ n Y ¼ P ðX i ¼ xi j C ¼ cl Þ
ð2Þ
i¼1
This assumption simplifies the estimation of the classconditional probabilities from the training data. Notice that one does not estimate the denominator in Eq. (1) since it is independent of the class. Instead, one normalizes the nominator term to 1 over all classes. Bayesian network models are widely used for discriminative prediction tasks such as classification. In recent years, it has been recognized, both theoretically and experimentally, that in many situations it is better to use a matching ‘discriminative’ or ‘supervised’ learning algorithm such as conditional likelihood maximization (Friedman, Geiger, & Goldszmidt, 1997; Greiner, Grove, & Schuurmans, 1997; Kontkanen, Myllymaki, & Tirri, 2001; Ng & Jordan, 2001). Naive Bayesian classifiers have been proven successful in many domains, despite the simplicity of the model and the restrictiveness of the independent assumptions it made. Naive Bayesian algorithm handles only categorical data, but could not reasonably express the probability between two numeric values and preserve the structure of numeric values. Extended Naive Bayesian algorithm is used in data mining as a simple and effective classification algorithm. The ENB algorithm properly handles the mixed data. 3. Extended Naive Bayesian classification algorithm The ENB algorithm has been widely used in data mining as a simple and effective classification algorithm. It handles
1081
the mixed data. For a categorical attribute, the conditional probability that an instance belongs to a certain class c given that the instance has an attribute value A = a, P(C = c j A = a) is given by the following equation: P ðC ¼ cjA ¼ aÞ ¼
P ðC ¼ c \ A ¼ aÞ nac ¼ P ðA ¼ aÞ na
ð3Þ
where nac is the number of instances in the training set which has the class value c and an attribute value of a, while na is the number of instances which simply has an attribute value of a. Due to horizontal partitioning of data, each party has partial information about every attribute. Each party can locally compute the local count of instances. The global count is given by the sum of the local counts. For a numeric attribute, the necessary parameters are the mean l and variance r2 for all different classes. Again, the necessary information is split between the parties. In order to compute the mean, each party needs to sum the attribute values of the appropriate instances having the same class value. These local sums are added together and divided by the total number of instances having the same class to get the mean for that class value. The ENB clustering algorithm usually has the following steps Step 1. A training dataset requires xi parties, Ck class values and w attribute values. If there occur numeric data attributes, calculate the mean value and the variance value. However, the time is counted if categorical data attributes occur. Step 2. It calculates the probability of an instance having the class and the attribute value. If there occur numeric data attributes, the probability of attribute value xi in class Ck determines the probability according to the following equation: pðxi jC k Þ ¼ pðC k Þ 0
atti Y i¼1
pðwi;t jC k Þ
attj Y j¼iþ1
2 1
B C jX j X 0j j ðlj l0j Þ rffiffiffiffiffiffiffiffiffiffiffiffiffi pB mC @z P A: ^2j ^02 r r j þ 0 nj n j
ð4Þ If there occur categorical data attributes, the probability of attribute value xi domain value wi,t in class Ck determines the probability such that P(Cijx) > P(Cjjx), x is in class Ci; else x is in class Cj. The Bayesian approach to classify the new instance is to assign the most probable target value, P(Cijx), given the attribute values {w1, w2, . . ., wn} that describe the instance according the following equation: P ðC i jxÞ ¼
P ðC i \ xÞ P ðxjC i ÞP ðC i Þ ¼ P ðxÞ P ðxÞ
ð5Þ
1082
C.-C. Hsu et al. / Expert Systems with Applications 35 (2008) 1080–1083
Table 1 The ARR score in five datasets, six kinds of classification algorithm Algorithm
Credit approval
German credit
Hepatitis
Horse
Breast cancer
ARR
ENB
6.9245 Duda and Hart (1973) (6.2083, 8.1053)
3.0322 Duda and Hart (1973) (2.6022, 3.8551)
11.9333 Duda and Hart (1973) (7.0000, 27.0000)
5.2185 Friedman et al. (1997) (3.9286, 6.6667)
25.0150 Langley et al. (1992) 25.0150 (22.1000, 32.0000)
0.7667
MLP’s
5.8885 Greiner et al. (1997) (5.0702, 6.8636)
2.6528 Greiner et al. (1997) (2.2843, 2.8953)
5.5871 Greiner et al. (1997) (4.0909, 10.2000)
3.9193 Greiner et al. (1997) (2.6316, 5.2727)
29.2250 Duda and Hart (1973) (20.0000, 45.2000)
0.4000
CART
5.4120 Ng and Jordan (2001) (4.1642, 7.2381)
2.5758 Ng and Jordan (2001) (2.2212, 2.9412)
6.9730 Friedman et al. (1997) (3.3077, 10.2000)
26.4563 Duda and Hart (1973) (3.3125, 68.0000)
14.0205 Ng and Jordan (2001) (7.8846, 20.0000)
0.3867
DT
6.2979 Friedman et al. (1997) (4.7667, 7.8718)
2.5758 Ng and Jordan (2001) (2.2212, 2.9412)
8.1958 Langley et al. (1992) (4.0909, 17.6667)
21.1898 Langley et al. (1992) (5.2727, 33.5000)
14.8546 Greiner et al. (1997) (8.2400, 22.1000)
0.3567
The ENB classifier makes the simplification assumption that the attribute values are conditionally independent given the target value according the following equation: P ðxjC i Þ ¼
n Y
P ðxk jC i Þ:
ð6Þ
k¼1
The categorical values have been normalized before calculating the probability of each class. It determines the normalized according to the following equation: P 1 þ xi 2Ck N ðwi;t ; xi ÞpðC k jxi Þ P ðwi;t jC k Þ ¼ P PjV j jV j þ xi 2Ck t¼1 N ðwi;t ; xi ÞpðC k jxi Þ ð7Þ where jVj is the total domain value in the attribute value xi. Step 3. All parties calculate the probability of each class, according to the following equation: pi ¼
m Y
p½j
ð8Þ
j¼1
Step 4. It selects the maximal of the probability of each class according to the following equation: classCount
max ðp½iÞ i¼1
ð9Þ
The real mixed datasets Australian Credit Approval, German Credit Data, Hepatitis, Horse Colic and Breast Cancer from the UCI repository (Merz & Murphy, 1996). Australian Credit Approval has 690 records of 14 attributes, including eight categorical attributes and six numerical attributes. German Credit Data has 1000 records of 20 attributes, including thirteen categorical attributes and six numerical attributes. Hepatitis has 155 records of 19 attributes, including thirteen categorical attributes and six numerical attributes. Horse Colic has 368 records of 27 attributes, including twenty categorical attributes and seven numerical attributes. Breast Cancer has 699 records of 9 numerical attributes This study uses the average reciprocal rank (ARR) (Quinlan, 1986; Voorhees & Tice, 2000) as our evaluation metric. ARR is defined as shown in Eq. (10). Assume that the system only retrieves n relevant items and they are ranked as r1, r2, . . ., rn. n 1 X 1 ð10Þ ARR ¼ N i¼1 ri The ARR score for classification using the accuracy rate are shown in Table 1. The ARR score for classification using an average amount l, minimize value and maximize value. The accuracy rate is shown in Table 1. The higher the ARR values, the better the classification result. ENB has the highest ARR score. The results clearly show that the ENB algorithm has good classification quality. 5. Conclusions and future work
4. Experiments and results The algorithm is developed by using Java 2.0, Access database and SPSS Clementine 7.2. In the experiments, it presents the results of the ENB algorithm with mixed data and compares with other classification algorithms, ex. Decision Tree (DT) (Quinlan, 1993), Classification And Regression Tree (CART) (Breiman, Friedman, Olshen, & Stone, 1984), and Multiplayer Perceptions (MLP’s) (Simon, 1999).
The aim of this paper which demonstrates the technology for building classification algorithm from examples is fairly robust. This paper also proposes an efficient ENB algorithm for classification. Meanwhile, it achieves ARR value reappearing nearly 76% and it also endeavors to improve the accuracy in the classification. The further study can apply this algorithm in other time series databases, like financial databases.
C.-C. Hsu et al. / Expert Systems with Applications 35 (2008) 1080–1083
References Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and Regression Trees. The Wadsworth and Brooks. Duda, R. O., & Hart, P. E. (1973). Pattern Classification and Scene Analysis. John Wiley & Sons. Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian network classifiers. Machine Learning, 29(2), 131–163. Greiner, R., Grove, A., & Schuurmans, D. (1997). Learning Bayesian nets that perform well. In Proceedings of the thirteenth annual conference on uncertainty in artificial intelligence (pp. 198–207). San Francisco. Kontkanen, P., Myllymaki, P., & Tirri, H. (2001). Classifier learning with supervised marginal likelihood. In Proceedings of the seventeenth conference on uncertainty in artificial intelligence (pp. 277–284). San Francisco.
1083
Langley, P., Iba, W., & Thompson, K. (1992). An analysis of Bayesian classifiers. In Proceedings of the international conference on artificial intelligence. Merz, C.J., & Murphy, P. (1996). UCI repository of ML databases, http:// www.ics.uci.edu/~mlearn/MLRepository.html. Ng, A., & Jordan, M. (2001). On discriminative vs. generative classifiers: A comparison of logistic regression and Naive Bayes. Advances in neural information processing systems, 14, 605–610. Quinlan, J. R. (1986). Induction of decision tree. Machine Learning., 1, 81–106. Quinlan, J. R. (1993). C4.5: Programs for machine learning. San Mateo. CA: Morgan Kaufman. Simon, H. (1999). Neural Networks: A Comprehensive Foundation. Voorhees E.M., & Tice, D.M. (2000). The TREC-8 Question Answering Track Report. The Eighth Text Retrieval Conference (TREC-8).