Ordinal classification using Pareto fronts

ESWA 9932

No. of Pages 7, Model 5G

7 April 2015 Expert Systems with Applications xxx (2015) xxx–xxx 1

Contents lists available at ScienceDirect

Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa 4 5 3

Ordinal classification using Pareto fronts q

6

M.M. Stenina a, M.P. Kuznetsov a,⇑, V.V. Strijov a,b

7 8 9 10 1 1 2 2 13 14 15 16 17 18 19 20

a b

Moscow Institute of Physics and Technology, Institutskiy Lane 9, Dolgoprudny, Moscow 141700, Russia Dorodnicyn Computing Center of Russian Academy of Sciences, Vavilov St. 40, 119333 Moscow, Russia

a r t i c l e

i n f o

Article history: Available online xxxx Keywords: Ordinal classification Pareto front Expert estimations Binary relation

a b s t r a c t The paper presents an ordinal classification method using Pareto fronts. An object is described by a set of ordinal features assigned by experts. We describe the class boundaries by the set of Pareto fronts. We propose to predict the object class using the nearest Pareto front boundary. The proposed method is illustrated by a real-world application to the International Union for Conservation of Nature Red List species categorization. Ó 2015 Published by Elsevier Ltd.

22 23 24 25 26 27 28 29

30 31

1. Introduction

32

We investigate the supervised ordinal classification problem (Frank & Hall, 2001; Har-Peled, Roth, & Zimak, 2003), a setting referred to as learning to rank or ordinal regression. Several key assumptions are made about the investigated data structure within the ordinal classification setting. First, the target variable assumed to have an ordinal nature. As formulated in Chu and Sathiya Keerthi (2007), that setting bridges the gap between regression and classification problems. The second assumption corresponds to the structure of features describing the given data. We consider the ordinal nature of input variables, that is, each feature provides the non-strict ranking over the set of objects (Ben-David, Sterling, & Pao, 1989; Furnkranz & Hullermeier, 2003; Kotlowski & Slowinski, 2009). We also make the following assumptions about the feature structure (Strijov, Granic, Juric, Jelavic, & Maricic, 2011):

33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53

 the given set of features is sufficient to construct an adequate ordinal classification model;  the rule that ‘‘bigger is better’’ is applied, that is, larger feature values imply greater object preference. In current research we consider the important requirement of monotonicity between the input and output ordinal variables. Monotone relationships are frequently encountered in machine q This project was supported by the Ministry of Education and Science of the Russian Federation, RFMEFI60414X0041, and by the Russian Foundation for Basic Research, Grant 14-07-31042. ⇑ Corresponding author. Tel.: +7 (926) 832 01 97; fax: +7 (499) 783 33 27. E-mail addresses: [email protected] (M.M. Stenina), [email protected] (M.P. Kuznetsov), [email protected] (V.V. Strijov).

learning area (Kotlowski & Slowinski, 2009). Recent investigations (Duivesteijn & Feelders, 2008) showed that satisfying the monotonicity requirements leads to the better prediction quality. The problem of monotone ordinal classification arises in the area of information retrieval (Xia, Liu, Wang, Zhang, & Li, 2008). The most common methods are based on object pairwise comparisons. To solve such kind of problems, the modified support vector machine (Yue, Finley, Radlinski, & Joachims, 2007) and boosting (Freund, Iyer, Schapire, & Singer, 2003) are used. However, the important shortcoming of such boosting-like models is their complexity and non-interpretability in the context of the investigated field. In this paper we propose a new ordinal classification method that operates with ordinal input and target variables. The method constructs classification model basing on the object dominance concept. We propose to describe the ordinal class boundaries using the Pareto front idea (Nogin, 2003). We propose to construct the set of Pareto fronts corresponding to the ordinal classes and to predict the object class using the nearest Pareto front boundary. To build the correct Pareto front model we introduce the concept of separable sample and propose the new efficient method of its construction. The main advantage of the method is that it allows to construct simple interpretable models on the same level of prediction accuracy. We use the proposed classification method to categorize threatened animal and plant species included in the International Union for Conservation of Nature (IUCN) Red List. Each species on this list belongs to one of seven possible categories: extinct, extinct in the wild, critically endangered, endangered, vulnerable, near threatened, and least concern. This categorization is monotone with respect to the risk of extinction. The object-feature matrix for the categorization problem consists of ordinal species descriptions assigned the experts and class labels corresponding to the species categories. The problem is to

http://dx.doi.org/10.1016/j.eswa.2015.03.021 0957-4174/Ó 2015 Published by Elsevier Ltd.

Please cite this article in press as: Stenina, M. M., et al. Ordinal classification using Pareto fronts. Expert Systems with Applications (2015), http://dx.doi.org/ 10.1016/j.eswa.2015.03.021

54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85

ESWA 9932

No. of Pages 7, Model 5G

7 April 2015 2

M.M. Stenina et al. / Expert Systems with Applications xxx (2015) xxx–xxx

Table 1 The part of ordinal IUCN data in the object-feature matrix. Feature

Condition

Trend

Population

3 – big; 2 – small; 1 – critically small 2 – complex; 1 – simple

4 – grows; 3 – stable; 2 – reduces; 1 – reduces fast 2 – stable; 1 – disappear

Population structure

then the ði; kÞ-th matrix is 1. For strict total orders, the matrix is lower triangular with zeros on the diagonal. Element li of set Y corresponds to row i of the matrix. We define the loss value rð; Þ as the difference between strings of the matrix Y,

135 136 137 138 139

140

Y X 0 rðli ; li0 Þ ¼ jYði; kÞ  Yði ; kÞj:

ð3Þ

142

k¼1

100

construct a model that estimates a class label from the species description. Table 1 shows four features within the object-feature matrix: ‘‘Population condition’’, ‘‘Population trend’’, ‘‘Population structure condition’’, ‘‘Population structure trend’’. We show that the proposed Pareto front method allows to construct an accurate, stable and interpretable ordinal classification model for the problem of the IUCN categorization. In addition to the IUCN data, we test the method on the benchmark datasets. We compare the method with two ordinal classification algorithms from Frank and Hall (2001) and one method of classification with monotonicity constraints from Duivesteijn and Feelders (2008). Furthermore, to emphasize the ordinal nature of the algorithm, we make ordinal feature transformation of the benchmark datasets and compare the methods on the transformed data.

101

2. Problem setting

86 87 88 89 90 91 92 93 94 95 96 97 98 99

102

103 105 106

107 109 110 111 112 113 114 115 116 117 118

119 121 122

123 125 126

127 129 130 131 132 133 134

Consider the set of pairs

D ¼ fðxi ; yi Þg;

i 2 I ¼ f1; . . . ; mg;

consisting of objects xi and class labels yi . Each object >

xi ¼ ½xi1 ; . . . ; xij ; . . . ; xid  ;

3. Two-class Pareto classification

143

Consider a special two-class case of problem (1) such that Y ¼ fl1 ; l2 g ¼ f0; 1g; 0  1. That is, the sample D consists of objects with class labels denoted by 0 or 1. Let f be the monotonic function minimizing error (2) in the two-class case. To construct f in this case, we first define f ðxÞ on the separable ^ sample D,

144

^ ¼ fðxi ; y Þg; D i

152

is described by ordinal-scaled measurements. This means that the set of values for feature j is a finite ordered set Lj with a binary relation . In this paper, we consider only strict total orders, i.e., total, non-reflexive, asymmetric and transitive binary relations. However, the proposed methods can be generalized to the case of partial orders. The set of values Y for the class labels yi is also a finite strictly ordered set Y ¼ fl1 ; . . . ; lY g with a binary relation l1  . . .  lY . The problem is to find a monotonic function

u : x # y^;

ð1Þ

150

G

P

154 155 156 157 158 160 161 162

163 165

such that yn ¼ 0 for n 2 N , and yp ¼ 1 for p 2 P.

166

3.1. Dominance relation

167

We now introduce the concept of a dominance relation. This includes n-domination and p-domination. We say that object xn ¼ ½xn1 ; . . . ; xnd > n-dominates object xi ¼ ½xi1 ; . . . ; xid > ,

168

or xn n xi ; if xnj xij for each j ¼ 1; . . . ; d: say

that

object

169 170

171 173

>

xp ¼ ½xp1 ; . . . ; xpd 

p-dominates

object

174 175

176

or xp p xk ;

This function should minimize the error SðuÞ,

SðuÞ ¼

149

159

xk ¼ ½xk1 ; . . . ; xkd  ,

and y 2 Y:

148

inition of the mapping f will be extended to the entire sample D and to the whole set of objects X. For the two-class classification problem, we split the set of ^ into two subsets: object indices I^ of the separable sample D

>

x 2 X ¼ L1      Ld

147

153

We

where

146

^ is a subset of the entire sample D. The ‘‘separable samsuch that D ple’’ concept for ordered sets means that there exists a hull (or Pareto optimal front) corresponding to each class defined by the binary relation ‘‘’’ such that the hulls for two classes do not intersect. First, the function f will be defined on the separable sample set ^ such that the error function (2) is zero on D. ^ Second, the defD

I^ ¼ N

j 2 J ¼ f1; . . . ; dg;

i 2 I^ # I

145

if

X ^i Þ; rðyi ; y

ð2Þ

i2I

^i ¼ uðxi Þ, and rð; Þ is the loss function between elements of where y the ordered set Y. To define a loss function between elements of the set Y ¼ fl1 ; . . . ; lY jl1  . . .  lY g, we introduce a binary matrix Y (Table 2) describing the binary relations between elements of Y. If li  lk ,

Table 2 Matrix of an ordered set. Labels

l1

l2

...

lY1

lY

l1 l2 ... lY1 lY

0 1 ... 1 1

0 0 ... 1 1

... ... ... ... ...

0 0 ... 0 1

0 0 ... 0 0

xpj xkj

for each j ¼ 1; . . . ; d:

We can assume that an object neither n-dominates nor p-dominates itself,

x ¤n x;

x ¤p x:

178 179 180

181 183

Fig. 1 illustrates the dominance relation in the case of two features; the x-axis denotes feature values from set L1 , and the y-axis denotes feature values from set L2 . The yellow region indicates the n-dominance space for object xn and the p-dominance space for object xp . Object xn n-dominates each object xi from the corresponding n-dominance space, and object xp p-dominates each object xk from the corresponding p-dominance space.

184

3.2. Pareto front construction

191

Let us define the Pareto fronts, the sets describing class boundaries for the separable sample.

192

Please cite this article in press as: Stenina, M. M., et al. Ordinal classification using Pareto fronts. Expert Systems with Applications (2015), http://dx.doi.org/ 10.1016/j.eswa.2015.03.021

185 186 187 188 189 190

193

ESWA 9932

No. of Pages 7, Model 5G

7 April 2015 3

M.M. Stenina et al. / Expert Systems with Applications xxx (2015) xxx–xxx

9 8 7

6 Feature 2

Feature 2

8

4

2

6

3

5

1

4 3

2

2

0 0

2

4 6 Feature 1

1

8

Fig. 1. Dominance relation.

2

4

Feature 1

6

8

Fig. 3. Two-class classification example.

! 194 195 196

Definition 1. A set of objects xn ; n 2 N , is called Pareto front POFn if, for each element xn 2 POFn , there does not exist any x such that xn xn .

206

Definition 2. A set of objects xp ; p 2 P is called Pareto front POFp if, for each element xp 2 POFp , there does not exist any x such that x p xp . Fig. 2 illustrates Pareto fronts for the two-class separable sample. Each object is described by two features. The x-axis denotes feature values from set L1 , and the y-axis denotes feature values from set L2 . The green triangles and blue squares are the objects from different classes. The objects forming the Pareto fronts are denoted by red circles. The dotted line indicates the n-front class boundary, and the solid line indicates the p-front class boundary.

207

3.3. Two-class classification

208

We now use the constructed Pareto fronts and the corresponding class boundaries to define a monotone classifier. ^ assigns the class label 0 to an object x 2 X if Function f : x # y ~ -dominates x. Function f there exists an object xn 2 POFn that n assigns the class label 1 to an object x 2 X if there exists an object ~-dominates x. Thus, xp 2 POFp that p

198 199 200 201 202 203 204 205

209 210 211 212 213

214 216

217 218

219

f ðxÞ ¼



0; if there exists xn 2 POFn : xn n~ x; 1; if there exists xp 2 POFp : xp p~ x:

ð4Þ

If the set of such elements is empty, we extend the definition of f to the entire set X according to the nearest Pareto front:

arg min ðqðx; x ÞÞ ;

f ðxÞ ¼ f

221

x0 2POFn [POFp

where the sets POFn ; POFp include Pareto fronts and boundary points corresponding to the imaginary objects. The function q is defined by function (3) applied to the feature values: d X qðx; x Þ ¼ rðxj ; x0j Þ: 0

3.4. Separable sample construction

242

Consider the method of constructing a set I^ such that the func^ is monotone over the corresponding subsample. tion f : x # y Split the set of indices I into two subsets

243

G

6

lðxi Þ ¼

5

1

2

4

Feature 1

6

Fig. 2. Pareto fronts.

8

230 231 232 233 234 235 236 237 238 239 240 241

244 245

248 249

n 2 N;

and yp ¼ 1;

250 252

p 2 P:



253 254

255

#fxj jxi n xj ; j 2 Pg;

if i 2 N ;

#fxj jxi p xj ; j 2 N g; if i 2 P;

257

Table 3 Two-class classifier example.

2

229

246

P

4 3

225

228

Consider the power l of the set of objects dominated by object xi and belonging to the foreign class:

7

224

In other words, f classifies an object x according to the rule of the nearest POF if the object x is not dominated by any POF. Fig. 3 shows model data consisting of two-class objects. Objects in the first class are indicated by green triangles, and those in the second class are denoted by blue squares. Each object is described by two features. The x-axis shows feature values from set L1 , the yaxis represents those from set L2 . The classified objects are indicated by the black circles. Table 3 gives the classification results. The first column contains the object number, the second column contains the object coordinates, and the third column contains the classifier output. The label 0 implies that the object was classified as a green triangle, whereas a 1 denotes that the object was classified as a blue square.

yn ¼ 0;

8

223

227

such that

9

222

ð5Þ

j¼1

I ¼N

Feature 2

197

0

Object

x

f ðxÞ

1 2 3

(4, 5) (6, 7) (9, 6)

0 1 1

Please cite this article in press as: Stenina, M. M., et al. Ordinal classification using Pareto fronts. Expert Systems with Applications (2015), http://dx.doi.org/ 10.1016/j.eswa.2015.03.021

ESWA 9932

No. of Pages 7, Model 5G

7 April 2015 4 258 259 260

M.M. Stenina et al. / Expert Systems with Applications xxx (2015) xxx–xxx

where # denotes the power of the set. To find set I^ , we consequently eliminate defective objects from the entire sample D. 1: I ¼ P

F

N.

^ 2: return I^ ¼ P

F ^ N.

^ :¼ N ; {initialization} ^ :¼ P; N 3: I^ :¼ I ; P 4: while the sample has the objects xi ; i 2 I^ such that lðxi Þ > 0 do 5: ^i :¼ arg maxlðx Þ;

8: 9: 10:

^ :¼ P ^ n f^ig; P ^ ^ if i 2 N then ^ :¼ N ^ n f^ig. N

281

Fig. 4 shows the model sample set, where objects are described with two features. This sample set consists of two classes (green triangles and blue squares). Fig. 4(a) shows the sample set with defective objects at coordinates (1, 1), (5, 4), and (8, 6). The defective objects dominate objects in the opposite class. These objects are indicated by red circles. Fig. 4(b) shows the separable subsample obtained once the defective objects have been eliminated.

282

4. Ordinal classification

283

4.1. Ordinal classifier construction

276 277 278 279 280

where n 2 N u ; if yn lu ;

and 301

The ordinal classifier

302

uðxÞ ¼ uðf 1;2 ; . . . ; f Y1;Y ÞðxÞ; u : X ! Y;

303 305

is defined as follows:

306

307

minflu jf u;uþ1 ðxÞ ¼ 0g; if flu jf u;uþ1 ðxÞ ¼ 0g – ;; lu 2Y

if flu jf u;uþ1 ðxÞ ¼ 0g ¼ ;:

lY ;

ð6Þ 309

Table 4 illustrates the application of Eq. (6). The output of the ordinal classifier uðxÞ is the label lu of the first class when the classifier f u;uþ1 equals 0, whereas if the output f u;uþ1 is 1, we assign the label lY to the ordinal classification result. Fig. 5 shows an example for a set of objects from three different classes. The axes denote feature values describing the objects. The various objects are indicated by the red circles, green triangles and blue squares. The class boundaries corresponding to the n-fronts are indicated by the dotted line, and the solid line indicates the

I^ :¼ I^ n f^ig; ^ then if ^i 2 P

274 275

P uþ1 ;

p 2 P uþ1 ; if yp luþ1 :

uðxÞ ¼

i2I^

7:

G

(

i

6:

I^ ¼ N u

Table 4 Monotone classifier illustration. 1, 2 1

u  1; u 1

... ...

u; u þ 1 0

... ...

Y  1; Y 0

9 8

289

290 292 293 294 295 296 297 298

299

7

l1      lu  luþ1      lY :

Feature 2

288

Y ¼ fl1 ; . . . ; lu ; luþ1 ; . . . ; lY g;

We denote the class label indices by f1; . . . ; u; u þ 1; . . . ; Yg. The Pareto two-class classifier can be constructed as

^ 2 f0; 1g; f u;uþ1 : x # y

x 2 X;

4

1

9 8

7

7

6

6

5 4

2

2 6

8

Feature 1

6

8

4 3

Feature 1

4

5

3

4

2

Fig. 5. Pareto front, feature 1 is preferable to feature 2.

8

2

5

2

9

1

6

3

for each pair of adjacent classes u; u þ 1. To construct the two-class classifier, we split the sample into two parts: objects with class labels ^lu and objects with class labels
Feature 2

285 287

Consider a general case of the problem:

Feature 2

284

1

2

4

Feature 1

6

8

Fig. 4. Eliminating defective objects.

Please cite this article in press as: Stenina, M. M., et al. Ordinal classification using Pareto fronts. Expert Systems with Applications (2015), http://dx.doi.org/ 10.1016/j.eswa.2015.03.021

310 311 312 313 314 315 316 317 318

ESWA 9932

No. of Pages 7, Model 5G

7 April 2015 5

M.M. Stenina et al. / Expert Systems with Applications xxx (2015) xxx–xxx

(

Table 5 Ordinal classifier example. Number

Object, x

f 12 ðxÞ

f 23 ðxÞ

uðxÞ

1 2 3

(1, 1) (5, 4) (9, 9)

0 1 1

0 0 1

1 2 3

if f u;uþ1 ðxÞ ¼ 0; than f ðuþsÞðuþ1þsÞ ðxÞ ¼ 0 for each s : ðu þ 1 þ sÞ 6 Y; if f u;uþ1 ðxÞ ¼ 1; than f ðusÞðuþ1sÞ ðxÞ ¼ 1 for each s : ðu  sÞ P 1: ð7Þ

353

Definition 4. We say that Pareto fronts POFn ðuÞ and POFp ðu þ 1Þ do not intersect,

354

POFn ðuÞ

329

p-fronts. The classified objects are shown by black circles. Table 5 shows an example of the set of two-class classifiers f 1;2 ; f 2;3 included in the ordinal classifier uðxÞ for the illustrated sample. The first column contains object numbers, the second their coordinates, and the third and fourth columns contain the two-class classification results for the adjacent classes. A label of 0 in the third column means that the classifier f 1;2 assigned the object to the first class, and a label of 0 in the fourth column means that the classifier f 2;3 assigned the object to the second class. The final column contains the results of ordinal classification. The values of this column correspond to the output of the ordinal classifier.

330

4.2. Extension of Pareto front definition for ordinal classification

331

344

To construct the fronts between classes labeled as lu and luþ1 , we use objects with class labels l1 ; . . . ; lu to construct an n-front for class lu and objects with class labels luþ1 ; . . . ; lY to construct a p-front for class luþ1 . Therefore, the same objects belong to fronts for different classes, and a front for one class contains objects of different classes. Fig. 6 illustrates a sample containing three-class model data. Red circles denote objects in the first class, and green triangles denote those in the second. The figure shows that object (7, 2) from the first class belongs to the n-fronts of both the first and second classes. This implies that the definition of the n-front should be extended to objects whose class label is not greater than the n-front class label; the definition of the p-front should be extended to objects whose class label is not less than the p-front class label.

345

4.3. Admissible classifiers

346

In this section, we introduce the important property of admissibility for an ordinal classifier. We prove a theorem about the admissibility conditions of the constructed classifier u.

321 322 323 324 325 326 327 328

332 333 334 335 336 337 338 339 340 341 342 343

347 348 349 350

351

359

POFn ðuÞ

362

\

POFp ðu þ 1Þ ¼ ;:

Fig. 2 shows an example of non-intersecting Pareto fronts. Theorem 1. If the Pareto fronts do not intersect,

POFn ðuÞ

\

POFp ðu þ 1Þ ¼ ;;

POFn ðuÞ

\

POFp ðu þ 1Þ ¼ ;;

f 1;2 ðxÞ ¼ 0;

f 2;3 ðxÞ ¼ 1:

372 374 375 376

377 379

1. 9y 2 POFn ð1Þ : y n x. If y 2 POFn ð2Þ, then it follows that f 2;3 ðxÞ ¼ 0, which contradicts the assumption that f 2;3 ðxÞ ¼ 1. If y R POFn ð2Þ, then

383

w n y;

and it follows that

)

9y0 2 POFn ð1Þ; such that y0 ¼ arg

6

min

qðx; yÞ;

y2POFn ð1Þ[POFp ð2Þ

5 4

3

3

2

2 1

2

4

Feature 1

382

384 385 386

387 389 391 393

f 2;3 ðxÞ ¼ 0:

2. The fronts POFn ð1Þ and POFp ð2Þ do not dominate object x. In this case,

6

381

390

POFn ð2Þ 3 wn y n x

7

8

371

380

7

6

370

(The case f 1;2 ðxÞ ¼ 1; f 2;3 ðxÞ ¼ 0 is similar.) The result f 1;2 ðxÞ ¼ 0 can be obtained if one of the two following conditions holds:

8

Feature 1

u ¼ 1; 2;

and that there exists an object x such that the transitivity condition does not hold,

8

4

365 367

u ¼ 1; . . . ; Y  1;

369

9

2

364

Proof. We prove the theorem for the case z ¼ 3. For more classes, the proof is similar. Suppose that the Pareto fronts do not intersect,

9

1

363

368

Definition 3. The classifier u in Eq. (6) is said to be admissible if, for each function f u;uþ1 , the transitivity condition holds:

4

360

then the transitivity condition (7) holds for any classified object.

9w 2 POFn ð2Þ :

5

356 358

POFp ðu þ 1Þ ¼ ;;

if the boundaries of their dominance spaces do not intersect,

Feature 2

320

Feature 2

319

\

355

6

8

Fig. 6. An example of a common object for the two fronts.

Please cite this article in press as: Stenina, M. M., et al. Ordinal classification using Pareto fronts. Expert Systems with Applications (2015), http://dx.doi.org/ 10.1016/j.eswa.2015.03.021

394 395

396 398

ESWA 9932

No. of Pages 7, Model 5G

7 April 2015 6 399 400 401 402 403

404 406 407

408 410

M.M. Stenina et al. / Expert Systems with Applications xxx (2015) xxx–xxx

where q is the distance function (5). The assumption f 2;3 ðxÞ ¼ 1 holds in one of two possible cases. (a) 9t 2 POFp ð3Þ : t p x. The object t does not belong to POFp ð2Þ, because POFp ð2Þ does not dominate x. Then, it follows that

9t0 2 POFp ð2Þ :

t0 p t:

Therefore, we obtain a chain of domination inequalities,

t0 p t p x;

3. The separable sample construction procedure involves the similar comparison of all object pairs between the different classes, that also costs Oðm2 nÞ operations for the balanced-size classes. The iterative procedure of the defective objects elimination is also quadratic due to the memorization of all dominant elements for each object. However, this memorization makes the memory costs also quadratic. 4. In the ordinal-class case the total complexity is multiplied by the number of classes, so that the final complexity estimation is Oðm2 nKÞ, where K is the number of classes.

464 465 466 467 468 469 470 471 472 473 474

411 412 413 414 415 416 417 418 419 420 421 422 423 424

425 427 428 429

430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 446 445

and it follows that t0 p x. This contradicts the assumption that front POFp ð2Þ does not dominate object x. (b) Object x is not dominated by fronts POFn ð2Þ and POFp ð3Þ. In this case, object x is not dominated by any front POFn ðuÞ; POFp ðu þ 1Þ with u ¼ 1; 2. Note that there exists an object y1 2 POFn ð1Þ whose dominance space boundary contains the point y0 , where y0 is the nearest point to x in the sense of the Hamming distance (5). Object y1 can belong to front POFn ð2Þ. In this case, the distance between x and POFn ð2Þ is less than the distance between x and POFn ð1Þ. (Here, the distance between a point and a front means the distance between a point and the nearest point of a front.) If object y1 does not belong to front POFn ð2Þ, there exists an object

y2 2 POFn ð2Þ :

y2 n y1 :

However, y2 ¤n x, because the object x is not dominated by any front. Hence, it follows that the distance between object x and front POFn ð2Þ is less than the distance between object x and point y0 on front POFn ð1Þ. The proof for the pair of fronts POFp ð2Þ; POFp ð3Þ is similar. There exists an object w1 2 POFp ð2Þ such that the boundary of its dominance space contains a point w0 , where w0 is the nearest point to x in the sense of the Hamming distance (5). The distance between object x and front POFp ð3Þ is not less than the distance between x and w0 2 POFp ð2Þ. We have proved that the distance between x and POFn ð2Þ is not greater than the distance between x and POFn ð1Þ, and the distance between x and POFp ð3Þ is not less than the distance between x and POFp ð2Þ. From f 1;2 ðxÞ ¼ 0, it follows that the distance between x and POFn ð1Þ is less than the distance between x and POFp ð2Þ. Thus, x is nearer to POFn ð2Þ than to POFp ð3Þ. This contradicts the assumption that f 2;3 ðxÞ ¼ 1, and concludes the proof. h

447

451

Given that the method of Pareto fronts uses only separable samples, all fronts are disjoint. Therefore the ordinal classifier (6) is admissible and the transitivity condition (7) holds for any classified object.

452

4.4. Computational cost

453

To estimate the computational cost, we must compute the cost for the basic stages of the algorithm: the two-object comparison, the Pareto front construction, the separable sample construction, and the ordinal-class case.

448 449 450

454 455 456 457 458 459 460 461 462 463

1. The cost of two object comparison is OðnÞ, where n is the number of features. 2. The Pareto front construction method involves comparison of the all pairs of objects. For the balanced-size classes OðmÞ the comparison is Oðm2 nÞ, where m is the total number of objects. After the pairwise comparison the method finds the non-dominated objects using the Oðm2 Þ operations.

We see that the Pareto front construction procedure is quite costly, and one of our further directions is to reduce the cost. The possible solution is to use the class transitivity property to construct the Pareto fronts for all classes together to eliminate the multiplier K from the complexity estimation.

475

5. Experimental results

480

5.1. Benchmark datasets

481

We verify the proposed method on the different benchmark datasets. To test the method on the ordinal data, we make an additional monotone feature transformation for each dataset. We compare the method with two ordinal classification algorithms and one method of classification with monotonicity constraints. We used the following datasets from the UCI repository: Pyrimidines, Machine CPU, Housing, Computer Activity, Abalone and Car. All datasets except the last one correspond to the regression problem, so that we made a discretization of the target variable into five levels containing equal number of objects. The experiment scheme duplicates the one from Chu and Sathiya Keerthi (2007). We randomly partitioned each dataset into the training and test splits, as it shown in Table 6. The partition was repeated 100 times independently. To measure the quality we estimated the mean zero-one loss and the mean absolute loss on the test datasets. For comparison we used two classification algorithms, the decision tree J48 (Trees) and the Support Vector Machine with the linear kernel (SVM), that were combined with the ordinal classification scheme from Frank and Hall (2001). Furthermore, we used the nearest-neighbor classification method with monotonicity (KNN) from Duivesteijn and Feelders (2008). The results for the five original datasets (all except ‘‘Cars’’) are given in Table 7. The bolded numbers indicate whether the corresponding method was stat significantly better than the others. We see that the ordinal SVM outperformed all other methods due to the linear nature of features in the considered datasets. To investigate the method properties on the ordinal-scaled data, we made the ordinal transformation of the dataset features. As for the target variable, we discretized all features into five levels containing equal number of objects. The results for the transformed datasets (and for the ‘‘Cars’’ whose features were initially ordinal) are given in Table 8. The

482

Table 6 Description of the datasets. Dataset

#FEATURES

#OBJECTS

Training/Test

Pyrimidines MachineCPU Boston Computer Abalone Cars RedBook

27 6 13 21 8 6 101

74 209 506 8182 4177 1728 102

50/24 150/59 300/206 4000/4182 1000/3177 1000/728 100/1

Please cite this article in press as: Stenina, M. M., et al. Ordinal classification using Pareto fronts. Expert Systems with Applications (2015), http://dx.doi.org/ 10.1016/j.eswa.2015.03.021

476 477 478 479

483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514

ESWA 9932

No. of Pages 7, Model 5G

7 April 2015 M.M. Stenina et al. / Expert Systems with Applications xxx (2015) xxx–xxx Table 7 Experimental results: ordinal target, linear features. Dataset

Pyrimidines MachineCPU Boston Computer Abalone

Mean zero-one loss (0:01)

Mean absolute loss (0:01)

SVM

POF

Trees

KNN

SVM

POF

Trees

KNN

0.50 0.44 0.38 0.32 0.53

0.62 0.44 0.48 0.71 0.59

0.61 0.47 0.41 0.38 0.57

0.55 0.51 0.47 0.60 0.60

0.64 0.53 0.46 0.35 0.78

0.90 0.53 0.65 1.36 0.92

0.84 0.53 0.47 0.41 0.77

0.75 0.61 0.62 0.90 0.88

Table 8 Experimental results: ordinal target, ordinal features. Dataset

Pyrimidines MachineCPU Boston Computer Abalone Cars RedBook

Mean zero-one loss (0:01)

Mean absolute loss (0:01)

SVM

POF

Trees

KNN

SVM

POF

Trees

KNN

0.57 0.51 0.40 0.44 0.78 0.19 0.66

0.58 0.39 0.48 0.69 0.59 0.19 0.47

0.60 0.47 0.40 0.41 0.58 0.08 0.48

0.61 0.43 0.41 0.45 0.59 0.06 0.62

0.71 0.65 0.49 0.53 1.78 0.24 0.85

0.77 0.45 0.68 1.38 0.92 0.26 0.60

0.79 0.56 0.46 0.45 0.76 0.08 0.52

0.76 0.51 0.51 0.55 0.89 0.07 0.79

7

6. Summary and further research

532

We have proposed the ordinal classification method based on the construction of Pareto fronts, which describe boundaries between ordinal classes. The ordinal classifier is constructed as a superposition of two-class Pareto classifiers. Our algorithm was compared with some well-known algorithms, and demonstrated adequate results. It was used to solve the IUCN Red List categorization problem. The main advantage of the method is the simplicity and interpretability of the obtained models. At the same time, the computational experiments showed that the obtained models allow to get the comparable prediction quality with the state-of-the art ordinal classification methods. Further investigations will be devoted to extending the scope of the proposed algorithm. The classification method will be extended to the case of partial orders defined over the set of features and over the set of feature values. From a practical point of view, the partial-order case corresponds to incomplete information given by the experts. To describe the partial orders, we will use the idea of a partial order cone proposed in Kuznetsov and Strijov (2014). Another way to improve the proposed classification algorithm is to take into account the expert information about feature preferences. The preference relation defined over the set of features allows to restrict the set of admissible models and to achieve better prediction quality.

533

References

557

Ben-David, Arie, Sterling, Leon, & Pao, Yoh-Han (1989). Learning and classification of monotonic ordinal concepts. Computational Intelligence, 5(1), 45–49. Chu, Wei, & Sathiya Keerthi, S. (2007). Support vector ordinal regression. Neural Computing, 19(3), 792–815. Duivesteijn, Wouter, & Feelders, Ad (2008). Nearest neighbour classification with monotonicity constraints. In Walter Daelemans, Bart Goethals, & Katharina Morik (Eds.), Machine learning and knowledge discovery in databases. Lecture notes in computer science (Vol. 5211, pp. 301–316). Berlin Heidelberg: Springer. ISBN 978-3-540-87478-2. Frank, Eibe, & Hall, Mark (2001). A simple approach to ordinal classification. Springer. Freund, Yoav, Iyer, Raj, Schapire, Robert E., & Singer, Yoram (2003). An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4, 933–969. ISSN 1532-4435. Furnkranz, Johannes, & Hullermeier, Eyke (2003). Pairwise preference learning and ranking. Machine Learning ECML, 2003, 145–156. Har-Peled, S., Roth, D., & Zimak, D. (2003). Constraint classification for multiclass classification and ranking. In NIPS (pp. 785–792). Kotlowski, Wojciech, & Slowinski, Roman (2009). Rule learning with monotonicity constraints. In Proceedings of the 26th annual international conference on machine learning. ICML ’09 (pp. 537–544). New York, NY, USA: Dover. ISBN 978-1-60558516-1. Kuznetsov, M. P., & Strijov, V. V. (2014). Methods of expert estimations concordance for integral quality estimation. Expert Systems with Applications, 41(4), 1988–1996. . Nogin, V. D. (2003). The edgeworth-pareto principle and relative importance of criteria in the case of a fuzzy preference relation. Computational Mathematics and Mathematical Physics, 43(11), 1666–1676. Strijov, Vadim, Granic, Goran, Juric, Jeljko, Jelavic, Branka, & Maricic, Sandra Antecevic (2011). Integral indicator of ecological impact of the croatian thermal power plants. Energy, 36(7), 4144–4149. Xia, Fen, Liu, Tie-Yan, Wang, Jue, Zhang, Wensheng, & Li, Hang (2008). Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on machine learning. ICML ’08 (pp. 1192–1199). New York, NY, USA: ACM. ISBN 978-1-60558-205-4. Yue, Yisong, Finley, Thomas, Radlinski, Filip, & Joachims, Thorsten (2007). A support vector method for optimizing average precision. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. SIGIR ’07 (pp. 271–278). New York, NY, USA: ACM. ISBN 978-1-59593-597-7.

558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596

534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556

3

2

1

1

2

3

Fig. 7. Comparison of the computed and expert estimated categories.

519

SVM method is significantly worse due to the linearity of kernel, while the ‘‘Trees’’ and the ‘‘POF’’ methods demonstrate sufficiently good results. Another interesting observation is that the Trees method outperforms the other methods according to the mean absolute loss.

520

5.2. IUCN Red List dataset

521

The results for the IUCN Red List categorization are shown in Table 8 in the last raw. Unlike the other datasets, we used the leave-one-out partition scheme. The POF and the Trees methods demonstrate the best results according to the zero-one loss. Fig. 7 compares the categories computed by the POF algorithm using the leave-one-out results. The x-axis denotes the computed categories, and the y-axis represents the categories determined by the experts. The radius of each point is proportional to the number of objects with the corresponding computed and expert categories. For a significant number of objects, the computed category was the same as that assigned by the experts.

515 516 517 518

522 523 524 525 526 527 528 529 530 531

Please cite this article in press as: Stenina, M. M., et al. Ordinal classification using Pareto fronts. Expert Systems with Applications (2015), http://dx.doi.org/ 10.1016/j.eswa.2015.03.021

597