- Email: [email protected]
A method for solving the n-dimensional convex hull problem
Pattern Recognition Letters 2 (1983) 23-25 North-Holland
October 1983
A method for solving the n-dimensional convex hull problem Adam JOZWIK Institute of Biocybernetics and Biomedical Engineering, Polish Academy of Sciences, 00-818 Warsaw, KRN 55, Poland Received 21 March 1983
Abstract: A method is presented for finding all vertices and all hyperplanes containing the faces of a convex polyhedron spanned by a given finite set X in Euclidean space E n. The present paper indicates how this method can be applied to the investigation of linear separability of two given finite sets X 1 and X 2 in E n. In the case of linear separability of these sets the proposed method makes it possible to find the separating hyperplane.
Key words: Convex hull, linear separability, learning algorithm, linear classifier.
1. Introduction
BEGIN
AA+--{(a,a) c A : (a, xi)+ a_<0}; T h e c o n v e x hull C o ( X ) o f a finite set X in E n is a p o l y h e d r o n . It can be d e t e r m i n e d by a set o f vertices V as well as by the system o f linear inequalities o f the f o r m (ai, x ) + a i > _ O ,
i = 1 , 2 . . . . . l,
(1)
where 1 d e n o t e s the n u m b e r o f faces o f the p o l y hedron. O u r aim is to find the set V a n d the set A = {(ai, ai)}l=l for the given finite set X in E n. F o r c o n v e n i e n c e , we a s s u m e that dim X ± =0, X = {xi}m=l a n d the set {xi-xl}'/_+_l 1 f o r m s a basis in E n"
A / ' - - t h e n u m b e r o f elements in A A ; IF A 1 > 0 THEN BEGIN
o r d e r the elements o f A in such a w a y that A A =
A \ {(aj, aAb=/-At~ ; s'--0; C~0;
BS '--f; FORj=I--AI+I
Toj=I
BY STEPS OF 1 DO
BEGIN
C,--CU {x~ v: (aj, x)+ ~j =0}; FORk=I
TO k = I - - A I B Y S T E P S O F 1 DO
BEGIN
B ' - - { x ~ V: (aj, x ) + a j = 0
2. Description of the method
a n d ( a k, x ) + a~ = 0}; IF B c o n t a i n s n - 1 p o i n t s THEN BEGIN
(i) l,--n + 1. (ii) V,-- {xi}~=,. (iii) A,-- {(ai, ai)}~= 1 (each h y p e r p l a n e (a i, x) + cti = 0 c o n t a i n s a d i f f e r e n t c o m b i n a t i o n o f n p o i n t s in V, the v e c t o r a i indicates a h a l f s p a c e t h a t contains the r e m a i n i n g p o i n t o f V). (iv) FOR i = n + 2 TO i = m BY STEPS OF 1 DO
s ° s + 1; Bs,--B; BS~BSOB END END END
A~-A\AA
0167-8655/83/$3.00 © 1983, Elsevier Science Publishers B.V. (North-Holland)
23
Volume 2, Number 1
PATTERN R E C O G N I T I O N LETTERS
(delete the last AI elements of A);
linear inequalities
AA*--O;
((a,a),y)>-O
F O R j = I TO j=SBYSTEPSOF 1 DO ( a , a ) ' - - a n arbitrary pair such that O=/:a~E n, (a,x)+a=O VxeBsU{xi} =D and (a, xo)+a>O for Xoe V \ D (as a vector x0 we could also take the mean vector of X); AA*--AA U {(a, a)}; END;
l*--l-Al+s;
V*-(V\C)UBSU{xi}; A*-AUAA END END,
3. The linear separability problem Let XI and )(2 be two finite sets XI and X2 in
E n. We need to check whether these sets are linearly separable. For this reason we find the sets Vi and A i for the sets Xi, i = 1, 2, respectively. The sets X 1 and X 2 are not linearly separable if and only if
(a, oo)+a>O
or
ffvo~Vz V(a,a)6Al
Vy~ Y=h(VIUV2),
where
BEGIN
~oo~V1 V(a,a)eA2
October 1983
(a, oo)+a>O.
These sets are strictly linearly separable if and only if
VoEV1,.7(a,a)eA 2 ( a , o ) + a < 0
~'(x, 1)
Y=h(x)=(-(x,l)
if x e V1, if x ~ V2,
(a,~)~E n+l and ( x , l ) ~ E ~+1, i.e. the pairs (a,a) and (x, 1) are considered as vectors in E n+ 1. Thus, the hyperplane (a, x) + a = 0 in E n corresponds to the hyperplane ((a, a), y ) = 0 in E n+l passing through the origin of the coordinate system. Let us assume that d i m Y ± = 0 , Y={yi}m=l, {yi}'~+lI forms a basis in E ~+l and 0 ~ C o ( Y ) (XI and X2 linearly separable). Furthermore, let us assume that for each vector ( a , a ) e A the hyperplane ((a, a), y) = 0 contains a different face of the convex cone K ( V ) spanned by V, where A and V mean sets that appear in the modified procedure. The modified procedure is derived from the primary one by replacing: 1. all symbols x by y (X by Y) 2. all expressions (a,x)+a,
(a, xi)+a,
(aj, x)+czj, (a, xo)+a
by
((a, a), y), ((a, a), Yi), ((aj, aj), y), ((a, a), Y0) respectively. If V and A are the sets obtained as a result of the modified procedure then each hyperplane (a, x ) + a = 0 separates the sets X 1 and X 2, where (a, ~) e A. Furthermore, any hyperplane separating the sets X1 and X2 can be expressed in the form
(a., x) + a . = 0,
(2)
or
V o ~ V 2 J ( a , a ) 6 A 1 (a, o ) + a < 0 .
where /
In the remaining case the considered sets are linearly separable (but not strictly). We have assumed that X 1NX2=0. If the sets X l and X2 are linearly separable then we may be interested in finding the separating hyperplane. To solve this problem we can apply the modification below of the procedure presented in Section 2. The problem of the linear separability of the sets X 1 and X2 is equivalent to solving the system of 24
(a.,a,)= ~ J.i(ai, o:i), i
J.i>_O,
1
and A = {(ai, a/)}~= I. Let AXi be a set of all points in X1UX2 covered by the hyperplane (ai, x)+o~i=O and ( a i , ~ i ) E A . Then the hyperplane
( Aia i + ~.jaj, x) + 2i(~i -+-2j(~j = 0 covers only A X i O A X j. Thus, if X1 and X2 are strictly linearly separable and the all Ai are positive
Volume 2, Number 1
PATTERN RECOGNITION LETTERS
then the hyperplane defined by (2) strictly separates these sets. The set V X = { x ~ . X 1U X 2 : y = h(x) ~ V}
is the set of all points in XIt3X2 each of which contains a separating hyperplane. We can note that the sets X I and X 2 are strictly linearly separable if and only if the sets V X l and V X 2 are strictly linearly separable.
4. Concluding remarks The proposed method is very convenient when it is necessary to separate some pairs of sets Xi and X j , i ¢ j , especially in the case of a multiclass problem, i.e. if we are given a certain number R of the sets X s, s = 1, 2 . . . . . R, and R > 2. Sometimes we need to find V and A for the set X = X 1 t3X2 when V~and A i , i = 1, 2, for Xl and X 2 are already found. Thus we can start with V1 and A~ and continue the computations considering the points from V1t3 1/2 instead of all points of X. Furthermore, the presented algorithms substantially accelerate the procedures proposed by Kosinetz (1964) or by Wapnik and Tschervonenkis
October 1983
(1974), when applied to finding the optimum separating hyperplane (the distance between the set X ~ U X 2 and the separating hyperplane is maximum). The sets VX~ and V X 2 can then be used instead of X t and X 2. The above is especially pronounced in difficult cases, i.e. if the distance between Co(XI) and Co(X 2) is small. The procedure of Section 2 can also be developed for solving the general system of linear inequalities of the form (1) as well as linear programming problems. Thus, we can also solve the linearly inseparable case of the two sets XI and X2 by minimizing the perceptron criterion function (see Duda and Hart (1973)). Linear programming and linear inequalities problems as well as the minimizing of the perceptron criterion function will be subjects of separate papers.
References Kosinetz, B.N. (1964), About a learning algorithm for a linear machine (in Russian). In: Computing Technique and Programming, Vol. 3, LGU, Leningrad, pp. 80-83. Wapnik, W.N. and Tschervonenkis, A.I. (1974), Pattern Recognition Theory (in Russian). Nauka, Moscow, pp. 322-328. Duda, R.D. and P.E. Hart (1973), Pattern Classification and Scene Analysis, Wiley, New York, pp. 166-171.
25
October 1983
A method for solving the n-dimensional convex hull problem Adam JOZWIK Institute of Biocybernetics and Biomedical Engineering, Polish Academy of Sciences, 00-818 Warsaw, KRN 55, Poland Received 21 March 1983
Abstract: A method is presented for finding all vertices and all hyperplanes containing the faces of a convex polyhedron spanned by a given finite set X in Euclidean space E n. The present paper indicates how this method can be applied to the investigation of linear separability of two given finite sets X 1 and X 2 in E n. In the case of linear separability of these sets the proposed method makes it possible to find the separating hyperplane.
Key words: Convex hull, linear separability, learning algorithm, linear classifier.
1. Introduction
BEGIN
AA+--{(a,a) c A : (a, xi)+ a_<0}; T h e c o n v e x hull C o ( X ) o f a finite set X in E n is a p o l y h e d r o n . It can be d e t e r m i n e d by a set o f vertices V as well as by the system o f linear inequalities o f the f o r m (ai, x ) + a i > _ O ,
i = 1 , 2 . . . . . l,
(1)
where 1 d e n o t e s the n u m b e r o f faces o f the p o l y hedron. O u r aim is to find the set V a n d the set A = {(ai, ai)}l=l for the given finite set X in E n. F o r c o n v e n i e n c e , we a s s u m e that dim X ± =0, X = {xi}m=l a n d the set {xi-xl}'/_+_l 1 f o r m s a basis in E n"
A / ' - - t h e n u m b e r o f elements in A A ; IF A 1 > 0 THEN BEGIN
o r d e r the elements o f A in such a w a y that A A =
A \ {(aj, aAb=/-At~ ; s'--0; C~0;
BS '--f; FORj=I--AI+I
Toj=I
BY STEPS OF 1 DO
BEGIN
C,--CU {x~ v: (aj, x)+ ~j =0}; FORk=I
TO k = I - - A I B Y S T E P S O F 1 DO
BEGIN
B ' - - { x ~ V: (aj, x ) + a j = 0
2. Description of the method
a n d ( a k, x ) + a~ = 0}; IF B c o n t a i n s n - 1 p o i n t s THEN BEGIN
(i) l,--n + 1. (ii) V,-- {xi}~=,. (iii) A,-- {(ai, ai)}~= 1 (each h y p e r p l a n e (a i, x) + cti = 0 c o n t a i n s a d i f f e r e n t c o m b i n a t i o n o f n p o i n t s in V, the v e c t o r a i indicates a h a l f s p a c e t h a t contains the r e m a i n i n g p o i n t o f V). (iv) FOR i = n + 2 TO i = m BY STEPS OF 1 DO
s ° s + 1; Bs,--B; BS~BSOB END END END
A~-A\AA
0167-8655/83/$3.00 © 1983, Elsevier Science Publishers B.V. (North-Holland)
23
Volume 2, Number 1
PATTERN R E C O G N I T I O N LETTERS
(delete the last AI elements of A);
linear inequalities
AA*--O;
((a,a),y)>-O
F O R j = I TO j=SBYSTEPSOF 1 DO ( a , a ) ' - - a n arbitrary pair such that O=/:a~E n, (a,x)+a=O VxeBsU{xi} =D and (a, xo)+a>O for Xoe V \ D (as a vector x0 we could also take the mean vector of X); AA*--AA U {(a, a)}; END;
l*--l-Al+s;
V*-(V\C)UBSU{xi}; A*-AUAA END END,
3. The linear separability problem Let XI and )(2 be two finite sets XI and X2 in
E n. We need to check whether these sets are linearly separable. For this reason we find the sets Vi and A i for the sets Xi, i = 1, 2, respectively. The sets X 1 and X 2 are not linearly separable if and only if
(a, oo)+a>O
or
ffvo~Vz V(a,a)6Al
Vy~ Y=h(VIUV2),
where
BEGIN
~oo~V1 V(a,a)eA2
October 1983
(a, oo)+a>O.
These sets are strictly linearly separable if and only if
VoEV1,.7(a,a)eA 2 ( a , o ) + a < 0
~'(x, 1)
Y=h(x)=(-(x,l)
if x e V1, if x ~ V2,
(a,~)~E n+l and ( x , l ) ~ E ~+1, i.e. the pairs (a,a) and (x, 1) are considered as vectors in E n+ 1. Thus, the hyperplane (a, x) + a = 0 in E n corresponds to the hyperplane ((a, a), y ) = 0 in E n+l passing through the origin of the coordinate system. Let us assume that d i m Y ± = 0 , Y={yi}m=l, {yi}'~+lI forms a basis in E ~+l and 0 ~ C o ( Y ) (XI and X2 linearly separable). Furthermore, let us assume that for each vector ( a , a ) e A the hyperplane ((a, a), y) = 0 contains a different face of the convex cone K ( V ) spanned by V, where A and V mean sets that appear in the modified procedure. The modified procedure is derived from the primary one by replacing: 1. all symbols x by y (X by Y) 2. all expressions (a,x)+a,
(a, xi)+a,
(aj, x)+czj, (a, xo)+a
by
((a, a), y), ((a, a), Yi), ((aj, aj), y), ((a, a), Y0) respectively. If V and A are the sets obtained as a result of the modified procedure then each hyperplane (a, x ) + a = 0 separates the sets X 1 and X 2, where (a, ~) e A. Furthermore, any hyperplane separating the sets X1 and X2 can be expressed in the form
(a., x) + a . = 0,
(2)
or
V o ~ V 2 J ( a , a ) 6 A 1 (a, o ) + a < 0 .
where /
In the remaining case the considered sets are linearly separable (but not strictly). We have assumed that X 1NX2=0. If the sets X l and X2 are linearly separable then we may be interested in finding the separating hyperplane. To solve this problem we can apply the modification below of the procedure presented in Section 2. The problem of the linear separability of the sets X 1 and X2 is equivalent to solving the system of 24
(a.,a,)= ~ J.i(ai, o:i), i
J.i>_O,
1
and A = {(ai, a/)}~= I. Let AXi be a set of all points in X1UX2 covered by the hyperplane (ai, x)+o~i=O and ( a i , ~ i ) E A . Then the hyperplane
( Aia i + ~.jaj, x) + 2i(~i -+-2j(~j = 0 covers only A X i O A X j. Thus, if X1 and X2 are strictly linearly separable and the all Ai are positive
Volume 2, Number 1
PATTERN RECOGNITION LETTERS
then the hyperplane defined by (2) strictly separates these sets. The set V X = { x ~ . X 1U X 2 : y = h(x) ~ V}
is the set of all points in XIt3X2 each of which contains a separating hyperplane. We can note that the sets X I and X 2 are strictly linearly separable if and only if the sets V X l and V X 2 are strictly linearly separable.
4. Concluding remarks The proposed method is very convenient when it is necessary to separate some pairs of sets Xi and X j , i ¢ j , especially in the case of a multiclass problem, i.e. if we are given a certain number R of the sets X s, s = 1, 2 . . . . . R, and R > 2. Sometimes we need to find V and A for the set X = X 1 t3X2 when V~and A i , i = 1, 2, for Xl and X 2 are already found. Thus we can start with V1 and A~ and continue the computations considering the points from V1t3 1/2 instead of all points of X. Furthermore, the presented algorithms substantially accelerate the procedures proposed by Kosinetz (1964) or by Wapnik and Tschervonenkis
October 1983
(1974), when applied to finding the optimum separating hyperplane (the distance between the set X ~ U X 2 and the separating hyperplane is maximum). The sets VX~ and V X 2 can then be used instead of X t and X 2. The above is especially pronounced in difficult cases, i.e. if the distance between Co(XI) and Co(X 2) is small. The procedure of Section 2 can also be developed for solving the general system of linear inequalities of the form (1) as well as linear programming problems. Thus, we can also solve the linearly inseparable case of the two sets XI and X2 by minimizing the perceptron criterion function (see Duda and Hart (1973)). Linear programming and linear inequalities problems as well as the minimizing of the perceptron criterion function will be subjects of separate papers.
References Kosinetz, B.N. (1964), About a learning algorithm for a linear machine (in Russian). In: Computing Technique and Programming, Vol. 3, LGU, Leningrad, pp. 80-83. Wapnik, W.N. and Tschervonenkis, A.I. (1974), Pattern Recognition Theory (in Russian). Nauka, Moscow, pp. 322-328. Duda, R.D. and P.E. Hart (1973), Pattern Classification and Scene Analysis, Wiley, New York, pp. 166-171.
25