- Email: [email protected]
Appendix E Mathematical Results
APPENDIX
E Mathematical Results
E.l. Lemma E.1.1 (A
MATRIX RESULTS
(Matrix inversion lemma)
+ BC)-l =
A-I - A-IB(I
Proof See Problem 5.2.
Remark £.1.1
+ CA-IBrICA- I
(E.Ll )
V
We note that, if B is a column (= g, say) and C is a row
(= hT, say), then Lemma E.Ll reduces to (A-+ ghTr l =
T
fI - A-I (1 + hTA-Ig) gh JA- I
(E.l.2)
The implication of Eq, (E.l.2) is that the inversion of a matrix plus a diad reduces to a simple scalar division provided the inverse of the original matrix is known. (See also the solution to Problem 5.2). V
Result E.l.l
If A is partitioned as A =
fAll A 2l
A l2 A 22
]
(E.1.3) 241
E.
242
MATHEMATICAL RESULTS
where A, A 11, and A 22 are all square and nonsingular, then
A- I = (A11 - A 12A 2"lA2.) - 1 i -(A 11 - A12A2"lA21fIA12AilJ --.------------------.=.-1-------=--1-- -----=.- L- -f----------.. . --------------:.-1- -----":-1----- ---(A 22 - A 2IA 11 A 12) f - (A 22 - A 21A11 A 1 2 ) A 21A11 ! (E.l.4)
Proof Simply show AA - I = 1 (exercise).
\1
Result E.l.2 If A is as defined in (E.1.3), then det A = det A 22 ' det(A 11 - A12AilA2.)
== det A 11 . det(A 22
(E.l.5) (E.l.6)
A 2IAi" A 12)
-
Proof A Il
fA 21
A12jf 1 A 22 -AilA21
0 j=f(A l l - A I2Ail A2.) Ai21 0
A 12Ailj 1
Therefore det A . det Ail = det(A ll - A 12 Ail A 21 ). Equation (E.l.5) follows since det Ail = l/(det A 22}. Similarly, Eq. (El.6) may be established. \1 Result E.l.3 det(I
+ BC) = det(I + CB)
(E.1.7)
Proof Consider the matrix A =
f~ ~ J
The result follows from Result El.2.
\1
Result E.l.4
(I
+ BC}-IB =
B(I
+ csv :
(E.1.8)
Proof From Lemma ELl, (I
+ BC}-IB =
+ CBflCB = + CBfl(1 + CB -
B - B(I
= B(I
+ CBfICB) CB) = B(I + CBfl
B(I - (I
\1
E.2.
VECTOR AND MATRIX DIFFERENTIATION RESULTS
243
E.2. VECTOR AND MATRIX DIFFERENTIATION RESULTS The following conventions are adopted throughout the book:
°
(1) a4J/ao, where 4J is a scalar and a vector, denotes a row vector with ith element a4J/aoi, Oi being the ith component of 0. (2) ap/ao, where and f3 are vectors, denotes a matrix with ijth element ap;/aO j where Pi' OJ are the ith and jth elements of P and 0, respectively. . (3) a4J/aM, where 4J is a scalar and M is a matrix, denotes a matrix with ijth element a4J/aMij where Mij is the ijth element of M.
°
We now have the following results. (See also [70].) Result E.2.1
a
aM log det M = M- T
(E.2.1 )
where M is any square nonsingular matrix.
Proof M- 1
= adj M/det M
Therefore M(adj M) = (det M)I. Taking log of iith element gives log det M = log( ~ Mik(adj M)ki) or
alog det M
(adj
M)ji
aMi}
(since (adj M)ki does not depend upon M ij )
= (adj M)ji = M- T det M Result E.2.2
(E.2.2)
244
E.
MATHEMATICAL RESULTS
Proof
Hence
Hence (where e, is a null vector save for 1 in the ith position) Hence
e
-;l-
uM u
trace WM- 1
= trace
rW-;l-OM-l~
uM u
=
-trace[WM- 1eie/M- 1]
-- -ej TM- 1WM- 1 ei
--
_
e,T[M- 1WM- 1]Tej
Hence
Result E.2.3 If M is a positive definite matrix with unit diagonal (M ii = 1, i = 1, ... , n), then det M achieves its maximum value for M = I. Proof For all positive x we have the following inequality:
log x < x - I '
with equality iff x = 1
Ii'=
1 Ai' where Al ... An are the eigenvalues of M. Now log det M = Hence using the above inequality.
log det M ::; trace M - n with equality iff Al = A2 = ... = An' Hence log det M achieves its maximum value for M = I. 'V E.3. CARATHEODORY'S THEOREM Throughout this section S denotes a subset of R". Definition £.3.1 The convex hull of the set S is the set rt of all convex combinations of elements of S, i.e.• (t
=
J\clc =
i
k=1
AiSj, 0::; Ai::; 1,
i
k=1
Ai = 1, Si
E
S\ )
E.4.
INEQUALITIES
245
or, more generally,
= lCIC = JSS dA(S), dA(S)
'6
~ 0, J.dA(S) = 1, S E s)
An alternative definition is that '6 is the smallest convex set containing S.
\7
We now have the following theorem due to Caratheodory [210]: Theorem E.3.1 Let S be any set in R" and '6 its convex hull. Then if and only if c can be expressed as a convex combination of n + 1 elements of S, i.e., if and only if
CE ct
"+1 C
=
L AjS
j=
[210].
Proof See
j ,
1
\7
Theorem E.3.2 If S is closed and bounded, then its convex hull '6 is closed and bounded.
[210].
Proof See
\7
Definition £.3.2 Aface of a convex set '6 is a convex subset ct', such that every line segment in '6 with interior point E '6' also has both endpoints in «. \7 '6.
Definition £.3.3 \7
Theorem E.3.3 extreme points.
An extreme point of "6. is a zero-dimensional face of A closed bounded convex set is the convex hull of its
[210].
Proof See
\7
Theorem E.3.4 Any boundary point of a closed bounded convex set '6 can be expressed as a convex combination of n or less extreme points of '6. Proof See
[210].
\7
EA. Result EA.1
If M
Proof See [156].
INEQUALITIES
°
= rxM 1 + (1 - rx)M 2' < a < 1, then det M > det MI' det M~I -.) \7
E Mathematical Results
E.l. Lemma E.1.1 (A
MATRIX RESULTS
(Matrix inversion lemma)
+ BC)-l =
A-I - A-IB(I
Proof See Problem 5.2.
Remark £.1.1
+ CA-IBrICA- I
(E.Ll )
V
We note that, if B is a column (= g, say) and C is a row
(= hT, say), then Lemma E.Ll reduces to (A-+ ghTr l =
T
fI - A-I (1 + hTA-Ig) gh JA- I
(E.l.2)
The implication of Eq, (E.l.2) is that the inversion of a matrix plus a diad reduces to a simple scalar division provided the inverse of the original matrix is known. (See also the solution to Problem 5.2). V
Result E.l.l
If A is partitioned as A =
fAll A 2l
A l2 A 22
]
(E.1.3) 241
E.
242
MATHEMATICAL RESULTS
where A, A 11, and A 22 are all square and nonsingular, then
A- I = (A11 - A 12A 2"lA2.) - 1 i -(A 11 - A12A2"lA21fIA12AilJ --.------------------.=.-1-------=--1-- -----=.- L- -f----------.. . --------------:.-1- -----":-1----- ---(A 22 - A 2IA 11 A 12) f - (A 22 - A 21A11 A 1 2 ) A 21A11 ! (E.l.4)
Proof Simply show AA - I = 1 (exercise).
\1
Result E.l.2 If A is as defined in (E.1.3), then det A = det A 22 ' det(A 11 - A12AilA2.)
== det A 11 . det(A 22
(E.l.5) (E.l.6)
A 2IAi" A 12)
-
Proof A Il
fA 21
A12jf 1 A 22 -AilA21
0 j=f(A l l - A I2Ail A2.) Ai21 0
A 12Ailj 1
Therefore det A . det Ail = det(A ll - A 12 Ail A 21 ). Equation (E.l.5) follows since det Ail = l/(det A 22}. Similarly, Eq. (El.6) may be established. \1 Result E.l.3 det(I
+ BC) = det(I + CB)
(E.1.7)
Proof Consider the matrix A =
f~ ~ J
The result follows from Result El.2.
\1
Result E.l.4
(I
+ BC}-IB =
B(I
+ csv :
(E.1.8)
Proof From Lemma ELl, (I
+ BC}-IB =
+ CBflCB = + CBfl(1 + CB -
B - B(I
= B(I
+ CBfICB) CB) = B(I + CBfl
B(I - (I
\1
E.2.
VECTOR AND MATRIX DIFFERENTIATION RESULTS
243
E.2. VECTOR AND MATRIX DIFFERENTIATION RESULTS The following conventions are adopted throughout the book:
°
(1) a4J/ao, where 4J is a scalar and a vector, denotes a row vector with ith element a4J/aoi, Oi being the ith component of 0. (2) ap/ao, where and f3 are vectors, denotes a matrix with ijth element ap;/aO j where Pi' OJ are the ith and jth elements of P and 0, respectively. . (3) a4J/aM, where 4J is a scalar and M is a matrix, denotes a matrix with ijth element a4J/aMij where Mij is the ijth element of M.
°
We now have the following results. (See also [70].) Result E.2.1
a
aM log det M = M- T
(E.2.1 )
where M is any square nonsingular matrix.
Proof M- 1
= adj M/det M
Therefore M(adj M) = (det M)I. Taking log of iith element gives log det M = log( ~ Mik(adj M)ki) or
alog det M
(adj
M)ji
aMi}
(since (adj M)ki does not depend upon M ij )
= (adj M)ji = M- T det M Result E.2.2
(E.2.2)
244
E.
MATHEMATICAL RESULTS
Proof
Hence
Hence (where e, is a null vector save for 1 in the ith position) Hence
e
-;l-
uM u
trace WM- 1
= trace
rW-;l-OM-l~
uM u
=
-trace[WM- 1eie/M- 1]
-- -ej TM- 1WM- 1 ei
--
_
e,T[M- 1WM- 1]Tej
Hence
Result E.2.3 If M is a positive definite matrix with unit diagonal (M ii = 1, i = 1, ... , n), then det M achieves its maximum value for M = I. Proof For all positive x we have the following inequality:
log x < x - I '
with equality iff x = 1
Ii'=
1 Ai' where Al ... An are the eigenvalues of M. Now log det M = Hence using the above inequality.
log det M ::; trace M - n with equality iff Al = A2 = ... = An' Hence log det M achieves its maximum value for M = I. 'V E.3. CARATHEODORY'S THEOREM Throughout this section S denotes a subset of R". Definition £.3.1 The convex hull of the set S is the set rt of all convex combinations of elements of S, i.e.• (t
=
J\clc =
i
k=1
AiSj, 0::; Ai::; 1,
i
k=1
Ai = 1, Si
E
S\ )
E.4.
INEQUALITIES
245
or, more generally,
= lCIC = JSS dA(S), dA(S)
'6
~ 0, J.dA(S) = 1, S E s)
An alternative definition is that '6 is the smallest convex set containing S.
\7
We now have the following theorem due to Caratheodory [210]: Theorem E.3.1 Let S be any set in R" and '6 its convex hull. Then if and only if c can be expressed as a convex combination of n + 1 elements of S, i.e., if and only if
CE ct
"+1 C
=
L AjS
j=
[210].
Proof See
j ,
1
\7
Theorem E.3.2 If S is closed and bounded, then its convex hull '6 is closed and bounded.
[210].
Proof See
\7
Definition £.3.2 Aface of a convex set '6 is a convex subset ct', such that every line segment in '6 with interior point E '6' also has both endpoints in «. \7 '6.
Definition £.3.3 \7
Theorem E.3.3 extreme points.
An extreme point of "6. is a zero-dimensional face of A closed bounded convex set is the convex hull of its
[210].
Proof See
\7
Theorem E.3.4 Any boundary point of a closed bounded convex set '6 can be expressed as a convex combination of n or less extreme points of '6. Proof See
[210].
\7
EA. Result EA.1
If M
Proof See [156].
INEQUALITIES
°
= rxM 1 + (1 - rx)M 2' < a < 1, then det M > det MI' det M~I -.) \7