- Email: [email protected]
Refresher on Matrix Theory
Appendix A Refresher on Matrix Theory This appendix presents some matrix results that are used in the book.
A.l
Definitions and Some Basic Properties of Matrices
A real matrix A of dimension m x n (m-by-n) is defined as an
•••
ain
A =
(A.l.l) flml
where aij is a real number. Denote R"^^^ as all real m x n matrices, then we can express the matrix in ( A . l . l ) more compactly as ^ G M"^^". A real matrix with dimension n x 1 is called a real n-vector: Xi (A.1.2)
Similarly, this can also be expressed as x G M " . The basic matrix computations include the transposition:
A' =
fl^ii • • • a-mi (A.1.3) ain 329
APPENDIX
330
A. REFRESHER
ON MATRIX
THEORY
the addition of two matrices of the same dimension A-\-B=
I a n +611 :
•• • •..
I
•••
^ m l I O'pT^i
ain + bin : CLrnn
(A.1.4)
1 Omn
the multipUcation by a scalar a
aA=
\
aau :
• • • OiCiin • •. :
[ aami
(A.1.5)
' • • ocamn
and the multipHcation of two matrices A e '.
and B G ]
fc=l
AB =
{A.1.6) L
Zl (^mkhn
fc=l
A sqaure matrix has equal number of rows and columns. The inverse of a square non-singular matrix A is denoted as A~^ and satisfies A~^A = AA~^ = I. Here / is the identity matrix with diagnal elements being I's and off-diagonal elements O's. It is easy to show t h a t IA = AI = A. The trace of a square matrix A is the sum of its diagonal elements and is denoted as trA. Denote d e t ^ as the determinant of A and did]A as the adjugate of A, then A-'
=
did] A
(A.1.7)
detA
Let A and B be matrices with compatible dimensions and their inverses exist, we have (ABf = A^B^
(A.1.9)
and (AB)-^ =A''B-^
(A.1.10)
T h e m a t r i x inversion l e m m a . Let A, B, C and D be matrices with compatible dimensions and the inverses of A and C exist. Then {A + BCD)-^
= A-^ - A-^B{C-^
-h
DA'^By^DA-^
This can be proved by multiply the right hand side in (A. 11) by yl H- BCD, identity matrix.
(A.LU) which results in the
A.2.
EIGENVALUES AND EIGENVECTORS
A.2
331
Eigenvalues and Eigenvectors
Eigenvalues and eigenvectors. Let A be a square n x n matrix. The eigen values of A are denoted as Ai, A2, ..., A„ and they are the solutions to the characteristic equation det{A-XI)
(A.2.1)
The right eigenvector U corresponding to the eigenvalue A^ is the non-zero solution to Ati = XiU, or, {A - XiI)U = 0
(A.2.2)
Similarly, the corresponding left eigenvector Qi is the nonzero solution to qlA = X,qJ, or, qJ{A-\I)
=0
(A.2.2)
The eigen values A^ of the real matrix A has following properties: 1. The eigen values of the real matrix A are either real or in complex conjugate pairs. 2. The sum of the eigenvalues of A equals the trace of A n
iiA = Yl^i
(A.2.3)
i=l
3. The product of the eigenvalues of A is equals to its determinant n
det yl = J l Xi
(A.2.4)
1=1
4. The eigen values of a diagonal matrix A=
an
0
•••
^22
•• •
0
are equal to its diagonal elements an, 022, •••, cinn-
A.3
The Singular Value Decomposition and QR Factorization
Singular Value Decomposition (SVD) A matrix Q G W^"""^ is orthogonal if Q^Q = I.
332
APPENDIX A. REFRESHER ON MATRIX
If a matrix Q = [gi
Qm] €
THEORY
^"^, with Qi e R"^, then qi form an orthonormal basis for
The existence of the SVD is summarized in the next result. Theorem A.3.1 (Golub and van Loan, 1989). Let A e R'' matrices: and
n] G]
then there exists orthogonal
V^ = [^1 • • • t'n] ^ J
such that, 0 10
U^AV =
€ W^"""" p < min{m, n}
0 0
(A.3.1)
|0
where o^i > a2 > - - - > o-p > 0 and where some of the zero matrices may be empty. The SVD of a matrix allows to characterize a number of geometric properties of a matrix representing a linear transformation. These are the (co-)range and (co-)null space or (co-)kernel of a matrix and are defined as: Let Ae .
^, then we have: == {y e W^ly = Ax for some x G W} = dim(range(^)) = {x e R"" : Ax = 0} = kern(yl^) = range(^^)
range(^) rank(yi) kern(^) co-range(>l) co-kern(y4)
(A.3.2)
Given a matrix A, a basis for the different geometric spaces defined in Definition A.3.1 can reliably be derived from the SVD of A. Let the SVD of A e M'^''" be denoted as in Theorem A.3.1 and let p < n, then range(^) rank(^) keni{A) co-range(yl) co-kern(^)
= = = = =
spanjiii, • • • , Up} C M"^ p span{i;p+i, • • • , f„} C M" span{iip+i, • • • ,Um} C W^ span{f i, • • • , Vp} C M^
(A.3.3)
Finally, one important application to be used in Chapter 8 is the column (or row) compression of a low rank matrix. Let ^ be as given in Theorem A.3.1, then the column compression of A is given as: AV = [Ai 0]
(A.3.4)
with Ai a full column rank matrix. With the SVD of A, the matrix Ai is explicitly given as: Ai
= [cFiUi
•••
cFpUp]
(A.3.5)
A.4.
THE HANKEL
A.3.2
MATRIX
OF A LINEAR
PROCESS
333
The Rank of a Product of Two Matrices
Though the SVD is a numerically reliable tool to assess the (numerical) rank of a matrix, for analysis of the rank of a matrix that is the product of two matrices a useful result is the so called Sylvester ^s inequality [65]. L e m m a A . 3 . 2 (Sylvester's inequality, Kailath, 1980). Let Mi G i ? ^ ' ' ^ and Ms G R^'^'P, p{Mi) + piM2) - n < p{MiM2)
A.3.3 Let Abe
< min(p(Mi),/9(M2))
then: (A.3.5)
The QR Factorization of a Matrix Si m X n matrix with 'm> n. The QR factorization of A is given by
A = QR
(A.3.6)
where Q is an m x m orthogonal matrix and R is a,n m x n upper triangular matrix.
A.4
T h e Hankel Matrix of a Linear Process
In recent years, the Hankel matrix has played a very important role in identification, model reduction and robust control (//QO control). The Hankel matrix x of a discrete-time process G{q) is defined as the double infinite matrix Gi G2 Gs
G2 Gs G3 G4 G4 G5
(A.4.1)
where {Gk}k=i,--,00 is the impulse response matrix of G{q). Denote n as the minimal order of G{q) (also called Mcmillan degree which is the minimum order of state space realisation of G{q)), then it is well known t h a t rank{x)
= n
(A.4.2)
The n singular values of x , hi{G), • • • , hs{G),, are called Hankel singular values (HSV) of the process G{q)] and, by convention, they are ordered as hiiG)
> h2{G) >•••>
h^+i{G)
where hi{G) is also called the Hankel norm of G{q).
(A.4.3)
A.l
Definitions and Some Basic Properties of Matrices
A real matrix A of dimension m x n (m-by-n) is defined as an
•••
ain
A =
(A.l.l) flml
where aij is a real number. Denote R"^^^ as all real m x n matrices, then we can express the matrix in ( A . l . l ) more compactly as ^ G M"^^". A real matrix with dimension n x 1 is called a real n-vector: Xi (A.1.2)
Similarly, this can also be expressed as x G M " . The basic matrix computations include the transposition:
A' =
fl^ii • • • a-mi (A.1.3) ain 329
APPENDIX
330
A. REFRESHER
ON MATRIX
THEORY
the addition of two matrices of the same dimension A-\-B=
I a n +611 :
•• • •..
I
•••
^ m l I O'pT^i
ain + bin : CLrnn
(A.1.4)
1 Omn
the multipUcation by a scalar a
aA=
\
aau :
• • • OiCiin • •. :
[ aami
(A.1.5)
' • • ocamn
and the multipHcation of two matrices A e '.
and B G ]
fc=l
AB =
{A.1.6) L
Zl (^mkhn
fc=l
A sqaure matrix has equal number of rows and columns. The inverse of a square non-singular matrix A is denoted as A~^ and satisfies A~^A = AA~^ = I. Here / is the identity matrix with diagnal elements being I's and off-diagonal elements O's. It is easy to show t h a t IA = AI = A. The trace of a square matrix A is the sum of its diagonal elements and is denoted as trA. Denote d e t ^ as the determinant of A and did]A as the adjugate of A, then A-'
=
did] A
(A.1.7)
detA
Let A and B be matrices with compatible dimensions and their inverses exist, we have (ABf = A^B^
(A.1.9)
and (AB)-^ =A''B-^
(A.1.10)
T h e m a t r i x inversion l e m m a . Let A, B, C and D be matrices with compatible dimensions and the inverses of A and C exist. Then {A + BCD)-^
= A-^ - A-^B{C-^
-h
DA'^By^DA-^
This can be proved by multiply the right hand side in (A. 11) by yl H- BCD, identity matrix.
(A.LU) which results in the
A.2.
EIGENVALUES AND EIGENVECTORS
A.2
331
Eigenvalues and Eigenvectors
Eigenvalues and eigenvectors. Let A be a square n x n matrix. The eigen values of A are denoted as Ai, A2, ..., A„ and they are the solutions to the characteristic equation det{A-XI)
(A.2.1)
The right eigenvector U corresponding to the eigenvalue A^ is the non-zero solution to Ati = XiU, or, {A - XiI)U = 0
(A.2.2)
Similarly, the corresponding left eigenvector Qi is the nonzero solution to qlA = X,qJ, or, qJ{A-\I)
=0
(A.2.2)
The eigen values A^ of the real matrix A has following properties: 1. The eigen values of the real matrix A are either real or in complex conjugate pairs. 2. The sum of the eigenvalues of A equals the trace of A n
iiA = Yl^i
(A.2.3)
i=l
3. The product of the eigenvalues of A is equals to its determinant n
det yl = J l Xi
(A.2.4)
1=1
4. The eigen values of a diagonal matrix A=
an
0
•••
^22
•• •
0
are equal to its diagonal elements an, 022, •••, cinn-
A.3
The Singular Value Decomposition and QR Factorization
Singular Value Decomposition (SVD) A matrix Q G W^"""^ is orthogonal if Q^Q = I.
332
APPENDIX A. REFRESHER ON MATRIX
If a matrix Q = [gi
Qm] €
THEORY
^"^, with Qi e R"^, then qi form an orthonormal basis for
The existence of the SVD is summarized in the next result. Theorem A.3.1 (Golub and van Loan, 1989). Let A e R'' matrices: and
n] G]
then there exists orthogonal
V^ = [^1 • • • t'n] ^ J
such that, 0 10
U^AV =
€ W^"""" p < min{m, n}
0 0
(A.3.1)
|0
where o^i > a2 > - - - > o-p > 0 and where some of the zero matrices may be empty. The SVD of a matrix allows to characterize a number of geometric properties of a matrix representing a linear transformation. These are the (co-)range and (co-)null space or (co-)kernel of a matrix and are defined as: Let Ae .
^, then we have: == {y e W^ly = Ax for some x G W} = dim(range(^)) = {x e R"" : Ax = 0} = kern(yl^) = range(^^)
range(^) rank(yi) kern(^) co-range(>l) co-kern(y4)
(A.3.2)
Given a matrix A, a basis for the different geometric spaces defined in Definition A.3.1 can reliably be derived from the SVD of A. Let the SVD of A e M'^''" be denoted as in Theorem A.3.1 and let p < n, then range(^) rank(^) keni{A) co-range(yl) co-kern(^)
= = = = =
spanjiii, • • • , Up} C M"^ p span{i;p+i, • • • , f„} C M" span{iip+i, • • • ,Um} C W^ span{f i, • • • , Vp} C M^
(A.3.3)
Finally, one important application to be used in Chapter 8 is the column (or row) compression of a low rank matrix. Let ^ be as given in Theorem A.3.1, then the column compression of A is given as: AV = [Ai 0]
(A.3.4)
with Ai a full column rank matrix. With the SVD of A, the matrix Ai is explicitly given as: Ai
= [cFiUi
•••
cFpUp]
(A.3.5)
A.4.
THE HANKEL
A.3.2
MATRIX
OF A LINEAR
PROCESS
333
The Rank of a Product of Two Matrices
Though the SVD is a numerically reliable tool to assess the (numerical) rank of a matrix, for analysis of the rank of a matrix that is the product of two matrices a useful result is the so called Sylvester ^s inequality [65]. L e m m a A . 3 . 2 (Sylvester's inequality, Kailath, 1980). Let Mi G i ? ^ ' ' ^ and Ms G R^'^'P, p{Mi) + piM2) - n < p{MiM2)
A.3.3 Let Abe
< min(p(Mi),/9(M2))
then: (A.3.5)
The QR Factorization of a Matrix Si m X n matrix with 'm> n. The QR factorization of A is given by
A = QR
(A.3.6)
where Q is an m x m orthogonal matrix and R is a,n m x n upper triangular matrix.
A.4
T h e Hankel Matrix of a Linear Process
In recent years, the Hankel matrix has played a very important role in identification, model reduction and robust control (//QO control). The Hankel matrix x of a discrete-time process G{q) is defined as the double infinite matrix Gi G2 Gs
G2 Gs G3 G4 G4 G5
(A.4.1)
where {Gk}k=i,--,00 is the impulse response matrix of G{q). Denote n as the minimal order of G{q) (also called Mcmillan degree which is the minimum order of state space realisation of G{q)), then it is well known t h a t rank{x)
= n
(A.4.2)
The n singular values of x , hi{G), • • • , hs{G),, are called Hankel singular values (HSV) of the process G{q)] and, by convention, they are ordered as hiiG)
> h2{G) >•••>
h^+i{G)
where hi{G) is also called the Hankel norm of G{q).
(A.4.3)