Matrix Analysis for Statistics

An up–to–date version of the complete, self–contained introduction to matrix analysis theory and practice

Providing accessible and in–depth coverage of the most common matrix methods now used in statistical applications, Matrix Analysis for Statistics, Third Edition features an easy–to–follow theorem/proof format. Featuring smooth transitions between topical coverage, the author carefully justifies the step–by–step process of the most common matrix methods now used in statistical applications, including eigenvalues and eigenvectors; the Moore–Penrose inverse; matrix differentiation; and the distribution of quadratic forms.

An ideal introduction to matrix analysis theory and practice, Matrix Analysis for Statistics, Third Edition features:

New chapter or section coverage on inequalities, oblique projections, and antieigenvalues and antieigenvectors Additional problems and chapter–end practice exercises at the end of each chapter Extensive examples that are familiar and easy to understand Self–contained chapters for flexibility in topic choice Applications of matrix methods in least squares regression and the analyses of mean vectors and covariance matrices

Matrix Analysis for Statistics, Third Edition is an ideal textbook for upper–undergraduate and graduate–level courses on matrix methods, multivariate analysis, and linear models. The book is also an excellent reference for research professionals in applied statistics.

James R. Schott, PhD, is Professor in the Department of Statistics at the University of Central Florida. He has published numerous journal articles in the area of multivariate analysis. Dr. Schott s research interests include multivariate analysis, analysis of covariance and correlation matrices, and dimensionality reduction techniques.

Preface xiii

1 A Review of Elementary Matrix Algebra 1

1.1 Introduction 1

1.2 Definitions and Notation 2

1.3 Matrix Addition and Multiplication 3

1.4 The Transpose 4

1.5 The Trace 5

1.6 The Determinant 5

1.7 The Inverse 9

1.8 Partitioned Matrices 12

1.9 The Rank of a Matrix 14

1.10 Orthogonal Matrices 15

1.11 Quadratic Forms 17

1.12 Complex Matrices 18

1.13 Random Vectors and Some Related Statistical Concepts 20

Problems 29

2 Vector Spaces 35

2.1 Introduction 35

2.2 Definitions 35

2.3 Linear Independence and Dependence 42

2.4 Matrix Rank and Linear Independence 45

2.5 Bases and Dimension 49

2.6 Orthonormal Bases and Projections 53

2.7 Projection Matrices 58

2.8 Linear Transformations and Systems of Linear Equations 65

2.9 The Intersection and Sum of Vector Spaces 73

2.10 Oblique Projections 75

2.11 Convex Sets 79

Problems 84

3 Eigenvalues and Eigenvectors 95

3.1 Introduction 95

3.2 Eigenvalues, Eigenvectors, and Eigenspaces 95

3.3 Some Basic Properties of Eigenvalues and Eigenvectors 99

3.4 Symmetric Matrices 105

3.5 Continuity of Eigenvalues and Eigenprojections 114

3.6 Extremal Properties of Eigenvalues 116

3.7 Additional Results Concerning Eigenvalues Of Symmetric

Matrices 122

3.8 Nonnegative Definite Matrices 128

3.9 Antieigenvalues and Antieigenvectors 140

Problems 142

4 Matrix Factorizations and Matrix Norms 153

4.1 Introduction 153

4.2 The Singular Value Decomposition 154

4.3 The Spectral Decomposition of a Symmetric Matrix 161

4.4 The Diagonalization of a Square Matrix 167

4.5 The Jordan Decomposition 171

4.6 The Schur Decomposition 173

4.7 The Simultaneous Diagonalization of Two Symmetric Matrices 176

4.8 Matrix Norms 182

Problems 189

5 Generalized Inverses 199

5.1 Introduction 199

5.2 The Moore Penrose Generalized Inverse 200

5.3 Some Basic Properties of the Moore Penrose Inverse 203

5.4 The Moore Penrose Inverse of a Matrix Product 209

5.5 The Moore Penrose Inverse of Partitioned Matrices 213

5.6 The Moore Penrose Inverse of a Sum 217

5.7 The Continuity of the Moore Penrose Inverse 219

5.8 Some Other Generalized Inverses 222

5.9 Computing Generalized Inverses 229

Problems 235

6 Systems of Linear Equations 243

6.1 Introduction 243

6.2 Consistency of a System of Equations 244

6.3 Solutions to a Consistent System of Equations 247

6.4 Homogeneous Systems of Equations 254

6.5 Least Squares Solutions to a System of Linear Equations 256

6.6 Least Squares Estimation For Less Than Full Rank Models 262

6.7 Systems of Linear Equations and The Singular Value Decomposition 266

6.8 Sparse Linear Systems of Equations 268

Problems 273

7 Partitioned Matrices 279

7.1 Introduction 279

7.2 The Inverse 280

7.3 The Determinant 282

7.4 Rank 290

7.5 Generalized Inverses 292

7.6 Eigenvalues 296

Problems 301

8 Special Matrices and Matrix Operations 309

8.1 Introduction 309

8.2 The Kronecker Product 310

8.3 The Direct Sum 317

8.4 The Vec Operator 318

x CONTENTS

8.5 The Hadamard Product 323

8.6 The Commutation Matrix 333

8.7 Some Other Matrices Associated With the Vec Operator 340

8.8 Nonnegative Matrices 346

8.9 Circulant and Toeplitz Matrices 356

8.10 Hadamard and Vandermonde Matrices 362

Problems 366

9 Matrix Derivatives and Related Topics 381

9.1 Introduction 381

9.2 Multivariable Differential Calculus 381

9.3 Vector and Matrix Functions 384

9.4 Some Useful Matrix Derivatives 390

9.5 Derivatives of Functions of Patterned Matrices 393

9.6 The Perturbation Method 395

9.7 Maxima and Minima 402

9.8 Convex and Concave Functions 406

9.9 The Method of Lagrange Multipliers 410

Problems 416

10 Inequalities 427

10.1 Introduction 427

10.2 Majorization 427

10.3 CauchySchwarz Inequalities 437

10.4 Hölder s Inequality 439

10.5 Minkowski s Inequality 443

10.6 The ArithmeticGeometric

Mean Inequality 444

Problems 446

11 Some Special Topics Related to Quadratic Forms 451

11.1 Introduction 451

11.2 Some Results on Idempotent Matrices 451

11.3 Cochran s Theorem 456

11.4 Distribution of Quadratic Forms in Normal Variates 459

11.5 Independence of Quadratic Forms 465

11.6 Expected Values of Quadratic Forms 470

11.7 The Wishart Distribution 478

Problems

References 499

Index 507

James R. Schott, PhD, is Professor in the Department of Statistics at the University of Central Florida. He has published numerous journal articles in the area of multivariate analysis. Dr. Schott s research interests include multivariate analysis, analysis of covariance and correlation matrices, and dimensionality reduction techniques.