Digital Signal Processing (DSP) with Python Programming

The parameter estimation and hypothesis testing are the basic tools in statistical inference. These techniques occur in many applications of data processing., and methods of Monte Carlo have become an essential tool to assess performance. For pedagogical purposes the book includes several computational problems and exercices. To prevent students from getting stuck on exercises, detailed corrections are provided.

Preface ix Notations and Abbreviations xi A Few Functions of Python (R) xiii Chapter 1. Useful Maths 1 1.1. Basic concepts on probability 1 1.2. Conditional expectation 10 1.3. Projection theorem 11 1.3.1. Conditional expectation 14 1.4. Gaussianity 14 1.4.1. Gaussian random variable 14 1.4.2. Gaussian random vectors 15 1.4.3. Gaussian conditional distribution 16 1.5. Random variable transformation 18 1.5.1. General expression 18 1.5.2. Law of the sum of two random variables 19 1.5.3. -method 20 1.6. Fundamental theorems of statistics 22 1.7. A few probability distributions 24 Chapter 2. Statistical Inferences 29 2.1. First step: visualizing data 29 2.1.1. Scatter plot 29 2.1.2. Histogram/boxplot 30 2.1.3. Q-Q plot 32 2.2. Reduction of dataset dimensionality 34 2.2.1. PCA 34 2.2.2. LDA 36 2.3. Some vocabulary 40 2.3.1. Statistical inference 40 2.4. Statistical model 41 2.4.1. Notation 42 2.5. Hypothesis testing 43 2.5.1. Simple hypotheses 45 2.5.2. Generalized likelihood ratio test (GLRT) 50 2.5.3. 2 goodness-of-fit test 57 2.6. Statistical estimation 58 2.6.1. General principles 58 2.6.2. Least squares method 62 2.6.3. Least squares method for the linear model 64 2.6.4. Method of moments 81 2.6.5. Maximum likelihood approach 84 2.6.6. Logistic regression 100 2.6.7. Non-parametric estimation of probability distribution 103 2.6.8. Bootstrap and others 107 Chapter 3. Inferences on HMM 113 3.1. Hidden Markov models (HMM) 113 3.2. Inferences on HMM 116 3.3. Filtering: general case 117 3.4. Gaussian linear case: Kalman algorithm 118 3.4.1. Kalman filter 118 3.4.2. RTS smoother 127 3.5. Discrete finite Markov case 129 3.5.1. Forward-backward formulas 130 3.5.2. Smoothing formula at one instant 133 3.5.3. Smoothing formula at two successive instants 134 3.5.4. HMM learning using the EM algorithm 135 3.5.5. The Viterbi algorithm 137 Chapter 4. Monte-Carlo Methods 141 4.1. Fundamental theorems 141 4.2. Stating the problem 141 4.3. Generating random variables 144 4.3.1. The cumulative function inversion method 144 4.3.2. The variable transformation method 147 4.3.3. Acceptance-rejection method 149 4.3.4. Sequential methods 151 4.4. Variance reduction 156 4.4.1. Importance sampling 156 4.4.2. Stratification 160 4.4.3. Antithetic variates 164 Chapter 5. Hints and Solutions 167 5.1. Useful maths 167 5.2. Statistical inferences 170 5.3. Inferences on HMM 226 5.4. Monte-Carlo methods 251 Bibliography 261 Index 263