Stat basics#
Probability rules, common distributions, the limit theorems, and how estimators trade bias for variance.
About this chapter
This chapter is the probability and estimation toolkit the rest of the book leans on. It opens with the rules for expectations, variances and covariances, then a reference table of common distributions with their densities, means, variances and moment generating functions.
Two limit theorems explain why averages are so well behaved: the law of large numbers says the sample mean settles on the true mean, and the central limit theorem says its error is roughly normal with spread \(\sigma/\sqrt{n}\). The rest is about estimating parameters from data. An estimator’s mean squared error splits exactly into bias squared plus variance, and the last section sets out the two ways of thinking about estimation, Bayesian and frequentist, following Jordan’s book.
Basics#
The rules used everywhere else in the book: independence, conditional probability and Bayes’ theorem, and how expectations, variances and covariances behave under sums and linear maps. The law of total variance splits \(V[Y]\) into the part that \(X\) explains and the part it leaves unexplained.
mutually exclusive: \(P(AB)=0\)
independence \(A \perp B\) means \(P(AB) = P(A)P(B)\)
conditional independence \(A \perp B\:\vert \:C\) means \(P(AB\vert C) = P(A\vert C) P(B\vert C)\)
conditional prob: \(P(A\vert B) = \frac{P(AB)}{P(B)} = \frac{P(B\vert A)P(A)}{\sum P(B\vert A)P(A)}\) (Bayes’ thm)
\(E[X] = \int P(x)x dx\)
\(E[h(X)] \approx h(E[X])\)
\(V[X] = E[(x-\mu)^2] = E[x^2]-E[x]^2\)
for unbiased estimate, divide by n-1
\(V(X_1-X_2) = V(X_1) + V(X_2)\) if \(X_1,X_2\) independent
\(V(a_1X_1+...+a_nX_n) = \sum_{i=1}^{n}\sum_{j=1}^{n}a_ia_jcov(X_i,X_j)\)
\(V[h(X)] \approx h'(E[X])^2 V[X]\)
standard deviation \(\sigma(X)=\sqrt{V[X]}\)
standard error = error of the mean = \(\sigma / \sqrt n\)
\(Cov[X,Y] = E[(X-\mu_X)(Y-\mu_Y)] = E[XY]-E[X]E[Y]\)
\(Cov(aX+bY,Z) = aCov(X,Z)+bCov(Y,Z)\)
\(Corr(Y,X) = \rho = \frac{Cov(Y,X)}{s_xs_y}\)
\(Corr(aX+b,cY+d) = Corr(X,Y)\) if a and c have same sign
\(R^2 = \rho^2\)
skewness = \(E[(\frac{X-\mu}{\sigma})^3]\)
law of total expectation: \(E[X] = E_Y[E(X\vert Y)]\)
law of total variance: \(V[Y] =\cblue{\underbrace{E[V(Y\vert X)]}_{\text{unexplained variance}}} + \corange{\underbrace{V(E[Y\vert X])}_{\text{explained variance}}}\)
Error bars#
always write what you use: standard dev, standard error, or 95% confidence interval = 1.96 * standard error
can get prediction intervals for on-line data using conformal prediction
nonconformity measure - how unusual an examples looks relative to previous examples
Inter-rater agreement#
Raw agreement between two raters overstates how well they agree, since some agreement happens by chance. Cohen’s kappa compares the observed agreement \(p_o\) with the chance agreement \(p_e\): it is 0 when they are equal and 1 for perfect agreement.
cohen’s kappa - measures how well different raters agree (just taking fraction may be too simple, because they might agree by chance)
from -1 to 1 (1 is perfect agreement)
\(\kappa = 1 - \frac{1 - p_o}{1-p_e}\) where \(p_o\) is the relative observed agreement among raters (identical to accuracy), and \(p_e\) is the hypothetical probability of chance agreement, using the observed data to calculate the probabilities of each observer randomly seeing each category
Inequalities#
Bounds that hold for any distribution. Markov’s inequality needs only the mean; Chebyshev’s also uses the variance and gives a better bound on how far \(X\) strays from \(\mu\). Jensen’s compares \(f(E[X])\) with \(E[f(X)]\) for convex \(f\).
cauchy-schwarz: \(\vert x \cdot y\vert \leq \vert \vert x\vert \vert \:\vert \vert y\vert \vert\)
\(E[XY]^2 \leq E[X^2] E[Y^2]\)
jensen’s: \(f(E[X]) \leq E[f(X)]\) for convex \(f\)
triangle: \(\vert \vert x + y \vert \vert \leq \vert \vert x \vert \vert + \vert \vert y \vert \vert\)
markov’s: \(P(X \geq a) \leq \frac{E[X]}{a}\)
X is typically running time of the algorithm
if we don’t have E[X], can use upper bound for E[X]
chebyshev’s: \(P(\vert X-\mu\vert \geq a) \leq \frac{Var[X]}{a^2}\)
utilizes the variance to get a better bound
Moment-generating function#
The MGF \(M_X(t) = E(e^{tX})\) holds all the moments at once: its \(r\)-th derivative at 0 is \(E(X^r)\). The MGF of a sum of independent variables is the product of their MGFs, which makes it handy for sums.
\(M_X(t) = E(e^{tX})\)
derivatives yield moments: \(\frac{d^r}{dX^r} M_X (0) = E(X^r)\)
sometimes \(\ln[M_x(t)]\) yields \(\mu\) and \(V(X)\)
\(Y = aX+b \implies M_y(t) = e^{bt}M_x(at)\)
\(Y = a_1X_1+a_2X_2 \implies M_Y(t) = M_{X_1}(a_1t)M_{X_2}(a_2t)\) if \(X_i\) independent
ordered statistics - variables \(Y_i\) such that \(Y_i\) is the ith smalless
Distributions#
A reference table. Each row gives a distribution’s CDF, its density or mass function, mean, variance and moment generating function, discrete distributions first.
PMF: \(f_X(x) = P(X=x)\)
PDF: \(P(a \leq X \leq b) = \int_a^b f(x) dx\)
these distributions are from the probability and statistics cookbook by matthias vallentin
discrete |
\(F_X(x)\) |
\(f_X(x)\) |
\(\mathbb{E}[X]\) |
\(\mathbb{V}[X]\) |
\(M_X(s)\) |
|---|---|---|---|---|---|
uniform \(\text{Unif}\{a, \dots, b\}\) |
\(\begin{cases} 0 & x < a \\ \frac{\lfloor x \rfloor - a + 1}{b - a} & a \le x \le b \\ 1 & x > b \end{cases}\) |
\(\frac{I(a \le x \le b)}{b - a + 1}\) |
\(\frac{a+b}{2}\) |
\(\frac{(b-a+1)^2 - 1}{12}\) |
\(\frac{e^{as} - e^{-(b+1)s}}{s(b-a)}\) |
bernoulli \(\text{Bern}(p)\) |
\((1-p)^{1-x}\) |
\(p^x (1-p)^{1-x}\) |
\(p\) |
\(p(1-p)\) |
\(1 - p + pe^s\) |
binomial \(\text{Bin}(n, p)\) |
\(I_{1-p}(n-x, x+1)\) |
\(\binom{n}{x} p^x (1-p)^{n-x}\) |
\(np\) |
\(np(1-p)\) |
\((1 - p + pe^s)^n\) |
multinomial \(\text{Mult}(n, p)\) |
\(\frac{n!}{x_1! \dots x_k!} p_1^{x_1} \cdots p_k^{x_k} \quad \sum_{i=1}^k x_i = n\) |
\(np_i\) |
\(np_i(1-p_i)\) |
\(\left(\sum_{i=0}^k p_i e^{s_i}\right)^n\) |
|
hypergeometric \(\text{Hyp}(N, m, n)\) |
\(\approx \Phi\left(\frac{x - np}{\sqrt{np(1-p)}}\right)\) |
\(\frac{\binom{m}{x}\binom{N-m}{n-x}}{\binom{N}{n}}\) |
\(\frac{nm}{N}\) |
\(\frac{nm(N-n)(N-m)}{N^2(N-1)}\) |
|
neg. binomial \(\text{NBin}(r, p)\) |
\(I_p(r, x+1)\) |
\(\binom{x+r-1}{r-1} p^r (1-p)^x\) |
\(r\frac{1-p}{p}\) |
\(r\frac{1-p}{p^2}\) |
\(\left(\frac{p}{1-(1-p)e^s}\right)^r\) |
geometric \(\text{Geo}(p)\) |
\(1 - (1-p)^x \quad x \in \mathbb{N}^+\) |
\(p(1-p)^{x-1} \quad x \in \mathbb{N}^+\) |
\(\frac{1}{p}\) |
\(\frac{1-p}{p^2}\) |
\(\frac{pe^s}{1-(1-p)e^s}\) |
poisson \(\text{Po}(\lambda)\) |
\(e^{-\lambda} \sum_{i=0}^x \frac{\lambda^i}{i!}\) |
\(\frac{\lambda^x e^{-\lambda}}{x!}\) |
\(\lambda\) |
\(\lambda\) |
\(e^{\lambda(e^s - 1)}\) |
continuous |
\(F_X(x)\) |
\(f_X(x)\) |
\(\mathbb{E}[X]\) |
\(\mathbb{V}[X]\) |
|---|---|---|---|---|
uniform \(\text{Unif}(a, b)\) |
\(\begin{cases} 0 & x < a \\ \frac{x-a}{b-a} & a < x < b \\ 1 & x > b \end{cases}\) |
\(\frac{I(a < x < b)}{b - a}\) |
\(\frac{a+b}{2}\) |
\(\frac{(b-a)^2}{12}\) |
normal \(\mathcal{N}(\mu, \sigma^2)\) |
\(\Phi(x) = \int_{-\infty}^x \phi(t)\, dt\) |
\(\phi(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left\{-\frac{(x-\mu)^2}{2\sigma^2}\right\}\) |
\(\mu\) |
\(\sigma^2\) |
log-normal \(\ln\mathcal{N}(\mu, \sigma^2)\) |
\(\frac{1}{2} + \frac{1}{2} \text{erf}\left[\frac{\ln x - \mu}{\sqrt{2\sigma^2}}\right]\) |
\(\frac{1}{x\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{(\ln x - \mu)^2}{2\sigma^2}\right\}\) |
\(e^{\mu + \sigma^2/2}\) |
\((e^{\sigma^2} - 1) e^{2\mu + \sigma^2}\) |
multi norm \(\text{MVN}(\mu, \Sigma)\) |
\((2\pi)^{-k/2} \vert\Sigma\vert^{-1/2} e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1} (x-\mu)}\) |
\(\mu\) |
\(\Sigma\) |
|
student’s t \(\text{Student}(\nu)\) |
\(I_x\left(\frac{\nu}{2}, \frac{\nu}{2}\right)\) |
\(\frac{\Gamma\left(\frac{\nu+1}{2}\right)}{\sqrt{\nu\pi}\,\Gamma\left(\frac{\nu}{2}\right)} \left(1 + \frac{x^2}{\nu}\right)^{-(\nu+1)/2}\) |
\(0\) |
\(\begin{cases} \frac{\nu}{\nu-2} & \nu > 2 \\ \infty & 1 < \nu \le 2 \end{cases}\) |
chi-square \(\chi^2_k\) |
\(\frac{1}{\Gamma(k/2)} \gamma\left(\frac{k}{2}, \frac{x}{2}\right)\) |
\(\frac{1}{2^{k/2}\Gamma(k/2)} x^{k/2-1} e^{-x/2}\) |
\(k\) |
\(2k\) |
F \(\text{F}(d_1, d_2)\) |
\(I_{\frac{d_1 x}{d_1 x + d_2}}\left(\frac{d_1}{2}, \frac{d_1}{2}\right)\) |
\(\frac{\sqrt{\frac{(d_1 x)^{d_1} d_2^{d_2}}{(d_1 x + d_2)^{d_1 + d_2}}}}{x \text{B}\left(\frac{d_1}{2}, \frac{d_1}{2}\right)}\) |
\(\frac{d_2}{d_2 - 2}\) |
\(\frac{2 d_2^2 (d_1 + d_2 - 2)}{d_1 (d_2 - 2)^2 (d_2 - 4)}\) |
exponential \(\text{Exp}(\beta)\) |
\(1 - e^{-x/\beta}\) |
\(\frac{1}{\beta} e^{-x/\beta}\) |
\(\beta\) |
\(\beta^2\) |
gamma \(\text{Gamma}(\alpha, \beta)\) |
\(\frac{\gamma(\alpha, x/\beta)}{\Gamma(\alpha)}\) |
\(\frac{1}{\Gamma(\alpha)\beta^\alpha} x^{\alpha-1} e^{-x/\beta}\) |
\(\alpha\beta\) |
\(\alpha\beta^2\) |
inv. gamma \(\text{InvGamma}(\alpha, \beta)\) |
\(\frac{\Gamma\left(\alpha, \frac{\beta}{x}\right)}{\Gamma(\alpha)}\) |
\(\frac{\beta^\alpha}{\Gamma(\alpha)} x^{-\alpha-1} e^{-\beta/x}\) |
\(\frac{\beta}{\alpha-1} \quad \alpha > 1\) |
\(\frac{\beta^2}{(\alpha-1)^2(\alpha-2)} \quad \alpha > 2\) |
dirichlet \(\text{Dir}(\alpha)\) |
\(\frac{\Gamma\left(\sum_{i=1}^k \alpha_i\right)}{\prod_{i=1}^k \Gamma(\alpha_i)} \prod_{i=1}^k x_i^{\alpha_i - 1}\) |
\(\frac{\alpha_i}{\sum_{i=1}^k \alpha_i}\) |
\(\frac{\mathbb{E}[X_i](1 - \mathbb{E}[X_i])}{\sum_{i=1}^k \alpha_i + 1}\) |
|
beta \(\text{Beta}(\alpha, \beta)\) |
\(I_x(\alpha, \beta)\) |
\(\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)} x^{\alpha-1} (1-x)^{\beta-1}\) |
\(\frac{\alpha}{\alpha+\beta}\) |
\(\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}\) |
weibull \(\text{Weibull}(\lambda, k)\) |
\(1 - e^{-(x/\lambda)^k}\) |
\(\frac{k}{\lambda}\left(\frac{x}{\lambda}\right)^{k-1} e^{-(x/\lambda)^k}\) |
\(\lambda\Gamma\left(1 + \frac{1}{k}\right)\) |
\(\lambda^2\Gamma\left(1 + \frac{2}{k}\right) - \mu^2\) |
pareto \(\text{Pareto}(x_m, \alpha)\) |
\(1 - \left(\frac{x_m}{x}\right)^\alpha \quad x \ge x_m\) |
\(\alpha \frac{x_m^\alpha}{x^{\alpha+1}} \quad x \ge x_m\) |
\(\frac{\alpha x_m}{\alpha - 1} \quad \alpha > 1\) |
\(\frac{x_m^\alpha}{(\alpha-1)^2(\alpha-2)} \quad \alpha > 2\) |
multivariate gaussian
2 parameterizations (\(x \in \mathbb{R}^n\))
canonical parameterization: \(p(x\vert\mu, \Sigma) = \frac{1}{(2\pi )^{n/2} \vert\Sigma\vert^{1/2}} \exp\left[ -\frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu) \right]\)
moment parameterization: \(p(x\vert\eta, \Omega) = \text{exp}\left( a + \eta^T x - \frac{1}{2} x^T \Omega x\right)\) ~ also called information parameterization
\(\Omega = \Sigma^{-1}\)
\(\eta = \Sigma^{-1} \mu\)
joint distr - split parameters into block matrices
want to block diagonalize the matrix
Schur complement of matrix M w.r.t. H: \(M/H\)
\(\mu = \begin{bmatrix} \mu_1 \\ \mu_2 \end{bmatrix}\)
\(\Sigma = \begin{bmatrix} \Sigma_{11} & \Sigma_{12}\\ \Sigma_{21} & \Sigma_{22} \end{bmatrix}\)
\(p(x_1, x_2) = \underbrace{p(x_1\vert x_2)}_{\text{conditional}}\cdot\underbrace{p(x_2)}_{\text{marginal}}\)
marginal
\(\mu_2^m = \mu_2\)
\(\Sigma_2^m = \Sigma_{22}\)
conditional
\(\mu_{1\vert 2}^c = \mu_1 + \Sigma_{12}\Sigma_{22}^{-1} (x_2 - \mu_2)\)
\(\Sigma_{1\vert 2}^c = \Sigma_{11} - \Sigma_{12} \Sigma_{22}^{-1} \Sigma_{21}\)
Law of large numbers#
Both results are about the sample mean \(\bar X\) of \(n\) independent draws with mean \(\mu\) and variance \(\sigma^2\). Its variance is \(\sigma^2/n\), which is the whole story behind both theorems.
Law of large numbers#
equivalent statements
\(E(\bar{X}-\mu)^2 \to 0\) as \(n \to \infty,\)
\(P(\vert\bar{X}-\mu\vert \geq \epsilon) \to 0\) as \(n \to \infty\)
\(T_o = X_1+...+X_n, E(T_o) = n\mu , V(T_o) = n\mu ^2\)
implications
\(E(\bar{X}) = \mu\)
\(V(\bar{X}) = \frac{\sigma_x^2}{n}\)
Central limit thm#
The law of large numbers says where \(\bar X\) ends up. The central limit theorem says what its error looks like: standardized by \(\sigma/\sqrt{n}\), \(\bar X\) has approximately a standard normal distribution when \(n\) is large.
2 characterizations
random sample means have a normal distr. if n is large
\(\lim_{n\to\infty}P(\frac{\bar{X}-\mu}{\sigma/\sqrt{n}}\leq z)=P(Z\leq z) = \Phi(z)\)
implications
\(X_1*..*X_n\) has approximately lognormal distribution if all \(P(X_i>0)\)
Bias and point estimation#
An estimator \(\hat\theta\) is a statistic used to guess a parameter \(\theta\). It can miss in two ways: by being off on average (bias) or by changing a lot from one sample to the next (variance). The decomposition below shows that mean squared error is exactly the sum of the two.
point estimator \(\hat{\theta}\) - statistic that predicts a parameter
point estimate - single number prediction
bias: \(E(\hat{\theta}) - \theta\)
more complex models (more nonzero parameters) have lower bias, higher variance
if high bias, train and test error will be very close (model isn’t complex enough)
after unbiased we want MVUE (minimum variance unbiased estimator)
need inductive inference property: must make prior assumptions in order to classify unseen instances
define inductive bias of a learner as the set of additional assumptions B sufficient to justify its inductive inferences as deductive inferences
bias types
preference bias = search bias - models can search entire space (e.g. NN, decision tree)
restriction bias = language bias - models that can’t express entire space (e.g. linear)
consistent: \(\hat{\theta_n} \to some \: value\)
basically it converges to a number (can still be biased)
bias/variance trade-off
MSE - mean squared error - \(E[(\hat{\theta}-\theta)^2]\) = \(\cblue{V(\hat{\theta})}+\corange{[E(\hat{\theta})-\theta]^2}\)
decomposition: \(MSE = \corange{Bias^2} + \cblue{Variance}\)
which can be shown as follows
\[\begin{split} \begin{aligned} MSE(\hat{\theta}) &= E\left[\hat{\theta} - \theta\right]^2 \\ &= E\left[\hat{\theta} - E(\hat{\theta}) + E(\hat{\theta}) - \theta\right]^2 \quad \text{(add and subtract } E(\hat{\theta})\text{)} \\ &= \cblue{E\left[\hat{\theta} - E(\hat{\theta})\right]^2} + \cgreen{2E\left[(\hat{\theta} - E(\hat{\theta}))(E(\hat{\theta}) - \theta)\right]} \\ &\quad + \corange{E\left[E(\hat{\theta}) - \theta\right]^2} \\ &= \cblue{Var(\hat{\theta})} + \cgreen{2(E(\hat{\theta}) - \theta)E\left[\hat{\theta} - E(\hat{\theta})\right]} \\ &\quad + \corange{\left(E(\hat{\theta} - \theta)\right)^2} \quad \text{(} E(\hat{\theta}) \text{ and } \theta \text{ are constant)} \\ &= \cblue{Var(\hat{\theta})} + \corange{Bias(\hat{\theta})^2} \end{aligned} \end{split}\]where the last equality follows from the cross term vanishing: \(\cgreen{E\left[\hat{\theta} - E(\hat{\theta})\right]} = E(\hat{\theta}) - E(\hat{\theta}) = 0\)
defs
bias = approximation err
variance = estimation err
confidence intervals - take sample data + produce a range of values that likely contains population parameter you are interested in
confidence interval of the prediction is a range that likely contains the mean value of the dependent variable given specific values of the independent variables - usually wider because it is one point, not a mean
MLE example
MLE - maximize likelihood \(L(\theta) = p(X_1,...,X_n;\theta_1,...\theta_m)\) (the agreement with a chosen distribution)
\(\hat{\theta} = \)argmax \( L(\theta)\)
\(L(\theta)=P(X_1...X_n\vert\theta)=\prod_{i=1}^n P(X_i\vert\theta)\)
\(\log \: L(\theta)= \ell(\theta) = \sum \log P(X_i\vert\theta)\)
to maximize, set \(\frac{\partial \ell (\theta)}{\partial \theta} = 0\)
fisher information \(I(\theta)=V[\frac{\partial^2}{\partial\theta^2} \overbrace{\ln(f[x;\theta])}^{\text{Fisher score function}} ]\) (for n samples, multiply by n)
higher info \(\implies\) lower estimation error
\(\overbrace{\ln(f[x;\theta])}^{\text{Fisher score function}}\)
Overview - Jordan ch 5#
Probability goes from a model to data; statistics goes from data back to a model. Bayesians treat the parameter as a random variable and update a prior into a posterior. Frequentists treat it as fixed and judge estimators, like the maximum likelihood estimate, by how they behave over repeated samples.
from Michael Jordan’s “An Introduction to Probabilistic Graphical Models”
prob theory: given model \(\theta\), infer data \(X\)
statistics: given data \(X\), infer model \(\theta\)
2 statistical schools of thought: Bayesian and frequentist
Bayesian: \(\cgreen{\overbrace{p(\theta \vert x)}^{\text{posterior}}} = \frac{\cblue{\overbrace{p(x\vert\theta)}^{\text{likelihood}}} \corange{\overbrace{p(\theta)}^{\text{prior}}}}{p(x)}\)
assumes \(\theta\) is a RV, find its distr.
prior probability \(\corange{p(\theta)}\)= statistician’s uncertainty
posterior \(\cgreen{p(\theta\vert x)}\) is what you don’t observe
\(\hat{\theta}_{Bayes} = \int \theta \: p(\theta \vert x) d\theta\) ~ mean of the posterior
- \[\begin{split}\hat{\theta}_{MAP} = \underset{\theta}{\operatorname*{argmax}} \: \cgreen{p(\theta\vert x)} = \underset{\theta}{\operatorname*{argmax}} \: \cblue{p(x\vert \theta)} \corange{p(\theta)} \\\ = \underset{\theta}{\operatorname*{argmax}} \: [ \cblue{\log \: p(x\vert\theta)} + \corange{\log \: p(\theta)} ]\end{split}\]
like penalized likelihood: log-likelihood + log-prior penalty
bayesians prefer whole distr. rather than parameter estimates
frequentist - use estimators (ex. MLE)
no prior - only use priors when they correspond to objective frequencies of observing values
neyman / pearson
\(\hat{\theta}_{MLE} = \operatorname*{argmax}_\theta \: p(x\vert\theta)\)
really likelihood is whatever we model (ex. for discriminative models would be \(p(y\vert x, \theta)\))
3 problems#
Jordan’s three basic problems: density estimation (model \(P(X)\)), regression (model \(p(y\vert x)\)) and classification. Each is worked both ways, frequentist with the MLE and Bayesian with a prior, and with both parametric models and nonparametric ones such as kernel density estimators.
density estimation - given samples of X, estimate P(X)
ex. univariate Gaussian density estimation
frequentist
derive MLE for mean and variance
bayesian
assume distr. for \(\mu\)
ex. \(p(\mu) \sim N(\mu_0, \tau^2)\)
derive MAP for mean and variance (assuming some prior)
can use plate to show repeated element
ex. discrete, multinomial prob. distr.
derive MLE
\(P(x\vert \theta) \sim\)multionomial distr.
derive MAP
want to be able to plug in posterior as prior recursively
this requires a Dirichlet prior to multiply the multinomial
Dirichlet: \(p(\theta) = C(\alpha) \theta_1^{\alpha_1 - 1}\cdot \cdot \cdot \theta_M^{\alpha_M-1}\)
ex. mixture models - \(p(x\vert\theta)=\sum_k \alpha_k f_k (x\vert\theta_k)\)
here \(f_k\) represent densities (mixture components)
\(\alpha_k\) are weights (mixing proportions)
can do inference on this - given x, figure out which cluster it fits into better
learning requires EM
can be used nonparametrically - mixture seive
however, means are allowed to vary
solving with random projection: project to low dim and keep track of means etc.
ex. nonparametric density estimation
ex. kernel density estimator - stacking up mass
each point contributes a kernel function \(k(x,x_n, \lambda)\)
\(x_n\) is location, \(\lambda\) is smoothing
\(\hat{p}(x) = \frac{1}{N}\sum_n k(x,x_n,\lambda)\)
nonparametric models sometimes called infinite-dimensional
regression - want \(p(y \vert x)\)
conditional mixture model - variable z can be used to pick out regions of input space where different regression functions are used
\(p(y_n\vert x_n,\theta) = \sum_k p(y_n\vert z_n^k = 1, x_n, \theta) \cdot p(z_n^k=1\vert x_n,\theta)\)
nonparametric regression
ex. kernel regression \(\hat{f}(x) = \frac{\sum_{i=1}^N k(x, x_i) \cdot y_i}{\sum_{m=1}^N k(x, x_j)}\)
classification
ex. Gaussian class-conditional densities
posterior probability is logistic function
clustering - use mixture models
Model selection / averaging#
Choosing among models of different complexity. Bayesians compare models by \(p(m\vert x)\) or average predictions over them. Frequentists can’t compare by likelihood alone, since it always prefers the more complex model, and use criteria such as AIC or cross-validation instead.
bayesian
for model m, want to maximize \(p(m\vert x) = \frac{p(x\vert m) p(m)}{p(x)}\)
usually, just take \(m\) that maximizes \(p(m\vert x)\)
model averaging: \(p(x_{new}\vert x) = \int dm \int d\theta \: p(x_{new}\vert \theta, m) p(\theta\vert x, m) p(m\vert x)\)
otherwise integrate over \(\theta, m\) - model averaging
frequentist
can’t use MLE - will always prefer more complex models
use some criteria such as KL-divergence, AIC, cross-validation
Key ideas
The variance of the sample mean is \(\sigma^2/n\), so its standard error shrinks like \(\sigma/\sqrt{n}\).
The law of large numbers says \(\bar X \to \mu\); the central limit theorem says \((\bar X - \mu)/(\sigma/\sqrt{n})\) is approximately standard normal for large \(n\).
An estimator’s mean squared error is exactly bias squared plus variance, and more complex models trade lower bias for higher variance.
The MLE maximizes \(\sum \log P(X_i\vert\theta)\). MAP adds a log-prior, which acts as a penalty on the log-likelihood.
Bayesians treat \(\theta\) as a random variable and report its posterior; frequentists treat it as fixed and judge estimators over repeated samples.