testing

wonderful summarizing blog post

# basics

• data snooping - decide which hypotheses to test after examining data
• null hypothesis $H_0$ vs alternative hypothesis $H_1$
• types
• simple hypothesis $\theta = \theta_0$
• composite hypothesis $\theta > \theta_0$ or $\theta < \theta_0$
• two-sided test: $H_0: \theta = \theta_0 : vs. : H_1 \theta \neq \theta_0$
• one-sided test: $H_0: \theta \leq \theta_0 : vs. : H_1: \theta > \theta_0$
• significance levels
• stat. significant: p = 0.05
• highly stat. significant: p = 0.01
• errors
• $\alpha$ - type 1 - reject $H_0$ but $H_0$ true
• $\beta$ - type 2 - fail to reject $H_0$ but $H_0$ false
• p-value = probability, calculated assuming that the null hypothesis is true, of obtaining a value of the test statistic at least as contradictory to $H_0$ as the value calculated from the available sample
• power: $1 - \beta$
• bonferroni procedure - we are doing 3 tests with 5% confidence, so we actually do 5/3% for each test in order to restrict everything to 5% total
• Benjamini–Hochberg procedure - controls for false discovery rate
• note: ranking is often more important than actual FDR control (because we just need to know what experiments to do)

# gaussian theory

• normal theory: assume $\epsilon_i$ ~ $N(0, \sigma^2)$
• distributions
• suppose $Z_1, …, Z_n$ ~ iid N(0, 1)
• chi-squared: $\chi_d^2$ ~ $\sum_i^d U_i^2$ w/ d degrees of freedom
• $(d-1)S^2/\sigma^2 \text{ proportional to } \chi_{d-1}^2$
• student’s t: $U_{d+1} / \sqrt{d^{-1} \sum_1^d U_i^2}$ w/ d degress of freedom
• t-test: test if mean is nonzero
• test null $\theta_k=0$ w/ $t = \hat{\theta}k / \hat{SE}$ where $SE = \hat{\sigma} \cdot \sqrt{\Sigma{kk}^{-1}}$
• t-test: reject if |t| is large
• when n-p is large, t-test is called the z-test
• under null hypothesis t follows t-distr with n-p degrees of freedom
• here, $\hat{\theta}$ has a normal distr. with mean $\theta$ and cov matrix $\sigma^2 (X^TX)^{-1}$
• e independent of $\hat{\theta}$ and $||e||^2 ~ \sigma^2 \chi^2_d$ with d = n-p
• observed stat. significance level = P-value - area of normal curve beyond $\pm \hat{\theta_k} / \hat{SE}$
• if 2 vars are statistically significant, said to have independent effects on Y
• f-test: test if any of non-zero means
• null hypothesis: $\theta_i = 0, i=p-p_0, …, p$
• alternative hypothesis: for at least one $i \in {p-p_0, …, p}, : \theta_i \neq 0$
• $F = \frac{(||X\hat{\theta}||^2 - ||X\hat{\theta}^{(s)}||^2) / p_0}{||e||^2 / (n-p)}$ where $\hat{\theta^{(s)}}$ has last $p_0$ entries 0
• under null hypothesis, $||X\hat{\theta}||^2 - ||X\hat{\theta}^{(s)}||^2$ ~ $U$, $||e||^2$ ~ $V$, $F$ ~ $\frac{U/p_0}{V/(n-p)}$ where $U : indep : V$, $U$ ~ $\sigma^2 \chi^2{p_0}$, $V$ ~ $\sigma^2 \chi{n-p}^2$
• there is also a partial f-test

# statistical intervals

• interval estimates come with confidence levels
• $Z=\frac{\bar{X}-\mu}{\sigma / \sqrt{n}}$
• For p not close to 0.5, use Wilson score confidence interval (has extra terms)
• confidence interval - if multiple samples of trained typists were selected and an interval constructed for each sample mean, 95 percent of these intervals contain the true preferred keyboard height
• frequentist idea

# tests on hypotheses

• Var($\bar{X}-\bar{Y})=\frac{\sigma_1^2}{m}+\frac{\sigma_2^2}{n}$
• tail refers to the side we reject (e.g. upper-tailed=$H_a:\theta>\theta_0$
• we try to make the null hypothesis a statement of equality
• upper-tailed - reject large values
• $\alpha$ is computed using the probability distribution of the test statistic when $H_0$ is true, whereas determination of b requires knowing the test statistic distribution when $H_0$ is false
• type 1 error usually more serious, pick $\alpha$ level, then constrain $\beta$
• can standardize values and test these instead

# testing LR coefficients

• confidence interval construction
• confidence interval (CI) is range of values likely to include true value of a parameter of interest
• confidence level (CL) - probability that the procedure used to determine CI will provide an interval that covers the value of the parameter - if we remade it 100 times, 95 would contain the true $\theta_1$
• $\hat{\beta_0} \pm t_{n-2,\alpha /2} * s.e.(\hat{\beta_0})$
• for $\beta_1$
• with known $\sigma$
• $\frac{\hat{\beta_1}-\beta_1}{\sigma(\hat{\beta_1})} \sim N(0,1)$
• derive CI
• with unknown $\sigma$
• $\frac{\hat{\beta_1}-\beta_1}{s(\hat{\beta_1})} \sim t_{n-2}$
• derive CI

# ANOVA (analysis of variance)

• y - called dependent, response variable
• x - independent, explanatory, predictor variable
• notation: $E(Y|x^) = \mu_{Y\cdot x^} =$ mean value of Y when x = $x^*$
• Y = f(x) + $\epsilon$
• linear: $Y=\beta_0+\beta_1 x+\epsilon$
• logistic: $odds = \frac{p(x)}{1-p(x)}=e^{\beta_0+\beta_1 x+\epsilon}$
• we minimize least squares: $SSE = \sum_{i=1}^n (y_i-(b_0+b_1x_i))^2$
• $b_1=\hat{\beta_1}=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} = \frac{S_{xy}}{S_{xx}}$
• $b_0=\bar{y}-\hat{\beta_1}\bar{x}$
• $S_{xy}=\sum x_iy_i-\frac{(\sum x_i)(\sum y_i)}{n}$
• $S_{xx}=\sum x_i^2 - \frac{(\sum x_i)^2}{n}$
• residuals: $y_i-\hat{y_i}$
• SSE = $\sum y_i^2 - \hat{\beta}_0 \sum y_i - \hat{\beta}_1 \sum x_iy_i$
• SST = total sum of squares = $S_{yy} = \sum (y_i-\bar{y})^2 = \sum y_i^2 - (\sum y_i)^2/n$
• $r^2 = 1-\frac{SSE}{SST}=\frac{SSR}{SST}$ - proportion of observed variation that can be explained by regression
• $\hat{\sigma}^2 = \frac{SSE}{n-2}$
• $T=\frac{\hat{\beta}1-\beta_1}{S / \sqrt{S{xx}}}$ has a t distr. with n-2 df
• $s_{\hat{\beta_1}}=\frac{s}{\sqrt{S_{xx}}}$
• $s_{\hat{\beta_0}+\hat{\beta_1}x^} = s\sqrt{\frac{1}{n}+\frac{(x^-\bar{x})^2}{S_{xx}}}$
• sample correlation coefficient $r = \frac{S_{xy}}{\sqrt{S_xx}\sqrt{S_{yy}}}$
• this is a point estimate for population correlation coefficient = $\frac{Cov(X,Y)}{\sigma_X\sigma_Y}$
• make fisher transformation - this test statistic also tests correlation
• degrees of freedom
• one-sample T = n-1
• T procedures with paired data - n-1
• T procedures for 2 independent populations - use formula ~= smaller of n1-1 and n2-1
• variance - n-2
• use z-test if you know the standard deviation—