Testing#
Some notes on hypothesis testing, confidence intervals, and ANOVA.
About this chapter
A test asks whether the data would be surprising if the null hypothesis \(H_0\) were true. The basics section sets up the vocabulary: type 1 error \(\alpha\), type 2 error \(\beta\), power \(1 - \beta\), p-values, and the corrections needed when many tests are run at once (Bonferroni, Benjamini-Hochberg).
The Gaussian theory section derives the t-test for one regression coefficient and the F-test for a group of them from the chi-squared and t distributions. The later sections cover what a confidence interval does and doesn’t promise, confidence intervals for regression coefficients, and the formulas of simple linear regression (\(S_{xx}\), \(S_{xy}\), SSE, SST, \(r^2\)) that ANOVA is built on.
Chapter outline: 6 sections
- Basics1 interactiveA test fixes a null hypothesis H_0 and asks how surprising the data would be if it were true.
- Gaussian theoryWith normal errors, \hat\theta is normal and the residual sum of squares is \sigma^2\chi^2_{n-p}, independent of \hat\theta.
- Statistical intervalsAn interval estimate comes with a confidence level, and the level describes the procedure, not one interval: over repeated samples, 95% of the intervals built this way contain the true value.
- Tests on hypotheses1 interactiveIn practice \alpha is picked first, which sets where the test starts rejecting.
- Testing LR coefficientsThe same construction for a regression coefficient.
- ANOVA (analysis of variance)Least squares with one predictor, written with the sums S_{xx}, S_{xy} and S_{yy}.
wonderful summarizing blog post
Basics#
A test fixes a null hypothesis \(H_0\) and asks how surprising the data would be if it were true. Two errors are possible: rejecting a true \(H_0\) (type 1, rate \(\alpha\)) and keeping a false one (type 2, rate \(\beta\)). Running many tests at once needs a correction, or type 1 errors pile up.
data snooping - decide which hypotheses to test after examining data
null hypothesis \(H_0\) vs alternative hypothesis \(H_1\)
types
simple hypothesis \(\theta = \theta_0\)
composite hypothesis \(\theta > \theta_0\) or \(\theta < \theta_0\)
two-sided test: \(H_0: \theta = \theta_0 \: vs. \: H_1 \theta \neq \theta_0\)
one-sided test: \(H_0: \theta \leq \theta_0 \: vs. \: H_1: \theta > \theta_0\)
significance levels
stat. significant: p = 0.05
highly stat. significant: p = 0.01
errors
\(\alpha\) - type 1 - reject \(H_0\) but \(H_0\) true
\(\beta\) - type 2 - fail to reject \(H_0\) but \(H_0\) false
p-value = probability, calculated assuming that the null hypothesis is true, of obtaining a value of the test statistic at least as contradictory to \(H_0\) as the value calculated from the available sample
power: \(1 - \beta\)
adjustments
bonferroni procedure - we are doing 3 tests with 5% confidence, so we actually do 5/3% for each test in order to restrict everything to 5% total
Benjamini–Hochberg procedure - controls for false discovery rate
note: ranking is often more important than actual FDR control (because we just need to know what experiments to do)
Gaussian theory#
With normal errors, \(\hat\theta\) is normal and the residual sum of squares is \(\sigma^2\chi^2_{n-p}\), independent of \(\hat\theta\). Ratios of these give the t-statistic for one coefficient and the F-statistic for a group of coefficients.
normal theory: assume \(\epsilon_i\) ~ \(N(0, \sigma^2)\)
distributions
suppose \(Z_1, ..., Z_n\) ~ iid N(0, 1)
chi-squared: \(\chi_d^2\) ~ \(\sum_i^d U_i^2\) w/ d degrees of freedom
\((d-1)S^2/\sigma^2 \text{ proportional to } \chi_{d-1}^2\)
student’s t: \(U_{d+1} / \sqrt{d^{-1} \sum_1^d U_i^2}\) w/ d degress of freedom
t-test: test if mean is nonzero
test null \(\theta_k=0\) w/ \(t = \hat{\theta}_k / \hat{SE}\) where \(SE = \hat{\sigma} \cdot \sqrt{\Sigma_{kk}^{-1}}\)
t-test: reject if |t| is large
when n-p is large, t-test is called the z-test
under null hypothesis t follows t-distr with n-p degrees of freedom
here, \(\hat{\theta}\) has a normal distr. with mean \(\theta\) and cov matrix \(\sigma^2 (X^TX)^{-1}\)
e independent of \(\hat{\theta}\) and \(\vert \vert e\vert \vert ^2 ~ \sigma^2 \chi^2_d\) with d = n-p
observed stat. significance level = P-value - area of normal curve beyond \(\pm \hat{\theta_k} / \hat{SE}\)
if 2 vars are statistically significant, said to have independent effects on Y
f-test: test if any of non-zero means
null hypothesis: \(\theta_i = 0, i=p-p_0, ..., p\)
alternative hypothesis: for at least one \(i \in \{p-p_0, ..., p\}, \: \theta_i \neq 0\)
\(F = \frac{\cblue{(\vert \vert X\hat{\theta}\vert \vert ^2 - \vert \vert X\hat{\theta}^{(s)}\vert \vert ^2)} / p_0}{\corange{\vert \vert e\vert \vert ^2} / (n-p)}\) where \(\hat{\theta^{(s)}}\) has last \(p_0\) entries 0
under null hypothesis, \(\cblue{\vert \vert X\hat{\theta}\vert \vert ^2 - \vert \vert X\hat{\theta}^{(s)}\vert \vert ^2}\) ~ \(\cblue{U}\), \(\corange{\vert \vert e\vert \vert ^2}\) ~ \(\corange{V}\), \(F\) ~ \(\frac{\cblue{U}/p_0}{\corange{V}/(n-p)}\) where \(\cblue{U} \: indep \: \corange{V}\), \(\cblue{U}\) ~ \(\sigma^2 \chi^2_{p_0}\), \(\corange{V}\) ~ \(\sigma^2 \chi_{n-p}^2\)
there is also a partial f-test
Statistical intervals#
An interval estimate comes with a confidence level, and the level describes the procedure, not one interval: over repeated samples, 95% of the intervals built this way contain the true value.
interval estimates come with confidence levels
\(Z=\frac{\bar{X}-\mu}{\sigma / \sqrt{n}}\)
For p not close to 0.5, use Wilson score confidence interval (has extra terms)
confidence interval - if multiple samples of trained typists were selected and an interval constructed for each sample mean, 95 percent of these intervals contain the true preferred keyboard height
frequentist idea
Tests on hypotheses#
In practice \(\alpha\) is picked first, which sets where the test starts rejecting. \(\beta\) then depends on how far the truth is from \(H_0\) and on the sample size.
Var(\(\bar{X}-\bar{Y})=\frac{\sigma_1^2}{m}+\frac{\sigma_2^2}{n}\)
tail refers to the side we reject (e.g. upper-tailed=\(H_a:\theta>\theta_0\)
we try to make the null hypothesis a statement of equality
upper-tailed - reject large values
\(\alpha\) is computed using the probability distribution of the test statistic when \(H_0\) is true, whereas determination of b requires knowing the test statistic distribution when \(H_0\) is false
type 1 error usually more serious, pick \(\alpha\) level, then constrain \(\beta\)
can standardize values and test these instead
Testing LR coefficients#
The same construction for a regression coefficient. With \(\sigma\) known, \((\hat\beta_1 - \beta_1)/\sigma(\hat\beta_1)\) is standard normal; with \(\sigma\) estimated it has a \(t_{n-2}\) distribution, and the interval is the estimate plus or minus \(t_{n-2,\alpha/2}\) standard errors.
confidence interval construction
confidence interval (CI) is range of values likely to include true value of a parameter of interest
confidence level (CL) - probability that the procedure used to determine CI will provide an interval that covers the value of the parameter - if we remade it 100 times, 95 would contain the true \(\theta_1\)
\(\hat{\beta_0} \pm t_{n-2,\alpha /2} * s.e.(\hat{\beta_0}) \)
for \(\beta_1\)
with known \(\sigma\)
\(\frac{\hat{\beta_1}-\beta_1}{\sigma(\hat{\beta_1})} \sim N(0,1)\)
derive CI
with unknown \(\sigma\)
\(\frac{\hat{\beta_1}-\beta_1}{s(\hat{\beta_1})} \sim t_{n-2}\)
derive CI
ANOVA (analysis of variance)#
Least squares with one predictor, written with the sums \(S_{xx}\), \(S_{xy}\) and \(S_{yy}\). The total sum of squares splits into the part explained by the regression and SSE, and \(r^2\) is the explained fraction.
y - called dependent, response variable
x - independent, explanatory, predictor variable
notation: \(E(Y\vert x^*) = \mu_{Y\cdot x^*} =\) mean value of Y when x = \(x^*\)
Y = f(x) + \(\epsilon\)
linear: \(Y=\beta_0+\beta_1 x+\epsilon\)
logistic: \(odds = \frac{p(x)}{1-p(x)}=e^{\beta_0+\beta_1 x+\epsilon}\)
we minimize least squares: \(SSE = \sum_{i=1}^n (y_i-(b_0+b_1x_i))^2\)
\(b_1=\hat{\beta_1}=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} = \frac{S_{xy}}{S_{xx}}\)
\(b_0=\bar{y}-\hat{\beta_1}\bar{x}\)
\(S_{xy}=\sum x_iy_i-\frac{(\sum x_i)(\sum y_i)}{n}\)
\(S_{xx}=\sum x_i^2 - \frac{(\sum x_i)^2}{n}\)
residuals: \(y_i-\hat{y_i}\)
SSE = \(\sum y_i^2 - \hat{\beta}_0 \sum y_i - \hat{\beta}_1 \sum x_iy_i\)
SST = total sum of squares = \(S_{yy} = \sum (y_i-\bar{y})^2 = \sum y_i^2 - (\sum y_i)^2/n\)
\(r^2 = 1-\frac{\corange{SSE}}{SST}=\frac{\cblue{SSR}}{SST}\) - proportion of observed variation that can be explained by regression
\(\hat{\sigma}^2 = \frac{SSE}{n-2}\)
\(T=\frac{\hat{\beta}_1-\beta_1}{S / \sqrt{S_{xx}}}\) has a t distr. with n-2 df
\(s_{\hat{\beta_1}}=\frac{s}{\sqrt{S_{xx}}}\)
\(s_{\hat{\beta_0}+\hat{\beta_1}x^*} = s\sqrt{\frac{1}{n}+\frac{(x^*-\bar{x})^2}{S_{xx}}}\)
sample correlation coefficient \(r = \frac{S_{xy}}{\sqrt{S_{xx}}\sqrt{S_{yy}}}\)
this is a point estimate for population correlation coefficient = \(\frac{Cov(X,Y)}{\sigma_X\sigma_Y}\)
make fisher transformation - this test statistic also tests correlation
degrees of freedom
one-sample T = n-1
T procedures with paired data - n-1
T procedures for 2 independent populations - use formula ~= smaller of n1-1 and n2-1
variance - n-2
use z-test if you know the standard deviation—
Key ideas
The p-value is the probability, computed assuming \(H_0\), of a test statistic at least as contradictory to \(H_0\) as the one observed.
\(\alpha\) is picked first. \(\beta\) then depends on how far the truth is from \(H_0\) and on the sample size, and the power is \(1 - \beta\).
Running many tests inflates false positives. Bonferroni keeps the chance of any false rejection at \(\alpha\); Benjamini-Hochberg controls the false discovery rate.
With normal errors, the t-statistic tests one regression coefficient and the F-statistic tests a group of them.
A 95% confidence interval is a statement about the procedure. Over repeated samples, 95% of the intervals contain the true parameter.
In simple regression SST = SSR + SSE, and \(r^2\) = SSR/SST is the fraction of variation the regression explains.