This is about one kind of hypothesis testing. It’s not about evaluating how good a particular model is at prediction, it’s about evaluating the hypothesis underlying different models.
Setup and F-Test definition
We have data of the form \((\vec x_i, y_i)\in \mathbb R^{p+1}\times \mathbb R\) and we assume the first coordinate of \(\vec x_i\) is \(1\) \(\vec x_i = (1, x_1,...,x_p)\) (where \(x_0\) represents the constant in the linear model). Assume \(i = 1,...,n\). Goal is to compare two different models.
Null hypothesis: Our \(H_0\) is that \(Y_{\vec x_i} \sim N(\beta_0 + \beta_1x_1 + ... + \beta_qx_q + 0x_{q+1} + ... + 0x_p, \theta^2)\), i.e. that \(Y\) is generated from a normal distribution that only depends on \(x_1,...,x_q\).
Alternative hypothesis: Our \(H_a\) is that \(Y_{\vec x_i} \sim N(\beta_0 + \beta_1x_1 + ... + \beta_px_p)\).
Theorem to prove: If the null hypothesis is true, then the following “F-Statistic” \[\frac{(\text{RSS}_0 - \text{RSS}_a)/(p-q)}{\text{RSS}_a/(n-p-1)}\] is \(F_{p-q,n - p -1}\) distributed. Here \(\text{RSS}\) is the Residual Sum of Squares, so \(\text{RSS}_0 = \sum_i(y_i - \tilde y_i)^2\) where \(\tilde y_i\) is the predicted value of \(y_i\) by the null hypothesis given \(\vec x_i\).
Note that once you fit your model and obtain your \(\hat \beta_j\) you’ll predict \(Y_{x_i} \sim \hat\beta_0 + \hat\beta_1 x_1 + ... + \hat \beta_q + \epsilon_i\) with \(\epsilon_i\sim N(0,\theta^2)\).
Some distribution definitions
Definition
If \(X_1,...,X_d \sim N(0,1)\) and independent then \(X_1^2 + ... + X^2_d\) is said to by \(\chi^2_d\) distributed.
We say that \(W\) is \(F_{d_1,d_2}\) distributed if \(W\sim \frac{U_1/d_1}{U_2/d_2}\) with \(U_1 \sim \chi_{d_1}^2\) and \(U_2 \sim \chi_{d_2}^2\) with \(U_1\) and \(U_2\) independent.
Observation about multivariate normal distributions
(Compare the following with Cochrane’s Theorem.) Say that \(X\sim N(\mu, \sigma^2 I_n)\) (here we’re saying that \(X\) is normally distributed in \(\mathbb R^n\) with spherically symmetric covariance matrix \(\sigma^2I_n\)) . Let \(u_1,...,u_n\) be an orthonormal basis for \(\mathbb R^n\). Then \(X\cdot u_i \sim N(\mu\cdot u_i, \sigma^2)\) and the collection \(\{X\cdot u_i\}\) is independent.
Proof
The multivariate normal pdf in this case is
\begin{align*} f_X(x) &= \frac{1}{\sqrt{2\pi \sigma^2}}\cdot \exp\left(-\frac{|x - \mu|^2}{2\sigma^2}\right) \\ &= \frac{1}{\sqrt{2\pi \sigma^2}} \cdot \exp\left(\frac{\sum^n_{i=1}|(x\cdot u_i - \mu\cdot u_i)u_i|^2}{2\sigma^2}\right) \\ &= \frac{1}{\sqrt{2\pi\sigma^2}}\cdot \prod_{i=1}^n\exp\left(\frac{(x\cdot u_i - \mu\cdot u_i)^2}{2\sigma^2}\right) \end{align*}