(Lecture notes from the Erdos institute data science bootcamp.) Presume there is a true relationship \[Y = f(X) + \epsilon\] where \(f:\mathbb R^p\to \mathbb R\) is some function and \(\epsilon\) is some error term independent of \(X\) with expected value \(0\) and variance \(\sigma^2\).
A learning algorithm takes some training data \(D = \{(x_i,y_i)\}_{i=1}^n\) and produces a fitted model \(\hat f_D\). These \(x_i\) are realizations of the random vector variable \(X\), and \(y_i = f(x_i) + \epsilon\).
We are now going to consider the variability in the fitted \(\hat f_D\).
If we consider many different datasets \(D\), then we’ll get different functions \(\hat f_D\), so this process is also a sampling process for \(\hat f\). We get variability in our model dependent on \(D\). \[\text{Expected Generalization Error} = \mathbb E_{x,y,D} \left[\big(\hat f_D(x) - y\big)^2\right]\] Here we’re treating the samples \(D\) as another random variable. Let \[\overline f = \mathbb E_{D}(\hat f_D).\] This is now a function \(\mathbb R^p\to \mathbb R\) as well. It can be shown that this expected generalization error splits into three terms involving this \(\overline f\) as follows: \[\mathbb E_{x,D}\left[\big( \hat f_D(x) - \overline f(x)\big)^2\right] + \mathbb E_x\left[\big(\overline f(x) - f(x)\big)^2\right] + \sigma^2\] We give these quantities the following names:
- The variance of the learning algorithm is \(\mathbb E_{x,D}\left[\big( \hat f_D(x) - \overline f(x)\big)^2\right]\). It’s a variance looking thing, the expected value of the squared difference between a thing and its mean. It measures how far a particular fitted model \(\hat f_D\) is likely to be from the mean fitted model \(\overline f\) averaged over all possible \(x\). It’s the variance of our model.
- The squared bias of the learning algorithm is \(\mathbb E_x\left[\big(\overline f(x) - f(x)\big)^2\right]\). It gives us a measure of how far the mean fitted moel \(\overline f\) is from the true relationship \(f\).
- The term \(\sigma^2\) is called the irreducible error. When we pick a single new observation \((x,y)\) we have \(y = f(x) + \epsilon\). So there is a lower bound on how well any observation is going to be able to generalize.
In the real world, we don’t know the distribution underlying our \(X\) or our \(Y\). We get to fit a single \(\hat f_D\), but we never see the true \(f\).
We can approximate the generalized expected error using training/test splits on the data.