Bias-variance Decomposition

The notation used is as follows:

Symbol	Notation
$D$	the dataset
$x$	the sample
$y_{D}$	the observation of $x$ in $D$ , affected by noise
$y$	the real value of $x$
$\bar{y}$	the mean of the real values
$f$	the model learned with $D$
$f (x)$	the prediction of $f$ with $x$
$\bar{f} (x)$	the expectation of prediction of $f$ with $x$
$l (f (x), y_{D})$	the loss function, chosen to be squared error

By assuming that the observation errors averages to $0$ , the expectation of the error will be $\begin{matrix} (1) & \begin{aligned} E_{x \sim D} & [l (f (x), y_{D})] = E [(f (x) - y_{D})^{2}] = E {[(f (x) - \bar{f} (x) + (\bar{f} (x) - y_{D})]^{2}} \\ = E [(f (x) - \bar{f} (x))^{2}] + E [(\bar{f} (x) - y_{D})^{2}] + 2 E [(f (x) - \bar{f} (x)) (\bar{f} (x) - y_{D})] \\ = E [(f (x) - \bar{f} (x))^{2}] + E {[(\bar{f} (x) - y) + (y - y_{D})]^{2}} \\ + 2 \underset{0}{\underset{⏟}{E [f (x) - \bar{f} (x)]}} E [\bar{f} (x) - y_{D}] \\ = E [(f (x) - \bar{f} (x))^{2}] + E [(\bar{f} (x) - y)^{2}] + E [(y - y_{D})^{2}] + 2 E [(\bar{f} (x) - y) (y - y_{D})] \\ = E [(f (x) - \bar{f} (x))^{2}] + E [(y - y_{D})^{2}] + E {[(\bar{f} (x) - \bar{y}) + (\bar{y} - y)]^{2}} \\ + 2 E [\bar{f} (x) - y] \underset{0}{\underset{⏟}{E [y - y_{D}]}} \\ = E [(f (x) - \bar{f} (x))^{2}] + E [(y - y_{D})^{2}] + E {[(\bar{f} (x) - \bar{y}) + (\bar{y} - y)]^{2}} \\ = E [(f (x) - \bar{f} (x))^{2}] + E [(y - y_{D})^{2}] + E [(\bar{f} (x) - \bar{y})^{2}] + E [(\bar{y} - y)^{2}] + 2 E [(\bar{f} (x) - \bar{y}) (\bar{y} - y)] \\ = E [(f (x) - \bar{f} (x))^{2}] + E [(y - y_{D})^{2}] + E [(\bar{f} (x) - \bar{y})^{2}] + E [(\bar{y} - y)^{2}] \\ + 2 E [\bar{f} (x) - \bar{y}] \underset{0}{\underset{⏟}{E [\bar{y} - y]}} \\ = \underset{v a r i a n c e}{\underset{⏟}{E [(f (x) - \bar{f} (x))^{2}]}} + \underset{b i a s^{2}}{\underset{⏟}{E [(\bar{f} (x) - \bar{y})^{2}]}} + \underset{n o i s e}{\underset{⏟}{E [(y - y_{D})^{2}]}} + \underset{s c a t t e r}{\underset{⏟}{E [(\bar{y} - y)^{2}]}} \end{aligned} \end{matrix}$ 5 ways to achieve right balance of Bias and Variance in ML model | by Niwratti Kasture | Analytics Vidhya | Medium

Last updated on Jan 7, 2022