Law of Total Variance

Conditional Expectation

Let \(X\) and \(Y\) be two discrete random variables. The conditional probability function of \(X\) given \(Y=y\) is \[ \Pr(X=x|Y=y) = \frac{\Pr(X=x, Y=y)}{P(Y=y)} \] Thus the conditional expectation of \(X\) given that \(Y=y\) is \[ \E(X|Y=y) \coloneq \sum_x x \Pr(X=x|Y=y) \] Clearly the conditional expectation \(\E(X|Y)\) is a function of \(Y\), or put it another way, a random variable depending on \(Y\), instead of \(X\).

Conditional Variance

Conditional variance can be similarly defined. \(\Var(X|Y=y)\) is the conditional variance of \(X\) given \(Y=y\) and \(\Var(X|Y)\) is a random variable depending on \(Y\): \[ \Var(X|Y) \coloneq \E[(X - \mu_{X|Y})^2 | Y] = \E(X^2|Y) - \E(X|Y)^2 \]

Laws of Total Expectation and Variance

If all the expectations below exist, then for any random variable \(X\) and \(Y\), we have \[ \E(X) = \E_{y \sim p_Y} [\E(X|Y=y)] \quad \textbf{Law of Total Expectation} \] and \[ \Var(X) = \E_{y \sim p_Y} [\Var(X|Y=y)] + \Var_{y \sim p_Y} [\E(X|Y=y)] \quad \textbf{Law of Total Variance} \]