Gaussian Distribution

Gaussian Distribution

One-dimensional

Suppose \(1\)-d random variable \(x \sim N(\mu, \sigma^2)\), then its density function is \[ p(x) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}(\frac{x - \mu}{\sigma})^2} \]

To verify that it integrates to \(1\), \[ \begin{aligned} \int_{-\infty}^{+\infty} p(x) \d x &= \sqrt{(\int_{-\infty}^{+\infty} p(x) \d x) \cdot (\int_{-\infty}^{+\infty} p(y) \d y)} \\ &= \sqrt {\int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} p(x)p(y) \d x \d y} \\ \end{aligned} \]

Let \(I^2 = \int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty} p(x)p(y) \d x \d y\), \[ \begin{gather} \int_{-\infty}^{+\infty} p(x) \d x = \sqrt{I^2} \\ I^2 = \int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2} (\frac{x - \mu}{\sigma})^2} \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2} (\frac{y - \mu}{\sigma})^2} \d x \d y \end{gather} \] Let \(u = \frac{x-\mu}{\sigma}, v = \frac{y-\mu}{\sigma}\), \[ \begin{aligned} I^2 &= \int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2}u^2}\frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2}v^2} \d u \d v \\ &= \frac{1}{2\pi} \int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} e^{-\frac{1}{2}(u^2 + v^2)} \d u \d v \end{aligned} \] Let \(u = r \sin \theta, v = r \cos \theta\), \[ \begin{aligned} \int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} e^{-\frac{1}{2}(u^2+v^2)} \d u \d v &= \int_{0}^{2\pi} \int_{0}^{+\infty} e^{-\frac{1}{2} (r^2 \sin^2\theta + r^2 \cos^2\theta)} r \d r \d\theta \\ &= \int_{0}^{2\pi} \int_{0}^{+\infty} -e^{-\frac{1}{2} r^2} \d(-\frac{1}{2} r^2) \d\theta \\ &= \int_{0}^{2\pi}-e^{t}\Big|_{t=0}^{t=-\infty}d\theta \\ &= \int_{0}^{2\pi}d\theta \\ &= 2\pi \end{aligned} \] Therefore, \(I^2 = \frac{1}{2\pi}2\pi = 1\), \(\int_{-\infty}^{+\infty}p(x)dx = \sqrt{I^2} = 1\)

Independent standard \(n\)-dimensional

Let \(Z = [Z_1, Z_2, ..., Z_n]^T\), suppose \(Z_i, Z_j (i,j=1,...,n \and i \ne j)\) are independent and \(Z_i (i=1,...,n)\) observes standard Gaussian distribution, we can derive the joint distribution density function for random variable \(Z\) to be \[ \begin{aligned} \newcommand{z}{\mathrm{z}} p(\z) &= p(z_1, z_2, ..., z_n) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2} z_i^2} \\ &= \frac{1}{(2\pi)^{\frac{n}{2}}} e^{-\frac{1}{2}\z^T\z} \\ \end{aligned} \]

First-order correlated \(n\)-dimensional

We have given the joint distribution function of independent \(n\)-dimensional standard Gaussian distribution. What if \(n\) dimensions of \(Z\) are not standard, are not independent with each other, but are correlated only in first order?

We may begin with standard Gaussian random variables \(X = [X_1, \dots, X_n]\). Then we can shift \(X\) by \(\mu\) and linearly transform it with an invertible matrix \(B^{-1}\). \(Z\) mentioned above will just be such a matrix as \(Z = B^{-1} (X - \mu)\) and \[ \begin{gather} X = B^{-1}(Z - \mu) \sim \mathcal{N}(0, I) \\ p_X(\x) = \frac{1}{(2\pi)^{\frac{n}{2}}} e^{-\frac{1}{2} \x^T\x} \end{gather} \]

Suppose \(Z\) is to take on values in \(\mathcal{Z}\), which is a subset of \(\R^{n}\), \[ \newcommand{z}{\mathrm{z}} P_Z(Z \in \mathcal{Z}) = \int_\mathcal{Z} p_Z(\z) \d \z \] Since \(Z = f(X) = BX + \mu\) and \(B\) is invertible, the mapping \(X \to Z\) is one-to-one, therefore the multivariate Jacobian transformation is \[ J(X \to Z) = B^{-1} \\ \] with its determinant \(J = |J(X \to Z)| = |B^{-1}| = |B|^{-1}\). Note that \[ \begin{aligned} |J| &= \sqrt{|B|^{-1}|B|^{-1}} \\ &= \sqrt{|B|^{-1}|B^T|^{-1}} \\ &= \sqrt{|BB^T|^{-1}} \\ &= |BB^T|^{-\frac{1}{2}} \end{aligned} \] Therefore, \[ \begin{aligned} P_Z (Z \in \mathcal{Z}) &= P_X (X \in f^{-1}(\mathcal{Z})) \\ P_Z (Z \in \mathcal{Z}) &= \int_{f^{-1}(\mathcal{Z})} p_X(\x) \d \x \\ &\Downarrow_{\x = f^{-1}(\z)} \\ \int_\mathcal{Z} p_Z (\z) \d \z &= \int_\mathcal{Z} p_X (f^{-1}(\z)) |J| \d \z \\ \end{aligned} \]

\[ \begin{aligned} &p_Z (\z) = p_X (f^{-1}(\z))|J| = p_X (B^{-1} (\z - \mu) |J| \\ &= \frac{1}{(2\pi)^{\frac{n}{2}}} e^{-\frac{1}{2} (\z-\mu)^T(B^{-1})^T B^{-1} (\z-\mu)} |BB^T|^{-\frac{1}{2}} \\ &= \frac{1}{(2\pi)^{\frac{n}{2}}} e^{-\frac{1}{2} (\z-\mu)^T (B^T)^{-1} B^{-1} (\z-\mu)} |BB^T|^{-\frac{1}{2}} \\ &= \frac{1}{(2\pi)^{\frac{n}{2}}} e^{-\frac{1}{2} (\z-\mu)^T (BB^T)^{-1} (\z-\mu)} |BB^T|^{-\frac{1}{2}} \\ &= \frac{1}{(2\pi)^{\frac{n}{2}}|BB^T|^\frac{1}{2}} e^{-\frac{1}{2} (\z-\mu)^T (BB^T)^{-1} (\z-\mu)} \\ \end{aligned} \]

Also note that \[ \begin{aligned} \Sigma_Z &= E[(\z-\mu) (\z-\mu)^T] \\ &= E[B X X^T B^T] \\ &= B E[X X^T] B^T \\ &= B I B^T \\ &= B B^T \end{aligned} \]

Thus, \[ p_Z(\z) = \frac{1}{\sqrt{(2\pi)^n|\Sigma_Z|}} e^{-\frac{1}{2} (\z-\mu)^T \Sigma_Z^{-1} (\z-\mu)} \]

External Materials

为什么高斯分布概率密度函数的积分等于1 - 知乎 (zhihu.com) || 多元高斯分布完全解析 - 知乎 (zhihu.com)