Quadratic Form

Quadratic Form

Quadratic form involves many concepts like real symmetric matrix, positive definiteness and singular value decomposition. It can be quite helpful to glue these things together.

A quadratic function \(f\) of \(n\) variables, or say a vector \(\x\) of length \(n\), is the sum of second-order terms: \[ f(\x) = \sum_{i=1}^n \sum_{j=1}^n c_{ij} x_i x_j \]

The above summation can be simplified as matrix product \(\x^T A \x\) where \(A\) is \(n \times n\) and \(a_{ij} = c_{ij}\). \(\x^T A \x\) is called the quadratic form of \(A\).

In essence, there is a nicer formulation for \(A\). Firstly define the \(n \times n\) matrix \(A\) such that \(a_{ij} = \frac{1}{2} (c_{ij} + c_{ji})\). It suffices to show \(A\) is a real symmetric matrix and \[ f(\x) = \x^T A \x \]

Thus the discussion of quadratic form usually encompasses the real symmetric matrix.

Positive Definiteness

Let \(A\) be a real symmetric matrix. \(A\) is positive definite if and only if the quadratic form of \(A\) is positive. Specifically, for every \(\x \ne 0\), \(\x^T A \x > 0\).

• Theorem

  A real symmetric matrix \(A\) is positive definite if and only if \(A\)’s eigenvalues are positive.

• Proof

  • Necessity

    For every \(A\)’s eigenpair \((\lambda, \v)\), we have \(\v^T A \v = \lambda \v^T \v > 0 \Rightarrow \lambda > 0\).

  • Sufficiency

    Take \(A\)’s spectral decomposition as \(A = Q \Lambda Q^T\) where \(Q Q^T = I\). For every \(\x > 0\), we have \[ \x^T A \x = \overbrace{\x^T Q}^{y^T} \Lambda \overbrace{Q^T \x}^{y} > 0 \]

One possible reason why we would like to define positive definiteness on real symmetric matrix is that, any real matrix can be easily decomposed into the addition of a real symmetric matrix and a real skew-symmetric matrix whose quadratic form is zero: \[ A = \frac{A + A^T}{2} + \frac{A - A^T}{2} \] Since \(\frac{A - A^T}{2}\)’s quadratic form is zero and it makes no contribution to \(A\)’s, the only component of interest will be the real symmetric \(\frac{A + A^T}{2}\). So why not just focus on the real symmetric matrix?