Spectral Normalization
Spectral normalization of an \(M \times N\) matrix \(A\) is defined as \[ ||A||_2 = \max_{\mathrm z} \frac{||A\mathrm z||_2}{||\mathrm z||_2} = \sqrt{\lambda_{\max}(A^TA)} = \sigma_{\max}(A) \] where \(\rm z \in \R^N\), \(\lambda_{\max}(A^TA)\) is the maximum eigenvalue of matrix \(A^TA\), which is exactly \(A\)’s largest singular value \(\sigma_{\max}(A)\).
To prove it, firstly note that: \[ \max_{\mathrm z} \frac{||A\mathrm z||_2}{||\mathrm z||_2} \iff \max_{\mathrm z} \frac{||A\mathrm z||^2_2}{||\mathrm z||^2_2} \] We may force a constraint on \(\mathrm z\) such that \(||\mathrm z||^2_2 = 1\). This is because \[ \frac{||Ac\mathrm z||^2_2}{||c\mathrm z||^2_2} = \frac{c^2||A\mathrm z||^2_2}{c^2||\mathrm z||^2_2} = \frac{||A\mathrm z||^2_2}{||\mathrm z||^2_2} \] The problem becomes \[ \begin{gather} \max_{\mathrm z} \frac{||A\mathrm z||^2_2}{||\mathrm z||^2_2} = ||A\mathrm z||^2_2 = \mathrm z^TA^TA\mathrm z \\ s.t. ||\mathrm z||^2_2 = 1 \end{gather} \] This can be solved by Lagrange multiplier, where the Lagrangian function will be \[ L(\mathrm z, \lambda) = \mathrm z^TA^TA\mathrm z + \lambda(||\mathrm z||^2_2 - 1) \]
The extrapolation of the spectral normalization will be related to Rayleigh quotient.