Information Theory, Inference and Learning Algorithms
-
Introduction
The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point (Claude Shannon, 1948). Similar to that stated above, the fundamental subject of machine learning is about with best effort recovering the original data using the so-called model or algorithm.
-
Source Coding Theorem
Notations and Concepts An ensemble \(X\) (we specifically use the random variable symbol to denote the ensemble) is a triplet \((X, \newcommand{A}{\mathcal A} \newcommand{P}{\mathcal P} \A_X, \P_X)\), where \(X\) denotes
-
Symbol Code
Other than block code where symbols are encoded in chunks, symbol code will assign each symbol a unique codeword. Among the codeword schemes, we prefer those where no codeword is a prefix of any other codeword.