The transition matrix is:
An alphabet is a set of elements called letters or symbols. If the number of elements is finite, the alphabet is said to be discrete.
example:
A well known alphabet is the binary
alphabet which contains both binary digits "0" and "1".
In Information Theory, a channel is a mathematical model for the devices which allow to transmit data from a location to another. Discrete memoryless channels are the simplest class of channels. They consist of:
The binary symmetric
channel with cross over probability p.
The transition matrix is:
If we denote X (resp Y) the input (resp output), we get:
The capacity of a channel is defined as the largest average information provided about the input X by the output Y (IYàX), maximized over all input probability laws. As it can be showed that IYàX = IXàY, we denote I(X;Y) = IYàX = IXàY (it is called average mutual information between X and Y), we get:
example:
capacity of the binary
symmetric channel with cross over probability p.
The calculations lead to:
From this sketch, we can see that the capacity is maximum when p = 0 or p = 1 (in these cases we know which one of the symbols has been transmitted). The capacity is zero for p = 0.5 since Y does not provide any information about the input symbol which may be either "0" or "1" with the same probability.
Shannon's second theorem shows that the capacity of a
channel is the maximum amount of information that can be transmitted through
it, with an arbitrarily small but non zero probability of error.
A gaussian channel is a discrete time channel with output Y linked to input X so that Y = X + Z , where Z is a gaussian random variable with mean 0 and variance N.
If we assume an average power constraint P on the input, it can be showed that the capacity (in bits per second) is:
An information source is a device which is able to deliver randomly a series of symbols from an alphabet. If the alphabet contains a finite number of elements, the source is said to be discrete, otherwise it is continuous.
example:
A computer connected to the world wide web can be considered as a binary source (the adjective binary refers to the number of elements in the alphabet) if we look at the outcomes before the modem (modulator demodulator). After the modem, the information source is continuous.
