Self-Information and Bits

Derive logarithmic surprise from probability, compare bits and nats, interpret fractional ideal code lengths, handle zero-probability failures, and add conditional costs across sequences.

Self-information measures the surprise of one observed outcome relative to a stated probability model. A likely outcome carries little new information when it occurs; an unlikely outcome carries more. The logarithm makes independent information additive and gives the quantity an operational interpretation as an ideal code length.

Surprise Depends on Probability

Suppose an outcome $x$ has probability $p(x)$. Its self-information, also called surprisal, is

Three boundary behaviors agree with ordinary reasoning about surprise:

• if $p(x)=1$, then $I(x)=0$ because the outcome was certain;
• if $0<p(x)<1$, then $I(x)>0$;
• as $p(x)$ approaches zero, $I(x)$ grows without bound.

For a weather model with $p(\text{rain})=0.2$, observing rain carries

Observing no rain under the same model carries

Rain is more informative in this technical sense because the model considered it less likely. This statement depends on the model and does not say that rain is more important to every person or decision.

Why Information Uses a Logarithm

A useful numerical measure of surprise should behave in three ways:

1. it should vary continuously as probability changes;
2. a less probable outcome should carry more information; and
3. independent outcomes should have additive information.

For independent outcomes $x$ and $y$, $p(x,y)=p(x)p(y)$. A logarithm converts this product into a sum:

Under mild regularity conditions, these requirements lead to a negative logarithm up to a positive choice of scale. The logarithm is therefore not an arbitrary decoration added to probability: it expresses how independent possibilities combine.

Two independent fair coin results have joint probability $1/4$. Their information is $-\log_2(1/4)=2$ bits, equal to one bit for each result. Ten independent fair coin results similarly carry ten bits.

The Logarithm Base Sets the Unit

With base $2$, information is measured in bits. Each bit corresponds to one binary distinction in an ideal description. With base $e$, the unit is the nat. Base $10$ gives hartleys.

The units are converted by a constant factor:

Halving a probability always adds one bit:

An equation or loss reported without its logarithm base may still be useful, but its numerical unit is incomplete. Many mathematical and machine-learning libraries use natural logarithms, so their log losses are measured in nats unless converted.

Ideal Code Length May Be Fractional

The probability $p(x)=0.3$ gives

An isolated binary codeword cannot literally contain $1.737$ bits. The value is an ideal length. Prefix-free codes for individual symbols use whole-number lengths, while block codes and arithmetic coding can approach fractional average lengths over long sequences.

For example, a sequence of $100$ occurrences of outcomes that each receive probability $0.3$ has ideal total length

The sequence can use a whole number of bits while its average approaches $1.737$ bits per outcome. Information theory commonly makes statements about long-run or expected length rather than promising an exact fractional codeword for one symbol.

A Model Assigns the Surprise

Self-information is not a property of an outcome by itself. It is calculated from a probability distribution. If model $q_A$ assigns probability $0.8$ to an observed token, its cost is

If model $q_B$ assigns probability $0.1$ to the same observed token, its cost is

Model B pays three extra bits for that outcome. A model is therefore penalized strongly when it assigns little probability to what occurs. This is the local, one-outcome form of logarithmic loss.

Zero Probability Creates Infinite Loss

If a model assigns $q(x)=0$ to an outcome that occurs, then

The issue is not merely numerical. A zero probability says that the event is impossible under the model, so a code based on that model has not reserved a finite representation for it. In practice, probabilistic systems must handle support carefully. Smoothing, probability floors, or model structures with positive probability can prevent unintended zeros, although each choice changes the model and must be justified.

A zero-probability outcome under the true source distribution is different: it never occurs under that source and therefore contributes nothing to an average over source outcomes. Problems arise when the source can produce an outcome to which the predictive model assigns zero.

Sequence Information Uses Conditional Probabilities

Independence is not required to calculate the information in a sequence. The probability chain rule writes

Taking a negative logarithm turns the product into a sum:

Suppose a model assigns

The sequence probability is $(1/2)(1/4)(1/8)=1/64$. Its information is

which also equals $1+2+3$ bits from the three conditional predictions. This additive token-level form is the basis of autoregressive language-model log loss.

One Outcome and an Average Are Different

Self-information describes one realized outcome under a stated model. It does not measure semantic meaning, practical importance, truth, or usefulness. A random identifier may be highly surprising yet convey little useful structure, while a predictable warning may be extremely important to act upon.

Before observing an outcome, we can average its possible self-information values using their source probabilities. That expected surprise is entropy, the subject of the next lesson. Keeping the distinction clear prevents us from calling the surprise of one rare event “the entropy” and from confusing a model's local log loss with the source's average uncertainty.