Surprise is the single event quantity. Entropy averages it under the true distribution. Cross entropy averages it using the probabilities of the model. KL is the extra cost of using the model. Negative log likelihood and binary cross entropy are the forms this takes for particular observations and distributions.

Requirements for a surprise function

Start with one event $x$. Suppose we know its probability $p(x)$, and we want a number that says how surprising it is to see that event happen.

So we want a function from probability to surprise. If the event has probability $p(x)$, write its surprise as $S(p(x))$.

The function should satisfy three basic requirements.

First, rarer events should be more surprising. If $x_1$ is less likely than $x_2$, then seeing $x_1$ should carry more surprise:

Second, independent surprises should add. If two independent events happen together, their joint probability is the product of their probabilities:

The surprise of seeing both should be the surprise of seeing one plus the surprise of seeing the other:

Third, the function should be continuous. A tiny change in probability should not create a sudden jump in surprise.

These requirements are already very restrictive. They say surprise should decrease as probability increases, turn multiplication of independent probabilities into addition of surprises, and behave smoothly.

Finding a function that satisfies these requirements

The second requirement is the important one. For independent events, probabilities multiply:

But surprises should add:

So the surprise function has to turn multiplication into addition:

That is exactly what logarithms do:

Continuity rules out strange discontinuous functions that satisfy the same algebraic identity but do not behave like a measurement. So the surprise function must be a constant multiple of a logarithm:

Probabilities are between 0 and 1, so $\log p$ is non-positive. But surprise should be non-negative, and rarer events should have larger surprise. That fixes the sign:

The constant $k$ must be positive and only chooses the unit. If we choose base 2 and set $k = 1$, surprise is measured in bits:

Entropy as average surprise across a distribution

Surprise belongs to an event after it happens. But before sampling from a distribution, we can still ask how surprised we expect to be.

Suppose $X$ is sampled from a distribution $p$. If the outcome is $x$, its surprise is:

But outcome $x$ happens with probability $p(x)$. So the expected surprise is the weighted average of all possible surprises:

Substituting the surprise function gives:

or equivalently:

This expected surprise is the entropy of the distribution:

A low entropy distribution is concentrated on a few outcomes whereas a high entropy distribution tends to be more uniform.

Cross entropy as expected surprise using another distribution

Entropy assumes the probabilities used to measure surprise are the same probabilities that generate the data. If the true distribution is $p$, then outcome $x$ has surprise $-\log_2 p(x)$, and the average surprise is:

But a model usually does not know $p$. It has its own distribution $q$. If the world still samples outcomes from $p$, but the model measures surprise using $q$, then the surprise of outcome $x$ becomes:

The average is still taken over what actually happens, so outcomes are weighted by $p(x)$:

This is cross-entropy. It is the expected surprise when reality follows $p$, but the probabilities used to encode or predict it come from $q$.

If $q = p$, cross-entropy becomes ordinary entropy:

If $q$ puts too little probability on events that actually happen, those events become overly surprising under the model, and cross-entropy increases.

KL divergence as extra surprise

Cross entropy tells us how surprised we are when reality follows $p$, but we measure surprise using $q$.

Entropy tells us how surprised we would be if we used the correct distribution $p$.

So the extra surprise from using $q$ instead of $p$ is:

Now expand both terms:

Combine the sums:

Use the logarithm rule:

This quantity is called the KL divergence from $p$ to $q$:

So cross entropy splits into two parts:

The first part is the unavoidable surprise in the data. The second part is the extra surprise paid because the model distribution is not the true distribution.

Why minimizing cross entropy also minimizes KL divergence

The decomposition also explains why minimizing cross-entropy and minimizing KL divergence often mean the same thing in training.

Suppose $p$ is the data distribution and $q$ is the model distribution. The data distribution is fixed. The model can change $q$, but it cannot change $p$.

In:

the entropy term $H(p)$ does not depend on the model. It is an additive constant. So choosing the model that minimizes cross-entropy is the same as choosing the model that minimizes KL divergence from the data distribution to the model distribution:

Cross entropy is the model-surprise view. KL divergence is the distribution-mismatch view. The optimization is the same because they differ only by the fixed entropy of the data.

Negative log likelihood as cross entropy with one hot targets

In classification, there is a small shift in what the distribution is over. Once an input $x$ is fixed, the model predicts a distribution over labels:

For example, the labels might be cat, dog, and horse. If the training label says the image is a cat, we usually define the target distribution as one-hot:

The cross-entropy for this one example is:

All labels except cat have weight zero, so their terms disappear:

So ordinary classification loss is still cross-entropy. We have just chosen a target distribution that puts all probability mass on the observed label.

Across a dataset, we average this model surprise over examples. Here $x_i$ is the input and $y_i$ is its ground-truth label:

Soft labels fit the same formula. If an image is ambiguous, or if a teacher model gives probabilities, the target distribution does not have to put all mass on one class. Multiple labels can contribute to the same cross-entropy.

Thus, cross-entropy with one-hot targets is equivalent to negative log-likelihood.

Binary cross entropy as bernoulli negative log likelihood

Binary cross-entropy is cross-entropy for a binary target. If the model predicts $\hat p = q_\theta(y = 1 \mid x)$, then the two-class model distribution is:

For a binary target $y$, the target distribution is:

So binary cross-entropy is:

With hard binary labels, where $y$ is either 0 or 1, BCE is also the NLL. With soft binary labels, BCE is still cross-entropy, but it is no longer one-hot NLL.