Binary Cross-Entropy

Binary cross-entropy is used for binary classification when the model outputs a probability for class 1.

Let:

y = target, either 0 or 1
p = predicted probability of class 1

The binary cross-entropy loss is:

loss = -[y log(p) + (1 - y) log(1 - p)]

For this chapter, focus on the two cases.

If the true target is 1:

loss = -log(p)

If the true target is 0:

loss = -log(1 - p)

The loss is small when the model assigns high probability to the correct class.

Small numerical check

If the target is 1 and the model predicts p = 0.8, then:

loss = -log(0.8)

This is smaller than:

-log(0.2)

because predicting 0.8 for the true class is better than predicting 0.2.

Exercise: Choose the smaller loss

The target is 1. Which predicted probability gives smaller binary cross-entropy: 0.9 or 0.2? Enter the smaller-loss probability.

Compute it first, then check your number.

HintTrue class is 1

For target 1, the loss is -log(p).

SolutionWork it out

Since the target is 1, the model should assign high probability to class

  1. 0.9 gives a smaller loss than 0.2.
Exercise: Target zero case

The target is 0, and the model predicts probability p = 0.1 for class 1. What probability did the model assign to the correct class 0?

Compute it first, then check your number.

HintBinary probabilities sum to one

Class 0 probability is 1 - p.

SolutionWork it out

The model assigns p = 0.1 to class 1, so it assigns 1 - 0.1 = 0.9 to class 0.