LLM Primer

Solutions

MATH-C11-C-001

A certain event has no uncertainty, so its entropy is 0 bits.

MATH-C11-C-002

Use the joint entropy identity:

H(X,Y)=H(X)+H(Y∣X)=4+2=6H(X, Y) = H(X) + H(Y \mid X) = 4 + 2 = 6

MATH-C11-C-003

Since 0.125 = 1/8 = 2^-3:

log⁡2(0.125)=−3\log_2(0.125) = -3

So:

−log⁡2(0.125)=3-\log_2(0.125) = 3

MATH-C11-C-004

Perplexity from loss in bits is:

2loss=24=162^{\text{loss}} = 2^4 = 16

MATH-C11-C-005

Mutual information is:

I(X;Y)=H(Y)−H(Y∣X)=7−3=4I(X;Y) = H(Y) - H(Y \mid X) = 7 - 3 = 4

MATH-C11-C-006

For a one-hot target, cross-entropy reads the probability assigned to the correct, or target, class. The loss becomes:

−log⁡qcorrect-\log q_{\text{correct}}

MATH-C11-C-007

No. KL divergence is directional. In general:

DKL(p∥q)≠DKL(q∥p)D_{\mathrm{KL}}(p \parallel q) \ne D_{\mathrm{KL}}(q \parallel p)

MATH-C11-C-008

No. Perplexity depends on the dataset and tokenizer. If the tokenization changes, the prediction units change, so the numbers should not be compared casually.

MATH-C11-C-009

No. Mutual information says that knowing one variable reduces uncertainty about another. It does not by itself show that one variable causes the other.

MATH-C11-C-010

Yes. At each position, a language model predicts a distribution over possible next tokens. That is classification over the vocabulary for that prediction step.

MATH-C11-C-011

Enter 1.

Low entropy means the model's probability distribution is concentrated. It does not prove the concentrated probability is on the correct outcome.

MATH-C11-C-012

Enter 1.

Accuracy treats both predictions as correct if the top class is right. Cross-entropy still distinguishes them by the probability assigned to the target class.

MATH-C11-C-013

Enter 1.

In D_KL(p || q), the terms are weighted by p. The mismatch is measured from p's reference view.

MATH-C11-C-014

Enter 1.

Perplexity is computed from average negative log-likelihood on a specific evaluation setup. If the dataset or tokenizer changes, the prediction units change, so the numbers should not be compared casually.

MATH-C11-C-015

Enter 1.

Good prediction and compression can show that a model has learned useful regularities. They do not by themselves prove truthfulness, safety, alignment, or usefulness in a particular setting.