Exercises

Practice architecture cards, sparse edge counts, long-context claims, KV cache and MoE ledgers, multimodal fusion, state-space classification, scaling, compute, and system routing.

Each exercise is standalone. State assumptions with every numerical answer and do not turn an analytical result into an unmeasured performance claim.

Q1. Complete an architecture card

A decoder keeps its blocks and objective but replaces dense causal attention with a local causal window. State the changed component and one invariant.

Choose the changed component and an invariant

The visibility graph changed; scaled attention over allowed edges and the surrounding decoder block remain.

The training objective changed; the visibility graph is the invariant.

Only retrieval outside the model changed; every layer is invariant.

Select one choice, then check.

Hint

Separate allowed edges from the operation performed on allowed edges.

Solution

The visibility graph changed. Scaled attention over allowed keys and the surrounding decoder block remain.

Q2. Compare dense and local edges

For $T=6$ and $w=2$ , calculate dense causal and local causal edge counts.

Your answer

Answer it first, then check.

Hint

Use

T(T+1)/2

and sum

[1,2,3,3,3,3]

Solution

Dense attention has

6(7)/2=21

edges. Local attention has

1+2+3+3+3+3=15

Q3. Audit a long-context claim

A benchmark places one key near the start of a 32,000-token prompt and shows 90% retrieval accuracy. It does not test multi-step reasoning. Which claims are supported and unsupported?

Choose the supported and unsupported claims

Distant retrieval in that benchmark is supported; general long-context reasoning is unsupported.

The result establishes reliable long-context reasoning for every task.

Only input acceptance is established; the retrieval measurement provides no additional evidence.

Select one choice, then check.

Hint

Do not replace a retrieval measurement with a broader reasoning claim.

Solution

Distant retrieval under the benchmark conditions is supported. General long-context reasoning is not established.

Q4. Compare MHA and MQA cache size

For $B=1,L=6,T=20,d_h=8$ , calculate K/V cache scalars for MHA with eight K/V heads and MQA with one K/V head.

Your answer

Answer it first, then check.

Hint

Use

2LBh_{kv}Td_h

twice.

Solution

MHA stores

2(6)(1)(8)(20)(8)=15360

scalars. MQA stores

2(6)(1)(1)(20)(8)=1920

Q5. Audit an MoE parameter claim

Eight experts have 100 parameters each, a router has 32, and top-1 routing selects one expert. Give total stored parameters and selected expert parameters per token. Name one omitted cost.

Choose the complete ledger

832 total; 100 selected expert parameters; routing is one omitted cost.

100 total; 100 active; no other work is omitted.

832 total; all 800 expert parameters are active for each token.

Select one choice, then check.

Hint

Add all experts and router for storage; then select one expert.

Solution

Total storage is

8(100)+32=832

. One token selects 100 expert parameters. Routing, dispatch, communication, capacity, or imbalance is an acceptable omitted cost.

Q6. Separate tokenization from fusion

Model A projects image patches and processes only image records. Model B uses separate image and text encoders joined by cross-attention. Which change defines B beyond image tokenization?

Choose one

A cross-modal fusion contract joins separate streams.

Only the image patch size changed.

Only the K/V cache dtype changed.

Select one choice, then check.

Hint

Look for the operation that connects modalities.

Solution

B adds a cross-modal fusion contract through cross-attention.

Q7. Distinguish a sequence alternative

A model has no query-key scores and carries a recurrent state through the sequence. Is it a sparse Transformer merely because it processes tokens?

Choose one

No. It uses an alternative recurrent state-space sequence mixer.

Yes. Any sequence model without dense attention is a sparse Transformer.

Yes. Its recurrent state is simply a differently shaped Transformer KV cache.

Select one choice, then check.

Hint

Sparse attention retains query-key interactions on fewer edges.

Solution

No. It uses an alternative recurrent state-space sequence mixer rather than a sparse attention graph.

Q8. Calculate and limit a scaling fit

Under $L(C)=AC^{-\alpha}$ , two illustrative measurements are $(C,L)=(1,3)$ and $(9,1)$ . Calculate $\alpha$ . Does this validate predictions beyond $C=9$ ?

Your answer

Answer it first, then check.

Hint

\log(3/1)/\log(9/1)=\log 3/\log 9

Solution

\alpha=1/2=0.5

. Two points do not validate behavior beyond the measured range.

Q9. Compare equal-compute allocations

Under $C/k=ND=6000$ , compare $(N,D)=(100,60)$ and $(200,30)$ . Which uses more compute, and which must have lower loss?

Choose the compute and loss conclusion

They use equal simplified compute; neither is guaranteed to have lower loss.

The 200-parameter allocation uses more compute and must have lower loss.

The 60-token allocation uses more compute and must have lower loss.

Select one choice, then check.

Hint

Multiply each pair, then ask whether an empirical loss relation was provided.

Solution

Both products equal 6000. Neither allocation is guaranteed to have lower loss without an empirical model and its conditions.

Q10. Route four LLM changes

Route these changes to their primary system layers: instruction tuning, vector-database lookup, temperature 0.7, and tensor-parallel serving.

Choose the four layers in order

post-training; retrieval and tools; inference policy; infrastructure

base architecture; post-training; retrieval and tools; evaluation

pretraining data; product policy; base architecture; inference policy

Select one choice, then check.

Hint

Ask whether each change alters learned behavior, external evidence, logit selection, or execution placement.

Solution

The layers are post-training; retrieval and tools; inference policy; and infrastructure.

Q1. Complete an architecture card

Q2. Compare dense and local edges

Q3. Audit a long-context claim

Q4. Compare MHA and MQA cache size

Q5. Audit an MoE parameter claim

Q6. Separate tokenization from fusion

Q7. Distinguish a sequence alternative

Q8. Calculate and limit a scaling fit

Q9. Compare equal-compute allocations

Q10. Route four LLM changes

Pause and reflect