Milestone 6 of 7

Inspect and ablate one prediction

Verify an internal trace, inspect one prediction, intervene on one named component, and bound the interpretation.

Turn one model output into a controlled interpretation experiment. The goal is not to assign a dramatic name to a component; it is to establish a trace, change one declared value, and make the smallest claim the evidence supports.

Milestone goal

Choose one checkpoint, prompt pair, output position, target, alternative, scalar metric, recorded sites, and replacement rule. Produce a faithful trace, one intervention table, a bounded interpretation, an alternative account, and one test that could distinguish the accounts.

Freeze the question

The checked reference uses:

clean:     <bos> X Y
corrupted: <bos> Y X
position:  final prompt position
metric:    logit(X) - logit(Y)

Both prompts are valid corpus sequences. “Corrupted” names the comparison run relative to this X-over-Y question; it does not mean malformed input.

The clean margin is 6.4618 and the corrupted margin is -5.7350. Preserve the complete probability rows as well as the selected margin.

Prove trace fidelity before interpreting it

An explicit trace may calculate attention separately so that it can retain per-head values. Compare its complete logits with the ordinary model forward pass. The checked maximum difference is $9.5367\times10^{-7}$ under tolerance $10^{-5}$ , and every attention row sums to one within $10^{-6}$ .

Record exact sites and shapes:

Recorded value	Shape
residual record	$(1,3,4)$
attention weights	$(1,2,3,3)$
per-head write in model space	$(1,2,3,4)$
MLP preactivation and activation	$(1,3,8)$
logits	$(1,3,8)$

If fidelity fails, fix masking, positions, normalization order, model mode, or axis labels before using the trace as evidence.

Separate three observations

Inspect at least:

one attention row, including its value and output path;
one attention or MLP write to the residual stream;
one intermediate residual projected through the frozen final norm and vocabulary readout.

The final query in layer 2 head 1 assigns weight 0.7685 to position 1 in the checked clean run. The diagnostic X-minus-Y projection changes sign after layer 2 attention. These are useful observations, but neither alone proves a mechanism.

Define and run the intervention

Replace one named component's model-space write with zero or another declared baseline. Measure

\Delta m=m_{intervention}-m_{clean}.

State layer, head or branch, position, tensor site, replacement, metric, and all other values held fixed. Zeroing layer 2 head 1 changes the checked margin from 6.4618 to -2.4767. This supports dependence under that zero-write replacement; it does not prove the head is sufficient, uniquely responsible, or aligned with a human concept.

Add a negative control and positive endpoint where possible. The reference patches identical position-0 state for zero recovery and the complete clean final residual for full recovery.

Question. Calculate an ablation effect

The clean logit margin is 4.2 and the ablated margin is 1.1. Calculate $\Delta m$ .

Your answer

Compute it first, then check your number.

Hint

Use intervention minus clean.

Solution

\Delta m=1.1-4.2=-3.1

Write the claim in layers

Use four separate fields:

Observation: the recorded value and where it occurred.
Intervention: the replacement and measured output change.
Interpretation: a proposed role no stronger than the tests.
Alternatives and limits: baseline dependence, interactions, prompt scope, and other explanations.

Do not call the largest attention weight an explanation. Do not turn a large zero-ablation effect into sufficiency. Do not call an MLP coordinate a feature without examples and interventions that test that label.

Acceptance gate

Continue only when:

the traced and ordinary logits agree within the declared tolerance;
every retained tensor has a site, axis convention, shape, and prompt;
baseline output and full probability row are preserved;
the intervention changes exactly one declared value;
controls, alternative explanations, and scope limits are recorded;
the final paragraph names a follow-up test that could disprove or refine the interpretation.

Deliverable: trace-fidelity result, internal evidence table, intervention and control table, bounded interpretation, alternative account, and next discriminating test.