Neurons, Features, Polysemanticity, and Circuits
Distinguish an MLP neuron, its activation, a hypothesized feature, polysemanticity, superposition, and a circuit. Inspect active coordinates without assigning semantic labels unsupported by the tiny corpus.
Interpretability vocabulary is easy to blur. These terms refer to different objects.
Separate the Objects
- A neuron is one coordinate in a chosen activation basis, such as one of the eight ReLU outputs in this model's MLP.
- An activation is the numerical value of that coordinate for a particular input and position.
- A feature is a hypothesized input property or computational quantity represented in model state. It may use one neuron, several neurons, or a direction not aligned with individual coordinates.
- A polysemantic neuron responds to several apparently unrelated features under an interpretation procedure.
- A circuit is a hypothesis about connected components and operations that implement a behavior.
Finding an active neuron does not identify a feature. Naming a feature does not establish a circuit.
Inspect the Tiny MLP without Naming Concepts
At the clean prompt's final position, the largest positive ReLU activations are:
| Layer | Neuron index | Activation |
|---|---|---|
| 1 | 6 | 1.2145 |
| 1 | 4 | 0.5132 |
| 2 | 5 | 1.2145 |
| 2 | 3 | 0.6806 |
| 2 | 6 | 0.2452 |
This table establishes which coordinates are active in one run. The four sequence patterns in the training corpus are too narrow to support a rich semantic label. A stronger study would collect activating and non-activating examples, inspect outgoing effects, intervene, and test held-out cases.
A Basis Coordinate Need Not Be the Natural Unit
A residual-stream coordinate is a model dimension, not automatically a neuron-like semantic unit. Some coordinated changes of representation and surrounding weights can preserve a network's function, although normalization, nonlinearities, and other architectural details constrain those changes. MLP hidden activations do have a parameter-defined coordinate basis, but features can still be distributed or share coordinates.
Elhage et al. (2022) demonstrate superposition and both monosemantic and polysemantic neurons in specified toy models. They explicitly leave uncertain which detailed geometry and dynamics transfer to real networks. Use the work as a model of a possible mechanism and evidence standard, not proof that every active coordinate in this decoder has the same explanation.
Q1. Name the measured object
For one prompt, MLP coordinate 6 has ReLU output 1.2145. What has been measured: a neuron, an activation, a feature, or a circuit?
Answer it first, then check.
Hint
Solution
Treat Circuit Diagrams as Testable Models
A proposed circuit should name reads, writes, positions, components, and the behavioral metric. It should predict ablations or patches that were not used to draw the diagram. Otherwise the diagram summarizes observations rather than explaining computation.