Intermediate Activations

Intermediate activations are the values inside the network between input and output.

They matter because they show what the network is computing before the final prediction.

For a one-hidden-layer MLP:

z1 = xW1 + b1
h = relu(z1)
scores = hW2 + b2

Here:

  • z1 is the hidden pre-activation;
  • h is the hidden activation;
  • scores are the final raw outputs.

The hidden activation h is often the most useful place to inspect the model's internal representation.

Batch shape

For a batch:

X shape:  (batch, input_features)
W1 shape: (input_features, hidden_units)
b1 shape: (hidden_units)

Then:

H shape: (batch, hidden_units)

The batch size stays. The feature dimension changes.

Exercise: Batch hidden shape

X has shape (12, 6) and W1 has shape (6, 8). What is the shape of the hidden activation matrix H?

Compute it first, then check your number.

HintBatch stays first

(12, 6) x (6, 8) gives (12, 8).

SolutionWork it out

XW1 has shape (12, 8). The bias has shape (8) and broadcasts across the batch. ReLU keeps the shape, so H has shape (12, 8).

Exercise: Activation keeps shape

If a ReLU is applied to a pre-activation matrix with shape (12, 8), what is the output shape?

Compute it first, then check your number.

HintElementwise transform

ReLU changes values, not the number of entries.

SolutionWork it out

ReLU is applied element by element, so the output has the same shape: (12, 8).