Contour Plots and Loss Surfaces

Use contour lines and filled intervals to inspect a function of two inputs on a flat parameter plane. Build matching coordinate and loss grids, read contour values and spacing, overlay parameter paths, and state what a two-dimensional slice leaves out.

A contour plot shows a function of two numerical inputs on a flat coordinate plane. Each contour line joins input pairs that produce the same function value. Filled regions can use color to show the intervals between those lines.

In machine learning, the two inputs can represent two model parameters and the function value can represent loss. A contour plot then answers questions such as where loss is small, how rapidly it changes in different directions, and how an optimization path moves across the parameter plane.

Begin with a function of two parameters

Consider the loss

It receives one weight and one bias, then returns one nonnegative number. The two squared terms are both zero when

weight = 2
bias = -1

so the minimum loss is 0 at the point (2, -1).

Evaluate a few nearby points before plotting:

loss(2, -1) = 0² + 2(0²) = 0
loss(3, -1) = 1² + 2(0²) = 1
loss(2,  0) = 0² + 2(1²) = 2

A one-unit change in bias raises the loss more than a one-unit change in weight near the minimum because the bias term is multiplied by 2.

Evaluate many parameter pairs on a grid

Matplotlib needs the loss at many coordinate pairs. Start with one-dimensional coordinate arrays:

np.meshgrid creates two matching two-dimensional arrays:

weight_grid, bias_grid = np.meshgrid(weight_values, bias_values)

Their values are:

weight_grid                 bias_grid
[[1, 2, 3],                 [[-2, -2, -2],
 [1, 2, 3],                  [-1, -1, -1],
 [1, 2, 3]]                   [ 0,  0,  0]]

Moving across a row changes weight. Moving down a column changes bias. Applying the loss expression to these arrays evaluates all nine coordinate pairs element by element.

The loss grid is:

[[3, 2, 3],
 [1, 0, 1],
 [3, 2, 3]]

Its center entry corresponds to (weight, bias) = (2, -1) and has loss 0. The three grids must have the same shape because one position represents one weight, one bias, and their loss.

Sample a finer grid

A visible contour needs more than nine samples. np.linspace can supply evenly spaced coordinates:

The resulting grids have shape (121, 121). The function is still the same mathematical object. The grid is a finite sample used to draw it.

A finer grid can produce smoother boundaries but requires more computation and memory. It does not repair a wrong function, wrong parameter range, or wrong axis meaning.

Draw level lines and filled intervals

contour draws lines at selected values:

contourf fills the intervals between levels:

The levels are function values, not parameter coordinates. A contour labeled 2 contains every sampled (weight, bias) pair where the interpolated loss is 2.

Overlay an optimization path

The next example combines filled intervals, labeled contour lines, the exact minimum, and a short sequence of parameter values moving toward that minimum. Edit one path point and inspect which levels it crosses.

The path loss decreases at every supplied step. The plotted path is an example, not the result of an optimization algorithm in this lesson. Later optimization material can generate such points and place them on the same kind of plot.

Read contour shape and spacing

The contours are ellipses centered at (2, -1). They are narrower in the bias direction because equal-sized bias changes affect this loss more strongly than equal-sized weight changes.

Contour spacing also carries information. When adjacent levels differ by a fixed loss amount:

• closely spaced lines mean a short movement crosses a larger loss change;
• widely spaced lines mean a longer movement is needed for the same loss change.

The direction of fastest local increase crosses a contour rather than following along it. Moving along one contour keeps the function value constant. In a smooth function, the gradient is a vector that collects the rate of change with respect to each input. It points perpendicular to a contour line, toward increasing values; the negative gradient points toward decreasing values.

This geometric statement depends on axis scale. If one axis is visually stretched or uses different units, judge numerical direction from the coordinates and function rather than appearance alone.

Choose levels and limits deliberately

Automatic contour levels are convenient for exploration, but explicit levels make comparisons reproducible. Use the same coordinate range, level values, and color scale when comparing two loss surfaces.

The chosen parameter range can hide important structure. A minimum outside the displayed region will not appear. A very wide range can compress the local shape near a minimum. Record:

• the weight and bias ranges;
• the number of grid samples;
• the contour levels;
• the color scale and limits;
• the formula or computation that produced the loss grid.

These settings are part of the figure's meaning.

State what two visible parameters leave out

This loss genuinely has two parameters. A neural network can have millions or billions. A two-dimensional contour plot of a larger model can show only:

• two selected parameters while others remain fixed;
• two chosen directions through parameter space;
• or a separate low-dimensional toy function.

Such a plot can build geometric intuition and inspect a chosen slice. It cannot show every direction, interaction, minimum, flat region, or saddle point in the full parameter space. Label the slice and avoid presenting it as a complete map of training.

loss = (weight - 2)² + 2(bias + 1)²

Read a contour as coordinates, levels, and a slice

To interpret a contour plot, verify the two coordinate axes, reconstruct how the grid was sampled, read each line as a constant function value, and inspect the chosen levels and limits. For a high-dimensional model, also state exactly which two parameters or directions the visible slice represents.