Implement ReLU

Why a network needs a bend

A neuron can first form a weighted sum and add a bias. If every layer only repeats that operation, the layers can be combined into a single affine transformation. Adding a nonlinear function between layers lets the network represent relationships that a single affine transformation cannot.

ReLU, short for rectified linear unit, is one such function. It replaces a negative input with zero and keeps a positive input unchanged. Zero stays zero.

ReLU(x) = max(0, x)

For x = −2, the larger of 0 and −2 is 0. For x = 3, it is 3. Before moving the slider, predict where the output will be when x crosses zero.

Your task

Write relu(values) in relu.py. It receives a flat Python list of finite integers or floats and returns a new list with ReLU applied to every element. Preserve the order and length. Do not change the original list. An empty list returns a new empty list.

Use Python’s standard library; a loop or comprehension is enough. Nested lists, NaN, and infinity are outside this problem. This first task does not implement a tensor library or gradients.

relu([-2, 0, 3])       # [0, 0, 3]
relu([-0.5, 0.25, -4]) # [0, 0.25, 0]
relu([])               # []

A vector here is represented by a flat list: the function handles each value independently. It does not choose the largest value in the whole list.

Run uses Pyodide in a separate Web Worker. It stops long-running code, bounds displayed output, and disables network access before your code starts. The public checks compare expected outputs and verify that the input list was not changed.

Think before checking

Would replacing each value with its absolute value work? Try −2. Would dropping negative entries work? Compare the output length with the input length.

def relu(values):
    result = []
    for value in values:
        result.append(value if value > 0 else 0)
    return result

What this does—and does not—solve

On the positive branch, the derivative with respect to the input is 1. On the negative branch, it is 0. The active branch avoids the small local derivative of a saturated sigmoid, but ReLU does not guarantee healthy gradients throughout a network. A unit can remain inactive, and the derivative at zero needs an explicit convention when implementing backpropagation.

Our task concerns forward values only. Passing its checks is evidence for that small contract, not mastery of training a neural network.

A small extension

Change the negative branch to return 0.1 * x. Predict the output for [-2, 0, 3] before running your code. Explain what changed in the graph and why this new function is no longer ReLU.