Convexity
Convexity is a useful special case.
For a convex function, any local minimum is also a global minimum.
Many deep learning loss surfaces are not convex, but convexity still gives a clean reference point.
Why Learn It?
Convex problems are easier to reason about.
They help us understand why some optimization guarantees exist in simpler settings and why deep learning is harder to analyze.
In a convex problem, the landscape has no bad local basin that traps you above a better faraway basin. That does not mean every algorithm is fast, but it makes the target easier to reason about.
Convexity is a property of the objective shape, not of the optimizer. An optimizer can be used on a non-convex loss, but the clean convex guarantee no longer applies.
Working Intuition
A convex bowl has one basin.
A non-convex landscape may have many basins, flat regions, and saddle points.
A saddle point is not a minimum or maximum. It can slope down in one direction and up in another. In high-dimensional optimization, saddle-like regions are one reason the landscape can be hard to picture.
This matters in neural networks because "gradient is near zero" does not always mean "we found the best point." It may mean the optimizer is in a flat or saddle-like region.
For a convex function, is every local minimum also a global minimum?
Enter 1 for yes, 0 for no.
Compute it first, then check your number.
Hint
Solution
Yes. For a convex function, a local minimum is global. Enter 1. Convexity
removes the possibility of a worse local basin hiding above a better faraway
minimum.
Are most deep learning loss surfaces assumed to be convex?
Answer it first, then check.
Hint
The lesson says deep learning usually leaves the convex comfort zone.
Solution
No. Deep learning loss surfaces are usually not assumed to be convex.
In a convex function, can a local minimum be worse than another faraway minimum?
Answer it first, then check.
Hint
For convex functions, local minima are global minima.
Solution
No. If the point is a local minimum of a convex function, it is also global.
Can a saddle point slope down in one direction and up in another?
Answer it first, then check.
Hint
A saddle is not simply a low point or high point.
Solution
Yes. A saddle point can have different slope behavior in different directions.
Enter 1 if convex guarantees should not be assumed for a general deep neural
network loss.
Compute it first, then check your number.
Hint
The lesson uses convexity as a reference case.
Solution
Enter 1. Deep neural network losses are usually non-convex, so convex
guarantees should not be assumed by default.
Before Moving On
Convexity is a clean special case, not the default shape of deep learning.