🎓 A1Learner
🔍
0 of 47 lessons completed

Julia Optimization from Scratch

Optimization from scratch is building a search for a best value by hand, without a ready-made solver. The search takes small steps downhill for a fixed number of steps.

Fixed steps move a guess toward the lowest point

The function takes ten steps of size 0.1. The printout shows the rounded guess.

function descend(x, step, n)
    for i in 1:n
        x = x - step * 2.0 * (x - 3.0)
    end
    return x
end
println(round(descend(0.0, 0.1, 10), digits=6))

A smaller step size needs more steps

The function takes twenty steps of size 0.05. The printout shows the rounded guess.

function descend(x, step, n)
    for i in 1:n
        x = x - step * 2.0 * (x - 3.0)
    end
    return x
end
println(round(descend(0.0, 0.05, 20), digits=6))

Descent works from a different lowest point

The function takes ten steps toward a lowest point at minus two. The printout shows the rounded guess.

function descend_to_low(x, step, n)
    for i in 1:n
        x = x - step * 2.0 * (x + 2.0)
    end
    return x
end
println(round(descend_to_low(0.0, 0.1, 10), digits=6))

Example

function descend(x, step, n)
    for i in 1:n
        x = x - step * 2.0 * (x - 3.0)
    end
    return x
end
println(round(descend(0.0, 0.1, 10), digits=6))
println(round(descend(0.0, 0.1, 20), digits=6))

Key points

  • Gradient: the gradient is the slope of a function at a point, and it points uphill.
  • Step size: a step size is the fixed distance moved on each step.
  • Descent: descent moves against the gradient, so the value falls.
  • Fixed steps: a fixed count of steps gives the same result on every run.
  • Quadratic: a quadratic is a function with a squared term and one lowest point.

In this example, the function moves against the slope at each step. It runs for ten steps and then for twenty steps. The longer run lands closer to three.

Note

a step size that is too large can overshoot the lowest point.

Try it Yourself

Edit the code, then press Run.

Running is simulated · JuliaHub

Exercise

Which direction does descent move relative to the gradient?