Steepest descent moves along , and the whole game is the step size . Implement three pieces (Murphy, Probabilistic Machine Learning):
exact_step(A, b, theta, d) for the quadratic : return the step that minimizes over , as a float. If (for example at the optimum), return 0.0.
armijo_backtrack(f, grad, theta, d, eta0=1.0, c=1e-4, rho=0.5, max_halvings=60): try and return the first that passes the Armijo test
If none of the first max_halvings candidates passes, return the last candidate tried, .
steepest_descent(f, grad, theta0, n_iter, rule="armijo", eta=0.1, A=None, b=None): run n_iter iterations of with and
rule="fixed": = eta;rule="exact": from exact_step (the objective is the quadratic given by A, b);rule="armijo": from armijo_backtrack with its default constants, restarting at every iteration.Return all iterates as a float array of shape (n_iter + 1, D), starting with theta0.