Browse and solve 1,200+ machine learning coding challenges on Deep-ML. Filter by difficulty (beginner, intermediate, advanced) and category (linear algebra, machine learning, deep learning, NLP, computer vision). Write Python solutions in the browser and get instant feedback against test cases.
Deep-ML | LaProp Step and the Margin Equilibrium It Shares with AdamWLaProp Step and the Margin Equilibrium It Shares with AdamW
LaProp Step and the Margin Equilibrium It Shares with AdamW
MediumDeep Learning
Example:
Input:
```python
import numpy as np
_r = lambda v: (np.round(np.asarray(v, dtype=float), 4) + 0.0).tolist()
theta = np.array([1.0, -2.0])
m = np.zeros(2); s = np.zeros(2)
theta, m, s = laprop_step(theta, np.array([0.5, -0.25]), m, s, 1, 0.1, 0.9, 0.99, 1e-8, 0.1)
print(_r(theta), _r(m), _r(s))
theta, m, s = laprop_step(theta, np.array([0.1, 0.1]), m, s, 2, 0.1, 0.9, 0.99, 1e-8, 0.1)
print(_r(theta), _r(m), _r(s))
```
At t = 1 the bias corrections undo the (1 - beta) factors, so s_hat = |g| and the normalised gradient is g / (|g| + 1e-8), about (1, -1). The update is lr * (sign(g) + 0.1 * theta): theta_0 = 1 - 0.1 * 1.1 = 0.89 and theta_1 = -2 + 0.1 * 1.2 = -1.88. At t = 2 LaProp averages the normalised gradients 0.1 / s_hat, giving 0.8191 and -1.8415; averaging raw gradients the Adam way would give 0.8006 and -1.8266.
A Fix for Softmax and Cross-Entropy predicts where logit margins stop growing: at the stationary point of AdamW every coordinate satisfies gi/(∣gi∣+ϵ)+λθi=0, so with ∣gi∣∝e−m the margin levels off at m∗(ϵ)≈log(1/ϵ). The learning rate and the two decay rates cancel, so RMSProp and LaProp share the same stationary point. To test the prediction at margins where gi2 would underflow in float32, the paper trains with LaProp (Ziyin et al., 2020) and tracks v directly instead of squaring the gradient. Implement that optimizer step and the stationary residual.
laprop_step(theta, g, m, s, t, lr, beta1, beta2, eps, weight_decay)
theta, g, m, s are float arrays of one shape (elementwise everywhere); t is the 1-based step count (t = 1 for the first step); the rest are Python floats. Return (theta_new, m_new, s_new) as new arrays; do not modify the inputs.
s is the running root-mean-square of the gradient, i.e. nt for the second moment nt=β2nt−1+(1−β2)gt2 in the LaProp paper. Update it without ever forming gt2 or s2 (a hypotenuse of β2st−1 and 1−β2gt), so that a gradient of 10−200 still produces the correct st:
st=β2st−12+(1−β2)gt2,s^t=st/1−β2t
LaProp normalises the gradient before accumulating momentum (Adam normalises after), with ϵ added outside the root and the bias correction of the second moment applied inside the momentum update, exactly as in Ziyin et al.:
mt=β1mt−1+(1−β1)s^t+ϵgt,m^t=1−β1tmt
Decoupled weight decay, as in AdamW: θt=θt−1−η(m^t+λθt−1) with lr=η and weight_decay=λ.
eps may be 0.0; the tests then never pass a zero gradient.
stationary_residual(g, theta, lam, eps)
Return the elementwise residual of the paper's stationarity condition, gi/(∣gi∣+ϵ)+λθi, as a float array. It is zero exactly at the stationary point of AdamW, and because η, β1, β2 cancel there, at LaProp's as well: at AdamW's stationary point the optimizer state is m=g, v=g2; at LaProp's it is m=g/(∣g∣+ϵ), s=∣g∣, and in both cases the parameter update vanishes.
Hints
numpy.hypot(a, b) computes a2+b2 without overflow or underflow in the squares.
Python's 0.98 ** 1000000 underflows silently to 0.0, so large t simply switches the bias corrections off.
In a one-parameter toy where the margin is the parameter, g=−σ(−m) and λ>0 pull against each other; with λ=0 nothing stops the margin, but each step shrinks like e−m/ϵ once ∣g∣<ϵ.