Watch the Gradient Stop Summing to Zero
The gradient of cross-entropy with respect to the logits is q−p, which sums to zero because both q and the one-hot target p sum to one. Barron et al. (2026) show that the standard implementation breaks this identity once the logit margin passes the dtype's precision cliff: the target entry rounds to zero while the others stay positive. Their fix never forms qt−1. Implement both ways in a chosen dtype and measure the sum.
standard_ce_grad(z, t, dtype)
z is a 1-D array of K >= 2 logits, t the target index, dtype one of np.float16, np.float32, np.float64.
- Cast
z to dtype and do every step in that dtype: shift by the maximum, exponentiate, normalise to q, then return q−onehot(t) as an array of dtype.
fixed_ce_grad(z, t, dtype)
- Same inputs and dtype rules. Return the paper's gradient as an array of
dtype, with m the logit margin zt−log∑j=tezj:
gj=1+e−mezj−zt(j=t),gt=−j=t∑gj
- The margin and log(1+e−m) must be computed stably in the dtype: no overflow or warning for any input, including a target that is far below the leading logit (
z = [0, 100, 0], t = 0) or logits of magnitude 1000. In the paper's example z = [9, -9, -9], t = 0 in float32 the result is about [-3.046e-8, 1.523e-8, 1.523e-8].
gradient_sum(g)
- Return the sum of the entries of
g, accumulated in float64, as a Python float.
Hints
- 1+e−mezj−zt equals e(zj−zt)−softplus(−m), and the exponent is never positive, so this form cannot overflow.
- softplus(x)=max(x,0)+log(1+e−∣x∣); use
np.log1p for the second term so that a value like 3×10−8 survives in float32.
- Do not obtain ∑j=tgj as (sum of all entries) minus (the target entry): at a large margin the target entry is 1 up to rounding, and the subtraction loses exactly the quantity you need. Zero the target slot first, then sum.