Where Cross-Entropy Stops Learning

Barron, Gnanapragasam and Tschannen (2026) show that the standard softmax cross-entropy has a silent numerical bug: once a target logit is far enough ahead of the rest, the target probability rounds to exactly 1 in floating point and its gradient becomes exactly zero, while the other gradients stay positive. The distance that matters is the logit margin, and the margin at which rounding kicks in is the precision cliff of the dtype. Implement both.

logit_margin(z, t)

• z is a 1-D array of K >= 2 logits (any real values, including magnitudes like 1000), t an integer index.
• Return the margin as a Python float (as defined in the paper):

• The log-sum-exp runs over the other logits only. It must be computed stably: no overflow, no numpy warning on stderr.

precision_cliff(dtype)

• dtype is one of np.float16, np.float32, np.float64.
• Return, rounded to 4 decimals, the smallest margin $m \geq 0$ at which $1/(1 + e^{-m})$ evaluates to exactly 1 when the arithmetic is done in that dtype: cast $e^{-m}$ to dtype, add dtype(1), take the reciprocal, compare with dtype(1). This is the margin at which the standard target gradient $q_t - 1$ rounds to zero.
• Find it by bisection on a float64 m over [0, 100]; only the rounding goes through the dtype. Stop once the bracket is narrower than 1e-9.

Hints

• The saturation test is monotone in $m$: false below the cliff, true above it, so bisection needs only a bracket.
• For the margin, drop the target, shift the remaining logits by their maximum, and add the shift back after the log.
• Rounding $1/(1 + e^{-m})$ once at the end, in float64, gives a different (wrong) cliff: the loss of the $e^{-m}$ term happens in the addition to 1, so that step must be done in the dtype.