Two SAE variants avoid the shrinkage by deciding which latents fire separately from how much they fire. Implement their forward passes and the straight-through gradient that trains a JumpReLU threshold.
jumprelu(pre, theta)
pre where pre > theta, else 0. theta is a (n,) vector of per-latent thresholds (strictly positive).jumprelu_sae_forward(X, W_enc, b_enc, theta, W_dec, b_dec)
pre = X @ W_enc + b_enc, then Z is the JumpReLU of pre with thresholds theta, and Xhat = Z @ W_dec + b_dec. Return (pre, Z, Xhat).gated_sae_forward(X, W_gate, b_gate, W_mag, b_mag, W_dec, b_dec)
pi = X @ W_gate + b_gate, active mask pi > 0; magnitude path mag = max(X @ W_mag + b_mag, 0); Z = mask * mag; Xhat = Z @ W_dec + b_dec. Return (pi, Z, Xhat).l0_loss(pre, theta)
pre > theta, as a float.l0_theta_pseudograd(pre, theta, eps)
theta using a rectangle kernel of width eps: for each latent, . Return the (n,) array.shrinkage_gap(X, W_enc, b_enc, W_dec, b_dec, lam)
lam * (fraction of entries of max(pre, 0) that are positive) as a float, the average shrinkage per latent entry.pre - theta) above the threshold.