A memory layer stores key-value slots and reads the top- of them per token. Sparse memory finetuning learns new data by updating only the slots that the new batch accesses far more often than pretraining data did, ranked by a TF-IDF score. Implement the ranking, the mask and the masked update.
access_counts(topk_indices, n_slots)
topk_indices is an (n_tokens, k) integer array of the slots each token retrieved. Return the (N,) count of accesses per slot.slot_tfidf(tf, df, n_background)
tf is the (N,) access count in the new batch, df the (N,) number of background (pretraining) batches, out of n_background, in which each slot was accessed at least once. Return (the +1 keeps never-seen slots finite), as a float array.sparse_update_mask(scores, t)
(N,) mask marking the t highest-scoring slots (ties by lower index first, stable sort on -scores). If t is zero or no slot has a positive score, the mask is all False; slots with score <= 0 are never selected.apply_sparse_update(values, grads, mask, lr)
values is (N, d), grads (N, d). Return a copy of values where masked rows are moved by -lr * grads and all other rows are unchanged.interference_rate(background_topk, mask)
background_topk is an (n_tokens, k) array of slots retrieved by background tokens. Return the fraction of background tokens that retrieved at least one updated slot, as a Python float.np.bincount(flat_indices, minlength=n_slots) counts accesses in one call.