When each KV entry serves as both key and value, rotating the values with RoPE leaks absolute position into the attention output; the fix is to counter-rotate the output by the negative query position. Implement two functions.
rope_partial(x, positions, rope_dims=64, base=10000.0)
x: (n, d); positions: n numbers (may be negative or non-integer).d - rope_dims columns are unchanged. In the last rope_dims columns, pair is the adjacent columns of that block, rotated by : .(n, d) float array; rope_dims = 0 returns an unchanged copy.partial_rope_attention(q, kv, q_pos, kv_pos, rope_dims=64, base=10000.0)
q: (n_q, d) with positions q_pos; kv: (n_kv, d) with positions kv_pos. No mask.q at q_pos and kv at kv_pos; scores (the full d); softmax over entries; output is the weighted sum of the rotated entries, rotated once more with position -q_pos[t] for query t.(output, weights) with shapes (n_q, d) and (n_q, n_kv).Shifting all positions by one constant must leave both results unchanged (up to float error).