Distance, not place

Attention scores how much one word listens to another with a dot product: the first word’s query (what it looks for) times the other’s key (what it offers), as in “Who should “it” listen to?”. On their own they don’t say where the words are. Write tag(vector, position): change a 2D query or key so it includes its word’s position, such that the score between a query at i and a key at j depends only on how far apart they are, not on where in the sentence they sit.