In scaled dot-product attention, raw attention scores are computed as the dot product between queries and keys, then divided by the square root of the key dimension before applying softmax. This scaling counteracts the tendency of dot products to grow large in magnitude as the dimension d_k increases, which would otherwise push softmax into regions with extremely small gradients.
Implement a function scaled_attention_weights(Q, K) that:
Q @ K.T.sqrt(d_k), where d_k is the last dimension of K.Args:
Q: numpy array of shape (n_q, d_k)K: numpy array of shape (n_k, d_k)Return the attention weight matrix of shape (n_q, n_k) as a nested Python list, with each value rounded to 4 decimal places.