Implement masked self-attention, a variation of the attention mechanism used in autoregressive sequence modeling tasks such as text generation. The mask ensures that each token can only attend to itself and preceding tokens.
A helper function compute_qkv(X, W_q, W_k, W_v) is already implemented for you. Your task is to implement masked_attention(Q, K, V, mask) which computes the masked self-attention output.
Inputs:
Q (Query): np.ndarray of shape (seq_len, d_model)K (Key): np.ndarray of shape (seq_len, d_model)V (Value): np.ndarray of shape (seq_len, d_model)mask: np.ndarray of shape (seq_len, seq_len), with 0 at allowed positions and -inf at blocked positionsOutput:
(seq_len, d_model)Formula:
where is the number of columns of Q and the softmax is applied to each row. Use a numerically stable softmax (subtract each row's maximum before exponentiating): the test inputs produce very large scores.