In the Transformer decoder, every layer contains an encoder-decoder attention sub-layer: the queries come from the decoder's own states, while the keys and values come from the output of the encoder stack. This is how each target position looks over the whole source sentence.
Implement cross_attention(decoder_x, encoder_out, W_q, W_k, W_v, src_pad_mask=None):
decoder_x has shape (tgt_len, d_model): the decoder states that act as queries.encoder_out has shape (src_len, d_model): the final encoder output that supplies keys and values.W_q and W_k have shape (d_model, d_k), W_v has shape (d_model, d_v).src_pad_mask, when given, is a boolean or 0/1 array of shape (src_len,) with True/1 at padded source positions. Those positions must receive an attention weight of exactly 0 for every query, and the remaining weights of each row must still sum to 1. Assume at least one source position is unmasked.Return the tuple (output, weights) where output has shape (tgt_len, d_v) and weights has shape (tgt_len, src_len).
-np.inf before the softmax; np.exp(-np.inf) is exactly 0.0.(tgt_len, src_len).