In several Transformer architectures, token embeddings are scaled by the square root of the embedding dimension before being passed to subsequent layers. This scaling keeps the magnitude of the embeddings on a comparable scale to the positional encodings and helps stabilize training.
Implement a function scaled_embedding_lookup(token_ids, embedding_matrix) that:
token_ids (a 2D list / array of shape (batch_size, seq_len)) in embedding_matrix (shape (vocab_size, emb_dim)).sqrt(emb_dim).You may assume all token ids are valid indices into embedding_matrix.