EmbeddingGemma learns from Gemini Embedding by matching embeddings rather than output distributions: the student's vectors for the query, the positive and the hard negative are each pulled toward the teacher's, with equal weight. A relational variant matches the teacher's similarity structure instead of its coordinates. Implement both in PyTorch.
embedding_match_loss(S, T)
S and T have shape (B, d). Return the mean over rows of .relational_distill_loss(S, T, tau=0.05)
softmax(Tn @ Tn.T / tau)[i], the student's is log_softmax(Sn @ Sn.T / tau)[i]. Return the KL divergence from the teacher to the student, averaged over rows (F.kl_div with reduction="batchmean").geometric_distill_loss(student, teacher)
student and teacher are dicts with keys "q", "p" and "n" holding (B, d) tensors. Return the sum of embedding_match_loss over the three keys, each with weight 1.DistillHead(d_student, d_teacher)
nn.Module with one nn.Linear(d_student, d_teacher, bias=False) called proj; forward(x) returns F.normalize(self.proj(x), dim=1). It lets a narrower student be matched to a wider teacher.F.cosine_similarity(S, T, dim=1) gives the per-row cosines.