A short-video ranker predicts several events at once (finish, like, share, follow) and blends them into one score. MMoE shares a pool of experts across the tasks and gives every task its own softmax gate over the experts, so tasks can weight the shared computation differently. Implement the model and the blending step.
MMoE(input_dim, n_experts, expert_hidden, n_tasks) (an nn.Module)
self.experts: an nn.ModuleList of n_experts networks, each Linear(input_dim, expert_hidden), ReLU, Linear(expert_hidden, expert_hidden), in that order.self.gates: an nn.ModuleList of n_tasks modules Linear(input_dim, n_experts) (with bias); the gate weights are the softmax over the expert dimension.self.towers: an nn.ModuleList of n_tasks modules Linear(expert_hidden, 1).forward(x) with x of shape (B, input_dim) returns logits of shape (B, n_tasks): for task t, the gate-weighted sum of the expert outputs goes through tower t.gate_weights(x) returns a tensor of shape (n_tasks, B, n_experts) with the softmax gate values.multitask_bce(logits, labels, task_weights=None)
labels has shape (B, n_tasks) of 0/1 floats. Return the sum over tasks of task_weights[t] times the mean binary cross-entropy of task t computed from the logits (default weight 1 for every task).blend_and_rank(probs, weights)
probs is (n_items, n_tasks) of per-task probabilities, weights a list of length n_tasks. Compute the blended score sum_t weights[t] * probs[:, t] and return the item indices sorted by blended score descending (ties by lower index) as a Python list.(B, n_experts, expert_hidden); the gate for a task is then a batched weighted sum with torch.einsum or a matmul.F.binary_cross_entropy_with_logits keeps the loss stable for large logits.