In a disaggregated multimodal deployment the vision encoder runs on its own GPUs, one encoder per GPU, several GPUs per machine. The prefill side picks a machine; on that machine the image goes to the GPU currently holding the fewest patches, because an image's encoder cost is its patch count, not its byte size.
route_images(images, gpus_per_machine, patch_size)
images is a list of dicts with "width", "height" and "machine" (the machine the prefill side already chose), in arrival order.(width // patch_size) * (height // patch_size).{"assignment": [(machine, gpu, patches), ...], "loads": [[...], ...]} where loads[m][g] is the final patch count on each GPU.loads table of n_machines x gpus_per_machine zeros; n_machines is one more than the largest machine index seen.min(range(g), key=...) returns the first minimum, which is the tie-break required.