Three Benchmarks, No Training
MTEB scores an embedding model on tasks that use the vectors directly. Implement three of them.
zero_shot_classify(X, label_embeddings)
X is (n, d), label_embeddings is (c, d), one vector per class description. Return the list of predicted class indices by highest cosine similarity (lower index on ties).
spearman(a, b)
- The Spearman rank correlation of two equal-length lists: rank each (ties get the average of the ranks they would occupy), then compute the Pearson correlation of the ranks. Rounded to 4 decimals.
sts_score(E1, E2, gold)
E1[i] and E2[i] are the two sentences of pair i; gold[i] is the human similarity. Return spearman between the per-pair cosine similarities and gold.
v_measure(labels_true, labels_pred, beta=1.0)
- From the contingency counts compute homogeneity h=1−H(C∣K)/H(C) and completeness c=1−H(K∣C)/H(K) with natural-log entropies, each defined as 1 when the denominator entropy is 0, and v=(1+β)hc/(βh+c) (0 when h+c=0). Return
(h, c, v) rounded to 4 decimals.
Hints
- Average ranks: sort, then for each run of equal values assign the mean of their 1-based positions.
- H(C∣K)=−∑c,knncklognknck over non-zero cells.