Before neural word embeddings, word vectors came from factorising count matrices. Implement the pipeline. vocab is a list of words; a word's row index is its position in vocab.
cooccurrence(tokens, vocab, window): the (V, V) count matrix where counts the positions with tokens[t] = word , tokens[s] = word and window. Tokens not in vocab are skipped (they still occupy their positions).
ppmi(C): positive pointwise mutual information, with the row sums, the column sums and the total; entries with (or an empty row or column) are . Natural log.
lsa_embeddings(M, k): from the SVD return (first singular directions), after flipping the sign of each column of so that its entry of largest absolute value is positive.
nearest(word, E, vocab, topn): the topn other words with the highest cosine similarity to word's row of E, best first (ties: smaller index), as a list of strings.
analogy(a, b, c, E, vocab): solve "a is to b as c is to?": the word, other than a, b and c, whose vector has the highest cosine similarity with .