Word representations: from co-occurrence counts to word2vec

Many useful functions take text as input. Given a movie review, we may want its sentiment, \(\hat y = \text{sentiment}(\text{words})\), as a label or a score. A machine learning model learns \(\hat y = f(\mathbf{x})\), and every model in the course so far has assumed \(\mathbf{x}\) is a vector of numbers. So before any learning, each word (or the whole input stream) has to be turned into a vector. The slide shows a short review mapped to a vector such as \([5.7, 1.2, 2.3, -10.2, 4.5, \dots]\).

The running example is a toy corpus of four sentences:

1. Human machine interface for computer applications
2. User opinion of computer system response time
3. User interface management system
4. System engineering for improved response time

The vocabulary \(V\) is the set of unique words across all the input streams (sentences or documents). Here it has 15 words: human, machine, interface, for, computer, applications, user, opinion, of, system, response, time, management, engineering, improved. The goal of the lecture is a vector for every word in \(V\).

The slides acknowledge three sources that are good further reading: Xin Rong, word2vec Parameter Learning Explained; Yoav Goldberg and Omer Levy, word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method; and Sebastian Ruder's blog posts on word embeddings.

The simplest recipe numbers the words (human = 0, machine = 1, interface = 2, …) and gives word \(i\) a vector of size \(|V|\) with a 1 in position \(i\) and 0 in the other \(|V| - 1\) positions. Only one entry is ever "hot". For example machine \(= [0, 1, 0, \dots, 0]\).

Two problems make these vectors useless as representations:

• Size. \(|V|\) is very large: about 50K for the Penn Treebank (PTB) corpus and 13M for Google's 1T corpus. Each vector stores \(|V|\) numbers, all but one of them 0.
• No notion of similarity. We would like cat and dog (both domestic animals) to be closer to each other than cat and truck. But the one-hot vectors of any two different words are orthogonal:

So euclid dist(cat, dog) = euclid dist(dog, truck) = \(\sqrt2\), and both cosine similarities are 0. The vectors carry nothing about what the words mean, while the course has always looked for representations that capture something essential about the entity (as with PCA and the autoencoders of Lecture 7).

One-hot vectors are sparse: one bit on. A distributed representation spreads a word's meaning over many entries. The starting point is J. R. Firth's remark from 1957, "You shall know a word by the company it keeps." To learn what bank means, look at the words that appear around it. In a sentence describing a bank as a financial institution that accepts deposits from the public and creates credit, the neighbours financial, deposits and credit describe it well. Representations built this way are called distributional similarity based representations.

A co-occurrence matrix is a terms × terms matrix. Entry \((i, j)\) counts how often term \(j\) appears in the context of term \(i\), where the context is a window of \(k\) words on either side. Rows are called words and columns contexts, so it is also called a word × context matrix. For the toy corpus with \(k = 2\) the slide shows:

The slide's table is a partial, illustrative one (for example user and system are more than two words apart in both sentences that contain them). The interactive figure in the SVD section counts the windows exactly, so some of its entries differ.

Each row of the matrix is a vector for its word, and each column a vector for its context. The sets of words and contexts can be the same or different. For instance, stop words such as for, the, and, a say little about a word, so we may keep fewer columns and leave them out. The lecture switches between both choices.

These rows are distributed, but they are still sparse M2 · 4:37: a word appears with only a few of the \(|V|\) other words, so most entries of a row are 0.

First, some fixable problems. Stop words are very frequent, so their counts are very high. Over all of Wikipedia, the, and or for may appear thousands of times in the window of machine, far more often than system or user. The counts are then skewed toward stop words. The lecture lists three fixes:

1. Ignore very frequent words: drop their columns.
2. Use a threshold \(t\) (say \(t = 100\)): \(X_{ij} = \min(\text{count}(w_i, c_j),\, t)\), where \(w\) is a word and \(c\) a context. "More than 100" is all we need to know about such a pair.
3. Use PMI instead of the count. The pointwise mutual information of a word and a context is

PMI is low when both words are frequent on their own but rarely appear together. It is high when they appear together nearly every time they appear at all, as when each occurs 100 times and they co-occur all 100 times. Frequent stop words get no credit just for being frequent.

One problem remains: if \(\text{count}(w, c) = 0\), then \(\operatorname{PMI}(w, c) = \log 0 = -\infty\). Two fixes:

Positive PMI (PPMI) also sets negative values to 0, so every cell is non-negative. The lecture offers only a rough intuition for this M2 · 9:07: a negative PMI says two words co-occur less often than chance, and it is unclear what a "negative relation" between words would mean. Even antonyms such as good and bad often appear in the same sentence. Replacing the counts by PMI gives the slide's matrix:

The severe problem: size M2 · 10:11

All of these are minor fixes. The real problem is unchanged from one-hot vectors: the rows are still very high dimensional (\(|V|\) entries), still very sparse (less so than one-hot), and they grow with the vocabulary, from 50K for PTB to 13M for Google 1T. The remedy is dimensionality reduction with the singular value decomposition (SVD). The lecture picks SVD over PCA because the matrix need not be square (for example after dropping stop-word columns), and SVD is the generalization of PCA to rectangular matrices.

From here on, \(X\) means the co-occurrence matrix with PPMI values, \(X_{\text{PPMI}}\), of size \(m \times n\) (\(m\) words, \(n\) contexts). Many variants of the co-occurrence matrix have been tried; PPMI is the most popular and most stable. SVD writes \(X\) as a product of three matrices, and the product can be expanded as a sum of rank-1 matrices:

The columns \(\mathbf{u}_i\) of \(U\) and \(\mathbf{v}_i\) of \(V\) are unit vectors, and \(\Sigma\) is diagonal with \(\sigma_1 \ge \sigma_2 \ge \dots\). Each term \(\sigma_i\mathbf{u}_i\mathbf{v}_i^\top \in \mathbb{R}^{m\times n}\) is the outer product of an \(m \times 1\) vector and a \(1 \times n\) vector, a matrix of rank 1. With all the terms the product equals \(X\). By the SVD theorem, stopping after the first term gives the best rank-1 approximation of \(X\), stopping after two terms the best rank-2 approximation, and so on. We write \(\hat X\) for the truncated sum.

"Approximation" should be read as compression. \(X\) has \(m \times n\) entries. The rank-1 approximation is rebuilt from only \(m + n + 1\) numbers: \(\mathbf{u} \in \mathbb{R}^m\), \(\mathbf{v} \in \mathbb{R}^n\) and \(\sigma \in \mathbb{R}\). The theorem guarantees that \(\mathbf{u}_1, \mathbf{v}_1, \sigma_1\) hold the most information in \(X\) (like the first principal component), and each later term \(\sigma_2\mathbf{u}_2\mathbf{v}_2^\top, \sigma_3\mathbf{u}_3\mathbf{v}_3^\top, \dots\) holds less and less important information.

An analogy: compressing colours M3 · 4:15

Suppose 8 bits encode shades of green. The last four bits say "green" and the first four say how light or dark:

In 8 bits the four codes look different. Asked to compress to 4 bits (as SVD compresses \(m \times n\) numbers into \(m + n + 1\)), we keep the most important bits, the ones that say "green", because telling green from red matters more than very light from very dark. After compression the similarity that was only slightly latent becomes obvious: all four become 1 0 1 1. SVD gives a similar guarantee: it keeps the most important information, and in doing so it exposes latent similarities between words.

The slide reconstructs the PPMI matrix from a low-rank SVD (the rank is not stated; it is more than 1 or 2). Two things change. The matrix is now dense. And cells that were 0 become nonzero:

Human–user and system–machine were 0 in \(X\): those words never appear together in the corpus. But system and machine both appear with the same kind of words (in the lecture's words, interface, install, run), and so do human and user. That shared company is information the matrix already contains. Forced to compress, SVD keeps the most important structure, and under that structure these pairs come out related. The latent co-occurrence between {system, machine} and {human, user} has become visible. Note that \(\hat X\) can contain negative entries, such as −0.13 for (for, user), although \(X\) has none.

Rows as vectors, \(XX^\top\) as similarities M3 · 7:36

Each row of \(X\) is a word's representation, so the \((i, j)\) entry of \(XX^\top\) is the dot product of row \(i\) and row \(j\). In the slide's small example, rows \([1, 2, 3]\) and \([1, 3, 5]\) give \(1 + 6 + 15 = 22\). A dot product is a cosine similarity before dividing by the two norms, so \(XX^\top\) roughly holds the similarities between all pairs of words.

With the original \(X\), the slide reports \(\cos(\text{human}, \text{user}) = 0.21\). With the reconstruction, \(\hat X\hat X^\top\) has entries 25.4 (human, human), 17.11 (user, user) and 6.84 (human, user), so

The compressed representation has learnt that human and user are more similar than the raw counts suggested.

After the SVD we still have to choose a vector for word \(i\). The \(i\)-th row of \(\hat X\) is better than the row of \(X\), since it has picked up the latent similarities. But \(\hat X\) is still \(m \times n\): each row still has \(n \approx |V|\) entries, even though it was built from far fewer numbers. What we want are vectors with far fewer dimensions whose dot products (similarities) equal those of the rows of \(\hat X\).

Define \(W_{\text{word}} = U\Sigma\), using the truncated factors. Since \(\Sigma\) is diagonal, \(\Sigma^\top = \Sigma\), and \(V^\top V = I\) because the columns of \(V\) are orthonormal:

The lecture uses \(m\), \(n\) and \(|V|\) interchangeably; when words and contexts are the same set, \(m = n = |V|\).

This was the standard way to get word vectors in NLP before deep learning M3 · 13:04: build a co-occurrence matrix, apply the PMI/PPMI heuristics, run SVD, and keep the top 100 to 300 dimensions. The result is a large compression that still gives related words a higher similarity, and the vectors can be fed to downstream tasks. The lecture says they will be put to use a few lectures later; the sequence models from Lecture 11 on read text one word vector at a time.

The methods so far are count-based models: they use co-occurrence counts. Nothing was learnt in the neural-network sense (apart from computing singular vectors). The rest of the lecture covers prediction-based models, which learn word representations directly as the parameters of a network trained on a prediction task. The plan:

1. The continuous bag of words (CBOW) model.
2. The skip-gram model with negative sampling (the famous word2vec).
3. GloVe word embeddings, a hybrid of count and prediction.
4. How to evaluate word embeddings.
5. Finally, the observation that good old SVD does just fine.

The task: predict the next word

Consider the task of predicting the \(n\)-th word given the previous \(n - 1\) words. Type "he sat on a" on a phone and the keyboard suggests "chair". This is multi-class classification: the input \(\mathbf{x}\) is the previous words and the output \(\mathbf{y}\) is a one-hot vector over the \(|V|\) words of the vocabulary. At test time the predicted distribution can supply the top few words as suggestions.

The training data comes for free. Take every \(n\)-word window in a corpus (the slides show 4-word windows over a film plot summary), use the first \(n - 1\) words as \(\mathbf{x}\) and the last as \(\mathbf{y}\), and slide the window along. For popular languages, all of Wikipedia is available, with no labelling effort. Choosing \(n\) is old ground in language modelling: the lecture notes that the last three words are usually enough to predict the next one, and that the default tool uses 7. For illustration we take \(n = 2\): predict the second word from the first.

Two questions follow: how to model this task, and what it has to do with learning word representations.

Use a feedforward network. The input is the one-hot vector \(\mathbf{x} \in \mathbb{R}^{|V|}\) of the context word, say sat. The output is a probability distribution over all \(|V|\) words, \(P(\text{he} \mid \text{sat}), P(\text{chair} \mid \text{sat}), \dots, P(\text{on} \mid \text{sat})\), and for this training window we want \(P(\text{on} \mid \text{sat})\) to be high. In between is one hidden layer \(\mathbf{h} \in \mathbb{R}^k\). The parameters are two matrices,

assuming that words and context words come from the same vocabulary. Structurally this is the MNIST classifier again (784 inputs, a hidden layer, 10 classes), with \(|V|\) inputs and \(|V|\) classes.

When \(\mathbf{x}\) is one-hot, \(W_{\text{context}}\mathbf{x}\) is simply the column of \(W_{\text{context}}\) where \(\mathbf{x}\) has its 1. The slide's example, with the second word on:

So there is a one-to-one correspondence between the words and the columns of \(W_{\text{context}}\): the \(i\)-th column is selected exactly when the \(i\)-th word is the input. We treat the \(i\)-th column as the representation of context \(i\). These columns are not known in advance. They are parameters, learnt by training.

The output function for multi-class classification is the softmax. If on is the \(i\)-th word of the vocabulary,

\(W_{\text{word}}^\top\mathbf{h}\) is the vector of \(|V|\) unnormalized scores; the slide writes it as \(W_{\text{word}}\mathbf{h}\). Its \(i\)-th entry is the \(i\)-th column of \(W_{\text{word}}\) dotted with \(\mathbf{h}\).

Since \(\mathbf{h}\) is the column of \(W_{\text{context}}\) for sat (say word \(j\)), \(P(\text{on} \mid \text{sat})\) is proportional to the exponential of the dot product between the \(j\)-th column of \(W_{\text{context}}\) and the \(i\)-th column of \(W_{\text{word}}\). The probability of word \(i\) depends on the \(i\)-th column of \(W_{\text{word}}\), so that column is also a representation of word \(i\).

Every word therefore gets two vectors: one for when it appears as a context (a column of \(W_{\text{context}}\)) and one for when it is the target (a column of \(W_{\text{word}}\)). SVD did the same: \(U\Sigma\) represented the words and \(V\) the contexts. The two methods learn the matrices differently, but the roles match.

From here on, \(c\) is the index of the context word (sat) and \(w\) the index of the correct output word (on). \(\mathbf{u}_c\) is column \(c\) of \(W_{\text{context}}\) and \(\mathbf{v}_w\) is column \(w\) of \(W_{\text{word}}\). The loss is the cross-entropy, which for a one-hot target keeps only the term of the correct word (softmax and cross-entropy were derived in Lecture 4):

Training is ordinary backpropagation with gradient descent. As the lecture points out, code written for MNIST would do, with only the input and output sizes changed.

Take one training pair \((c, w)\) and find the update for \(\mathbf{v}_w\). Expand the log of the fraction, then differentiate. Only the \(w' = w\) term of the sum involves \(\mathbf{v}_w\), and the derivative of the log-sum is that term's share of the sum times \(\mathbf{u}_c\):

The rule has a clean reading:

• If \(\hat y_w \to 1\), we already predict the right word, and \(\mathbf{v}_w\) is (almost) not updated.
• If \(\hat y_w \to 0\), \(\mathbf{v}_w\) gets a fraction of \(\mathbf{u}_c\) added to it. Adding one vector to another increases the cosine similarity between them, the same effect as the perceptron update in Lecture 2.

So training maximizes the cosine similarity between a word's vector \(\mathbf{v}_w\) and the vectors \(\mathbf{u}_c\) of the context words it appears with.

Not in the lecture: the same steps give \(\nabla_{\mathbf{v}_{w'}} = \hat y_{w'}\,\mathbf{u}_c\) for every other word \(w' \ne w\), so the wrong words' vectors are pushed away from \(\mathbf{u}_c\), each in proportion to the probability it was wrongly given.

Why similar words end up close M5 · 22:53

Take two words \(w\) and \(w'\) that tend to appear in similar contexts \(c\), such as dog and cat, which both appear with eats, sleeps and pet. Training gives both \(\mathbf{v}_{\text{dog}}\) and \(\mathbf{v}_{\text{cat}}\) a high cosine similarity with those context vectors, and so, transitively, a high cosine similarity with each other.

One gap remains M5 · 24:07. We never wanted dog close to eats, since they are not similar words. The resolution is that \(\mathbf{v}_{\text{dog}}\) moves toward \(\mathbf{u}_{\text{eats}}\), a column of the other matrix. Within \(W_{\text{word}}\), the matrix whose columns we use as the representations, dog and cat end up close, and dog and eats need not.

The lecture is explicit that this is only a reasonable intuition. There is no formal proof that it is what happens, and that is one criticism of word2vec: it works very well without a clear reason, while SVD rests on a principle. The neat picture also gets harder to see once there are more context words.

In practice the window has \(d\) words, not 2: \(d - 1\) context words predict one word (the lecture mentions 4 or 7, and has seen 11). With two context words, predict on from he and sat. Concatenate their one-hot vectors into \(\mathbf{x} \in \mathbb{R}^{2|V|}\) (in general \(\mathbb{R}^{(d-1)|V|}\)) and multiply by \([W_{\text{context}}, W_{\text{context}}] \in \mathbb{R}^{k\times 2|V|}\), two copies of \(W_{\text{context}}\) side by side. The product is the sum of the two selected columns:

In the slide's 3-word vocabulary the first block switches on sat (word 2) and the second he (word 3). Nobody actually performs this expensive multiplication: if he is word \(i\) and sat word \(j\), just look up those two columns and add them. The order of the context words does not matter, which is why the model is a bag of words: once the words are in a bag, their order is lost.

The output is unchanged: with \(k\) now the index of on,

The loss depends on \(W_{\text{word}}\) and on \(\mathbf{u}_{c_1}, \dots, \mathbf{u}_{c_{d-1}}\), and these are the parameters backpropagation updates: all columns of \(W_{\text{word}}\), because every column appears in the softmax denominator, but only the \(d - 1\) columns of \(W_{\text{context}}\) that were looked up. No gradient reaches the other columns of \(W_{\text{context}}\), so real implementations never touch them.

The slide leaves the update for \(\mathbf{v}_w\) as an exercise. Not in the lecture: the derivation is the same with \(\mathbf{u}_c\) replaced by \(\mathbf{h}\), so \(\nabla_{\mathbf{v}_w} = -\mathbf{h}(1 - \hat y_w)\) and \(\mathbf{v}_w \leftarrow \mathbf{v}_w + \eta\,\mathbf{h}(1 - \hat y_w)\): the target moves toward the sum of its context vectors. Each looked-up context column receives the same gradient \(\partial\mathcal{L}/\partial\mathbf{h} = -\big(\mathbf{v}_w - \sum_{w'}\hat y_{w'}\mathbf{v}_{w'}\big)\).

The problem: the softmax is expensive M5 · 35:08

The input side reduces to a lookup, but the output side does not. The softmax denominator sums over every word in the vocabulary, so each training step multiplies a \(|V| \times k\) matrix by a \(k\)-vector and exponentiates \(|V|\) scores. With \(|V|\) at 50K, 100K or 13M, that is not feasible at scale. The next modules give three ways around it.

CBOW predicts an output word from a bag of context words. The skip-gram model does the reverse: given the input word on, predict the context words on both sides of it, he, sat, a, chair. The roles of the matrices swap: the input word goes through \(W_{\text{word}}\), and the outputs are scored with \(W_{\text{context}}\).

With only one context word, this is the same network as before, and we get the same update rule, now for \(\mathbf{u}_c\) instead of \(\mathbf{v}_w\). With several context words there are several predictions, and the loss is a sum of cross-entropies:

The gradient of a sum is the sum of the gradients, so each term is handled exactly as in the one-word case. Skip-gram inherits CBOW's problem, only worse: a full softmax for each of the \(d - 1\) predictions. Three solutions follow: negative sampling, contrastive estimation and hierarchical softmax.

Build two sets of (word, context) pairs, both free to construct:

• \(D\): all correct pairs \((w, c)\) in the corpus. From "sat on a chair": (sat, on), (sat, a), (sat, chair), (on, a), (on, chair), (a, chair), and the same pairs reversed.
• \(D'\): incorrect pairs \((w, r)\), made by pairing \(w\) with a random context word \(r\) that has never appeared with it: (sat, oxygen), (sat, magic), (chair, sad), (chair, walking).

As before, \(\mathbf{v}_w\) is the vector of the word and \(\mathbf{u}_c\) of the context. Let the random variable \(z\) say whether a pair is correct (\(z = 1\)) or not, and model its probability with the sigmoid of a dot product:

We want \(p(z = 1)\) high for every pair in \(D\) and \(p(z = 0)\) high for every pair in \(D'\). The parameters \(\theta\) are all the word vectors \(\mathbf{v}_w\) and context vectors \(\mathbf{u}_c\). Maximizing the probability of all the pairs at once means maximizing a product, and taking logs turns it into a sum:

The first sum is largest when the dot products of true pairs are large: context vectors are pulled toward the vectors of the words they appear with, and, transitively as before, words that share contexts end up close. The second sum adds what the lecture stresses as the new ingredient: words that never appear in the same context are pushed apart. There is no expensive softmax anywhere.

How many negatives, and which ones M7 · 1:55

• \(k\) negatives per positive. In the original paper, Mikolov et al. sample \(k\) negative pairs for every positive pair, so \(|D'| = k\,|D|\). This reflects a natural imbalance: most pairs of words in a vocabulary never occur together, so incorrect pairs vastly outnumber correct ones. \(k\) is a hyperparameter, tuned on a validation set, for instance by checking that cat–dog or cat–feline come out closer than cat–truck.
• Drawing \(r\). The simplest choice is uniform over the words that never appear with \(w\). Better to account for frequency and draw frequent words more often, with the unigram distribution \(p(r) = \text{count}(r)/N\), where \(N\) is the total number of words in the corpus. word2vec uses a modified unigram distribution, raised to the power 3/4, a value found to give the best performance:

Not in the lecture: the power flattens the distribution a little. Two words with unigram probabilities 0.9 and 0.1 are drawn with probabilities about 0.84 and 0.16, so rarer words are sampled somewhat more often than their frequency alone would give.

The lecture adds a remark that matters later M7 · 6:31. word2vec had a huge effect on NLP: word vectors became the standard features for almost every application. Later, careful analysis showed that its heuristics (the 3/4 power, the value of \(k\), the learning rate) must be tuned well for it to beat SVD, and that adding similar knobs to SVD makes its vectors just as good. See evaluation.

The second way around the softmax uses the same idea of true versus random. Take a positive window, "He sat on a chair", and make a negative one by replacing a word with a random word: "He sat abracadabra a chair". A feedforward network takes the vectors of the word pair, \(\mathbf{v}_c\) for sat and \(\mathbf{v}_w\) for on (\(2d\) numbers for \(d\)-dimensional vectors), passes them through a hidden layer \(W_h \in \mathbb{R}^{2d\times h}\), and outputs a single score \(s\) through \(W_{\text{out}} \in \mathbb{R}^{h\times 1}\). The same network on (sat, abracadabra) gives a score \(s_c\) for the corrupted window. The output layer has one unit instead of \(|V|\).

The loss is built up from a wishlist:

• We want \(s > s_c\), so try to maximize \(s - s_c\).
• A tiny gap is not convincing: \(s = 0.99\) against \(s_c = 0.98\) barely separates the windows. Ask for a margin \(m\) (with scores in \([0, 1]\), perhaps 0.3 or 0.4), that is, maximize \(s - (s_c + m)\).
• Once \(s > s_c + m\) the network is doing its job, so the loss should be 0 and nothing should be backpropagated.

The slide writes the objective as "maximize \(\max(0, s - (s_c + m))\)". Read literally, that would keep rewarding ever larger gaps. The behaviour the lecture describes (no loss and no update once \(s > s_c + m\)) is the hinge loss in the box. Standard name: a margin ranking loss.

The third solution is a clever trick, less obvious than the other two, used in NLP applications where speed matters. Keep the input side (one-hot sat, \(W_{\text{context}}\), \(\mathbf{h} = \mathbf{v}_c\); note that the context vector is now called \(\mathbf{v}_c\)), but replace the output layer by a binary tree:

• The tree has \(|V|\) leaves, one per word, and there is a unique path from the root to each leaf.
• Let \(l(w_1), l(w_2), \dots, l(w_p)\) be the nodes on the path from the root to word \(w\), and let \(\pi(w)\) be a binary vector with \(\pi(w)_k = 1\) if the path branches left at node \(l(w_k)\) and 0 otherwise. In the figure, \(\pi(\text{on}) = (1, 0, 0)\): left, right, right. For a balanced tree the path has \(\log_2|V|\) nodes (3 for 8 words).
• Each internal node \(i\) carries a vector \(\mathbf{u}_i\). The parameters are \(W_{\text{context}}\) and \(\mathbf{u}_1, \mathbf{u}_2, \dots\). A binary tree with \(|V|\) leaves has \(|V| - 1\) internal nodes, so this is in effect the same number of parameters as \(W_{\text{word}}\).

The probability of predicting a word is the probability of taking its path from the root:

Each branching probability is a sigmoid of the dot product between the context vector and the vector of the node, \(\mathbf{u}_i\):

Training to maximize these probabilities makes the context vector \(\mathbf{v}_c\) highly similar to \(\mathbf{u}_i\) at nodes where the path to the target branches left, and dissimilar where it branches right. Contexts that appear with the same words get pushed the same way at the same nodes. If cat and dog both appear with sleep, both move toward or away from the same node vectors, so their representations become similar: the same transitive argument as before.

With the tree, \(p(w \mid \mathbf{v}_c)\) takes \(|\pi(w)|\) dot products and sigmoids instead of the \(|V|\) the softmax requires. As for building the tree, even a random arrangement of the words on the leaves works well in practice M9 · 12:05. (The lecture leaves as a quiz question whether a random tree causes any problem.)

Not in the lecture: because the two branches at every node sum to 1, the probabilities of all \(|V|\) leaves also sum to 1. The tree gives a proper distribution over words without ever computing a normalizing sum.

Count-based methods (SVD) compute word representations from global co-occurrence counts over the whole corpus. Predict-based methods learn representations through a prediction task, one (word, context) pair at a time. GloVe (global vectors) combines the two: learn vectors by gradient descent, but make them faithful to the global counts.

\(X_{ij}\) encodes important global information about how words \(i\) and \(j\) co-occur: global because it is computed from the entire corpus (whether as counts, PMI or PPMI). A dot product \(\mathbf{v}_i^\top\mathbf{v}_j\) measures how well two words go together, and so does \(P(j \mid i)\), the probability of seeing \(j\) in the context of \(i\). GloVe asks the two to agree:

Similarly \(\mathbf{v}_j^\top\mathbf{v}_i = \log X_{ij} - \log X_j\), using \(X_{ij} = X_{ji}\). The left sides are equal, so add the two equations:

\(\log X_i\) depends only on word \(i\) and \(\log X_j\) only on word \(j\). To give the model more freedom, replace them by word-specific biases \(b_i\) and \(b_j\) that are learnt too: \(\mathbf{v}_i^\top\mathbf{v}_j + b_i + b_j = \log X_{ij}\). The left side is the value predicted from the parameters, the right side the value computed from the corpus, so minimize the squared difference:

This objective weighs all co-occurrences equally, which is a drawback. The fix is a weighting function \(f(X_{ij})\), chosen so that neither rare nor frequent words are over-weighted:

The lecturer corrects the notation in the video M10 · 6:36: as in the other models there should be a word vector and a separate context vector, \(\mathbf{v}_i^\top\mathbf{u}_j\). Then cat and feline both move toward the context vectors of the words they co-occur with, and so toward each other.

Not in the lecture: \(f(0) = 0\), so pairs that never co-occur, where \(\log X_{ij}\) would be \(-\infty\), drop out of the sum.

Any of the models (CBOW, skip-gram, GloVe, each with its variants) produces word vectors. Three kinds of test compare them:

Semantic relatedness

Ask humans to judge how related each of many word pairs is, say on a scale from 0 to 1. For the same pairs compute the model's cosine similarity,

and compute the correlation between the two sets of scores. Model 1 is better than Model 2 if \(\operatorname{corr}(S_{\text{model1}}, S_{\text{human}}) > \operatorname{corr}(S_{\text{model2}}, S_{\text{human}})\). The same test can choose hyperparameters: keep the setting whose vectors give the higher correlation.

Synonym detection M11 · 2:31

Resources such as WordNet supply synonyms. Each test item gives a term and a few candidates, one of them the correct synonym and the rest distractors. The model picks

and its accuracy over many items (60 right out of 100 is 60%) compares models.

Analogy M11 · 3:35

Semantic analogy: brother is to sister as grandson is to ?. Find the nearest neighbour of \(\mathbf{v}_{\text{sister}} - \mathbf{v}_{\text{brother}} + \mathbf{v}_{\text{grandson}}\); it should be granddaughter. Syntactic analogy: work is to works as speak is to ?. The nearest neighbour of \(\mathbf{v}_{\text{works}} - \mathbf{v}_{\text{work}} + \mathbf{v}_{\text{speak}}\) should be speaks. The first two tasks use dot products; analogies use arithmetic on the vectors.

Count versus predict: which is best M11 · 5:44

• Baroni et al. [2014] found that predict models consistently outperform count models on all tasks.
• Levy et al. [2015] did a more thorough analysis (in the instructor's opinion, because it gave SVD a fair chance). word2vec has tuned knobs, such as the 3/4 power on the sampling distribution and the number \(k\) of negative samples, so they gave SVD analogous ones: raise the counts to a tunable power before computing PMI, and an adjustment that mirrors the \(k\) negative samples. With these, good old SVD does better than the prediction-based models on similarity tasks, though not on analogy tasks.

Most NLP applications need similarity far more often than analogies, so in most cases SVD would be just fine. That is the twist promised at the start.

SVD is a matrix factorization of the co-occurrence matrix. Levy et al. also showed, under some assumptions (the lecture gives the intuition, not the proof), that word2vec implicitly does a matrix factorization. word2vec learns two matrices, \(W_{\text{context}}\) and \(W_{\text{word}}\), and their product is a matrix \(M\) closely tied to PMI:

The slide writes \(M = W_{\text{context}} * W_{\text{word}}\); the lecturer points out that a transpose is needed. With both matrices of size \(k \times |V|\), the matrix of all word–context dot products is \(W_{\text{word}}^\top W_{\text{context}}\).

So word2vec factorizes a shifted PMI matrix, very much like what SVD does. Take the PMI matrix, subtract \(\log k\) from every entry, run SVD, and you essentially get back the word2vec representations. At an intuitive level this is expected: both methods rely on the same principle, that words appearing together (the distributional idea) should get related representations. SVD turns co-occurrence into counts and factorizes them; word2vec turns it into updates.

word2vec is still preferred in most applications, for computational reasons. SVD needs the eigenvectors of a huge matrix such as \(X^\top X\). Even with very good libraries this costs roughly \(O(n^{2.\text{something}})\), and the full high-dimensional PMI matrix must be held in memory. word2vec is iterative by design: each gradient step updates only a few parameters, never the whole model at once, and with negative sampling, contrastive estimation or hierarchical softmax each step is cheap.

One-hot vector

A \(|V|\)-dimensional vector with a single 1 at the word's index; any two are \(\sqrt2\) apart with cosine 0.

Co-occurrence matrix

Word × context matrix whose \((i, j)\) entry counts how often context \(j\) falls within \(k\) words of word \(i\).

PMI

\(\log\frac{\text{count}(w,c)\,N}{\text{count}(w)\,\text{count}(c)}\): how much more often two words co-occur than chance would predict.

PPMI

PMI with negative values (and the \(-\infty\) of zero counts) replaced by 0; PMI₀ replaces only the zero-count cells.

Truncated SVD

\(\hat X = \sum_{i \le k}\sigma_i\mathbf{u}_i\mathbf{v}_i^\top\), the best rank-\(k\) approximation; it exposes latent co-occurrences.

\(W_{\text{word}} = U\Sigma\)

\(k\)-dimensional word vectors with \(W_{\text{word}}W_{\text{word}}^\top = \hat X\hat X^\top\); \(W_{\text{context}} = V\).

Prediction-based model

Learns word vectors as the parameters of a network trained on a prediction task whose data comes free from raw text.

CBOW

Predicts a word from the sum of its context vectors, \(\mathbf{h} = \sum\mathbf{u}_{c_i}\), with a softmax over \(|V|\).

Skip-gram

Predicts each context word from the input word; the loss is a sum of cross-entropies.

Negative sampling

Maximize \(\log\sigma(\mathbf{u}_c^\top\mathbf{v}_w)\) on true pairs and \(\log\sigma(-\mathbf{u}_r^\top\mathbf{v}_w)\) on \(k\) random pairs drawn from \(p(r)^{3/4}\).

Contrastive estimation

Score a true and a corrupted window and minimize \(\max(0, m - (s - s_c))\).

Hierarchical softmax

A word's probability is the product of sigmoids along its root-to-leaf path in a binary tree: about \(\log_2|V|\) terms.

GloVe

Fits \(\mathbf{v}_i^\top\mathbf{v}_j + b_i + b_j\) to \(\log X_{ij}\) by weighted least squares with weights \(f(X_{ij})\).

Word analogy

Evaluation by vector arithmetic: the nearest neighbour of \(\mathbf{v}_{\text{sister}} - \mathbf{v}_{\text{brother}} + \mathbf{v}_{\text{grandson}}\) should be granddaughter.