How does a neural network turn words into numbers?
Type a sentence. Watch it become pieces, then rows of numbers, then points on a map where meaning is distance.
Each piece gets a number from a fixed list. “##” marks a piece that continues the previous one. Case is ignored.
The list is looked up in a table, one row per vocabulary entry. No single number means anything by itself.
Closest to “queen”
- princess84%
- mother82%
- woman72%
Similarity is the angle between two rows of numbers. 100% means the same direction.
“queen” is one whole entry in the vocabulary. Its numbers put it next to princess, mother, woman.
Tap a dot on the map to inspect any word. The vocabulary is tiny on purpose. This is an illustration, not a trained model.
A model never sees letters.
Before anything else, the text is cut into pieces called tokens. Common words stay whole. Rare words are split into parts the model has met before, so “unbelievable” becomes “un”, “believ”, “able”.
Every piece has a number in a fixed list, the vocabulary. That number is only a label, like a locker number. It says nothing about meaning yet. Real vocabularies hold from 30,000 to 200,000 entries.
Try a real tokenizerEach label unlocks a row of numbers.
Inside the model there is a big table with one row per vocabulary entry. A row is called an embedding. Here a row has ten numbers. In real models it has hundreds or thousands.
Before training, every row is random noise. During training, the model guesses the next token billions of times. Each miss nudges the rows a little. Words that show up in similar company end up with similar rows.
Nobody labels the numbers. One number on its own means nothing. Together they fix a position in space, and position is what the rest of the network works with.
Meaning is distance.
Similar words sit close together: “kitten” next to “puppy”, “Paris” next to “London”. Directions matter too. The step from “man” to “woman” is roughly the same as the step from “king” to “queen”. So you can do arithmetic with words.
Closest words to the result: queen 95%, mother 87%, princess 86%. The three input words are left out.
Each strip is one row of numbers. Subtracting removes what the two words share; adding puts in a new direction.
This trick made word vectors famous in 2013. Today’s language models still start the same way: a lookup from token to row. Everything the model knows about a word begins with that row.
The paper that showed king − man + woman ≈ queenThen context moves the points.
The table gives each token the same row every time. That cannot be the whole story: “bank” by a river and “bank” with money are different things. So the layers that follow mix every row with its neighbours in the sentence. The point for “bank” drifts toward water or toward money.
At the very end, the model scores every vocabulary entry against the final position, picks the next token, and the number turns back into text. Words in, numbers throughout, words out.
A longer illustrated walk-throughA small note about this demo
Nothing here is trained. The vocabulary has about ninety whole words plus a few word pieces, letters, digits and punctuation. The tokenizer is a greedy longest-match, in the spirit of WordPiece, and it ignores case. Token ids are scrambled so they look as arbitrary as real ones.
Each word vector is built from ten hand-picked features, such as “animal” or “royal”, with small deterministic noise added so the numbers look learned. That is why the map and the arithmetic behave the way real embeddings do. Fragments, symbols and Cyrillic letters get placeholder vectors near the middle of the space.
The map is a fixed linear projection of the ten numbers onto a plane, not a learned layout like t-SNE or UMAP. Similarity is cosine similarity, shown as a percentage. The arithmetic demo leaves the three input words out of the search for the closest match, which is the usual convention.