Explainer
Embeddings
Lists of numbers that place words, sentences or images so that similar meanings sit close together.
An embedding turns a piece of content into a long list of numbers. Picture each list as a point on a map where similar meanings sit near each other. Real embeddings have hundreds or thousands of directions instead of a map's two (GPT-3's had 12,288, 3blue1brown.com), but the idea of nearness is the same. The useful property is distance. Texts with related meanings end up near each other, even when they share no words, so "How do I reset my password?" lands close to "I forgot my login". Search tools use this to find passages by meaning instead of exact wording, and recommendation systems use it to find similar items. Inside a language model, every token is first turned into an embedding, and the model's layers keep refining those lists of numbers as they take in more context.
Going deeper
The word2vec paper (Mikolov et al., 2013) helped make learned word vectors widely used. It describes earlier work by the same group showing that "King" minus "Man" plus "Woman" lands nearest to "Queen", and in its own tests it leaves the input words themselves out of such searches (arxiv.org).