Attention Is All You Need, Animated — The Paper Every LLM Is Built On

TOKENS → VECTORS

Words don't go into a neural network. Numbers do.

FROM WORDS TO NUMBERS.

Before anything can attend to anything else, a sentence has to become math. Each token — here, roughly one word — is looked up in an embedding table: a matrix with one learned row per vocabulary entry. That row becomes the token's starting vector, 512 numbers wide in the paper's base model.

Real models split words into subword pieces (byte-pair encoding, WordPiece, SentencePiece) so rare words don't blow up the vocabulary. This demo tokenizes on whitespace instead, for clarity — type a short sentence and watch it drop into vector form.

WHY VECTORS AT ALL?

A vector can sit near other similar vectors in space — that's the whole trick. Nothing here "understands" English; addition and dot products do all the work from here on.

TOKENIZATIONWhitespace here · BPE/WordPiece in real models

EMBEDDING DIM d_model512 (paper's base model)

VOCAB SIZE (paper)≈37,000 shared BPE tokens

LOOKUPOne learned row per token, trained by backprop

LIVE SIMULATIONSIM/01 · TOKENS → VECTORS

TOKENS 6 · SHOWING 8 OF 512 DIMS

Type a sentence, press run. Each column is one token's vector: height is magnitude, blue is positive, red is negative — 8 illustrative dimensions standing in for the real 512.

A SENSE OF ORDER

Attention has no idea what order words came in. This is how it finds out.

NO ORDER WITHOUT HELP.

Attention compares every token to every other token at once — nothing about the mechanism cares whether "dog bites man" or "man bites dog" came first. So before anything else happens, the paper adds a fixed positional encoding vector to every token embedding: sine on even dimensions, cosine on odd ones, at wavelengths that stretch geometrically from 2π out to 10000·2π.

Each dimension oscillates at its own frequency, so nearby positions get nearby vectors. The sinusoid form even lets a trained model extrapolate to sequence lengths it never saw, since PE(pos+k) is a linear function of PE(pos).

THE EXACT FORMULA

PE(pos, 2i) = sin(pos / 10000^2i/d)
PE(pos, 2i+1) = cos(pos / 10000^2i/d)

TYPEFixed sinusoid, not learned

WHY SIN & COSRelative offsets become a linear function

ALTERNATIVELearned position embeddings (BERT, GPT)

RANGEEvery value lies in [−1, 1]

LIVE SIMULATIONSIM/02 · POSITIONAL ENCODING

POSITION 0 OF 16 · DIM 16

SEQUENCE LENGTH16

SHORTLONG

DIMENSIONS SHOWN16

FEWMANY

Exact values, not illustrative. Top: every position × dimension cell, coloured by value. Bottom: two dimensions unrolled as waves — low dimensions oscillate fast, high dimensions slow. The amber line scrubs through positions automatically.

QUERY, KEY, VALUE

Every token asks a question, offers an answer, and carries a payload.

EVERY TOKEN ASKS A QUESTION.

Each token is projected into three vectors by three learned matrices: a query (what am I looking for?), a key (what do I contain, that others might want?), and a value (what do I actually carry, if picked?). Comparing one token's query against every token's key gives a compatibility score — how much attention that token should pay to each other token.

The scores pass through softmax to become a probability distribution that sums to 1, then that distribution weights a sum of every token's value vector. The result is a new vector for each token — a blend of the whole sentence, weighted by relevance.

WHY DIVIDE BY √d_k?

Dot products of long vectors can grow large, which pushes softmax into a region where its gradient is nearly flat. Dividing by √d_k keeps scores in a range softmax can still learn from — toggle it off on the right and watch the distribution collapse.

FORMULAAttention(Q,K,V) = softmax(QKᵀ/√dₖ)V

d_k IN THE PAPER64 per head

COMPLEXITYO(n²·d) per layer, n = sequence length

MASKINGDecoder self-attention hides future tokens

LIVE SIMULATIONSIM/03 · CLICK A TOKEN TO QUERY IT

QUERYING cat

Click any token. Line thickness and the bars below are its softmax attention weight over every token, including itself. Turn scaling off to see why the paper insists on it.

MANY HEADS, ONE SENTENCE

Eight parallel conversations about the same six words.

EIGHT HEADS, EIGHT VIEWPOINTS.

One attention pattern only captures one kind of relationship. The paper instead runs h = 8 independent attention heads in parallel, each with its own learned Q/K/V projections into a smaller 64-dimensional space, then concatenates all eight outputs back into one 512-wide vector and passes it through one more learned matrix, Wᴼ.

In trained models, heads tend to specialise — some track adjacent words, some track subject↔verb agreement across a clause, some attend almost entirely to a single "sink" token. Nobody tells them to; it falls out of training.

WHY NOT ONE BIG HEAD?

A single 512-dimensional attention pattern averages everything into one blend. Splitting into 8 smaller heads costs the same total compute (8×64=512) but lets each head commit to a different, sharper pattern instead of compromising.

HEADS h8 (base model) · 16 (big model)

PER-HEAD DIM d_k=d_v64

OUTPUT PROJECTIONWᴼ ∈ ℝ^(512×512)

COSTSame total FLOPs as one full-width head

LIVE SIMULATIONSIM/04 · MULTI-HEAD ATTENTION

HEAD 1 OF 4 SPOTLIGHTED

NUMBER OF HEADS h4

FEWERMORE (8 = PAPER'S BASE MODEL)

Each grid is one head's attention matrix over the same six-token sentence — row attends to column. Patterns here are hand-shaped archetypes (local, sink, diagonal…), not a trained checkpoint. Click a head to spotlight it.

STACK IT UP

Six of these layers, twice over — once reading, once writing.

SIX LAYERS DEEP, TWICE OVER.

The full Transformer is two stacks of N = 6 identical layers. The encoder reads the source sentence: each layer is self-attention, then a position-wise feed-forward network (512→2048→512), each wrapped in a residual connection and layer normalization. The decoder adds a third sublayer — cross-attention, where queries come from the sentence being generated but keys and values come from the encoder's output.

The decoder's self-attention is masked: position i can only attend to positions ≤ i, so the model can't peek at words it hasn't generated yet. That's what keeps training in parallel and generating one token at a time consistent with each other.

RESIDUALS AND NORM

Every sublayer computes LayerNorm(x + Sublayer(x)) — the raw input skips around the sublayer and is added back before normalizing. That skip is why 6-layer stacks (and modern 96-layer ones) are trainable at all: gradients keep a direct path back to the input.

LAYERS N6 encoder + 6 decoder (base)

FFN INNER WIDTH2048

PARAMETERS65M (base) · 213M (big)

NORMALIZATIONLayerNorm around every sublayer

LIVE SIMULATIONSIM/05 · ENCODER–DECODER STACK

N = 6 LAYERS PER STACK · WATCH THE PULSE CLIMB

LAYERS N6

SHALLOWPAPER'S N=6

Bottom to top is input to output. The amber pulse is data climbing the stack; the amber diagonal is cross-attention pulling from the encoder into every decoder layer; the green curves are residual skips.

ONE TOKEN AT A TIME

The whole stack runs again for every single word it writes.

ONE WORD. THEN THE WHOLE THING AGAIN.

A trained Transformer doesn't write a sentence — it writes one token, appends it to what it's already written, and reruns the entire stack on the longer sequence to write the next one. That's why longer outputs cost more: token N needs N passes through every layer that came before it, not one.

The decoder's output at the last position is projected to vocabulary-size logits and passed through softmax, giving a probability for every possible next token. Temperature reshapes that distribution before a token is picked — low temperature sharpens it toward the top choice, high temperature flattens it toward uniform.

GREEDY VS. SAMPLING

Always picking the single highest-probability token (greedy decoding) is deterministic but can loop or flatten out; sampling from the distribution adds variety at the cost of predictability. Most production chat models sample, with temperature tuned per use case.

OUTPUT LAYERLinear (d_model→|vocab|) + softmax

DECODING STRATEGIESGreedy · sampling · beam search

COST PER TOKENOne full forward pass, every layer

THIS DEMO9-word toy vocabulary, hand-set logits

PRIMARY SOURCES

▸ THE PAPER ON ARXIV ▸ NEURIPS 2017 PROCEEDINGS ▸ THE ILLUSTRATED TRANSFORMER

LIVE SIMULATIONSIM/06 · AUTOREGRESSIVE GENERATION

GENERATED: the

TEMPERATURE0.70

SHARPFLAT

Watch it write, one token per step. The bars are the toy model's next-token distribution; the amber bar is what gets appended. A tiny hand-written lookup table stands in for real logits — the loop itself is exactly how generation works.

READ THE EQUATION

Six pieces, one line, most of modern AI.

TERM EXPLORERSELECT A TERM TO READ IT IN WORDS

· / )

—

Select a term above.

Q	Query — a learned projection of the current token: "what am I looking for?"
K	Key — a learned projection of every token: "what do I offer, if compared against?"
V	Value — a learned projection of every token: "what do I actually contribute, if picked?"
d_k	Dimension of the key/query vectors — 64 in the paper's base model, per head.
h	Number of parallel attention heads — 8 in the base model, 16 in the big model.
d_model	Width of every token vector flowing through the network — 512 in the base model.

This single line, run 8 times in parallel per layer and stacked 6 layers deep, is most of what a Transformer does. Everything else — embeddings, positions, feed-forward layers, residuals — moves information into and out of this equation.

◈ BEHIND THE VISUALS

SIM/01 and SIM/03 use a deterministic pseudo-embedding — a sine function seeded by each word's characters — standing in for a real, learned 512-dimensional embedding table and learned Q/K/V projection matrices. The shapes are illustrative; no weights were trained.

SIM/02 plots the paper's exact sinusoidal positional-encoding formula, PE(pos,2i)=sin(pos/10000^(2i/d)), PE(pos,2i+1)=cos(pos/10000^(2i/d)) — that one is exact, not illustrative.

SIM/04's eight attention-pattern grids are hand-shaped archetypes (local, sink, diagonal, and similar patterns reported in interpretability studies of real trained models), not sampled from an actual checkpoint. SIM/05 draws the paper's stated layer count, sublayer order, and residual/cross-attention wiring exactly; the travelling pulse is a pacing device, not a measured signal. SIM/06 generates text from a tiny hand-written nine-word vocabulary and lookup table, not a trained language model — the mechanism (softmax over logits, temperature, greedy vs. sampling) is real, the "opinions" behind the numbers are invented.

ATTENTION IS
ALL YOU NEED.

◈ WHAT THIS PAGE SHOWS — AND WHAT IT DOES NOT