The walkthrough begins before attention: how text becomes numbers, what an embedding can store, and why the same starting vector is not enough for every sentence. It then builds attention from one dot product to the complete matrix operation.
A model never receives meaning directly. It receives rows of learned numbers, then updates those rows using context.
01 / 09
First: how a model represents a word.
A computer cannot calculate with the idea of “bank.” A tokenizer first splits text into tokens and assigns each token an integer ID. An embedding table then turns each ID into a vector: an ordered row of learned numbers.
textbanka word here; sometimes a token is only part of a word
→
token ID4821an index, not a meaningful quantity
→
embedding lookupE[4821]select row 4821 from a learned table
→
input embedding[0.12, -0.44, 0.71, 0.09]four dummy coordinates; real vectors are usually much wider
Embedding table
One learned row per vocabulary token. Before the sequence is processed, the same token ID retrieves the same starting row.
Position information
A separate signal tells the model where the token occurs. Without it, content-only attention cannot distinguish reordered tokens.
Contextual representation
After attention and later layers, the row at that position has been updated using other tokens. It is now context-dependent.
The exact same word [BANK] points to different concepts. The evidence that separates them lives in the surrounding tokens. I generated this collage of bank images using gpt-image-2.
same starting lookup for bankXbank = [0.12, -0.44, 0.71, 0.09]shape: 1 × 4
attention weights for bankAbank = [0.04, 0.42, 0.28, 0.06, 0.08, 0.12]shape: 1 × n · one coefficient per displayed word
For this sentence, road and curves receive the largest weights. Attention computes the update as AbankV. A Transformer block later combines that update with the input through its residual path.
The context mechanism
Attention connects a position to useful neighbours.
The current position is matched with the other positions, those matches become weights, and the value rows are combined as a weighted average.
1 connect with neighbours→2 weighted average→3 new contextualised word
02 / 09
Before attention, every translation had to pass through one fixed-width state.
This is the historical problem attention first addressed. It was not literally “one token storing the sentence.” It was one vector, usually the encoder RNN or LSTM’s final hidden state, acting as the only summary available to the decoder.
RNN
Reads tokens in order. At step t, it combines the current token with the previous hidden state: ht = f(xt, ht-1).
LSTM
Adds gates and a memory cell so information can survive longer. It improves recurrence, but a basic sequence-to-sequence decoder still receives one final fixed-width summary.
Hidden state
A vector of numbers representing what the recurrent network has carried forward so far. It is model memory, not a word and not a token.
Before attention: one route through c
encoder reads left to rightevery decoder step reuses the same summary
Theh₁
→
studenth₂
→
solvedh₃
→
the problemh₄
all source details→c = h₄ one fixed-size vector→decoder
DerSchülerlöstedie Aufgabe
The recurrence has already blended earlier information into h₄. The problem is the interface: names, actions, word order, and long-distance details must all survive in the same fixed number of coordinates, and the decoder cannot revisit h₁, h₂, or h₃ directly.
With attention: keep every encoder state available
cᵢ = .05h₁ + .10h₂ + .18h₃ + .67h₄
For “Aufgabe,” the decoder gives the largest weight to the source state for “problem.” It still forms a soft mixture, so other states can contribute.
The encoder could remain recurrent
Attention was first added on top of RNN and LSTM encoders. The encoder still read tokens one after another and produced h₁, h₂, …, hₙ.
Each hidden state kept a fixed width
Every hⱼ has the same number of coordinates, such as 512. A longer sentence creates more state vectors, not wider state vectors.
The decoder gained direct access
At output step i, weights αij combine all encoder states into a fresh context vector ci, instead of forcing every step to reuse only the final state.
From recurrent attention to the Transformer
Early attention kept the recurrent encoder and removed the single-vector interface bottleneck by letting every decoding step read all encoder states. The Transformer later removed recurrence from the central sequence-mixing path: self-attention lets every position directly exchange information with every permitted position.
03 / 09
Now attention becomes a learned retrieval operation.
We have a sequence of vectors and a reason to update them. Before doing arithmetic, give the three names a job: a query asks what the current position needs, keys advertise what each position can match, and values carry the content that will be mixed in.
Q
Query · the request
For the token currently being updated, this vector encodes what kind of evidence would be useful. A query for bank might match nearby words about roads, water, or money.
“I need context that clarifies this position.”
K
Key · the match signal
Every position produces a key. Dot products compare the current query with every key and create one score per candidate position.
“Here is what I can match.”
V
Value · the payload
Every position also produces a value. Once the scores become weights, those value rows are averaged in the proportions the query requested.
“Here is the information to retrieve.”
Attention in one sentencecompare the query with every key → normalize the matches → mix the corresponding values
Q, K, and V are learned numeric views, not human-readable labels. The model discovers useful directions because training rewards predictions that use them.
SLIDE 1 OF 10Start with a sequence
SLIDE 01
04 / 09
The equation, with shapes and numbers.
Attention is a sequence of ordinary matrix operations. The transpose makes the inner dimensions line up. The softmax is applied across each row, so each query distributes one unit of weight over the keys it is allowed to see.
1scores = QKT(nq × dk) · (dk × nk) = nq × nk
2scaled = scores / √dkthe shape stays nq × nk
3masked = scaled + MM also has shape nq × nk: 0 keeps an allowed score and −∞ blocks a disallowed pair
4weights = softmax(masked)row-wise normalization; shape stays nq × nk
Attention(Q,K,V) = softmax(QKT / √dk + M)V M = 0 when no positions are masked
Qnumber of queries × key width
Knumber of keys × key width
Vnumber of values × value width
Anumber of queries × number of keys
The reason for dividing by √dk. A dot product adds one product for each coordinate. As the width grows, raw scores tend to grow too. Large logits push softmax toward almost one-hot rows, which makes gradients less useful. Scaling keeps the values in a workable range.
The meaning of the output. Row i of A is a set of coefficients. Multiplying it by V takes the same coefficient-weighted sum of the value rows. The output is a new representation, not a pointer to one original token.
learned during training
Parameters that persist
The embedding table and projection matrices WQ, WK, WV, and WO are adjusted by gradient descent. They are reused across sentences after training.
produce
computed for this input
Activations that change
Q, K, V, scores, attention weights, and output vectors are recomputed from the current sequence. This is why a different context produces different numbers.
05 / 09
Self-attention, one matrix operation at a time.
The complete calculation is stacked vertically so the dimensions remain visible from input to output. Move through the nine stages; the active operation is highlighted while the full pipeline stays in place.
STEP 1 OF 9
Project the queries
Multiply every input row by WQ.
06 / 09
Self-attention, cross-attention, and masks reuse the same calculation.
The operation stays the same. What changes is where Q, K, and V come from, and which positions are allowed to contribute.
Self-attention
All three projections start from the same sequence X, but use separate learned matrices:
Q = XWQ K = XWK V = XWV
A token can read other tokens in the same sequence. In a bidirectional encoder, it can read both left and right context.
In ordinary retrieval, a separate search request asks about a fixed collection. In self-attention, each token supplies its own request, and every token can be part of the collection.
Cross-attention
The decoder supplies the query. The encoder supplies keys and values:
Q = YWQ K = XWK V = XWV
This lets a target-language position retrieve information from the source sequence. For example, a decoder position generating a word about the river can retrieve the source phrase containing river bank.
Causal masking
For next-token prediction, position i may read positions up to i, but not future positions.
100110111
The zeros replace scores with negative infinity before softmax. A padding mask is a different mask: it hides padding tokens rather than future tokens.
Choose a token
For the selected token it, the bars show one attention row. In a real model these numbers are learned from the data.
Q = itrow sum 1.000
07 / 09
Multiple heads learn different relationships in parallel.
A single attention map has one set of weights for a layer. Multi-head attention runs several smaller attention operations, each with its own projections. One head may prefer nearby syntax while another gives more weight to a longer reference. The head outputs are concatenated and passed through an output projection, so the layer returns to the original model width.
Xsequence representation
→
Q, K, Vseparate projections per head
→→
concatthen multiply by WO
HEAD 1
Nearby syntax
This head gives high weight to nearby positions. It might learn a pattern such as a determiner relating to the noun that follows.
MultiHead(Q,K,V) = Concat(head₁, …, headh)WO
With dmodel = 8 and two heads, each head can work in a four-dimensional subspace. The smaller width keeps the total amount of representation constant while allowing different learned similarity spaces.
One complete head
I use three learned projections before scoring.
I feed the same position-aware sequence X into three linear projections. The results differ because WQ, WK, and WV are separate learned matrices.
Q and K provide matching features. V carries the content that the attention weights combine. Scaling by √dk keeps the dot products in a useful range.
complete headQKT (n × n) → A (n × n) · V (n × dv) = Z (n × dv)
Second worked example
Two heads, one four-dimensional input.
Use three tokens from a bank sentence: river, bank, flooded. To keep the arithmetic visible, the example uses selector projections. Head 1 reads the first two features and head 2 reads the last two. Real heads use learned projections that can mix every input feature.
X · 3 tokens × 4 features
river1010bank0101flooded1100
Each row is one position. The two heads see different two-dimensional views of that row.
Head 1 attention weights A¹
For river: [0.401, 0.198, 0.401] · V¹ = [0.802, 0.599]
same row after each head
head 1 output[0.802, 0.599]
+
head 2 output[0.503, 0.248]
→
concatenate[0.802, 0.599, 0.503, 0.248]
The concatenated row is four numbers again. An output matrix WO can rotate and mix those four channels before the residual connection.
Separate projections create separate maps
Both heads read the same tokens, but their independently learned WQ and WK matrices produce different comparisons.
Shapes from input to output
X: 3 × 4 → each head Q,K,V: 3 × 2 → each map A: 3 × 3 → each head output 3 × 2 → concatenated output 3 × 4.
Attention maps show coefficients
“Head 1 found the subject” is too strong. A bright cell only says that this learned head assigned a larger coefficient in this layer and this example.
08 / 09
What the operation costs in a real model.
The attention formula is compact, but its intermediate tensors can be large. These constraints shape how modern systems store and schedule the computation.
n²
Pairwise scores
For a sequence of length n, QKT contains n × n scores. Doubling the context length roughly quadruples the number of pairs.
sequence length grows →KV
Autoregressive cache
During generation, the model stores key and value rows from earlier tokens so it does not recompute them at every step. The cache grows with layers, heads, tokens, and head width.
tile
FlashAttention
FlashAttention computes the same exact attention result in tiles. It keeps the running softmax statistics while moving smaller blocks through memory, so the full score matrix does not need to be materialized at once.
Modern variants preserve the same core roles
Architectures such as DeepSeek's multi-head latent attention reduce the amount of key/value state that must be cached. Kimi K3 is described as using a hybrid design that combines different attention patterns. The memory and routing strategy changes, while queries still request information, keys still provide matches, values still carry content, and masks still control visibility.
09 / 09
The mechanism in one view.
The input lookup for bank begins the same in every sentence. Attention creates a context-dependent update by scoring keys, normalizing those scores, and taking a weighted sum of value rows.
Attention is learned retrieval
For each query row, the model compares the query with every key row. The scores become a probability-like row after softmax. The output is the weighted sum of the value rows. During training, the projection matrices and the rest of the model learn which comparisons help the task.
Heatmaps show routing, not reasoning
A bright cell means that one query assigned a larger coefficient to one key in that layer and head. It does not prove that the model used a human concept, and it does not show the complete computation. Different heads and layers can produce different maps.
Shapes make matrix errors visible
If Q is nq × dk and K is nk × dk, then QKT is nq × nk. If V is nk × dv, then weights V returns nq × dv.
Multi-head attention
Every head receives the same input rows but has separate projection matrices. It produces its own attention map and filtered value matrix. Concatenation puts those smaller outputs side by side, and WO mixes them back into the model width.
These operations produce the context-dependent rows. The matrix dimensions shown throughout the page identify exactly which rows and features are being combined.
Created for this page
I created the diagrams, matrix examples, and interactive walkthrough. The collage of bank images was generated using gpt-image-2.
