One query reading four keys — edge weight IS the softmax weight, so the figure shows where the token looks before the steps say why. Evidence: heads sharpen as recall climbs over training; the ladder is context windows across four eras on a log scale.
Who knows what 'it' means?
Imagine reading a story out loud in class. When you hit the word 'it', you stop and ask: who knows what 'it' is? Every word in the sentence raises a hand — but 'cat' waves really hard and 'the' barely lifts a finger. You listen mostly to the hands waving hardest, and that's how you know 'it' is the cat.
Attention is just deciding whose hand to listen to.
out = softmax(QKᵀ / √d) · V
RNN
1997
~100 TOKENS
Transformer
2017
512 TOKENS
GPT-2 era
2019
1K TOKENS
Long context
2024
10⁶ TOKENS
Token bars are log-scale — each step is 10×