← Sugeerth Murugesan Portfolio

Transformer Attention Visualization

Scaled dot-product self-attention, computed live on a worked example — the Query–Key–Value mechanism that powers transformers.

Attention(Q,K,V) = softmax( Q·K⊤ / √dk ) · V

Every token is linearly projected into three vectors. The query asks “what am I looking for?”, each key advertises “what do I offer?”, and the value carries the information actually mixed into the output. A query scores itself against every key, those scores are normalized with softmax into attention weights, and the output is the weighted sum of the values. Everything below is the real arithmetic — no mock data.

Query  Q = X·WQ

The token currently being updated. It probes the sequence for relevant context.

Key  K = X·WK

One per token. The dot product Q·K measures how relevant each token is to the query.

Value  V = X·WV

The payload. Attention weights blend the values into the new representation.

Interactive self-attention

Sentence: “the cat sat on the mat” — tokenized into 6 tokens. Pick a query token, switch attention heads, and drag the temperature slider to see softmax sharpen or flatten.

dk = 4  ·  scaling ÷ √4 = 2

1 · Attention-weight heatmap

Rows = query tokens, columns = key tokens. Each row sums to 1. The selected query row is outlined.

low
high attention weight

2 · Attention as weighted connections

bertviz-style arcs from the query token. Arc width & opacity encode the attention weight.

3 · The softmax step for query

Raw scores  Q·K⊤ / √dk
→
exp(score / T)
→
Attention weights (softmax, Σ=1)

How the numbers are produced

Honest, reproducible pipeline — this is exactly what the code does on each render:

  1. Each of the 6 tokens has a fixed 8-dimensional embedding vector X (illustrative but deterministic; the duplicate “the” gets a small positional offset so positions 0 and 4 differ).
  2. Each head has its own learned-looking projection matrices WQ, WK, WV (8×4). We compute Q = X·WQ, K = X·WK, V = X·WV.
  3. Scores Sij = (Qi · Kj) / √dk, with dk = 4, so we divide by 2.
  4. Attention weights Ai = softmax(Si / T) per query row — T is the temperature slider.
  5. Output Oi = Σj Aij Vj — the new representation of token i, shown under view 2.

Head 1 is tuned toward local/adjacent attention, Head 2 toward content similarity (the two “the” tokens attend to each other), and Head 3 is more diffuse — mirroring the specialized heads observed in real transformers.