How a Transformer uses attention and context, how its parts fit together, and why it matters for language and image models.
What is a Transformer?
A Transformer is a neural-network architecture that uses attention to combine information from different parts of its input.
An architecture is the arrangement of the model's parts and the connections between them. Training adjusts the numbers within that arrangement.
Think of a recipe and a prepared dish. The recipe describes how to put things together; the dish is a particular result. Similarly, two models can share a Transformer architecture yet behave differently because they were trained differently.
Where does it fit?
Machine learning is the method of learning from data. A model is what we train and use. Transformer describes one way to organise that model.
The large language model article describes models that work with language. Here we look at an architecture used to build many of them.
Transformer and LLM describe different things: the construction and the kind of model. A Transformer can also work with images. The Vision Transformer, for example, represents an image as a sequence of patches.
Why does context matter?
Consider these two sentences:
“The bank approved my loan.”
“We sat on the bank beside the river.”
The word “bank” is the same, but its meaning changes. “Loan” points towards a financial institution; “river” points towards the edge of a waterway.
A useful representation of a word must therefore depend on more than the word alone. It needs information from the surrounding text.
This picture would illustrate the problem the model must solve. It would not prove that a particular model uses those exact links.
What is attention?
Attention is a calculation that gives different weights to different pieces of information, then mixes that information using the weights.
For a rough analogy, imagine asking a group which bus goes to the railway station. You would give more weight to the person familiar with that route than to someone discussing a different journey.
The model does not consciously ask or listen. It compares numerical representations. In self-attention, the information being compared comes from positions within the same sequence.
The standard calculation uses three learnt representations:
- Query: what is being matched at this position.
- Key: what each available position offers for matching.
- Value: the information each position contributes to the mixture.
Query–key comparisons produce scores. The model converts these into weights and uses them to combine the values. Several attention heads perform different learnt comparisons; their results are combined.
A head is not necessarily a neatly labelled specialist in grammar or facts. These roles are learnt numerical patterns, rather than jobs assigned by a programmer.
How does text pass through it?
For a language Transformer, the broad sequence is:
- Split the text into tokens. These may be words, parts of words or punctuation.
- Represent the tokens with numbers. Each token becomes a vector: a list of numerical values.
- Include position information. Word order matters. “The dog chased the boy” and “The boy chased the dog” contain the same words but describe different events.
- Mix and process information. Attention combines information across permitted positions. A feed-forward network processes each position's representation.
- Repeat through layers. Later layers work with representations produced by earlier ones.
- Produce the task's output. In a next-token language model, an output layer produces probabilities for possible next tokens.
Residual connections carry information around the sublayers, while normalisation helps keep training stable. Attention is a central component, but it is not the whole network.
Why was it an important change?
Vaswani and colleagues introduced the Transformer in 2017 in Attention Is All You Need, demonstrating it on translation tasks.
Recurrent networks pass a changing internal state along a sequence. Transformers allow much more of the work across sequence positions to happen in parallel during training. Attention also gives distant positions a direct way to exchange information.
This helped make larger-scale training practical. It did not mean that generating a reply became a single simultaneous operation.
In ordinary autoregressive generation, the model produces a next token, adds it to the available text and repeats. Parallel training and token-by-token generation can both be true.
Are all Transformers alike?
Three common arrangements help explain the differences:
- Encoder-only: builds representations using context on both sides within the supplied input. BERT is an example; it can be adapted for tasks such as classification and extracting answers.
- Decoder-only: uses a causal mask so a position cannot access later tokens. GPT-style language models use this arrangement to continue text.
- Encoder–decoder: an encoder processes the input, and a decoder generates output while attending to the encoder's representations. This was the original Transformer's arrangement.
A causal mask prevents a next-token model from looking ahead at the answer during training. In an encoder–decoder model, attention to the separate encoder output is called cross-attention.
What are its limits?
Full self-attention compares pairs of positions. Doubling the sequence length gives roughly four times as many pairs, so long inputs can be expensive. This describes the attention calculation, not a promise that the whole system's cost always rises by exactly four times.
The architecture also does not decide whether a statement is true. A model can use context effectively and still give a wrong answer. Training data, the learning objective and the surrounding software all matter.
Nor does the word “attention” establish human awareness. Here it names a mathematical operation.
What follows?
A Transformer gives a model a way to represent each part of an input in relation to other parts.
That is the useful idea to remember: the representation changes with the context. The architecture makes these relationships computable; training determines which patterns the model learns.
References
Vaswani and colleagues — Attention Is All You Need (2017): the original architecture, attention calculation and training comparison.
Google Machine Learning Crash Course — Transformers: an introduction to language models and Transformer components.
Devlin and colleagues — BERT (2018): an encoder model using context on both sides.
Dosovitskiy and colleagues — An Image is Worth 16x16 Words (2020): applying Transformers to image patches.
Ai disclosure: written with the help of AI (ChatGPT). You are encouraged to point out errors and omissions.
