Transformer Architecture: meaning and definition
Transformer Architecture in plain language
Self-attention lets the model use information from other allowed positions in the input. Which positions are allowed depends on the design: a model that predicts the next word cannot look at words that come later.
The original transformer used an encoder and a decoder for translation. Later models use only an encoder, only a decoder, or both, depending on the task.
How it works
Represent tokens and position
Convert input tokens into vectors and provide a way to account for their positions.
Combine context through attention
Attention weights determine how information from available positions contributes to a representation.
Transform and produce task output
Layers also apply other transformations. The output head and training objective determine the task.
Illustrative example · Fictional data
The same word in two fictional sentences
- Input
- “The bank approved the request” and “They sat beside the river bank.”
- Process
- Read the word in the context of the sentence around it, instead of giving it one fixed meaning every time.
- Expected output
- Context-sensitive representations that can support different interpretations in downstream tasks.
- How to check it
- This shows the idea of reading a word in context. It is not a measured attention map, or proof that any model understands either sentence correctly.
Where it helps
- Language modelling and translation.
- Context-sensitive text representations.
- Architectures adapted for visual and other sequence-like inputs.
Encoder, decoder and encoder–decoder
- Encoder
- Builds input representations, often using context from both directions.
- Decoder-only
- Commonly generates sequentially with causal masking.
- Encoder–decoder
- Encodes a source and generates a corresponding output sequence.
Multi-head attention and feed-forward layers
Attention works out how words relate to each other, using query, key and value calculations. Several attention heads can each pick up different useful relationships, and their results are combined. Do not assume a particular head always stands for one human idea, such as grammar.
A transformer block also includes a feed-forward network applied at each position, shortcut connections and normalisation. Stacking blocks transforms the representation again and again. Position information and the attention mask decide which relationships the model can use. Attention alone is not the whole design.
Training, generation and image patches
When training a model that predicts the next word, many positions can be learned from at once, while masking stops it seeing future words. When it generates text, each new word depends on the words before it, so the answer comes out one word at a time. Training in parallel does not mean the model writes a whole answer at once.
A Vision Transformer can treat an image as a sequence of small patches instead of words. Position information helps keep track of where each patch sits. This shows attention can work on more than text; it does not reveal the unpublished design of every commercial model that handles images.
Related reading and learning pathways
Common pitfalls
- An attention weight is not automatically a faithful explanation of a prediction.
- Longer context capacity does not guarantee every detail is used correctly.
- Different transformer variants have different computational and input constraints.
Put the idea to work
Try it yourself
Create two sentences using the same ambiguous word. Identify which surrounding words resolve its meaning, then explain why this is a language exercise rather than a measured model explanation.
Common questions
Is a transformer the same as an LLM?
No. The transformer is a design that many LLMs use, but transformer models do other jobs too. BERT, for example, is a transformer used to understand and sort text rather than to write freely. Vision transformers use the same design for images.
Does a transformer always see future tokens?
No. Attention masks can restrict access. Causal generation uses prior context rather than future tokens. Encoder models such as BERT can attend to the whole input at once, while decoder models such as GPT generate one token at a time and only attend to earlier tokens.
Your next step
Continue learning
Connected concepts
Sources and further reading
Examples on this page are teaching exercises using fictional information. They are not client case studies or measured outcomes.