Transformers and attention (intuition)
Comes up in: AI engineering
This concept is being written. It gets the full five steps — explanation, predict, interview angle, practice with tests, and review cards. Here is exactly what it will cover, so you can study it elsewhere in the meantime.
What it covers
- embeddings
- self-attention in plain words
- why attention cost grows with context
- positional information
- decoder-only models
- cite Vaswani et al. 2017
Words you will know
What you will practise
- interactive attention visualiser trace
Interview questions this answers
- Explain attention in simple terms.
- What are reasoning models and when are they worth the extra cost?