Tokens, context windows and next-token prediction
Comes up in: AI engineering, Recruiter screen
This concept is being written. It gets the full five steps — explanation, predict, interview angle, practice with tests, and review cards. Here is exactly what it will cover, so you can study it elsewhere in the meantime.
What it covers
- tokenisation (subwords
- non-English text often costs more tokens)
- context window
- next-token prediction
- why models hallucinate
- knowledge cutoff
- counting tokens with the provider's API
Words you will know
What you will practise
- compare token counts for English, Hindi and code
Interview questions this answers
- What is a token? Why does token count matter?
- Why do LLMs hallucinate? How do you reduce it in a product?