How inference serving works
Comes up in: AI engineering, AI system design
This concept is being written. It gets the full five steps — explanation, predict, interview angle, practice with tests, and review cards. Here is exactly what it will cover, so you can study it elsewhere in the meantime.
What it covers
- prefill vs decode
- the KV cache and its memory growth with context length
- TTFT vs throughput
- continuous batching
- prefix (prompt) caching on the server
- speculative decoding
- back-of-the-envelope latency and cost maths
Words you will know
What you will practise
- estimate latency and cost for 3 workloads
Interview questions this answers
- What is the KV cache and why does long context cost more?