Streaming, timeouts, retries and rate limits
Comes up in: AI engineering, AI system design
This concept is being written. It gets the full five steps — explanation, predict, interview angle, practice with tests, and review cards. Here is exactly what it will cover, so you can study it elsewhere in the meantime.
What it covers
- streaming for long outputs
- timeouts
- retrying 429/5xx/overloaded with backoff and jitter (SDK defaults)
- rate limit headers
- idempotency of side-effecting flows
- refusal/stop reasons to check before using content
- fallbacks between models
Words you will know
What you will practise
- resilient client wrapper with tests using a fake server