Evals: why and how
Comes up in: AI engineering, AI system design, Take-home
This concept is being written. It gets the full five steps — explanation, predict, interview angle, practice with tests, and review cards. Here is exactly what it will cover, so you can study it elsewhere in the meantime.
What it covers
- evals as tests for probabilistic systems
- eval types (code-graded, model-graded, human)
- building datasets from real traffic and synthetic data
- error analysis first
- precision/recall/F1 and confusion matrices for classifiers
- sample size, confidence intervals and run-to-run variance
- regression evals in CI
- online A/B tests
Words you will know
What you will practise
- build an eval suite for the site's grader (shared with ARCHITECTURE §9)
Interview questions this answers
- A model upgrade made some answers worse. How would you have caught it?