Engrain by ByteRush

AGT.09 4 concepts

Evals and LLMOps

Measure and operate LLM systems.

  1. AGT.09.01Evals: why and howCoreMid-levelBeing written
  2. AGT.09.02LLM-as-judge done rightCoreSeniorBeing written
  3. AGT.09.03Observability, tracing and monitoringMid-levelBeing written
  4. AGT.09.04Agent evalsCoreSeniorBeing written

Interview questions in this module

  1. How do you evaluate an LLM feature before launch?
  2. What are the pitfalls of LLM-as-judge and how do you calibrate it?
  3. How do you monitor an LLM system in production?
  4. How do you evaluate an agent, not just a single response?
  5. A model upgrade made some answers worse. How would you have caught it?