Start with the task, not the model
An eval only helps if it measures something the product actually needs. Start by defining the real task: summarization, answer generation, routing, extraction, refusal behavior, or something else. Then ask what a good outcome looks like for that task.
Build a representative prompt set
Use examples that reflect real user intent, not just easy happy paths. Include common requests, important edge cases, and prompts that should trigger uncertainty, clarification, or refusal.
Choose a rubric that matches the job
Some tasks need exactness. Others need helpfulness within a quality range. Your rubric might include groundedness, factual accuracy, completeness, policy compliance, citation quality, or fallback behavior depending on the feature.
Set thresholds before you compare versions
If you wait until after seeing the results to decide what counts as “good enough,” the eval becomes a debate tool instead of a quality tool. Define acceptable ranges and red lines first.
Review failures like test failures
An eval should not stop at a score. Review where the system failed, look for clusters, and turn recurring miss patterns into better prompts, better retrieval, tighter guardrails, or clearer fallback behavior.
Use evals for regression, not just launch
The real value of evals is comparison over time. Run them when prompts change, models change, retrieval changes, or guardrail logic changes so quality regressions surface before users find them.