How to evaluate the response quality, tool use accuracy, and safety of an AI Agent quantitatively. Covers an evaluation system you can use right away in practice, from building a Golden Set to LLM-as-Judge and regression tests.
Advanced
Updated Oct 1, 2026
~3 min read
6 sections
1 code examples
Learning Paths that include this guide
This guide is one step in the paths below. Study it together with the steps before and after.
Understanding Agent Evaluation as a real-world workflow
How to evaluate the response quality, tool use accuracy, and safety of an AI Agent quantitatively. Covers an evaluation system you can use right away in practice, from building a Golden Set to LLM-as-Judge and regression tests. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.
Key perspective
AI / LLM systems
Look beyond the model and the prompt: connect the data flow, evaluation, and post-deployment operating metrics in one view.
Building a Golden SetLLM-as-Judge evaluationAgent regression testsProduction quality monitoring
Architecture diagrams
To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Agent Evaluation.
Learning flow
Rendering diagram…
Architecture view
Rendering diagram…
Why evaluation is hard
When you first open Agent Evaluation, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.
An LLM Agent can produce different output for the same input every time, and often there is no single fixed "right answer". Because the assert equal style of unit testing does not work, without a separate evaluation system it is hard to notice quality degradation after an Agent is deployed.
Building a Golden Set
Building a Golden Set is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
First build a set of question and expected-result pairs that represent real user question patterns. Organizing it by difficulty and type makes it easy to see which cases are weak.
Difficulty
Share
Example
Easy (Happy path)
50%
Questions solved by a standard single tool call
Medium (Multi-step)
30%
Questions that need several tools combined in order
Hard (Edge case)
20%
Ambiguous questions, handling a tool failure
Quantitative metrics
Quantitative metrics is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
When grading against the Golden Set, do not look at one score only; tracking several metrics together is what tells you which part is weak.
Metric
What it measures
Task Success Rate
Whether the goal was actually achieved
Tool Call Accuracy
Whether the right tool was called with the right arguments
Faithfulness
Whether a RAG answer is really grounded in the retrieved documents
Latency (p50/p95)
The distribution of time to respond
Cost per task
The token cost of completing one task
LLM-as-Judge
Here you look at LLM-as-Judge alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
People cannot grade every response, so a separate LLM is given clear grading criteria and evaluates automatically. The more concretely the criteria are given as a checklist, the closer the agreement with human judgment.
llm_judge.pyPYTHON
JUDGE_PROMPT = """Grade the following Agent response on a scale of 0 to 5.Question: {question}Expected result: {expected}Agent response: {actual}Grading criteria:- 5: effectively identical to the expected result, with clear grounding- 3: on the right track, but some information is missing or inaccurate- 0: completely wrong, or answered with no groundingReply with the score as a number only."""def judge(question: str, expected: str, actual: str) -> int: score = call_llm(JUDGE_PROMPT.format( question=question, expected=expected, actual=actual)) return int(score.strip())
Regression tests
This section covers Regression tests from a practical perspective. Instead of memorizing the concept, focus on the situations in which you would reach for it.
Every time you swap the model or edit a prompt, re-run the whole golden set and compare the scores with the previous version. Wired into CI, it catches quality drops before deployment.
Production monitoring
This section covers Production monitoring from a practical perspective. Instead of memorizing the concept, focus on the situations in which you would reach for it.
The golden set is for pre-deployment verification; real traffic needs separate sampled evaluation. Draw some real responses at random, grade them with LLM-as-Judge, and track score trends and outliers.
Agent Evaluation: frequently asked questions
How do I evaluate an AI agent?
Look at the process as well as whether the final answer is right. Measure whether the right tools were called with the right arguments, whether there were unnecessary steps, and how much time and money it took. Build an evaluation set of representative tasks and rerun the same set whenever the prompt or the model changes.
Is it acceptable to evaluate LLM answers with an LLM?
It is useful at a scale people cannot grade by hand, but it should not be trusted blindly. Give the evaluator concrete criteria, and first check how well its verdicts agree with a sample graded by people. Items with a clear right answer are checked more accurately and cheaply with rules or code.
How large should an evaluation dataset be, and how do I build it?
A few dozen cases drawn from real usage are enough to start. Include failed cases and edge cases, not only easy ones, and keep adding problems found in production. Representing the real usage distribution matters more than size.