Don't start with a huge eval set — turn failures into regression tests
Soon after building an agent, you run into a deceptively hard question: what should I test it with? Force yourself to write ten cases and they quickly start sounding alike; plan for a hundred and you usually end up with an empty spreadsheet. An eval set is less like an answer key you finish before launch and more like an operating record that grows whenever the agent gets something wrong.
Five cases are enough to begin
Do not try to cover the world on day one. Start with two ordinary inputs you expect to see often, two boundary cases likely to confuse the model, and one input where a wrong answer would cause real trouble. For a research agent, that might mean two everyday market questions, two questions with scarce or conflicting sources, and one where presenting an old figure as current could spoil a decision.
A good first case is rarely clever. It sounds like something a teammate would actually paste into the product, and you can explain in one sentence why failure matters. If the first set is made entirely of exotic edge cases, its score may look sophisticated while saying very little about everyday quality.
Write pass conditions, not the perfect answer
LLM answers change slightly from run to run. Pinning the entire expected output to one sentence makes a correct answer fail because its wording changed, while a wrong answer with familiar phrasing may pass. It is more useful to record what every acceptable answer must contain than to write a model response for the agent to imitate.
A competitor-research case, for example, can require two company names (must_include), at least two distinct sources (min_sources), and a criterion such as ‘explain which teams each product fits, rather than merely listing features.’ Keywords and source counts can be checked mechanically; a judge can assess the quality of the explanation. Keeping those responsibilities separate makes a failed report much easier to diagnose.
Let AI propose cases, not decide what good means
AI-generated eval cases are genuinely useful for getting past the blank page. A model can scan the graph's tools and branches and suggest inputs a person missed. But generated cases tend to be tidy and reasonable. They do not naturally sound like customers who omit the subject, use an ambiguous date, or squeeze contradictory constraints into one sentence.
That is why AI-generated cases in London System Agent arrive as drafts and are never saved automatically. You rewrite the question in a real user's voice, raise conditions that are too easy, remove unsupported expectations, and save each case explicitly. The cost of generation is small compared with the cost of letting a bad pass criterion approve several releases.
Do not close the tab on a bad run
When an agent produces a disappointing answer in production, the most expensive response is to tweak the prompt and close the tab. If that input never enters the test set, the same problem can quietly return three months later when the model changes or another node is edited. Fixes fade from memory; regression cases survive the next change.
Save the failed input with clear expectations and tag the reason briefly. In London System Agent, you can leave feedback on a run and promote a template run to a golden regression case; saved agents can also have cases added directly. The button is not the important part. The habit is: bad answer → record the reason → save a case → fix → run it again.
Stub and Real answer different questions
You do not need a live model call every time. A Stub evaluation checks deterministic contracts such as required keywords and minimum source counts quickly and without model cost. Right after changing the graph or a branch, that is enough to catch a surprising number of obvious regressions.
A Real evaluation executes the agent with a live model and uses a judge for natural-language criteria. It costs money and carries some variance, so it is better reserved for release candidates and meaningful prompt changes than every structural edit. A Stub pass means the contract still holds. A Real pass means the quality bar was met for this model and these inputs. They should not be read as the same 100 percent.
Failure density matters more than case count
An eval set is not a warehouse that only grows. Merge cases that test the same fault, delete cases for behavior you no longer support, and rewrite criteria whose verdict keeps flipping. Tags such as ‘boundary,’ ‘sources,’ ‘forbidden behavior,’ and ‘real incident’ also reveal which kind of failure is still missing from the set.
Ten cases rescued from real incidents can be more useful than a hundred ordinary questions. The latter checks an average demo; the former keeps the team from paying for the same mistake twice. The point of an eval set is not to manufacture a high score. It is to stop familiar failures from coming back.
One useful thing to do today
Pick one saved agent and recall a recent input whose answer bothered you. Put that input in a case, then add required keywords, a source count, or a criterion a person can judge. Run it once in Stub and, if the case matters enough, once in Real.
Those first five cases will look different a few weeks from now. That is healthy. As the product meets reality, the eval set should change with it. Turning one failure into one test today is much more useful than waiting for a perfect evaluation plan.
Your turn to build
Compile your first agent graph from a single sentence.