Evaluation-Driven Development
A methodology for building AI features where each feature is treated as a testable hypothesis and the pull request is gated on an offline evaluation pipeline -- gold-standard and synthetic datasets scored by code-metric and LLM-as-judge evaluators -- instead of on traditional unit tests.
Context
Evaluation-Driven Development (EDD) was the deep dive on Episode 31 of the ADI Pod, drawn from a Decoding AI piece by Paul Easton and Alejandro Aboy. The premise: a non-deterministic AI feature can’t be gated by a unit test the way deterministic code can, because the same input can produce a different output run to run. EDD replaces the unit-test gate with an offline eval pipeline — in the article’s case built on Opik — and treats every feature as a hypothesis you measure rather than a spec you assert.
The moving parts the episode pulled out:
- Gold-standard vs. synthetic datasets — hand-curated examples with known-good answers, versus generated ones that cover more ground cheaply. You want both, and you want to know which is which.
- Code-metric vs. LLM-as-judge evaluators — deterministic scorers (exact match, regex, latency) where the answer is checkable, and a model grading the output where it isn’t.
- An “aggression” dial — a tunable setting for how harsh the LLM-as-judge reviewer is, i.e. how big a jerk your automated reviewer gets to be before it starts failing borderline cases.
Why It Matters
Shimin’s framing on the show was that moving from unit tests to evals is like moving from Newtonian physics to quantum mechanics: the old rules don’t vanish, they stop being the right level of description once the system is probabilistic by design. EDD is the non-deterministic era’s analog to test-driven development — you build the measurement in before the feature, and you let the eval, not a green checkmark on a unit test, decide whether the PR ships.
It also fits the show’s recurring throughline that the leverage in AI engineering keeps moving up the stack: the hard part is no longer writing the feature but specifying how you’ll know it works, which is exactly the judgment a human still has to supply.
Related Concepts
- Verification debt — what accumulates when AI output ships without the eval gate EDD insists on
- Benchmaxxed — the failure mode of optimizing to the eval instead of the goal, the risk any eval-first practice has to guard against
- Loop engineering — the harness around the agent; EDD is the quality gate you wire into that loop