Evals are the unit tests of AI
The EDD harness is the eval-harness skill that ships inside ECC (Everything Claude Code). It brings Eval-Driven Development to AI coding agents: you write the pass/fail criteria BEFORE the agent writes a line, then run those evals continuously. Instead of eyeballing an agent's output and hoping, you measure it. An eval is just a check with a known answer, and the harness gives you a shared shape for capability checks, regression checks, graders, and reliability scores.
- Define expected behavior BEFORE implementation.
- Run evals continuously, not once at the end.
- Track regressions on every change.
- Measure reliability with pass@k, not vibes.
Because AI output is non-deterministic
An agent that passes once can fail the next run on the same prompt. Manual review does not scale and it does not catch quiet regressions. EDD turns 'looks right' into a number you can gate on. Define the evals once, and every model swap, prompt edit, or skill change gets scored against the same bar. That is what lets you raise how much you delegate without losing the thread: the eval gate, not your eyeballs, decides what ships.
- Catches regressions a human reviewer would miss.
- Makes model and prompt changes comparable, run to run.
- Replaces 'I think it works' with pass@3 = 0.92.
- Deterministic graders beat asking the model 'did you get it right?'.
Define, implement, evaluate, report
The workflow is a tight loop. First you DEFINE the evals (capability and regression) before coding. Then you IMPLEMENT to pass them. Then you EVALUATE by running every grader and recording PASS/FAIL. Then you REPORT: a small summary with the pass@k metrics and a ship/no-ship status. Evals are first-class artifacts, versioned with the code in .claude/evals/.
## EVAL DEFINITION: add-auth
### Capability Evals
1. handleAuth is exported from src/auth.ts
2. validateEmail is exported from src/auth.ts
### Regression Evals
1. existing health() export still present
### Success Metrics
- pass@3 >= 0.90 for capability evals
- pass^3 = 1.00 for regression evalsCapability evals vs regression evals
Capability evals ask: can the agent do something it could not do before? They carry a task, success criteria, and an expected output. Regression evals ask the opposite: did a change break something that already worked? They pin a baseline (a SHA or checkpoint) and re-run the existing tests. You want capability evals to clear pass@3 >= 0.90, and regression evals to clear pass^3 = 1.00 on release-critical paths.
- Capability: new ability, task + criteria + expected output.
- Regression: protect what works, pinned to a baseline.
- Capability target: pass@3 >= 0.90.
- Regression target: pass^3 = 1.00 for critical paths.
Code, rule, LLM, and human graders
A grader is the thing that turns an output into PASS or FAIL. The harness defines four. Code-based graders are deterministic checks (a grep, a passing test, a clean build) and they are preferred whenever possible. Rule graders apply regex or schema constraints. LLM graders use a model as judge against a rubric, scored 1 to 5 with reasoning, for open-ended output. Human graders flag security or ambiguous changes for manual review. The rule of thumb: deterministic beats probabilistic, and security never gets fully automated.
- Code grader: grep, npm test, npm run build. Deterministic.
- Rule grader: regex / schema constraints.
- LLM grader: model-as-judge, rubric scored 1 to 5.
- Human grader: manual review for security and ambiguity.
# PASS if the function is exported, FAIL otherwise
grep -q "export function handleAuth" src/auth.ts && echo PASS || echo FAIL
# PASS if the build is clean
npm run build && echo PASS || echo FAILpass@k and pass^k
Reliability is the whole point, so the harness measures it directly. pass@k means 'at least one success in k attempts': pass@1 is first-try success rate, pass@3 is success within three tries. pass^k is the stricter bar, 'all k trials succeed', used for critical paths. You chase pass@3 >= 0.90 for capability work and pass^3 = 1.00 for release-critical regressions. The anti-patterns are just as important: do not overfit prompts to known examples, do not measure only the happy path, and do not let flaky graders into a release gate.
- pass@1: first-attempt success rate.
- pass@3: at least one success in three tries (target >= 0.90).
- pass^3: all three trials pass (target 1.00 on critical paths).
- Watch cost and latency drift while chasing pass rates.
Install and run your first eval
The eval-harness is a skill: 'installing' it means making it available to your agent and following its pattern. Pull ECC, copy the skill into your skills directory, and confirm the frontmatter. Then run the EDD loop: write the definition, implement, and run a code-based grader. No service to stand up, no daemon. The harness is a discipline plus a shared artifact layout.
git clone https://github.com/affaan-m/ECC
mkdir -p .claude/skills
cp -r ECC/skills/eval-harness .claude/skills/
head -5 .claude/skills/eval-harness/SKILL.md # name: eval-harnessecho 'export function handleAuth(){ return true; }' > src/auth.ts
grep -q 'export function handleAuth' src/auth.ts && echo PASS || echo FAILWhat happened when we ran it
These are real numbers from running the harness on the Hetzner build box on 2026-06-09, not a copied example. We copied the skill, confirmed its frontmatter, then ran the full EDD loop end to end with deterministic code-based graders, plus a negative control to prove the grader actually discriminates. We also validated the four evaluator-rag example artifacts that ship in ECC.
- Frontmatter asserts: name PASS, EDD present PASS, pass@k present PASS.
- EDD loop graders: 4/4 PASS (handleAuth, validateEmail, syntax, health regression).
- pass@1 = 1.00, pass@3 = 1.00, pass^3 = 1.00 over 3 trials.
- Negative control correctly FAILS on a missing impl (grader discriminates).
- ECC evaluator-rag artifacts parse: report, verifier, scenario, trace = 4/4 valid JSON.
- Verifier example: accepted candidate score 0.94, rejected candidate score 0.21.
grep -q 'export function handleAuth' src/auth.ts && echo PASS # PASS
grep -q 'export function validateEmail' src/auth.ts && echo PASS # PASS
node --check src/auth.ts && echo PASS # PASS
grep -q 'export function health' src/auth.ts && echo PASS # PASS (regression)
# -> 4/4 passed | pass^3 = 1.00The validator layer for a multi-agent fleet
In a multi-agent fleet, EDD is the 'close the loop' doctrine made concrete. It is the validator layer that sits between a builder agent and shipping: the eval gate, not a human, decides what passes. That is what lets you raise autonomy without rubber-stamping delegated work. It maps straight onto fleet KPIs (Streak up, Attempts down) and onto a learning loop like Petey's proceval / prociprover, where low-scoring runs get a patch and re-scored. Deterministic graders are the backstop that makes a /goal or overnight loop trustworthy.
- The gate between 'agent built it' and 'we shipped it'.
- Raises Size and lowers Presence without rubber-stamping.
- Maps to fleet KPIs: Streak up, Attempts down.
- Feeds a learning loop: low scores -> patch -> re-score.