Which defect classes does your eval suite actually catch?

Every fleet member is a deterministic model broken in exactly one documented way at a stated, seeded rate — an answer key. Each suite archetype below is executed by its real engine against every member. A low detection rate is not an opinion about the suite; it is a measured hole, reproducible from the repo with one command.

Reading the board: green ≥ 0.9 · amber 0.5–0.9 · red < 0.5 — this defect class passes the suite invisibly. A defect the suite cannot even see in principle certifies nothing about the suite — every member's defective response provably differs from its clean twin (see the fleet's certification tests).

What is audited here is a suite archetype, honestly named — the checks an engineer would write for this task format, run by the real tool named in the section header. No named public benchmark's own dataset or checker is being scored.

Disclosure that matters: the diligent archetypes were written by the fleet's author with full knowledge of the task's ground-truth markers. Their 1.000 rows demonstrate that the ceiling is reachable — every defect here is catchable by a suite that checks everything the task makes checkable. They do not demonstrate that typical suites reach it; the naive row is the archetype most suites start from, and it is blind to five of six classes. Reproduce: python audit/run_audit.py.