Model providers ship silent updates. A prompt that worked can quietly get worse with no version bump and no error. This tracks it: a frozen, deterministically-graded suite runs against live models on a schedule, every score is kept, and a drop from one run to the next is flagged. No LLM-as-judge — the grader is exact-match, so a score change means the model moved.
Loading…
…or click any model below to toggle it. Your selection is remembered.
| Model | Latest accuracy | Δ vs previous | Status | When |
|---|
The moments behind the chart — what happened, when, and how to see it in the data. Newest first.
A fixed suite of 22 unambiguous tasks (instruction-following, factual recall, arithmetic, multi-step reasoning, counting, string manipulation, formatting, refusal calibration) runs against each model — deterministic where the model allows it. Some tasks are deliberately chosen failure modes for strong models (letter-counting, numeric vs. lexical sorting, 9.9 vs 9.11), so even a flagship has room to slip. Each answer is graded by exact match, regex, or a numeric compare — never by another model. Runs are stored in eval-history, which computes the run-to-run comparison. The suite is versioned and frozen; a chart only means something if the questions never change under it.
Open, auditable, reproducible: the suite, the graders, and the runner are all in the repo.