Model Bench
Empirical cross-provider LLM benchmark — 22,522 trials × 64 models × 9 tasks, with subscription-vs-public-API cost columns side-by-side and industry-benchmark side tracks.
Overview
A benchmarking site I built to answer one question well: which subscription-based LLM is worth its slot on my dashboard, and which pay-as-you-go API is the better substitute. Every public benchmark either (a) scores only pay-as-you-go APIs (so the results don’t tell me what a Claude Max / ChatGPT Plus / ollama-routed model would actually do on my workloads) or (b) makes vague quality claims with no cost column (so I can’t compare a $20/month subscription to a $5 in API spend). This benchmark measures both, on the same tasks, in the same week, scored by a judge from a different model family.
What’s measured
Every (model × task) cell runs N times and is graded on four axes:
- Quality — judge score 0–5 from a model outside the cell’s
provider family (so Claude doesn’t grade Claude). 9 task types:
code_debug,code_gen_long,json_strict,summarize,creative_write,reasoning_multistep,agentic_prompt,refactor_existing_code,agentic_tool_use. - Cost — two columns:
WIRE-USD— what the call actually cost at the provider API ($0.0000 for flat-rate subscriptions like Anthropic Max, ChatGPT Plus, ollama-routed)1P-EST— what the same token traffic would cost at public-API rates, computed fromfirst_party_pricing.json. This is the apples-to-apples column.
- Latency — wall-clock end-to-end including any reasoning trace tokens.
- Pass rate — share of trials where the judge returned
passes_key_criteria: true. - Cross-judge sanity — a second judge (Fable, claude-fable-5)
re-scored 203 trials across 61 high-variance cells (plus 1 sample
per model) to detect judge-specific drift. Fable is on average
0.51 points more generous than the primary judge; the biggest
divergence is
creative_write(+1.93 mean). - Industry side tracks — GPQA Diamond, IFEval, MMLU-Pro, and HumanEval+ waves run alongside the custom 9-task harness so subscription models can be compared to public leaderboard norms.
Current snapshot (live aggregate, 2026-07-26)
22,522 trials × 64 models × 9 tasks from
data/full_scorecard.json on herman/model-benchmark-site. Rank
table below is the top of leaderboard_mean_score. Canonical live board:
models.hermanity.dev.
| # | Model | Provider | Mean Score | Pass Rate | Trials |
|---|---|---|---|---|---|
| 1 | claude-fable | claude-cli | 4.44 | 88% | 660 |
| 2 | claude-opus-5-high | claude-cli | 4.44 | 87% | 190 |
| 3 | claude-opus-5 | claude-cli | 4.39 | 89% | 90 |
| 4 | ollama-cl/gemma4 | skynet | 4.38 | 85% | 52 |
| 5 | grok-4.5-direct | xai-direct | 4.33 | 81% | 140 |
| 6 | mistral/magistral-small | skynet | 4.32 | 82% | 34 |
| 7 | ollama-cl/deepseek-v4-flash | skynet | 4.29 | 82% | 34 |
| 8 | ollama-cl/qwen3-coder-next | skynet | 4.19 | 79% | 52 |
Top-of-board quality is still tight (about 0.25 points across the dense
mid-pack once low-n rows are set aside), but the cost spread remains
enormous across subscription / public-API / local-ollama routing.
claude-fable and claude-opus-5-high currently share the mean-score
lead with real sample depth; lower-n Skynet/ollama rows just under them
are early signal, not settled ranking. Recent scale waves still in the
board: Claude Opus 5 launch route, Kimi Coding API provider + HumanEval+
side track, plus earlier underutilized-cloud / Grok HAM waves.
Why judge-from-different-family matters
When the judge is in the same provider family as the trial model, the results drift toward “every model ties at the top.” Claude judging Claude gives 4.5 ± 0.2 even on objectively worse outputs. Cross-family judging widens the spread enough that the ranking is actually useful for dashboard slot decisions.
What’s in the repo
Two coupled repos:
herman/model-benchmark-site— this Hugo siteherman/model-benchmark— the harness + scoring script + raw trial output.python3 harness.py --models <subset> --tasks <subset> --reps 2 --parallel 4runs the trials;python3 scorecard.pyaggregates;python3 sync_to_site.pycopies data into this Hugo site asdata/*.json.
The trial prompts and judge rubrics are checked into
tasks.json / judge_rubric.json so every score is reproducible.
Live
- Site: https://models.hermanity.dev/
- Methods:
/methodology/on the site — explains the judge-from-different-family protocol, the 1P-EST pricing column, and the per-task leaderboard.
Status
Active — refreshed whenever I add a model to my active set or
notice a quality regression in a workflow I rely on. Current
methodology is v4 (cross-judge Fable sanity-check on 542 high-variance
cells + 2 new task types refactor_existing_code and
agentic_tool_use); v3 added the cross-family judging + 1P-EST
cost columns; v2 added the agentic_prompt task class in 2026-06.