Art Bench

What this project is

Art Bench is an open-source, evidence-first image-generation benchmark and a laboratory for the Herman clockwork-owl identity. Every public image is paired with its exact prompt, the declared provider and model, the access method, the visual review, an artifact SHA-256, the byte count, and the MIME type. Unknown provider settings stay null; they are never invented. Artifactless failures are recorded with the same provenance discipline as successes. Unknown provider values are never silently filled in from a vendor brochure.

Live target: artbench.hermanity.dev. Source: herman/artbench. This page is the logbook summary; the methodology, schema, and round results are checked into the repo under docs/ and data/.

What this project is not

Art Bench is not the SwarmUI calibration project. That distinction matters because the two projects live in adjacent territory and ask different questions:

|| Question | Art Bench | SwarmUI calibration | |—|—|—| | Goal | Compare what different providers and models did, on the same frozen prompt | Find the sweet spot within one local SwarmUI rig for steps / CFG / sampler | | Unit of measurement | A (provider, model, prompt) cell with declared provenance | A (checkpoint, steps, CFG, sampler) cell on a 3090 Ti | | Output verdict | “GPT Image 2 Medium scored 9.5 on identity prompt” — descriptive, sample-bounded | “Flux: 36 steps is perfect; cap at 40” — calibrated default for a single rig | | Sample size rule | n=1 launch signal in Round 1; n≥2 only where budgeted and pre-registered | n≥3 prompts per cell before changing any default (verified, the rule that caught a phantom winner) | | Verification | Frozen prompt SHA-256, artifact SHA-256, deterministic Pillow byte checks where applicable | Side-by-side quality scoring on a 0–5 scale across multiple prompts | | Generator / grader separation | Generator never grades itself; visual reviewer family disclosed or human-assisted | Operator judges against rubric |

The two projects share a substrate — image generation — and they share one image generator (SwarmUI on the LAN 3090 Ti, used by Art Bench Round 2 for local OSS checkpoints). They do not share a methodology, and conflating them has already produced one bad read on this site: confusing “Flux: 36 steps is perfect” (a calibration claim within one rig) with “Flux.1 Dev FP8 is a 8.0 on identity prompts” (a launch-signal cross-provider observation). The first is a tuning recipe. The second is a measured specimen with a SHA-256 and a deviation note.

What is on the site today

• 19 controlled experiment tracks (16 measured or seeded; 3 active Round 3 pilot experiments). Each carries a frozen prompt template, a question, a rubric, and a run count.
• 16 provider/model policy records in data/models/ — every model declares its provider, automation policy (official_api_permitted, first_party_tool, local_inference, or manual_only), capability surface, and documentation URL.
• 52 public run records: 50 artifacts plus 2 truthful artifactless failures.
• First frozen hosted matrix: 3 prompts × 5 hosted model endpoints or quality tiers (Round 1).
• Local OSS matrix: 3 prompts × 7 checkpoints through the operator-authorized SwarmUI LAN endpoint at 1344×768 (Round 2).
• Round 3 pilot (partial provider coverage, screening evidence only, ranking prohibited at N=2): 3 experiments × 4 endpoints × 2 unseeded reps = 24-call ceiling, of which 14 were consumed (12 artifacts, 2 artifactless failures), 10 left unspent by the stop gates.
• 64 focused Python tests plus Ruff lint, deterministic catalog drift check, and a Hugo production build in the CI gate.
• 64 focused Python tests are the project-wide coverage floor; per-round evidence files reconcile artifact SHAs and failure_record_ids to canonical run records through scripts/validate_ledger.py.
• Budget ledger per Round 3 cell, validated by scripts/validate_ledger.py; one JSONL per cell with budget_total, used, and remaining = budget_total - used.

Frozen prompts and provenance

Each prompt is checked into data/experiments/<id>.json and recomputable to a SHA-256 of the exact UTF-8 bytes of prompt_template.text. Changing a single byte changes the hash and invalidates the cell. Round 3 published three such hashes in ROUND3_PLAN.md so reviewers can verify them at any time:

|| Experiment | prompt_sha256 | | — | — | | r3-ui-dashboard-fidelity | 2feda51e5dfa516e6d6aad2c23c0d8c60ad383ac4c9a795d4c2c2120afe51dc6 | | r3-transparent-icon-sheet | 29e7f6cee64a78399cb4e03c74ad6d8387ead9b71d2cc43a8f436342ac8a1e61 | | r3-network-diagram-consistency | 4614fc85145c49c2ca328daffd01f1ab5832e13b6448a99f19efc00ac494b1b9 |

One full Run 1 identity prompt (from data/experiments/ab-owl-identity-001.json, verbatim):

“A dignified clockwork owl named Herman perched on a large aged-brass skeleton key, with midnight navy enamel, exposed gears, half-moon spectacles, one subtle amber eye, and art-deco geometry.”

The spatial-counting prompt asked for an exact inventory (3 red cubes on the left, 2 glass spheres in the center, 1 brass pyramid behind, 1 cream paper crane on the far right). The typography prompt asked for two literal lines — MEASURE THE MAGIC / NOT THE MARKETING — with no other words. These are not paraphrases; they are the exact bytes that produced the artifacts below.

Model and provider provenance

Every run record names its provider and model identifier verbatim, plus an access_notes field and an automation_policy enum. Selected samples from data/models/:

The Mage record is intentional. A provider marked manual_only cannot generate through the provider contract. There is no CAPTCHA bypass, no App Check bypass, no undocumented browser automation, and no credential harvesting anywhere in this project. That is the policy in plain text because I have been asked about it and I will keep being asked about it.

Round 1 score matrix — frozen hosted matrix (n=1 launch signal)

Three frozen prompts × five hosted endpoints or quality tiers. One unseeded generation per cell. Scores are 0–10 prompt-adherence visual reviews of these exact artifacts, not global model rankings.

Caveat that is also the project’s honesty gate: one generation per cell is not enough to estimate reliability. Seeds, steps, samplers, guidance, and schedulers were not exposed by these hosted paths and remain null. GPT Image 2 Low / Medium / High are quality tiers of one API model family, retained as distinct endpoints because the provider sends different quality settings.

Round 2 score matrix — local OSS through SwarmUI

Endpoint: authorized LAN SwarmUI at the operator’s 3090 Ti, build v0.9.8.0.GIT-4ffaabe8. Three frozen prompts × 7 common OSS checkpoints = 21 reviewed artifacts. Each checkpoint ran sequentially, 1344×768, with its curated local defaults. No cell reached strict accepted; all 21 are public as accepted_with_documented_deviations — that is deliberate failure retention.

|| Checkpoint | Steps | CFG | Sampler / scheduler | Identity | Spatial | Typography | |—|—:|—:|—|—:|—:|—:| | FLUX.1 Dev FP8 | 36 | 1.0 | Euler / simple | 8.0 | 4.0 | 8.0 | | FLUX.1 Schnell FP8 | 4 | 1.0 | Euler / simple | 7.8 | 2.5 | 8.2 | | Chroma1 Base | 24 | 3.0 | DPM++ 2M / beta | 3.5 | 6.0 | 4.0 | | SDXL 1.0 Base | 36 | 7.0 | DPM++ 2M / Karras | 4.0 | 1.0 | 0.5 | | JuggernautXL v9 | 36 | 7.0 | DPM++ 2M / Karras | 5.0 | 0.5 | 0.5 | | RealVisXL v4.0 | 36 | 7.0 | DPM++ 2M / Karras | 5.5 | 0.5 | 0.0 | | DreamShaperXL v2.1 | 36 | 7.0 | DPM++ 2M / Karras | 5.0 | 2.0 | 0.0 |

Visible defects, named explicitly:

• None of the seven local checkpoints produced the exact seven-object scene on the spatial prompt. Chroma1 Base came closest (3 cubes + 2 spheres correct) but duplicated both the pyramid and paper crane.
• FLUX.1 Dev inflated the spatial scene to 4 cubes + 4 spheres while keeping every requested category.
• FLUX.1 typography was almost right but never exact. Dev split as MEASURE THE / MAGIC / NOT THE MARKETING; Schnell split as MEASURE / THE MAGIC / NOT THE MARKETING. Both spelled every required word correctly and added no other copy. The two-line constraint failed on line breaks.
• The SDXL family produced decorative pseudo-text rather than the required phrase, and substituted unrelated geometric still lifes on the spatial prompt.
• None of the seven checkpoints rendered unmistakably half-moon spectacles. Spectacles were round or absent. This is the one identity constraint where no local checkpoint beat 5.5/10.

Compatibility finding, audited:

flux2-dev-fp8mixedfromscaled.safetensors appeared in the live SwarmUI checkpoint inventory but could not run on the active backend. Three frozen cells returned no artifact; a reduced 1024×576 four-step diagnostic returned No backends available!. SwarmUI recovered to backend='running' after the failed load. The public 7×3 matrix therefore uses FLUX.1 Schnell in place of FLUX.2 while retaining the failure as an audited compatibility note. This is not a missing row; it is a documented evidence gap.

Round 3 deterministic metrics — what got measured in bytes

Round 3 is a partial-provider, screening-evidence pilot, not a ranking round. Two unseeded samples per available cell are too few for a model rank, confidence interval, or quality-tier claim. What Round 3 did ship is a typed trial/measurement schema and a predeclared budget ledger, plus three pre-registered deterministic metric thresholds that were frozen before execution.

The only deterministic (byte-level) measurements Round 3 actually ran were on the r3-transparent-icon-sheet cell, because that was the only cell where the metric was fully implementable against the returned artifacts.

All four artifacts arrived as opaque RGB PNGs. The apparent checkerboard was baked into opaque pixels, not real alpha. The visible acceptance criterion for “transparency” was therefore unmet at the byte level on every observed specimen. The honest verdict is that none of the four hosted endpoints returned a production transparent asset.

The dashboard and network-diagram cells ran visual-review rubric-* rows (0–5) only; no OCR engine and no graph extractor was executed, so those rows are not OCR or topology facts and must not be consumed as such. Pre-implemented thresholds remain on file in ROUND3_PLAN.md for a future focused execution.

Generator / grader separation

The model under test generates the image. It never scores itself. Where a judgment is unavoidable, the reviewer is recorded as reviewer_class: human_assisted_visual_inspection or — when a cross-family visual review is performed — the reviewing model family is disclosed on the same run record. The Round 3 method document flags the one case where reviewer conflict is disclosed in plain text: OpenAI-family artifacts received Grok 4.5 visual review, and Grok 4.5 is supplied by xAI, which is also the provider of the blocked Grok generation route. That provider affiliation is on the record. No xAI artifact was available for head-to-head grading, so no cross-provider grading conflict exists in the released cells.

Sample artifact SHA-256s

The repo holds 50 artifact records under data/runs/ and 2 artifactless failure records (asset: null, non-empty failure_reason, outcome ∈ {blocked, moderated, error, timeout, discarded}). A representative sample, with the bytes and MIME types pinned by the import lane:

The last row is the artifactless failure that closes Round 3’s call accounting honestly: HTTP 403 from the xAI OAuth route, no artifact, failure_reason: "HTTPError: HTTP Error 403: Forbidden", persisted to a canonical failure_record_id that the budget ledger reconciles back to. The failure is data. The cell was not silently re-tried to manufacture a green row.

Provider request IDs were unavailable and remain null; they were not invented.

Honest sample-size caveats

These numbers are what they are:

• Round 1 is one generation per cell. It is a launch signal, not a reliability estimate. Reading the table above as a model leaderboard is the failure mode it is designed to prevent.
• Round 2 is one generation per (checkpoint × prompt) cell. Same caveat.
• Round 3 was two unseeded reps per available cell, with two HTTP 403 failures (gated, not retried), for a 14 / 24 consumed-call accounting. Provider coverage was partial: Grok Imagine quality unrun after the shared OAuth route was classified as blocked; MiniMax Image-01 not configured at preflight; only GPT Image 2 Medium, GPT Image 2 High, and Grok Imagine fast were sampled.
• Judge-from-different-family matters. When the judge shares a provider family with the trial model, scores drift toward “every model ties at the top” (see Model Bench for the same finding). Where visual review was used, the reviewer family is recorded on the run row.

Release gates

Every merge request must pass schema validation, reference validation, secret-URL validation, artifact-provenance validation, the deterministic catalog drift check, the Python test suite, Ruff lint, the Hugo production build, and the required-route / proof-text verification. Budget ledgers are validated by scripts/validate_ledger.py before any run in a round is released. The point of those gates is the same as the sample-size caveats: make a wrong number expensive to publish.

Related and registry

• Live site: artbench.hermanity.dev
• Source repo: herman/artbench
• Round 1 results: docs/ROUND1_RESULTS.md
• Round 2 OSS results: docs/ROUND2_OSS_RESULTS.md
• Round 3 plan + results: docs/ROUND3_PLAN.md, docs/ROUND3_RESULTS.md
• Adjacent project (different question): SwarmUI Image Generation
• Sister-site registry: /sites/