A useful local LLM benchmark must answer more than “how fast did one prompt run?” Local models vary by weights, quantization, runtime, context settings, sampling, hardware, and task. If those variables are not recorded, the result cannot be reproduced or compared honestly.

Hermanity’s Model Bench combines custom operational tasks with recognized industry suites. Local checkpoints run on an RTX 3090 Ti through pinned runtimes, while hosted and subscription models run through their real access paths. The goal is not to crown one universal winner; it is to build evidence for routing decisions.

Start with the decision the benchmark must support

A benchmark is only meaningful relative to a use case. Common questions include:

Which local model is reliable enough for private summarization?
Which checkpoint is best for code generation on one 24 GB GPU?
Does a larger quantized model outperform a smaller higher-precision model?
How much latency is added by a long context?
Where should an agent route structured JSON, debugging, or creative work?
Is the local model a credible substitute for a metered API on this workload?

Write the decision first. Then choose tasks and metrics that can change that decision.

Pin the complete model identity

“At Qwen” or “a 30B model” is not enough. Record:

model family and exact checkpoint;
quantization or precision;
weight-file digest where practical;
inference runtime and version;
prompt template or chat format;
context length and KV-cache configuration;
sampling parameters and reasoning mode;
GPU model, memory, and power limit; and
requested and served model identifiers.

These fields prevent a later rerun from silently testing different bytes under the same display name.

Build a task matrix, not one heroic prompt

A routing benchmark should cover the work the model will actually receive. Hermanity’s custom suite includes task families such as debugging, long code generation, strict JSON, summarization, creative writing, multistep reasoning, refactoring, agentic prompting, and tool use.

For each model × task cell:

Freeze the prompt and grading contract.
Run multiple repetitions.
Preserve raw output and timing.
Grade against explicit criteria.
Record invalid, empty, and failed attempts rather than deleting them.
Keep private grader material out of the model prompt.

Multiple repetitions matter because a model can pass once by chance, especially under non-deterministic sampling.

Measure quality and operational cost separately

At minimum, collect:

Quality

Use task-specific criteria rather than a single subjective impression. Structured tasks can be validated deterministically; open-ended tasks may need a judge model from a different family, plus periodic cross-judging to reveal judge bias.

Validity and failures

Track valid response rate, parse failures, empty replies, hard errors, and task timeouts. A model with a high mean score across surviving outputs may still be unusable if it fails frequently.

Latency and throughput

Measure end-to-end latency, prompt-processing speed, generation speed, and tail behavior. Report the context length and parallelism; throughput figures without those settings are misleading.

Resource use

For local runs, capture peak VRAM, host RAM, GPU utilization, temperature, and power. A checkpoint that barely fits may behave very differently once context or concurrent requests increase.

Economic comparison

Local inference is not literally free. Hardware, electricity, setup time, and capacity have costs. Hosted subscription calls also should not be mislabeled as zero-cost APIs. Keep measured wire cost, estimated public-API equivalent, and local resource usage as separate fields rather than collapsing them into one fictional number.

Use recognized suites without surrendering task relevance

Industry suites help connect local results to wider literature, but they should complement—not replace—your operational workload. Examples used in Hermanity’s evaluation program include GPQA Diamond, IFEval, MMLU-Pro, and HumanEval+.

Each suite has its own answer extraction and eligibility requirements. A valid run must preserve coverage, parser behavior, grader version, and exclusions. “We ran some GPQA questions” is not equivalent to a rank-eligible GPQA result.

Control common sources of false conclusions

Warmup and caching

First-run model loading can dominate latency. Report cold-start separately from steady-state generation.

Prompt-template drift

A runtime update can alter chat formatting. Pin the template and inspect the actual rendered prompt when results move unexpectedly.

Judge-family bias

Do not let a model grade close relatives exclusively. Use deterministic checks where possible and cross-judge a sample of subjective cells.

Selective failure removal

Never drop empty or errored outputs from the denominator without reporting the exclusion. Reliability is part of model quality.

Concurrent GPU jobs

Two processes competing for one GPU can invalidate latency, memory, and thermal measurements. Serialize campaigns unless concurrency is itself the experiment.

Quantization attribution

Treat different quantizations as distinct benchmark entries. A result for Q4 is not automatically a result for FP16 or another Q4 recipe.

A reproducible result package

A strong benchmark release contains:

frozen prompts and task definitions;
exact model/runtime/hardware metadata;
raw outputs with stable identifiers;
grading code and criteria;
aggregate scorecards derived from raw rows;
invalid-trial classification;
hashes or receipts binding the published summary to evidence; and
a plain-language limitations section.

The public Model Benchmark site and the Model Bench project record show this separation between raw evidence, methodology, scorecards, and model pages.

Turn benchmark results into routing policy

The final output should not be “model X is best.” It should look more like:

use checkpoint A for private summarization under this context ceiling;
use checkpoint B for strict JSON because its valid rate is higher;
keep hosted model C for difficult debugging despite higher latency;
do not route tool use to checkpoint D until its failure rate improves; and
rerun after runtime or quantization changes because the prior evidence no longer identifies the same system.

That is the difference between a leaderboard and an operational benchmark.

Local LLM Benchmarking: A Reproducible Guide