projects / bunsens-lab
A charcoal lab mouse in cyan safety goggles watches a controlled blue flame under a glass observation dome.
INFRASTRUCTURE ·

Bunsen's Lab

An isolated agent laboratory where every optimization hypothesis is measured against a falsifier before it gets anywhere near production. Eight experiment lanes, zero unauthorized mutations.

Overview

Bunsen’s Lab (live at lab.hermanity.dev) is the public notebook for work performed by Bunsen — a sandbox Hermes Agent built to heat-test changes before they get anywhere near Herman.

Bunsen is a charcoal lab mouse in cyan safety goggles, a navy coat, and a controlled blue flame. The mascot is playful; the operating boundary is not. Bunsen has a scoped model key, no Honcho memory service, no copied Herman identity credentials, and no authority to mutate Herman.

The site translates raw JSON, timing traces, test logs, and patch diffs into pages a human can audit quickly. Every result comes from an isolated Bunsen profile or worktree. Nothing published here means a change has been promoted to Herman. Candidate status means the experiment produced evidence worth review — not permission to ship it.

The lab rule

A result is not an authorization. Experiments run on Bunsen, in transient service units, identity-free profiles, or detached worktrees. Herman remains untouched until the operator explicitly approves a candidate.

What every experiment records

  1. Hypothesis — what should change and why.
  2. Baseline — the pre-change number or failure mode.
  3. Intervention — the smallest isolated candidate.
  4. Result — measured runs, tests, latency, and process state.
  5. Falsifier — the observation that would overturn the conclusion.
  6. Promotion boundary — what still has to be true before Herman changes.

The eight experiment lanes

ExperimentStatus
Candidate release topologypass
Model routing receiptspass
Fork reconciliationpass
Lifecycle safetypass-with-observation
Honcho config cache-bustingpromoted-to-fork
Parallel npm auditscandidate
Source-patch dispositionreconciled
Web-capability fallbackmeasured

That status vocabulary matters. It is not a pass/fail binary:

  • pass — the predeclared claim survived its falsifier.
  • measured — useful evidence, but not a promotion candidate.
  • candidate — worth human review; not installed.
  • hold — evidence exists, but a required gate is still unresolved.
  • promoted-to-fork — merged into the downstream fork after its exact-head gates; still not installed on Herman.
  • reconciled — competing or stale findings were classified into a durable disposition.
  • pass-with-observation — the safety claim passed while an adjacent workload criterion missed.

What a real experiment looks like

The model routing receipts experiment is a good representative. Three fresh, identity-free Bunsen profiles exercised the same scoped Skynet key: a primary model, a direct fallback, and an intentionally invalid primary with a known fallback. The claim was that a configured fallback can succeed functionally while observability reports only the requested model. The falsifier: fail the claim if the fallback response succeeds but the receipt names the invalid primary, the canonical config hash changes, or the model is absent from the authenticated live roster.

The forced fallback ran, produced the expected response, and the receipt correctly named the fallback model — not the invalid primary. The canonical Bunsen config SHA-256 was unchanged before and after. The claim survived.

Evidence handling

Raw evidence stays on the lab VM and in the agent’s evidence workspace. Public pages contain sanitized summaries and SHA-256 handles only. A sensitive-name scanner checks values associated with TOKEN, KEY, SECRET, and PASSWORD keys. Scanner failures are treated as evidence failures, not formatting warnings.

Test interpretation is equally conservative: targeted green tests prove the changed surface behaves as intended. They do not erase unrelated full-suite failures. A candidate remains held until the applicable repository test and lint gates are green in a clean promotion worktree.

Qualification

Phase 3 is a live qualification in progress — re-baselining the downstream fork on upstream and qualifying the candidate under real, non-offline load. The current qualification state is published on the lab homepage, with evidence handles retained in the artifact index.

Why this project exists

When an agent can write code, open MRs, and self-merge in its personal namespace, the gap between “it works in my sandbox” and “it’s installed on production” needs to be explicit, not implicit. Bunsen’s Lab is that gap. The falsifier column is the difference between an experiment and a vibe.