Chapter 9 — Quality and evidence: how Rosetta knows it works

Most platforms have tests. Rosetta additionally has an empirical program: ledgered runs of its own pipelines against realistic engagements, scored by versioned instruments, with findings that demonstrably flow back through governance into shipped releases. This chapter covers both layers — the conventional verification and the empirical lane — because the second is the more architecturally distinctive.

9.1 Conventional verification

Summarizing what Chapters 4 and 7 detailed, so the reader has one inventory:

• Producer: path-filtered CI over ten jobs — shellcheck, Terraform validation, schema and seed smoke loads against a live Neo4j, the Markdown↔Cypher round-trip check, MySQL migration idempotency re-runs, the 39 invariant gates (closed enums, edge contracts, evidence tethering, partition isolation, purpose requirements, linkage), installer smoke against the published route, plugin-bundle consistency.
• Consumer: 1,028 Go test functions under the race detector; OpenAPI linting; config-schema checks for the classifier/flow/twin/governance engines; the behavioral corpus replay — 62 recorded request/response pairs replayed against a real stack as a parity contract (ADR-GO-010); 116 .http smoke requests.
• Plugins: a validation gate on plugin structure (CI-enforced after a real incident class), per-skill eval harnesses in the rosetta-agent plugin, and the research/archaeology pipelines' own validation stages.

One verification lesson has been institutionalized and deserves an assessor's attention because it is subtle: CI environments must match production's restrictions, or they miss real failures. Production Neo4j runs a restricted apoc allowlist; the Cypher-syntax CI job runs unrestricted apoc and therefore cannot catch an allowlist violation — only the corpus-replay lane (restricted, like production) does. The platform documents this inside the metamodel itself as a standing trap note. Finding a system that knows precisely which of its own gates can lie to it is rare.

9.2 The evidence archive

The empirical lane lives in a dedicated member (rosetta-tests, charter ratified as DT-RX-023) that is explicitly not a software test suite — it is the run archive: eight major laps at snapshot (seven numbered test campaigns plus a controlled bake-off), and the first run of the formalized improvement loop. Its conventions are the archive's architecture:

• Locked inputs. Every lap's inputs are hash-pinned; changing an input means a new run directory, never an edit.
• Pinned toolchains. Each run's manifest records the exact platform, server, MCP, metamodel, plugin, rubric, and scorer versions it ran on — so run-over-run deltas are attributable rather than anecdotal.
• Append-only run logs, per-run scorecards, retrospectives, and preserved baseline outputs from the comparison run.

The instruments are deliberately kept in the umbrella, not the archive: a versioned quality rubric (a maturity ladder plus hard gates plus scored dimensions), a JSON schema for score files, and a three-tier scoring scheme — mechanical (a stdlib-only comparator script), model-judged (versioned judge prompts), and human. The yardstick is versioned independently of every repo it measures (DT-RX-019/020), and the rubric's own prior version is preserved verbatim in an archive folder — the measurement system applies the provenance discipline to itself.

9.3 The loop, demonstrated

What elevates this from process documentation to evidence is that the loop has observable output:

• The remediation train. One campaign's retrospective produced twelve findings; the findings became ratified cross-repo decisions (the DT-id prefixing, the provenance lane, and the reachable rehearsal lane among them); the decisions shipped as a coordinated release across producer, server, and MCP surface — with the production migration gated and ledgered. Finding → DT → ADR → release, traceable end to end in the record.
• The bake-off. A controlled two-arm comparison (vanilla agent vs. Rosetta-equipped, same dossier, plan-only) scored the Rosetta arm ahead — and surfaced the platform's most important known limitation: twins transmit blind spots with authority. A medium-confidence claim entered the twin and emerged in downstream renders as flat fact, with zero gap items to catch it. The finding was written up with the same rigor as the wins.
• The structural response. That finding did not stay a retro bullet: at snapshot it is an open cross-repo decision (fact-level bitemporality — assertions invalidated, never overwritten, with an as-of read surface), which is the governance system metabolizing its own evaluation output into schema evolution.

9.4 What the empirical record supports — and what it doesn't yet

An assessor should be precise about the claim strength here. The record supports: that the pipelines run end-to-end on realistic engagements; that output quality is measured, versioned, and improving across laps; that the feedback loop from finding to shipped fix operates; and that the organization reports its failures with unusual candor (the honest labeling is itself verifiable — simulated stamps, "Not Met" scorecard rows, and retro findings are in the artifacts).

The record does not yet contain: an independent third-party run (every lap was executed by the same operator-plus-agent team that built the system); a production engagement with a paying customer (the flagship engagements are simulated or internal dogfood); or statistically meaningful sample sizes (laps are n=1 by design, compared against a baseline lap). These are the right next experiments, not hidden gaps — the bake-off's own retro proposes the vanilla-challenger re-run, and the archive's structure is built for exactly this accumulation.

The verdict: Chapter 10.