# Chapter 9 — Quality and evidence: how Rosetta knows it works

*Most platforms have tests. Rosetta additionally has an **empirical program**: ledgered
runs of its own pipelines against realistic engagements, scored by versioned
instruments, with findings that demonstrably flow back through governance into shipped
releases. This chapter covers both layers — the conventional verification and the
empirical lane — because the second is the more architecturally distinctive.*

## 9.1 Conventional verification

Summarizing what Chapters 4 and 7 detailed, so the reader has one inventory:

- **Producer:** path-filtered CI over ten jobs — shellcheck, Terraform validation,
  schema and seed smoke loads against a live Neo4j, the Markdown↔Cypher round-trip
  check, MySQL migration idempotency re-runs, the **39 invariant gates** (closed
  enums, edge contracts, evidence tethering, partition isolation, purpose
  requirements, linkage), installer smoke against the published route, plugin-bundle
  consistency.
- **Consumer:** 1,028 Go test functions under the race detector; OpenAPI linting;
  config-schema checks for the classifier/flow/twin/governance engines; the
  **behavioral corpus replay** — 62 recorded request/response pairs replayed against
  a real stack as a parity contract (ADR-GO-010); 116 `.http` smoke requests.
- **Plugins:** a validation gate on plugin structure (CI-enforced after a real
  incident class), per-skill eval harnesses in the rosetta-agent plugin, and the
  research/archaeology pipelines' own validation stages.

One verification lesson has been institutionalized and deserves an assessor's
attention because it is subtle: **CI environments must match production's
restrictions, or they miss real failures.** Production Neo4j runs a restricted apoc
allowlist; the Cypher-syntax CI job runs unrestricted apoc and therefore *cannot*
catch an allowlist violation — only the corpus-replay lane (restricted, like
production) does. The platform documents this inside the metamodel itself as a
standing trap note. Finding a system that knows precisely which of its own gates can
lie to it is rare.

## 9.2 The evidence archive

The empirical lane lives in a dedicated member (`rosetta-tests`, charter ratified as
DT-RX-023) that is explicitly **not a software test suite** — it is the run archive:
eight major laps at snapshot (seven numbered test campaigns plus a controlled
bake-off), and the first run of the formalized improvement loop. Its conventions are
the archive's architecture:

- **Locked inputs.** Every lap's inputs are hash-pinned; changing an input means a
  new run directory, never an edit.
- **Pinned toolchains.** Each run's manifest records the exact platform, server, MCP,
  metamodel, plugin, rubric, and scorer versions it ran on — so run-over-run deltas
  are attributable rather than anecdotal.
- **Append-only run logs**, per-run scorecards, retrospectives, and preserved
  baseline outputs from the comparison run.

The instruments are deliberately kept in the umbrella, not the archive: a versioned
**quality rubric** (a maturity ladder plus hard gates plus scored dimensions), a JSON
schema for score files, and a three-tier scoring scheme — mechanical (a stdlib-only
comparator script), model-judged (versioned judge prompts), and human. The yardstick
is versioned independently of every repo it measures (DT-RX-019/020), and the rubric's
own prior version is preserved verbatim in an archive folder — the measurement system
applies the provenance discipline to itself.

## 9.3 The loop, demonstrated

What elevates this from process documentation to evidence is that the loop has
observable output:

- **The remediation train.** One campaign's retrospective produced twelve findings;
  the findings became ratified cross-repo decisions (the DT-id prefixing, the
  provenance lane, and the reachable rehearsal lane among them); the decisions
  shipped as a coordinated release across producer, server, and MCP surface — with
  the production migration gated and ledgered. Finding → DT → ADR → release,
  traceable end to end in the record.
- **The bake-off.** A controlled two-arm comparison (vanilla agent vs.
  Rosetta-equipped, same dossier, plan-only) scored the Rosetta arm ahead — and
  surfaced the platform's most important known limitation: **twins transmit blind
  spots with authority.** A medium-confidence claim entered the twin and emerged in
  downstream renders as flat fact, with zero gap items to catch it. The finding was
  written up with the same rigor as the wins.
- **The structural response.** That finding did not stay a retro bullet: at snapshot
  it is an open cross-repo decision (fact-level bitemporality — assertions
  invalidated, never overwritten, with an as-of read surface), which is the
  governance system metabolizing its own evaluation output into schema evolution.

## 9.4 What the empirical record supports — and what it doesn't yet

An assessor should be precise about the claim strength here. The record **supports**:
that the pipelines run end-to-end on realistic engagements; that output quality is
measured, versioned, and improving across laps; that the feedback loop from finding to
shipped fix operates; and that the organization reports its failures with unusual
candor (the honest labeling is itself verifiable — simulated stamps, "Not Met"
scorecard rows, and retro findings are in the artifacts).

The record does **not yet** contain: an independent third-party run (every lap was
executed by the same operator-plus-agent team that built the system); a production
engagement with a paying customer (the flagship engagements are simulated or internal
dogfood); or statistically meaningful sample sizes (laps are n=1 by design, compared
against a baseline lap). These are the right next experiments, not hidden gaps — the
bake-off's own retro proposes the vanilla-challenger re-run, and the archive's
structure is built for exactly this accumulation.

*The verdict: [Chapter 10](10-evaluation.md).*