Verifying AI-Written Code: A Playbook

2026-08-11 · act101 · testing, agents

The agent finishes and reports back: "Refactored the settlement path, extracted the retry logic, behavior unchanged." The diff is clean. The tests are green. You merge it. Three weeks later a payment double-fires in production, and the postmortem lands on a guard clause that used to sit inside a try and now sits outside it. The change compiled. The suite passed. Nothing you had would have caught it, because you were never testing that boundary in the first place.

"The tests pass" is a real signal, but it answers a narrower question than teams give it credit for: the code you thought to test still does what you asserted. It says nothing about the code you didn't test, and an agent rewriting a function does not confine its edits to the paths your suite happens to exercise. A passing suite proves the assertions held. It does not prove the change the agent actually made preserved behavior.

Verification asks that second, harder question directly. Given two versions of a function — before and after the agent touched it — did the contract survive? Did the control flow keep its shape? Did a cleanup path quietly disappear? And, most importantly, it is built to answer "I could not tell" out loud, as a distinct verdict from "yes." A verifier that always says looks equivalent is worse than no verifier at all, because teams learn to trust it and then it is wrong once.

If you want the whole discipline in one sentence: classify what the change is, verify whether it preserved behavior, and gate the merge on the evidence — while never painting "unknown" green. act101 implements that as a family of differential verification tools an agent runs against its own diff, each returning a three-valued verdict that discloses exactly which dimensions it was able to judge.

Why verifying agent-written change needs its own discipline

Review and testing were designed around a human writing code at human speed. Both assumptions break when an agent writes the change.

The old backstop was a senior engineer reading every meaningful diff. That was already the bottleneck when humans wrote every line. It cannot scale to code generated all day by a process that starts each session with a clean memory and a bias toward looking finished.

The new backstop is differential and structural. Take the two versions, compare their modeled behavior directly, and return a verdict that says what it checked and what it couldn't. The verifier is the skeptical reviewer the agent's own confidence can't replace.

The verification loop, end to end

agent edits ─► classify ─► verify ─► gate ─► merge
    ▲            (what      (did it   (is it
    │            changed?)  survive?) safe?)
    └──────── remediate the named divergence ────────┘

There are several tools in the loop, but only three operating stages.

  1. Classify the change. For each touched function, decide whether the hunk is format, signature, or behavior. Most edits are not behavior changes; naming them cheaply lets you spend scrutiny where it belongs.
  2. Verify preservation. For the changes that matter, compare the two versions dimension by dimension — contract, control-flow shape, side effects, cross-language parity — and get a Preserved / Diverged / Unknown verdict per function.
  3. Gate the merge. Combine each verdict with whether a test actually reaches the change, and produce one decision: Merge, Review, Block, or Unknown — with Unknown held distinct from Merge so "we couldn't judge this" never masquerades as "this is fine."

The back-edge is the point. A blocking verdict does not say "behavior bad" and hand a human a wall of diff. It names the function and the dimension that moved — raises count 1 → 0, dropped cleanup, branch shape changed with no test reaching it — so the engineer or the agent can repair that specific boundary and re-run the same check. The evidence survives the session as a receipt; the correction re-enters the loop.

This sits beside tests and review rather than replacing them. Tests assert intended behavior on the paths you chose. Review judges whether the change is a good idea. Verification answers a third question neither was built for: did this edit preserve the behavior that was already there? Keep the three separate and each stays trustworthy.

Stage 1 · classify the change

Not every diff deserves the same suspicion. The cheapest useful move is to sort each function's change into a kind, so a reviewer — human or agent — knows where to look before looking.

verify_diff_semantics takes two versions of a file and classifies each changed function's hunks as one of:

The classification runs on model diffs, not string diffs: signature via an interface comparison, behavior via a control-flow and effect comparison, format via "AST-equal but text-different." A local rename that a text diff would flag as a large change collapses to Format, because nothing about the function's modeled behavior moved. That is the entire value — it stops the eye from spending its budget on reflow and points it at the one hunk that changed a branch.

For a whole pull request, summarize_pr aggregates the per-function classifications across every touched symbol into counts: how many signature changes, how many behavior changes, how many format-only. A PR that is forty format-only hunks and one behavior change is a very different review than its line count suggests, and the summary makes that legible before anyone opens the diff.

Classification is triage, not judgment. Format-only is a strong hint a hunk is safe; it is not yet a proof that behavior survived. It tells you where the next stage needs to run, not that the next stage can be skipped.

Stage 2 · verify preservation

Once you know a function's behavior changed — or you simply want proof it didn't — the differential verifiers compare the two versions directly. Each returns a three-valued verdict and, on every result, a modeled_kinds field that discloses which dimensions it was actually able to judge for the grammars in play.

verify_contract_preserved checks a function's public and behavioral contract across the two versions: signature arity, return shape, the set of side effects, control-flow shape, and raises. The verdict is preserved, broken{dimensions}, or unknown{dimensions}. The rule that makes it trustworthy: it never reports preserved on a dimension the grammar does not model. If the language's control-flow modeling can't see a branch construct, that dimension comes back unknown, not a quiet pass.

verify_behavioral_equivalence compares the two versions' control-flow graphs — the count and shape of branch, loop, exception, and return edges — and returns equivalent, changed{dimensions}, or unknown{reason}. Same branch-and-loop skeleton, same modeled behavior. This is the tool an agent runs after an extract-function or an inline to prove it moved code without moving behavior.

verify_side_effects diffs the effect signature between versions: which reads, writes, allocations, and blocking calls were added or removed. It flags one class by name — dropped cleanup: an allocation still happens on both sides, but a matching write or blocking teardown was removed. That is the shape of the classic leak the agent introduces while "simplifying" a resource path, and it is exactly the pattern a green test suite sails straight past.

verify_port_parity is the cross-language verifier. It compares a source symbol and its port dimension by dimension — signature arity, return presence, effects, control-flow shape, raises — and, crucially, judges a dimension only when both grammars model it. An effect kind one language expresses and the other doesn't is excluded from the comparison rather than reported as a divergence. It will only return preserved when at least two dimensions were jointly modeled and matched; a matched signature alone is not parity evidence and comes back unknown. A single real divergence dominates and yields diverged.

Port parity has an optional execution tier for the languages where it is safe to run untrusted-to-you-but-yours code: for TypeScript/JavaScript (node) and Python (python3), when the runtime is present and the signature is JSON-representable, it can drive both versions with a handful of deterministic inputs and diff the outputs, upgrading a structural verdict to an observed one. It runs each side under a resource boundary — CPU, memory, and file-size caps — and, if any input's output diverges, forces the verdict to diverged. Two honesty rules govern it: this is a resource boundary, not a security sandbox (there is no network isolation, and it is scoped to your own source and ported code, not adversarial third-party code), and it never claims execution parity it did not actually run. Any eligibility gap — a compiled language, a missing runtime, a signature that won't serialize — falls back cleanly to the structural verdict with the reason recorded.

The through-line across all four is the same: the verdict distinguishes preserved from diverged from I couldn't judge this dimension, and it tells you which was which. Structural verification is deliberately not compiler-grade — it compares the shape of control flow and effects, not alias-aware execution paths or precise exception targets. It would rather return unknown than launder a guess into a green check.

Stage 3 · gate the merge

Classification and per-function verdicts are inputs. A merge needs one decision.

Two tools produce it. verify_test_impact finds the minimal set of tests whose call graph transitively reaches the changed symbols — the honest answer to "is this change actually covered?", derived from the call graph rather than from a coverage report's optimism. It walks from the change outward, and where it cannot resolve a callee or hits its depth cap, it says so rather than pretending the frontier is empty.

The gate then composes the change class, the test reach, and the effect findings into a single verdict by a first-match rule set:

The verdicts carry an exit contract for CI: Merge = 0, Block = 1, Review = 2, Unknown = 3. Read that carefully — the codes are deliberately not ordered so that "anything nonzero is failure." Review and Unknown are their own states with their own numbers, and a CI script must branch on the exact code, not on != 0. The overall verdict for a pull request is the most severe verdict across all its changed functions: one blocked function blocks the PR.

Exit 3 must never be painted green. "We could not judge this change" and "this change preserved behavior" are different statements, and collapsing them is how a verification gate quietly becomes theater. Unknown is a request for a human or a better-modeled grammar, not a pass.

Because the whole surface is exposed to the agent through MCP, the agent runs this stage on itself before it ever reaches CI. A model that finishes an extraction can classify its own hunks, verify the contract held, check whether a test reaches the change, and read the gate verdict — all before claiming the work is done. CI stays the backstop, not the first place anyone learns a new divergence appeared.

The agent does not get to certify its own diff by describing it. "I preserved the behavior" is a claim generated by the thing that made the change. "Contract preserved, CFG equivalent, dropped-cleanup false, one test reaches it, gate: Merge" is a receipt. Insist on the receipt.

The merge gate

A verification verdict is one input to a merge decision, not a substitute for tests, review, or security gates. The minimum that makes this input trustworthy:

  1. The comparison is real. Both versions parse and the target function is found in each. A verifier that couldn't build one side returns Unknown, never a pass.
  2. The verdict names its dimensions. Every result carries modeled_kinds. A preserved that silently skipped the dimension you cared about is not the reassurance it looks like — read what was judged.
  3. Unknown stays its own state. Tier denial, analysis failure, and unmodeled constructs surface as Unknown and are triaged, not swept into the pass bucket by a != 0 check.
  4. Test reach is derived, not assumed. "Covered" means a test's call graph reaches the change, not that overall coverage is high somewhere else in the file.
  5. Every block carries before → after evidence. A failing gate names the function and the dimension that moved. That is what an engineer acts on; a bare red X is what they learn to override.
  6. Port parity claims match what ran. A structural parity verdict says structural; an execution-enriched one says execution. The two are labeled differently because they prove different things.
  7. The receipt survives the job. If verification ran, its evidence is persisted against the function's before/after hashes so a later reviewer can trust it without re-running — and discard it the moment the code changes underneath it.

A pull request green on these conditions has preserved the behavioral dimensions the verifier could model. It may still be a bad feature, a security regression, or a behavior change on a dimension no grammar models yet. Keep the claim narrow. Narrow claims are the ones teams keep trusting.

Best practices, in plain English

Failure modes & gotchas

These are the ways a verification gate turns into decoration — or worse, a confident false pass.

The common thread is honesty. A verifier earns authority only when its preserved is narrow and provable, its Unknown is loud, and its evidence survives the diff it describes. The first time a team learns the gate said "equivalent" about a change that wasn't, they stop reading its verdicts — even after the gap is fixed.

Cost, and the agent budget

Structural verification is cheap where it matters. Comparing two versions of a function is a pair of analyses on a bounded span, not a whole-repository crawl, so an agent can verify each function it touches without turning a refactor into a batch job. Classification and contract checks are fast enough to run inline, per function, in the agent's own session.

The one stage with real cost is execution-tier port parity, because it spawns subprocesses. That cost is opt-in and bounded: it runs only for eligible interpreter languages with serializable signatures, only a handful of deterministic input cases, and only under CPU, memory, and file-size caps. Everywhere else, parity stays structural and free of a runtime. You pay for execution exactly where it buys you an observed verdict, and nowhere else.

For the agent, the valuable output is not the two function versions reread into context. It is a compact verdict: the change class, the preservation result, the deciding dimension, and whether a test reaches it. That compact receipt is what lets a model self-gate cheaply — it spends a few structural comparisons, not a re-ingestion of the codebase, to earn the right to say a change is done.

How act101 implements the loop

The discipline is describable without a product: take two versions, compare modeled behavior, and gate on a verdict that admits what it couldn't judge. The hard part is producing one comparable behavioral signature across languages, refactors, ports, agent sessions, and CI — and refusing to fake it when the inputs aren't comparable.

act101 builds these verdicts from the same structural analyzers it uses for the rest of its work:

The verification family is exposed through MCP as the agent's surface — verify_diff_semantics, verify_contract_preserved, verify_behavioral_equivalence, verify_side_effects, verify_test_impact, and summarize_pr — with bisect_regression to find the commit that introduced a behavior change (semantic bisect ships in Architecture Edition), and durable receipts as content-addressed evidence of a run. The verification family ships in Engineering Edition: the same tier that gives the agent deterministic refactor operations gives it the means to prove those refactors preserved behavior. Cross-language verify_port_parity, and behavioral equivalence run in cross-language port mode, ship in Enterprise Edition, alongside the porting state machine they exist to police — because a migration you cannot verify is a migration you cannot trust.

The verdict is evidence, not reassurance. Same two versions, same modeled dimensions in; same three-valued verdict out. When the versions aren't comparable, or a dimension isn't modeled, the tool returns Unknown instead of smoothing the gap into a prettier "equivalent."

The maturity ladder

Tick what the team does today, not what the rollout plan hopes for.

Zero to three: verification by vibes and a green suite. Four to six: behavior changes get real scrutiny but the loop is manual. Seven to nine: the agent and CI both meet the same evidence. Ten: every change — refactor or port, human or agent — arrives at the merge with a verdict that says what it preserved and admits what it couldn't judge.

A reasonable 30 / 60 / 90-day plan

  1. Days 1–30 — classify and observe. Turn on diff-semantics classification and summarize_pr for agent-authored pull requests. Do not block anything yet. Learn the ratio of format-to-behavior in your agent's output, and start reading modeled_kinds so the team internalizes what "preserved" does and doesn't cover.
  2. Days 31–60 — verify the behavior class. For every behavior-class change, run contract and equivalence checks, and run side-effect diffs on resource-handling paths. Have agents self-verify during refactors, still without CI authority. Begin blocking on dropped cleanup — it is the finding least likely to cost you a false positive while you build trust.
  3. Days 61–90 — gate with intent. Wire the four gate verdicts into CI, handling each explicitly and keeping Unknown its own state. Persist receipts. For teams doing migration work, add verify_port_parity to the port pipeline and enable the execution tier where the languages qualify. Track every gate override as a policy bug, not as normal operation.

At ninety days the win is not a green board. It is a team that can say, of any agent-written change, this preserved behavior on the dimensions we can model, a test reaches it, and here is the receipt — and, just as valuably, can point at the changes where the honest answer was "we could not tell," and knows a human looked.

What differential testing got right (and what structural verification adds)

The idea underneath this is old and well-earned. Differential testing — run two versions on the same inputs, compare the outputs — has caught compiler bugs and validated rewrites for decades. Equivalence checking is a staple of hardware verification, where "the optimized circuit behaves like the reference" is a property you prove, not hope. The instinct is right: to trust a transformation, compare the thing before with the thing after and insist they match.

Those techniques assumed you could run both versions on shared inputs. That assumption holds for a compiler and a circuit. It frays for a function mid-refactor with no isolated harness, and it breaks entirely for a port where the two versions don't share a language, a runtime, or a test. What was missing was a way to compare modeled behavior structurally — control-flow shape, effects, contract, cross-language parity — when execution isn't available, and to be honest, dimension by dimension, about the limits of that model.

That is the axis structural verification adds. It compares what it can model directly, runs the code where running it is both possible and cheap, and returns a verdict that separates preserved from diverged from couldn't tell. The shortest version is the useful one: classify the change, verify it preserved behavior, gate the merge on the evidence — and when the evidence runs out, say so, rather than call it green. Do that on the cadence agents write code, and "the agent said it was fine" stops being the last word before production finds out otherwise.

Discuss on X