# Tests Are the Verification Layer: Proving Agent Generated Changes

> 🗓️ **Last updated: September 2026**

The coding agent reports that the feature is complete. The project compiles. Four tests pass. The fifth test was deleted because it represented "legacy behavior." This is one way to achieve a green build, in the same sense that removing the battery improves a smoke alarm's noise profile.

Generated code does not need a new definition of correctness. It needs a stronger and more explicit path to the definition the system already has.

## Verification is not model evaluation

An LLM evaluation asks whether a model or agent behaves acceptably across a set of tasks. Software verification asks whether this particular change preserves the required behavior of this particular system. Series 3 covered the first problem. This article is about the second.

The distinction prevents a common mistake. A strong benchmark score does not make an individual patch correct. A weak benchmark score does not make a passing, reviewed patch incorrect. The unit of evidence is the change and the repository around it.

## Build a verification ladder

Every task should climb a known sequence of checks. The exact tools vary, but the layers are stable.

1.  Formatting and generated file checks
    
2.  Compilation and static analysis
    
3.  Focused unit tests
    
4.  Contract and integration tests
    
5.  End to end behavior where the risk requires it
    
6.  Security, compatibility, and performance checks for affected surfaces
    

Run cheap, local checks first so obvious defects return quickly. Run broad checks before merge. The task contract should identify which layers are required. The agent may discover that additional checks are prudent, but it may not decide that required checks feel unnecessary today.

## Protect the oracle

A test is useful only if its expected result comes from an independent requirement. When an agent changes production code and rewrites the failing assertion to match, the build may become green without the software becoming right.

Treat changes to existing tests as a separate review concern. Require an explanation that ties the changed expectation to a contract, requirement, approved behavior change, or corrected defect in the test. A test update is not suspicious by default. An unexplained test update is.

For defects, prefer a regression test that fails before the implementation change and passes after it. For new behavior, connect acceptance criteria to observable tests. For refactoring, require existing behavior to remain stable and use characterization tests where coverage is weak.

## Verify more than the happy path

Coding agents are very good at satisfying the example placed directly in front of them. Production defects often live one input beyond that example. Include boundary values, invalid state, cancellation, retries, concurrency, authorization, serialization, and partial failure according to the domain.

Use property based or generative tests when invariants matter more than individual examples. Use contract tests for service boundaries. Use snapshot tests carefully; a large snapshot approved without inspection is simply a text file that has learned to say yes.

## Make hidden checks legitimate

Some acceptance checks should remain outside the agent's immediate edit surface. This prevents the implementation and its only judge from being produced by the same reasoning process. Hidden checks are common in benchmarks, but the production equivalent is broader: contract suites owned by another component, policy tests, compatibility checks, or independently maintained acceptance tests.

Hidden does not mean mysterious. Engineers should know the categories of behavior being protected. Surprise requirements are poor governance. Independent evidence is good verification.

## Record the evidence

The pull request should state exactly which commands ran, their results, and which checks could not run. Continuous integration remains the authoritative execution environment. Local success is useful feedback, not a substitute for clean execution on the target platform.

Store test reports and relevant artifacts. For a UI change, retain screenshots or visual comparisons where appropriate. For an API change, retain contract results. For a migration, retain forward and rollback evidence. "Tests passed" is less useful than it sounds when nobody can identify the tests.

## Failure modes

The first failure is compile driven development. Compilation proves that the compiler accepted the program. The compiler has never claimed to understand the business.

The second is test laundering. The agent modifies the test until the implementation passes, with no external evidence that behavior should change.

The third is flaky acceptance. Repeated reruns eventually create a green check. Fix or quarantine the flaky test under an explicit policy; do not convert probability into proof.

The fourth is broad change with narrow verification. A shared package changes, but only the local unit suite runs. Verification scope should follow impact, not convenience.

## The architect's checklist

1.  Define a standard verification ladder for the repository.
    
2.  Tie required checks to the task's risk and affected surfaces.
    
3.  Review changed expectations separately from implementation changes.
    
4.  Require regression evidence for defects and contract evidence for interfaces.
    
5.  Run authoritative checks in a clean continuous integration environment.
    
6.  Preserve named commands, reports, and unresolved failures in the handoff.
    

The agent can write the patch and some of its tests. It cannot be the sole author of reality. That position remains occupied by the system's contracts, users, and occasionally production at three in the morning.

## Sources

1.  [SWE-bench repository and benchmark documentation](https://github.com/SWE-bench/SWE-bench)
    
2.  [Anthropic on evaluating AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)