Shipping software
How I review AI-generated changes before shipping
The review procedure behind my own tools: define the result, inspect the diff, verify findings, and test the customer-visible behavior.
Randall Morgan · · 4 min read
Define the result before inspecting the code
I start with the behavior the change is meant to produce. For a billing fix, that includes what happens when a request fails or is repeated. For a page update, it includes what a visitor can read and which action they can take. That gives the review a concrete standard beyond whether the code looks reasonable.
A change can compile and still solve the wrong problem. I compare the implementation with the original requirement and check whether unrelated behavior changed.
Read the diff and the surrounding behavior
I inspect changed files alongside the code they call. Boundaries matter: browser to server, one account to another, payment event to credit ledger, and local draft to published content. A reassuring explanation from the coding agent is useful context, but the implementation and observed result are the evidence.
The review should identify concrete failures: the input, the affected path, and the consequence. Vague suggestions and stylistic preferences should not displace a real correctness issue.
Verify the second opinion
My open-source Codex review skill asks a separate reviewer for findings and then checks them against the code. Findings are classified as confirmed, debatable, or false alarms before they are presented. A second reviewer can spot something the first missed, but can also misunderstand context.
The deeper bug-search skill is reserved for difficult state, concurrency, or data-flow problems. More reviewers are useful only when their claims are checked. The number of agents involved is not a quality metric.
Test the consequence of the change
I choose verification around the likely failure. An authorization change needs evidence that another account cannot reach the data. A checkout or refund change needs its failure paths checked. A website change needs the built pages inspected on a narrow screen and its links followed.
Tests and screenshots describe a particular version and environment. I keep that scope attached to the result rather than claiming that a passing local check proves every production condition.
- Check the intended result and a meaningful failure case.
- Inspect the customer-visible output, not only the test log.
- Keep publication and consequential actions deliberate.
- Document remaining uncertainty before release.
Leave the next person a usable handoff
The final handoff should say what changed, why it changed, what was checked, and what remains unresolved. This makes a later correction easier and gives a reviewer enough context to decide whether the change is ready to ship.