Passing Tests Is Not the Same as Being Accepted
One of the easiest mistakes in AI-assisted work is to let a passing test suite close a question it was never designed to answer.
The build passes. The types check. The page renders. The agent reports success. Everyone feels the gravitational pull of done.
But “it passed” is incomplete without another question:
It passed which layer?
I now think about acceptance as a ladder. Each rung answers a different question, and evidence from one rung should not be used to make claims about the next.
1. Build validation
Does the artifact compile, render, or assemble?
This catches broken imports, invalid schemas, syntax errors, missing assets, and other structural failures. It is necessary and usually cheap. It tells me that the thing can exist in the target environment.
It does not tell me that the thing is correct.
2. Deterministic checks
Does the artifact satisfy the rules we know how to encode?
Tests, lint rules, validators, type systems, accessibility checks, security scanners, and snapshot comparisons live here. Good checks turn important expectations into repeatable evidence. They are especially valuable in AI workflows because they do not get persuaded by confident prose.
But deterministic checks can only enforce the contract we actually wrote. They do not notice the requirement we forgot, the assumption embedded in the fixture, or the product question we never translated into a test.
An AI can also generate tests that faithfully confirm its own misunderstanding. Coverage rises while confidence should not.
3. Smoke checks
Does the critical path work in something resembling the real environment?
Open the page. Click the link. Submit the form. Restart the process. Use the actual runtime. Check the production-shaped route rather than only the isolated function.
Smoke checks catch integration failures that unit tests miss. They are often where “works on my machine” becomes a more useful claim: “worked once across the intended path.”
That is still a narrow claim. A successful smoke check is not proof of reliability, usability, or resilience.
4. Human review
Does the result serve the purpose?
This is where taste, coherence, and meaning enter. A human reads the article, rehearses the presentation, reviews the interaction, or examines the decision. They ask whether the artifact says the right thing—not merely whether it contains valid components.
For a presentation, a render proves that every slide exists. Rehearsal reveals whether the speaker can own the argument. For a research brief, valid citations prove that sources are attached. Human review reveals whether the evidence supports the conclusion. For a product, a working flow proves that the buttons connect. Human use reveals whether the flow makes sense.
Human review is not a ceremonial sign-off after the “real” technical work. It is a different evaluation instrument.
5. Real-world acceptance
Does the result keep working for the people and conditions it was built for?
This may require actual users, production traffic, repeated runs, failure recovery, support feedback, or time. It is the layer where hidden assumptions meet reality.
A controlled pilot may show that a workflow is understandable. It does not automatically prove that it is safe for sensitive data. Three successful runs may show that the mechanism works. They do not establish long-term reliability. A user saying “this looks good” may accept the direction without accepting the operational burden.
Real-world acceptance is specific. Who accepted it? For what use? Under which conditions? What remains unproven?
Keep the receipts separate
The problem is not that teams skip all these layers. It is that the evidence gets compressed into a single green status.
“QA passed” may mean the build completed, two fixtures passed, and one person glanced at the page. That sentence hides three different claims and leaves two more unanswered.
I prefer a small acceptance record:
- Build: passed in the target runtime.
- Deterministic checks: named checks and their results.
- Smoke: exact path exercised and environment used.
- Human review: reviewer, purpose, and decision.
- Real-world use: observed conditions, failures, and open risks.
This does not need to become bureaucracy. A few precise lines are more useful than a polished paragraph claiming the work is complete.
AI makes the distinction more important
AI systems are unusually good at producing artifacts that carry the visual signals of completion. They produce the code, the tests, the documentation, and the success summary in one motion. The package feels internally consistent because the same system generated all the pieces.
That consistency can be deceptive.
The remedy is not distrust of every output. It is evaluation independence. Use deterministic tools where possible. Exercise the real path. Ask a human to judge the actual purpose. Record what was observed and what remains assumed.
A passing test suite is good news. It means a defined set of expectations held under a defined set of conditions.
Acceptance is the larger decision: this is fit for this purpose, for these people, within these boundaries.
The first is evidence for the second. It is not a substitute for it.