AI moves testing from brittle scripts to durable intent and evidence.
Explore the course
A BOOK BY JASON ARBON
Test at the speed of AI.Software is easy to generate.
Confidence is hard.
How AI Tests Software is a field guide to using AI to inspect, explore, test, challenge, and qualify the software that people and AI agents build. For a deep dive into testing AI itself—including AI-based functionality and systems—read my companion book, Testing AI. Explore the TestBucks practice website and companion materials as you read.
AI CHECKS•AI TESTS•AI CONFIDENCE
A substantial field guide.From AI checks to release confidence.Illustrative hardcover rendering.
THE QUALITY PROBLEM HAS BECOME THE CONTROL POINT OF SOFTWARE.
01 / THE THESIS
AI is subsuming
the old testing stack.
Test frameworks, recorded scripts, test-case databases, issue trackers, and reporting dashboards were abstractions built so people could manage software complexity. AI can work directly from intent, source, screenshots, logs, traces, and the current product state.
The reusable asset is no longer a fragile click sequence. It is the promise, the evidence contract, the authority boundary, and the decision the result can change.
The coding agent can orchestrate validation, but independent validation has to qualify the result.
AI will spend more compute validating, comparing, repairing, and monitoring software than initially generating it.
02 / A BETTER VOCABULARY
Four terms for a new quality system.
Stop using “AI testing” as one vague bucket. Name the capability, the interaction, the workflow, and the decision.
01
AI Checks
Use screenshots, logs, network activity, DOM, traces, and source to inspect quality evidence without touching product state.
02
AI Tests
Drive browsers, devices, APIs, data, and the whole system to prove a user or business intent.
03
AI Test Harnesses
Run the long-lived workflow: context, planning, execution, challenge, reporting, and AI Confidence.
04
AI Confidence
Decide whether the evidence is enough to ship, canary, hold, repair, or gather more proof.
03 / THE LOOP
Generate. Validate.
Learn. Repeat.
An AI Test Harness turns a fresh repository, a change, a URL, or an existing suite into a stateful quality workflow. It can discover risk, generate candidates, execute them through any adapter, challenge its own findings, and continue until AI Confidence converges.
LONG-RUNNING QA WORKFLOW
- 01Context + riskWhat changed? What matters?
- 02ExecuteRun the right checks and tests.
- 03ChallengeTry to disprove the findings.
RESULTAI ConfidenceShip, hold, or gather more evidence.
04 / INDEPENDENCE
The creator can ask
for the test. It cannot
be the only judge.
No one buys a house without an independent inspector. No one accepts a bridge because the contractor says the concrete looks good. AI-generated software needs the same separation.
The coding agent orchestrates from the richest possible context. An independent cloud validation plane gathers its own evidence, challenges the result, and qualifies what is ready to ship.
DRAFT READERS & CONTRIBUTORS
A better book, thanks to you.
Thank you to the readers who volunteered to help shape this draft and agreed to be recognized here. Your time, questions, and feedback matter to me.
A star marks feedback incorporated into the book so far.
- Anna Zybina
- Billy Perez
- Chris Williams
- Christian Manna Guimma
- Gaurav Khurana
- Gokul Sridharan
- Jorge Diz
- Maneesh Maddala
- Michael BOST★
- Nicholas Shaw
- Nikos V. Kamvysis
- Nimesha Weerasinghe
- Pradeep Verma
- Rahul Parwal
- Rohit Sharma
- Ronak Ray
- Saqib Javed
- Shadi Moghaddasi, MSc
- Sumit Chaudhary
- Victor Ematin★
Listed with permission. Recognition is a thank-you, not an endorsement. This list will grow as I confirm permissions and incorporate more feedback.
START HERE
Test at the speed of AI.
Qualify what ships.
Put the ideas into practice, and share what you learn.