AI Made Development Faster. Testing Needs to Stop Living in Spreadsheets.

A developer built testboat, a structured testing system that treats test artifacts like code, to address the bottleneck of proving what was tested and whether a release is safe as AI agents accelerate development. The tool creates a .testboat directory with connected YAML files for requirements, test cases, execution plans, bugs, and reports, enabling teams to query release evidence instead of relying on memory.

AI agents are making software development faster. That is great. But there is a problem I do not think we are talking about enough: testing is not speeding up in the same way. In many teams, testing is still held together by spreadsheets, meeting notes, screenshots, chat messages, and the memory of a few experienced QA engineers. That worked when delivery was slower. It becomes fragile when one developer can use multiple agents to change code across several modules in a single afternoon. The bottleneck is no longer "can we write more test cases?" The bottleneck is: Can the team prove what was tested, why it was tested, what failed, what was fixed, and whether the release is safe? That is the problem I built testboat for. The sentence I worry about most is not: We did not test this. At least that is honest. The dangerous sentence is: I think we tested this. That sentence usually means the team has test artifacts, but they are disconnected: Each piece may be useful on its own. But when a Tech Lead asks, "Which requirements are not covered?" or a founder asks, "Can we release today?", the team has to reconstruct the answer manually. That is not a testing process. That is institutional memory under pressure. AI agents are very good at increasing throughput. They can: But faster change creates more testing uncertainty. If an agent changes the authentication module, what should be rerun? If a test fails, is it a product bug, a flaky automation script, or an environment issue? If a developer says "fixed", has the failed test actually been rerun? If a release report says "main flows passed", where is the evidence? Without a structured system, QA becomes the human buffer. Tech Leads become risk translators. Founders buy uncertainty with every release. That is not sustainable. testboat treats test artifacts like code. It creates a .testboat/ directory in your project: .testboat/ .active draft/ strategy.yaml tags.yaml cases/ TC-001.yaml bugs/ BUG-001.yaml executions/ plans/ results/ execution-matrix.yaml automate/ reports/ The important part is not "YAML is nice." The important part is connection . A requirement connects to a test case through req id . A test case connects to an execution plan. An execution plan connects to an automation script. A result connects back to the test case. A bug can connect to both the test case and the failing result. The latest execution state is summarized in an execution matrix. Reports are generated from the same artifacts, not written from memory. That changes the conversation. Instead of asking: Did we test login? You can ask: Show me every auth test case, its latest result, open bugs, and whether the release exit criteria passed. QA should not have to be the team's memory database. With testboat , a test case is a structured file: id: TC-001 title: Login with wrong password returns 401 status: ready priority: P1 automation: to-automate tags: sprint: v1.0 type: functional module: auth req id: STORY-001 steps: - action: Enter wrong password expected: API returns 401 expected result: User sees a clear error message It is diffable. It is reviewable. It has a state: php draft - ready - pass / fail / blocked / skipped That means QA can maintain testing facts instead of constantly answering questions from memory. Tech Leads need quality gates, not just good intentions. testboat validate runs pre-report checks: That last part matters. Your strategy.yaml can define severity rules and exit criteria. For example, P0 and P1 bugs must be zero before release. So the report is not just a nice HTML page. It is generated after the system checks whether the release evidence is healthy enough. This is the kind of thing that can eventually belong in CI. Founders do not need to read every test case. But they do need release confidence. "Main flows passed" is not enough. The useful questions are: testboat generates strategy, sprint, and closure reports from the actual test artifacts. That gives leadership evidence instead of vibes. The goal is not to replace QA. The goal is to give AI agents a testing workflow they can follow. testboat enable creates agent-specific instructions for tools like Claude, Copilot, Cursor, Kiro, and others. An agent can then follow a repeatable SOP: That is the difference between "AI wrote some tests" and "AI participated in the testing lifecycle." If the auth module changed, you should not ask: Can someone test login? You should be able to do this: testboat case list --module auth testboat matrix show Then rerun the affected tests and record results: testboat result record TC-001 pass --type automated --by "AI" If a failure appears: testboat bug add \ --title "Wrong password returns 500 instead of 401" \ --tc TC-001 \ --severity major \ --priority P1 And after the fix, the bug should not jump straight to "closed." It should move through retest: php fixed - pending-retest - verified - closed That is the loop teams need when development is moving faster. AI is making code cheaper to produce. That does not automatically make releases safer. If anything, it makes weak testing systems more visible. The next layer of AI engineering is not just faster code generation. It is turning the surrounding engineering practices into systems that agents can participate in. Testing is one of those practices. That is why I built testboat . Not to generate more test cases. To make testing traceable, reviewable, versioned, validated, and reportable. pip install testboat testboat init testboat enable cursor testboat strategy create Project: https://github.com/lijma/testboat https://github.com/lijma/testboat Docs: https://lijma.github.io/testboat/ https://lijma.github.io/testboat/ How does your team know a release is actually ready? Is that answer stored in a system, or mostly in people's heads?