I Let AI Write 300 Tests in a Weekend. Here's What Actually Happened.

title: "" published: false tags: 

I want to tell you the honest version of this, because the internet is full of the other version. You know the one: someone points an AI tool at their codebase, tests materialize, coverage hits ninety percent, and a triumphant screenshot follows. That is a real thing that can happen. It is also about a third of the actual story, and the missing two thirds are where all the useful lessons live.

So here is my weekend of letting AI generate three hundred tests, including the parts that do not screenshot well.

Friday night: the honeymoon

The first few hours were genuinely astonishing. I fed a service's API schema into the tooling and watched it produce dozens of test cases faster than I could read them. Happy paths, malformed inputs, boundary values, missing fields. Stuff that would have taken me a full day of tedious typing appeared in minutes, and a lot of it was good. Not all of it, but a lot.

If I had stopped Friday night and posted a screenshot, this would be one of those triumphant threads. The coverage number really did leap. The tests really did run. It really did feel like magic.

I have since learned that the coverage number leaping is the least trustworthy moment in the whole process.

Saturday: reading what it actually wrote

Saturday was when I made myself read every generated test instead of trusting the green checkmarks, and the mood shifted.

Maybe a fifth of them were quietly asserting the wrong thing. Not broken, which would have been obvious. Confidently wrong. The model had looked at the current behavior of an endpoint and written a test asserting that this behavior was correct, when in a couple of cases the current behavior was itself the bug. A test that certifies a defect is worse than no test, because it stands guard over the mistake and fails you the moment someone tries to fix it.

Another chunk were technically valid but useless. Three slightly different tests all checking the same happy path, none touching the concurrency case that actually scared me. The generation had pooled where generation is easy, which is exactly where I needed it least.

I spent Saturday surveying what was out there to see whether this was a tool problem or a me problem. Working through the current best ai testing tools made it clear the good ones are converging on the same honest posture: they draft, they do not decide. The output is a proposal for a human to accept, edit, or reject, and the tools that pretend otherwise are the ones setting you up for a confidently-wrong Saturday of your own.

Sunday: the workflow that actually worked

By Sunday I had a process that felt right, and it was less "AI writes my tests" and more "AI writes a very fast first draft that I aggressively edit."

The reframe that unlocked it was understanding what these systems are structurally good at. The current wave of generative ai testing tools is excellent at volume and terrible at judgment, and once you stop expecting judgment from them, they become genuinely great. I let the model handle everything mechanical: the permutations, the test data, the boilerplate, the fifteen variations of an input I would have gotten bored writing by hand. Then I kept, for myself, the two things it cannot do. Deciding what correct actually means for our business. And ranking which risks deserve the deep coverage.

Concretely: generate wide, then curate hard. Let it draft two hundred cases. Delete eighty. Fix the assertions on forty. Hand-write the dozen that guard the genuinely dangerous paths, because those are exactly the ones the model underweights. What is left is a suite far larger and far better than I would have produced alone in the time, but only because a human stood between the generation and the commit.

What I would tell Friday-night me

The technology is real and I am not going back. Sunday-me ships more tests, of higher quality, than any previous version of me. But the framing that the internet sold me was backwards.

AI did not replace the thinking. It replaced the typing, which turns out to have never been the hard part. The hard part was always deciding what deserves to be tested and what correct means, and the moment you outsource that to the model, you get three hundred tests that certify whatever the code currently does, bugs included, with total confidence.

Generate wide. Curate hard. Never let a machine define correctness for the paths that can hurt you. Do that, and a weekend of AI-generated tests is one of the best trades in modern software. Skip the curation, and you have built an elaborate machine for defending your own bugs.

Have you had the "confidently wrong generated test" experience yet? I am curious whether mine was a fluke or a rite of passage.