I Don't Trust a System That Grades Its Own Homework

I’ve watched this movie before. Not this exact one — but I’ve been in enough rooms where a system announced it was fine while it quietly wasn’t, and I’ve learned what that silence costs.

Here’s a habit I have that I’m not proud of.

My business runs on a system I built. It writes, it schedules, it publishes, it routes. That’s the entire point of it — I built it so I would stop being the person doing those things at eleven at night. And it works. It has worked for months.

And still, every time it tells me something went out, I go look.

Not a spot check. Not a weekly audit. Every time. I open the page, with my own eyes, and confirm the thing exists in the world. A system I designed, that I trust enough to run my company on, and I am still standing behind it with a clipboard like a man who doesn’t quite believe the elevator is going to stop at his floor.

I know exactly why I do it. One day the system told me a post had published. The confirmation was clean. The timestamp was there. I went to look anyway, for no particular reason, and the post wasn’t there. It had never been there.

Nothing burned down. No customer noticed. Nobody sent me an angry email. What it cost me was smaller than that, and worse: I stopped being able to believe the thing I’d built. The reports kept arriving. I just couldn’t use them anymore. Every green checkmark after that was a claim, not a fact, and I had no way to tell the difference without getting up and looking.

That’s the part nobody warns you about. AI doesn’t usually fail by exploding. It fails by telling you it succeeded. And once you’ve caught it doing that even once, you can’t un-catch it. You are now the verification layer, permanently, doing by hand the exact work you bought the machine to take off your plate.

The first wave that talks back

I’ve ridden a few of these waves now. Dot-com. Cloud. Mobile. Data. Each one arrived with the same promise and the same pattern: an enormous amount of real capability, wrapped in an enormous amount of nonsense, and a gap in the middle where the people actually building things had to figure out on their own which was which.

This wave is bigger and faster than any of them. It’s also the first one where the tools are confident. A bad database migration didn’t tell you it went well. A broken deploy didn’t send you a summary of its own success. This generation does. It reports, it summarizes, it reassures, and it does all of that whether or not anything happened.

That’s a genuinely new problem, and most of the advice being sold right now doesn’t touch it. The courses teach you to prompt. The platforms teach you to build. Almost nobody is teaching the thing that actually determines whether any of it survives contact with a real business: how do you know it worked?

Building the check into the system

So I did the only thing I know how to do, which is build the answer and then live inside it.

The system that runs this company has verification wired into it. Not as a feature — as the premise. Nothing counts as done because it said it was done. Things count as done when something other than the thing itself confirms it. That sounds obvious written down. It is not how most AI workflows are built, including most of mine, for longer than I’d like to admit.

The habit is still there. I still go look. But now the system looks too, and it looks first, and it tells me when it finds a gap instead of waiting for me to stumble into one. Which means the checking I do by hand is slowly becoming redundant — and the day I catch myself not doing it anymore will be the day I know it’s finished.

Why this is public

That’s what this journal is.

I run multiple businesses on this system. It is not a demo, it is not a case study I’m writing about someone else, and it is not finished. I’m going to document what happens as I run it: what breaks, what the fix was, what I got wrong, what I’d tell you to do differently. In public, while it’s happening, not cleaned up afterward into a tidy lesson.

Some of it will be unflattering. That’s most of the value. Every operator I talk to is quietly convinced they’re the only one whose AI setup is held together with tape and hope, and that belief is doing more damage than the tape.

You’re not behind. The tools really are like this. What’s missing isn’t your discipline — it’s a layer nobody sold you, because it’s not exciting and it doesn’t demo well.

I’ll be writing about that layer for a while.

If you want the ground floor of it — what’s actually worth learning right now and what’s noise — the AI Playbook is free and it’s where I’d start. No pitch attached to it. It’s the thing I wish someone had handed me before I learned all of this the expensive way.

The wave comes in. And the wave keeps coming.