The Day Everything Stopped

The message was polite. A customer on the platform wanted to know why their notifications had stopped. Not angry — curious. They’d been getting a steady stream of updates about their orders, and then, at some point they couldn’t quite place, the stream had ended.

I went to check the system. It said it was fine.

What I’d built

Here’s what the system was. A scheduled service I built that sent order-status emails — daily, weekly, or monthly, depending on what each customer had asked for. Thousands of customers on the platform.

It ran on a timer: pull the list of notifications that were due, look up each customer’s details from another service, build the email, send it, mark it done. Simple on paper.

And it had, technically, kept running. Every day it woke up, did its rounds, wrote completed in the log, and went back to sleep. What it hadn’t done, for weeks, was send that customer anything.

The reason was small. When the service it depended on came back empty for a given customer — no data — my code did exactly what I’d told it to do: it wrote one line to a log and moved on to the next one. No error, because nothing had errored. No alert, because I hadn’t written one. I’d built the system to handle two states, working and broken, and this was neither. It was working, and producing nothing.

Nobody was reading that log. Why would they? The system said it was fine.

Twice more

I’d like to tell you I fixed it and that was that. It happened twice more, on the same system.

The second time, a data-consolidation job mangled the field that held who each email should go to. Where there should have been an address, there was a garbled structure. The sender looked at it, found nothing it could send to, and — you can guess — skipped it. On time. Every scheduled run. Reporting success.

That one lasted days, and I found it only because I was in the database looking for something else entirely and noticed the addresses didn’t look like addresses.

The third time, a single bad record — one value in the wrong shape, where a list was expected — broke the code that read the schedule, and every notification in the system stopped. All of them. A customer told me.

Three failures. Zero alerts. The system reported success on every run, and not once did it tell me. A customer found two of them. I stumbled on the third.

It was working, and producing nothing.

What I finally understood

Here’s what I finally understood, and I wish it hadn’t taken three rounds.

“Nothing to send” and “can’t reach the source” are different events. One is a Tuesday. The other is a fire. I had built a system that couldn’t tell them apart — it treated no data and no way to get data as the same shrug — and then I trusted its report on how things were going.

That’s the trap, and it isn’t specific to my system or my industry. Most automation is built for the happy path and the crash. The thing in between — running fine, producing nothing, reporting success — isn’t a state anybody designs for, because it doesn’t feel like failure. It feels like a quiet day.

So I changed two things.

Your automation has a version of this

Your automation has a version of this. I’d bet on it. Somewhere in your business there is a process that runs on a schedule, reports that it ran, and has never once been asked whether it actually did anything.

If the answer you get back is “it says it’s fine,” you don’t have monitoring. You have a system that grades its own homework.

If I can’t write that query, the feature isn’t ready for production. It’s ready for a customer to find.

I’ve put together a free, self-paced course for anyone running a business on AI and automation tools — the AI Playbook. It ends exactly where this story does: with proof that your system actually works.

The wave comes in. And the wave keeps coming.