The month I lost a day of customer data at $40K MRR: a data-loss diary (2026)

I crossed $40K MRR on a Thursday. I remember because I took a screenshot of the dashboard, sent it to my partner, and let myself feel, for about an hour, like the hard part was behind me.

The next night I deleted a day of my customers' data. Not all of it. Just one day. It turned out one day was more than enough to lose my biggest account and teach me the single most expensive lesson of the whole journey: a backup you have never restored is not a backup. It is a hope.

This is a composite diary. The MRR figures are self-reported and lightly rounded, and the founder is a blend of a few people I have interviewed who lived through the same kind of night. The sequence is exact, because it happens more than anyone admits.

The four-minute migration

The feature was small. A new field on the records our customers create every day, plus a backfill to populate it for existing rows. I had run the script against staging twice. Four minutes, clean, no errors.

At around 1am, tired and a little proud, I opened a terminal to run it against production. What I did not notice was that the shell still had staging's habits and production's connection string. The backfill had a bug I had never hit on staging's tiny dataset: under a certain condition it rewrote the wrong column, and for a large slice of rows it overwrote real customer content with an empty value.

The script finished in under a minute. It reported success. I closed the laptop.

I learned what had actually happened at 8am, from a support email with a subject line I still think about: "Where did our stuff go?"

When I reached for the backup

Here is the part I had rehearsed in my head a hundred times and never once in real life.

I had backups. Of course I had backups. Nightly database dumps, running on a schedule, quietly, for over a year. I had watched them exist. I had never watched one come back.

When I went to restore, two things fell out of the sky at once. The most recent nightly dump had completed at roughly 3am the previous night, about 22 hours before the incident. And a newer, more frequent backup job I thought I had set up months earlier had been failing silently since a credential rotation. No alert. No red dot. Just a job that returned early and logged nothing anyone read.

So the freshest clean copy of my customers' data was almost a full day old.

The backups were not the problem. The problem was that I had never once tried to bring one back, so I did not know what "restore" actually cost until the meter was running.

I was living, at small scale, the exact failure that the GitLab January 31, 2017 database incident made famous. In that 2017 outage an engineer accidentally removed the primary database directory, and when the team went to recover they discovered that of their several backup and replication methods, none were reliably working. They restored from a manual snapshot taken roughly six hours earlier and permanently lost the data in between, affecting thousands of projects and comments. Five ways to recover, and the one that saved them was a copy someone happened to take by hand. I had read that post-mortem years before. I had nodded along. I had changed nothing.

The 22 hours I could not get back

I made the call at 9am: restore from the 3am-the-night-before dump, accept the loss, and stop the bleeding. The alternative was trying to reconstruct clobbered rows by hand from application logs, which would have taken days and still been wrong.

The math of it is brutal and simple. Every record my customers had created in that roughly 22-hour window was gone. Not corrupted, not degraded. Gone, because the clean copy predated it.

There is a term for the number I had never bothered to define: RPO, or recovery point objective. It is the honest answer to "how much data can this business afford to lose?" I had never answered it, so the universe answered it for me: about a day. If I had been asked out loud beforehand, I would have said "an hour, absolutely no more." The gap between the answer I would have given and the answer I actually had is the whole story.

The email I did not want to send

I sent one email to every affected account, from my own address, that morning. No legal softening. It said what broke, that it was my mistake and not theirs, exactly which window of data was lost, what I had already done, and what I could not undo. I offered the month free and, where it helped, a call.

Most people were kind. Software people especially. Several replied with some version of "we have all been there," which I did not deserve and was grateful for anyway.

One customer did not reply at all.

The customer I lost

My largest account was a small agency paying about $1,600 per month, a little over four percent of my $40K MRR and by far my biggest single logo. They used my product to hold work they did for their own clients. The data lost in that 22-hour window included a day of that work, work they had already billed, work they now had to redo and explain.

They cancelled the following week. No anger, just a short note that they needed something they could fully trust with client data, and that they understood. That sentence, "we understand," was worse than a complaint would have been.

Two smaller accounts followed them out the door over the next month, spooked by the incident email. MRR slid from $40K to about $37K. It took me almost three months to climb back over $40K, and I never did get that agency back.

What I changed, in the order it should have been done the first time

I did not add heroics. I added the boring things I had skipped.

First, I made restores routine instead of theoretical. Once a month I now spin up a throwaway database, restore the latest backup into it, and run a script that checks the data is actually there. The first drill took an afternoon and surfaced two more silent problems. An untested backup is a guess, and I had been guessing for a year.

Second, I shortened how much I could ever lose. I moved the critical data onto managed storage with continuous point-in-time recovery. On systems like Amazon RDS point-in-time recovery, transaction logs are shipped every few minutes and you restore into a brand new instance at almost any second inside the retention window (AWS docs, accessed 2026). The same incident on that setup costs you minutes of data, not a day. My RPO went from "about 24 hours, discovered live" to "about 5 minutes, on purpose."

Third, I put guardrails between me and production. Destructive and backfill scripts now run inside a transaction I have to consciously commit, require a typed confirmation of the environment name, and are never run from a shell that can silently point at prod. The 1am version of me could not be trusted, so I removed the sharp edges he could reach.

Fourth, I made failure loud. Every backup job now pages me if it does not finish, and a green check only appears after a restore has been verified, not after a dump has been written.

If you take one thing from this

Do not audit your backups. Restore one. This week, before you need to, spin up a scratch database, pull your newest backup into it, and confirm your real data is actually sitting there. You will learn more in that one hour than in a year of watching backup jobs report success. The founders who survive their worst night are not the ones with the most backups. They are the ones who found out, on a calm Tuesday of their own choosing, exactly what a restore costs.

I found out at 8am on the worst morning of my company, from a customer, in the form of a question I could not answer. Pick the Tuesday instead.

Keep reading

• The month I migrated off Firebase at $26K MRR: the other time infrastructure decided the shape of my month.
• The month my biggest integration partner 10x'd its API price at $46K MRR: what single-vendor dependency costs when the ground moves under you.