Introduction

After a long 12-hour workday, we deployed the changes during a peer-programming session, and to our delight, everything was running smoothly. We watched as the scripts executed flawlessly, but then a fatal error message popped up on our screen: “CRITICAL: Config not found”.

We immediately dived into debugging, and soon discovered that our config database was empty. How could this have happened? Well, prior to deployment, I had the bright idea of cleaning up our old databases by dropping them. In hindsight, that decision proved to be a grave mistake.

Our conversation went something like this:

Me: “I can drop the old databases, right? Nothing should happen.”

Colleague: “Yeah, but does it auto-create them?”

Me: “Of course. What could possibly go wrong?”

Colleague: “What could possibly go wrong?”

In a moment of overconfidence, we had confirmed that nothing could go wrong. Unfortunately, we only ran database backups once a day, and we made the significant changes that day. As a result, we had to rewrite, re-modify, and re-upload all of our configs from old backups.

Lessons learned: never underestimate the power of Murphy’s Law, and always have a backup plan in place