Tech Horror Stories
True stories from the trenches.
I Fixed a Typo in a Comment and the Entire Build Pipeline Broke Because CI Parses Comments as Config
Fixed a typo in a comment. The build broke. CI parses comments for config. The typo was the actual deployment instruction.
The Production SSH Key Was in a Public GitHub Repo for 4 Years and Nobody Noticed
Production SSH key in a public repo since 2022. 4,000 clones. Zero malicious logins. Hackers probably felt bad for us.
We Migrated From MySQL to Postgres to SQLite to MongoDB to a Google Sheet in 18 Months
MySQL to Postgres to SQLite to Mongo to Sheets. Each migration took 6 months. We ended up back where we started.
Our Entire Monitoring Stack Crashed Because It Generated Too Many Alerts About Itself
Monitoring generated 47,000 alerts about its own health. We missed the alert about the actual database being on fire.
I Pushed a Hotfix Using My Phone While at a Wedding and Now the Bride Is Not Speaking to Me
Hotfix deployed from a wedding. Typed with one hand holding champagne. The bride is still furious. The hotfix worked.
The Firewall Rule Was Just an HTML Comment That Says Please Do Not Hack Us
The firewall config was just HTML comments saying please do not. Our security audit found this in 47 milliseconds.
I Accidentally DoSed Our Own API with an Infinite Retry Loop and Called It Resilience Engineering
Retry loop spawned new retries. 14,000 requests per second from our own API. But auto-scaling passed the stress test.
Our Production Database Runs on a Raspberry Pi in a Bathroom and Nobody Knows
Raspberry Pi runs 47,000 queries per day from the bathroom. Boss thinks it is a Postgres cluster. For 2 years.
I Replaced Our Entire Test Suite with console assert and Nobody Noticed for 6 Months
Deleted the test framework. Replaced with console.assert in production. QA team laid off. Quality is unchanged.
The CDN Cached a 500 Error Page for 24 Hours and Our Site Was Just Broken for Everyone
CDN cached a 500 error for 24 hours. Users thought we went out of business. We were just deploying on a Friday.