Tech Horror Stories
True stories from the trenches.
Our Monorepo Has 47 package json Files and Not a Single One Is Up to Date
47 package.json files. 14 versions of React. One package depends on left-pad. In the year 2026.
The Production Bug Only Happens When the CEO Is Looking at the Dashboard
The bug only manifests when the CEO opens the dashboard. We think it is his browser extensions. Too afraid to ask.
Our Redis Instance Ran Out of Memory and Started Evicting Auth Tokens of Logged In Users
Redis OOM killed auth tokens. Every logged-in user was suddenly logged out. Support got 14,000 tickets in 10 minutes.
I Disabled CORS for Debugging and Forgot to Re enable It for 8 Months
CORS was disabled for a quick local test. The commit message said TEMPORARY. That was 8 months ago.
The Payment System Was Rounding Cents to the Nearest Dollar and Pocketing the Difference Since 2019
The payment system rounded $2.99 to $3.00 and kept the cent. Since 2019. We owe users approximately $47,000.
Our Rate Limiter Limited Legitimate Users While a Botnet Brute Forced Passwords for 6 Months
Botnet brute-forced 47,000 accounts. The rate limiter was blocking users who typed too fast in the search bar.
The Intern Deployed to Production on Day One and It Went Better Than Our Usual Deploys
The intern's first deploy was flawless. Our senior engineer deployed later that day and took down prod for 3 hours.
I Ran rm rf on Production and the Only Backup Was a Screenshot
rm -rf / on a Friday. The only backup was a Slack screenshot of the old schema. We rebuilt everything from memory.
The Backup System Was Backing Up the Backup Logs Not the Actual Data
We backed up 14TB of log files. The actual database was not included. Nobody checked the backup script until today.
Our SSL Certificate Expired and Nobody Knew Because We Pinned the Old One
SSL cert expired in 2024. We pinned it in 2023. Users see a warning every time. We told them to just click proceed.