The Payment System Was Rounding Cents to the Nearest Dollar and Pocketing the Difference Since 2019
The payment system rounded $2.99 to $3.00 and kept the cent. Since 2019. We owe users approximately $47,000.
Our Rate Limiter Limited Legitimate Users While a Botnet Brute Forced Passwords for 6 Months
Botnet brute-forced 47,000 accounts. The rate limiter was blocking users who typed too fast in the search bar.
The Intern Deployed to Production on Day One and It Went Better Than Our Usual Deploys
The intern's first deploy was flawless. Our senior engineer deployed later that day and took down prod for 3 hours.
I Ran rm rf on Production and the Only Backup Was a Screenshot
rm -rf / on a Friday. The only backup was a Slack screenshot of the old schema. We rebuilt everything from memory.
The Backup System Was Backing Up the Backup Logs Not the Actual Data
We backed up 14TB of log files. The actual database was not included. Nobody checked the backup script until today.
Our SSL Certificate Expired and Nobody Knew Because We Pinned the Old One
SSL cert expired in 2024. We pinned it in 2023. Users see a warning every time. We told them to just click proceed.
I Found a Comment That Said Do Not Touch This and I Touched It
I touched it. The build broke. The senior dev sighed and said yeah that happens. He did not elaborate.
The Database Migration That Never Ended: A 47 Hour Saga
47 hours of database migration. 14 rollbacks. The migration script is now longer than the application itself.
Our Legacy System Runs on a Mainframe That Requires a Blood Sacrifice to Reboot
The mainframe requires a ritual reboot. Only one retiree knows the sequence. He lives in a cabin without phone service.
I Inherited a Codebase Where Every Function Is Called doTheThing
Every function is named doTheThing. There are 47 versions. Nobody knows which one actually does the thing.