The most expensive software bug in history...
Knight Capital’s 2012 collapse shows how a reused feature flag, inconsistent server deployment, and a mistaken rollback turned one dormant trading function into a catastrophic software bug that executed millions of bad trades in 45 minutes, costing the firm $440 million and leading to its sale.
MAIN POINTS FROM TRANSCRIPT
- Knight Capital was a major Wall Street market maker processing about $20 billion in trades daily.
- Engineers reused an old feature flag tied to a dormant “Power Peg” test function instead of creating a new one.
- A manual deployment left one of eight servers unpatched, so the flag activated different code across machines.
- Panic rollback spread the bad code everywhere, causing millions of trades and massive financial losses.
TAKEAWAYS
- Old code and unused flags can become dangerous when repurposed without careful cleanup.
- Manual, inconsistent deployments create severe risk in systems that require exact uniformity.
- Fast detection matters, but bad assumptions during incident response can make failures much worse.
- Financial software bugs can have enormous real-world consequences in minutes, not hours or days.