On August 1st 2012, Knight Capital Group lost $440 million in 45 minutes of high-frequency trading.

The deployment of a software update to their trading platform was faulty. Their subsequent efforts to manually rollback the deployment were catastrophically fatal. There’s a detailed Securities and Exchange Commission filing that describes in lurid technical detail the full scale of the disaster. Triggering an investigation by the SEC is a great way to get your system deeply audited. Continue reading