The $440M Software Test Disaster: What Happened?

I recently wrote about how a bad round of software testing cost Wall Street trading firm Knight Capital an estimated $440 million – enough to almost put the company out of business. In that article, I speculated that Knight might be bailed out if the SEC allowed them to unwind the trades, basically taking a mulligan on the 45-minute debacle. Turns out that ain’t gonna happen. (Insert mea culpa here: I also mistakenly reversed the terms ‘bid’ and ‘ask’ in that story. Which is shameful, given that I have both graduate and undergraduate degrees in finance. I am a constant disappointment to myself and everyone around me. That said, on with the story.)

Knight was left squirming on the hook by US regulators and subsequently forced to find money to cover the losses from third parties. In return for floating Knight $400 million, six other Wall Street firms will be paid a 2% preferred stock dividend and, if they like, be able to convert those preferred shares into enough common stock to own 75% of the company – a pretty sweet deal for a company that was a solid market player before last Wednesday.

Tyler Durden’s blog posts at Zerohedge have been both fast and solid on this story. Here he posts some of the highlights from an interview with Knight CEO Tom Joyce. One of the quotes from Joyce: “We have to do a better job on our testing environment.” Yeah, I think I’d make that a priority, or at least move it farther up the list. It’s an understatement of such magnitude that I’m at a loss to come up with an apt comparison. Maybe if Napoleon had said, “We need to do a better job of scouting out our enemies” after Waterloo? But I’m drawing a blank right now.

So what happened IT-wise with Knight Capital?

Read More

Bad Algorithm Burns $440 Million; Worst Software Testing Day EVER

Knight Capital, a firm that specializes in executing trades for retail brokers, took $440 million in cash losses Wednesday due to a faulty test of new trading software. In headlines it’s being called a trading “glitch,” which isn’t nearly as accurate as the term I’d use: “f**king disaster.” The broad outline of the story is here and more colorful, bloody details are here.

Briefly, here’s what happened: Knight Capital’s worst day in IT started Wednesday morning with a test run of their new trading software. An old pal of mine who’s following the story closely (and is also deep in both IT and trading) told me that the company set up the software to work with only a few stocks. They also set the buy/sell points well outside where the markets were currently trading to ensure that nothing would actually execute.

But somehow – and this will probably the be the subject of several lawsuits, books, and maybe even a Broadway musical – the software didn’t behave as expected. It went out and did what it was designed to do: execute lots and lots of trades very, very quickly.

Read More

Radio Free HPC, Episode 3: Is LINPACK Obsolete?

A conversation with Jack Dongarra (of Top500 list fame) at ISC’12 is the inspiration for a spirited discussion of whether the venerable LINPACK benchmark has outlived its usefulness. Dan accuses Henry of cursing the darkness; then they talk about what…

Read More

Radio Free HPC, Episode 2: When Technology Lies

What happens when technology lies to you? Dan rants about discovering that his exercise machine has been lying to him for more than a year now, but the episode (thankfully) evolves into a discussion of benchmarks and benchmarking. Bonus: one…

Read More

Radio Free HPC, Episode 1: Data Integrity

Join Rich Brueckner of insideHPC, Henry Newman of Instrumental, Inc. and our own Dan Olds for this series of podcasts about all things HPC. In this first episode, the guys talk about Data Integrity and the menace of Silent Data…

Read More

ISC’12: Tsinghua, NUDT Triumph in Kluster Kamph

The 2012 ISC Student Cluster Challenge ended last week, and it’s high time I write about the official announcement of the awards and analyze some of the results.

Each of the five teams put in months of work designing its cluster and learning how to optimize it to run the various benchmarks and application codes featured in the competition. They also had to stay on their toes, since the organizers were adding a secret ‘surprise’ application to the slate of apps to be run on Tuesday and Wednesday.

The award winners were announced at the closing ceremony on Wednesday afternoon. Gilad Shainer of the HPC Advisory Council, and overall organizer of the competition, acted as emcee. Short but inspirational addresses were given by ISC’s Thomas Meuer and Gerd Buettner from chief sponsor Airbus. The students were on the edges of their chairs as they waited out the program formalities in order to learn the results. ‘Read more’ to see results and team photos.

Read More

ISC’12: Last Day: Stony Brook Struggles, but Overcomes

Stony Brook University (profile here) has had their work cut out for them at the 2012 ISC Student Cluster Competition. From the very beginning, they had software problems with their cluster, requiring them to concentrate on troubleshooting and repairing while…

Read More

ISC’12: Last Day: Will GPUs Put NUDT Over the Top?

China’s National University of Defense Technology (NUDT, profile here) is staying true to its heritage by being the only team to use a hybrid CPU/GPU system at the ISC 2012 Student Cluster Competition in Hamburg. NUDT, like Team Tsinghua, had…

Read More

ISC’12: Last Day: Tsinghua Looks for Win

The Tsinghua University team (profile here) didn’t take the easy way to the ISC 2012 Student Cluster Competition in Hamburg. They had to fight their way through five other universities, all competing to carry the Chinese flag, in an intra-country…

Read More