So what’s the best way to make important decisions? Hunches? Intuition? Magic 8 Ball? Or relying on data analysis to reveal the best choice? Ok, it’s kind of a stupid question; most anyone answering it seriously will say, “Yeah, of course you want to use data to figure out what to do, you dumbass.” (Not that everyone would call me a dumbass in their response  –  just the people who know me personally).

While everyone says that using data to make decisions is the only way to manage, I’d submit that very few organizations actually do it. I mostly see firms relying on hunches and intuition, and then using data (if forced to; they often don’t) to justify, adjust, and calibrate the course of action that they’ve already decided to pursue.

The problem is that humans have a number of innate cognitive traits that don’t let us easily choose non-intuitive alternatives – even when our intuition is demonstrably wrong. A couple of Deloitte consultants wrote an interesting article that talks about this unfortunate fact of life in a conversational way, via real-world examples. They also discuss how data-led decision making pays off.

What I (and a bunch of smarter, more articulate, and more successful people) see happening is real, data-led decision making coming to enterprises large and small. This trend will impact IT shops in that they will see rapidly increasing demand from the business side for more data, more compute capacity, and results delivered in near-real time.

When you look at the workloads and requirements, they look a lot like typical HPC stuff. To me, this means that the trend of HPC technology crossing over into the business world is going to accelerate even faster. While we (sadly) can’t make ourselves smarter, maybe using vast amounts of data can stop us from doing so many outright stupid things. It’s something to hope for, at least.