VMware press released their benchmark suite today. The new benchmark, cunningly named VMmark;, is the first publicly released x86 multi-workload system performance test. Briefly, the suite tests server performance by running six common workloads simultaneously – including email (Exchange), a java server, a standby server, an Apache web server running on Linux, an Oracle database, and a file server. More details on workloads, methodology, and metrics are here…

Sun and Dell have already published VMmark results (Sun results, Dell results), and other x86 vendors are sure to follow. At first glance, this all seems like a pretty cool thing – and something for everyone to like: system vendors get a new number they can use to beat each other over the head with, guys like me get something new to talk about, VMware gets the chance to set the terms of battle in virtualization performance evaluation, and customers get to be on the receiving end of a virtual (and literal) blizzard of PowerPoint slides and press releases touting VMmark scores. While we are generally in favor of stuff that gives customers more data points about the relative performance of the systems, we\’re a little concerned that VMmark may be the second coming of TPC-C…a very long in the tooth OLTP benchmark that bears almost no relevance to real-world system performance and is increasingly less useful for system comparisons. Over time, vendors (system, o/s, database) have gotten so good at tuning for TPC-C workloads, and actually designing systems to hit TPC-C benchmarks, that the winning TPC-C configs and associated throughput is as different from real-world results as drag racers are from family sedans. We would hate to see this happen with VMmark. Towards that end, VMware has some pretty strict configuration and reporting requirements – a good thing, but some of the requirements may be a little too restrictive, particularly those that may discourage the development and use of other virtualization benchmarks.