Why do some VM's obtain astronomical scores on CaffeineMark 2.5?Some Java Virtual Machines (VM's) have begun to post extremely high scores on the the CaffeineMark 2.5 benchmark. In some cases, individual scores were as high as 1 million! The reason for these high scores is that the Loop and Method tests were too optimizable. The Just-In-Time compilers realize that the calculations performed in these tests are not used, so the VM's don't execute the computations. While the high scores are an indicator of superior performance, the benchmark is misleading in these cases because real world performance is not 10,000 times better than it was a year ago. The optimizability problems of CaffeineMark 2.5 have been fixed in the new CaffeineMark 3.0. Also, the overall score is calculated using a geometric mean instead of a weighted average. This results in performance numbers that correlate with typical user experience. |