Benchmarks in Java (Know what u're measuring)

Accurate Benchmark

It's often needed to know whether the certain operation is faster than another one(especially to argue with somebody on code review:)).

Probably the most of the fast self-written benchmark tests are incorrect because of side factors, that were not considered.

Here are few points we have to consider before start doing better:

• JVM warmup at start an execution is slow and becomes faster and faster until it goes to steady-state;
• Just In Time Compiler optimizes the code, so your measures become wrong;
• Garbage Collector GC could act during your measure so it'll affect the results;
• Class loading First execution time usually includes the loading of all classes it uses, cause JVM typically loads classes only when they're first used;
• False Sharing Might happen when u try to read 2 adjacent fields from 2 or more threads at the same time (those could contend with each other on the hardware level)
• Caching CPU cache, File system cache, Memory access optimizations.

There is a fairly simple and useful Code Tool by Open JDK called JMH supposed to simplify & optimize the benchmarks writing & running (references are in the end of page). ---- ## Elapsed Time Measure Basics The most of the benchmarks are based on the time period between the execution start & end: long startTime = System.currentTimeMillis(); task.run(); long elapsedTime = System.currentTimeMillis() - startTime; System.out.println("Execution time: " + elapsedTime + " milliseconds."); Or a bit better one is with average among multiple executions: long startTime = 0; long endTime = 0; long elapsedTotal = 0; for (var i = 0; i < repetitions; i++) { startTime = System.currentTimeMillis(); task.run(); long endTime = System.currentTimeMillis(); elapsedTotal += (endTime - startTime); } long average = elapsedTotal / repetitions; System.out.println("Average execution time across " + repetitions + ": " + (elapsedTotal / repetitions)); But we already know, that the results above are not accurate at all in java. ---- ## Code Optimization Pitfalls ### Dead Code **Always return result of your calculations *(avoid `void`)* ** Dead *(unused)* code will be eliminated, so the results would be wrong. public void wrongTest(int a, int b) { a + b; } public int rightTest(int a, int b) { return a + b; } ### Constant folding **Always read the test input from a state object.** JIT would optimize your result if it's predictable and does not depend on objects state. So, the simplest way to prevent it is to separate your constants to the instance fields. public void wrongTest() { list.add(1000); } public boolean rightTest(int thousand) { return list.add(thousand); } ### Loops optimization **Try to avoid the loops in your micro-benchmarks at all.** JIT is smart and might optimize the loops in different ways. Anyway, if you've decided to use the loop, ensure that the pitfalls above are avoided. for (var i = 0; i < repetitions; i++) { count++; } Watch-out `++` is the same like `+=1` where one is a constant. for (var i = 0; i < repetitions; i++) { count += one; } ### Code deoptimization **Do not mix up benchmarks within the same JVM run.** If JVM speculates that `run()` always dispatches to the same target class, and it can generate very efficient native code. But this assumption might become invalidated with the 2nd benchmark which introduces another implementation of `java.lang.Runnable`. So, that will force JVM to de-optimize previously generated code & slow down the execution time. ---- ## Base Line Resources allocation, initialization & access also consumes the execution time. **If your test method contains some statements that are not related to the actual logic that u're trying to test, measure those execution time separately and rely on the diff.** For example in the code sample from above in Time Measurement Basics, the `Calling System.currentTimeMillis()` has an effect on throughput as well, and our benchmarks needs to accurately subtract the execution time taken for each of these calls. So, our baseline test method for that case would be following: public long testBaseline(){ long resultTime = System.currentTimeMillis(); return resultTime; } ... System.out.println("Execution time: " + (taskElapsed - baselineElapsed*2) + " milliseconds."); ---- ## Conclusions He're a few points that'll improve our benchmark accuracy: * Always read the test input from a dynamic variables. * Always return result in your tested methods. * Avoid Loops in micro-benchmarks. * Execute task enough times to ensure that it's in steady-state. * Separate logic not related to the certain tested operation under the baseline. * Use nanoseconds in micro-benchmarks. * Mixing benchmarks within the same JVM run is wrong. * Ensure that compiler & gc are not doing unexpected things *(always run with `-XX:+PrintCompilation` & `-verbose:gc`)*. * Be aware of initialization, optimization & de-optimization effects. **Always know what exactly you want to measure.** Most of the time your micro-benchmarks won't reflect your production code. If you want to make an accurate assumption according to the certain part of your production code, you should test those part exactly as it's. Because, in the real world the jit, gc, etc. will do their actual work, so in this case you can't really rely on the measurements whereas those were disabled. ---- ## see Also * [Open JDK - JMH Code Tool](http://openjdk.java.net/projects/code-tools/jmh/) * [Open JDK - JMH Samples](http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/) * [Stackoverflow - Correct microbenchmark](http://stackoverflow.com/questions/504103/how-do-i-write-a-correct-micro-benchmark-in-java) * [Writing a Java Micro-benchmark](http://www.kdgregory.com/index.php?page=java.microBenchmark) * [Oracle - Avoiding Benchmarking Pitfalls on the JVM](http://www.oracle.com/technetwork/articles/java/architect-benchmarking-2266277.html) * [Introduction to JMH](http://java-performance.info/jmh/) * [Robust Java Bench-marking](http://www.ibm.com/developerworks/java/library/j-benchmark1/index.html) * [JavaScript execution & memory control](http://bloomlab.blogspot.com.ee/2015/11/best-practice-executions-memory-control.html)