On May 15, 2015, at 1:50 PM, Marshall Abrams <[log in to unmask]> wrote:

> If I want to compare the speed of different versions of the same MASON simulation (Java version vs. Clojure rewrite, Clojure vs. Clojure with optimizations, etc.), is it reasonable to simply compare the "Rate:" outputs in a few runs of the simulations?

People mostly are about wall-clock time in this context.  So I'd do this:

1. Do not run with the GUI.  Just from the comman line.
2. Run for a long time, so the JIT compilation is optimized out.
3. Run with a fairly large memory footprint.

There are some other options you can tweak in the JVM but that'd be a good start.  You're not going to improve things by more than a factor of 2 anyway, and the Java and Clojure versions are probably further than that apart, at least initially, so it'll give you a good estimate.

> There are libraries that will run something multiple times, each time with sufficient burn-in for the JIT compiler, and average over the results.  In Clojure, Criterium is the standard one used for comparing speed of functions, but I don't think it can be used with pure Java.  Would I be better off going that route, or do you feel that the rate output is good enough (with the preceding caveats).

If you are firing up using Clojure, you'll have to use some outer testing facility like this.  If you're running using MASON's doLoop() facility, you could do something like this (in Unix):

time java sim.app.heatbugs.HeatBugs -repeat 30 -until 1000000

Then divide the result by 30.

> One odd thing that I've noticed is on one of my machines (the slower one) after the first 10 or 20 seconds, the rates often drop just a little bit and remains hovering around the lower value.  This is very consistent.  

Sounds about right.  Often caused by memory bounds, among other things.  I see it a lot.

> (I'm playing with versions of the students simulation from Chapter 2 of the manual.  With type hints and the latest version of Clojure, I have a Clojure version that's close to half the speed of the Java version, rather than the ~1/170 ratio with my first, unoptimized version.   I'm using Java 1.6 (not worth explaining), so maybe the ratio would be worse with a more recent javac.  This is all on the assumption that the answer to my first question is "yes", of course.)

Half the speed is awesome.

For grins, you might try whipping it up in Kawa too and compare the too.

Sean