I've been evaluating ECJ for possible use in a large scale cloud computing based evolutionary computation project for the optimization of AIs in highly complex wargames.
What makes this a hard problem is that:
1. The evaluations are expensive - a mean of 400 seconds per evaluation on a one core 3.5 ghz processor.
2. The evaluations are noisy - a better AI can still lose to worse AI, and often does
3. The evaluation run times also have a large variance from approximately 80 seconds up to 1000 seconds.
As evolutionary approaches, I'm leaning to steady-state EDA type algorithms as a seemingly good fit for the problem domain.
All was looking good in the evaluation of ECJ until what seems like a fatal problem in the last sentence of section 6.1.6 Noisy Distributed Problems in the ECJ Owners manual :
"There’s no equivalent to this hack in Asynchronous Evolution: you’ll just have to ask a machine to test the individual 5 times."
Unfortunately that would seem to significantly reduce the ability to fan out evaluations to reduce elapsed clock time per evaluation which would significantly increase "time travel" - ie where evaluated individuals re-enter a population as candidates for inclusion at a much later time than they were created for evaluation.
Is another hack possible to spread out evaluations where one needs to run multiple tests to get a good-enough estimator of an individual? i might even be willing to do the hacking.
--