Good thoughts Eric.

however:

1.  I could specify multicore VMs on the cloud service, BUT that would be wasteful of paid for CPU.  For example: suppose I set the Test N Times to 4, and used a 4 core VM ... because of the very substantial variance in run times of the simulations, there would be unused cores as the evaluations than ran to the shorter end of the duration curve finished.  Economic efficiency is one of the key design goals for the project.\: only pay for core that you are actively running.  

My plan is to use singe core VMs.  Interestingly, the cloud services charge "per core" and a 4 core VM costs exactly 4 times the cost of a 1 core VM.  To minimize paying for unused cores it sure seems  that 1 core VMs makes the most sense.  I've also heard anecdotally that there is sometimes inadequate CPU<->Memory bandwidth available in the multicore VMs, and that could be a choke point for my compute and memory_access bound simulator.

Another problem is that it is likely I'd want to gradually increase the Test N Times number as evolution progresses, making efficient core use even more difficult.

2.  I could try and use a large population instead of doing  doing multiple tests, but I hate to restrict my search space for Evolutionary Algorithms that way.  

I currently have my own home-brew python evolver that  is parrallelized to the extent that it can run multiple evaluations in parallel on a single multi-core machine, not too hard to extend that to multiple systems, BUT it would be nice to get all the other stuff that comes with ECJ.

Perhaps another approach would be to extend (ie hack) the single machine steady-state system to make it asynchronous and write my own distributed evaluation runner. 

I don't need any of the fancy distributed evolution that the ECJ slaves can do: just a simple running of a shell level command and the capture and parse of the console output and then return the Win/Lose/Draw data back to the Master.  If I did it myself, rather than using sockets for communication (always a potential source of trouble!), I'd use one of the queue services that the Cloud Platform provides.

Anybody have any opinions on how hard it would be to make the ECJ steady-state mechanism asynchronous?

On Tue, Oct 25, 2016 at 2:29 PM, Eric 'Siggy' Scott <[log in to unmask]> wrote:
Jim,

Tis an interesting problem you raise.  It's true that the "hack" ECJ's generational algorithms use for re-evaluation doesn't make any sense in a steady-state model.

I don't know how people have handled multiple tests in parallel steady-state EAs before.  I don't recall seeing any discussion of it in the literature.

It seems to me that there are two options, though, that could save you the trouble of implementing a distribution multiple-testing scheme:
  1. If your evaluation function doesn't exhaust all of a node's resources, run multiple tests in parallel on the same node.  This is easy to do inside your implementation of the Problem class.

    Your heavy-duty simulations probably eat up all your nodes' processors, though, so this might not help your application.

  2. Ramp up the population size.  In some cases, given the same computational resources, using a large population can be just as effective at washing out the effects of noise as multiple testing.

    You can see if your application falls into this category by using a fixed budget of fitness evaluations and seeing if it makes more progress with a big population, or with multiple testing.  If the latter truly works much better, then that's a sign that it could be worth your effort to modify ECJ's steady-state master-slave model to support distributed multiple testing.
Just my two cents.  Sean et all will be more familiar with what it might take to implement the feature itself.

Siggy

On Tue, Oct 25, 2016 at 1:34 PM, Jim Rutt <[log in to unmask]> wrote:
I've been evaluating ECJ for possible use in a large scale cloud computing based evolutionary computation project for the optimization of AIs in highly complex wargames.   

What makes this a hard problem is that:

1.  The evaluations are expensive - a mean of 400 seconds per evaluation on a one core 3.5 ghz processor.  
2.  The evaluations are noisy - a better AI can still lose to worse AI, and often does
3.  The evaluation run times also have a large variance from approximately 80 seconds up to 1000 seconds.

As evolutionary approaches, I'm leaning to steady-state EDA type algorithms as a seemingly good fit for the problem domain.  

All was looking good in the evaluation of ECJ until what seems like a fatal problem in the last sentence of section 6.1.6 Noisy Distributed Problems in the ECJ Owners manual :

"There’s no equivalent to this hack in Asynchronous Evolution: you’ll just have to ask a machine to test the individual 5 times."

Unfortunately that would seem to significantly reduce the ability to fan out evaluations to reduce elapsed clock time per evaluation  which would significantly increase "time travel" - ie where evaluated individuals  re-enter a population as candidates for inclusion at a much later time than they were created for evaluation.  

Is another hack possible to spread out evaluations where one needs to run multiple tests to get a good-enough estimator of an individual?   i might even be willing to do the hacking.




--
Jim Rutt
JPR Ventures



--

Ph.D student in Computer Science, George Mason University
CFO and Web Director, Journal of Mason Graduate Research



--
===========================
Jim Rutt
JPR Ventures