Usually HotSpot needs about 1.5 times the RAM that your program is  
running in.  So some back-of-the-napkin:

Let's say your individuals are taking 500 nodes apiece (fairly  
large).  At 32 bytes per node, you've got 16000 bytes per  
individual.  At 2000 individuals, that's 32 megs.  Since you have two  
generations at a time, you've got 64 megs.  Let's say HotSpot needs  
twice that in heap to GC properly.  That puts you at 128 megs.  And  
no sweet spot, mind you -- EC problems scan through the entire  
population once per generation, so it's all got to be in RAM.

If you used a packed array storage format like lil-gp, you could get  
this to 4 bytes per node, so this whole thing could run in 32 megs  
(or less if you don't run in Java).  But we don't offer a packed  
array storage format, and I originally didn't write one because it's  
a little pokey and overly complex to breed with; we'd of course be  
interested in any one someone might whip up.

One quick way to save some RAM: use steady-state evolution rather  
than generational evolution.  That'll cut your RAM utilization in half.

Sean

On Jan 30, 2006, at 5:36 AM, Dignum, Stephen A N wrote:

> Thanks Sean,
>
> You're probably right in that there is extensive writing to disk at  
> this point.
>
> I've run the apps on two different windows machines, one laptop  
> with a whole load of memory and one old desktop with relatively  
> little and they both seem to slow down at the same point which  
> seems odd. I guess this has more to do with Windows' implementation  
> of java than the configuration of the PC. Both are running XP.
>
> My main problem seems to be with the ant example, which I've simply  
> doubled the generations* & population size. As some experiments are  
> run with far greater population sizes I was surprised that this was  
> an issue even if there was serious bloat.
>
> A test on a Linux/UNIX machine is required!
>
> Stephen
>
> * This seems odd but I'm doing some research in this area
>
> ________________________________
>
> From: ECJ Evolutionary Computation Toolkit on behalf of Sean Luke
> Sent: Sun 29/01/2006 19:14
> To: [log in to unmask]
> Subject: Re: Speed/Space
>
>
>
> On Jan 29, 2006, at 10:07 AM, Dignum, Stephen A N wrote:
>
>> I'm working through some of the GP examples in the apps directory.
>> All work fine until I start to up the population size (doubling to
>> 2048). I then get an out-of-memory error.
>
> Actually, we use 2048 all the time and don't have much trouble, so
> that's interesting to hear.  Which examples are creating difficulty
> for you?
>
>> To get around this I increase the standard java memory parameters
>> (Xmx, Xms, XX: etc), This works but if I then increase the number
>> of generations past about 65 I get a sudden reduction in
>> performance which isn't due to increase in program size i.e. number
>> of nodes stays about the same. There may be a large increase in
>> program depth but from my understanding of how ECJ works (by
>> looking through the code!) it is the number of nodes that effects
>> performance not depth.
>
> ECJ's memory usage is fairly constant.  In addition to a few small
> constant elements, just after breeding time (the worst memory usage
> case) ECJ has the current population _and_ the previous population
> which was generated from it.
>
> In the GP examples, individuals in the population have a constant
> overhead each: the big loss comes from GPNodes, which take up (last
> time I checked) about 32 bytes apiece.  Compared to other methods,
> it's a memory hog.  I got into significant discussion of this, I
> think, in the last tutorial.
>
> So as the individuals grow in size due to bloat, it's perfectly
> plausible for ECJ to be growing beyond your standard heap size.  But
> a sudden reduction in performance shouldn't ever happen if it's not
> commensurate with a sudden increase in GP nodes.  There is one
> possibility though: you could be out of your sweet spot and are now
> swapping badly.  Could this be problems specific to Windows?  We
> typically run on Linux and OS X.
>
> You can see the current memory usage and swap information by running
> like this
>
>         java -verbose:gc ...
>
> Sean