ECJ has never had a testing suite and that's a huge and embarassing failure of the library.  I have been fortunate that the library has exhibited only a few bugs, never any showstoppers, and has had a lot of eyeballs on it.  

I'm not asking for big test suites at this stage -- though if someone would undertake one I have ideas about how to build one (I'd prefer to build one using the existing parameter file mechanism, which would make it easier to integrate directly into ECJ, rather than something like JUnit).  There's a lot of challenges involved in developing test suites for a stochastic system: that's easily a software engineering thesis right there (some of you might see this as an opportunity).  But some simple unit tests or sanity checks would be very welcome.

But what would benefit me right now is for people to attack the changes I made in ECJ and try to find something which messed up.  I think I got it right but made a *lot* of changes.  Places to look:

	- Am I using doubles now but calling the wrong methods in RandomChoice, or using RandomChoiceChooser rather than RandomChoiceChooserD?

	- Am I writing and reading the doubles to dataInput and dataOutput, or still writing floats?

	- Am I calling the right methods in MersenneTwisterFast?

	- How about methods in ParameterDatabase?

	- Are your experiments still seeming to run right?

... etc.  Basically I'm trying to establish an informal bug bounty in the form of eternal glory if anyone can find errors in the code I've posted.

Sean 


On Feb 19, 2014, at 10:27 PM, Eric Scott <[log in to unmask]> wrote:

> "I don't think any set of deterministic standard tests can replace testing
> on the diversity of platforms and situations where ECJ is used."
> 
> For high-level integration testing, this is true.  But each step of the process can be unit tests by stubbing out stochastic behavior (though this can be tricky).
> 
> Siggy
> 
> 
> On Wed, Feb 19, 2014 at 9:21 PM, Raymond Shpeley <[log in to unmask]> wrote:
> I agree with Siggy. Looking at the tests for GP alone is a fair chunk of work, let
> alone the entire library. Ideally, people who have seen and worked with these
> demo apps before should be testing them. Still, some quick scouring is likely
> worth the effort.
> 
> With the information space (or search space as Koza calls it) of GC being so
> huge, I don't think any set of deterministic standard tests can replace testing
> on the diversity of platforms and situations where ECJ is used.
> 
> Still, one must start some place.
> 
> -- ray
> 
> On Wed, 19 Feb 2014 16:37:58 -0500, Eric 'Siggy' Scott <[log in to unmask]>
> wrote:
> 
> >It would of course be great to have CI and test suites for ECJ.
> > Retrofitting tests to the entire library is of course a monumental
> >undertaking, however.
> >
> >Also, in this case, Sean is asking if the example apps yield "basically the
> >same results" -- qualitatively.  Setting up quantitative tests of
> >qualitative criteria to see if the behavior of stochastic simulations
> >changes is difficult and usually needs done carefully on a case-by-case
> >basis.
> >
> >Siggy
> >
> 
> 
> 
> -- 
> 
> Ph.D student in Computer Science
> George Mason University
> http://mason.gmu.edu/~escott8/