Regarding the general idea of a test suite (sorry if this is distracting from the immediate float->double topic): > I'd prefer to build one using the existing parameter file mechanism, which > would make it easier to integrate directly into ECJ, rather than something > like JUnit I recently wrote JUnit tests for a slightly modified SimpleEvaluator. I struggled at first with setting up the fixture (ex. should I use param files?), but ended up settling on a combination of A) manual initialization of just the components of EvolutionState that were required for the test, and B) using a bare-bones ParameterDatabase as a dependency-lookup pattern<http://xunitpatterns.com/Dependency%20Lookup.html> when the class under test needed to create other objects (ex. Evaluator.setup() initializing a Problem). The resulting fixture (reproduced below) was a lot easier to set up than I'd feared. It is self-contained (using no external ".params" files), and it wasn't hard to get >90% branch coverage for an Evaluator (that's another advantage of JUnit -- coverage metric tools like EclEmma work out of the box). If the class under test references a piece of global state we forgot it needed, we get an NPE (rather than a cheerful, faulty test). The tests confirmed the bug in SimpleEvaluator I wrote about earlier this month<http://listserv.gmu.edu/cgi-bin/wa?A2=ind1402&L=ECJ-INTEREST-L&P=R170> . I'd happily contribute tests for a couple classes -- if they were made part of the repository, we could let the suite grow over time. Siggy private final static String PROBLEM_DOUBLE_NAME = "ecjapp.doubles.TestSimpleGroupedProblem"; private final static String BAD_PROBLEM_DOUBLE_NAME = "ecjapp.doubles.TestSimpleProblem"; private EvolutionState state; private SimpleGroupedEvaluator sut; @Before public void setUp() { this.state = getFreshState(); this.sut = new SimpleGroupedEvaluator(); // Each test needs to call setup (perhaps after tweaking the parameters) } private static EvolutionState getFreshState() { final EvolutionState state = new SimpleEvolutionState(); // Set up just the parameters needed for the SUT to initialize itself state.parameters = getParams(); state.evalthreads = 1; // We need errors to throw exceptions (rather than exit the program) so we can verify them. state.output = Evolve.buildOutput(); state.output.setThrowsErrors(true); // Set up a population state.population = new Population(); state.population.subpops = new Subpopulation[] { new Subpopulation() }; state.population.subpops[0].individuals = getIndividuals(); return state; } private static ParameterDatabase getParams() { final ParameterDatabase parameters = new ParameterDatabase(); // Parameters needed by Evaluator.setup() parameters.set(new Parameter("eval." + Evaluator.P_PROBLEM), PROBLEM_DOUBLE_NAME); // Parameters needed by SimpleGroupedEvaluator.setup() parameters.set(new Parameter("eval." + SimpleGroupedEvaluator.P_CLONE_PROBLEM), "false"); parameters.set(new Parameter("eval." + SimpleGroupedEvaluator.P_NUM_TESTS), "1"); return parameters; } private static Individual[] getIndividuals() { final Individual[] individuals = new Individual[4]; individuals[0] = new TestIndividual(0); individuals[1] = new TestIndividual(1); individuals[2] = new TestIndividual(2); individuals[3] = new TestIndividual(3); return individuals; } @After public void tearDown() { this.state = null; this.sut = null; } On Thu, Feb 20, 2014 at 5:00 PM, Sean Luke <[log in to unmask]> wrote: > ECJ has never had a testing suite and that's a huge and embarassing > failure of the library. I have been fortunate that the library has > exhibited only a few bugs, never any showstoppers, and has had a lot of > eyeballs on it. > > I'm not asking for big test suites at this stage -- though if someone > would undertake one I have ideas about how to build one (I'd prefer to > build one using the existing parameter file mechanism, which would make it > easier to integrate directly into ECJ, rather than something like JUnit). > There's a lot of challenges involved in developing test suites for a > stochastic system: that's easily a software engineering thesis right there > (some of you might see this as an opportunity). But some simple unit tests > or sanity checks would be very welcome. > > But what would benefit me right now is for people to attack the changes I > made in ECJ and try to find something which messed up. I think I got it > right but made a *lot* of changes. Places to look: > > - Am I using doubles now but calling the wrong methods in > RandomChoice, or using RandomChoiceChooser rather than RandomChoiceChooserD? > > - Am I writing and reading the doubles to dataInput and > dataOutput, or still writing floats? > > - Am I calling the right methods in MersenneTwisterFast? > > - How about methods in ParameterDatabase? > > - Are your experiments still seeming to run right? > > ... etc. Basically I'm trying to establish an informal bug bounty in the > form of eternal glory if anyone can find errors in the code I've posted. > > Sean > > > On Feb 19, 2014, at 10:27 PM, Eric Scott <[log in to unmask]> wrote: > > > "I don't think any set of deterministic standard tests can replace > testing > > on the diversity of platforms and situations where ECJ is used." > > > > For high-level integration testing, this is true. But each step of the > process can be unit tests by stubbing out stochastic behavior (though this > can be tricky). > > > > Siggy > > > > > > On Wed, Feb 19, 2014 at 9:21 PM, Raymond Shpeley <[log in to unmask]> > wrote: > > I agree with Siggy. Looking at the tests for GP alone is a fair chunk of > work, let > > alone the entire library. Ideally, people who have seen and worked with > these > > demo apps before should be testing them. Still, some quick scouring is > likely > > worth the effort. > > > > With the information space (or search space as Koza calls it) of GC > being so > > huge, I don't think any set of deterministic standard tests can replace > testing > > on the diversity of platforms and situations where ECJ is used. > > > > Still, one must start some place. > > > > -- ray > > > > On Wed, 19 Feb 2014 16:37:58 -0500, Eric 'Siggy' Scott <[log in to unmask]> > > wrote: > > > > >It would of course be great to have CI and test suites for ECJ. > > > Retrofitting tests to the entire library is of course a monumental > > >undertaking, however. > > > > > >Also, in this case, Sean is asking if the example apps yield "basically > the > > >same results" -- qualitatively. Setting up quantitative tests of > > >qualitative criteria to see if the behavior of stochastic simulations > > >changes is difficult and usually needs done carefully on a case-by-case > > >basis. > > > > > >Siggy > > > > > > > > > > > -- > > > > Ph.D student in Computer Science > > George Mason University > > http://mason.gmu.edu/~escott8/ > -- Ph.D student in Computer Science George Mason University http://mason.gmu.edu/~escott8/