Print

Print


Regarding the general idea of a test suite (sorry if this is distracting from the immediate float->double topic):
 
I'd prefer to build one using the existing parameter file mechanism, which would make it easier to integrate directly into ECJ, rather than something like JUnit

I recently wrote JUnit tests for a slightly modified SimpleEvaluator.  I struggled at first with setting up the fixture (ex. should I use param files?), but ended up settling on a combination of A) manual initialization of just the components of EvolutionState that were required for the test, and B) using a bare-bones ParameterDatabase as a dependency-lookup pattern when the class under test needed to create other objects (ex. Evaluator.setup() initializing a Problem).

The resulting fixture (reproduced below) was a lot easier to set up than I'd feared.  It is self-contained (using no external ".params" files), and it wasn't hard to get >90% branch coverage for an Evaluator (that's another advantage of JUnit -- coverage metric tools like EclEmma work out of the box).  If the class under test references a piece of global state we forgot it needed, we get an NPE (rather than a cheerful, faulty test).

The tests confirmed the bug in SimpleEvaluator I wrote about earlier this month.

I'd happily contribute tests for a couple classes -- if they were made part of the repository, we could let the suite grow over time.

Siggy



    private final static String PROBLEM_DOUBLE_NAME = "ecjapp.doubles.TestSimpleGroupedProblem";
    private final static String BAD_PROBLEM_DOUBLE_NAME = "ecjapp.doubles.TestSimpleProblem";
    private EvolutionState state;
    private SimpleGroupedEvaluator sut;   
  
    @Before
    public void setUp() {
        this.state = getFreshState();
        this.sut = new SimpleGroupedEvaluator(); // Each test needs to call setup (perhaps after tweaking the parameters)
    }
    
    private static EvolutionState getFreshState() {
        final EvolutionState state = new SimpleEvolutionState();
        // Set up just the parameters needed for the SUT to initialize itself
        state.parameters = getParams();
        state.evalthreads = 1;
        
        // We need errors to throw exceptions (rather than exit the program) so we can verify them.
        state.output = Evolve.buildOutput();
        state.output.setThrowsErrors(true); 
        
        // Set up a population
        state.population = new Population();
        state.population.subpops = new Subpopulation[] { new Subpopulation() };
        state.population.subpops[0].individuals = getIndividuals();
        return state;
    }
    
    private static ParameterDatabase getParams() {
        final ParameterDatabase parameters = new ParameterDatabase();
        // Parameters needed by Evaluator.setup()
        parameters.set(new Parameter("eval." + Evaluator.P_PROBLEM), PROBLEM_DOUBLE_NAME);
        
        // Parameters needed by SimpleGroupedEvaluator.setup()
        parameters.set(new Parameter("eval." + SimpleGroupedEvaluator.P_CLONE_PROBLEM), "false");
        parameters.set(new Parameter("eval." + SimpleGroupedEvaluator.P_NUM_TESTS), "1");
        return parameters;
    }
    
    private static Individual[] getIndividuals() {
        final Individual[] individuals = new Individual[4];
        individuals[0] = new TestIndividual(0);
        individuals[1] = new TestIndividual(1);
        individuals[2] = new TestIndividual(2);
        individuals[3] = new TestIndividual(3);
        return individuals;
    }
    
    @After
    public void tearDown() {
        this.state = null;
        this.sut = null;
    }


On Thu, Feb 20, 2014 at 5:00 PM, Sean Luke <[log in to unmask]> wrote:
ECJ has never had a testing suite and that's a huge and embarassing failure of the library.  I have been fortunate that the library has exhibited only a few bugs, never any showstoppers, and has had a lot of eyeballs on it.

I'm not asking for big test suites at this stage -- though if someone would undertake one I have ideas about how to build one (I'd prefer to build one using the existing parameter file mechanism, which would make it easier to integrate directly into ECJ, rather than something like JUnit).  There's a lot of challenges involved in developing test suites for a stochastic system: that's easily a software engineering thesis right there (some of you might see this as an opportunity).  But some simple unit tests or sanity checks would be very welcome.

But what would benefit me right now is for people to attack the changes I made in ECJ and try to find something which messed up.  I think I got it right but made a *lot* of changes.  Places to look:

        - Am I using doubles now but calling the wrong methods in RandomChoice, or using RandomChoiceChooser rather than RandomChoiceChooserD?

        - Am I writing and reading the doubles to dataInput and dataOutput, or still writing floats?

        - Am I calling the right methods in MersenneTwisterFast?

        - How about methods in ParameterDatabase?

        - Are your experiments still seeming to run right?

... etc.  Basically I'm trying to establish an informal bug bounty in the form of eternal glory if anyone can find errors in the code I've posted.

Sean


On Feb 19, 2014, at 10:27 PM, Eric Scott <[log in to unmask]> wrote:

> "I don't think any set of deterministic standard tests can replace testing
> on the diversity of platforms and situations where ECJ is used."
>
> For high-level integration testing, this is true.  But each step of the process can be unit tests by stubbing out stochastic behavior (though this can be tricky).
>
> Siggy
>
>
> On Wed, Feb 19, 2014 at 9:21 PM, Raymond Shpeley <[log in to unmask]> wrote:
> I agree with Siggy. Looking at the tests for GP alone is a fair chunk of work, let
> alone the entire library. Ideally, people who have seen and worked with these
> demo apps before should be testing them. Still, some quick scouring is likely
> worth the effort.
>
> With the information space (or search space as Koza calls it) of GC being so
> huge, I don't think any set of deterministic standard tests can replace testing
> on the diversity of platforms and situations where ECJ is used.
>
> Still, one must start some place.
>
> -- ray
>
> On Wed, 19 Feb 2014 16:37:58 -0500, Eric 'Siggy' Scott <[log in to unmask]>
> wrote:
>
> >It would of course be great to have CI and test suites for ECJ.
> > Retrofitting tests to the entire library is of course a monumental
> >undertaking, however.
> >
> >Also, in this case, Sean is asking if the example apps yield "basically the
> >same results" -- qualitatively.  Setting up quantitative tests of
> >qualitative criteria to see if the behavior of stochastic simulations
> >changes is difficult and usually needs done carefully on a case-by-case
> >basis.
> >
> >Siggy
> >
>
>
>
> --
>
> Ph.D student in Computer Science
> George Mason University
> http://mason.gmu.edu/~escott8/



--

Ph.D student in Computer Science
George Mason University
http://mason.gmu.edu/~escott8/