Print

Print


Regarding the general idea of a test suite (sorry if this is distracting
from the immediate float->double topic):


> I'd prefer to build one using the existing parameter file mechanism, which
> would make it easier to integrate directly into ECJ, rather than something
> like JUnit


I recently wrote JUnit tests for a slightly modified SimpleEvaluator.  I
struggled at first with setting up the fixture (ex. should I use param
files?), but ended up settling on a combination of A) manual initialization
of just the components of EvolutionState that were required for the test,
and B) using a bare-bones ParameterDatabase as a dependency-lookup
pattern<http://xunitpatterns.com/Dependency%20Lookup.html> when
the class under test needed to create other objects (ex. Evaluator.setup()
initializing a Problem).

The resulting fixture (reproduced below) was a lot easier to set up than
I'd feared.  It is self-contained (using no external ".params" files), and
it wasn't hard to get >90% branch coverage for an Evaluator (that's another
advantage of JUnit -- coverage metric tools like EclEmma work out of the
box).  If the class under test references a piece of global state we forgot
it needed, we get an NPE (rather than a cheerful, faulty test).

The tests confirmed the bug in SimpleEvaluator I wrote about earlier this
month<http://listserv.gmu.edu/cgi-bin/wa?A2=ind1402&L=ECJ-INTEREST-L&P=R170>
.

I'd happily contribute tests for a couple classes -- if they were made part
of the repository, we could let the suite grow over time.

Siggy



    private final static String PROBLEM_DOUBLE_NAME =
"ecjapp.doubles.TestSimpleGroupedProblem";
    private final static String BAD_PROBLEM_DOUBLE_NAME =
"ecjapp.doubles.TestSimpleProblem";
    private EvolutionState state;
    private SimpleGroupedEvaluator sut;

    @Before
    public void setUp() {
        this.state = getFreshState();
        this.sut = new SimpleGroupedEvaluator(); // Each test needs to call
setup (perhaps after tweaking the parameters)
    }

    private static EvolutionState getFreshState() {
        final EvolutionState state = new SimpleEvolutionState();
        // Set up just the parameters needed for the SUT to initialize
itself
        state.parameters = getParams();
        state.evalthreads = 1;

        // We need errors to throw exceptions (rather than exit the
program) so we can verify them.
        state.output = Evolve.buildOutput();
        state.output.setThrowsErrors(true);

        // Set up a population
        state.population = new Population();
        state.population.subpops = new Subpopulation[] { new
Subpopulation() };
        state.population.subpops[0].individuals = getIndividuals();
        return state;
    }

    private static ParameterDatabase getParams() {
        final ParameterDatabase parameters = new ParameterDatabase();
        // Parameters needed by Evaluator.setup()
        parameters.set(new Parameter("eval." + Evaluator.P_PROBLEM),
PROBLEM_DOUBLE_NAME);

        // Parameters needed by SimpleGroupedEvaluator.setup()
        parameters.set(new Parameter("eval." +
SimpleGroupedEvaluator.P_CLONE_PROBLEM), "false");
        parameters.set(new Parameter("eval." +
SimpleGroupedEvaluator.P_NUM_TESTS), "1");
        return parameters;
    }


    private static Individual[] getIndividuals() {
        final Individual[] individuals = new Individual[4];
        individuals[0] = new TestIndividual(0);
        individuals[1] = new TestIndividual(1);
        individuals[2] = new TestIndividual(2);
        individuals[3] = new TestIndividual(3);
        return individuals;
    }

    @After
    public void tearDown() {
        this.state = null;
        this.sut = null;
    }



On Thu, Feb 20, 2014 at 5:00 PM, Sean Luke <[log in to unmask]> wrote:

> ECJ has never had a testing suite and that's a huge and embarassing
> failure of the library.  I have been fortunate that the library has
> exhibited only a few bugs, never any showstoppers, and has had a lot of
> eyeballs on it.
>
> I'm not asking for big test suites at this stage -- though if someone
> would undertake one I have ideas about how to build one (I'd prefer to
> build one using the existing parameter file mechanism, which would make it
> easier to integrate directly into ECJ, rather than something like JUnit).
>  There's a lot of challenges involved in developing test suites for a
> stochastic system: that's easily a software engineering thesis right there
> (some of you might see this as an opportunity).  But some simple unit tests
> or sanity checks would be very welcome.
>
> But what would benefit me right now is for people to attack the changes I
> made in ECJ and try to find something which messed up.  I think I got it
> right but made a *lot* of changes.  Places to look:
>
>         - Am I using doubles now but calling the wrong methods in
> RandomChoice, or using RandomChoiceChooser rather than RandomChoiceChooserD?
>
>         - Am I writing and reading the doubles to dataInput and
> dataOutput, or still writing floats?
>
>         - Am I calling the right methods in MersenneTwisterFast?
>
>         - How about methods in ParameterDatabase?
>
>         - Are your experiments still seeming to run right?
>
> ... etc.  Basically I'm trying to establish an informal bug bounty in the
> form of eternal glory if anyone can find errors in the code I've posted.
>
> Sean
>
>
> On Feb 19, 2014, at 10:27 PM, Eric Scott <[log in to unmask]> wrote:
>
> > "I don't think any set of deterministic standard tests can replace
> testing
> > on the diversity of platforms and situations where ECJ is used."
> >
> > For high-level integration testing, this is true.  But each step of the
> process can be unit tests by stubbing out stochastic behavior (though this
> can be tricky).
> >
> > Siggy
> >
> >
> > On Wed, Feb 19, 2014 at 9:21 PM, Raymond Shpeley <[log in to unmask]>
> wrote:
> > I agree with Siggy. Looking at the tests for GP alone is a fair chunk of
> work, let
> > alone the entire library. Ideally, people who have seen and worked with
> these
> > demo apps before should be testing them. Still, some quick scouring is
> likely
> > worth the effort.
> >
> > With the information space (or search space as Koza calls it) of GC
> being so
> > huge, I don't think any set of deterministic standard tests can replace
> testing
> > on the diversity of platforms and situations where ECJ is used.
> >
> > Still, one must start some place.
> >
> > -- ray
> >
> > On Wed, 19 Feb 2014 16:37:58 -0500, Eric 'Siggy' Scott <[log in to unmask]>
> > wrote:
> >
> > >It would of course be great to have CI and test suites for ECJ.
> > > Retrofitting tests to the entire library is of course a monumental
> > >undertaking, however.
> > >
> > >Also, in this case, Sean is asking if the example apps yield "basically
> the
> > >same results" -- qualitatively.  Setting up quantitative tests of
> > >qualitative criteria to see if the behavior of stochastic simulations
> > >changes is difficult and usually needs done carefully on a case-by-case
> > >basis.
> > >
> > >Siggy
> > >
> >
> >
> >
> > --
> >
> > Ph.D student in Computer Science
> > George Mason University
> > http://mason.gmu.edu/~escott8/
>



-- 

Ph.D student in Computer Science
George Mason University
http://mason.gmu.edu/~escott8/