Regarding the general idea of a test suite (sorry if this is distracting
from the immediate float->double topic):
> I'd prefer to build one using the existing parameter file mechanism, which
> would make it easier to integrate directly into ECJ, rather than something
> like JUnit
I recently wrote JUnit tests for a slightly modified SimpleEvaluator. I
struggled at first with setting up the fixture (ex. should I use param
files?), but ended up settling on a combination of A) manual initialization
of just the components of EvolutionState that were required for the test,
and B) using a bare-bones ParameterDatabase as a dependency-lookup
pattern<http://xunitpatterns.com/Dependency%20Lookup.html> when
the class under test needed to create other objects (ex. Evaluator.setup()
initializing a Problem).
The resulting fixture (reproduced below) was a lot easier to set up than
I'd feared. It is self-contained (using no external ".params" files), and
it wasn't hard to get >90% branch coverage for an Evaluator (that's another
advantage of JUnit -- coverage metric tools like EclEmma work out of the
box). If the class under test references a piece of global state we forgot
it needed, we get an NPE (rather than a cheerful, faulty test).
The tests confirmed the bug in SimpleEvaluator I wrote about earlier this
month<http://listserv.gmu.edu/cgi-bin/wa?A2=ind1402&L=ECJ-INTEREST-L&P=R170>
.
I'd happily contribute tests for a couple classes -- if they were made part
of the repository, we could let the suite grow over time.
Siggy
private final static String PROBLEM_DOUBLE_NAME =
"ecjapp.doubles.TestSimpleGroupedProblem";
private final static String BAD_PROBLEM_DOUBLE_NAME =
"ecjapp.doubles.TestSimpleProblem";
private EvolutionState state;
private SimpleGroupedEvaluator sut;
@Before
public void setUp() {
this.state = getFreshState();
this.sut = new SimpleGroupedEvaluator(); // Each test needs to call
setup (perhaps after tweaking the parameters)
}
private static EvolutionState getFreshState() {
final EvolutionState state = new SimpleEvolutionState();
// Set up just the parameters needed for the SUT to initialize
itself
state.parameters = getParams();
state.evalthreads = 1;
// We need errors to throw exceptions (rather than exit the
program) so we can verify them.
state.output = Evolve.buildOutput();
state.output.setThrowsErrors(true);
// Set up a population
state.population = new Population();
state.population.subpops = new Subpopulation[] { new
Subpopulation() };
state.population.subpops[0].individuals = getIndividuals();
return state;
}
private static ParameterDatabase getParams() {
final ParameterDatabase parameters = new ParameterDatabase();
// Parameters needed by Evaluator.setup()
parameters.set(new Parameter("eval." + Evaluator.P_PROBLEM),
PROBLEM_DOUBLE_NAME);
// Parameters needed by SimpleGroupedEvaluator.setup()
parameters.set(new Parameter("eval." +
SimpleGroupedEvaluator.P_CLONE_PROBLEM), "false");
parameters.set(new Parameter("eval." +
SimpleGroupedEvaluator.P_NUM_TESTS), "1");
return parameters;
}
private static Individual[] getIndividuals() {
final Individual[] individuals = new Individual[4];
individuals[0] = new TestIndividual(0);
individuals[1] = new TestIndividual(1);
individuals[2] = new TestIndividual(2);
individuals[3] = new TestIndividual(3);
return individuals;
}
@After
public void tearDown() {
this.state = null;
this.sut = null;
}
On Thu, Feb 20, 2014 at 5:00 PM, Sean Luke <[log in to unmask]> wrote:
> ECJ has never had a testing suite and that's a huge and embarassing
> failure of the library. I have been fortunate that the library has
> exhibited only a few bugs, never any showstoppers, and has had a lot of
> eyeballs on it.
>
> I'm not asking for big test suites at this stage -- though if someone
> would undertake one I have ideas about how to build one (I'd prefer to
> build one using the existing parameter file mechanism, which would make it
> easier to integrate directly into ECJ, rather than something like JUnit).
> There's a lot of challenges involved in developing test suites for a
> stochastic system: that's easily a software engineering thesis right there
> (some of you might see this as an opportunity). But some simple unit tests
> or sanity checks would be very welcome.
>
> But what would benefit me right now is for people to attack the changes I
> made in ECJ and try to find something which messed up. I think I got it
> right but made a *lot* of changes. Places to look:
>
> - Am I using doubles now but calling the wrong methods in
> RandomChoice, or using RandomChoiceChooser rather than RandomChoiceChooserD?
>
> - Am I writing and reading the doubles to dataInput and
> dataOutput, or still writing floats?
>
> - Am I calling the right methods in MersenneTwisterFast?
>
> - How about methods in ParameterDatabase?
>
> - Are your experiments still seeming to run right?
>
> ... etc. Basically I'm trying to establish an informal bug bounty in the
> form of eternal glory if anyone can find errors in the code I've posted.
>
> Sean
>
>
> On Feb 19, 2014, at 10:27 PM, Eric Scott <[log in to unmask]> wrote:
>
> > "I don't think any set of deterministic standard tests can replace
> testing
> > on the diversity of platforms and situations where ECJ is used."
> >
> > For high-level integration testing, this is true. But each step of the
> process can be unit tests by stubbing out stochastic behavior (though this
> can be tricky).
> >
> > Siggy
> >
> >
> > On Wed, Feb 19, 2014 at 9:21 PM, Raymond Shpeley <[log in to unmask]>
> wrote:
> > I agree with Siggy. Looking at the tests for GP alone is a fair chunk of
> work, let
> > alone the entire library. Ideally, people who have seen and worked with
> these
> > demo apps before should be testing them. Still, some quick scouring is
> likely
> > worth the effort.
> >
> > With the information space (or search space as Koza calls it) of GC
> being so
> > huge, I don't think any set of deterministic standard tests can replace
> testing
> > on the diversity of platforms and situations where ECJ is used.
> >
> > Still, one must start some place.
> >
> > -- ray
> >
> > On Wed, 19 Feb 2014 16:37:58 -0500, Eric 'Siggy' Scott <[log in to unmask]>
> > wrote:
> >
> > >It would of course be great to have CI and test suites for ECJ.
> > > Retrofitting tests to the entire library is of course a monumental
> > >undertaking, however.
> > >
> > >Also, in this case, Sean is asking if the example apps yield "basically
> the
> > >same results" -- qualitatively. Setting up quantitative tests of
> > >qualitative criteria to see if the behavior of stochastic simulations
> > >changes is difficult and usually needs done carefully on a case-by-case
> > >basis.
> > >
> > >Siggy
> > >
> >
> >
> >
> > --
> >
> > Ph.D student in Computer Science
> > George Mason University
> > http://mason.gmu.edu/~escott8/
>
--
Ph.D student in Computer Science
George Mason University
http://mason.gmu.edu/~escott8/
