Testing realizers, ideas and coorperation

Since BML realizers very quickly turn into complicate software engineering projects, some ways of (automaticly) testing them is essential. It would be nice to share some ideas on testing and perhaps share test sets.

In Elckerlyc (HMI, University of Twente's realizer: http://hmi.ewi.utwente.nl/showcases/Elckerlyc) we currently run (j)unit tests on the parser/constraint finder and on the scheduler. We check if the constraint finder finds the correct constraints and if the scheduler assigns valid times to motion units and speech units. We also check if warnings occur on the planning level (mainly stuff like starttime > endtime). Currently we do this for the 24 bml files at www.herwinvanwelbergen.nl/bml/bmltest.zip. This test set is slowly growing. All new planning or scheduling bugs we encounter are reproduced in unit test cases and minimal bml scripts, typically before they are fixed.

I'm now looking at some way to check the actual execution of BML: at least timing and what behavior should be active at what time, but could also be scenarios like removing gaze targets, evoking events, etc. Maybe using faster-than-real-time mockup planners and players and junit testing and/or making use of the BML feedback mechanism. Using the latter we could actually make a generic testsuite for different realizers.