NewRegressionFramework

We'd like to revamp the regression tests bu moving to a new framework. This page is intended to host a discussion of features and design for the new framework.

Desirable features

• Ability to add regressions via EXTRAS
  • For example, move eio tests into eio module so we don't try to run them when it's not compiled in
• Ability to not run regressions for which binaries or other inputs aren't available
  • With maybe some nice semi-automated way of downloading binaries when they're publicly available
• Better categorization of tests, and ability to run tests by category, e.g.:
  • by CPU model
  • by ISA
  • by Ruby protocol
• Better checkpoint testing
  • some of this doesn't really depend on the regression framework, just needs new tests
  • e.g., integrating util/checkpoint-tester.py
• Support for random testing (e.g., for background testing processes)
  • Random latencies?
  • Random testing a la memory testers but with different seeds, longer intervals
• Decouple from SCons somewhat
  • Avoid having scons dependency bugs force unnecessary re-running of tests, particularly for update-refs

Implementation ideas

Just ideas... no definitive decisions have been made yet.

• Use Python's unittest module, or something that extends it such as nose
• Use SCons to manage dependencies between binaries/test inputs and test results, but in a different SCons invocation (i.e., in its own SConstruct/SConscript)