Difference between revisions of "Regression Tests"

Revision as of 21:06, 6 October 2006

Running Regressions

Running the M5 regression tests is the recommended way to make sure that any changes to the simulator still comply with M5. Regression Testing is performed using SCons to help guide which tests are run. All current tests are located in the: 'm5/tests/' directory.

The basic usage for running a regression is:

% scons build/<arch>_<mode>/tests/<binary_type>/<test_directory>

Here is an example of running all of the 'quick' regression tests for the ALPHA architecture in syscall-emulation (SE) mode.

% scons build/ALPHA_SE/tests/debug/quick
...
**** build/ALPHA_SE/tests/debug/quick/00.hello/alpha/linux/o3-timing PASSED!
...
**** build/ALPHA_SE/tests/debug/quick/00.hello/alpha/linux/simple-atomic PASSED!
...
**** build/ALPHA_SE/tests/debug/quick/00.hello/alpha/linux/simple-timing PASSED!
...
**** build/ALPHA_SE/tests/debug/quick/00.hello/alpha/tru64/o3-timing PASSED!
...

Creating Your Own Regressions

Adding a regression test is done by adding a new directory to the 'm5/tests' directory and filling that directory up with the appropriate reference files. Every regression test needs these files in order to compare the current changes of M5 with the 'known working state:

• stdout - Standard Output Stream
• stderr - Standard Error Stream
• config.ini - M5 Configuration File
• config.out - M5 Configuration Output
• m5stats.txt - Statistics From M5 Simulation

One needs to run the M5 with the appropriate binary in order to get the aforementioned simulation files. For example, a command line to collect simulation data for a hello world regression test might be:

% build/ALPHA_SE/m5.debug configs/example/se.py --cmd=tests/test-progs/hello/bin/alpha/hello 
  >stdout 2>stderr

Creating the 'hello world' regression test directory and copying over the reference output might then follow this command line sequence:

% mkdir tests/quick/00.hello
% mkdir tests/quick/00.hello/ref
% cp config.ini tests/quick/00.hello/ref
% cp config.out tests/quick/00.hello/ref
% cp m5stats.txt tests/quick/00.hello/ref
% cp stdout tests/quick/00.hello/ref
% cp stderr tests/quick/00.hello/ref

The last thing one needs to do is add a 'test.py' configuration file that specifies the workload for the regression test. For this 'hello world' example, the 'test.py' file would just contain:

root.system.cpu.workload = LiveProcess(cmd='hello', executable = binpath('hello'))

Once the test.py is copied to the 'hello world' regression test directory, the individual regression test can be run:

% cp test.py tests/quick/00.hello
% scons build/ALPHA_SE/tests/debug/quick/00.hello

Updating Regression Tests

Once you've made and verified changes to the simulator, you can update any regression tests with the "update" option:

% scons build/ALPHA_SE/tests/debug/quick/00.hello --update