The UML test suite

Site Home Page The UML Wiki UML Community Site The UML roadmap What it's good for Case Studies Kernel Capabilities Downloading it Running it Compiling Installation Skas Mode Incremental Patches Test Suite Host memory use Building filesystems Troubles User Contributions Related Links The ToDo list Projects Diary Thanks Contacts
Tutorials The HOWTO (html) The HOWTO (text) Host file access Device inputs Sharing filesystems Creating filesystems Resizing filesystems Virtual Networking Management Console Kernel Debugging UML Honeypots gprof and gcov Running X Diagnosing problems Configuration Installing Slackware Porting UML IO memory emulation UML on 2G/2G hosts Adding a UML system call Running nested UMLs
How you can help Overview Documentation Utilities Kernel bugs Kernel projects
Screenshots A virtual network An X session
Transcripts A login session A debugging session Slackware installation
Reference Kernel switches Slackware README
Papers ALS 2000 paper (html) ALS 2000 paper (TeX) ALS 2000 slides LCA 2001 slides OLS 2001 paper (html) OLS 2001 paper (TeX) ALS 2001 paper (html) ALS 2001 paper (TeX) UML security (html) LCA 2002 (html) WVU 2002 (html) Security Roundtable (html) OLS 2002 slides LWE 2005 slides
Fun and Games Kernel Hangman Disaster of the Month
| ### The UML test suite The UML test suite is a set of Perl scripts which I use as a regression test on new UML patches. It comprises a test harness, which organizes and runs the suite, and the suite itself, which is a directory hierarchy which groups similar tests. The tests to be done during a particular test run are specified by the parent directory, if you want all of the tests in that directory to run, an individual test, or any combination of directories and files. The tests are able to specify the properties of the UML in which they are to run. The test harness compiles and runs UMLs according to the test requirements, trying to minimize the number of builds and boots required to satisfy all of the tests' requirements. The tests vary widely in what they test and how they work. The simplest ones run a command (which they may have compiled themselves) and check the return value. Others run gdb and make it step through a test program to check that breakpoints work correctly. These use an already-booted UML, and don't need any special setup of the UML. In contrast, there are tests which have requirements of the UML. For example, the boot_filesystems test checks that a user-specified set of filesystems boots correctly. So, it requests an unbooted UML for its use, but puts no more requirements on it. Others test various configurable options, and request a UML with certain options enabled, which they boot. Finally, there are tests which stress the kernel generally in order to make sure that it is stable under a hard workload. The kbuild test, which runs a kernel build, is an example of this.
Why you should run it
The main reason would be to do a sanity-check on UML when compiled locally, with whatever unusual config options you favor, and run on your hardware. UML is somewhat sensitive to toolchain changes, and, to a less extent, to the host hardware. Users regularly shake out bugs that don't appear during development, and these are often exposed by different toolchains and hardware. Secondly, if UML fails mysteriously somehow, a test may exercise the same bug. In this case, the test is likely to be much more specific than your workload, and the search for the bug much more constrained. It would also be a help to the UML project in general to have the tests run as often as possible, and on as a great a variety of hosts as possible. This will expose bugs, and get them fixed, more quickly.
How to run it
> First, download the latest version from the test suite section of the UML download page. `host% wget http://www.user-mode-linux.org/mirror/uml_tests_20040726.tar.bz2` Uncompress and extract it `host% bunzip2 uml_tests_20040726.tar.bz2` `host% tar xf uml_tests_20040726.tar` Set up a ~/.umltest, which is the config file needed to tell the harness about the external resources, such as filesystems, and other information it will need in order to run. The file contains a perl hash, with different types of information under different keys. If you don't know perl, don't panic, because the syntax is simple enough. Mine looks like this: `
>     {
>         kernel_pool => "/home/jdike/linux/2.4/um",
>         uml => {
>             ubd => { 0 => "/home/jdike/roots/debian_22" },
>             mem => "64M",
>             cow_ubds => 1,
>             prompts => { "/home/jdike/roots/debian_22" => "usermode:.*#" },
>         },
>         tests => { "filesystems/boot_filesystems" =>
>                    { filesystems => [ "/home/jdike/roots/debian_22" ],
>                      time => 0
>                    },
>                    "stress/kbuild" =>
>                    { kernel_pool => "/home/jdike/roots/kernel",
>                      time => 0 },
>         },
>     }
` The kernel_pool tells the harness the location of a UML source pool. This will be where it builds whatever UMLs it needs. The uml section specifies the default UML command. This will be used for any tests which don't have special requirements. The ubd hash within that is a mapping of device numbers to files, and this one will turn into "ubd0=/home/jdike/roots/debian_22" on the command line. The prompts hash specifies a regular expression which matches the shell prompts for the filesystems that will be booted. The tests section provides test-specific information. The keys here are the filenames of the tests under the tests/ directory, and without the ".pl" extension. They can ask for arbitrary information here. I specify one filesystem for the boot_filesystems test to boot, and an image of a filesystem containing a kernel source tree for the kernel build test. Now, the tests can be run as follows `host% perl test.pl` This will run the full test suite, reporting an overall status when it's done. A full log of the run will appear in tests.log. This will contain all output from the UMLs, including boot logs and anything the tests do. Specific tests can be run by adding them to the command line `perl test.pl stress/kbuild` Adding a directory to the command line will cause all of the tests within the that directory to be run. So, `perl test.pl functional` will run all the tests under functional/, but nothing else.
How to add tests
> The tests are all under the "tests" directory, each in a separate perl file. Here is a very simple test, commented: `
>     # Virtually all tests will need these two modules.  The first defines
>     # the Test object, an instance of which is returned from here.  The
>     # second is a library of useful utilities which essentially all of the
>     # tests use.
>     use UML::Test;
>     use UML::Testlib;

>     # strict is good
>     use strict;

>     # This function is the actual test.  It will be run inside a booted
>     # UML, at the shell.  run_exit_0 is from TestLib.  It runs the dd
>     # command within the UML object given to the test, returning if the
>     # command had an exit status of 0, and dying if not.  In this case,
>     # the harness will trap that and report the failure.
>     sub run {
>         my $test = shift;
>         my $uml = shift;

>         run_exit_0($uml, "dd if=/dev/mem of=/dev/null");
>     }

>     # This constructs the Test object and returns it.  This specifies that
>     # this test be run in a booted UML (state => "up") and that the test
>     # is contained in the 'run' function.
>     UML::Test->new(state => "up", run => \&run);
> ` This illustrates the two essential features of a test The function which executes the actual test The construction of the test object, which specifies the test function and any requirements that the test makes on the UML. The test can also request an unbooted UML with 'state => "up"', if the purpose of the test is to check that UML boots. It can also request a UML with special configuration options. The initrd test does this like so: `
>     UML::Test->new(state => "up",
>                    run => \&run,
>                    kconfig => { MMAPPER => "y" },
>                    config => { "uml/iomem" => "mmapper,$file" } );
` The kconfig option requests that it be given a UML with CONFIG_MMAPPER set to "y", and the config option adds 'iomem=mmapper,$file' to the UML command line.

Hosted at