Greg testing framework - Using greg

Go to the first, previous, next, last section, table of contents.

# Using `greg`

The Greg framework is designed to be used in two ways - as an embedded system from within any application which is linked with the Guile library, or stand-alone using the command-line `greg` driver script. For both of these methods of usage the test cases are written the same way and the expected output is the same.

## Output

While Greg may produce more verbose output in response to various settings, the basic output of a test run is a series of lines describing the success/failure state of each testcase encountered, followed by a summary of all test cases.

In `normal' mode, only unexpected results are displayed, but in `verbose' output mode, results for all results are displayed.

`greg` flags the outcome of each test case with a line consisting of one of the following codes followed by a colon, a space, and then the testcase description.

`PASS`
  The most desirable outcome: the test succeeded, and was expected to succeed.
`UPASS`
  A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.
`FAIL`
  A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to locate the bug.
`XFAIL`
  A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is `UNSUPPORTED` instead.
`UNRESOLVED`
  Output from a test requires manual inspection; the test suite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.
`UNTESTED`
  A test case is not yet complete, and in particular cannot yet produce a `PASS` or `FAIL`. You can also use this outcome in dummy "tests" that note explicitly the absence of a real test case for a particular property.
`UNSUPPORTED`
  A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine.

## Files and directories

A Greg test run expects to find files and directories in a certain layout (modeled on that used by DejaGNU).

The test source directory (normally your current directory) is expected to contain one or more tool directories. Each tool directory should contain one or more test scripts. In fact any file in a tool directory which has a .scm extension is assumed to be a Guile test script.

When a normal Greg test run is done, Greg goes through each tool directory in turn and loads each test script in turn.

You may set the Guile variable greg-tools or use the --tools ... command-line option to specify a list of tools directories to use rather than assuming that all subdirectories are tool directories. If you do this, the tools are tested in the order in which they appear in the list rather than the default order (ASCII sorted by name).

You may set the Guile variable greg-files or use the --files ... command-line option to specify a list of file names to use rather than assuming that all .scm files in each tool directory are test scripts. If you do this, the files are loaded in the order in which they appear in the list.

As a (minor) complication to this simple layout, Greg permits the use of begin.grg and end.grg scripts in both the main source directory and in each tool directory. These scripts permit you to add any initialisation and cleanup code you want. Typically (for non-embedded testing) you would use a begin.grg script to start the application to be tested.

If begin.grg exists in the main source directory, it will be loaded before any tools are tested.

If end.grg exists in the main source directory, it will be loaded after all the tools are tested.

If begin.grg exists in a tool directory, it will be loaded before any test scripts in that directory are loaded.

If end.grg exists in a tool directory, it will be loaded after all the test scripts in that directory are loaded.

## Embedded usage

Greg is designed primarily for embedded usage - any application that uses Guile as it's scripting language should be able to use Greg to test itself.

To this end - Greg provides a Guile module containing definitions of various procedures (used to run tests) and variables (used to modify the behavior of a test run).

Before trying to use any part of Greg, You need to load the Greg module with (use-modules (ice-9 greg))

The main procedure to run Greg tests is (greg-test-run). You can use this to run tests in much the same way as the greg script is used to run tests from the command-line. The behavior of this procedure is modified by setting the following top-level variables -

- greg-debug The greg-debug variable is a boolean used to determine whether to output extra debug information. The debug information is written to a .dbg file. Test scripts may use the greg-dlog procedure to output debug information.
- greg-files The greg-files variable is a list of strings used to specify the names of the test scripts to be run. If this list is empty, all the files with the `.scm` extension in a tool subdirectory are loaded. If the list contains names, the named scripts are loaded in the order in which they occur in the list rather than the default order (ASCII sorted by name).
  The names of scripts in this list should not contain the `.scm` extension - it will be added automatically by Greg.
- greg-obj-dir The greg-obj-dir variable is a string naming the directory in which Greg expects to find binaries to be run as child processes.
  By default, this is the current directory (`.`).
- greg-out-dir The greg-out-dir variable is a string naming the directory in which Greg will generate log and debug output files.
  By default, this is the current directory (`.`).
- greg-posix The greg-posix variable is a boolean used to determine whether to produce strict posix output, or permit non-posix extensions.
- greg-src-dir The greg-src-dir variable is a string naming the directory in which Greg will look for tool subdirectories and where it expects to find the main initialisation and cleanup scripts (begin.grg and end.grg) for a test run.
  By default, this is the current directory (`.`).
- greg-tools The greg-tools variable is a list of strings used to specify the names of the tools to be tested (ie the names of the subdirectories of the main source directory). If this list is empty, all the subdirectories of the main source directory (normally `.`) are assumed to be valid tool directories. If the list contains names, the named subdirectories are used in the order in which they occur in the list.
- greg-verbose The greg-verbose variable is a number used to determine the level of detail produced in the log output. At zero (the default), only minimal information is output (unexpected testcase results and a summary of the number of testcases passed). At one, the result of each testcase is reported and a more detailed summary is output. At two, entry and exit of tool directories and test files is reported.

## Command-line usage

`greg` is the executable test driver for Greg. This is a Guile script that you can use to run tests from the command line. The command-line options that you can pass to `greg` are listed below.

`greg` returns an exit code of `1` if any test has an unexpected result; otherwise (if all tests pass or fail as expected) it returns `0` as the exit code.

This is the full set of command line options that `greg` recognizes.

    greg --tool tool ...
    [ --debug ]
    [ --file script ... ]
    [ --help ]
    [ --objdir path ]
    [ --outdir path ]
    [ --posix ]
    [ --srcdir path ]
    [ -v | --verbose ]  [ -V | --version ]

`--tool tool ...`
  tool specifies what set of tests to run. It provides a list of subdirectories (each corresponding to a tool) in which test scripts can be found. Initialisation code (including executable tool startup) for each tool may be placed in ``begin.grg'` in the appropriate tool subdirectory. Cleanup code may be placed in ``end.grg'`. For example, including --tool gcc on the `greg` command line runs tests from the `gcc` subdirectory. The order in which the tools are tested will be the same as the order in which the tool names occur on the command line.
`--file script ...`
  Specify the names of testsuites to run. By default, `greg` runs all tests for the tool, but you can restrict it to particular testsuites by giving the names of the .scm `guile` scripts that control them. You do not need to supply the .scm file extension - it will be assumed. testsuite may not include path information; use plain filenames. The order in which the test scripts are run will be the same as the order in which the script names occur on the command line.
`--debug`
  Turns on the `expect` internal debugging output. Debugging output is displayed as part of the `greg` output, and logged to a file called ``tests.dbg'`. The extra debugging output does not normally appear on standard output.
`--help`
  Prints out a short summary of the `greg` options, then exits (even if you also specify other options).
`--objdir path`
  Use path as the top directory containing any auxiliary compiled test code. This defaults to ``.'`. Use this option to locate pre-compiled test code. You can normally prepare any auxiliary files needed with `make`.
`--outdir path`
  Write output logs in directory path. The default is ., the directory where you start `greg`. This option affects only the log and the debug files ``tool.log'` and ``tool.dbg'`.
`--srcdir path`
  Use path as the top directory for test scripts to run. `greg` looks in this directory for any subdirectory whose name matches any toolname (specified with --tool).
`--verbose`
`-v`
  By default, `greg` shows only the output of tests that produce unexpected results; that is, tests with status FAIL (unexpected failure), UPASS (unexpected success), or ERROR (a severe error in the test case itself). Specifying --verbose to see output for tests with status PASS (success, as expected) and XFAIL (failure, as expected). It also causes a more detailed summary to be displayed.
  Specifying --verbose more than once causes more detail to be displayed.
`--version`
`-V`
  Prints out the version numbers of Greg, and Guile, then exits without running any tests.

Go to the first, previous, next, last section, table of contents.