Greg testing framework - Greg in brief

Go to the first, previous, next, last section, table of contents.

# Greg in brief

Greg is a framework for testing other programs and libraries. Its purpose is to provide a single front end for all tests and to be a small, simple framework for writing tests. Greg leverages off the Guile language to provide all the power (and more) of other test frameworks with greater simplicity and ease of use.

The simplicity of the Greg framework makes it easy to write tests for any program, but it was specifically written for use with GNUstep-Guile to permit direct testing of the GNUstep libraries without the necessity to run a separate driver program.

The core functionality of Greg is a Guile module which can be loaded into any software with an embedded Guile interpreter. Any program which uses Guile as it's scripting language can therefore use Greg to test itself directly!

For testing external programs, Greg provides a compiled module that may be dynamically linked into Guile to permit you to run an application as a child process on a pseudo-terminal. In conjunction with the standard Guile `expect' module, this lets you test external programs.

Also provided is greg - a Guile script to invoke the Greg test framework in much the same way that runtest is used in DejaGNU.

All tests have the same output format (enforced by the greg-testcase procedure). Greg's output is designed to be both readable and readily parsed by other software, so that it can be used as input to customised testing processes.

Greg provides most of the functionality of DejaGNU but is rather simpler. It omits specific support for cross-platform/remote testing since this is really rather trivial to add where required and tends to vary from site to site so much that an attempt at a generic solution is pretty pointless. What Greg does do, is provide hooks to let you easily introduce site specific code for handling those sorts of situations.

The current version of Greg can normally be found on GNU ftp sites and at http://www.tiptree.demon.co.uk/gstep/guile/greg-0.6.tar.gz with documentation online at http://www.tiptree.demon.co.uk/gstep/guile/gregdoc_toc.html

or, for the bleeding edge - availably by anonymous cvs as part of the GNUstep-Guile package in the GNUstep project -

CVSROOT=":pserver:anoncvs@cvs.net-community.com:/gnustep"
export CVSROOT
cvs login (password is `anoncvs')
cvs -z3 checkout guile

## How to run a Greg testsuite

To run tests from an existing collection, try running

    make check

If the `check` target exists, it usually saves you some trouble--for instance, it can set up any auxiliary programs or other files needed by the tests.

Alternatively, if you are in the top-level source directory of an existing testsuite (ie. there are subdirectories containing files with a `.scm` extension), you can get the greg script to test all the tools in the directory by typing -

    greg

If you have a test suite that is intended to be used for `embedded' testing - You need to start the application to be tested, gain access to it's Guile command line (or other guile interface) and enter the commands -

    (use-modules (ice-9 greg))
    (greg-test-all)

## A trivial example of a testcase

Each Greg test is a `Guile` script; the tests vary widely in complexity, depending on the nature of the tool and the feature tested.

    ;
    ; GNUstep-guile interface library test
    ;
    ; Create an object using the NSString [stringWithCString:] method and
    ; check that the resulting object is of the correct class.
    ;
    (greg-testcase "The `stringWithCString:' method creates an NSString object" #t
    (lambda ()
      (define obj ([] "NSString" stringWithCString: "Hello world"))
      (gstep-bool ([] obj isKindOfClass: ([] "NSString" class)))
    ))

Though brief, this example is a complete test. It illustrates some of the main features of Greg test scripts:

- The test case does not deal with accessing the library - the Guile interpreter in use is linked with the library. If you were testing an external application which needed to be started up as a child process running on a pseudo-terminal, the process startup would have been performed in a separate begin.grg script.
- You use the Guile programming language to write the script and, in this case, the builtin macros and procedures (such as `gstep-bool` and `[]` provided by the GNUstep-Guile library.
- Test cases use the procedure `greg-testcase` to record the test outcome. You pass this procedure a string (`assertion') describing the testcase, a boolean specifying the expected outcome, and a `thunk' (parameterless procedure) that performs the actual test.
  The `thunk' is normally a `lambda expression' - you can look these up in the Guile/Scheme documentation - or just follow the example.
- As much code as possible is placed inside the `thunk' passed to `greg-testcase` so that, in the event of an error, the `greg-testcase` procedure can trap it and report an unresolved test.
  The code - `(define obj ([] "NSString" stringWithCString: "Hello world"))` could have appeared outside the testcase, but that would have been less safe.

## Why Greg does what it does

Greg was written to support regression testing for the GNUstep libraries. It was inspired by an earlier test framework (by Ovidiu Predescu) that used DejaGNU along with a `driver' program (to make the calls to the library) and a suite of TcL scripts to control the driver.

There were three main problems (inherent in the nature of DejaGNU) with that approach -

- Writing/Debugging tests was a pain - errors in the TcL code of a testcase could all too easily interfere with the TcL routines that interfaced to the driver program.
  Solution - use another language where errors in testcases are localised.
- Writing/Debugging tests was a pain - you needed to attach gdb to the driver program that was running as a child of the DejaGNU process in order to debug errors in the driver/library under test.
  Solution - `embedded testing' make the test framework and the code under test part of the process, so it's easy to run the whole thing under gdb.
- The driver program and TcL scripts required quite a bit of support when adding new tests.
  Solution - `embedded testing' we needed to remove the intermediary layers and test as directly as possible.

So, something different was required, a test framework in a safer, simpler language that made it easy to create thin interfaces to libraries, so simplifying the task of producing testcases.

Of course, the good points of DejaGNU needed to be retained - clear output, Posix compliance, the ability to test separate programs as well as libraries.

A couple of additional goals seemed worthwhile -

- Make the test framework flexible so that applications could easily use it for self-testing.
- Give the framework a similar structure to, and a user interface like that of DejaGNU, so that people with a background in testing could easily adopt it.

Hopefully, Greg meets all its goals.

## A POSIX conforming test framework

This section is copied almost directly from the DejaGNU documentation with minor modifications -

Greg is believed to conform to the POSIX standard for test frameworks.

POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance.

The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says "The sun shall shine", a corresponding assertion might be "The sun is shining." A test based on this assertion would pass or fail depending on whether it is daytime or nighttime. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.

As there is no test suite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages, and their definitions. Four messages are supported for a required feature of POSIX conforming systems, and a fifth for a conditional feature. Greg supports the use of all five output messages; in this sense a test suite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:

`PASS`
  A test has succeeded. That is, it demonstrated that the assertion is true.
`UPASS`
  POSIX 1003.3 does not incorporate the notion of unexpected passes, so for strict POSIX, `PASS`, instead of `UPASS`, is returned for test cases which were not expected to pass but did. This means that `PASS` is in some sense more ambiguous than if `UPASS` is also used.
`FAIL`
  A test has produced the bug it was intended to capture. That is, it has demonstrated that the assertion is false. The `FAIL` message is based on the test case only. Other messages are used to indicate a failure of the framework.
`XFAIL`
  POSIX 1003.3 does not incorporate the notion of expected failures, so for strict POSIX, `FAIL`, instead of `XFAIL`, is returned for test cases which were expected to fail and did not. This means that `FAIL` is in some sense more ambiguous than if `XFAIL` is also used.
`UNRESOLVED`
  A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion; this outcome requires that a human being go over results, to determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should resolved to `PASS` or `FAIL` before a test run can be considered finished. Note that for POSIX, each assertion must produce a test result code. If the test isn't actually run, it must produce `UNRESOLVED` rather than just leaving that test out of the output. With Greg this is not a problem - any unexpected termination of a greg-testcase procedure will produce `UNRESOLVED`. Here are some of the ways a test may wind up `UNRESOLVED`:
  - A test's execution is interrupted.
  - A test does not produce a clear result. This is usually because there was a Guile exception thrown while processing the test. This usually requires a human being to examine the output to determine what really happened--and to improve the test case.
  - A test depends on a previous test, which fails.
  - The test was set up incorrectly and failed to produce a boolean value.
`UNTESTED`
  A test was not run. This is a placeholder, used when there is no real test case yet.

The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional:

`UNSUPPORTED`
  There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented.

Greg uses the same output procedures to produce these messages for all test suites, and these procedures are already known to conform to POSIX 1003.3. For a Greg test suite to conform to POSIX 1003.3, you must set the variable `greg-posix` to be true (or run the `greg` command with the `--posix` flag). Doing this will ensure that non-posix extensions are not used.

Go to the first, previous, next, last section, table of contents.