Greg testing framework - Writing tests

Go to the first, previous, next, last section, table of contents.

# Writing tests

### types of testsuite

There are three types of situation where Greg may be used as a test framework -

- Embedded testing
  In this case, you have an application that uses Guile as it's scripting language - so you have provided an interface between the internals of your program and the Guile interpreter. Your testcases can use that interface to test your program directly, and you run the tests by loading the Greg framework and then doing (greg-test-run) A testsuite for embedded use does not normally need begin.grg or end.grg since the application to be tested will be running the tests on itself - so it doesn't need to control child processes.
- Library testing
  When you want to test a library, you need to write a tiny program to start up the Guile command-line interpreter with the library to be tested inked in to the program via a via a set of stubs. The resulting program can then be used to run tests on the directly, just like normal embedded testing. There is a tool called G-Wrap (by Chris Lee), which will generate stubs to import your library functions into Guile. The home page of this tool is at http://www.cs.cmu.edu/~chrislee/g-wrap/ For testing Objective-C libraries, you should use the GNUstep-Guile interface library (http://www.tiptree.demon.co.uk/gstep) which provides a fairly complete interface between the Guile and Objective-C languages (using the GNU objective-C runtime).
- External testing
  When you want to test an external program, you use Greg in much the same way that you would use DejaGNU. That is - you run the program to be tested as a separate child process and your test cases consist of Guile code to send commands to the child process and wait for responses from it. The child process is normally started by a begin.grg script using the (greg-child) procedure.

### testsuite file layout

A testsuite for a tool is a directory containing one or more test script files and (optionally) begin.grg and end.grg files to handle initialisation and cleanup.

Each script file has a .scm extension and contains Guile (Scheme) code, but you do not need to know much about the Guile programming language to write most tests.

A script file will contain one or more testcases - each of which constitutes a test of a single well defined feature of the tool that the script is meant to test. A testcase is always written using the greg-testcase procedure.

### greg-testcase

The greg-testcase procedure takes three arguments -

- The assertion
  This is a string describing the test to be performed. It should take the form of an assertion that is expected to hold true.
- The expected outcome
  This is a boolean value - #t if the test is expected to return #t (ie. the assertion is expected to be proved correct), and #f if the test is expected to return #f (ie. the assertion is expected to be proved to be incorrect).
- The test itself
  This is a thunk (a Guile procedure that takes no arguments) and is normally a lambda expression. The return value of this procedure should be #t if the assertion is proved correct, #f if it is proved incorrect.

The Guile programming language permits the thunk to return in four ways -

- Return the boolean value - #t
  In which case, the test has succeeded, the assertion has proved correct, and Greg will record either an expected success (PASS) or an unexpected success (UPASS) depending on the expected outcome passed to greg-testcase.
- Return the boolean value - #f
  In which case, the test has failed, the assertion has proved incorrect, and Greg will record either an expected failure (XFAIL) or an unexpected failure (FAIL) depending on the expected outcome passed to greg-testcase.
- Return a non-boolean value
  In which case, the test is unresolved (UNRESOLVED).
- Throw an exception
  In which case Greg's behavior depends on the exception thrown -
  - 'pass
    Greg acts as if the test had returned the boolean #t
  - 'fail
    Greg acts as if the test had returned the boolean #f
  - 'unsupported
    Greg reports this case as unsupported (UNSUPPORTED)
  - 'untested
    Greg reports this case as untested (UNTESTED)
  - (quit)
    If the script calls the (quit) primitive - an exception is raised. In this special case, the test is reported as unresolved (UNRESOLVED) and no further tests are executed. The testrun is terminated.
  - any other exception
    Greg reports this case as unresolved (UNRESOLVED)

As there are no other ways in which the thunk may be exited, it is impossible for a testcase to produce a result that doesn't fit into the framework (unless your testcase manages either to crash Guile or enter an infinite loop - in which case you won't get any output).

The value returned by the greg-testcase procedure is a boolean - #t if the test resulted in an expected pass, #f otherwise.
You can use this return value to make the execution of subsequent testcases dependent on the success of an earlier testcase.

    ;
    ; A testcase to check an instance of numeric addition
    ;
    (greg-testcase "One plus One is two" #t
    (lambda ()
      (eq? (+ 1 1 ) 2)
    ))

    ;
    ;  The above testcase will generate output -
    ;  `PASS: One plus One is two'
    ;

### Multiple testcases

It is normal to have more than one testcase in a file and this produces no problems - the only thing to watch out for is communicating information between testcases -

The scope of variables defined in the thunk in a greg-testcase procedure call is that thunk - the variable will not be visible to the next testcase.

So - to pass information from one testcase to the next it is necessary to define variables that can be seen in each testcase. The way to do this is normally to define these variables at the start of the file and then use the set! procedure within each testcase to set a value for a variable to be passed to the next testcase.

    (define arith-ok #f)

    ;
    ; A testcase to check an instance of numeric addition
    ;
    (greg-testcase "One plus One is two" #t
    (lambda ()
      (if (eq? (+ 1 1 ) 2)
        (begin (set! arith-ok #t) #t)
        #f)
    ))

    ;
    ; A testcase to check arithmetic - only supported if we have addition.
    ;
    (greg-testcase "X multiplied by 2 is X plus X" #t
    (lambda ()
      (if arith-ok
        (eq? (+ 1 1) (* 1 2))
        (throw 'unsupported))
    ))

Of course, if (as above) the only information you want to pass from a testcase is whether the test succeeded or not, you can use the return value from the greg-testcase procedure directly -

    (if
      (greg-testcase "One plus One is two" #t
        (lambda ()
          (eq? (+ 1 1 ) 2)
        )
      )
      (greg-testcase "X multiplied by 2 is X plus X" #t
        (lambda ()
          (eq? (+ 1 1) (* 1 2))
        )
      )
      (greg-dlog "Arithmetic operations not supported\n")
    )

### External tests

When Greg is used to test an external application, you usually want to run that application as a child process on a pseudo-terminal and handle tests sending a sequence of commands to the application and reading anticipated output from the application.

#### Starting a child process

Greg provides the greg-child procedure to start up a child process on a pseudo-terminal. You would usually call this procedure in the begin.grg file in your tool directory, but you could call it at the start of each script to get a new child process for each script.

The greg-child procedure expects one argument (the name of the program to be executed) followed by any number of additional arguments which are the arguments to be passed to the child process.
If the program name does not begin with a slash, Greg will look in the directory specified in greg-obj-dir to find it (by default the current directory).
If you want your normal PATH to be searched for the program, you should use -
`(greg-child "/bin/sh" "-c" "foo args")`
to get the shell to execute program foo with arguments args.

The greg-child procedure will automatically close down the I/O channels to any process previously started and wait for that process to die. If the old child process is badly behaved and will not die, this can result in Greg hanging - in this case you may need to explicitly kill the old child by another method before starting the new child process (this is one of the uses of the end.grg script).

As a special case, you can use an empty string as the program name - if you do this, another copy of the guile process will be created as a child and the value returned by greg-child in the child process will be a list containing the single number 0 (in the parent it will be a list containing the input port, output port and process id of the child). You can use this information to get the child copy of the process to be the program under test. This is useful for embedded testing where you want to test the I/O capabilities of the program.

NB. The greg-child procedure is implemented on top of the new primitive pty-child. This primitive is used to create a new child process on the end of a pseudo-terminal. Arguments and return values are as for greg-child.

- When successfully starting a new program a three item list is returned in the parent - input port to read from the child, output port to write to the child, process id of child.
- When returning in a child process because the program name was an empty string a list of one item (zero) is returned.
- After a failure to create a child process, an empty list is returned.

#### Sending to a child process

The greg-send procedure is provided to send instructions to a child process. This procedure takes one or more string arguments and sends them to the child process (if one exists).

#### Reading from a child process

The greg-recv macro is used to read data from a child process. This procedure actually provides a simple front-end to the expect module. You can use the expect module facilities directly if you want more control than is offered by greg-recv.

The greg-recv macro expects one or more lists as arguments - each list containing a string (a pattern to match) and a result to return on a successful match. The value returned by greg-recv is the result for the pattern actually matched.
If no pattern is matched within the timeout period then an empty list is returned (unless you use (set! expect-timeout-proc xxx) to override Gregs timeout handler.
If no pattern is matched before an end-of-file is read, then an empty list is returned (unless you use (set! expect-eof-proc xxx) to override Gregs end-of-file handler.

Go to the first, previous, next, last section, table of contents.