GCL 2.6.2 tests

GCL 2.6.2 RELEASE NOTES
The GCL team is happy to announce the release of version 2.6.2, the latest achievement in the 'stable' series. While strictly speaking a bug-fix only release, 2.6.2 incorporates several major improvements over the last stable release, 2.5.3. Some highlights:

- The development of a 'lisp compiler torture tester' by GCL developer Paul Dietz which repeatedly compiles randomly generated forms of specifiable length to test the compiler for correctness.
- The application of several significant corrections to the GCL lisp compiler to remove every known instance of miscompilation uncovered by this tester. To our knowledge, GCL is alone with CLISP in passing this torture test for runs of effectively indefinite length.
- Major performance improvements were applied to the lisp compiler to enable it to complete random tests of great length in a reasonable amount of time.
- Corrections to the GCL core files to enable very large image sizes in 64 bits, in which more than a billion cons cells can be allocated. Current 64bit options include amd64, ia64, and alpha running most flavors of GNU/Linux.
- Corrections to the heap scaling behavior of the garbage collector, resulting in significant performance gains in many instances.
- Support for the latest gcc and binutils versions on all platforms but mingw
- The elimination of many instances of unnecessary internal garbage generation bringing the associated performance gains
- Native support for execstack protected linux kernels, such as on Fedora core systems
- Native support for FreeBSD, OpenBSD, and MacOSX
- Static function pointer support to stabilize dynamic library usage on Itanium systems
- Transparent readline initialization when compiled in
- Support for profiling via gprof
- Automatic disabling of SGC (stratified garbage collection) if the image is executed on a kernel not supporting fault address recovery
- Remove a memory leak associated with heavy bignum usage via the introduction of SGC contiguous pages
- Several significant internal bug fixes, epecially in the mingw port.
- Alter the build process to perform a full self compile with full function proclamation at build time.
- GCL now compiles Axiom from scratch and carries it to all supported platforms with the current exception of mingw
- GCL's ANSI build now in use for its first end-user application -- maxima (current cvs)
- New 64bit platform support -- amd64, with full native object relocation

The full changelog can be found in the source tree in the file 'debian/changelog'.
The GCL team has subjected this release to a wide variety of tests and benchmarks. While all such results are necessarily incomplete, one can nevertheless usefully summarize the approximate state of affairs as follows:

- GCL is about as portable as CLISP
- The GCL lisp compiler is about as robust/correct as that of CLISP, at least as measured by the random tester, which at present only covers a mostly integer subset of lisp.
- GCL is about as fast as CMUCL
- GCL plays a major role in carrying the primary large open source lisp end user applications to a wide variety of systems
- GCL is still the least ANSI compliant of the freely available lisp systems, though a modest level of compliance has been achieved in this release. Much greater compliance has been achieved in the 2.7.x (cvs unstable) series yet to be officially released.

The specific test results are arranged in the following table. Some terms need defining:

BFD | the method of relocating compiled lisp object modules into the running executable using the BFD library
custreloc | the method of relocating compiled lisp object modules into the running executable using the native GCL code. This method as well as the BFD method preserve the module loading across image saving and re-execution
dlopen | the method of dynamically linking in compiled lisp object modules into the existing session only via the system dynamic linker loader, ld.so.
SGC | Stratified Garbage Collection -- an optional accelerated generational garbage collection algorithm employing read-only memory
CLtL1 | Common Lisp, the Language vol I, referring to the book of the same name by Steele defining a widely used lisp language standard prior to the ANSI standardization process in 1994.
ANSI | the work in progress image build attempting to eventually extend traditional GCL into full ANSI complaince
Ansi tests | the results of the work in progress ansi compliance test suite written by GCL developer Paul Dietz presented as the number of failures divided by the total number of tests run
Random tests | the results of the random 'compiler torture tester' presented as the number of tests/the size of the random forms/the number of variables passed to the random function

In the table below, green denotes a pass, yellow denotes an as yet unimplemented option, and red indicates failure. Blank cells indicate tests that have not been run.

System | CPU | Self Build | BFD | dlopen | custreloc | Preferred Linking | SGC | CLtL1 | ANSI | ANSI tests | Random tests | Maxima 5.9.0/CLtL1 (4) | Maxima CVS/ANSI (4) | ACL2 2.8/CLtL1 (5) | Axiom CVS/CLtL1 (6) | nqthm CLtL1 | pc-nqthm CLtL1
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
Debian GNU/Linux (sid) | i386 |  |  |  |  | bfd or custreloc |  |  |  | 303/ 10697 | 50000/10000/8 500000/1000/8 |  |  |  |  |  | (setq si::*multiply-stacks* 16)
Debian GNU/Linux (sid) | sparc |  |  |  |  | bfd or custreloc |  |  |  | 303/ 10697 |  |  |  |  |  |  |
Debian GNU/Linux (sid) | powerpc |  |  |  |  | bfd |  |  |  | 303/ 10697 |  |  |  |  |  |  |
Debian GNU/Linux (sid) | amd64 |  |  |  |  | bfd |  |  |  | 303/ 10697 |  |  |  |  |  |  |
Debian GNU/Linux (sid) | arm |  |  |  |  | bfd |  |  |  | 303/ 10697 |  |  |  |  |  |  |
Debian GNU/Linux (sid) | m68k |  |  |  |  | bfd |  |  |  | 303/ 10697 |  |  |  |  |  |  |
Debian GNU/Linux (sid) | s390 |  |  |  |  | bfd |  |  |  | 303/ 10697 |  |  |  |  |  |  |
Debian GNU/Linux (sid) | ia64 |  |  |  |  | dlopen |  |  |  | 303/ 10697 |  |  |  |  | (1) |  |
Debian GNU/Linux (sid) | hppa | -O0 |  |  |  | dlopen |  |  |  | 303/ 10697 |  |  |  |  | (1) |  |
Debian GNU/Linux (sid) | mips |  |  |  |  | dlopen |  |  |  | 303/ 10697 |  |  |  |  | (1) |  |
Debian GNU/Linux (sid) | mipsel |  |  |  |  | dlopen |  |  |  | 303/ 10697 |  |  |  |  | (1) |  |
Debian GNU/Linux (sid) | alpha |  |  |  |  | dlopen |  |  |  | 303/ 10697 |  |  |  |  | (1) |  |
Fedora FC1 | i386 |  |  |  |  | bfd or custreloc |  |  |  | 303/ 10697 | 12000/1000/8 |  |  |  |  |  |
Solaris | sparc |  |  |  |  | bfd or custreloc |  |  |  | 303/ 10697 | 4000/1000/8 (4) |  |  |  |  |  |
Windows MINGW(a) | i386 |  |  |  |  | custreloc |  |  |  | 303/ 10697 | 57000/1000/8 |  |  |  | (2) |  |
MacOSX | powerpc |  |  |  |  | bfd | (3) |  |  | 303/ 10697 |  |  |  |  |  |  |
OpenBSD | i386 |  |  |  |  | bfd |  |  |  | 303/ 10697 |  |  |  |  |  |  |
FreeBSD | i386 |  |  |  |  | custreloc |  |  |  | 303/ 10697 |  |  |  |  |  |  |

Notes:
(1) dlopen builds use file descriptors for each object load. The step in the Axiom build process which regenerates its databases consumes more than the conventional maximum of 1024 file descriptors available by default on most UNIX systems.
(2) An AXIOMsys executable can be produced, and is basically functional, but experiences sporadic errors of a type as yet unknown.
(3) This is known to work on at least some versions of the OS, but others report a hang (infinite loop) when enabling SGC. It is possible that this is due to a mprotect bug in older versions of the Darwin system shared libraries. 'compatibility version of user 6.0.0' appears to work.
(4) On this machine, the underlying gcc was old (3.0) and segfaulted outside of GCL when attempting to compile its produced C code after a few thousand iterations.
(a) The preferred build environment for Mingw/Windows is gcc 3.3.1, binutils 2.14.90, and the latest msys release.
The following table presents the results of the popular gabriel benchmarks of three freely available lisp systems, GCL, CLISP and CMUCL. Times are presented as multiples of the time GCL took in completing the tests. Green indicates tests on which GCL is the fastest, while red indicates tests on which GCLwas not the fastest. The benchmark code can be found in ftp://ftp.ma.utexas.edu/gcl/gabriel.tgz. The number of test iterations has been increased by a factor of 400 to overcome granularity issues on modern machines. The '(print (time ...))' statements around each test iteration were removed, again due to granularity and relative i/o load. Likewise the special init.lsp file conventionally used to preallocate GCL memory in such cases was removed as it is now mostly obsolete. Finally the tests were modified slightly to place the optimization declamations at the top of each file being compiled as suggested by a CMUCL expert.
As with any benchmark, results can vary somewhat with the details of the executing machine. With lisp in particular, the ratios of the cache sizes, cpu speed, and memory bandwidths can impact such tests significantly. We present the results for two popular configurations below. While the precise details of the differences are as yet known, it is speculated that the first result is more dominated by in-cache cpu performance, while the latter is more dominated by memory access efficiency.

Dual Intel Xeon 2.4Ghz, 512 Mb, Linux 2.4.20 | Athlon XP 3000+ (2.1Ghz), 512 Mb, Linux 2.4.26 |
Benchmark | GCL 2.6.2 | CMUCL 18e-9 | CLISP 2.33-2
BOYER | 1.000 | 2.200 | 9.869
BROWSE | 1.000 | 2.240 | NA
CTAK | 1.000 | 0.230 | 1.890
DDERIV | 1.000 | 2.148 | 2.909
DERIV | 1.000 | 2.083 | 3.640
DESTRU-MOD | 1.000 | 2.043 | 9.880
DESTRU | 1.000 | 1.168 | 5.743
DIV2 | 1.000 | 2.222 | 3.911
FFT-MOD | 1.000 | 1.585 | 206.057
FFT | 1.000 | 1.544 | 176.088
FPRINT | 1.000 | 2.136 | 3.742
FREAD | 1.000 | 1.746 | 2.111
FRPOLY | 1.000 | 1.524 | 5.112
PUZZLE-MOD | 1.000 | 10.824 | 41.618
PUZZLE | 1.000 | 11.324 | 37.671
STAK | 1.000 | 1.536 | 9.836
TAK-MOD | 1.000 | 1.465 | 15.053
TAK | 1.000 | 1.486 | 14.629
TAKL | 1.000 | 1.419 | 14.965
TAKR | 1.000 | 1.933 | 12.327
TPRINT | 1.000 | 0.937 | 1.263
TRAVERSE | 1.000 | 0.875 | 8.378
TRIANG-MOD | 1.000 | 7.067 | 26.814
TRIANG | 1.000 | 1.281 | 18.565
GEOMETRIC AVERAGE | 1.00 | 1.86 | 10.33
MEDIAN | 1.00 | 1.67 | 9.87
Benchmark | GCL 2.6.2 | CMUCL 18e | CLISP 2.33
BOYER | 1.000 | 0.892 | 6.316
BROWSE | 1.000 | 0.965 | NA
CTAK | 1.000 | 0.435 | 3.489
DDERIV | 1.000 | 0.822 | 1.579
DERIV | 1.000 | 0.651 | 1.639
DESTRU-MOD | 1.000 | 0.812 | 4.779
DESTRU | 1.000 | 0.550 | 3.239
DIV2 | 1.000 | 0.599 | 1.525
FFT-MOD | 1.000 | 2.655 | 337.207
FFT | 1.000 | 1.923 | 251.026
FPRINT | 1.000 | 2.322 | 3.508
FREAD | 1.000 | 1.890 | 1.900
FRPOLY | 1.000 | 1.013 | 3.606
PUZZLE-MOD | 1.000 | 5.976 | 20.350
PUZZLE | 1.000 | 5.472 | 19.387
STAK | 1.000 | 1.655 | 8.064
TAK-MOD | 1.000 | 1.382 | 14.775
TAK | 1.000 | 1.399 | 14.514
TAKL | 1.000 | 1.281 | 12.877
TAKR | 1.000 | 1.735 | 15.500
TPRINT | 1.000 | 2.008 | 1.674
TRAVERSE | 1.000 | 0.770 | 8.013
TRIANG-MOD | 1.000 | 6.639 | 25.182
TRIANG | 1.000 | 1.186 | 16.948
GEOMETRIC AVERAGE | 1.00 | 1.40 | 8.46
MEDIAN | 1.00 | 1.33 | 8.01

Many improvements are planned for the 2.7.x development series time permitting, the most important of which is to complete the task of building an ANSI compliant GCL image.