# Platform Independent Extensions

## Customizing &clisp; Process Initialization and Termination

### Cradle to Grave

What is done when

1. ### Initialization

   1. Parse command line arguments until the first positional argument (see `:SCRIPT` in image).

   2. Load the &mem-image;.

   3. Install internal signal handlers.

   4. Initialize time variables.

   5. Initialize locale-dependent encodings.

   6. Initialize stream variables.

   7. Initialize pathname variables.

   8. Initialize &ffi-pac;.

   9. Initialize modules.

   10. Run all functions in &init-hooks;.

   11. Say hi, unless suppressed by &opt-q;.

   12. Load &RC-file;, unless suppressed by &opt-norc;.

2. ### The actual work

   Handle command line options: file loading and/or compilation, form evaluation, script execution, &repl;.

3. ### Finalization (executed even on abnormal exit due to `kill`)

   1. Unwind the &STACK;, executing cleanup forms in &unwind-protect;.

   2. Run all functions in &fini-hooks;.

   3. Call &fresh-line; on the standard streams.

   4. Say bye unless suppressed by &opt-q;.

   5. Wait for a keypress if requested by `-w`.

   6. Close all open &file-stream-t;s.

   7. Finalize modules.

   8. Close all open DLLs.

### Customizing Initialization

&init-hooks; is run like this:

    (&ignore-errors; (&mapc; #'&funcall; &init-hooks;))

#### The difference between &init-hooks; and &init-function;

- &init-hooks; are always run regardless of the command line options before even the banner is printed.

- The &init-function; is run only if the &repl; is ever entered and just before the first prompt is printed.

### Customizing Termination

&fini-hooks; is run like this:

    (&mapc; #'&funcall; &fini-hooks;)

## Saving an Image

The function `(&savemem; &optional-amp; (&filename-r; "lispinit.mem") &key-amp; :KEEP-GLOBAL-HANDLERS :QUIET :INIT-FUNCTION :LOCKED-PACKAGES :START-PACKAGE :EXECUTABLE :NORC :SCRIPT :DOCUMENTATION :VERBOSE)` saves the running &clisp;'s memory to the file &filename-r;; extension &mem-file; is recommended (when &filename-r; does not have an extension, &mem-file; extension is automatically added unless the file being created is an executable).

`:QUIET`

- If this argument is not &nil;, the startup banner and the good-bye message will be suppressed, as if by &opt-q;.

  This is ¬-e; recommended for interactive application delivery, please append your banner to ours (using &init-function;) instead of replacing it.

`:VERBOSE`

- Print a message after writing the file. This argument defaults to &savemem-verbose;; initial value is &t;.

`:NORC`

- If this argument is not &nil;, the &RC-file; loading will be suppressed, as if by &opt-norc;.

`:INIT-FUNCTION`

- This argument specifies a function that will be executed at startup of the saved image, before entering the standard &repl; (but after all other initialization, see cradle-grave); thus, if you want to avoid the &repl;, you have to call &exit; at the end of the init function yourself (this does not prevent &fini-hooks; from being run).

  See the manual for passing command line arguments to this function.

  See also &init-hooks; and &fini-hooks;.

`:SCRIPT`

- This options determines the handling of positional arguments when the image is invoked.

  - If it is &t;, then the first positional argument is the script name and the rest is placed into &args;, as described in quickstart-unix.

  - It it is &nil;, then all positional arguments are placed into &args; to be handled by the &init-function;.

  This option defaults to &t; when &init-function; is &nil; and to &nil; when &init-function; is non-&nil;.

&documentation-k;

- The description of what this image does, printed by the `-help-image` olption.

  Defaults to `(&documentation; &init-function; '&function-doc;)`

`:LOCKED-PACKAGES`

- This argument specifies the packages to lock before saving the image; this is convenient for application delivery, when you do not want your users to mess up your product. This argument defaults to &sys-pack-list;.

`:START-PACKAGE`

- This argument specifies the starting value of &package-var; in the image being saved, and defaults to the current value of &package-var;.

`:KEEP-GLOBAL-HANDLERS`

- When non-&nil;, the currently established global handlers (either with &set-global-handler; or with &opt-on-error;) are inherited by the image. Defaults to &nil;, so that

      &sh-prompt; clisp -i myfile -x '(&savemem;)'

  will produce an image without any global handlers inherited from the batch mode of the above command.

`:EXECUTABLE`

- When non-&nil;, the saved file will be a standalone executable. In this case, the &mem-file; extension is not added. On &win32; and &cygwin; the extension `#P".exe"` is added instead.

  Additionally, if this argument is &zero;, the standard &clisp; command line options will ¬-e; be processed by the executable but will be placed into &args; instead. This is convenient for application delivery, so that your &clisp;-based application can accept, e.g., &opt-x;. To override this feature of the image, you have to prefix the options with `"--clisp"`, e.g., use `--clisp-x` instead of &opt-x;. This, given such a &clisp;-based application, you can get to an ordinary &clisp; &repl; by doing

      &sh-prompt; application --clisp-x '(&savemem; "myclisp" :executable t :init-function nil)'
      &sh-prompt; ./myclisp
      [1]> (! 20)
      2432902008176640000

  These instructions are also printed by `--clisp--help`.

  Of course, this feature opens a security hole if the application is running `setuid` root, therefore &clisp; resets the effective group and user IDs to the real ones if it sees a `"--clisp-*"` option.

You can use this memory image with the &opt-M; option. On &unix; systems, you may compress it with &gnu; &gzip; to save disk space.

### Image Portability

Memory images are ¬-e; portable across different platforms (in contrast with platform-independent &fasl-file; files). They are ¬-e; even portable across &linkset;s: image saved using the &full; &linkset; cannot be used with the &base; &rt;:

    &sh-prompt; clisp -K full -x '(&savemem;)'
    &sh-prompt; clisp -K base -M lispinit.mem
    base/lisp.run: initialization file `lispinit.mem' was not created by this version of CLISP runtime

See also /.

## Quitting &clisp;

The functions

- `(&exit; &optional-amp; &status-r;)`
- `(EXT:QUIT &optional-amp; &status-r;)`
- `(EXT:BYE &optional-amp; &status-r;)`

- all synonymous - terminate &clisp;. If &status-r; is non-&nil;, &clisp; aborts with the supplied numeric error &status-r;, i.e., the OS environment is informed that the &clisp; session did not succeed.

Final delimiters also terminate &clisp;.

## Internationalization of &clisp;

## Glossary

Internationalization (i18n)

- preparing a program so that it can use multiple national languages and national cultural conventions without requiring further source code changes.

Localization (l10n)

- providing the data - mostly textual translations - necessary for an internationalized program to work in a particular language and with particular cultural conventions.

&clisp; is internationalized, and is localized for the languages English, German, French, Spanish, Dutch, Russian, and Danish. &clisp; also supports internationalized Lisp programs, through &ggettext;, see i18n-mod.

### The Language

The facilities described in this section will work only for the languages for which &clisp; itself is already localized.

The language &clisp; uses to communicate with the user can be one of

- `ENGLISH`
- `DEUTSCH` (i.e., German)
- `FRANÇAIS` (i.e., French)
- `ESPAÑOL` (i.e., Spanish)
- `NEDERLANDS` (i.e., Dutch)
- `РУССКИЙ` (i.e. Russian)
- `DANSK` (i.e., Danish)

This is controlled by the &symbol-macro; &curr-lang;
, which can be set at run time as well as using the &opt-L; command line option. If you wish to change the locale directory at run time too, you can do that by setting &curr-lang; to a &cons-t; cell, whose &car; is the language (a &symbol-t;, one of the above), and whose &cdr; is the new locale directory.

More languages can be defined through the macro &deflang;
: `(&deflang; &lang-r;)`. For such an additional language to take effect, you must install the corresponding message catalog, or translate the messages yourself, using &ggettext; and &emacs; (or &xemacs;) po-mode.

This works only for strings. For arbitrary language-dependent Lisp objects, you define one through the macro &def-i-l;
: `(&def-i-l; &symbol-r; &optional-amp; (`default-language` &t;))` and add language-dependent values through the macro &defloc;
: `(&defloc; &symbol-r; &lang-r; `value-form`)` (One such form for each language. Languages without an assigned value will be treated like the default-language.) You can then access the localized value by calling &localized;
: `(&localized; &symbol-r; &optional-amp; &lang-r;)`

## Encodings

### Introduction

An encoding describes the correspondence between &character-t;s and raw bytes during input/output via &stream-t;s with &stream-element-type; &character-t;.

An &encoding; is an object composed of the following facets:

character set

- This denotes both the set of &character-t;s that can be represented and passed through the I/O channel, and the way these characters translate into raw bytes, i.e., the map between sequences of &character-t; and &ubyte-8; in the form of &string-t;s and &ubyte-vec; as well as character and byte &stream-t;s. In this context, for example, &utf-8; and &ucs-4; are considered different, although they can represent the same set of characters.

&line-term; mode

- This denotes the way newline characters are represented.

&encoding;s are also &type-glo;s. As such, they represent the set of characters encodable in the character set. In this context, the way characters are translated into raw bytes is ignored, and the line terminator mode is ignored as well. &typep; and &subtypep; can be used on encodings:

    (&subtypep; &utf-8; &utf-16;)
    &t; ;
    &t;
    (&subtypep; &utf-16; &utf-8;)
    &t; ;
    &t;
    (&subtypep; CHARSET:ASCII CHARSET:ISO-8859-1)
    &t; ;
    &t;
    (&subtypep; CHARSET:ISO-8859-1 CHARSET:ASCII)
    &nil; ;
    &t;

### 1:1 encodings

Encodings which define a bijection between character and byte sequences are called &enc1-1;s. &iso-8859-1; is an example of such an encoding: any byte sequence corresponds to some character sequence and vice versa. &ascii;, however, is ¬-e; a &enc1-1;: there are no characters for bytes in the range [128;255]. &utf-8; is ¬-e; a &enc1-1; either: some byte sequences do not correspond to any character sequence.

### Character Sets

&non-unicode-only;

- Only one character set is understood: the platform's native (8-bit) character set. See characters.

&unicode-only;

- The following character sets are supported, as values of the corresponding (constant) symbol in the &charset-pac; package:

  ### Symbols in package &charset-pac;

  1. `UCS-2` ≡ `UNICODE-16` ≡ `UNICODE-16-BIG-ENDIAN`, the 16-bit basic multilingual plane of the &unicode; character set. Every character is represented as two bytes.

  2. `UNICODE-16-LITTLE-ENDIAN`

  3. `UCS-4` ≡ `UNICODE-32` ≡ `UNICODE-32-BIG-ENDIAN`, the 21-bit &unicode; character set. Every character is represented as four bytes. This encoding is used by &clisp; internally.

  4. `UNICODE-32-LITTLE-ENDIAN`

  5. `UTF-8`, the 21-bit &unicode; character set. Every character is represented as one to four bytes. &ascii; characters represent themselves and need one byte per character. Most Latin/Greek/Cyrillic/Hebrew characters need two bytes per character. Most other characters need three bytes per character, and the rarely used remaining characters need four bytes per character. This is therefore, in general, the most space-efficient encoding of all of Unicode.

  6. `UTF-16`, the 21-bit &unicode; character set. Every character in the 16-bit basic multilingual plane is represented as two bytes, and the rarely used remaining characters need four bytes per character. &charset-glibc-libiconv;

  7. `UTF-7`, the 21-bit &unicode; character set. This is a stateful 7-bit encoding. Not all &ascii; characters represent themselves. &charset-glibc-libiconv;

  8. `JAVA`, the 21-bit &unicode; character set. &ascii; characters represent themselves and need one byte per character. All other characters of the basic multilingual plane are represented by `\u`nnnn`` sequences (`nnnn` a hexadecimal number) and need 6 bytes per character. The remaining characters are represented by `\u`xxxx`\u`yyyy`` and need 12 bytes per character. While this encoding is very comfortable for editing Unicode files using only &ascii;-aware tools and editors, it cannot faithfully represent all &unicode; text. Only text which does not contain `\u` (backslash followed by lowercase Latin u) can be faithfully represented by this encoding.

  9. `ASCII`, the well-known US-centric 7-bit character set (American Standard Code for Information Interchange - &ascii;).

  10. `ISO-8859-1`, &ascii-iso-ext; Afrikaans, Albanian, Basque, Breton, Catalan, Cornish, Danish, Dutch, English, Færoese, Finnish, French, Frisian, Galician, German, Greenlandic, Icelandic, Irish, Italian, Latin, Luxemburgish, Norwegian, Portuguese, Ræto-Romanic, Scottish, Spanish, and Swedish languages.

      This encoding has the nice property that

          (&loop; :for i :from 0 :to &char-code-limit; :for c = (&code-char; i)
            :always (&or-m; (¬-f; (&typep; c CHARSET:ISO-8859-1))
                        (&equalp; (&convert-string-to-bytes; (&string; c) CHARSET:ISO-8859-1)
                                (&vector; i))))
          T

      i.e., it is compatible with &clisp; &code-char;/&char-code; in its own domain.

  11. `ISO-8859-2`, &ascii-iso-ext; Croatian, Czech, German, Hungarian, Polish, Slovak, Slovenian, and Sorbian languages.

  12. `ISO-8859-3`, &ascii-iso-ext; Esperanto and Maltese languages.

  13. `ISO-8859-4`, &ascii-iso-ext; Estonian, Latvian, Lithuanian and Sami (Lappish) languages.

  14. `ISO-8859-5`, &ascii-iso-ext; Bulgarian, Byelorussian, Macedonian, Russian, Serbian, and Ukrainian languages.

  15. `ISO-8859-6`, suitable for the Arabic language.

  16. `ISO-8859-7`, &ascii-iso-ext; Greek language.

  17. `ISO-8859-8`, &ascii-iso-ext; Hebrew language (without punctuation).

  18. `ISO-8859-9`, &ascii-iso-ext; Turkish language.

  19. `ISO-8859-10`, &ascii-iso-ext; Estonian, Icelandic, Inuit (Greenlandic), Latvian, Lithuanian, and Sami (Lappish) languages.

  20. `ISO-8859-13`, &ascii-iso-ext; Estonian, Latvian, Lithuanian, Polish and Sami (Lappish) languages.

  21. `ISO-8859-14`, &ascii-iso-ext; Irish Gælic, Manx Gælic, Scottish Gælic, and Welsh languages.

  22. `ISO-8859-15`, &ascii-iso-ext; ISO-8859-1 languages, with improvements for French, Finnish and the Euro.

  23. `ISO-8859-16` &ascii-iso-ext; Rumanian language.

  24. `KOI8-R`, &ascii-iso-ext; Russian language (very popular, especially on the internet).

  25. `KOI8-U`, &ascii-iso-ext; Ukrainian language (very popular, especially on the internet).

  26. `KOI8-RU`, &ascii-iso-ext; Russian language. &charset-libiconv;

  27. `JIS_X0201`, a character set for the Japanese language.

  28. `MAC-ARABIC`, &ascii-pl-ext;.

  29. `MAC-CENTRAL-EUROPE`, &ascii-pl-ext;.

  30. `MAC-CROATIAN`, &ascii-pl-ext;.

  31. `MAC-CYRILLIC`, &ascii-pl-ext;.

  32. `MAC-DINGBAT`, a platform specific character set.

  33. `MAC-GREEK`, &ascii-pl-ext;.

  34. `MAC-HEBREW`, &ascii-pl-ext;.

  35. `MAC-ICELAND`, &ascii-pl-ext;.

  36. `MAC-ROMAN` ≡ `MACINTOSH`, &ascii-pl-ext;.

  37. `MAC-ROMANIA`, &ascii-pl-ext;.

  38. `MAC-SYMBOL`, a platform specific character set.

  39. `MAC-THAI`, &ascii-pl-ext;.

  40. `MAC-TURKISH`, &ascii-pl-ext;.

  41. `MAC-UKRAINE`, &ascii-pl-ext;.

  42. `CP437`, a DOS oldie, &ascii-pl-ext;.

  43. `CP437-IBM`, an IBM variant of `CP437`.

  44. `CP737`, a DOS oldie, &ascii-pl-ext;, &good-for; the Greek language.

  45. `CP775`, a DOS oldie, &ascii-pl-ext;, &good-for; some Baltic languages.

  46. `CP850`, a DOS oldie, &ascii-pl-ext;.

  47. `CP852`, a DOS oldie, &ascii-pl-ext;.

  48. `CP852-IBM`, an IBM variant of `CP852`.

  49. `CP855`, a DOS oldie, &ascii-pl-ext;, &good-for; the Russian language.

  50. `CP857`, a DOS oldie, &ascii-pl-ext;, &good-for; the Turkish language.

  51. `CP860`, a DOS oldie, &ascii-pl-ext;, &good-for; the Portuguese language.

  52. `CP860-IBM`, an IBM variant of `CP860`.

  53. `CP861`, a DOS oldie, &ascii-pl-ext;, &good-for; the Icelandic language.

  54. `CP861-IBM`, an IBM variant of `CP861`.

  55. `CP862`, a DOS oldie, &ascii-pl-ext;, &good-for; the Hebrew language.

  56. `CP862-IBM`, an IBM variant of `CP862`.

  57. `CP863`, a DOS oldie, &ascii-pl-ext;.

  58. `CP863-IBM`, an IBM variant of `CP863`.

  59. `CP864`, a DOS oldie, &good-for; the Arabic language.

  60. `CP864-IBM`, an IBM variant of `CP864`.

  61. `CP865`, a DOS oldie, &ascii-pl-ext;, &good-for; some Nordic languages.

  62. `CP865-IBM`, an IBM variant of `CP865`.

  63. `CP866`, a DOS oldie, &ascii-pl-ext;, &good-for; the Russian language.

  64. `CP869`, a DOS oldie, &ascii-pl-ext;, &good-for; the Greek language.

  65. `CP869-IBM`, an IBM variant of `CP869`.

  66. `CP874`, a DOS oldie, &ascii-pl-ext;, &good-for; the Thai language.

  67. `CP874-IBM`, an IBM variant of `CP874`.

  68. `WINDOWS-1250` ≡ `CP1250`, &ascii-pl-ext;, heavily incompatible with ISO-8859-2.

  69. `WINDOWS-1251` ≡ `CP1251`, &ascii-pl-ext;, heavily incompatible with ISO-8859-5, &good-for; the Russian language.

  70. `WINDOWS-1252` ≡ `CP1252`, a platform specific extension of the ISO-8859-1 character set.

  71. `WINDOWS-1253` ≡ `CP1253`, &ascii-pl-ext;, gratuitously incompatible with ISO-8859-7, &good-for; the Greek language.

  72. `WINDOWS-1254` ≡ `CP1254`, a platform specific extension of the ISO-8859-9 character set.

  73. `WINDOWS-1255` ≡ `CP1255`, &ascii-pl-ext;, gratuitously incompatible with ISO-8859-8, suitable for the Hebrew language. &charset-glibc-libiconv;

  74. `WINDOWS-1256` ≡ `CP1256`, &ascii-pl-ext;, &good-for; the Arabic language.

  75. `WINDOWS-1257` ≡ `CP1257`, &ascii-pl-ext;.

  76. `WINDOWS-1258` ≡ `CP1258`, &ascii-pl-ext;, &good-for; the Vietnamese language. &charset-glibc-libiconv;

  77. `HP-ROMAN8`, &ascii-pl-ext;.

  78. `NEXTSTEP`, &ascii-pl-ext;.

  79. `EUC-JP`, a multibyte character set for the Japanese language. &charset-glibc-libiconv;

  80. `SHIFT-JIS`, a multibyte character set for the Japanese language. &charset-glibc-libiconv;

  81. `CP932`, a Microsoft variant of `SHIFT-JIS`. &charset-glibc-libiconv;

  82. `ISO-2022-JP`, a stateful 7-bit multibyte character set for the Japanese language. &charset-glibc-libiconv;

  83. `ISO-2022-JP-2`, a stateful 7-bit multibyte character set for the Japanese language. This character set is only available on platforms with &glibc; 2.3 or newer or &libiconv;.

  84. `ISO-2022-JP-1`, a stateful 7-bit multibyte character set for the Japanese language. &charset-libiconv;

  85. `EUC-CN`, a multibyte character set for simplified Chinese. &charset-glibc-libiconv;

  86. `HZ`, a stateful 7-bit multibyte character set for simplified Chinese. &charset-libiconv;

  87. `GBK`, a multibyte character set for Chinese, &charset-glibc-libiconv;

  88. `CP936`, a Microsoft variant of `GBK`. &charset-glibc-libiconv;

  89. `GB18030`, a multibyte character set for Chinese, &charset-glibc-libiconv;

  90. `EUC-TW`, a multibyte character set for traditional Chinese. &charset-glibc-libiconv;

  91. `BIG5`, a multibyte character set for traditional Chinese. &charset-glibc-libiconv;

  92. `CP950`, a Microsoft variant of `BIG5`. &charset-glibc-libiconv;

  93. `BIG5-HKSCS`, a multibyte character set for traditional Chinese. &charset-glibc-libiconv;

  94. `ISO-2022-CN`, a stateful 7-bit multibyte character set for Chinese. &charset-glibc-libiconv;

  95. `ISO-2022-CN-EXT`, a stateful 7-bit multibyte character set for Chinese. &charset-glibc-libiconv;

  96. `EUC-KR`, a multibyte character set for Korean. &charset-glibc-libiconv;

  97. `CP949`, a Microsoft variant of `EUC-KR`. &charset-glibc-libiconv;

  98. `ISO-2022-KR`, a stateful 7-bit multibyte character set for Korean. &charset-glibc-libiconv;

  99. `JOHAB`, a multibyte character set for Korean used mostly on &dos;. &charset-glibc-libiconv;

  100. `ARMSCII-8`, &ascii-iso-ext; Armenian. &charset-glibc-libiconv;

  101. `GEORGIAN-ACADEMY`, &ascii-iso-ext; Georgian. &charset-glibc-libiconv;

  102. `GEORGIAN-PS`, &ascii-iso-ext; Georgian. &charset-glibc-libiconv;

  103. `TIS-620`, &ascii-iso-ext; Thai. &charset-glibc-libiconv;

  104. `MULELAO-1`, &ascii-iso-ext; Laotian. &charset-libiconv;

  105. `CP1133`, &ascii-iso-ext; Laotian. &charset-glibc-libiconv;

  106. `VISCII`, &ascii-iso-ext; Vietnamese. &charset-glibc-libiconv;

  107. `TCVN`, &ascii-iso-ext; Vietnamese. &charset-glibc-libiconv;

  108. `BASE64`, encodes arbitrary byte sequences with 64 &ascii; characters

              ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

       as specifined by &mime;; 3 bytes are encoded with 4 characters, line breaks are inserted after every 76 characters.

       While this is not a traditional character set (i.e., it does not map a set of characters in a natural language into bytes), it does define a map between arbitrary byte sequences and certain character sequences, so it falls naturally into the &encoding; class.

  Only on &gnu; systems with &glibc; 2.2 or better and other systems (&unix; and &win32;) on which the &libiconv; &c-lang; library has been installed

- The character sets provided by the library function &iconv; can also be used as encodings. To create such an encoding, call &make-encoding; with the character set name (a string) as the `:CHARSET` argument.

  When an &encoding; is available both as a &enc-built-in; and through &iconv;, the &enc-built-in; is used, because it is more efficient and available across all platforms.

  These encodings are not assigned to global variables, since there is no portable way to get the list of all character sets supported by &iconv;.

  On standard-compliant &unix; systems (e.g., &gnu; systems, such as &gnu;/&linux; and &gnu;/&hurd;) and on systems with &libiconv; you get this list by calling the program: `iconv -l`.

  The reason we use only &glibc; 2.2 or &libiconv; is that the other &iconv; implementations are broken in various ways and we do not want to deal with random &clisp; crashes caused by those bugs. If your system supplies an &iconv; implementation which passes the &libiconv;'s test suite, please report that to clisp-list and a future &clisp; version will use &iconv; on your system.

### Line Terminators

The line terminator mode can be one of the following three keywords:

&unix-k;

- Newline is represented by the &ascii; &lf-c; character (U000A).

&mac-k;

- Newline is represented by the &ascii; &cr-c; character (U000D).

&dos-k;

- Newline is represented by the &ascii; &cr-c; followed by the &ascii; &lf-c;.

Windows programs typically use the &dos-k; line terminator, sometimes they also accept &unix-k; line terminators or produce &mac-k; line terminators.

The &http; protocol also requires &dos-k; line terminators.

The line terminator mode is relevant only for output (writing to a &file-pipe-socket-s;). During input, all three kinds of line terminators are recognized. See also clhs-newline.

### Function &make-encoding;

The function `(&make-encoding; &key-amp; :CHARSET :LINE-TERMINATOR :INPUT-ERROR-ACTION :OUTPUT-ERROR-ACTION)` returns an &encoding;. The `:CHARSET` argument may be an encoding, a string, or &default-k;. The possible values for the &line-term; argument are the keywords &unix-k;, &mac-k;, &dos-k;.

The `:INPUT-ERROR-ACTION` argument specifies what happens when an invalid byte sequence is encountered while converting bytes to characters. Its value can be &error-k;, &ignore-k; or a character to be used instead. The &unicode; character #\uFFFD is typically used to indicate an error in the input sequence.

The `:OUTPUT-ERROR-ACTION` argument specifies what happens when an invalid character is encountered while converting characters to bytes. Its value can be &error-k;, &ignore-k;, a byte to be used instead, or a character to be used instead. The &unicode; character #\uFFFD can be used here only if it is encodable in the character set.

### Function &enc-charset;

&unicode-only;

The function `(&enc-charset; &encoding-r;)` returns the charset of the &encoding-r;, as a &symbol-t; or a &string-t;.

`(&string; (&enc-charset; &encoding-r;))` is not necessarily a valid &mime; name.

### Default encodings

Besides every &file-pipe-socket-s; containing an encoding, the following &symbol-macro; places contain global &encoding;s:

### &symbol-macro; &def-file-enc;

The &symbol-macro; &place; &def-file-enc; is the encoding used for new &file-pipe-socket-s;, when no &extfmt; argument was specified.

&unicode-only;

- The following are &symbol-macro; places.

  &path-enc;

- is the encoding used for converting filenames in the file system (represented with byte sequences by the OS) to lisp &pathname-t; components (&string-t;s). If this encoding is incompatible with some file names on your system, file system access (e.g., &directory;) may &signal; &error-t;s, thus extreme caution is recommended if this is ¬-e; a &enc1-1;. Sometimes it may not be obvious that the encoding is involved at all. E.g., on &win32;:

      (&parse-namestring; (&string; #\ARMENIAN_SMALL_LETTER_RA))
      *** - PARSE-NAMESTRING: syntax error in filename "ռ" at position 0

  when &path-enc; is &utf-16; because then `#\ARMENIAN_SMALL_LETTER_RA` corresponds to the 4 bytes `#(255 254 124 5)` and the byte `124` is ¬-e; a valid byte for a &win32; file name because it means `|` in &ascii;.

  The set of valid pathname bytes is determined by the &autoconf; test `src/m4/filecharset.m4` at configure time. While rather stable for the first 127 bytes, on &win32; it varies wildly for the bytes 128-256, depending on the OS version and the file system.

  The &line-term; mode of &path-enc; is ignored.

&term-enc;

- is the encoding used for communication with the terminal, in particular by &terminal-io-var;.

&misc-enc;

- is the encoding used for access to &env-var;s, command line options, and the like. Its &line-term; mode is ignored.

&foreign-enc;

- is the encoding for strings passed through the &ffi-pac; (some platforms only). If it is a &enc1-1;, i.e. an encoding in which every character is represented by one byte, it is also used for passing characters through the &ffi-pac;.

The default encoding objects are initialized according to &opt-E;.

#### Reminder

You have to use &letf;/&letf-star; for &symbol-macro;s; &let;/&let-star; will ¬-e; work!

#### Default &line-term;

The &line-term; facet of the above &encoding;s is determined by the following logic: since &clisp; understands all possible &line-term;s on input (see clhs-newline), all that matters is what &line-term; do most other programs expect?

&unix-only;

- If a non-0 `O_BINARY` &cpp; constant is defined, we assume that the OS distinguishes between text and binary files, and, since the encodings are relevant only for text files, we thus use &dos-k;; otherwise the default is &unix-k;.

&win32-only;

- Since most &win32; programs expect CRLF, the default &line-term; is &dos-k;.

This boils down to the following code in `src/encoding.d`:

     #if defined(WIN32) || (defined(UNIX) && (O_BINARY != 0))

##### Default &line-term; on &cygwin;

Both of the above tests pass on &cygwin;, so the default &line-term; is &dos-k;. If you so desire, you can change it in your &RC-file;.

### Converting between strings and byte vectors

Encodings can also be used to convert directly between strings and their corresponding byte vector representation according to that encoding.

`(&convert-string-from-bytes; &vec-r; &encoding-r; &key-amp; &start-k; &end-k;)`

- converts the subsequence of &vec-r; (a &ubyte-vec;) from &start-r; to &end-r; to a &string-t;, according to the given &encoding-r;, and returns the resulting string.

`(&convert-string-to-bytes; &string-r; &encoding-r; &key-amp; &start-k; &end-k;)`

- converts the subsequence of &string-r; from &start-r; to &end-r; to a &ubyte-vec;, according to the given &encoding-r;, and returns the resulting byte vector.

## Generic streams

This interface is &clisp;-specific and now obsolete. Please use the Gray streams interface instead.

Generic streams are user programmable streams. The programmer interface:

`(`gstream:make-generic-stream` `controller`)`

- returns a generic stream.

`(`gstream:generic-stream-controller` &stream-r;)`

- returns a private object to which generic stream methods dispatch. The typical usage is to retrieve the object originally provided by the user in `gstream:make-generic-stream`.

`(`gstream:generic-stream-p` &stream-r;)`

- determines whether a stream is a generic stream, returning &t; if it is, &nil; otherwise.

In order to specify the behavior of a generic stream, the user must define &clos; methods on the following &clos; generic functions. The function `gstream:generic-stream-&x-r;` corresponds to the &cl; function `&x-r;`. They all take a controller and some number of arguments.

`(gstream:generic-stream-read-char `controller`)`

- Returns and consumes the next character, &nil; at end of file. Takes one argument, the controller object.

`(gstream:generic-stream-peek-char `controller`)`

- Returns the next character, &nil; at end of file. A second value indicates whether the side effects associated with consuming the character were executed: &t; means that a full &read-char; was done, &nil; means that no side effects were done. Takes one argument, the controller object.

`(gstream:generic-stream-read-byte `controller`)`

- Returns and consumes the next integer, &nil; at end of file. Takes one argument, the controller object.

`(gstream:generic-stream-read-char-will-hang-p `controller`)`

- This generic function is used to query the stream's input status. It returns &nil; if `gstream:generic-stream-read-char` and `gstream:generic-stream-peek-char` will certainly return immediately. Otherwise it returns true.

`(gstream:generic-stream-write-char `controller` &ch-r;)`

- The first argument is the controller object. The second argument is the character to be written.

`(gstream:generic-stream-write-byte `controller` `by`)`

- The first argument is the controller object. The second argument is the integer to be written.

`(gstream:generic-stream-write-string `controller` &string-r; &start-r; &len-r;)`

- Writes the subsequence of &string-r; starting from &start-r; of length &len-r;. The first argument is the controller object.

`(gstream:generic-stream-clear-input `controller`)`

`(gstream:generic-stream-clear-output `controller`)`

`(gstream:generic-stream-finish-output `controller`)`

`(gstream:generic-stream-force-output `controller`)`

`(gstream:generic-stream-close `controller`)`

- Take one argument, the controller object.

## Weak Objects

Recall two terms: An object is called "alive" as long as it can be retrieved by the user or program, through any kind of references, starting from global and local variables. (Objects that consume no heap storage, also known as "immediate objects", such as &character-t;s, &fixnum-t;s, and &short-float-t;s, are alive indefinitely.) An object is said to be &gc;ed when its storage is reclaimed, at some moment after it becomes "dead".

### Weak Pointers

A &weak-pointer; is an object holding a reference to a given object, without keeping the latter from being &gc;ed.

### Weak Pointer API

`(`EXT:MAKE-WEAK-POINTER` &value-r;)`

- returns a &fresh; &weak-pointer; referring to &value-r;.

`(`EXT:WEAK-POINTER-P` &object-r;)`

- returns true if the &object-r; is of type &weak-pointer;.

`(`EXT:WEAK-POINTER-VALUE` `weak-pointer`)`

- returns two values: The original value and &t;, if the value has not yet been &gc;ed, else &nil; and &nil;. It is &setf;-able: you can change the value that the weak pointer points to.

Weak pointers are useful for notification-based communication protocols between software modules, e.g. when a change to an object &x-r; requires a notification to an object &y-r;, as long as &y-r; is alive.

### Weak Lists

A &weak-list; is an ordered collection of references to objects that does ¬-e; keep the objects from being &gc;ed. It is semantically equivalent to a list of &weak-pointer;s, however with a more efficient in-memory representation than a plain list of &weak-pointer;s would be.

### Weak List API

`(`EXT:MAKE-WEAK-LIST` &list-r;)`

- creates a &weak-list; pointing to each of the elements in the given &list-r;.

`(`EXT:WEAK-LIST-P` &object-r;)`

- returns true if the &object-r; is of type &weak-list;.

`(`EXT:WEAK-LIST-LIST` `weak-list`)`

- returns a &list-t; of those objects from the `weak-list` that are still alive.

`(&setf; (`EXT:WEAK-LIST-LIST` `weak-list`) &list-r;)`

- replaces the list of objects stored by the `weak-list`.

Weak lists are useful for notification based communication protocols between software modules, e.g. when a change to an object &x-r; requires a notification to objects &k1-r;, &k2-r;, ..., as long as such a particular &kn-r; is alive.

A &weak-list; with a single element is semantically equivalent to a single &weak-pointer;.

### Weak And Relations

A weak and relation is an ordered collection of references to objects, that does ¬-e; keep the objects from being &gc;ed, and which allows access to all the objects as long as all of them are still alive. As soon as one of them is &gc;ed, the entire collection of objects becomes empty.

### Weak And Relation API

`(`EXT:MAKE-WEAK-AND-RELATION` &list-r;)`

- creates a &weak-and-relation; between the objects in the given &list-r;.

`(`EXT:WEAK-AND-RELATION-P` &object-r;)`

- returns true if the &object-r; is of type &weak-and-relation;.

`(`EXT:WEAK-AND-RELATION-LIST` `weak-and-relation`)`

- returns the list of objects stored in the `weak-and-relation`. The returned list must not be destructively modified.

&weak-and-relation;s are useful to model relations between objects that become worthless when one of the objects dies.

A &weak-and-relation; with a single element is semantically equivalent to a &weak-pointer;.

### Weak Or Relations

A weak or relation is an ordered collection of references to objects, that keeps all objects from being &gc;ed as long as one of them is still alive. In other words, each of them keeps all others among them from being &gc;ed. When all of them are unreferenced, the collection of objects becomes empty.

### Weak Or Relation API

`(`EXT:MAKE-WEAK-OR-RELATION` &list-r;)`

- creates a &weak-or-relation; between the objects in the given &list-r;.

`(`EXT:WEAK-OR-RELATION-P` &object-r;)`

- returns true if the &object-r; is of type &weak-or-relation;.

`(`EXT:WEAK-OR-RELATION-LIST` `weak-or-relation`)`

- returns the list of objects stored in the `weak-or-relation`. The returned list must not be destructively modified.

&weak-or-relation;s are useful to model relations between objects that do not become worthless when one of the objects dies.

A &weak-or-relation; with a single element is semantically equivalent to a &weak-pointer;.

### Weak Associations

A weak association is a mapping from an object called &key-r; to an object called &value-r;, that exists as long as the key is alive. In other words, as long as the key is alive, it keeps the value from being &gc;ed.

### Weak Association API

`(`EXT:MAKE-WEAK-MAPPING` &key-r; &value-r;)`

- creates a &weak-mapping;.

`(`EXT:WEAK-MAPPING-P` &object-r;)`

- returns true if the object is of type &weak-mapping;.

`(`EXT:WEAK-MAPPING-PAIR` `weak-mapping`)`

- returns three values: the original key, the original value, and &t;, if the key has not yet been &gc;ed, else &nil;, &nil;, &nil;.

`(`EXT:WEAK-MAPPING-VALUE` `weak-mapping`)`

- returns the value, if the key has not yet been &gc;ed, else &nil;.

`(&setf; (`EXT:WEAK-MAPPING-VALUE` `weak-mapping`) &value-r;)`

- replaces the value stored in the `weak-mapping`. It has no effect when the key has already been &gc;ed.

Weak associations are useful to supplement objects with additional information that is stored outside of the object.

### Weak And Mappings

A weak and mapping is a mapping from a tuple of objects called &keys-r; to an object called &value-r;, that does ¬-e; keep the keys from being &gc;ed and that exists as long as all keys are alive. As soon as one of the keys is &gc;ed, the entire mapping goes away.

### Weak And Mapping API

`(`EXT:MAKE-WEAK-AND-MAPPING` &keys-r; &value-r;)`

- creates a &weak-and-mapping; between the &keys-r; objects in the given list and the given &value-r;. The &keys-r; list must be non-empty.

`(`EXT:WEAK-AND-MAPPING-P` &object-r;)`

- returns true if the &object-r; is of type &weak-and-mapping;.

`(`EXT:WEAK-AND-MAPPING-PAIR` `weak-and-mapping`)`

- returns three values: the list of keys, the value, and &t;, if none of the keys have been &gc;ed, else &nil;, &nil;, &nil;. The returned keys list must not be destructively modified.

`(`EXT:WEAK-AND-MAPPING-VALUE` `weak-and-mapping`)`

- returns the value, if none of the keys have been &gc;ed, else &nil;.

`(&setf; (`EXT:WEAK-AND-MAPPING-VALUE` `weak-and-mapping`) &value-r;)`

- replaces the value stored in the `weak-and-mapping`. It has no effect when some key has already been &gc;ed.

&weak-and-mapping;s are useful to model properties of sets of objects that become worthless when one of the objects dies.

A &weak-and-mapping; with a single key is semantically equivalent to a weak association.

### Weak Or Mappings

A weak or mapping is a mapping from a tuple of objects called &keys-r; to an object called &value-r;, that keeps all keys and the value from being &gc;ed as long as one of the keys is still alive. In other words, each of the keys keeps all others among them and the value from being &gc;ed. When all of them are unreferenced, the entire mapping goes away.

### Weak Or Mapping API

`(`EXT:MAKE-WEAK-OR-MAPPING` &keys-r; &value-r;)`

- creates a &weak-or-mapping; between the &keys-r; objects in the given list and the given &value-r;. The &keys-r; list must be non-empty.

`(`EXT:WEAK-OR-MAPPING-P` &object-r;)`

- returns true if the &object-r; is of type &weak-or-mapping;.

`(`EXT:WEAK-OR-MAPPING-PAIR` `weak-or-mapping`)`

- returns three values: the list of keys, the value, and &t;, if the keys have not yet been &gc;ed, else &nil;, &nil;, &nil;. The returned keys list must not be destructively modified.

`(`EXT:WEAK-OR-MAPPING-VALUE` `weak-or-mapping`)`

- returns the value, if the keys have not yet been &gc;ed, else &nil;.

`(&setf; (`EXT:WEAK-OR-MAPPING-VALUE` `weak-or-mapping`) &value-r;)`

- replaces the value stored in the `weak-or-mapping`. It has no effect when the keys have already been &gc;ed.

&weak-or-mapping;s are useful to model properties of sets of objects that do not become worthless when one of the objects dies.

A &weak-or-mapping; with a single key is semantically equivalent to a weak association.

### Weak Association Lists

A weak association list is an ordered collection of pairs, each pair being built from an object called &key-r; and an object called &value-r;. The lifetime of each pair depends on the type of the weak &alist;:

&key-k;

- The pair exists as long as the &key-r; is not &gc;ed. As long as the &key-r; is alive, it prevents the &value-r; from being &gc;ed.

&value-k;

- The pair exists as long as the &value-r; is not &gc;ed. As long as the &value-r; is alive, it prevents the &key-r; from being &gc;ed.

&key-and-value-k;

- The pair exists as long as the &key-r; and the &value-r; are alive.

&key-or-value-k;

- The pair exists as long as the &key-r; or the &value-r; are alive. As long as the &key-r; is alive, it prevents the &value-r; from being &gc;ed, and as long as the &value-r; is alive, it prevents the &key-r; from being &gc;ed.

In other words, each pair is:

&key-k;

- a &weak-mapping; from the &key-r; to the &value-r;,

&value-k;

- a &weak-mapping; from the &value-r; to the &key-r;,

&key-and-value-k;

- a &weak-and-relation; of the &key-r; and the &value-r;,

&key-or-value-k;

- a &weak-or-relation; of the &key-r; and the &value-r;.

### Weak Association List API

`(`EXT:MAKE-WEAK-ALIST` :type :initial-contents)`

- creates a &weak-alist;. The &type-r; argument must be one of the four aforementioned types; the default is &key-k;. The `initial-contents` argument must be an &alist;.

`(`EXT:WEAK-ALIST-P` &object-r;)`

- returns true if the &object-r; is of type &weak-alist;.

`(`EXT:WEAK-ALIST-TYPE` `weak-alist`)`

- returns the type of the `weak-alist`.

`(`EXT:WEAK-ALIST-CONTENTS` `weak-alist`)`

- returns an &alist; that corresponds to the current contents of the `weak-alist`.

`(&setf; (`EXT:WEAK-ALIST-CONTENTS` `weak-alist`) `contents`)`

- replaces the contents of a `weak-alist`. The `contents` argument must be an &alist;.

`(`EXT:WEAK-ALIST-ASSOC` &item-r; `weak-alist` [:test] [:test-not] [:key])`

- is equivalent to `(&assoc; &item-r; (`EXT:WEAK-ALIST-CONTENTS` `weak-alist`) [:test] [:test-not] [:key])`.

`(`EXT:WEAK-ALIST-RASSOC` &item-r; `weak-alist` [:test] [:test-not] [:key])`

- is equivalent to `(&rassoc; &item-r; (`EXT:WEAK-ALIST-CONTENTS` `weak-alist`) [:test] [:test-not] [:key])`.

`(`EXT:WEAK-ALIST-VALUE` &item-r; `weak-alist` [:test] [:test-not])`

- is equivalent to `(&cdr; (`EXT:WEAK-LIST-ASSOC` &item-r; `weak-alist` [:test] [:test-not]))`.

`(&setf; (`EXT:WEAK-ALIST-VALUE` &item-r; `weak-alist` [:test] [:test-not]) &value-r;)`

- replaces the value stored for &item-r; in a `weak-alist`. When a pair with the given &item-r; as key does not exist or has already been &gc;ed, a new pair is added to the &alist;.

Weak associations lists are useful to supplement objects with additional information that is stored outside of the object, when the number of such objects is known to be small.

### Weak Hash Tables

A weak &hash-table-t; is an unordered collection of pairs, each pair being built from an object called &key-r; and an object called &value-r;. There can be only one pair with a given &key-r; in a weak &hash-table-t;. The lifetime of each pair depends on the type of the weak &hash-table-t;

&key-k;

- The pair exists as long as the &key-r; is not &gc;ed. As long as the &key-r; is alive, it prevents the &value-r; from being &gc;ed.

&value-k;

- The pair exists as long as the &value-r; is not &gc;ed. As long as the &value-r; is alive, it prevents the &key-r; from being &gc;ed.

&key-and-value-k;

- The pair exists as long as the &key-r; and the &value-r; are alive.

&key-or-value-k;

- The pair exists as long as the &key-r; or the &value-r; are alive. As long as the &key-r; is alive, it prevents the &key-r; from being &gc;ed, and as long as the &value-r; is alive, it prevents the &key-r; from being &gc;ed.

In other words, each pair is:

&key-k;

- a &weak-mapping; from the &key-r; to the &value-r;,

&value-k;

- a &weak-mapping; from the &value-r; to the &key-r;,

&key-and-value-k;

- a &weak-and-relation; of the &key-r; and the &value-r;,

&key-or-value-k;

- a &weak-or-relation; of the &key-r; and the &value-r;.

See also make-hash.

Weak &hash-table-t;s are useful to supplement objects with additional information that is stored outside of the object. This data structure scales up without performance degradation when the number of pairs is big.

Weak &hash-table-t;s are also useful to implement canonicalization tables.

## Finalization

Calling `(&finalize; &object-r; &func-r;)` has the effect that when the specified object is being &gc;ed, `(&funcall; &func-r; &object-r;)` will be executed.

Calling `(&finalize; &object-r; &func-r; `guardian`)` has a similar effect, but only as long as the `guardian` has not been &gc;ed: when &object-r; is being &gc;ed, `(&funcall; &func-r; &object-r; `guardian`)` will be executed. If the `guardian` is &gc;ed before &object-r; is, nothing happens.

### Note

The time when the &object-r; is being &gc;ed is not defined deterministically. (Actually, it might possibly never occur.) It denotes a moment at which no references to &object-r; exist from other Lisp objects. When the &func-r; is called, &object-r; (and possibly `guardian`) enter the arena of live Lisp objects again.

No finalization request will be executed more than once.

## The Prompt

&clisp; prompt consists of 3 mandatory parts: start, body, and finish; and 2 optional parts: break, which appears only during debugging (after &break; or &error;), and step, which appears only during &step;ping. Each part is controlled by a custom variable, which can be either a &string-t; or a &function-t; of no arguments returning a &string-t; (if it is something else - or if the return value was not a &string-t; - it is printed with &princ;). In the order of invocation:

&prompt-start;

- Defaults to an empty string.

&prompt-step;

- Used only during &step;ping. Defaults to `Step n` , where &n-r; is the stepping level as returned by &step-level;.

&prompt-break;

- Used only inside break loop (during debugging). Defaults to `Break n` , where &n-r; is the break level as returned by &break-level;.

&prompt-body;

- Defaults to `package[n]` where &pack-r; is the shortest (nick)name (as returned by &package-shortest-name;) of the current package &package-var; if it is ¬-e; the same as it was in the beginning (determined by &prompt-new-package;) or if it does not contain symbol &t;, (it is assumed that in the latter case you would want to keep in mind that your current package is something weird); and &n-r; is the index of the current prompt, kept in &command-index;;

&prompt-finish;

- Defaults to `>` .

To facilitate your own custom prompt creation, the following functions and variables are available:

&break-level;

- This &function-t; returns current &break;/&error; level.

&step-level;

- This &function-t; returns current &step; level.

&prompt-new-package;

- This &function-t; returns &package-var; or &nil; if the current package is the same as it was initially.

&package-shortest-name;

- This &function-t; takes one argument, a &package-t;, and returns its shortest name or nickname.

&command-index;

- contains the current prompt number; it is your responsibility to increment it (this variable is bound to 0 before saving the &mem-image;).

## Maximum ANSI CL compliance

Some &ansi-cl; features are turned off by default for convenience and backward compatibility. They can be switched on, all at once, by setting the &symbol-macro; &ansi; to &t;, or they can be switched on individually. Setting &ansi; to &t; implies the following:

1. Setting &pathprint; to &t;.

2. Setting &spacecharprint; to &t;.

3. Setting &fixnum-char-ansi; to &t;.

4. Setting &count-ansi; to &t;.

5. Setting &pathmerge; to &t;.

6. Setting &parsename; to &t;.

7. Setting &flocont; to &t;.

8. Setting &floratcont; to &t;.

9. Setting &phasecont; to &t;.

10. Setting &loop-ansi; to &t;.

11. Setting &pr-empty-arr-ansi; to &t;.

12. Setting &pr-unreadable-ansi; to &t;.

13. Setting &defun-accept-spelalist; to &nil;.

If you run &clisp; with the &opt-ansi; switch or set the &symbol-macro; &ansi; to &t; and then save &mem-image;, then all subsequent invocations of &clisp; with this image will be as if with &opt-ansi; (regardless whether you actually supply the &opt-ansi; switch). You can always set the &symbol-macro; &ansi; to &nil;, or invoke &clisp; with the &opt-traditional; switch, reversing the above settings, i.e.,

1. Setting &pathprint; to &nil;.

2. Setting &spacecharprint; to &nil;.

3. Setting &fixnum-char-ansi; to &nil;.

4. Setting &count-ansi; to &nil;.

5. Setting &pathmerge; to &nil;.

6. Setting &parsename; to &nil;.

7. Setting &flocont; to &nil;.

8. Setting &floratcont; to &nil;.

9. Setting &phasecont; to &nil;.

10. Setting &loop-ansi; to &nil;.

11. Setting &pr-empty-arr-ansi; to &nil;.

12. Setting &pr-unreadable-ansi; to &nil;.

13. Setting &defun-accept-spelalist; to &t;.

## Additional Fancy Macros and Functions

&clisp; comes with some extension macros, mostly defined in the file ¯os3-lisp; and loaded from the file &init-lisp; during `make`:

### Macro ðe;

`(ðe; &val-type-r; &form-r;)` enforces a type check in both interpreted and compiled code.

### Macros &letf; & &letf-star;

These macros are similar to &let; and &let-star;, respectively, except that they can bind &place;s, even &place;s with &mul-val;. Example:

    (letf (((values a b) form)) ...)

is equivalent to

    (multiple-value-bind (a b) form ...)

while

    (letf (((first l) 7)) ...)

is approximately equivalent to

     (&let-star; ((#:g1 l) (#:g2 (first #:g1)))
       (&unwind-protect; (&progn; (&setf; (first #:g1) 7) ...)
          (&setf; (first #:g1) #:g2)))

### Macro `EXT:MEMOIZED`

`(`EXT:MEMOIZED` &form-r;)` memoizes the &pri-val; of &form-r; from its first evaluation.

### Macro &with-collect;

Similar to the &loop;'s `COLLECT` construct, except that it is looks more "Lispy" and can appear arbitrarily deep. It defines local macros (with ¯olet;) which collect objects given to it into lists, which are then returned as &mul-val;. E.g.,

    (ext:with-collect (c0 c1)
      (dotimes (i 10) (if (oddp i) (c0 i) (c1 i))))
    (1 3 5 7 9) ;
    (0 2 4 6 8)

returns two &list-t;s as &mul-val;.

### Macro `EXT:COMPILE-TIME-VALUE`

Sometimes one may want to call an expensive function at compilation time and write the &pri-val; into the &fasl-file; file, thus speeding up loading the &fasl-file; file. E.g., let your file `primes.lisp` be

    (defun primes-list (limit)
      "Return the list of all primes smaller than LIMIT."
      ...)
    (defvar *all-primes* (compile-time-value (primes-list &most-positive-fixnum;)))

Then

`(&load; "primes.lisp")`

- will ¬-e; call `primes-list` and `*all-primes*` will be &nil;.

`(&compile-file; "primes.lisp")`

- will call `primes-list` (and will probably take a long time) and will write the resulting list into `(&compile-file-pathname; "primes.lisp")`

`(&load; (&compile-file-pathname; "primes.lisp"))`

- will ¬-e; call `primes-list` but `*all-primes*` will be the list computed during compilation.

An alternative is to save a &mem-image;, which is faster than &fasl-file; file but less portable.

### Macro `EXT:WITH-GENSYMS`

Similar to its namesake from Paul Graham's book On Lisp, this macro is useful for writing other macros:

    (with-gensyms ("FOO-" bar baz zot) ...)

expands to

    (let ((bar (gensym "FOO-BAR-"))
          (baz (gensym "FOO-BAZ-"))
          (zot (gensym "FOO-ZOT-")))
      ...)

### Function `EXT:REMOVE-PLIST`

Similar to &remove; and &remf;, this function removes some properties from a &plist;. It is non-destructive and thus can be used on &rest-amp; arguments to remove some keyword parameters, e.g.,

    (defmacro with-foo ((&key-amp; foo1 foo2) &body-amp; body)
      `(... ,foo1 ... ,foo2 ... ,@body))
    (defmacro with-foo-bar ((&rest-amp; opts &key-amp; bar1 bar2
                             &allow-other-keys-amp;)
                            &body-amp; body)
      `(with-foo (,@(remove-plist opts :bar1 :bar2)
         ... ,bar1 ... ,bar2 ... ,@body)))
    (defun foo-bar ()
      (with-foo-bar (:bar1 1 :foo2 2) ...))

here `WITH-FOO` does not receive the `:BAR1 1` argument from `FOO-BAR`.

### Macros `EXT:WITH-HTML-OUTPUT` and `EXT:WITH-HTTP-OUTPUT`

Defined in &inspect-lisp;, these macros are useful for the rudimentary &http; server defined there.

### Function &open-http; and macro `EXT:WITH-HTTP-INPUT`

Defined in `clhs.lisp`, they allow downloading data over the Internet using the &http; protocol. `(&open-http; url &key-amp; &if-does-not-exist; :LOG)` opens a &sock; connection to the `url` host, sends the `GET` request, and returns two values: the &socket-stream; and content length. `(EXT:WITH-HTTP-INPUT (&var-r; url) &body-amp; body)` binds &var-r; to the &socket-stream; returned by &open-http; and executes the &body-r;. `(EXT:WITH-HTTP-INPUT ((&var-r; &cont-r;) url) &body-amp; body)` additionally binds &cont-r; to the content length.

&open-http; will check &http-proxy; on startup and parse the &env-var; `HTTP_PROXY` if &http-proxy; is &nil;.

The `:LOG` argument binds &http-log-stream;.

### Variable &http-log-stream;

Function &open-http; logs its actions to &http-log-stream; which is initially set to &terminal-io-var;.

### Function &browse-url;

Function `(&browse-url; url &key-amp; &browser-k; &out-k;)` calls a browser on the URL. `browser` (defaults to &browser;) should be a valid keyword in the &browsers; &alist;. &out-k; specifies the stream where the progress messages are printed (defaults to &standard-output-var;).

### Variable &http-proxy;

If you are behind a proxy server, you will need to set &http-proxy; to a &list-t; `(name:password host port)`. By default, the &env-var; `http_proxy` is used, the expected format is `"name:password@host:port"`. If no `#\@` is present, &name-r; and &pass-r; are &nil;. If no `#\:` is present, &pass-r; (or &port-r;) is &nil;.

Use function `(EXT:HTTP-PROXY &optional-amp; (&string-t; (&getenv; "http_proxy")))` to reset &http-proxy;.

### Function &canonicalize;

If you want to canonicalize a &value-r; before further processing it, you can pass it to &canonicalize; together with a &sequence-t; of &function-t;s: `(&canonicalize; &value-r; `functions` &key-amp; (test '&eql;) (max-iter 1024))` will call &func-r;s on &value-r; until it stabilizes under `test` (which should be a valid &hash-table-test;) and return the stabilized value and the number of iterations the stabilization required.

E.g., &new-clx; uses it together with `XLIB:*CANONICALIZE-ENCODING*` to fix the broken encoding names returned by the &X; (e.g., convert `"iso8859-1"` to `"ISO-8859-1"`) before passing them over to &make-encoding;. If you encounter an &encoding; &error-t; in &new-clx;, you can augment this variable to avoid it.

## Customizing &clisp; behavior

The user-customizable variables and functions are located in the package &custom-pac; and thus can be listed using `(&apropos; "" "CUSTOM")`:

- &ansi;
- `CUSTOM:*APPLYHOOK*`
- &apropos-do-more;
- &apropos-matcher;
- `CUSTOM:*BREAK-ON-WARNINGS*`
- &browser;
- &browsers;
- &clhs-root;
- &clhs-root-default;
- &fixnum-char-ansi;
- &compile-warn;
- &compiled-types;
- &curr-lang;
- &def-file-enc;
- &default-float-format;
- &default-tz;
- `CUSTOM:*DEFTYPE-DEPTH-LIMIT*`
- &defun-accept-spelalist;
- &dev-prefix;
- &editor;
- `CUSTOM:*EQ-HASHFUNCTION*`
- `CUSTOM:*EQL-HASHFUNCTION*`
- `CUSTOM:*EQUAL-HASHFUNCTION*`
- `CUSTOM:*ERROR-HANDLER*`
- `CUSTOM:*EVALHOOK*`
- &fill-indent-sexp;
- &fini-hooks;
- &flocont;
- &floratcont;
- &foreign-enc;
- &http-log-stream;
- &http-proxy;
- &impnotes-root;
- &impnotes-root-default;
- &init-hooks;
- `CUSTOM:*INSPECT-BROWSER*`
- `CUSTOM:*INSPECT-FRONTEND*`
- `CUSTOM:*INSPECT-LENGTH*`
- `CUSTOM:*INSPECT-PRINT-LENGTH*`
- `CUSTOM:*INSPECT-PRINT-LEVEL*`
- `CUSTOM:*INSPECT-PRINT-LINES*`
- &libdir;
- &load-comp;
- &load-echo;
- &load-lpt-db;
- &load-obs;
- &load-paths;
- &loop-ansi;
- &pathmerge;
- &misc-enc;
- `CUSTOM:*PACKAGE-TASKS-TREAT-SPECIALLY*`
- &parsename;
- &parsedot;
- &path-enc;
- &phasecont;
- &ppr-first-newline;
- &pr-closure;
- &pr-empty-arr-ansi;
- &pr-indent;
- &pr-sym-pack-prefix;
- &pathprint;
- `CUSTOM:*PRINT-PRETTY-FILL*`
- &pr-rpars;
- &spacecharprint;
- &pr-unreadable-ansi;
- &prompt-body;
- &prompt-break;
- &prompt-finish;
- &prompt-start;
- &prompt-step;
- &reopen;
- &err-pr-bt;
- &savemem-verbose;
- &count-ansi;
- &source-types;
- &suppress-check-redef;
- &suppress-similar-const-redef;
- &sys-pack-list;
- &term-enc;
- &trace-indent;
- &user-commands;
- `CUSTOM:*USER-MAIL-ADDRESS*`
- &warn-fpc;
- &warn-on-hashtable-needing-rehash-after-gc;
- `CUSTOM:*WITH-HTML-OUTPUT-DOCTYPE*`

### Note

Some of these variables are platform-specific.

You should set these variables (and do whatever other customization you see fit) in the file &config-lisp; in the build directory before building &clisp;. Alternatively, after building &clisp;, or if you are using a binary distribution of &clisp;, you can modify &config-lisp;, compile and load it, and then save the &mem-image;. Finally, you can create an &RC-file; which is loaded whenever &clisp; is started.

## Code Walker

You can use function &expand-form; to expand all the macros, &symbol-macro;s, etc, in a single form:

    (&expand-form; '(macrolet ((bar (x) `(print ,x)))
                        (macrolet ((baz (x) `(bar ,x)))
                          (symbol-macrolet ((z 3))
                            (baz z)))))
    (locally (print 3)) the expansion
    &t; indicator: some expansion has actually been done

This is sometimes called a code walker, except that a code walker would probably leave the ¯olet; and &symbol-macrolet; forms intact and just do the expansion.

Function &expand-form; is the exported part of the &clisp; interpreter (AKA &eval;), so it expands forms by assuming the &eval-when; situation `:EXECUTE` and is therefore unsuitable for forms that may later be passed to the compiler:

    (&expand-form; '(&eval-when; (:COMPILE-TOPLEVEL) (foo)))
    &nil; ;
    &t;
    (&expand-form; '(&eval-when; (:LOAD-TOPLEVEL) (foo)))
    &nil; ;
    &t;

## Platform Specific Extensions

## Random Screen Access

&unix-w32-only;

`(`SCREEN:MAKE-WINDOW`)`

- returns a `WINDOW-STREAM`. As long as this stream is open, the terminal is in cbreak/noecho mode. &terminal-io-var; should not be used for input or output during this time. (Use &with-kbd; and &kbd-in; instead.)

`(`SCREEN:WITH-WINDOW` . &body-r;)`

- binds &scr-win;
  to a `WINDOW-STREAM` and executes &body-r;. The stream is guaranteed to be closed when the body is left. During its execution, &terminal-io-var; should not be used, as above.

`(`SCREEN:WINDOW-SIZE` &ws-r;)`

- returns the window's size, as two values: height (= y&sub-max;+1) and width (= x&sub-max;+1).

`(`SCREEN:WINDOW-CURSOR-POSITION` &ws-r;)`

- returns the position of the cursor in the window, as two values: line (≥0, ≤y&sub-max;, 0 means top), column (≥0, ≤x&sub-max;, 0 means left margin).

`(`SCREEN:SET-WINDOW-CURSOR-POSITION` &ws-r; &line-r; &col-r;)`

- sets the position of the cursor in the window.

`(`SCREEN:CLEAR-WINDOW` &ws-r;)`

- clears the window's contents and puts the cursor in the upper left corner.

`(`SCREEN:CLEAR-WINDOW-TO-EOT` &ws-r;)`

- clears the window's contents from the cursor position to the end of window.

`(`SCREEN:CLEAR-WINDOW-TO-EOL` &ws-r;)`

- clears the window's contents from the cursor position to the end of line.

`(`SCREEN:DELETE-WINDOW-LINE` &ws-r;)`

- removes the cursor's line, moves the lines below it up by one line and clears the window's last line.

`(`SCREEN:INSERT-WINDOW-LINE` &ws-r;)`

- inserts a line at the cursor's line, moving the lines below it down by one line.

`(`SCREEN:HIGHLIGHT-ON` &ws-r;)`

- switches highlighted output on.

`(`SCREEN:HIGHLIGHT-OFF` &ws-r;)`

- switches highlighted output off.

`(`SCREEN:WINDOW-CURSOR-ON` &ws-r;)`

- makes the cursor visible, a cursor block in most implementations.

`(`SCREEN:WINDOW-CURSOR-OFF` &ws-r;)`

- makes the cursor invisible, in implementations where this is possible.

## External Modules

&unix-w32-only;

### Modules on &win32;

Everything described in the section will work verbatim on &win32; when using &cygwin; or &mingw;, except for one thing - you will need to replace the `run` extension in &lisp-run; with the &win32; executable extension `exe`.

For historical reasons, all examples appear to assume &unix; and use the `run` file type (extension) for the &clisp; &rt;. This does ¬-e; mean that they will not work on &win32;.

### Overview

External modules are a mechanism to add extensions (written in &c-lang;, for example) to &clisp;. Extending &clisp; using an external module requires creating a &modset; and adding it to an existing &linkset; using &clisp-link; to prodice a new &linkset; which contains the extension.

#### Module Set

A module
is a piece of code (&c-lang; or Lisp) which defines extra (non-core) Lisp objects, symbols and functions. Together with &link-sh;, which describes how to add the module to an existing &clisp;, it comprises a &modset;.

More formally, &modset;
is a directory containing:

&link-sh;

- some &sh; commands, which prepare the directory for linking, and set some &env-var;s, see mod-vars

all other files that define the module functionality

- needed by &link-sh;

In &link-sh; the &modset; directory is referred to as `$modulename/`.

A module name
must consist of the characters `A`-`Z`, `a`-`z`, `_`, `0`-`9`.

The module name clisp is reserved.

#### Linking Set

A &linkset;
is a collection of files (&rt;, &mem-image; &c) which allows performing two major tasks:

1. Running &clisp;: to run a &clisp; contained in some &linkset; &dir-r;, call

       &sh-prompt; &dir-r;/lisp.run &opt-M; &dir-r;/lispinit.mem

   or

       &sh-prompt; clisp &opt-K; &dir-r;

   (recommended, since it also passes `-B` to the &rt;).

2. Adding a &modset; to create a new &linkset; which will contain the module functionality.

The &clisp; build directory contains three &linkset;s in directories `boot`, &base;, and &full;, and a &clisp; installation normally contains two &linkset;s: &base;, and &full;

More formally, a &linkset; is a directory containing at least these files:

&lisp-run;

- the executable &rt;

&lispinit;

- the &mem-image;

`modules.h`

- the list of modules contained in this &linkset;

`modules.o`

- the compiled list of modules contained in this &linkset;

`makevars`

- some &sh; commands, setting the variables

  &varlist-table;

  `CC`

- the &c-lang; compiler

`CPPFLAGS`

- flags for the &c-lang; compiler, when preprocessing or compiling

`CFLAGS`

- flags for the &c-lang; compiler, when compiling or linking

`CLFLAGS`

- flags for the &c-lang; compiler, when linking

`LIBS`

- libraries to use when linking (either present in the &linkset; directory, or system-wide)

`X_LIBS`

- additional &X; libraries to use

`RANLIB`

- the ranlib command

`FILES`

- the list of files needed when linking

all the `FILES`

- listed in `makevars`

#### Operating on &modset;s and &linkset;s

Adding one or several &modset;s to a &linkset;s produces a new &linkset; which contains the functionality of those modules in addition to all the functionality present in the original &linkset;. This is done by the &clisp-link; shell script, installed in &libdir;, so that you can invoke it with

    &sh-prompt; `&clisp-cmd; &opt-b;`/&clisp-link;

#### &clisp-link; commands

create

- The command

      &sh-prompt; &clisp-link; create &mod-r; &file-r; ...

  creates a &modset; in &mod-r; directory which refers (via symbolic links) to files &file-r;... The files are expected to be modules of their own.

add

- The command

      &sh-prompt; &clisp-link; add &source-r; &dest-r; &mod-r; ...

  combines the &linkset; in directory &source-r; and the &module;s in directories &mod-r;... to a new &linkset;, in the directory &dest-r; which is newly created.

run

- The command

      &sh-prompt; &clisp-link; run &source-r; &mod-r; ...

  runs the &linkset; in directory &source-r;, with the &module;s in directories &mod-r;... If &clisp; has been built with the configuration option &with-dynmod;, the loading will be performed dynamically. Otherwise - this is much slower - a temporary &linkset; will be created and deleted afterwards.

#### Requirements

&clisp-link; needs a link kit directory containing:

- `"modules.c"`

- &clisp-h;

&clisp-link; expects to find these files in a subdirectory `linkkit/` of the current directory. or in its own installation directory (i.e., &libdir;). This can be overridden by the &env-var; `CLISP_LINKKIT`.

See also mod-set-example.

#### Module set variables

The following variables should be defined in &link-sh;.

`NEW_FILES`

- the space-separated list of object files that belong to the &modset; and will belong to every new &linkset; linked with this &modset;.

`NEW_LIBS`

- the space-separated list of object files, libraries or &c-lang; compiler switches that need to be passed to the &c-lang; compiler when linking the &lisp-run; belonging to a new &linkset;.

`NEW_MODULES`

- the space-separated list of the module names belonging to the &modset;. Normally, every &c-file; file in the &modset; defines a module of its own. The module name is usually derived from the file name.

&mod-load;

- the space-separated list of Lisp files to load before building the &lispinit; belonging to a new &linkset;.

&mod-preload; (optional)

- the space-separated list of Lisp files to load into an intermediate &lispinit; file, before building the &lispinit; belonging to a new &linkset;. This variable is usually used to create (or unlock) the Lisp &package-t;s which must be present when the new &c-file; files are initialized. E.g., the &def-call-in; functions must reside in already defined packages; see ex-call-in. You can find a live example in `modules/syscalls/preload.lisp` and `modules/syscalls/link.sh.in`.

  If you are unlocking a package, you must also &delete; it from &sys-pack-list; (see image) here and re-add it to &sys-pack-list; in one of the &mod-load; files. See, e.g., `modules/i18n/preload.lisp` and `modules/i18n/link.sh.in`.

### Module initialization

Each module has two initialization functions:

void `module__&name-r;__init_function_1` (struct module_t* module)

- called only once when &clisp; discovers while loading a &mem-image; that there is a module present in the executable (&lisp-run;) which was not present at the time the image was saved. It can be used to create Lisp objects, e.g. functions or keywords, and is indeed used for that purpose by &modprep;.

  You do ¬-e; have to define this function yourself; &modprep; and &ffi-pac; will do that for you.

  If you use &ffi-pac;, `(&c-lines; :init-once ...)` will add code to this function.

  The &package-t;s must already exist and be unlocked, cf. &mod-preload;.

  If you are using &modprep; &and-e; defining your own init-once function, it must call the `module__&name-r;__init_function_1__modprep` function!

void `module__&name-r;__init_function_2` (struct module_t* module)

- called every time &clisp; starts. It can be used to bind names to foreign addresses, since the address will be different in each invocation of &clisp;, and is indeed used for that purpose by &ffi-pac; (e.g., by &def-call-out;). It can also be used to set parameters of the libraries to which the module interfaces, e.g., the &pcre-mod; module sets `pcre_malloc` and `pcre_free`.

  You do ¬-e; have to define this function yourself; &modprep; and &ffi-pac; will do that for you.

  If you use &ffi-pac;, `(&c-lines; :init-always ...)` will add code to this function.

&name-r; is the module name.

See also custom-init-fini.

### Module finalization

Each module has a finalization function

void `module__&name-r;__fini_function` (struct module_t* module)

- called before exiting &clisp;.

  You do ¬-e; have to define this function yourself; &modprep; and &ffi-pac; will do that for you.

  If you use &ffi-pac;, `(&c-lines; :fini ...)` will add code to this function.

&name-r; is the module name.

See also custom-init-fini.

### Function &modinfo;

Function `(&modinfo; &optional-amp; &name-r; `verbose`)` allows one to inquire about what modules are available in the currently running image. When called without arguments, it returns the list of module names, starting with clisp. When &name-r; is supplied and names a module, 3 values are returned - &name-r;, `subr-count`, `object-count`. When `verbose` is non-&nil;, the full list of module lisp function names written in &c-lang; (`Subr`s) and the full list of internal lisp objects available in &c-lang; code are additionally returned for the total of 5 values.

When &name-r; is `:FFI`, returns the list of shared libraries opened using &library-k;. When `verbose` is non-&nil;, return the &alist; of DLL names and all foreign objects associated with it.

### Function &mod-dynload;

Only when compiled &with; configure flag &with-dynmod;.

Dynamic loading does not work on all operating systems (&dlopen; or equivalent is required).

&with-dynmod; precludes some optimizations which are enabled by default.

Function `(&mod-dynload; &filename-r; ({&name-r;}+))` loads a shared object file or library containing a number of named external &clisp; modules.

This facility cannot be used to access arbitrary shared libraries. To do that, use the &library-k; argument to &def-call-out; and &def-c-var; instead.

External modules for &clisp; are shared objects (dynamic libraries) that contain the `module__&name-r;__subr_tab` variable, among others. This serves to register external functions which operate on Lisp-level structures with &clisp;.

To use &dlopen; with modules, you should add `-fPIC` to the module's compilation options. Something like

    &sh-prompt; cc -shared -o &name-r;.so &name-r;.o

may be needed to produce the shared object file.

### Example

#### Create a &modset; with &glibc; bindings

To link in the &ffi-pac; bindings for the &gnu;/&linux; operating system, the following steps are needed. (Step 1 and step 2 need not be executed in this order.)

1. #### Create a new &modset;

       &sh-prompt; &clisp-link; create linux /&path-r;/bindings/linux.c

2. #### Modify the newly created `linux/&link-sh;`

   1. #### add `-lm` to the libraries

      replace

          NEW_LIBS="$file_list"

      with

          NEW_LIBS="$file_list -lm"

   2. #### load `linux.fas` before saving the &mem-image;

      replace

          TO_LOAD=''

      with

          TO_LOAD='/&path-r;/bindings/linux.fas'

3. #### Compile `linux.lisp`, creating `linux.c`

       &sh-prompt; clisp -c /&path-r;/bindings/linux.lisp

4. #### Create a new &linkset;

       &sh-prompt; &clisp-link; add linux base base+linux

5. #### Run and try it

       &sh-prompt; base+linux/lisp.run -M base+linux/lispinit.mem -x '(linux:stat "/tmp")'

### Module tools

There are some tools to facilitate easy module writing.

#### Modprep

If your module is written in &c-lang;, you can pre-process your sources with &modprep; in the &clisp; distribution and define lisp functions with the `DEFUN` macro:

    DEFUN(MY-PACKAGE:MY-FUNCTION-NAME, arg1 arg2 &key-amp; FOO BAR) {
      if (!boundp(STACK_0)) STACK_0 = fixnum(0); /* BAR */
      if (!boundp(STACK_1)) STACK_1 = fixnum(1); /* FOO */
      pushSTACK(`MY-PACKAGE::SOME-SYMBOL`); /* create a symbol in the package */
      pushSTACK(`#(:THIS :IS :A :VECTOR)`); /* some vector, created once */
      pushSTACK(``MY-PACKAGE::MY-FUNCTION-NAME``); /* double `` means FUNCTION */
      VALUES1(listof(7)); /* cons up a new list and clean up the STACK */
    }

Then `(MY-PACKAGE:MY-FUNCTION-NAME 'A 12 :FOO T)` will return (A 12 T 0 MY-PACKAGE::SOME-SYMBOL #(:THIS :IS :A :VECTOR) #<ADD-ON-SYSTEM-FUNCTION MY-PACKAGE:MY-FUNCTION-NAME>) (assuming you &export;ed `MY-FUNCTION-NAME` from MY-PACKAGE).

Note that the arguments are passed on the &STACK; (last argument being the top) which has to be cleaned up before exit.

Another useful macros are:

`DEFVAR`

- create a GC-visible private object

`DEFFLAGSET`

- define a &c-lang; function which will remove several flag arguments from the &STACK; and return the combined flag value

`DEFCHECKER`

- define a map from &cpp; constants to lisp symbols and functions that map between them, checking that the argument is appropriate

See `modules/syscalls/calls.c` and other included modules for more examples and file &modprep; for full documentation.

If you manipulate Lisp objects, you need to watch out for GC-safety.

#### clisp.h

If your module is written in &c-lang;, you will probably want to `#include "clisp.h"` to access &clisp; objects. You will certainly need to read &clisp-h; and some code in included modules, but here are some important hints that you will need to keep in mind:

- Lisp objects have type &object-t;.

- Variables of this type are invalidated by lisp memory allocation (`allocate_*()` functions) - but ¬-e; &c-lang; allocations (&malloc; et al) - and must be saved on the &STACK; using &cpp; macros `pushSTACK()`, `popSTACK()` and `skipSTACK()`.

- Access object slots using the appropriate `TheFoo()` macro, e.g., `TheCons(my_cons)->Car`, but first check the type with `consp()`.

- Arguments are passed on the &STACK;, as illustrated in the above example.

- Wrap your system calls in `begin_system_call()`/`end_system_call()` pairs. These macros, defined in &clisp-h;, save and restore registers used by &clisp; which could be clobbered by a system call.

  If the system call could block (e.g., &read-U;) you need to use `begin_blocking_system_call()` and `end_blocking_system_call()` instead. This will allow other threads to run while yours is inside the system call. This means that &gc;ion could happen while you are inside this system call and, thus, that all objects of type &object-t; are invalidated by the call. See also gc-safety and gc-mt.

#### Exporting

If your module uses &ffi-pac; to interface to a &c-lang; library, you might want to make your module package case-sensitive and use &exporting-lisp; in the &clisp; distribution to make &ffi-pac; forms and &defun;, &defmacro; at al export the symbols they define. See `modules/netica/`, `modules/matlab/` and `modules/bindings/` for examples.

### Trade-offs: &ffi-pac; vs. &c-lang; modules

When deciding how to write a module: whether to use &ffi-pac; or to stick with &c-lang; and &modprep;, one has to take into account several issues:

Speed: &c-lang; wins

- &ffi-pac; has a noticeable overhead: compare `RAWSOCK:HTONS` (defined in `modules/rawsock/rawsock.c`) with

      (&def-call-out; htons (:name "htons") (&library-k; :default)
        (&arguments-k; (s ffi:short)) (&ret-type-k; ffi:short) (&lang-k; :stdc))

  and observe that `RAWSOCK:HTONS` is almost 3 times as fast (this really does compare the &ffi-pac; overhead to the normal lisp function call because `htons` is computationally trivial). This difference will matter only if you call a simple function very many times, in which case it would make sense to put the loop itself into &c-lang;.

Portability: &c-lang; wins

- 1. &ffi-pac; is ¬-e; as widely ported as &clisp;, so it is possible that you will face a platform where &clisp; runs but &ffi-pac; is not present.

  2. It is much easier to handle portability in &c-lang;: observe the alternative implementations of `htonl` et al in `modules/rawsock/rawsock.c`.

  3. Certain &c-lang; structures have different layout on different platforms, and functions may take 64-bit arguments on some platforms and 32-bit arguments on others; so the &ffi-pac; code has to track those differences, while &c-lang; will mostly take care of these things for you.

Code size: &ffi-pac; wins

- You need to type much fewer characters with &ffi-pac;, and, if you use the &library-k; argument to &def-call-out; and &def-c-var;, you do not need to leave your &clisp; session to try out your code. This is a huge advantage for rapid prototyping.

UI: &c-lang; wins

- To produce a nice lispy UI (using &optional-amp; and &key-amp;word arguments etc), you will need to write wrappers to your &foreign-function-t;s, while in &c-lang; you can do that directly. The same goes for polymorphism: accepting different argument types (like, e.g., &resolve-host; does) would require a lisp wrapper for &foreign-function-t;s.

Learning curve: unclear

- If you are comfortable with &c-lang;, you might find the &clisp; &c-lang; module facilities (e.g., &modprep;) very easy to use.

  &clisp; &ffi-pac;, on the other hand, is quite high-level, so, if you are more comfortable with high-level languages, you might find it easier to write &ffi-pac; forms than &c-lang; code.

Safety: unclear

- One can get a segfault either way: if your &def-call-out; form does not describe the &c-lang; function's expectations with respect to the arguments and return values (including &allocation;), you will probably learn that the hard way. If the module is written in &c-lang;, all the opportunities to shoot oneself in the foot (and other body parts) are wide open (although well known to most &c-lang; users). However, with &c-lang;, one has to watch for GC-safety too.

The granularity of the choice is per function: the same module can use both &modprep; and &ffi-pac;.

It is not a good idea to have both `foo.lisp` and `foo.c` files in a module, because if you ever add an &ffi-pac; form to the former, &compile-file-my; will overwrite the latter.

### Modules included in the source distribution

A few modules come with the source distribution of &clisp; (but are not necessarily built in a particular binary distribution).

To use modules, read `unix/INSTALL` and build &clisp; in a directory `build-dir` with, e.g.,

    &sh-prompt; ./configure --with-module=pcre --with-module=clx/new-clx --cbc build-dir

then run it with

    &sh-prompt; ./build-dir/clisp &opt-K; full

This will create a &base; &linkset; with modules &i18n-mod;, ®exp-mod; and &syscalls-mod; (and maybe &readline-mod;); and a &full; &linkset; with modules &new-clx; and &pcre-mod; in addition to the 3 (or 4) &base; modules.

Here we list the included modules by their general theme. See ext-modules for individual module documentation.

#### Base Modules

The default build process includes the following modules in both &base; &and-e; &full; &linkset;s:

&i18n-mod;

- Internationalization of User Programs.

®exp-mod;

- The &posix; Regular Expressions matching, compiling, executing.

&syscalls-mod;

- Use some system calls in a platform-independent way.

&readline-mod; (only when both &readline; and &ffi-pac; are available)

- When &readline; and &ffi-pac; are available, some advanced readline and history features are exported using this module.

The composition of the &full; &linkset; depends on the platform and on the vendor preferences.

#### Database, Directory et al

&gdbm-mod;

- Interface to &gdbm; by Masayuki Onjo masayuki.onjo@gmail.com.

&berkeley-db-mod;

- Berkeley DB interface.

&dirkey-mod;

- Directory Access (LDAP, &win32; registry etc).

&postgresql-mod;

- Access &postgresql-link; from &clisp;.

&oracle-mod;

- Access &oracle-link; RDBMS from &clisp;; by &hin;.

#### Mathematics, Data Mining et al

&libsvm-mod;

- Build Support Vector Machine models using &libsvm-link; inside &clisp;.

&pari-mod;

- Interface to the computer algebra system &pari-link;.

&matlab-mod;

- Do matrix computations via MATLAB.

&netica-mod;

- Work with Bayesian belief networks and influence diagrams using &netica-link;.

#### Matching, File Processing et al

&pcre-mod;

- The &pcre-link; matching, compiling, executing.

&wildcard-mod;

- Shell (&sh;) globbing (&path-match;).

&zlib-mod;

- Compress &vector-t;s using &zlib-link;.

#### Communication, Networking

&rawsock-mod;

- Raw socket access.

&dbus-mod;

- Interface to &d-bus-link;.

&fastcgi-mod;

- Access &fastcgi-link; from &clisp;; by &hin;.

#### Graphics

&clx;

- Call Xlib functions from &clisp;. Two implementations are supplied:

  &mit-clx;, from MIT

- the standard implementation

&new-clx;, by &gilbert-baumann;

- faster, with additional features, but not quite complete yet. Please try it first and use &mit-clx; only if &new-clx; does not work for you. &new-clx; comes with several demos, please try them using

      &sh-prompt; clisp &opt-K; &full; &opt-i; modules/clx/new-clx/demos/clx-demos.lisp &opt-x; '(clx-demos:run-all-demos)'

  and follow the intructions.
This functionality is documented in the manual , also available in the &clisp; source distribution as `modules/clx/clx-manual.tar.gz`.

>k2-mod;

- Use >k; and &glade; to create GUI by James Bailey dgym.bailey@gmail.com.

#### Bindings

Call the operating system functions from &clisp;. The following platforms are supported:

&glibc-mod;

- &linux;/&glibc;

&win32-mod;

- &win32;

#### Toys and Games

&queens-mod;

- Compute the number of solutions to the &n-r;-queens problem on a &n-r;×&n-r; chessboard (a toy example for the users to explore the &clisp; &module; system).

`modules/clx/new-clx/demos/sokoban.lisp`

- a demo which comes with &new-clx;.

## The Foreign Function Call Facility

Many &unix;, &win32; platforms only.

### Introduction

This facility, also known as Foreign Language Interface, allows one to call a function implemented in &c-lang; from inside &clisp; and to do many related things, like inspect and modify foreign memory, define a callback (i.e., make a lisp function available to the &c-lang; world), etc. To use this facility, one writes a foreign function
description into an ordinary Lisp file, which is then compiled and loaded as usual; or just evaluates the appropriate form in the &repl;.

There are two basic ways to do define a foreign function:

1. Use &dlopen; and &dlsym; to get to the location of the function code in a dynamic library. To access this facility, pass the &library-k; option to &def-call-out; and &def-c-var;.

   Unfortunately, this functionality is not available on some operating systems, and, also, it offers only a part of the foreign functionality: &cpp; macros and `inline` functions cannot be accessed this way. On the other hand, this functionality is available in the &repl; and does not require a &c-lang; compiler.

2. Use a somewhat less direct way: when you do not use the &library-k; argument, &compile-file-my; produces a &c-file; file (in addition to a &fasl-file; and a &lib-file;). Then you compile (with a &c-lang; compiler) and link it into &clisp; (statically, linking it into `lisp.a`, or dynamically, loading it into a running &clisp; using &dlopen; and &dlsym;). This way you can use any functionality your foreign library exports, whether using ordinary functions, `inline` functions, or &cpp; macros (see dffi-ex-macro).

All symbols relating to the foreign function interface are exported from the package &ffi-pac;. To use them, `(&use-package; &ffi-pac;)`.

Special &ffi-pac; forms may appear anywhere in the Lisp file.

### Overview

These are the special &ffi-pac; forms. We have taken a pragmatic approach: the only foreign languages we support for now are &c-lang; and &the-ansi; &c-lang;.

Unless specifically noted otherwise, type specification parameters are ¬-e; evaluated, so that they can be compiled by &parse-c-type; into the internal format at macroexpansion time.

### High-level &ffi-pac; forms; &name-r; is any Lisp &symbol-t;; &cname-r; is a &string-t;

`(&def-c-type; &name-r; &optional-amp; &ctype-r;)`

- This form makes &name-r; a shortcut for &ctype-r;. Note that &ctype-r; may already refer to &name-r;. Forward declarations of types are not possible, however.

  When &ctype-r; is omitted, the type is assumed to be an integer, and its size and signedness are determined at link time, e.g., `(&def-c-type; size_t)`.

`(&def-c-var; &name-r; {&option-r;}*)`

- This form defines a &foreign-variable-t;. &name-r; is the Lisp name, a regular Lisp &symbol-t;.

  ### Options for &def-c-var;

  `(&name-k; &cname-r;)`

- specifies the name as seen from &c-lang;, as a &string-t;. If not specified, it is derived from the print name of the Lisp name.

`(&type-k; &ctype-r;)`

- specifies the variable's foreign type.

`(&ro-k; &boolean-t;)`

- If this option is specified and non-&nil;, it will be impossible to change the variable's value from within Lisp (using &setq; or similar).

`(:ALLOC &allocation;)`

- This option can be either &none-k; or &malloc-free-k; and defaults to &none-k;. If it is &malloc-free-k;, any values of type &c-string;, &c-ptr;, &c-ptr-null;, &c-array-ptr; within the foreign value are assumed to be pointers to &malloc;-allocated storage, and when &setq; replaces an old value by a new one, the old storage is freed using &free; and the new storage allocated using &malloc;. If it is &none-k;, &setq; assumes that the pointers point to good storage (not &c-NULL;!) and overwrites the old values by the new ones. This is dangerous (just think of overwriting a string with a longer one or storing some data in a &c-NULL; pointer...) and deprecated.

`(&library-k; &name-r;)`

- Specifies the (optional) dynamic library which contains the variable, the default is set by &default-foreign-library;.

`(&version-k; &version-r;)`

- Specifies the (optional) symbol version in the library (therefore, if &version-k; is supplied, &library-k; must also be supplied)

`(&documentation-k; &string-r;)`

- Specifies the (optional) &variable-doc; documentation.

`(&def-c-const; &name-r; {&option-r;}*)`

- This form defines a Lisp &constant; &name-r; whose value is determined at build time using an internal &foreign-function-t;.

  ### Options for &def-c-const;

  `(&name-k; &cname-r;)`

- specifies the name as seen from &c-lang;, as a &string-t;. If not specified, it is derived from the print name of the Lisp name.

`(&type-k; &ctype-r;)`

- specifies the constant's foreign type, one of

  - `FFI:INT`
  - &c-string;
  - &c-pointer;

`(`:GUARD` &string-r;)`

- specifies the &cpp; check to wrap around &cname-r;, defaults to `"defined(&cname-r;)"`; can be &nil; to omit the test. When the test fails, &name-r; is unbound.

`(&documentation-k; &string-r;)`

- Specifies the (optional) &variable-doc; documentation.

See also dffi-ex-macro.

`(&def-call-out; &name-r; {&option-r;}*)`

- This form defines a named call-out function (a foreign function called from Lisp: control flow temporarily leaves Lisp).

  ### Options for &def-call-out;

  `(&name-k; &cname-r;)`

- Any Lisp function call to `#'&name-r;` is redirected to call the &c-lang; function &cname-r;.

`(&arguments-k; {(&arg-r; &ctype-r; [¶m-mode; [&allocation;]])}*)`

`(&ret-type-k; &ctype-r; [&allocation;])`

- Argument list and return value, see allocation and param-mode.

`(&lang-k; &lang-r;)`

- See c-flavor.

`(:BUILT-IN &boolean-t;)`

- When the function is a &c-lang; built-in, the full prototype will be output (unless suppressed by &ffi-out-fun;).

`(&library-k; &name-r;)`

- Specifies the (optional) dynamic library which contains the function, the default is set by &default-foreign-library;.

`(&version-k; &version-r;)`

- Specifies the (optional) symbol version in the library (therefore, if &version-k; is supplied, &library-k; must also be supplied)

`(&documentation-k; &string-r;)`

- Specifies the (optional) &function-doc; documentation.

See also dffi-functions.

`(&def-call-in; &func-r; {&option-r;}*)`

- This form defines a callback
  - a named call-in function (i.e., a Lisp function called from the foreign language: control flow temporary enters Lisp)

  ### Options for &def-call-in;

  `(&name-k; &cname-r;)`

- Any &c-lang; function call to the &c-lang; function &cname-r; is redirected to call the &cl; function &func-r;, which should be a &funname;.

`(&arguments-k; {(&arg-r; &ctype-r; [¶m-mode; [&allocation;]])}*)`

`(&ret-type-k; &ctype-r; [&allocation;])`

- Argument list and return value, see allocation and param-mode.

`(&lang-k; &lang-r;)`

- See c-flavor.

See also dffi-functions.

`(&open-foreign-library; &name-r;)`

- Open (load) a shared foreign library.

  This is only needed if you want to test for presence of a library without creating a foreign object. When you create a &foreign-variable-t; or a &foreign-function-t; using &def-c-var; or &def-call-out; with a &library-k; argument, the library &name-r; is opened automatically.

`(&close-foreign-library; &name-r;)`

- Close (unload) a shared foreign library (opened by &open-foreign-library; or the &library-k; argument to &def-call-out; or &def-c-var;).

  If you modify your shared library, you need to use close it using &close-foreign-library; first. When you use the &foreign-variable-t; or the &foreign-function-t; which resides in the library &name-r;, it will be re-opened automatically.

`(&default-foreign-library; &lib-name;)`

- This macro sets the default &library-k; argument for &def-call-out; and &def-c-var;. &lib-name; should be &nil; (meaning use the &c-lang; file produced by &compile-file-my;), a &string-t;, or, depending on the underlying &dlsym; or `dlvsym` implementation, &default-k; or `:NEXT`.

  The default is set separately in each &comp-unit;, so, if you are interfacing to a single library, you can set this variable in the beginning of your lisp file and omit the &library-k; argument throughout the file.

`(&def-c-struct; &name-r; (&symbol-r; &ctype-r;)*)`

- This form defines &name-r; to be both a &structure-class; and a foreign &c-lang; type with the given slots. If this class representation overhead is not needed one should consider writing `(&def-c-type; &name-r; (&c-struct; {&list-t; | &vector-t;} (&symbol-r; &ctype-r;)*))` instead. &name-r; is a &symbol-t; (structure name) or a &list-t; whose &first; element is the structure name and the &rest; is options. Two options are supported at this time:

  ### Options for &def-c-struct;

  `:TYPEDEF`

- means that the name of this structure is a &c-lang; type defined with `typedef` elsewhere.

&external-k;

- means that this structure is defined in a &c-lang; header file that you include with, e.g., `(&c-lines; "#include <filename.h>~%")`.

These options determine how the struct is written to the &c-file;.

`(&def-c-enum; &name-r; {&symbol-r; | (&symbol-r; [&value-r;])}*)`

- This form defines &symbol-r;s as constants, similarly to the &c-lang; declaration `enum { &symbol-r; [= &value-r;], ... };`

  You can use `(`FFI:ENUM-FROM-VALUE` &name-r; &value-r;)` and `(`FFI:ENUM-TO-VALUE` &name-r; &symbol-r;)` to convert between the numeric and symbolic representations (of course, the latter function boils down to &symbol-value; plus a check that the &symbol-r; is indeed a constant defined in the &def-c-enum; &name-r;).

`(&c-lines; &fmt-r; {&arg-r;}*)`

- This form outputs the string `(&format; &nil; &fmt-r; {&arg-r;}*)` to the &c-lang; output file's top level. This is usually used to include the relevant header files, see def-c-struct-external and ffi-extern-output.

  When &fmt-r; is not a &string-t;, is should be a &symbol-t;, and then the &string-t; `(&format; &nil; {&arg-r;}*)` is added to the appropriate &c-lang; function:

  `:INIT-ALWAYS`

  `:INIT-ONCE`

- initialization function

`:FINI`

- finalization function

`(&element; &cplace-r; &index1-r; ... &indexn-r;)`

- Array element: If &cplace-r; is of foreign type `(&c-array; &ctype-r; (&dim1-r; ... &dimn-r;))` and 0 ≤ &index1-r; < &dim1-r;, ..., 0 ≤ &indexn-r; < &dimn-r;, this will be the &place; corresponding to `(&aref; &cplace-r; &index1-r; ... &indexn-r;)` or `&cplace-r;[&index1-r;]...[&indexn-r;]`. It is a &place; of type &ctype-r;. If &cplace-r; is of foreign type `(&c-array-max; &ctype-r; &dim-r;)` and 0 ≤ &index-r; < &dim-r;, this will be the &place; corresponding to `(&aref; &cplace-r; &index-r;)` or `&cplace-r;[&index-r;]`. It is a &place; of type &ctype-r;.

`(&deref; &cplace-r;)`

- Dereference pointer: If &cplace-r; is of foreign type `(&c-ptr; &ctype-r;)`, `(&c-ptr-null; &ctype-r;)` or `(&c-pointer; &ctype-r;)`, this will be the &place; the pointer points to. It is a &place; of type &ctype-r;. For `(&c-ptr-null; &ctype-r;)`, the &cplace-r; may not be &c-NULL;.

`(&slot; &cplace-r; &slot-name-r;)`

- Struct or union component: If &cplace-r; is of foreign type `(&c-struct; &class-r; ... (&slot-name-r; &ctype-r;) ...)` or of type `(&c-union; ... (&slot-name-r; &ctype-r;) ...)`, this will be of type &ctype-r;.

`(&cast; &cplace-r; &ctype-r;)`

- Type change: A &place; denoting the same memory locations as the original &cplace-r;, but of type &ctype-r;.

`(&offset; &cplace-r; &offset-r; &ctype-r;)`

- Type change and displacement: return a &place; denoting a memory locations displaced from the original &cplace-r; by an &offset-r; counted in bytes, with type &ctype-r;. This can be used to resize an array, e.g. of &ctype-r; `(&c-array; `uint16` &n-r;)` via `(&offset; &cplace-r; 0 '(&c-array; `uint16` &k-r;))`.

`(&c-var-addr; &cplace-r;)`

- Return the address of &cplace-r; as a Lisp object of type &foreign-address-t;. This is useful as an argument to foreign functions expecting a parameter of &c-lang; type &c-pointer;.

`(&c-var-object; &cplace-r;)`

- Return the &foreign-variable-t; object underlying the &cplace-r;. This is also an acceptable argument type to a &c-pointer; declaration.

`(&typeof; &cplace-r;)`

- returns the &ctype-r; corresponding to the &cplace-r;.

`(&sizeof; &ctype-r;)`

`(&sizeof; &cplace-r;)`

- The first form returns the size and alignment of the &c-lang; type &ctype-r;, measured in bytes.

  The second form returns the size and alignment of the &c-lang; type of &cplace-r;, measured in bytes.

`(&bitsizeof; &ctype-r;)`

`(&bitsizeof; &cplace-r;)`

- The first form returns the size and alignment of the &c-lang; type &ctype-r;, measured in bits.

  The second form returns the size and alignment of the &c-lang; type of &cplace-r;, measured in bits.

`(&foreign-address-unsigned; &f-ent;)`

`(&unsigned-foreign-address; &number-r;)`

- &foreign-address-unsigned; returns the &integer-t; address embodied in the Lisp object of type &foreign-address-t;, &foreign-pointer-t;, &foreign-variable-t; or &foreign-function-t;.

  &unsigned-foreign-address; returns a &foreign-address-t; object pointing to the given &integer-t; address.

`(&foreign-address; &f-ent;)`

- &foreign-address; is both a type name and a selector/constructor function. It is the Lisp object type corresponding to a &c-pointer; external type declaration, e.g. a call-out function with `(&ret-type-k; &c-pointer;)` yields a Lisp object of type &foreign-address-t;.

  The function extracts the object of type &foreign-address-t; living within any &foreign-variable-t; or &foreign-function-t; object. If the &f-ent; already is a &foreign-address-t;, it returns it. If it is a &foreign-pointer-t; (e.g. a base foreign library address), it encapsulates it into a &foreign-address-t; object, as suitable for use with a &c-pointer; external type declaration. It does not construct addresses out of &number-t;s, &unsigned-foreign-address; must be used for that purpose.

`(&foreign-variable; &f-ent; &ctypei-r; &key-amp; &name-r;)`

- This constructor creates a new &foreign-variable-t; from the given &foreign-address-t; or &foreign-variable-t; and the internal &c-lang; type descriptor (as obtained from &parse-c-type;). &name-r;, a &string-t;, is mostly useful for documentation and interactive debugging since it appears in the printed representation of the &foreign-variable-t; object, as in `#<&foreign-variable-t; "foo" #x0ADD4E55>`. In effect, this is similar to &cast; (or rather `(&offset; ... 0 ...)` for places), except that it works with &foreign-address; objects and allows caching of the internal &c-lang; types.

`(&foreign-function; &f-ent; &ctypei-r; &key-amp; &name-r;)`

- This constructor creates a &foreign-function-t; from the given &foreign-address-t; or &foreign-function-t; and the internal &c-lang; type descriptor (as obtained from `(&parse-c-type; '(&c-function; ...))`, in which case it is important to specify the &lang-k; because the expressions are likely to be evaluated at run time, outside the &comp-unit;). The &name-r;, a &string-t;, is mostly useful for documentation and interactive debugging since it appears in the printed representation of the &foreign-function-t; object, e.g., `#<&foreign-function-t; "foo" #x0052B060>`. It is inherited from the given &foreign-function-t; object when available.

  See also dffi-functions.

`(&validp; &f-ent;)`

`(&setf; (&validp; &f-ent;) &value-r;)`

- This predicate returns &nil; if the &f-ent; (e.g. the Lisp equivalent of a &c-pointer;) refers to a pointer which is invalid (e.g., because it comes from a previous Lisp session). It returns &t; if &f-ent; can be used within the current Lisp process (thus it returns &t; for all non-foreign arguments).

  You can invalidate a foreign object using `(&setf; &validp;)`. You cannot resurrect a zombie, nor can you kill a non-foreign object.

`(&foreign-pointer; &f-ent;)`

- &foreign-pointer; returns the &foreign-pointer-t; associated with the Lisp object of type &foreign-address-t;, &foreign-pointer-t;, &foreign-variable-t; or &foreign-function-t;.

`(&set-foreign-pointer; &f-ent; {&f-ent; | ©-k;})`

- &set-foreign-pointer; changes the &foreign-pointer-t; associated with the Lisp object of type &foreign-address-t;, &foreign-variable-t; or &foreign-function-t; to that of the other entity. With ©-k;, a &fresh; &foreign-pointer-t; is allocated. The original &f-ent; still points to the same object and is returned. This is particularly useful with `(&setf; &validp;)`, see ex-dffi-validity.

`(&with-foreign-object; (&var-r; &ctype-r; [`initarg`]) &body-r;)`

`(&with-c-var; (&var-r; &ctype-r; [`initarg`]) &body-r;)`

- These forms allocate space on the &c-lang; execution stack, bind respectively a &foreign-variable-t; object or a local &symbol-macro; to &var-r; and execute &body-r;.

  When `initarg` is not supplied, they allocate space only for `(&sizeof; &ctype-r;)` bytes. This space is filled with zeroes. E.g., using a &ctype-r; of &c-string; or even `(&c-ptr; (&c-array; `uint8` 32))` (!) both allocate space for a single pointer, initialized to &c-NULL;.

  When `initarg` is supplied, they allocate space for an arbitrarily complex set of structures rooted in &ctype-r;. Therefore, &c-array-max;, `#()` and `""` are your friends for creating a pointer to the empty arrays:

      (with-c-var (v '(c-ptr (c-array-max uint8 32)) #())
        (setf (element (deref v) 0) 127) v)

  &ctype-r; is evaluated, making creation of variable sized buffers easy:

      (with-c-var (fv `(c-array uint8 ,(length my-vector)) my-vector)
        (print fv))

`(&foreign-value; &foreign-variable;)`

`(&setf; (&foreign-value; &foreign-variable;) ...)`

- This functions converts the reference to a &c-lang; data structure which the &foreign-variable; describes, to Lisp. Such a reference is typically obtained from &allocate-shallow;, &allocate-deep;, &foreign-allocate; or via a `(&c-pointer; &ctype-r;)` &c-lang; type description. Alternatively, macros like &with-c-place; or &with-c-var; and the concept of foreign &place; hide many uses of this function.

  The &setf; form performs conversion from Lisp to &c-lang;, following to the &foreign-variable;'s type description.

`(&with-foreign-string; (&f-addr; `char-count` `byte-count` &string-r; &key-amp; &encoding-r; `null-terminated-p` &start-r; &end-r;) &body-amp; &body-r;)`

- This forms converts a Lisp &string-r; according to the &encoding-r;, allocating space on the &c-lang; execution stack. &encoding-r; can be any &encoding;, e.g. &utf-16; or &utf-8;, whereas &foreign-enc; must be an &ascii;-compatible encoding.

  &body-r; is then executed with the three variables &f-addr;, `char-count` and `byte-count` respectively bound to an untyped &foreign-address-t; (as known from the &c-pointer; foreign type specification) pointing to the stack location, the number of &character-t;s of the Lisp &string-r; that were considered and the number of &ubyte-8; bytes that were allocated for it on the &c-lang; stack.

  When `null-terminated-p` is true, which is the default, a variable number of zero bytes is appended, depending on the encoding, e.g. 2 for &utf-16;, and accounted for in `byte-count`, and `char-count` is incremented by one.

  The &foreign-address-t; object bound to &f-addr; is invalidated upon the exit from the form.

  A stupid example (a quite costly interface to `mblen`):

      (with-foreign-string (fv elems bytes string
                            :encoding charset:jis... :null-terminated-p nil
                            :end 5)
       (declare (ignore fv elems))
       (format t "This string would take ~D bytes." bytes))

`(&parse-c-type; &ctype-r;)`

`(&deparse-c-type; &ctypei-r;)`

- Convert between the external (&list-t;) and internal (&vector-t;) &c-lang; type representations (used by &describe-my;).

  Although you can memoize a &ctypei-r; (see memoized - but do not expect type redefinitions to work across memoization!), you cannot serialize it (write to disk) because deserialization loses object identity.

`(&allocate-shallow; &ctype-r; &key-amp; &count-k; &ro-k;)`

`(&allocate-deep; &ctype-r; &cont-r; &key-amp; &count-k; &ro-k;)`

`(&foreign-free; &f-ent; &key-amp; :FULL)`

`(&foreign-allocate; &ctypei-r; &key-amp; :INITIAL-CONTENTS &count-k; &ro-k;)`

- Macro &allocate-shallow; allocates `(&sizeof; &ctype-r;)` bytes on the &c-lang; heap and zeroes them out (like `calloc`). When &count-k; is supplied, &ctype-r; is substituted with `(&c-array; &ctype-r; &count-r;)`, except when &ctype-r; is &character-t;, in which case `(&c-array-max; &character-t; &count-r;)` is used instead. When &ro-k; is supplied, the Lisp side is prevented from modifying the memory contents. This can be used as an indication that some foreign side is going to fill this memory (e.g. via &read-U;).

  Returns a &foreign-variable-t; object of the actual &ctype-r;, whose address part points to the newly allocated memory.

  &allocate-deep; will call &c-lang; &malloc; as many times as necessary to build a structure on the &c-lang; heap of the given &ctype-r;, initialized from the given &cont-r;.

  E.g., `(&allocate-deep; '&c-string; "ABCDE")` performs 2 allocations: one for a &c-lang; pointer to a string, another for the contents of that string. This would be useful in conjunction with a `char**` &c-lang; type declaration. `(&allocate-shallow; '&c-string;)` allocates room for a single pointer (probably 4 bytes).

  `(&allocate-deep; '&character-t; "ABCDEF" :count 10)` allocates and initializes room for the type `(&c-array-max; &character-t; 10)`, corresponding to `char*` or, more specifically, `char[10]` in &c-lang;.

  Function &foreign-free; deallocates memory at the address held by the given &f-ent;. If `:FULL` is supplied and the argument is of type &foreign-variable-t;, recursively frees the whole complex structure pointed to by this variable.

  If given a &foreign-function-t; object that corresponds to a &clisp; callback, deallocates it. Callbacks are automatically created each time you pass a Lisp function via the &ffi-pac;.

  Use `(&setf; &validp;)` to disable further references to this address from Lisp. This is currently not done automatically. If the given pointer is already invalid, &foreign-free; (currently) &signal;s an &error-t;. This may change to make it easier to integrate with &finalize;.

  Function &foreign-allocate; is a lower-level interface as it requires an internal &c-lang; type descriptor as returned by &parse-c-type;.

`(&with-c-place; (&var-r; &f-ent;) &body-r;)`

- Create a &place; out of the given &foreign-variable-t; object so operations on places (e.g. &cast;, &deref;, &slot; etc.) can be used within &body-r;. &with-c-var; appears as a composition of &with-foreign-object; and &with-c-place;.

  Such a &place; can be used to access memory referenced by a &f-ent; object:

      (setq foo (allocate-deep '(c-array uint8 3) rgb))
      (with-c-place (place foo) (element place 0))

&ffi-out-fun;

`FFI:*OUTPUT-C-VARIABLES*`

- &clisp; will write the `extern` declarations for foreign functions (defined with &def-call-out;) and foreign variables (defined with &def-c-var;) into the output &c-file; (when the Lisp file is compiled with &compile-file-my;) unless these variables are &nil;. They are &nil; by default, so the `extern` declarations are ¬-e; written; you are encouraged to use &c-lines; to include the appropriate &c-lang; headers. Set these variables to non-&nil; if the headers are not available or not usable.

&ffi-guard;

- When this variable is non-&nil; at &compile-time;, &clisp; will guard the &c-lang; statements in the output file with &cpp; conditionals to take advantage of &autoconf; feature detection. E.g.,

      (&eval-when; (compile) (setq *foreign-guard* t))
      (&def-call-out; some-function (:name "function_name") ...)

  will produce

      # if defined(HAVE_FUNCTION_NAME)
        register_foreign_function((void*)&function_name,"function_name",1024);
      # endif

  and will compile and link on any system.

  This is mostly useful for product delivery when you want your module to build on any system even if some features will not be available.

  &ffi-guard; is initialized to &nil; for backwards compatibility.

&foreign-pointer-info;

- This is an interface to `dladdr` and it returns the 4 fields of `Dl_info` as &mul-val;.

### Low-level &ffi-pac; forms

`(&memory-as; &f-addr; &ctypei-r; &optional-amp; &offset-r;)`

`(&setf; (&memory-as; &f-addr; &ctypei-r; &optional-amp; &offset-r;) &value-r;)`

- This accessor is useful when operating with untyped foreign pointers (&foreign-address-t;) as opposed to typed ones (represented by &foreign-variable-t;). It allows to type and dereference the given pointer without the need to create an object of type &foreign-variable-t;.

  Alternatively, one could use `(&foreign-value; (&foreign-variable; &f-ent; &ctypei-r;))` (also &setf;able).

  Note that &ctypei-r; is the internal representation of a foreign type, thus &parse-c-type; is required with literal names or types, e.g. `(&memory-as; &f-addr; (&parse-c-type; '(&c-array; uint8 3)))` or `(&setf; (&memory-as; &f-addr; (&parse-c-type; 'uint32)) 0)`.

### (Foreign) &c-lang; types

Foreign &c-lang; types are used in the &ffi-pac;. They are ¬-e; regular &cl; types or &clos; classes.

A &ctype-r; is either a predefined &c-lang; type or the name of a type defined by &def-c-type;.

### the predefined &c-lang; types (&ctype-r;)

`simple-c-type`

- the simple &c-lang; types

  Lisp name | Lisp equivalent | &c-lang; equivalent | &ilu; equivalent | Comment
  &nil; | &nil; | `void` |  | as a result type only
  &boolean-t; | &boolean-t; | `int` | `BOOLEAN`
  &character-t; | &character-t; | `char` | `SHORT CHARACTER`
  `char` | &integer-t; | `signed char`
  `uchar` | &integer-t; | `unsigned char`
  `short` | &integer-t; | `short`
  `ushort` | &integer-t; | `unsigned short`
  `int` | &integer-t; | `int`
  `uint` | &integer-t; | `unsigned int`
  `long` | &integer-t; | `long`
  `ulong` | &integer-t; | `unsigned long`
  `uint8` | &ubyte-8; | `uint8` | `BYTE`
  `sint8` | &sbyte-8; | `sint8`
  `uint16` | &ubyte-16; | `uint16` | `SHORT CARDINAL`
  `sint16` | &sbyte-16; | `sint16` | `SHORT INTEGER`
  `uint32` | &ubyte-32; | `uint32` | `CARDINAL`
  `sint32` | &sbyte-32; | `sint32` | `INTEGER`
  `uint64` | `(&unsigned-byte-t; 64)` | `uint64` | `LONG CARDINAL` | does not work on all platforms
  `sint64` | `(&signed-byte-t; 64)` | `sint64` | `LONG INTEGER` | does not work on all platforms
  &single-float-t; | &single-float-t; | `float`
  &double-float-t; | &double-float-t; | `double`

&c-pointer;

- This type corresponds to what &c-lang; calls `void*`, an opaque pointer. When used as an argument, &nil; is accepted as a &c-pointer; and treated as &c-NULL;; when a function wants to return a &c-NULL; &c-pointer;, it actually returns &nil;.

`(&c-pointer; &ctype-r;)`

- This type is equivalent to what &c-lang; calls `&ctype-r; *`: a pointer to a single item of the given &ctype-r;. It differs from `(&c-ptr-null; &ctype-r;)` (see below) in that no conversion to and from Lisp will occur (beyond the usual one of the &c-lang; &c-NULL; pointer to or from Lisp &nil;). Instead, an object of type &foreign-variable-t; is used to represent the foreign &place;. It is assimilable to a typed pointer.

&c-string;

- This type corresponds to what &c-lang; calls `char*`, a zero-terminated string. Its Lisp equivalent is a string, without the trailing zero character.

`(&c-struct; &class-r; (&ident1-r; &ctype1-r;) ... (&identn-r; &ctypen-r;))`

- This type is equivalent to what &c-lang; calls `struct { &ctype1-r; &ident1-r;; ...; &ctypen-r; &identn-r;; }`. Its Lisp equivalent is: if &class-r; is &vector-t;, a &simple-vector-t;; if &class-r; is &list-t;, a &proper-list-glo;; if &class-r; is a symbol naming a structure or &clos; class, an instance of this class, with slots of names &ident1-r;, ..., &identn-r;.

  &class-r; may also be a &cons-t; of a &symbol-t; (as above) and a &list-t; of &def-c-struct; options.

`(&c-union; (&ident1-r; &ctype1-r;) ... (&identn-r; &ctypen-r;))`

- This type is equivalent to what &c-lang; calls `union { &ctype1-r; &ident1-r;; ...; &ctypen-r; &identn-r;; }`. Conversion to and from Lisp assumes that a value is to be viewed as being of &ctype1-r;.

`(&c-array; &ctype-r; &dim1-r;)`

`(&c-array; &ctype-r; (&dim1-r; ... &dimn-r;))`

- This type is equivalent to what &c-lang; calls `&ctype-r; [&dim1-r;] ... [&dimn-r;]`. Note that when an array is passed as an argument to a function in &c-lang;, it is actually passed as a pointer; you therefore have to write `(&c-ptr; (&c-array; ...))` for this argument's type.

`(&c-array-max; &ctype-r; &maxdim-r;)`

- This type is equivalent to what &c-lang; calls `&ctype-r; [&maxdim-r;]`, an array containing up to &maxdim-r; elements. The array is zero-terminated if it contains less than &maxdim-r; elements. Conversion from Lisp of an array with more than &maxdim-r; elements silently ignores the extra elements.

`(&c-function; (&arguments-k; {(&arg-r; `a-c-type` [¶m-mode; [&allocation;]])}*) (&ret-type-k; `r-c-type` [&allocation;]) (&lang-k; &lang-r;))`

- This type designates a &c-lang; function that can be called according to the given prototype `(`r-c-type` (*) (`a-c-type&sub-1;`, ...))`. Conversion between &c-lang; functions and Lisp functions is transparent, and &c-NULL;/&nil; is recognized and accepted.

`(&c-ptr; &ctype-r;)`

- This type is equivalent to what &c-lang; calls `&ctype-r; *`: a pointer to a single item of the given &ctype-r;.

`(&c-ptr-null; &ctype-r;)`

- This type is also equivalent to what &c-lang; calls `&ctype-r; *`: a pointer to a single item of the given &ctype-r;, with the exception that &c-lang; &c-NULL; corresponds to Lisp &nil;.

`(&c-array-ptr; &ctype-r;)`

- This type is equivalent to what &c-lang; calls `&ctype-r; (*)[]`: a pointer to a zero-terminated array of items of the given &ctype-r;.

The conversion of &c-string;, `(&c-array; &character-t; &dim1-r;)`, `(&c-array-max; &character-t; &maxdim-r;)`, `(&c-array-ptr; &character-t;)` is governed by &foreign-enc; and dimensions are given in bytes. The conversion of &character-t;, and as such of `(&c-ptr; &character-t;)`, or `(&c-ptr-null; &character-t;)`, as well as that of multi-dimensional arrays `(&c-array; &character-t; (&dim1-r; ... &dimn-r;))`, are governed by &foreign-enc; if the latter is a &enc1-1;, or by the &ascii; encoding otherwise.

Remember that the &c-lang; type `char` is a numeric type and does not use &character-t; &encoding;s.

### The choice of the &c-lang; flavor

&c-function;, &def-call-in;, &def-call-out; take a &lang-k; argument. The &lang-r; is either &c-k; (denotes K&R &c-lang;) or &stdc-k; (denotes &the-ansi; &c-lang;) or &stdc-sc-k; (denotes &the-ansi; &c-lang; with the stdcall calling convention). It specifies whether the &c-lang; function (caller or callee) has been compiled by a K&R &c-lang; compiler or by an &the-ansi; &c-lang; compiler, and possibly the calling convention.

The default language is set using the macro &default-foreign-language;
. If this macro has not been called in the current &comp-unit; (usually a file), a warning is issued and &stdc-k; is used for the rest of the unit.

### Foreign variables

Foreign variables
are variables whose storage is allocated in the foreign language module. They can nevertheless be evaluated and modified through &setq;, just as normal variables can, except that the range of allowed values is limited according to the variable's foreign type.

#### Equality of foreign values

For a foreign variable &x-r; the form `(&eql; &x-r; &x-r;)` is not necessarily true, since every time &x-r; is evaluated its foreign value is converted to a &fresh; Lisp value. Ergo, `(&setf; (&aref; &x-r; &n-r;) &y-r;)` modifies this &fresh; Lisp value (immediately discarded), ¬-e; the foreign data. Use &element; et al instead, see dffi-places.

Foreign variables are defined using &def-c-var; and &with-c-var;.

### Operations on foreign places

A &foreign-variable-t; &name-r; defined by &def-c-var;, &with-c-var; or &with-c-place; defines a &place;, i.e., a form which can also be used as argument to &setf;. (An lvalue in &c-lang; terminology.) The following operations are available on foreign places:

- &element;
- &deref;
- &slot;
- &cast;
- &offset;
- &c-var-addr;
- &c-var-object;
- &typeof;
- &sizeof;
- &bitsizeof;

### Foreign functions

Foreign functions are functions which are defined in the foreign language. There are named foreign functions
(imported via &def-call-out; or created via &def-call-in;) and anonymous foreign functions
; they arise through conversion of function pointers using &foreign-function;.

A call-out function
is a foreign function called from Lisp: control flow temporarily leaves Lisp. A call-in function
(AKA callback
) is a Lisp function called from the foreign language: control flow temporary enters Lisp.

The following operators define foreign functions:

- &def-call-in;
- &def-call-out;
- &foreign-function;

#### Callbacks and memory management

Callbacks (&c-lang; function calling Lisp function) create so-called trampolines
. A trampoline is a piece of &c-lang; code which knows how to call a particular Lisp function. (That is how all foreign language interfaces work, not just ours). The &c-lang; pointer that the foreign library receives is the pointer to this piece of code. These are ¬-e; subject to &gc;ion, as there is no protocol to tell the garbage collector when a given callback is not needed anymore (unlike with Lisp objects).

With callbacks to named functions (i.e., created by a &def-call-in; form where &func-r; is a &funname;) this is mostly harmless, since &func-r; is unlikely to be redefined.

With callbacks to anonymous functions (i.e., created by &foreign-function; or a &def-call-in; form where &func-r; argument is a &lambda-expr;), this might become an issue when they are produced dynamically and en masse, e.g. inside a loop, so that many trampolines are generated.

You can use &foreign-free; to free the trampoline associated with a &foreign-function-t; object, but when you pass a &lambda-expr; to a &def-call-out; as an argument of type &c-function;, such a trampoline &is-e; allocated, but you do ¬-e; get hold of the associated trampoline object, and thus you cannot (trivially) free it. Thus you may find it easier to create the &foreign-function-t; object first, pass it to the &def-call-out;, and then call &foreign-free; manually.

### Argument and result passing conventions

When passed to and from functions, allocation of arguments and results is handled as follows:

Values of &simple-c-type;, &c-pointer; are passed on the stack, with dynamic extent. The &allocation; is effectively ignored.

Values of type &c-string;, &c-ptr;, &c-ptr-null;, &c-array-ptr; need storage. The &allocation; specifies the allocation policy:

&none-k;

- no storage is allocated.

&alloca-k;

- allocation of storage on the stack, which has dynamic extent.

&malloc-free-k;

- storage will be allocated via &malloc; and released via &free;.

If no &allocation; is specified, the default &allocation; is &none-k; for most types, but &alloca-k; for &c-string; and &c-ptr; and &c-ptr-null; and &c-array-ptr; and for &out-k; arguments. The &malloc-free-k; policy provides the ability to pass arbitrarily nested structures within a single conversion.

### Call-out function arguments:

For arguments passed from Lisp to &c-lang;:

- &malloc-free-k;

- Lisp allocates the storage using &malloc; and never deallocates it. The &c-lang; function is supposed to call &free; when done with it.

&alloca-k;

- Lisp allocates the storage on the stack, with dynamic extent. It is freed when the &c-lang; function returns.

&none-k;

- Lisp assumes that the pointer already points to a valid area of the proper size and puts the result value there.

  This is dangerous and deprecated.

For results passed from &c-lang; to Lisp:

- &malloc-free-k;

- Lisp calls &free; on it when done.

&none-k;

- Lisp does nothing.

### Call-in function arguments:

For arguments passed from &c-lang; to Lisp:

- &malloc-free-k;

- Lisp calls &free; on it when done.

&alloca-k;

&none-k;

- Lisp does nothing.

For results passed from Lisp to &c-lang;:

- &malloc-free-k;

- Lisp allocates the storage using &malloc; and never deallocates it. The &c-lang; function is supposed to call &free; when done with it.

&none-k;

- Lisp assumes that the pointer already points to a valid area of the proper size and puts the result value there.

  This is dangerous and deprecated.

Passing &c-struct;, &c-union;, &c-array;, &c-array-max; values as arguments (not via pointers) is only possible to the extent the &c-lang; compiler supports it. Most &c-lang; compilers do it right, but some &c-lang; compilers (such as &gcc; on hppa, x86_64 and &win32;) have problems with this. The recommended workaround is to pass pointers; this is fully supported. See also clisp-devel (/).

### Parameter Mode

A function parameter's ¶m-mode; may be

&in-k; (means: read-only):

- The caller passes information to the callee.

&out-k; (means: write-only):

- The callee passes information back to the caller on return. When viewed as a Lisp function, there is no Lisp argument corresponding to this, instead it means an additional return value. Requires &allocation; = &alloca-k;.

&in-out-k; (means: read-write):

- Information is passed from the caller to the callee and then back to the caller. When viewed as a Lisp function, the &out-k; value is returned as an additional return value.

The default is &in-k;.

### Examples

#### Simple declarations and access

The &c-lang; declaration

    struct foo {
      int a;
      struct foo * b[100];
    };

corresponds to

    (&def-c-struct; foo
      (a int)
      (b (c-array (c-ptr foo) 100)))

The element access

    struct foo f;
    f.b[7].a

corresponds to

    (declare (type foo f))
    (foo-a (aref (foo-b f) 7)) or
    (slot-value (aref (slot-value f 'b) 7) 'a)

#### External &c-lang; variable and some accesses

    struct bar {
      short x, y;
      char a, b;
      int z;
      struct bar * n;
    };

    extern struct bar * my_struct;

    my_struct->x++;
    my_struct->a = 5;
    my_struct = my_struct->n;

corresponds to

    (&def-c-struct; bar
      (x short)
      (y short)
      (a char)
      (b char) or (b character) if it represents a character, not a number
      (z int)
      (n (c-ptr bar)))

    (&def-c-var; my_struct (:type (c-ptr bar)))

    (setq my_struct (let ((s my_struct)) (incf (slot-value s 'x)) s)) or
    (incf (slot my_struct 'x))
    (setq my_struct (let ((s my_struct)) (setf (slot-value s 'a) 5) s)) or
    (setf (slot my_struct 'a) 5)
    (setq my_struct (slot-value my_struct 'n)) or
    (setq my_struct (deref (slot my_struct 'n)))

#### Calling an external function

On &the-ansi; &c-lang; systems, `stdlib.h` contains the declarations:

    typedef struct {
      int quot;   /* Quotient */
      int rem;    /* Remainder */
    } div_t;
    extern div_t div (int numer, int denom);

This translates to

    (&def-c-struct; (div_t :typedef)
      (quot int)
      (rem int))
    (&default-foreign-language; :stdc)
    (&def-call-out; div (&arguments-k; (numer int) (denom int))
      (&ret-type-k; div_t))

Sample call from within Lisp (after running &clisp-link;):

    (div 20 3)
    #S(DIV_T :QUOT 6 :REM 2)

#### Another example for calling an external function

Suppose the following is defined in a file `cfun.c`:

    struct cfunr { int x; char *s; };
    struct cfunr * cfun (int i,char *s,struct cfunr * r,int a[10]) {
      int j;
      struct cfunr * r2;
      printf("i = %d\n", i);
      printf("s = %s\n", s);
      printf("r->x = %d\n", r->x);
      printf("r->s = %s\n", r->s);
      for (j = 0; j < 10; j++) printf("a[%d] = %d.\n", j, a[j]);
      r2 = (struct cfunr *) malloc (sizeof (struct cfunr));
      r2->x = i+5;
      r2->s = "A C string";
      return r2;
    }

It is possible to call this function from Lisp using the file `callcfun.lisp` (do not call it `cfun.lisp` - &compile-file-my; will overwrite `cfun.c`) whose contents is:

    (&defpackage; "TEST-C-CALL" (:use &cl-pac; &ffi-pac;))
    (&in-package; "TEST-C-CALL")
    (&eval-when; (compile) (setq &ffi-out-fun; t))
    (&def-c-struct; cfunr (x int) (s c-string))
    (&default-foreign-language; :stdc)
    (&def-call-out; cfun (&ret-type-k; (c-ptr cfunr))
      (&arguments-k; (i int)
                  (s c-string)
                  (r (c-ptr cfunr) :in :alloca)
                  (a (c-ptr (c-array int 10)) :in :alloca)))
    (defun call-cfun ()
      (cfun 5 "A Lisp string" (make-cfunr :x 10 :s "Another Lisp string")
            '#(0 1 2 3 4 5 6 7 8 9)))

Use the &module; facility:

    &sh-prompt; &clisp-link; create cfun callcfun.c
    &sh-prompt; cc -O -c cfun.c
    &sh-prompt; cd cfun
    &sh-prompt; ln -s ../cfun.o cfun.o
    Add cfun.o to NEW_LIBS and NEW_FILES in &link-sh;.
    &sh-prompt; cd ..
    &sh-prompt; base/lisp.run -M base/lispinit.mem -c callcfun.lisp
    &sh-prompt; &clisp-link; add base base+cfun cfun
    &sh-prompt; base+cfun/lisp.run -M base+cfun/lispinit.mem -i callcfun
    > (test-c-call::call-cfun)
    i = 5
    s = A Lisp string
    r->x = 10
    r->s = Another Lisp string
    a[0] = 0.
    a[1] = 1.
    a[2] = 2.
    a[3] = 3.
    a[4] = 4.
    a[5] = 5.
    a[6] = 6.
    a[7] = 7.
    a[8] = 8.
    a[9] = 9.
    #S(TEST-C-CALL::CFUNR :X 10 :S "A C string")
    >
    &sh-prompt; rm -r base+cfun

Note that there is a memory leak here: The return value `r2` of `cfun()` is &malloc;ed but never &free;d. Specifying

    (&ret-type-k; (c-ptr cfunr) :malloc-free)

is not an alternative because this would also `free(r2->x)` but `r2->x` is a pointer to static data.

The memory leak can be avoided using

    (&ret-type-k; (c-pointer cfunr))

instead, in conjunction with

    (defun call-cfun ()
      (let ((data (cfun ...)))
        (&unwind-protect; (&foreign-value; data)
          (&foreign-free; data :FULL nil))))

#### Accessing &cpp; macros

Suppose you are interfacing to a library `mylib.so` which defines types, macros and `inline` functions in `mylib.h`:

    #define FOO(x)  .....
    #define BAR ...
    struct zot { ... }
    inline int bar (int x) { ... }

To make them available from &clisp;, write these forms into the lisp file `my.lisp`:

    (&c-lines; "#include <mylib.h>
    int my_foo (int x) { return FOO(x); }
    int my_bar (int x) { return bar(x); }~%")
    (&def-c-const; bar)
    (&def-c-const; zot-size (:name "sizeof(struct zot)") (:guard nil))
    (&def-call-out; my-foo (:name "my_foo") (&arguments-k; (x ffi:int)) (&ret-type-k; ffi:int))
    (&def-call-out; my-bar (:name "my_bar") (&arguments-k; (x ffi:int)) (&ret-type-k; ffi:int))

Compiling this file will produce `my.c` and `my.fas` and you have two options:

1. Compile `my.c` into `my.o` with

       &sh-prompt; gcc -c my.c -lmylib

   and use &clisp-link; to create a new &clisp; &linkset;.

2. Add `(&library-k; "my.dll")` to the &def-call-out; forms, compile `my.c` into `my.so` (or `my.dll` on &win32;) with

       &sh-prompt; gcc -shared -o my.so my.c -lmylib

   and load `my.fas`.

Of course, you could have created `my1.c` containing

    #include <mylib.h>
    int my_foo (int x) { return FOO(x); }
    int my_bar (int x) { return bar(x); }

manually, but &c-lines; allows you to keep the definitions of `my_foo` and `my-foo` close together for easier maintenance.

#### Calling Lisp from &c-lang;

To sort an array of double-floats using the Lisp function &sort; instead of the &c-lang; library function `qsort`, one can use the following interface code `sort1.c`. The main problem is to pass a variable-sized array.

    extern void lispsort_begin (int);
    void* lispsort_function;
    void lispsort_double (int n, double * array) {
      double * sorted_array;
      int i;
      lispsort_begin(n); /* store #'sort2 in lispsort_function */
      sorted_array = ((double * (*) (double *)) lispsort_function) (array);
      for (i = 0; i < n; i++) array[i] = sorted_array[i];
      free(sorted_array);
    }

This is accompanied by `sort2.lisp`:

    (&defpackage; "FFI-TEST" (:use &cl-pac; &ffi-pac;))
    (&in-package; "FFI-TEST")
    (&eval-when; (compile) (setq &ffi-out-fun; t))
    (&def-call-in; lispsort_begin (&arguments-k; (n int))
      (&ret-type-k; nil)
      (&lang-k; :stdc))
    (&def-c-var; lispsort_function (:type c-pointer))
    (defun lispsort_begin (n)
      (setf (cast lispsort_function
                  `(c-function
                     (&arguments-k; (v (c-ptr (c-array double-float ,n))))
                     (&ret-type-k; (c-ptr (c-array double-float ,n))
                                   :malloc-free)))
            #'sort2))
    (defun sort2 (v)
      (declare (type vector v))
      (sort v #'<))

To test this, use the following test file `sorttest.lisp`:

    (&eval-when; (compile) (setq &ffi-out-fun; t))
    (&def-call-out; sort10
      (:name "lispsort_double")
      (&lang-k; :stdc)
      (&arguments-k; (n int)
                  (array (c-ptr (c-array double-float 10)) :in-out)))

Now try

    &sh-prompt; &clisp-link; create sort sort2.c sorttest.c
    &sh-prompt; cc -O -c sort1.c
    &sh-prompt; cd sort
    &sh-prompt; ln -s ../sort1.o sort1.o

Add `sort1.o` to `NEW_LIBS` and `NEW_FILES` in &link-sh;. Create a file `package.lisp` containing the form

    (&make-package; "FFI-TEST" :use '(&cl-pac; &ffi-pac;))

and add `package.lisp` to &mod-preload; in &link-sh;. Proceed:

    &sh-prompt; cd ..
    &sh-prompt; base/lisp.run -M base/lispinit.mem -c sort2.lisp sorttest.lisp
    &sh-prompt; &clisp-link; add base base+sort sort
    &sh-prompt; base+sort/lisp.run -M base+sort/lispinit.mem -i sort2 sorttest
    > (sort10 10 '#(0.501d0 0.528d0 0.615d0 0.550d0 0.711d0
                    0.523d0 0.585d0 0.670d0 0.271d0 0.063d0))
    #(0.063d0 0.271d0 0.501d0 0.523d0 0.528d0 0.55d0 0.585d0 0.615d0 0.67d0 0.711d0)
    &sh-prompt; rm -r base+sort

#### Calling Lisp from &c-lang; dynamically

Create a dynamic library `lispdll` (`#P".dll"` on &win32;, `#P".so"` on &unix;) with the following function:

    typedef int (*LispFunc)(int parameter);
    int CallInFunc(LispFunc f) {
      return f(5)+11;
    }

and call it from Lisp:

    (ffi:def-call-out callout
      (:name "CallInFunc")
      (&library-k; "lispdll.dll")
      (&arguments-k; (function-arg
                   (ffi:c-function (&arguments-k; (number ffi:int))
                                   (&ret-type-k; ffi:int) (&lang-k; :stdc))))
      (&ret-type-k; ffi:int)
      (&lang-k; :stdc))
    (defun f (x) (* x 2))
    F
    (callout #'f)
    21

#### Variable size arguments: calling `gethostname` from &clisp;

The standard &unix; function

       int gethostname
       char* name
       size_t length

follows a typical pattern of &c-lang; out-parameter convention: it expects a pointer to a buffer it is going to fill. So you must view this parameter as either &out-k; or &in-out-k;. Additionally, one must tell the function the size of the buffer. Here &len-r; is just an &in-k; parameter. Sometimes this will be an &in-out-k; parameter, returning the number of bytes actually filled in.

So &name-r; is actually a pointer to an array of up to &len-r; characters, regardless of what the poor `char*` &c-lang; prototype says, to be used like a &c-lang; `string` (&c-NULL;-termination). &unix; specifies that host names are limited to `HOST_NAME_MAX` bytes, which is, of course, system dependent, but it appears that 256 is sufficient.

In the present example, you can use allocation &alloca-k;, like you would do in &c-lang;: stack-allocate a temporary:

    (&def-call-out; gethostname
      (&arguments-k; (name (&c-ptr; (&c-array-max; ffi:char 256))
                        &out-k; &alloca-k;)
                  (length ffi:int))
      (&lang-k; :stdc) (&library-k; :default)
      (&ret-type-k; ffi:int))
    GETHOSTNAME
    (defun myhostname ()
      (multiple-value-bind (success name)&out-k; and &in-out-k; parameters are returned as &mul-val;
          (gethostname 256)
        (if (zerop success) name
          (error "~S: ~S: ~S" 'myhostname (os:errno) (os:strerror)))))See errno
    MYHOSTNAME
    (myhostname)
    #(97 98 97 122 111 110 107)

It is a &simple-vector-t;, not a &string-t;, because the &name-r; argument is an array of `char` (an &integer-t; type, see dffi-types), not `character`.

    (&def-call-out; gethostname
      (&arguments-k; (name (&c-ptr; (&c-array-max; character 256))
                        &out-k; &alloca-k;)
                  (length ffi:int))
      (&lang-k; :stdc) (&library-k; :default)
      (&ret-type-k; ffi:int))
    GETHOSTNAME
    (myhostname)
    "abazonk"

Now we have a different problem: if `gethostname` fails, then the buffer allocated for &name-r; will be filled with garbage, but it will still go through the string conversion &before-e; we can check the `success` status. If &foreign-enc; is &iso-8859-1;, this is not a problem since no real conversion is happening, but with &utf-8; an &error-t; may be &signal;ed. A safe approach is to pass to the foreign function our own stack-allocated buffer, and only convert the buffer to a string when the foreign function succeeds:

    (&def-call-out; gethostname
      (&arguments-k; (name &c-pointer;)
                  (length ffi:int))
      (&lang-k; :stdc) (&library-k; :default)
      (&ret-type-k; ffi:int))
    GETHOSTNAME
    (defun myhostname ()
      (&with-foreign-object; (name '(&c-array-max; character 256))
        (let ((success (gethostname name 256)))
          (if (zerop success) (&foreign-value; name)
            (error "~S: ~S: ~S" 'myhostname (os:errno) (os:strerror))))))
    MYHOSTNAME
    (myhostname)
    "abazonk"

Note that the &type-r; argument of &with-foreign-object; is evaluated, so we do not have to make any assumptions about `HOST_NAME_MAX`:

    (defun myhostname ()
      (let ((host-name-max (os:sysconf :host-name-max)))
        (&with-foreign-object; (name `(&c-array-max; character ,host-name-max))
          (let ((success (gethostname name host-name-max)))
            (if (zerop success) (&foreign-value; name)
              (error "~S: ~S: ~S" 'myhostname (os:errno) (os:strerror)))))))
    MYHOSTNAME
    (myhostname)
    "abazonk"

#### Accessing variables in shared libraries

Suppose one wants to access and modify variables that reside in shared libraries:

    struct bar {
      double x, y;
      double out;
    };

    struct bar my_struct = {10.0, 20.5, 0.0};

    double test_dll(struct bar *ptr)
    {
      return ptr->out = ptr->out + ptr->x + ptr->y;
    }

This is compiled to `libtest.so` (or `libtest.dll`, depending on your platform).

Use the following lisp code:

    (&use-package; &ffi-pac;)

    (&def-c-struct; bar
      (x double-float)
      (y double-float)
      (out double-float))

    (&def-call-out; get-own-c-float
      (&library-k; "libtest.so")
      (&lang-k; :stdc)
      (:name "test_dll")
      (&arguments-k; (ptr c-pointer :in :alloca))
      (&ret-type-k; double-float))

    (&def-c-var; my-c-var (:name "my_struct")
      (&library-k; "libtest.so") (:type (c-ptr bar)))

Note that `get-own-c-float` takes a &c-pointer;, not a `(&c-ptr; bar)` as the argument.

Now you can access call `get-own-c-float` on `my-c-var`:

    (&c-var-addr; my-c-var)
    #<FOREIGN-ADDRESS #x282935D8>
    (get-own-c-float (&c-var-addr; my-c-var))
    30.5d0
    (get-own-c-float (&c-var-addr; my-c-var))
    61.0d0
    (get-own-c-float (&c-var-addr; my-c-var))
    91.5d0
    (get-own-c-float (&c-var-addr; my-c-var))
    122.0d0

#### Controlling validity of resources

&set-foreign-pointer; is useful in conjunction with `(&setf; &validp;)` to limit the extent of external resources. Closing twice can be avoided by checking &validp;. All pointers depending on this resource can be disabled at once upon close by sharing their &foreign-pointer; using &set-foreign-pointer;.

    (&def-c-type; PGconn c-pointer)opaque pointer
    (&def-call-out; PQconnectdb (&ret-type-k; PGconn)
      (&arguments-k; (conninfo c-string)))
    (defun sql-connect (conninfo)
      (let ((conn (PQconnectdb conninfo)))
        (unless conn (error "NULL pointer"))
        ;; may wish to use &finalize; as well
        (&set-foreign-pointer; conn ©-k;)))
    (defun sql-dependent-resource (conn arg1)
      (&set-foreign-pointer; (PQxxx conn arg1) conn))
    (defun sql-close (connection)
      (when (&validp; connection)
        (PQfinish connection)
        (setf (&validp; connection) nil)
        T))

Sharing &foreign-pointer; goes both ways: invalidating the dependent resource will invalidate the primary one.

An alternative approach to resource management, more suitable to non-&ffi-pac; &module;s, is implemented in the &berkeley-db-mod; module, see bdb-close.

#### Floating point arrays

Save this code into `sum.c`:

    double sum (int len, double *vec) {
      int i;
      double s=0;
      for (i=0; i<len; i++) s+= vec[i];
      return s;
    }

and compile it with

    &sh-prompt; gcc -shared -o libsum.so sum.c

Now you can sum doubles:

    (&def-call-out; sum (:name "sum") (&library-k; "libsum.so") (&lang-k; :stdc)
      (&ret-type-k; ffi:double-float)
      (&arguments-k; (len ffi:int) (vec (&c-array-ptr; ffi:double-float))))
    (sum 3 #(1d0 2d0 3d0))
    6d0

#### More examples

You can find more information and examples of the &clisp; &ffi-pac; in the following clisp-list messages:

variable size values

- /

variable length arrays

- / /

Even more examples can be found in the file `tests/ffi.tst` in the &clisp; source distribution.

## Socket Streams

&unix-w32-only;

### Introduction

Sockets are used for interprocess communications by processes running on the same host as well as by processes running on different hosts over a computer network. The most common kind of sockets is Internet stream sockets, and a high-level interface to them is described here. A more low level interface that closely follows the &c-lang; system calls is also available, see rawsock.

Two main varieties of sockets are interfaced to:

active sockets

- correspond to &socket-stream;s which are &bidir-s; &stream-t;s

passive sockets

- correspond to &socket-server;s which are a special kind of objects that are used to allow the other side to initiate interaction with lisp.

#### Lisp &repl; server

Here is a simple lisp &repl; server that waits for a remote connection and evaluates forms read from it:

    (&let; ((server (&sose;)))
      (&format; t "~&Waiting for a connection on ~S:~D~%"
              (&sose-host; server) (&sose-port; server))
      (&unwind-protect;
          ;; infinite loop, terminate with &ctrl-c;
          (&loop; (&with-open-stream; (socket (&so-accept; server))
                  (&multiple-value-bind; (local-host local-port) (&sost-local; socket)
                    (&multiple-value-bind; (remote-host remote-port) (&sost-peer; socket)
                      (&format; &t; "~&Connection: ~S:~D -- ~S:~D~%"
                              remote-host remote-port local-host local-port)))
                  ;; loop is terminated when the remote host closes the connection or on &exit;
                  (&loop; (&when; (&eq; :eof (&so-status; (cons socket :input))) (&return;))
                        (&print; (&eval; (&read; socket)) socket)
                        ;; flush everything left in socket
                        (&loop; :for c = (&read-char-no-hang; socket nil nil) :while c)
                        (&terpri; socket))))
        ;; make sure server is closed
        (&sose-close; server)))

##### This opens a gaping security hole!

Functions like &shell;, &exec;, &run-cmd; will allow the remote host to execute arbitrary code with your permissions. While functions defined in lisp (like &run-cmd;) can be removed (using &fmakunbound;), the built-in functions (like &shell; and &exec;) cannot be permanently removed from the &rt;, and an experienced hacker will be able to invoke them even if you &fmakunbound; their names.

You should limit the socket server to local connections by passing the &string-t; `"127.0.0.1"` as the `:INTERFACE` argument to &sose;.

#### Lisp &http; client

Here are a couple of simple lisp &http; clients that fetch a web page and a binary file, and upload a file:

    (&defun; wget-text (host page file &optional-amp; (port 80))
      ;; &http; requires the &dos-k; &line-term;
      (&with-open-stream; (socket (&so-connect; port host &extfmt; &dos-k;))
         (&format; socket "GET ~A HTTP/1.0~2%" page)
         ;; dump the whole thing - header+data - into the output file
         (&with-open-file; (out file :direction :output)
           (&loop; :for line = (&read-line; socket nil nil) :while line
              :do (&write-line; line out)))))
    (&defun; wget-binary (host page file &optional-amp; (port 80))
      (&with-open-stream; (socket (&so-connect; port host &extfmt; &dos-k;))
        (&format; socket "GET ~A HTTP/1.0~2%" page)
        (&loop; :with content-length :for line = (&read-line; socket nil nil)
          ;; header is separated from the data with a blank line
          :until (&zerop; (&length; line)) :do
          (&when; (&string-eq; line #1="Content-length: " :end1 #2=&sharp-dot;(&length; #1#))
            (&setq; content-length (&parse-integer; line :start #2#))
          ;; this will not work if the server does not supply the content-length header
          :finally (&return; (&let; ((data (&make-array; content-length
                                                   :element-type '&ubyte-8;)))
                               ;; switch to binary i/o on socket
                               (&setf; (&stream-element-type; socket) '&ubyte-8;)
                               ;; read the whole file in one system call
                               (&rd-by-seq; data socket)
                               (&with-open-file; (out file :direction :output
                                                    &eltype; '&ubyte-8;)
                                 ;; write the whole file in one system call
                                 (&wr-by-seq; data out))
                               data))))))
    (&defun; wput (host page file &optional-amp; (port 80))
      (&with-open-stream; (socket (&so-connect; port host &extfmt; &dos-k;))
        (&with-open-file; (in file :direction :inptut &eltype; '&ubyte-8;)
          (&let-star; ((length (&file-length; in))
                 (data (&make-array; length :element-type '&ubyte-8;)))
            ;; some servers may not understand the "Content-length" header
            (&format; socket "PUT ~A HTTP/1.0~%Content-length: ~D~2%" page length)
            (&setf; (&stream-element-type; socket) '&ubyte-8;)
            (&rd-by-seq; data in)
            (&wr-by-seq; data socket)))
        ;; not necessary if the server understands the "Content-length" header
        (&sost-shut; socket :output)
        ;; get the server response
        (&loop; :for line = (&read-line; socket nil nil) :while line :collect line)))

### Socket API Reference

`(&sose; &optional-amp; &port-r; &key-amp; `:INTERFACE` `:BACKLOG`)`

- This function creates a socket an binds a port to the socket. The server exists to watch for client connect attempts. The optional argument is the port to use (non-negative &fixnum-t;). The `:BACKLOG` parameter defines maximum length of queue of pending connections (see `listen`) and defaults to 1. The `:INTERFACE` is either a &string-t;, interpreted as the IP address that will be bound, or a socket, from whose peer the connections will be made. Default is (for backward compatibility) to bind to all local interfaces, but for security reasons it is advisable to bind to loopback `"127.0.0.1"` if you need only local connections.

`(&sose-close; &sose-r;)`

- Closes down the server socket. Just like streams, &socket-server;s are closed at &gc;ion. You should not rely on this however, because &gc;ion times are not deterministic.

`(&sose-host; &sose-r;)`

`(&sose-port; &sose-r;)`

- Returns the host mask indicating which hosts can connect to this server and the port which was bound using &sose;.

`(&so-wait; &sose-r; &timeout-opt;)`

- Wait for a fixed time for a connection on the &sose-r; (a &socket-server;). Without a timeout argument, &so-wait; blocks indefinitely. When timeout is zero, poll. Returns &t; when a connection is available (i.e., &so-accept; will not block) and &nil; on timeout.

`(&so-accept; &sose-r; &key-amp; &eltype; &extfmt; &buffered; &timeout-k;)`

- Creates the server-side &bidir-s; &socket-stream; for the connection. Waits for an attempt to connect to the server for no more than &timeout-k; &seconds; (which may be a non-negative &real-t; or a list `(sec usec)` or a pair `(sec . usec)`). &signal;s an &error-t; if no connection is made in that time.

`(&so-connect; &port-r; &optional-amp; [&host-r;] &key-amp; &eltype; &extfmt; &buffered; &timeout-k;)`

- Attempts to create a client-side &bidir-s; &socket-stream;. Blocks until the server accepts the connection, for no more than &timeout-k; &seconds;. If it is 0, returns immediately and (probably) blocks on the next i/o operation (you can use &so-status; to check whether it will actually block).

`(&so-status; `socket-stream-or-list` &timeout-opt;)`

- Checks whether it is possible to read from or write to a &socket-stream; or whether a connection is available on a &socket-server; without blocking.

  This is similar to &listen;, which checks only one &stream-t; and only for input, and &so-wait;, which works only with &socket-server;s.

  We define &status-r; for a &socket-server; or a &socket-stream; to be &error-k; if any i/o operation will cause an &error-t;.

  Additionally, for a &socket-server;, we define &status-r; to be &t; if a connection is available, i.e., is &so-accept; will not block, and &nil; otherwise.

  Additionally, for a &socket-stream;, we define &status-r; in the given &direction-r; (one of &input-k;, &output-k;, and &io-k;) to be

  &varlist-table;

  ### Possible status values for various directions:

  &input-k; status:

- &varlist-table;

  &nil;

- reading will block

&input-k;

- some input is available

&eof-k;

- the stream has reached its end

&output-k; status:

- &varlist-table;

  &nil;

- writing will block

&output-k;

- output to the stream will not block

&io-k; status:

- output status | input status
  &nil; | &input-k; | &eof-k;
  &nil; | &nil; | &input-k; | &eof-k;
  &output-k; | &output-k; | &io-k; | &append-k;

### Possible values of `socket-stream-or-list`:

&socket-stream; or &socket-server;

- Returns the appropriate status, as defined above (&io-k; status for &socket-stream;)

`(&socket-stream; . &direction-r;)`

- Return the status in the specified direction

a non-empty list of the above

- Return a list of values, one for each element of the argument list (a la &mapcar;)

If you want to avoid &consing; up a &fresh; list, you can make the elements of `socket-stream-or-list` to be `(&sost-r; &direction-r; . &x-r;)` or `(&sose-r; . &x-r;)`. Then &so-status; will destructively modify its argument and replace &x-r; or &nil; with the status and return the modified list. You can pass this modified list to &so-status; again.

The optional arguments specify the timeout. &nil; means wait forever, &zero; means poll.

The second value returned is the number of objects with non-&nil; status, i.e., actionable objects. &so-status; returns either due to a timeout or when this number is positive, i.e., if the timeout was &nil; and &so-status; did return, then the second value is positive (this is the reason &nil; is ¬-e; treated as an empty &list-t;, but as an invalid argument).

Note that &so-status; may &signal; a &stream-error-t;. This happens if the &socket-stream; receives an `RST` packet, see `tests/econnreset.lisp`.

This is the interface to `select` (on some platforms, `poll`), so it will work on any &clisp; &stream-t; which is based on a &file-des;, e.g., &kbd-in; and &file-pipe-socket-s;s, as well as on raw sockets.

`(&sost-host; &sost-r;)`

`(&sost-port; &sost-r;)`

- These two functions return information about the &socket-stream;.

`(&sost-peer; &sost-r; [`do-not-resolve-p`])`

- Given a &socket-stream;, this function returns the name of the host on the opposite side of the connection and its port number; the server-side can use this to see who connected.

  When the optional second argument is non-&nil;, the hostname resolution is disabled and just the IP address is returned, without the FQDNFully Qualified Domain Name.

  &sost-raw-too;

`(&sost-local; &sost-r; [`do-not-resolve-p`])`

- The dual to &sost-peer; - same information, host name and port number, but for the local host. The difference from &sost-host; and &sost-port; is that this function asks the OS (and thus returns the correct trusted values) while the other two are just accessors to the internal data structure, and basically return the arguments given to the function which created the &sost-r;.

  &sost-raw-too;

`(&sost-shut; &sost-r; &direction-r;)`

- Some protocols provide for closing the connection in one &direction-r; using `shutdown`. This function provides an interface to this &unix; system call. &direction-r; should be &input-k; or &output-k;. Note that you should still call &close; after you are done with your &sost-r;; this is best accomplished by using &with-open-stream;.

  All &socket-stream;s are &bidi-s;s (i.e., both &input-stream-p; and &output-stream-p; return &t; for them). &sost-shut; breaks this and turns its argument stream into an &in-s; (if &direction-r; is &output-k;) or &out-s; (if &direction-r; is &input-k;). Thus, the following important invariant is preserved: whenever

  - a &stream-t; is open (i.e., &open-stream-p; returns &t;) and

  - a &stream-t; is an &in-s; (i.e., &input-stream-p; returns &t;)

  the &stream-t; can be read from (e.g., with &read-char; or &read-byte;). &sost-raw-too;

`(&so-opt; &sose-r; &rest-amp; {&option-r;}*)`

- Query and, optionally, set socket options using `getsockopt` and &setsockopt;. An &option-r; is a keyword, optionally followed by the new value. When the new value is not supplied, &setsockopt; is not called. For each option the old (or current, if new value was not supplied) value is returned. E.g., `(&so-opt; &sose-r; :SO-LINGER 1 :SO-RCVLOWAT)` returns 2 values: &nil;, the old value of the `:SO-LINGER` option, and 1, the current value of the `:SO-RCVLOWAT` option.

  &sost-raw-too;

`(&so-handles; &stream-r;)`

- Return the input and output OS &file-des;s of the &stream-r; as &mul-val;. See rawsock.

## Multiple Threads of Execution

Only in &clisp; built &with; compile-time flag `--with-threads`.

This functionality is experimental.

Use it at your own risk.

Discuss it on clisp-devel.

### Introduction

&clisp; uses the OS threads to implement multiple threads of execution. Two flavors are supported: &posix; and &win32;. Both are preemptive.

All symbols are exported from the package &mt-pac;, which has nicknames MT (for MultiThreading) and MP (for MultiProcessing). When this functionality is present, &features-my; contains the symbol `:MT`.

See also gc-mt.

### General principles

#### Parallelizability

A program developed for a single-threaded world which shares no application objects with programs running in other threads must run fine, without problems.

Specfically: if, in a single-threaded world, execution of program A before program B produces semantically the same results as execution of program B before program A, then in a multithreaded world, it is possible to run A and B simultaneously in different threads, and the result will be the same as in the two single-threaded cases (A before B, or B before A).

#### Summary

- If A and B have no common objects, then the implementation ensures that the principle is fulfilled.

- If A and B share some objects, the implementation allows the programs to satisfy the principle with little effort.

#### Special Variable Values

Every &dyn-var; has a global value that can be shared across all &thread;s.

Bindings of &dyn-var;s (via &let;/&let-star;/&multiple-value-bind;) are local to &thread;s, i.e. every &symbol-t; has a different value cell in each &thread;. &symbol-value-thread; can be used to inspect and modify these thread local bindings.

Threads do not inherit dynamic bindings from the parent thread.

Example:

    (defvar *global* 1)create a &glob-var;
    (defun thread-1 ()
       here *global* and (&symbol-value; *global*) will be 1 not 2!
      (setq *global* 5)change the &glob-var; value
      (let ((*global* 10))&thr-var; value is initialized
        (setq *global* 20)&thr-var; value is changed
        &glob-var; value is not accessible here (only via &symbol-value-thread;)
        )
      (setq *global* 30))&glob-var; value is modified again
    (let ((*global* 2))&thr-var; value is initialized
      (&make-thread; #'thread-1))

#### Packages

This information is likely to change in the near future

&package-t; objects have an internal mutex and are locked by &intern; before adding a symbol (if &find-symbol; fails). All modifications of internal package data are guarded by this mutex.

Currently there is one exception: &use-package; is not protected and thus not thread safe.

Symbol look-up within package currently is not guarded! This means that in certain cases &find-symbol; may return &nil; even though the symbol is accessible (this may happen when internal package hash tables are re-hashed at the time &find-symbol; performs look-up).

One crucial implication is that one cannot call &load; from different threads at the same time.

See also mt-clos.

#### &clos;

This information is likely to change in the near future

&clos; is ¬-e; thread-safe. &defclass;, &defgeneric;, &defmethod;, &defstruct; modify &clos; without any locking and may interfere with each other.

It is recommended that all code is &load;ed &before-e; any &thread;s are spawned.

See also mt-package.

#### Hash Tables, Sequences, and other mutable objects

If you want to shoot yourself, it is your responsibility to wear armor.

Nothing is ever locked automatically (automatic locking will impose an unjustifiable penalty on &hash-table-t;s and &sequence-t;s local to threads), so the user must use locks when sharing &hash-table-t;s, &sequence-t;s and user-defined mutable objects between threads.

This approach is consistent with the usual &cl; approach:

> The consequences are undefined when code executed during an object-traversing operation destructively modifies the object in a way that might affect the ongoing traversal operation...

> If an object O1 is used as a key in a hash table H and is then visibly modified with regard to the equivalence test of H, then the consequences are unspecified if O1 is used as a key in further operations on H...

#### Examples of thread-unsafe code

Here are some forms whose results are undefined if two threads evaluate them without locking:

    (&incf; (&gethash; x global-ht 0))see mt-mutable
    (&setf; (&aref; global-array ...) ...)ditto
    (&defmethod; &gf-r; (...) ...)see mt-clos
    (&load; &filename-r;)see mt-package
    (&read-from-string; &string-r;)ditto

### Thread API reference

&thread;

- The type of the object returned by &make-thread;.

  Each &thread; represent a separate computation, executed in parallel to each other.

`(&make-thread; &func-r; &key-amp; &name-k; :INITIAL-BINDINGS :CSTACK-SIZE :VSTACK-SIZE)`

- Start a new named &thread; running &func-r;.

  `:INITIAL-BINDINGS`

- an &alist; of `(&symbol-r; . &form-r;)`. The &form-r;s are &eval;uated in the context of the new thread and &symbol-r;s are bound to the result in the thread before &func-r; is called. The default value is &default-special-bindings;.

`:CSTACK-SIZE`

- the size in bytes of the control (&c-lang;) stack. If 0, the value is decided by the OS.

`:VSTACK-SIZE`

- the size of the &clisp; &STACK; in &object-t;s. The default value is calculated based on the &opt-m; option used when &clisp; was started. If 0, the value will be the same as that of the calling thread.

If &thread; creation fails (e.g., due to a lack of system memory), a &control-error-t; is &signal;ed.

Cf. `pthread_create`.

`(&threadp; &object-r;)`

- Check whether the object is of type &thread;.

`(&thread-yield; &thr;)`

- Relinquish the CPU. The &thr; is placed at the end of the run queue and another thread is scheduled to run.

  Cf. `sched_yield`.

`(&thread-interrupt; &thr; &key-amp; &function-k; &override-k; &arguments-k;)`

- Interrupt the normal execution flow in &thr; and ask it to &apply; &func-r; to &args-r;.

  Use `(&thread-interrupt; &thr; &function-k; &nil;)` to debug &thr; and `(&thread-interrupt; &thr; &function-k; &t;)` to terminate &thr;.

  The &override-k; argument overrides &with-deferred-interrupts; and should be used with extreme care.

  Threads can only be interrupted at a point where the &gc;ion can run, see gc-mt.

  Currently on &win32; blocking I/O cannot be interrupted. The interrupt will be handled after the call returns.

  Thread may be interrupted inside &unwind-protect;'s cleanup forms and on non-local exit from &func-r; - they may not execute entirely. In order to prevent this, &with-deferred-interrupts; is provided.

`(&thread-name; &thr;)`

- Return the &name-r; of the &thr;.

`(&thread-active-p; &thr;)`

- Return &nil; if the thread has already terminated and &t; otherwise.

  By the time this function returns &t;, &thr; may have already terminated anyway.

`(¤t-thread;)`

- Return the &thread; object encapsulating the caller.

`(&list-threads;)`

- Return the &list-t; of all currently running &thread;s.

  By the time this function returns, the set of actually running threads may have a single intersection with the return value - the ¤t-thread;.

&mutex;

- The type of the object return by &make-mutex;.

  This represents a lock, i.e., a way to prevent different threads from doing something at the same time, e.g., modifying the same object.

`(&mutexp; &object-r;)`

- Check whether the object is of type &mutex;.

`(&make-mutex; &key-amp; &name-k; `:RECURSIVE-P`)`

- Create new &mutex; object - not locked by any thread. &name-k; should be a &string-t; describing the mutex (this really helps debugging deadlocks). When `RECURSIVE-P` is non-&nil;, a recursive &mutex; is created i.e., a thread can acquire the mutex repeatedly (and should, of course, release it for each successful acquisition).

  Cf. `pthread_mutex_init`.

`(&mutex-name; &thr;)`

- Return the &name-r; of the &mutex;.

`(&mutex-lock; &mutex-r; &key-amp; &timeout-k;)`

- Acquire the &mutex-r;. If &mutex-r; is locked by another thread, the call blocks and waits up to &timeout-k; &seconds;. If &timeout-k; is not specified, waits forever.

  Return &t; on a successful locking of &mutex-r;, and &nil; on timeout.

  If the calling thread has already acquired &mutex-r;, then

  - if &mutex-r; is recursive, &t; is returned (for each recursive &mutex-lock; there should be a separate &mutex-unlock;);

  - If &mutex-r; is non-recursive an &err-sig; to avoid a deadlock.

  Cf. `pthread_mutex_lock`.

`(&mutex-unlock; &mutex-r;)`

- Release (unlock) &mutex-r;. If the calling thread is not locking &mutex-r;, an &err-sig;.

  Cf. `pthread_mutex_unlock`.

`(&mutex-owner; &mutex-r;)`

- Return the &thread; that owns (locks) &mutex-r;, or &nil; if &mutex-r; is not locked.

  By the time this function returns the &mutex-r; ownership may have changed (unless the owner is the ¤t-thread;). The function is mostly useful for debugging deadlocks.

`(&mutex-recursive-p; &mutex-r;)`

- Return a indicator whether &mutex-r; is recursive.

`(&with-lock; (&mutex-r;) &body-amp; &body-r;)`

- Execute &body-r; with &mutex-r; locked. Upon exit &mutex-r; is released. Return whatever &body-r; returns.

&exemption;

- The type of the object returned by &make-exemption;. These correspond to the &posix; condition variables, see `pthread.h`.

  These objects allow broadcasting state from one &thread; to the others.

`(&exemptionp; &object-r;)`

- Check whether the object is of type &exemption;.

`(&make-exemption; &key-amp; &name-k;)`

- Create a new &exemption; object. &name-k; should be a &string-t; describing the exemption (this really helps debugging deadlocks).

  Cf. `pthread_cond_init`.

`(&exemption-name; &thr;)`

- Return the &name-r; of the &exemp-r;.

`(&exemption-signal; &exemp-r;)`

- Signal &exemp-r; object, i.e. wake up a thread blocked on waiting for &exemp-r;.

  Cf. `pthread_cond_signal`.

`(&exemption-wait; &exemp-r; &mutex-r; &key-amp; &timeout-k;)`

- Wait for another &thread; to call &exemption-signal; or &exemption-broadcast; on &exemp-r;. &mutex-r; should be locked by the caller; otherwise an &err-sig;. The function releases the &mutex-r; and waits for &exemp-r;. On return &mutex-r; is acquired again.

  The function waits up to &timeout-k; &seconds;. If timeout is not specified, waits forever.

  Returns &t; if &exemp-r; was signaled and &nil; on timeout.

  On &posix; it is possible to have spurious wakeups, i.e., this function may return &t; even though no thread called &exemption-broadcast; or &exemption-signal;. Therefore, a common idiom for using this function is `(&loop; :while (some-condition-is-satisfied) :do (&exemption-wait; &exemp-r; &mutex-r;))`.

  Cf. `pthread_cond_wait`.

`(&exemption-broadcast; &exemp-r;)`

- Signal &exemp-r; to all threads waiting for it.

  Cf. `pthread_cond_broadcast`.

`(&y-or-n-p-timeout; seconds default &rest-amp; &args-r;)`

`(&yes-or-no-p-timeout; seconds default &rest-amp; &args-r;)`

- Similar to &y-or-n-p; and &yes-or-no-p;, but use &with-timeout; to return `DEFAULT` when no reply is given within timeout &seconds;.

`(&with-timeout; (seconds &body-amp; timeout-forms) &body-amp; &body-r;)`

- Execute &body-r;. If it does not finish for up to &seconds;, it is interrupted and `timeout-forms` are executed.

  Return the values of the last evaluated form in either &body-r; or `timeout-forms`.

  Since on timeout the current thread is interrupted, special care may be needed for ensuring proper cleanup in &body-r;. See &thread-interrupt; and &with-deferred-interrupts;.

`(&setf; (&symbol-value-thread; &symbol-r; &thr;) &value-r;)`

`(&symbol-value-thread; &symbol-r; &thr;)`

- Access or set the &thr-var; value. When &thr; is &t;, use the ¤t-thread;; if it is &nil;, use the &glob-var; binding. Returns two values: the &symbol-r; binding and an indicator: &nil; if not bound in the &thr;, &t; if bound, and &symbol-t; &makunbound; if the &thr-var; binding was removed with &makunbound;.

&default-special-bindings;

- The default `:INITIAL-BINDINGS` argument of &make-thread;.

`(&with-deferred-interrupts; &body-amp; &body-r;)`

- Defer thread interrupts (but ¬-e; thread preemption) while &body-r; is executed. If there is an interrupt while &body-r; is run, it is queued and will be executed after &body-r; finishes.

  Care is needed if waiting or blocking in &body-r;, since there is no way to interrupt it (in case of a deadlock). The macro was added to avoid partial &unwind-protect;'s cleanup forms evaluation in case they are interrupted with a non-local exit.

## Quickstarting delivery with &clisp;

This section describes three ways to turn &clisp; programs into executable programs, which can be started as quickly as executables written in other languages.

### Summary

&unix;

- &clisp; can act as a script interpreter.

Desktop environments such as &kde;, &gnome;, &macosx; or &win32;.

- Files created with &clisp; can be associated with the &clisp; executable so that clicking on them would make &clisp; execute the appropriate code.

&linux; kernel with `CONFIG_BINFMT_MISC=y`

- Associate the extensions &fasl-file; and &lisp-file; with &clisp;; then you can make the files executable and run them from the command line.

#### Multi-file applications

These three techniques apply to a single &lisp-file; or &fasl-file; file. If your application is made up of several &lisp-file; or &fasl-file; files, you can simply concatenate them (using `cat`) into one file; the techniques then apply to that concatenated file.

#### Lisp-less target

These three techniques assume that the target machine has &clisp; pre-installed and thus you can deliver just your own application, not &clisp; itself. If you want to deliver applications without assuming anything about your target box, you have to resort to creating executable &mem-image;s.

### Scripting with &clisp;

&unix-only;

On &unix;, a text file (&fasl-file; or &lisp-file;) can be made executable by adding a first line of the form

    #!&interp-r; [&interp-args;]

and using `chmod` to make the file executable.

### OS Requirements

&clisp; can be used as a script interpreter under the following conditions:

- The &interp-r; must be the full pathname of &clisp;. The recommended path is `/usr/local/bin/clisp`, and if &clisp; is actually installed elsewhere, making `/usr/local/bin/clisp` be a symbolic link to the real &clisp;.

- The &interp-r; must be a real executable, not a script. Unfortunately, in the binary distributions of &clisp; on Solaris, &clisp-cmd; is a shell script because a &c-lang; compiler cannot be assumed to be installed on this platform. If you do have a &c-lang; compiler installed, build &clisp; from the source yourself; `make install` will install &clisp-cmd; as a real executable.

- On some platforms, the first line which specifies the interpreter is limited in length:

  - max. 32 characters on SunOS 4,

  - max. 80 characters on HP-UX,

  - max. 127 characters on &linux;.

  Characters exceeding this limit are simply cut off by the system. At least 128 characters are accepted on Solaris, IRIX, AIX, OSF/1. There is no workaround: You have to keep the interpreter pathname and arguments short.

- On Solaris and HP-UX, only the first `interpreter-arg` is passed to the &interp-r;. In order to pass more than one option (for example, &opt-M; and &opt-C;) to &clisp;, separate them with no-break spaces instead of normal spaces. (But the separator between &interp-r; and &interp-args; must still be a normal space!) &clisp; will split the &interp-args; both at no-break spaces and at normal spaces.

### Script execution

- The script should contain Lisp forms, except in the `#!` line.

- The file is loaded normally, through the function &load-my; (in particular, the name of the script file, which is `$0` in &sh;, can be found in &load-truename-var; and &load-pathname-var;).

- Before it is loaded, the variable &args; is bound to a &list-t; of &string-t;s, representing the arguments given to the Lisp script (i.e., `$1` in &sh; becomes `(&first; &args;)` etc).

- The standard &unix; i/o facilities (see &stdio;) are used: &standard-input-var; is bound to &stdin;, &standard-output-var; to &stdout;, and &error-output-var; to &stderr;. Note dribble-script.

- The &cont-err;s will be turned into &warning-t;s (using &appease-cerrors;).

- Non-&cont-err;s and &ctrl-c; interrupts will terminate the execution of the Lisp script with an error status (using &exit-on-error;).

- If you wish the script's contents to be compiled during loading, add &opt-C; to the &interp-args;.

See also the manual.

### If nothing works

Another, quite inferior, alternative is to put the following into a file:

    #!/bin/sh
    exec clisp <<EOF
    (lisp-form)
    (another-lisp-form)
    (yet-another-lisp-form)
    EOF

The problem with this approach is that the return values of each form will be printed to &standard-output-var;. Another problem is that no user input will be available.

### Desktop Environments

&win32;, &gnome;, &kde;, &macosx; desktop platforms only.

#### Notations

Although we use &win32;-specific notation, these techniques work on other desktop environments as well.

There are two different ways to make &clisp; executables on desktop platforms.

1. Associate the &mem-file; extension with `c:\clisp\clisp.exe &opt-M; "%s"`.

2. Associate the &fasl-file; extension with `c:\clisp\clisp.exe &opt-i; "%s"` Alternatively, you may want to have a function `main` in your &fasl-file; files and associate the &fasl-file; extension with `c:\clisp\clisp.exe &opt-i; %s &opt-x; (main)`.

Then clicking on the compiled lisp file (with &fasl-file; extension) will load the file (thus executing all the code in the file), while the clicking on a &clisp; &mem-image; (with &mem-file; extension) will start &clisp; with the given &mem-image;.

On &win32;, &clisp; is distributed with a file `src/install.bat`, which runs `src/install.lisp` to create a file `clisp.lnk` on your desktop and also associates &fasl-file;, &lisp-file;, and &mem-file; files with &clisp;.

### Associating extensions with &clisp; via kernel

&linux; platforms only.

You have to build your kernel with `CONFIG_BINFMT_MISC=y` and `CONFIG_PROC_FS=y`. Then you will have a `/proc/sys/fs/binfmt_misc/` directory and you will be able to do (as `root`; you might want to put these lines into `/etc/rc.d/rc.local`):

    # echo ":CLISP:E::fas::/usr/local/bin/clisp:" >> /proc/sys/fs/binfmt_misc/register
    # echo ":CLISP:E::lisp::/usr/local/bin/clisp:" >> /proc/sys/fs/binfmt_misc/register

Then you can do the following:

    &sh-prompt; cat << EOF > hello.lisp
    (print "hello, world!")
    EOF
    &sh-prompt; clisp -c hello.lisp
    ;; Compiling file hello.lisp ...
    ;; Wrote file hello.fas
    0 errors, 0 warnings
    &sh-prompt; chmod +x hello.fas
    &sh-prompt; hello.fas

    "hello, world!"

Please read `/usr/src/linux/Documentation/binfmt_misc.txt` for details.

## Shell, Pipes and Printing

This section describes how &clisp; can invoke external executables and communicate with the resulting processes.

### Shell

&unix-only;

- `(&exec; &program-r; `arg&sub-1;` `arg&sub-2;` ...)` executes an external program. Its name is &program-r; (a full pathname). It is given the &string-t;s `arg&sub-1;`, `arg&sub-2;`, ... as arguments.

&unix-w32-only;

- `(&shell; [&command-r;])` calls the operating system's shell, the value of the &env-var; `SHELL` on &unix; and `COMSPEC` on &win32;. `(&shell;)` calls the shell for interactive use. `(&shell; &command-r;)` calls the shell only for execution of the one given &command-r;.

&unix-w32-only;

- The functions &run-cmd; and &run-prog; are the general interface to &shell; and the above:

  `(&run-cmd; &command-r; &key-amp; `:MAY-EXEC` `:INDIRECTP` &input-k; &output-k; &if-output-exists; &wait-k;)` runs a shell command (including shell built-in commands, like `DIR` on &win32; and `for/do/done` on &unix;).

  `(&run-prog; &program-r; &key-amp; `:MAY-EXEC` `:INDIRECTP` &arguments-k; &input-k; &output-k; &if-output-exists; &wait-k;)` runs an external program.

  &command-r;

- the shell command.

  &unix-only;

- The shell the command is passed to is the value of the &env-var; `SHELL`, which normally is &sh;. The command should be a simple command; a command list should be enclosed in "{ ... ; }" (for &sh;) or "( ... )" (for `/bin/csh`).

&program-r;

- the program. The directories listed in the &env-var; `PATH` will be searched for it.

&arguments-k;

- a list of arguments (&string-t;s) that are given to the program.

&input-k;

- where the program's input is to come from: either `:TERMINAL` (&stdin;, the default) or &stream-k; (a Lisp &stream-t; to be created) or a &path-des; (an input file) or &nil; (no input at all).

&output-k;

- where the program's output is to be sent to: either `:TERMINAL` (&stdout;, the default) or &stream-k; (a Lisp &stream-t; to be created) or a &path-des; (an output file) or &nil; (ignore the output).

&if-output-exists;

- what to do if the &output-k; file already exists. The possible values are &overwrite-k;, &append-k;, &error-k;, with the same meaning as for &open;. The default is &overwrite-k;.

&wait-k;

- whether to wait for program termination or not (this is useful when no i/o to the process is needed); the default is &t;, i.e., synchronous execution.

`:MAY-EXEC`

- pass `exec` to the underlying shell (&unix; only).

`:INDIRECTP`

- use a shell to run the command, e.g., `(&run-prog; "dir" :indirectp &t;)` will run the shell built-in command `DIR`. This argument defaults to &t; for &run-cmd; and to &nil; for &run-prog;. (&win32; only).

If &stream-k; was specified for &input-k; or &output-k;, a Lisp &stream-t; is returned. If &stream-k; was specified for both &input-k; and &output-k;, three Lisp &stream-t;s are returned, as for the function &mk-pipe-io;. Otherwise, the return value depends on the process termination status: if it exited on a signal or a core-dump, the signal number is returned as a negative &integer-t;, else, if it ended normally with 0 exit status, &nil; is returned; otherwise, the status is returned as a positive &integer-t;.

This use of &run-prog; can cause deadlocks, see &mk-pipe-io;.

### Pipes

&unix-w32-only;

- `(&mk-pipe-in; &command-r; &key-amp; &eltype; &extfmt; &buffered;)`

- returns an &in-s; that will supply the output from the execution of the given operating system command.

`(&mk-pipe-out; &command-r; &key-amp; &eltype; &extfmt; &buffered;)`

- returns an &out-s; that will pass its output as input to the execution of the given operating system command.

`(&mk-pipe-io; &command-r; &key-amp; &eltype; &extfmt; &buffered;)`

- returns three values. The &pri-val; is a &bidi-s; that will simultaneously pass its output as input to the execution of the given operating system command and supply the output from this command as input. The second and third value are the &in-s; and the &out-s; that make up the &bidi-s;, respectively.

  These three streams must be closed individually, see &iss052;.

  Improper use of this function can lead to deadlocks. Use it at your own risk!

  A deadlock occurs if the command and your Lisp program either both try to read from each other at the same time or both try to write to each other at the same time.

  To avoid deadlocks, it is recommended that you fix a protocol between the command and your program and avoid any hidden buffering: use &read-char;, &read-char-no-hang;, &listen;, &so-status; instead of &read-line; and &read; on the input side, and complete every output operation by a &finish-output;. The same precautions must apply to the called command as well.

### Printing

The macro &with-print;
:

    (&with-print; (&var-r; [&extfmt;])
      {&declaration-r;}*
      {&form-r;}*)

binds the variable &var-r; to an &out-s; that sends its output to the printer.

## Operating System Environment

Most modern operating systems support &env-var;s that associate strings (variables) with other strings (values). These variables are somewhat similar to the &special-dec; variables in &cl;: their values are inherited by the processes from their parent process.

You can access your OS &env-var;s using the function `(&getenv; &optional-amp; &string-r;)`, where &string-r; is the name of the &env-var;. When &string-r; is omitted or &nil;, all the &env-var;s and their values are returned in an &alist;.

You can change the value of existing &env-var;s or create new ones using `(&setf; (&getenv; &string-r;) &nval-r;)`.