3 Writing the Setup Script
Keywords: dist

Subsections

- 3.1 Listing whole packages
- 3.2 Listing individual modules
- 3.3 Describing extension modules
  - 3.3.1 Extension names and packages
  - 3.3.2 Extension source files
  - 3.3.3 Preprocessor options
  - 3.3.4 Library options
  - 3.3.5 Other options
- 3.4 Listing scripts
- 3.5 Listing additional files

#
3 Writing the Setup Script

The setup script is the centre of all activity in building, distributing, and installing modules using the Distutils. The main purpose of the setup script is to describe your module distribution to the Distutils, so that the various commands that operate on your modules do the right thing. As we saw in section 2.1 above, the setup script consists mainly of a call to `setup()`, and most information supplied to the Distutils by the module developer is supplied as keyword arguments to `setup()`.

Here's a slightly more involved example, which we'll follow for the next couple of sections: the Distutils' own setup script. (Keep in mind that although the Distutils are included with Python 1.6 and later, they also have an independent existence so that Python 1.5.2 users can use them to install other module distributions. The Distutils' own setup script, shown here, is used to install the package into Python 1.5.2.)

      #!/usr/bin/env python

      from distutils.core import setup

      setup(name="Distutils",
            version="1.0",
            description="Python Distribution Utilities",
            author="Greg Ward",
            author_email="gward@python.net",
            url="http://www.python.org/sigs/distutils-sig/",
            packages=['distutils', 'distutils.command'],
           )

There are only two differences between this and the trivial one-file distribution presented in section 2.1: more meta-data, and the specification of pure Python modules by package, rather than by module. This is important since the Distutils consist of a couple of dozen modules split into (so far) two packages; an explicit list of every module would be tedious to generate and difficult to maintain.

Note that any pathnames (files or directories) supplied in the setup script should be written using the Unix convention, i.e. slash-separated. The Distutils will take care of converting this platform-neutral representation into whatever is appropriate on your current platform before actually using the pathname. This makes your setup script portable across operating systems, which of course is one of the major goals of the Distutils. In this spirit, all pathnames in this document are slash-separated (MacOS programmers should keep in mind that the absence of a leading slash indicates a relative path, the opposite of the MacOS convention with colons).

This, of course, only applies to pathnames given to Distutils functions. If you, for example, use standard Python functions such as `glob.glob()` or `os.listdir()` to specify files, you should be careful to write portable code instead of hardcoding path separators:

          glob.glob(os.path.join('mydir', 'subdir', '*.html'))
          os.listdir(os.path.join('mydir', 'subdir'))

##
3.1 Listing whole packages

The packages option tells the Distutils to process (build, distribute, install, etc.) all pure Python modules found in each package mentioned in the packages list. In order to do this, of course, there has to be a correspondence between package names and directories in the filesystem. The default correspondence is the most obvious one, i.e. package `distutils` is found in the directory distutils relative to the distribution root. Thus, when you say `packages = ['foo']` in your setup script, you are promising that the Distutils will find a file foo/__init__.py (which might be spelled differently on your system, but you get the idea) relative to the directory where your setup script lives. (If you break this promise, the Distutils will issue a warning but process the broken package anyways.)

If you use a different convention to lay out your source directory, that's no problem: you just have to supply the package_dir option to tell the Distutils about your convention. For example, say you keep all Python source under lib, so that modules in the ``root package'' (i.e., not in any package at all) are right in lib, modules in the `foo` package are in lib/foo, and so forth. Then you would put

      package_dir = {'': 'lib'}

in your setup script. (The keys to this dictionary are package names, and an empty package name stands for the root package. The values are directory names relative to your distribution root.) In this case, when you say `packages = ['foo']`, you are promising that the file lib/foo/__init__.py exists.

Another possible convention is to put the `foo` package right in lib, the `foo.bar` package in lib/bar, etc. This would be written in the setup script as

      package_dir = {'foo': 'lib'}

A `package: dir` entry in the package_dir dictionary implicitly applies to all packages below package, so the `foo.bar` case is automatically handled here. In this example, having `packages = ['foo', 'foo.bar']` tells the Distutils to look for lib/__init__.py and lib/bar/__init__.py. (Keep in mind that although package_dir applies recursively, you must explicitly list all packages in packages: the Distutils will not recursively scan your source tree looking for any directory with an __init__.py file.)

##
3.2 Listing individual modules

For a small module distribution, you might prefer to list all modules rather than listing packages--especially the case of a single module that goes in the ``root package'' (i.e., no package at all). This simplest case was shown in section 2.1; here is a slightly more involved example:

      py_modules = ['mod1', 'pkg.mod2']

This describes two modules, one of them in the ``root'' package, the other in the `pkg` package. Again, the default package/directory layout implies that these two modules can be found in mod1.py and pkg/mod2.py, and that pkg/__init__.py exists as well. And again, you can override the package/directory correspondence using the package_dir option.

##
3.3 Describing extension modules

Just as writing Python extension modules is a bit more complicated than writing pure Python modules, describing them to the Distutils is a bit more complicated. Unlike pure modules, it's not enough just to list modules or packages and expect the Distutils to go out and find the right files; you have to specify the extension name, source file(s), and any compile/link requirements (include directories, libraries to link with, etc.).

All of this is done through another keyword argument to `setup()`, the extensions option. extensions is just a list of `Extension` instances, each of which describes a single extension module. Suppose your distribution includes a single extension, called `foo` and implemented by foo.c. If no additional instructions to the compiler/linker are needed, describing this extension is quite simple:

      Extension("foo", ["foo.c"])

The `Extension` class can be imported from `distutils.core`, along with `setup()`. Thus, the setup script for a module distribution that contains only this one extension and nothing else might be:

      from distutils.core import setup, Extension
      setup(name="foo", version="1.0",
            ext_modules=[Extension("foo", ["foo.c"])])

The `Extension` class (actually, the underlying extension-building machinery implemented by the `build_ext` command) supports a great deal of flexibility in describing Python extensions, which is explained in the following sections.

### 3.3.1 Extension names and packages

The first argument to the `Extension` constructor is always the name of the extension, including any package names. For example,

      Extension("foo", ["src/foo1.c", "src/foo2.c"])

describes an extension that lives in the root package, while

      Extension("pkg.foo", ["src/foo1.c", "src/foo2.c"])

describes the same extension in the `pkg` package. The source files and resulting object code are identical in both cases; the only difference is where in the filesystem (and therefore where in Python's namespace hierarchy) the resulting extension lives.

If you have a number of extensions all in the same package (or all under the same base package), use the ext_package keyword argument to `setup()`. For example,

      setup(...
            ext_package="pkg",
            ext_modules=[Extension("foo", ["foo.c"]),
                         Extension("subpkg.bar", ["bar.c"])]
           )

will compile foo.c to the extension `pkg.foo`, and bar.c to `pkg.subpkg.bar`.

### 3.3.2 Extension source files

The second argument to the `Extension` constructor is a list of source files. Since the Distutils currently only support C/C++ extensions, these are normally C/C++ source files. (Be sure to use appropriate extensions to distinguish C++ source files: .cc and .cpp seem to be recognized by both Unix and Windows compilers.)

However, you can also include SWIG interface (.i) files in the list; the `build_ext` command knows how to deal with SWIG extensions: it will run SWIG on the interface file and compile the resulting C/C++ file into your extension.

** SWIG support is rough around the edges and largely untested; especially SWIG support of C++ extensions! Explain in more detail here when the interface firms up. **

On some platforms, you can include non-source files that are processed by the compiler and included in your extension. Currently, this just means Windows message text (.mc) files and resource definition (.rc) files for Visual C++. These will be compiled to binary resource (.res) files and linked into the executable.

### 3.3.3 Preprocessor options

Three optional arguments to `Extension` will help if you need to specify include directories to search or preprocessor macros to define/undefine: `include_dirs`, `define_macros`, and `undef_macros`.

For example, if your extension requires header files in the include directory under your distribution root, use the `include_dirs` option:

      Extension("foo", ["foo.c"], include_dirs=["include"])

You can specify absolute directories there; if you know that your extension will only be built on Unix systems with X11R6 installed to /usr, you can get away with

      Extension("foo", ["foo.c"], include_dirs=["/usr/include/X11"])

You should avoid this sort of non-portable usage if you plan to distribute your code: it's probably better to write your code to include (e.g.) `<X11/Xlib.h>`.

If you need to include header files from some other Python extension, you can take advantage of the fact that the Distutils install extension header files in a consistent way. For example, the Numerical Python header files are installed (on a standard Unix installation) to /usr/local/include/python1.5/Numerical. (The exact location will differ according to your platform and Python installation.) Since the Python include directory--/usr/local/include/python1.5 in this case--is always included in the search path when building Python extensions, the best approach is to include (e.g.) `<Numerical/arrayobject.h>`. If you insist on putting the Numerical include directory right into your header search path, though, you can find that directory using the Distutils `sysconfig` module:

      from distutils.sysconfig import get_python_inc
      incdir = os.path.join(get_python_inc(plat_specific=1), "Numerical")
      setup(...,
            Extension(..., include_dirs=[incdir]))

Even though this is quite portable--it will work on any Python installation, regardless of platform--it's probably easier to just write your C code in the sensible way.

You can define and undefine pre-processor macros with the `define_macros` and `undef_macros` options. `define_macros` takes a list of `(name, value)` tuples, where `name` is the name of the macro to define (a string) and `value` is its value: either a string or `None`. (Defining a macro `FOO` to `None` is the equivalent of a bare `#define FOO` in your C source: with most compilers, this sets `FOO` to the string `1`.) `undef_macros` is just a list of macros to undefine.

For example:

      Extension(...,
                define_macros=[('NDEBUG', '1')],
                               ('HAVE_STRFTIME', None),
                undef_macros=['HAVE_FOO', 'HAVE_BAR'])

is the equivalent of having this at the top of every C source file:

      #define NDEBUG 1
      #define HAVE_STRFTIME
      #undef HAVE_FOO
      #undef HAVE_BAR

### 3.3.4 Library options

You can also specify the libraries to link against when building your extension, and the directories to search for those libraries. The `libraries` option is a list of libraries to link against, `library_dirs` is a list of directories to search for libraries at link-time, and `runtime_library_dirs` is a list of directories to search for shared (dynamically loaded) libraries at run-time.

For example, if you need to link against libraries known to be in the standard library search path on target systems

      Extension(...,
                libraries=["gdbm", "readline"])

If you need to link with libraries in a non-standard location, you'll have to include the location in `library_dirs`:

      Extension(...,
                library_dirs=["/usr/X11R6/lib"],
                libraries=["X11", "Xt"])

(Again, this sort of non-portable construct should be avoided if you intend to distribute your code.)

** Should mention clib libraries here or somewhere else! **

### 3.3.5 Other options

There are still some other options which can be used to handle special cases.

The extra_objects option is a list of object files to be passed to the linker. These files must not have extensions, as the default extension for the compiler is used.

extra_compile_args and extra_link_args can be used to specify additional command line options for the compiler resp. the linker command line.

export_symbols is only useful on windows, it can contain a list of symbols (functions or variables) to be exported. This option is not needed when building compiled extensions: the `initmodule` function will automatically be added to the exported symbols list by Distutils.

## 3.4 Listing scripts

So far we have been dealing with pure and non-pure Python modules, which are usually not run by themselves but imported by scripts.

Scripts are files containing Python source code, indended to be started from the command line. Distutils doesn't provide much functionality for the scripts: the only support Distutils gives is to adjust the first line of the script if it starts with `#!` and contains the word ``python'' to refer to the current interpreter location.

The scripts option simply is a list of files to be handled in this way.

## 3.5 Listing additional files

The data_files option can be used to specify additional files needed by the module distribution: configuration files, data files, anything which does not fit in the previous categories.

data_files specifies a sequence of (directory, files) pairs in the following way:

      setup(...
            data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']),
                        ('config', ['cfg/data.cfg'])])

Note that you can specify the directory names where the data files will be installed, but you cannot rename the data files themselves.

Each (directory, files) pair in the sequence specifies the installation directory and the files to install there. If directory is a relative path, it is interpreted relative to the installation prefix (Python's `sys.prefix` for pure-Python packages, `sys.exec_prefix` for packages that contain extension modules). Each file name in files is interpreted relative to the setup.py script at the top of the package source distribution. No directory information from files is used to determine the final location of the installed file; only the name of the file is used.

You can specify the data_files options as a simple sequence of files without specifying a target directory, but this is not recommended, and the `install` command will print a warning in this case. To install data files directly in the target directory, an empty string should be given as the directory.