# Creating eggs with extensions needing custom build settings

Sometimes, It’s necessary to provide extra control over how an egg is created. This is commonly true for eggs with extension modules that need to access libraries or include files.

The zc.recipe.egg:custom recipe can be used to define an egg with custom build parameters. The currently defined parameters are:

include-dirs
  A new-line separated list of directories to search for include files.
library-dirs
  A new-line separated list of directories to search for libraries to link with.
rpath
  A new-line separated list of directories to search for dynamic libraries at run time.
define
  A comma-separated list of names of C preprocessor variables to define.
undef
  A comma-separated list of names of C preprocessor variables to undefine.
libraries
  The name of an additional library to link with. Due to limitations in distutils and despite the option name, only a single library can be specified.
link-objects
  The name of an link object to link against. Due to limitations in distutils and despite the option name, only a single link object can be specified.
debug
  Compile/link with debugging information
force
  Forcibly build everything (ignore file timestamps)
compiler
  Specify the compiler type
swig
  The path to the swig executable
swig-cpp
  Make SWIG create C++ files (default is C)
swig-opts
  List of SWIG command line options

In addition, the following options can be used to specify the egg:

egg
  An specification for the egg to be created, to install given as a setuptools requirement string. This defaults to the part name.
find-links
  A list of URLs, files, or directories to search for distributions.
index

  The URL of an index server, or almost any other valid URL. :)

  If not specified, the Python Package Index, http://cheeseshop.python.org/pypi, is used. You can specify an alternate index with this option. If you use the links option and if the links point to the needed distributions, then the index can be anything and will be largely ignored. In the examples, here, we’ll just point to an empty directory on our link server. This will make our examples run a little bit faster.

environment
  The name of a section with additional environment variables. The environment variables are set before the egg is built.

To illustrate this, we’ll define a buildout that builds an egg for a package that has a simple extension module:

    #include <Python.h>
    #include <extdemo.h>

    static PyMethodDef methods[] = {};

    PyMODINIT_FUNC
    initextdemo(void)
    {
        PyObject *m;
        m = Py_InitModule3("extdemo", methods, "");
    #ifdef TWO
        PyModule_AddObject(m, "val", PyInt_FromLong(2));
    #else
        PyModule_AddObject(m, "val", PyInt_FromLong(EXTDEMO));
    #endif
    }

The extension depends on a system-dependent include file, extdemo.h, that defines a constant, EXTDEMO, that is exposed by the extension.

The extension module is available as a source distribution, extdemo-1.4.tar.gz, on a distribution server.

We have a sample buildout that we’ll add an include directory to with the necessary include file:

    >>> mkdir('include')
    >>> write('include', 'extdemo.h',
    ... """
    ... #define EXTDEMO 42
    ... """)

We’ll also update the buildout configuration file to define a part for the egg:

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = extdemo
    ...
    ... [extdemo]
    ... recipe = zc.recipe.egg:custom
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... include-dirs = include
    ...
    ... """ % dict(server=link_server))

    >>> print_(system(buildout), end='')
    Installing extdemo...

We got the zip_safe warning because the source distribution we used wasn’t setuptools based and thus didn’t set the option.

The egg is created in the develop-eggs directory not the eggs directory because it depends on buildout-specific parameters and the eggs directory can be shared across multiple buildouts.

    >>> ls(sample_buildout, 'develop-eggs')
    d  extdemo-1.4-py2.4-unix-i686.egg
    -  zc.recipe.egg.egg-link

Note that no scripts or dependencies are installed. To install dependencies or scripts for a custom egg, define another part and use the zc.recipe.egg recipe, listing the custom egg as one of the eggs to be installed. The zc.recipe.egg recipe will use the installed egg.

Let’s define a script that uses out ext demo:

    >>> mkdir('demo')
    >>> write('demo', 'demo.py',
    ... """
    ... import extdemo, sys
    ... def print_(*args):
    ...     sys.stdout.write(' '.join(map(str, args)) + '\\n')
    ... def main():
    ...     print_(extdemo.val)
    ... """)

    >>> write('demo', 'setup.py',
    ... """
    ... from setuptools import setup
    ... setup(name='demo')
    ... """)

    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... develop = demo
    ... parts = extdemo demo
    ...
    ... [extdemo]
    ... recipe = zc.recipe.egg:custom
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... include-dirs = include
    ...
    ... [demo]
    ... recipe = zc.recipe.egg
    ... eggs = demo
    ...        extdemo
    ... entry-points = demo=demo:main
    ... """ % dict(server=link_server))

    >>> print_(system(buildout), end='')
    Develop: '/sample-buildout/demo'
    Updating extdemo.
    Installing demo.
    Generated script '/sample-buildout/bin/demo'.

When we run the script, we’ll 42 printed:

    >>> print_(system(join('bin', 'demo')), end='')
    42

## Updating

The custom recipe will normally check for new source distributions that meet the given specification. This can be suppressed using the buildout non-newest and offline modes. We’ll generate a new source distribution for extdemo:

    >>> update_extdemo()

If we run the buildout in non-newest or offline modes:

    >>> print_(system(buildout+' -N'), end='')
    Develop: '/sample-buildout/demo'
    Updating extdemo.
    Updating demo.

    >>> print_(system(buildout+' -o'), end='')
    Develop: '/sample-buildout/demo'
    Updating extdemo.
    Updating demo.

We won’t get an update.

    >>> ls(sample_buildout, 'develop-eggs')
    -  demo.egg-link
    d  extdemo-1.4-py2.4-unix-i686.egg
    -  zc.recipe.egg.egg-link

But if we run the buildout in the default on-line and newest modes, we will. This time we also get the test-variable message again, because the new version is imported:

    >>> print_(system(buildout), end='')
    Develop: '/sample-buildout/demo'
    Updating extdemo.
    zip_safe flag not set; analyzing archive contents...
    Updating demo.
    ...

    >>> ls(sample_buildout, 'develop-eggs')
    -  demo.egg-link
    d  extdemo-1.4-py2.4-linux-i686.egg
    d  extdemo-1.5-py2.4-linux-i686.egg
    -  zc.recipe.egg.egg-link

## Controlling the version used

We can specify a specific version using the egg option:

    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... develop = demo
    ... parts = extdemo demo
    ...
    ... [extdemo]
    ... recipe = zc.recipe.egg:custom
    ... egg = extdemo ==1.4
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... include-dirs = include
    ...
    ... [demo]
    ... recipe = zc.recipe.egg
    ... eggs = demo
    ...        extdemo ==1.4
    ... entry-points = demo=demo:main
    ... """ % dict(server=link_server))

    >>> print_(system(buildout+' -D'), end='')
    Develop: '/sample-buildout/demo'
    ...

    >>> ls(sample_buildout, 'develop-eggs')
    -  demo.egg-link
    d  extdemo-1.4-py2.4-linux-i686.egg
    -  zc.recipe.egg.egg-link

### Controlling environment variables

To set additional environment variables, the environment option is used.

Let’s create a recipe which prints out environment variables. We need this to make sure the set environment variables are removed after the egg:custom recipe was run.

    >>> mkdir(sample_buildout, 'recipes')
    >>> write(sample_buildout, 'recipes', 'environ.py',
    ... """
    ... import logging, os, zc.buildout
    ...
    ... class Environ:
    ...
    ...     def __init__(self, buildout, name, options):
    ...         self.name = name
    ...
    ...     def install(self):
    ...         logging.getLogger(self.name).info(
    ...             'test-variable left over: %s' % (
    ...                 'test-variable' in os.environ))
    ...         return []
    ...
    ...     def update(self):
    ...         self.install()
    ... """)
    >>> write(sample_buildout, 'recipes', 'setup.py',
    ... """
    ... from setuptools import setup
    ...
    ... setup(
    ...     name = "recipes",
    ...     entry_points = {'zc.buildout': ['environ = environ:Environ']},
    ...     )
    ... """)

Create our buildout:

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... develop = recipes
    ... parts = extdemo checkenv
    ...
    ... [extdemo-env]
    ... test-variable = foo
    ...
    ... [extdemo]
    ... recipe = zc.recipe.egg:custom
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... include-dirs = include
    ... environment = extdemo-env
    ...
    ... [checkenv]
    ... recipe = recipes:environ
    ...
    ... """ % dict(server=link_server))
    >>> print_(system(buildout), end='')
    Develop: '/sample-buildout/recipes'
    Uninstalling demo.
    Uninstalling extdemo.
    Installing extdemo.
    Have environment test-variable: foo
    zip_safe flag not set; analyzing archive contents...
    Installing checkenv.
    ...

The setup.py also printed out that we have set the environment test-variable to foo. After the buildout the variable is reset to its original value (i.e. removed).

When an environment variable has a value before zc.recipe.egg:custom is run, the original value will be restored:

    >>> import os
    >>> os.environ['test-variable'] = 'bar'
    >>> print_(system(buildout), end='')
    Develop: '/sample-buildout/recipes'
    Updating extdemo.
    Updating checkenv.
    checkenv: test-variable left over: True

    >>> os.environ['test-variable']
    'bar'

Sometimes it is required to prepend or append to an existing environment variable, for instance for adding something to the PATH. Therefore all variables are interpolated with os.environ before the’re set:

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... develop = recipes
    ... parts = extdemo checkenv
    ...
    ... [extdemo-env]
    ... test-variable = foo:%%(test-variable)s
    ...
    ... [extdemo]
    ... recipe = zc.recipe.egg:custom
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... include-dirs = include
    ... environment = extdemo-env
    ...
    ... [checkenv]
    ... recipe = recipes:environ
    ...
    ... """ % dict(server=link_server))
    >>> print_(system(buildout), end='')
    Develop: '/sample-buildout/recipes'
    Uninstalling extdemo.
    Installing extdemo.
    Have environment test-variable: foo:bar
    zip_safe flag not set; analyzing archive contents...
    Updating checkenv.
    ...

    >>> os.environ['test-variable']
    'bar'
    >>> del os.environ['test-variable']

Create a clean buildout.cfg w/o the checkenv recipe, and delete the recipe:

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... develop = recipes
    ... parts = extdemo
    ...
    ... [extdemo]
    ... recipe = zc.recipe.egg:custom
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... include-dirs = include
    ...
    ... """ % dict(server=link_server))
    >>> print_(system(buildout), end='')
    Develop: '/sample-buildout/recipes'
    Uninstalling checkenv.
    Uninstalling extdemo.
    Installing extdemo...

    >>> rmdir(sample_buildout, 'recipes')

# Controlling develop-egg generation

If you want to provide custom build options for a develop egg, you can use the develop recipe. The recipe has the following options:

setup
  The path to a setup script or directory containing a startup script. This is required.
include-dirs
  A new-line separated list of directories to search for include files.
library-dirs
  A new-line separated list of directories to search for libraries to link with.
rpath
  A new-line separated list of directories to search for dynamic libraries at run time.
define
  A comma-separated list of names of C preprocessor variables to define.
undef
  A comma-separated list of names of C preprocessor variables to undefine.
libraries
  The name of an additional library to link with. Due to limitations in distutils and despite the option name, only a single library can be specified.
link-objects
  The name of an link object to link against. Due to limitations in distutils and despite the option name, only a single link object can be specified.
debug
  Compile/link with debugging information
force
  Forcibly build everything (ignore file timestamps)
compiler
  Specify the compiler type
swig
  The path to the swig executable
swig-cpp
  Make SWIG create C++ files (default is C)
swig-opts
  List of SWIG command line options

To illustrate this, we’ll use a directory containing the extdemo example from the earlier section:

    >>> ls(extdemo)
    -  MANIFEST
    -  MANIFEST.in
    -  README
    -  extdemo.c
    -  setup.py

    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... develop = demo
    ... parts = extdemo demo
    ...
    ... [extdemo]
    ... setup = %(extdemo)s
    ... recipe = zc.recipe.egg:develop
    ... include-dirs = include
    ... define = TWO
    ...
    ... [demo]
    ... recipe = zc.recipe.egg
    ... eggs = demo
    ...        extdemo
    ... entry-points = demo=demo:main
    ... """ % dict(extdemo=extdemo))

Note that we added a define option to cause the preprocessor variable TWO to be defined. This will cause the module-variable, ‘val’, to be set with a value of 2.

    >>> print_(system(buildout), end='')
    Develop: '/sample-buildout/demo'
    Uninstalling extdemo.
    Installing extdemo.
    Installing demo.
    ...

Our develop-eggs now includes an egg link for extdemo:

    >>> ls('develop-eggs')
    -  demo.egg-link
    -  extdemo.egg-link
    -  zc.recipe.egg.egg-link

and the extdemo now has a built extension:

    >>> contents = os.listdir(extdemo)
    >>> bool([f for f in contents if f.endswith('.so') or f.endswith('.pyd')])
    True

Because develop eggs take precedence over non-develop eggs, the demo script will use the new develop egg:

    >>> print_(system(join('bin', 'demo')), end='')
    2