# Repeatable buildouts: controlling eggs used

One of the goals of zc.buildout is to provide enough control to make buildouts repeatable. It should be possible to check the buildout configuration files for a project into a version control system and later use the checked in files to get the same buildout, subject to changes in the environment outside the buildout.

An advantage of using Python eggs is that dependencies of eggs used are automatically determined and used. The automatic inclusion of dependent distributions is at odds with the goal of repeatable buildouts.

To support repeatable buildouts, a versions section can be created with options for each distribution name who’s version is to be fixed. The section can then be specified via the buildout versions option.

To see how this works, we’ll create two versions of a recipe egg:

    >>> mkdir('recipe')
    >>> write('recipe', 'recipe.py',
    ... '''
    ... import sys
    ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n')
    ... class Recipe:
    ...     def __init__(*a): pass
    ...     def install(self):
    ...         print_('recipe v1')
    ...         return ()
    ...     update = install
    ... ''')

    >>> write('recipe', 'setup.py',
    ... '''
    ... from setuptools import setup
    ... setup(name='spam', version='1', py_modules=['recipe'],
    ...       entry_points={'zc.buildout': ['default = recipe:Recipe']},
    ...       )
    ... ''')

    >>> write('recipe', 'README', '')

    >>> print_(system(buildout+' setup recipe bdist_egg'))
    Running setup script 'recipe/setup.py'.
    ...

    >>> rmdir('recipe', 'build')

    >>> write('recipe', 'recipe.py',
    ... '''
    ... import sys
    ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n')
    ... class Recipe:
    ...     def __init__(*a): pass
    ...     def install(self):
    ...         print_('recipe v2')
    ...         return ()
    ...     update = install
    ... ''')

    >>> write('recipe', 'setup.py',
    ... '''
    ... from setuptools import setup
    ... setup(name='spam', version='2', py_modules=['recipe'],
    ...       entry_points={'zc.buildout': ['default = recipe:Recipe']},
    ...       )
    ... ''')

    >>> print_(system(buildout+' setup recipe bdist_egg'))
    Running setup script 'recipe/setup.py'.
    ...

and we’ll configure a buildout to use it:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))

If we run the buildout, it will use version 2:

    >>> print_(system(buildout), end='')
    Getting distribution for 'spam'.
    Got spam 2.
    Installing foo.
    recipe v2

We can specify a versions section that lists our recipe and name it in the buildout section:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ...
    ... [versions]
    ... spam = 1
    ... eggs = 2.2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))

Here we created a versions section listing the version 1 for the spam distribution. We told the buildout to use it by specifying release-1 as in the versions option.

Now, if we run the buildout, we’ll use version 1 of the spam recipe:

    >>> print_(system(buildout), end='')
    Getting distribution for 'spam==1'.
    Got spam 1.
    Uninstalling foo.
    Installing foo.
    recipe v1

Running the buildout in verbose mode will help us get information about versions used. If we run the buildout in verbose mode without specifying a versions section:

    >>> print_(system(buildout+' buildout:versions= -v'), end='')
    Installing 'zc.buildout', 'setuptools'.
    We have a develop egg: zc.buildout 1.0.0.
    We have the best distribution that satisfies 'setuptools'.
    Picked: setuptools = 0.6
    Installing 'spam'.
    We have the best distribution that satisfies 'spam'.
    Picked: spam = 2.
    Uninstalling foo.
    Installing foo.
    recipe v2

We’ll get output that includes lines that tell us what versions buildout chose a for us, like:

    zc.buildout.easy_install.picked: spam = 2

This allows us to discover versions that are picked dynamically, so that we can fix them in a versions section.

If we run the buildout with the versions section:

    >>> print_(system(buildout+' -v'), end='')
    Installing 'zc.buildout', 'setuptools'.
    We have a develop egg: zc.buildout 1.0.0.
    We have the best distribution that satisfies 'setuptools'.
    Picked: setuptools = 0.6
    Installing 'spam'.
    We have the distribution that satisfies 'spam==1'.
    Uninstalling foo.
    Installing foo.
    recipe v1

We won’t get output for the spam distribution, which we didn’t pick, but we will get output for setuptools, which we didn’t specify versions for.

You can request buildout to generate an error if it picks any versions:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... allow-picked-versions = false
    ...
    ... [versions]
    ... spam = 1
    ... eggs = 2.2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='')
    While:
      Installing.
      Checking for upgrades.
      Getting distribution for 'setuptools'.
    Error: Picked: setuptools = 0.6.30

We can name a version something else, if we wish, using the versions option:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... versions = release1
    ...
    ... [release1]
    ... spam = 1
    ... eggs = 2.2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='')
    Updating foo.
    recipe v1

We can also disable checking versions:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... versions =
    ...
    ... [versions]
    ... spam = 1
    ... eggs = 2.2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='')
    Uninstalling foo.
    Installing foo.
    recipe v2

## Easier reporting and managing of versions (new in buildout 2.0)

Since buildout 2.0, the functionality of the buildout-versions extension is part of buildout itself. This makes reporting and managing versions easier.

Buildout picks a version for setuptools and for the tests, we need to grab the version number:

    >>> import pkg_resources
    >>> req = pkg_resources.Requirement.parse('setuptools')
    >>> setuptools_version = pkg_resources.working_set.find(req).version

If you set the `show-picked-versions` option, buildout will print versions it picked at the end of its run:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... show-picked-versions = true
    ...
    ... [versions]
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='')
    Updating foo.
    recipe v2
    Versions had to be automatically picked.
    The following part definition lists the versions picked:
    [versions]
    setuptools = 0.6.99
    spam = 2

When everything is pinned, no output is generated:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... show-picked-versions = true
    ...
    ... [versions]
    ... setuptools = %s
    ... spam = 2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % (join('recipe', 'dist'), setuptools_version))
    >>> print_(system(buildout), end='')
    Updating foo.
    recipe v2

The Python package index is case-insensitive. Both http://pypi.python.org/simple/Django/ and http://pypi.python.org/simple/dJaNgO/ work. And distributions aren’t always naming themselves consistently case-wise. So all version names are normalized and case differences won’t impact the pinning:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... show-picked-versions = true
    ...
    ... [versions]
    ... setuptools = %s
    ... Spam = 2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % (join('recipe', 'dist'), setuptools_version))
    >>> print_(system(buildout), end='')
    Updating foo.
    recipe v2

Sometimes it is handy to have a separate file with versions. This is a regular buildout file with a single `[versions]` section. You include it by extending from that versions file:

    >>> write('my_versions.cfg',
    ... '''
    ... [versions]
    ... setuptools = %s
    ... spam = 2
    ... ''' % setuptools_version)
    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... extends = my_versions.cfg
    ... find-links = %s
    ... show-picked-versions = true
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='')
    Updating foo.
    recipe v2

If not everything is pinned and buildout has to pick versions, you can tell buildout to append the versions to your versions file. It simply appends them at the end.

    >>> write('my_versions.cfg',
    ... '''
    ... [versions]
    ... setuptools = %s
    ... ''' % setuptools_version)
    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... extends = my_versions.cfg
    ... update-versions-file = my_versions.cfg
    ... find-links = %s
    ... show-picked-versions = true
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='')
    Updating foo.
    recipe v2
    Versions had to be automatically picked.
    The following part definition lists the versions picked:
    [versions]
    spam = 2
    Picked versions have been written to my_versions.cfg

The versions file now contains the extra pin:

    >>> print_(open('my_versions.cfg').read())

    ...

    # Added by buildout at YYYY-MM-DD hh:mm:ss.dddddd
    spam = 2

And re-running buildout doesn’t report any picked versions anymore:

    >>> print_(system(buildout), end='')
    Updating foo.
    recipe v2

If you’ve enabled `update-versions-file` but not `show-picked-versions`, buildout will append the versions to your versions file anyway (without printing them to the console):

    >>> write('my_versions.cfg',
    ... '''
    ... [versions]
    ... setuptools = %s
    ... ''' % setuptools_version)
    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... extends = my_versions.cfg
    ... update-versions-file = my_versions.cfg
    ... find-links = %s
    ... show-picked-versions = false
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='')
    Updating foo.
    recipe v2
    Picked versions have been written to my_versions.cfg

The versions file contains the extra pin:

    >>> print_(open('my_versions.cfg').read())

    ...

    # Added by buildout at YYYY-MM-DD hh:mm:ss.dddddd
    spam = 2

Because buildout now includes buildout-versions’ (and part of the older buildout.dumppickedversions’) functionality, it warns if these extensions are configured.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = foo
    ... extensions = buildout-versions
    ...
    ... [foo]
    ... recipe = spam
    ... """)
    >>> print_(system(buildout), end='')
    While:
      Installing.
      Loading extensions.
      Error: Buildout now includes 'buildout-versions'
      (and part of the older 'buildout.dumppickedversions').
      Remove the extension from your configuration and look at the
      'show-picked-versions' option in buildout's documentation.