The easy_install module provides some functions to provide support for egg and script installation. It provides functionality at the python level that is similar to easy_install, with a few exceptions:
The easy_install module provides a function, install, for installing one or more packages and their dependencies. The install function takes 2 positional arguments:
It supports a number of optional keyword arguments:
The URL of an index server, or almost any other valid URL. :)
If not specified, the Python Package Index, http://pypi.python.org/simple/, is used. You can specify an alternate index with this option. If you use the links option and if the links point to the needed distributions, then the index can be anything and will be largely ignored. In the examples, here, we’ll just point to an empty directory on our link server. This will make our examples run a little bit faster.
The install method returns a working set containing the distributions needed to meet the given requirements.
We have a link server that has a number of eggs:
>>> print_(get(link_server), end='')
<html><body>
<a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br>
<a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br>
<a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br>
<a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br>
<a href="demo-0.4c1-py2.4.egg">demo-0.4c1-py2.4.egg</a><br>
<a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br>
<a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br>
<a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br>
<a href="du_zipped-1.0-pyN.N.egg">du_zipped-1.0-pyN.N.egg</a><br>
<a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br>
<a href="index/">index/</a><br>
<a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br>
</body></html>
Let’s make a directory and install the demo egg to it, using the demo:
>>> dest = tmpdir('sample-install')
>>> import zc.buildout.easy_install
>>> ws = zc.buildout.easy_install.install(
... ['demo==0.2'], dest,
... links=[link_server], index=link_server+'index/')
We requested version 0.2 of the demo distribution to be installed into the destination server. We specified that we should search for links on the link server and that we should use the (empty) link server index directory as a package index.
The working set contains the distributions we retrieved.
>>> for dist in ws:
... print_(dist)
demo 0.2
demoneeded 1.1
We got demoneeded because it was a dependency of demo.
And the actual eggs were added to the eggs directory.
>>> ls(dest)
d demo-0.2-py2.4.egg
d demoneeded-1.1-py2.4.egg
If we remove the version restriction on demo, but specify a false value for newest, no new distributions will be installed:
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest, links=[link_server], index=link_server+'index/',
... newest=False)
>>> ls(dest)
d demo-0.2-py2.4.egg
d demoneeded-1.1-py2.4.egg
If we leave off the newest option, we’ll get an update for demo:
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest, links=[link_server], index=link_server+'index/')
>>> ls(dest)
d demo-0.2-py2.4.egg
d demo-0.3-py2.4.egg
d demoneeded-1.1-py2.4.egg
Note that we didn’t get the newest versions available. There were release candidates for newer versions of both packages. By default, final releases are preferred. We can change this behavior using the prefer_final function:
>>> zc.buildout.easy_install.prefer_final(False)
True
The old setting is returned.
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest, links=[link_server], index=link_server+'index/')
>>> for dist in ws:
... print_(dist)
demo 0.4c1
demoneeded 1.2c1
>>> ls(dest)
d demo-0.2-py2.4.egg
d demo-0.3-py2.4.egg
d demo-0.4c1-py2.4.egg
d demoneeded-1.1-py2.4.egg
d demoneeded-1.2c1-py2.4.egg
Let’s put the setting back to the default.
>>> zc.buildout.easy_install.prefer_final(True)
False
We can supply additional distributions. We can also supply specifications for distributions that would normally be found via dependencies. We might do this to specify a specific version.
>>> ws = zc.buildout.easy_install.install(
... ['demo', 'other', 'demoneeded==1.0'], dest,
... links=[link_server], index=link_server+'index/')
>>> for dist in ws:
... print_(dist)
demo 0.3
other 1.0
demoneeded 1.0
>>> ls(dest)
d demo-0.2-py2.4.egg
d demo-0.3-py2.4.egg
d demo-0.4c1-py2.4.egg
d demoneeded-1.0-py2.4.egg
d demoneeded-1.1-py2.4.egg
d demoneeded-1.2c1-py2.4.egg
d other-1.0-py2.4.egg
>>> rmdir(dest)
Sometimes it’s useful to specify version information independent of normal requirements specifications. For example, a buildout may need to lock down a set of versions, without having to put put version numbers in setup files or part definitions. If a dictionary is passed to the install function, mapping project names to version numbers, then the versions numbers will be used.
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest, links=[link_server], index=link_server+'index/',
... versions = dict(demo='0.2', demoneeded='1.0'))
>>> [d.version for d in ws]
['0.2', '1.0']
In this example, we specified a version for demoneeded, even though we didn’t define a requirement for it. The versions specified apply to dependencies as well as the specified requirements.
If we specify a version that’s incompatible with a requirement, then we’ll get an error:
>>> from zope.testing.loggingsupport import InstalledHandler
>>> handler = InstalledHandler('zc.buildout.easy_install')
>>> import logging
>>> logging.getLogger('zc.buildout.easy_install').propagate = False
>>> ws = zc.buildout.easy_install.install(
... ['demo >0.2'], dest, links=[link_server],
... index=link_server+'index/',
... versions = dict(demo='0.2', demoneeded='1.0'))
Traceback (most recent call last):
...
IncompatibleConstraintError: Bad constraint 0.2 demo>0.2
>>> print_(handler)
zc.buildout.easy_install DEBUG
Installing 'demo >0.2'.
zc.buildout.easy_install ERROR
The constraint, 0.2, is not consistent with the requirement, 'demo>0.2'.
>>> handler.clear()
If no versions are specified, a debugging message will be output reporting that a version was picked automatically:
>>> ws = zc.buildout.easy_install.install( ... ['demo'], dest, links=[link_server], index=link_server+'index/', ... )>>> print_(handler) zc.buildout.easy_install DEBUG Installing 'demo'. zc.buildout.easy_install INFO Getting distribution for 'demo'. zc.buildout.easy_install INFO Got demo 0.3. zc.buildout.easy_install DEBUG Picked: demo = 0.3 zc.buildout.easy_install DEBUG Getting required 'demoneeded' zc.buildout.easy_install DEBUG required by demo 0.3. zc.buildout.easy_install INFO Getting distribution for 'demoneeded'. zc.buildout.easy_install DEBUG Running easy_install:... zc.buildout.easy_install INFO Got demoneeded 1.1. zc.buildout.easy_install DEBUG Picked: demoneeded = 1.1
- zc.buildout.easy_install DEBUG
- Installing ‘demo’.
- zc.buildout.easy_install DEBUG
- We have the best distribution that satisfies ‘demo’.
- zc.buildout.easy_install DEBUG
- Picked: demo = 0.3
- zc.buildout.easy_install DEBUG
- Getting required ‘demoneeded’
- zc.buildout.easy_install DEBUG
- required by demo 0.3.
- zc.buildout.easy_install DEBUG
- We have the best distribution that satisfies ‘demoneeded’.
- zc.buildout.easy_install DEBUG
- Picked: demoneeded = 1.1
>>> handler.uninstall() >>> logging.getLogger('zc.buildout.easy_install').propagate = True
We can request that we get an error if versions are picked:
>>> zc.buildout.easy_install.allow_picked_versions(False)
True
(The old setting is returned.)
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest, links=[link_server], index=link_server+'index/',
... )
Traceback (most recent call last):
...
UserError: Picked: demo = 0.3
>>> zc.buildout.easy_install.allow_picked_versions(True)
False
The function default_versions can be used to get and set default version information to be used when no version information is passes. If called with an argument, it sets the default versions:
>>> zc.buildout.easy_install.default_versions(dict(demoneeded='1'))
...
{...}
It always returns the previous default versions. If called without an argument, it simply returns the default versions without changing them:
>>> zc.buildout.easy_install.default_versions()
{'demoneeded': '1'}
So with the default versions set, we’ll get the requested version even if the versions option isn’t used:
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest, links=[link_server], index=link_server+'index/',
... )
>>> [d.version for d in ws]
['0.3', '1.0']
Of course, we can unset the default versions by passing an empty dictionary:
>>> zc.buildout.easy_install.default_versions({})
{'demoneeded': '1'}
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest, links=[link_server], index=link_server+'index/',
... )
>>> [d.version for d in ws]
['0.3', '1.1']
Setuptools allows metadata that describes where to search for package dependencies. This option is called dependency_links. Buildout has its own notion of where to look for dependencies, but it also uses the setup tools dependency_links information if it’s available.
Let’s demo this by creating an egg that specifies dependency_links.
To begin, let’s create a new egg repository. This repository hold a newer version of the ‘demoneeded’ egg than the sample repository does.
>>> repoloc = tmpdir('repo')
>>> from zc.buildout.tests import create_egg
>>> create_egg('demoneeded', '1.2', repoloc)
>>> link_server2 = start_server(repoloc)
Turn on logging on this server so that we can see when eggs are pulled from it.
>>> _ = get(link_server2 + 'enable_server_logging')
GET 200 /enable_server_logging
Now we can create an egg that specifies that its dependencies are found on this server.
>>> repoloc = tmpdir('repo2')
>>> create_egg('hasdeps', '1.0', repoloc,
... install_requires = "'demoneeded'",
... dependency_links = [link_server2])
Let’s add the egg to another repository.
>>> link_server3 = start_server(repoloc)
Now let’s install the egg.
>>> example_dest = tmpdir('example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['hasdeps'], example_dest,
... links=[link_server3], index=link_server3+'index/')
GET 200 /
GET 200 /demoneeded-1.2-pyN.N.egg
The server logs show that the dependency was retrieved from the server specified in the dependency_links.
Now let’s see what happens if we provide two different ways to retrieve the dependencies.
>>> rmdir(example_dest)
>>> example_dest = tmpdir('example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['hasdeps'], example_dest, index=link_server+'index/',
... links=[link_server, link_server3])
GET 200 /
GET 200 /demoneeded-1.2-pyN.N.egg
Once again the dependency is fetched from the logging server even though it is also available from the non-logging server. This is because the version on the logging server is newer and buildout normally chooses the newest egg available.
If you wish to control where dependencies come from regardless of dependency_links setup metadata use the ‘use_dependency_links’ option to zc.buildout.easy_install.install().
>>> rmdir(example_dest)
>>> example_dest = tmpdir('example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['hasdeps'], example_dest, index=link_server+'index/',
... links=[link_server, link_server3],
... use_dependency_links=False)
Notice that this time the dependency egg is not fetched from the logging server. When you specify not to use dependency_links, eggs will only be searched for using the links you explicitly provide.
Another way to control this option is with the zc.buildout.easy_install.use_dependency_links() function. This function sets the default behavior for the zc.buildout.easy_install() function.
>>> zc.buildout.easy_install.use_dependency_links(False)
True
The function returns its previous setting.
>>> rmdir(example_dest)
>>> example_dest = tmpdir('example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['hasdeps'], example_dest, index=link_server+'index/',
... links=[link_server, link_server3])
It can be overridden by passing a keyword argument to the install function.
>>> rmdir(example_dest)
>>> example_dest = tmpdir('example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['hasdeps'], example_dest, index=link_server+'index/',
... links=[link_server, link_server3],
... use_dependency_links=True)
GET 200 /demoneeded-1.2-pyN.N.egg
To return the dependency_links behavior to normal call the function again.
>>> zc.buildout.easy_install.use_dependency_links(True)
False
>>> rmdir(example_dest)
>>> example_dest = tmpdir('example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['hasdeps'], example_dest, index=link_server+'index/',
... links=[link_server, link_server3])
GET 200 /demoneeded-1.2-pyN.N.egg
The easy_install module provides support for creating scripts from eggs. It provides a function similar to setuptools except that it provides facilities for baking a script’s path into the script. This has two advantages:
The scripts method can be used to generate scripts. Let’s create a destination directory for it to place them in:
>>> import tempfile
>>> bin = tmpdir('bin')
Now, we’ll use the scripts method to generate scripts in this directory from the demo egg:
>>> import sys
>>> scripts = zc.buildout.easy_install.scripts(
... ['demo'], ws, sys.executable, bin)
the three arguments we passed were:
The bin directory now contains a generated script:
>>> ls(bin)
- demo
The return value is a list of the scripts generated:
>>> import os, sys
>>> if sys.platform == 'win32':
... scripts == [os.path.join(bin, 'demo.exe'),
... os.path.join(bin, 'demo-script.py')]
... else:
... scripts == [os.path.join(bin, 'demo')]
True
Note that in Windows, 2 files are generated for each script. A script file, ending in ‘-script.py’, and an exe file that allows the script to be invoked directly without having to specify the Python interpreter and without having to provide a ‘.py’ suffix.
The demo script run the entry point defined in the demo egg:
>>> cat(bin, 'demo')
#!/usr/local/bin/python2.7
import sys
sys.path[0:0] = [
'/sample-install/demo-0.3-py2.4.egg',
'/sample-install/demoneeded-1.1-py2.4.egg',
]
import eggrecipedemo
if __name__ == '__main__':
sys.exit(eggrecipedemo.main())
Some things to note:
Rather than requirement strings, you can pass tuples containing 3 strings:
- A script name,
- A module,
- An attribute expression for an entry point within the module.
For example, we could have passed entry point information directly rather than passing a requirement:
>>> scripts = zc.buildout.easy_install.scripts(
... [('demo', 'eggrecipedemo', 'main')], ws,
... sys.executable, bin)
>>> cat(bin, 'demo')
#!/usr/local/bin/python2.7
import sys
sys.path[0:0] = [
'/sample-install/demo-0.3-py2.4.egg',
'/sample-install/demoneeded-1.1-py2.4.egg',
]
import eggrecipedemo
if __name__ == '__main__':
sys.exit(eggrecipedemo.main())
Passing entry-point information directly is handy when using eggs (or distributions) that don’t declare their entry points, such as distributions that aren’t based on setuptools.
The interpreter keyword argument can be used to generate a script that can be used to invoke the Python interactive interpreter with the path set based on the working set. This generated script can also be used to run other scripts with the path set on the working set:
>>> scripts = zc.buildout.easy_install.scripts(
... ['demo'], ws, sys.executable, bin, interpreter='py')
>>> ls(bin)
- demo
- py
>>> if sys.platform == 'win32':
... scripts == [os.path.join(bin, 'demo.exe'),
... os.path.join(bin, 'demo-script.py'),
... os.path.join(bin, 'py.exe'),
... os.path.join(bin, 'py-script.py')]
... else:
... scripts == [os.path.join(bin, 'demo'),
... os.path.join(bin, 'py')]
True
The py script simply runs the Python interactive interpreter with the path set:
>>> cat(bin, 'py')
#!/usr/local/bin/python2.7
import sys
sys.path[0:0] = [
'/sample-install/demo-0.3-pyN.N.egg',
'/sample-install/demoneeded-1.1-pyN.N.egg',
]
_interactive = True
if len(sys.argv) > 1:
_options, _args = __import__("getopt").getopt(sys.argv[1:], 'ic:m:')
_interactive = False
for (_opt, _val) in _options:
if _opt == '-i':
_interactive = True
elif _opt == '-c':
exec(_val)
elif _opt == '-m':
sys.argv[1:] = _args
_args = []
__import__("runpy").run_module(
_val, {}, "__main__", alter_sys=True)
if _args:
sys.argv[:] = _args
__file__ = _args[0]
del _options, _args
__file__f = open(__file__)
exec(compile(__file__f.read(), __file__, "exec"))
__file__f.close(); del __file__f
if _interactive:
del _interactive
__import__("code").interact(banner="", local=globals())
If invoked with a script name and arguments, it will run that script, instead.
>>> write('ascript', r'''
... "demo doc"
... import sys
... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\n')
... print_(sys.argv)
... print_((__name__, __file__, __doc__))
... ''')
>>> print_(system(join(bin, 'py')+' ascript a b c'), end='')
['ascript', 'a', 'b', 'c']
('__main__', 'ascript', 'demo doc')
For Python 2.5 and higher, you can also use the -m option to run a module:
>>> if sys.version_info < (2, 5):
... print ('usage: pdb.py blah blah blah')
... else:
... print_(system(join(bin, 'py')+' -m pdb'), end='')
...
usage: pdb.py ...
>>> print_(system(join(bin, 'py')+' -m pdb what'), end='')
Error: what does not exist
An interpreter can also be generated without other eggs:
>>> scripts = zc.buildout.easy_install.scripts(
... [], [], sys.executable, bin, interpreter='py')
>>> cat(bin, 'py')
#!/usr/local/bin/python2.7
import sys
sys.path[0:0] = [
]
...
An additional argument can be passed to define which scripts to install and to provide script names. The argument is a dictionary mapping original script names to new script names.
>>> bin = tmpdir('bin2')
>>> scripts = zc.buildout.easy_install.scripts(
... ['demo'], ws, sys.executable, bin, dict(demo='run'))
>>> if sys.platform == 'win32':
... scripts == [os.path.join(bin, 'run.exe'),
... os.path.join(bin, 'run-script.py')]
... else:
... scripts == [os.path.join(bin, 'run')]
True
>>> ls(bin)
- run
>>> print_(system(os.path.join(bin, 'run')), end='')
3 1
The scripts that are generated are made executable:
>>> if sys.platform == 'win32':
... os.access(os.path.join(bin, 'run.exe'), os.X_OK)
... else:
... os.access(os.path.join(bin, 'run'), os.X_OK)
True
We can pass a keyword argument, extra paths, to cause additional paths to be included in the a generated script:
>>> foo = tmpdir('foo')
>>> scripts = zc.buildout.easy_install.scripts(
... ['demo'], ws, sys.executable, bin, dict(demo='run'),
... extra_paths=[foo])
>>> cat(bin, 'run')
#!/usr/local/bin/python2.7
import sys
sys.path[0:0] = [
'/sample-install/demo-0.3-py2.4.egg',
'/sample-install/demoneeded-1.1-py2.4.egg',
'/foo',
]
import eggrecipedemo
if __name__ == '__main__':
sys.exit(eggrecipedemo.main())
An “argument” keyword argument can be used to pass arguments to an entry point. The value passed is a source string to be placed between the parentheses in the call:
>>> scripts = zc.buildout.easy_install.scripts(
... ['demo'], ws, sys.executable, bin, dict(demo='run'),
... arguments='1, 2')
>>> cat(bin, 'run')
#!/usr/local/bin/python2.7
import sys
sys.path[0:0] = [
'/sample-install/demo-0.3-py2.4.egg',
'/sample-install/demoneeded-1.1-py2.4.egg',
]
import eggrecipedemo
if __name__ == '__main__':
sys.exit(eggrecipedemo.main(1, 2))
You can also pass script initialization code:
>>> scripts = zc.buildout.easy_install.scripts(
... ['demo'], ws, sys.executable, bin, dict(demo='run'),
... arguments='1, 2',
... initialization='import os\nos.chdir("foo")',
... interpreter='py')
>>> cat(bin, 'run')
#!/usr/local/bin/python2.7
import sys
sys.path[0:0] = [
'/sample-install/demo-0.3-py2.4.egg',
'/sample-install/demoneeded-1.1-py2.4.egg',
]
import os
os.chdir("foo")
import eggrecipedemo
if __name__ == '__main__':
sys.exit(eggrecipedemo.main(1, 2))
It will be included in interpreters too:
>>> cat(bin, 'py')
#!/usr/local/bin/python2.7
import sys
sys.path[0:0] = [
'/sample-install/demo-0.3-py3.3.egg',
'/sample-install/demoneeded-1.1-py3.3.egg',
]
import os
os.chdir("foo")
_interactive = True
...
Sometimes, you want to be able to move a buildout directory around and have scripts still work without having to rebuild them. We can control this using the relative_paths option to install. You need to pass a common base directory of the scripts and eggs:
>>> bo = tmpdir('bo')
>>> ba = tmpdir('ba')
>>> mkdir(bo, 'eggs')
>>> mkdir(bo, 'bin')
>>> mkdir(bo, 'other')
>>> ws = zc.buildout.easy_install.install(
... ['demo'], join(bo, 'eggs'), links=[link_server],
... index=link_server+'index/')
>>> scripts = zc.buildout.easy_install.scripts(
... ['demo'], ws, sys.executable, join(bo, 'bin'), dict(demo='run'),
... extra_paths=[ba, join(bo, 'bar')],
... interpreter='py',
... relative_paths=bo)
>>> cat(bo, 'bin', 'run')
#!/usr/local/bin/python2.7
import os
join = os.path.join
base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
base = os.path.dirname(base)
import sys
sys.path[0:0] = [
join(base, 'eggs/demo-0.3-pyN.N.egg'),
join(base, 'eggs/demoneeded-1.1-pyN.N.egg'),
'/ba',
join(base, 'bar'),
]
import eggrecipedemo
if __name__ == '__main__':
sys.exit(eggrecipedemo.main())
Note that the extra path we specified that was outside the directory passed as relative_paths wasn’t converted to a relative path.
Of course, running the script works:
>>> print_(system(join(bo, 'bin', 'run')), end='')
3 1
We specified an interpreter and its paths are adjusted too:
>>> cat(bo, 'bin', 'py')
#!/usr/local/bin/python2.7
import os
join = os.path.join
base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
base = os.path.dirname(base)
import sys
sys.path[0:0] = [
join(base, 'eggs/demo-0.3-pyN.N.egg'),
join(base, 'eggs/demoneeded-1.1-pyN.N.egg'),
'/ba',
join(base, 'bar'),
]
_interactive = True
if len(sys.argv) > 1:
_options, _args = __import__("getopt").getopt(sys.argv[1:], 'ic:m:')
_interactive = False
for (_opt, _val) in _options:
if _opt == '-i':
_interactive = True
elif _opt == '-c':
exec(_val)
elif _opt == '-m':
sys.argv[1:] = _args
_args = []
__import__("runpy").run_module(
_val, {}, "__main__", alter_sys=True)
if _args:
sys.argv[:] = _args
__file__ = _args[0]
del _options, _args
__file__f = open(__file__)
exec(compile(__file__f.read(), __file__, "exec"))
__file__f.close(); del __file__f
if _interactive:
del _interactive
__import__("code").interact(banner="", local=globals())
Most python libraries use the console_scripts entry point nowadays. But several still have a scripts=['bin/something'] in their setup() call. Buildout also installs those:
>>> distdir = tmpdir('distutilsscriptdir')
>>> distbin = tmpdir('distutilsscriptbin')
>>> ws = zc.buildout.easy_install.install(
... ['other'], distdir,
... links=[link_server], index=link_server+'index/')
>>> scripts = zc.buildout.easy_install.scripts(
... ['other'], ws, sys.executable, distbin)
>>> ls(distbin)
- distutilsscript
Like for console_scripts, the output is a list of the scripts generated. Likewise, on windows two files, an .exe and a script with -script.py appended, are generated:
>>> import os, sys
>>> if sys.platform == 'win32':
... scripts == [os.path.join(distbin, 'distutilsscript.exe'),
... os.path.join(distbin, 'distutilsscript-script.py')]
... else:
... scripts == [os.path.join(distbin, 'distutilsscript')]
True
It also works for zipped eggs:
>>> distdir2 = tmpdir('distutilsscriptdir2')
>>> distbin2 = tmpdir('distutilsscriptbin2')
>>> ws = zc.buildout.easy_install.install(
... ['du_zipped'], distdir2,
... links=[link_server], index=link_server+'index/')
>>> scripts = zc.buildout.easy_install.scripts(
... ['du_zipped'], ws, sys.executable, distbin2)
>>> ls(distbin2)
- distutilsscript
Distutils copies the script files verbatim, apart from a line at the top that looks like #!/usr/bin/python, which gets replaced by the actual python interpreter. Buildout does the same, but additionally also adds the sys.path like for the console_scripts.
>>> cat(distbin, 'distutilsscript')
#!/usr/local/bin/python2.7
# -*- coding: utf-8 -*-
"""Module docstring."""
from __future__ import print_statement
import sys
sys.path[0:0] = [
'/distutilsscriptdir/other-1.0-pyN.N.egg',
]
import os
import sys; sys.stdout.write("distutils!\n")
Note that there are several items that need to come first in such a script before buildout’s sys.path statements: a source encoding hint, a module docstring and __future__ imports. Buildout retains them in their proper place by looking at the first non-future import and placing its sys.path statement before that.
Due to the nature of distutils scripts, buildout cannot pass arguments as there’s no specific method to pass them to.
In some cases, a python 3 __pycache__ directory can end up in an internal EGG-INFO metadata directory, next to the script information we’re looking for. Buildout doesn’t crash on that:
>>> eggname = [name for name in os.listdir(distdir2)
... if name.endswith('egg')][0]
>>> scripts_metadata_dir = os.path.join(
... distdir2, eggname, 'EGG-INFO', 'scripts')
>>> os.mkdir(os.path.join(scripts_metadata_dir, '__dummy__'))
>>> scripts = zc.buildout.easy_install.scripts(
... ['du_zipped'], ws, sys.executable, distbin2)
>>> ls(distbin2)
- distutilsscript
Sometimes, we need to control how extension modules are built. The build function provides this level of control. It takes a single package specification, downloads a source distribution, and builds it with specified custom build options.
The build function takes 3 positional arguments:
It supports a number of optional keyword arguments:
The URL of an index server, or almost any other valid URL. :)
If not specified, the Python Package Index, http://pypi.python.org/simple/, is used. You can specify an alternate index with this option. If you use the links option and if the links point to the needed distributions, then the index can be anything and will be largely ignored. In the examples, here, we’ll just point to an empty directory on our link server. This will make our examples run a little bit faster.
Our link server included a source distribution that includes a simple extension, extdemo.c:
#include <Python.h>
#include <extdemo.h>
static PyMethodDef methods[] = {};
PyMODINIT_FUNC
initextdemo(void)
{
PyObject *m;
m = Py_InitModule3("extdemo", methods, "");
#ifdef TWO
PyModule_AddObject(m, "val", PyInt_FromLong(2));
#else
PyModule_AddObject(m, "val", PyInt_FromLong(EXTDEMO));
#endif
}
The extension depends on a system-dependent include file, extdemo.h, that defines a constant, EXTDEMO, that is exposed by the extension.
We’ll add an include directory to our sample buildout and add the needed include file to it:
>>> mkdir('include')
>>> write('include', 'extdemo.h',
... """
... #define EXTDEMO 42
... """)
Now, we can use the build function to create an egg from the source distribution:
>>> zc.buildout.easy_install.build(
... 'extdemo', dest,
... {'include-dirs': os.path.join(sample_buildout, 'include')},
... links=[link_server], index=link_server+'index/')
['/sample-install/extdemo-1.4-py2.4-unix-i686.egg']
The function returns the list of eggs
Now if we look in our destination directory, we see we have an extdemo egg:
>>> ls(dest)
d demo-0.2-py2.4.egg
d demo-0.3-py2.4.egg
d demoneeded-1.0-py2.4.egg
d demoneeded-1.1-py2.4.egg
d extdemo-1.4-py2.4-unix-i686.egg
Let’s update our link server with a new version of extdemo:
>>> update_extdemo()
>>> print_(get(link_server), end='')
<html><body>
<a href="bigdemo-0.1-py2.4.egg">bigdemo-0.1-py2.4.egg</a><br>
<a href="demo-0.1-py2.4.egg">demo-0.1-py2.4.egg</a><br>
<a href="demo-0.2-py2.4.egg">demo-0.2-py2.4.egg</a><br>
<a href="demo-0.3-py2.4.egg">demo-0.3-py2.4.egg</a><br>
<a href="demo-0.4c1-py2.4.egg">demo-0.4c1-py2.4.egg</a><br>
<a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br>
<a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br>
<a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br>
<a href="du_zipped-1.0-pyN.N.egg">du_zipped-1.0-pyN.N.egg</a><br>
<a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br>
<a href="extdemo-1.5.zip">extdemo-1.5.zip</a><br>
<a href="index/">index/</a><br>
<a href="other-1.0-py2.4.egg">other-1.0-py2.4.egg</a><br>
</body></html>
The easy_install caches information about servers to reduce network access. To see the update, we have to call the clear_index_cache function to clear the index cache:
>>> zc.buildout.easy_install.clear_index_cache()
If we run build with newest set to False, we won’t get an update:
>>> zc.buildout.easy_install.build(
... 'extdemo', dest,
... {'include-dirs': os.path.join(sample_buildout, 'include')},
... links=[link_server], index=link_server+'index/',
... newest=False)
['/sample-install/extdemo-1.4-py2.4-linux-i686.egg']
>>> ls(dest)
d demo-0.2-py2.4.egg
d demo-0.3-py2.4.egg
d demoneeded-1.0-py2.4.egg
d demoneeded-1.1-py2.4.egg
d extdemo-1.4-py2.4-unix-i686.egg
But if we run it with the default True setting for newest, then we’ll get an updated egg:
>>> zc.buildout.easy_install.build(
... 'extdemo', dest,
... {'include-dirs': os.path.join(sample_buildout, 'include')},
... links=[link_server], index=link_server+'index/')
['/sample-install/extdemo-1.5-py2.4-unix-i686.egg']
>>> ls(dest)
d demo-0.2-py2.4.egg
d demo-0.3-py2.4.egg
d demoneeded-1.0-py2.4.egg
d demoneeded-1.1-py2.4.egg
d extdemo-1.4-py2.4-unix-i686.egg
d extdemo-1.5-py2.4-unix-i686.egg
The versions option also influences the versions used. For example, if we specify a version for extdemo, then that will be used, even though it isn’t the newest. Let’s clean out the destination directory first:
>>> import os
>>> for name in os.listdir(dest):
... remove(dest, name)
>>> zc.buildout.easy_install.build(
... 'extdemo', dest,
... {'include-dirs': os.path.join(sample_buildout, 'include')},
... links=[link_server], index=link_server+'index/',
... versions=dict(extdemo='1.4'))
['/sample-install/extdemo-1.4-py2.4-unix-i686.egg']
>>> ls(dest)
d extdemo-1.4-py2.4-unix-i686.egg
The develop function is similar to the build function, except that, rather than building an egg from a source directory containing a setup.py script.
The develop function takes 2 positional arguments:
It supports some optional keyword argument:
We have a local directory containing the extdemo source:
>>> ls(extdemo)
- MANIFEST
- MANIFEST.in
- README
- extdemo.c
- setup.py
Now, we can use the develop function to create a develop egg from the source distribution:
>>> zc.buildout.easy_install.develop(
... extdemo, dest,
... {'include-dirs': os.path.join(sample_buildout, 'include')})
'/sample-install/extdemo.egg-link'
The name of the egg link created is returned.
Now if we look in our destination directory, we see we have an extdemo egg link:
>>> ls(dest)
d extdemo-1.4-py2.4-unix-i686.egg
- extdemo.egg-link
And that the source directory contains the compiled extension:
>>> contents = os.listdir(extdemo)
>>> bool([f for f in contents if f.endswith('.so') or f.endswith('.pyd')])
True
Normally, when distributions are installed, if any processing is needed, they are downloaded from the internet to a temporary directory and then installed from there. A download cache can be used to avoid the download step. This can be useful to reduce network access and to create source distributions of an entire buildout.
A download cache is specified by calling the download_cache function. The function always returns the previous setting. If no argument is passed, then the setting is unchanged. If an argument is passed, the download cache is set to the given path, which must point to an existing directory. Passing None clears the cache setting.
To see this work, we’ll create a directory and set it as the cache directory:
>>> cache = tmpdir('cache')
>>> zc.buildout.easy_install.download_cache(cache)
We’ll recreate our destination directory:
>>> remove(dest)
>>> dest = tmpdir('sample-install')
We’d like to see what is being fetched from the server, so we’ll enable server logging:
>>> _ = get(link_server+'enable_server_logging')
GET 200 /enable_server_logging
Now, if we install demo, and extdemo:
>>> ws = zc.buildout.easy_install.install(
... ['demo==0.2'], dest,
... links=[link_server], index=link_server+'index/')
GET 200 /
GET 404 /index/demo/
GET 200 /index/
GET 200 /demo-0.2-py2.4.egg
GET 404 /index/demoneeded/
GET 200 /demoneeded-1.1.zip
>>> zc.buildout.easy_install.build(
... 'extdemo', dest,
... {'include-dirs': os.path.join(sample_buildout, 'include')},
... links=[link_server], index=link_server+'index/')
GET 404 /index/extdemo/
GET 200 /extdemo-1.5.zip
['/sample-install/extdemo-1.5-py2.4-linux-i686.egg']
Not only will we get eggs in our destination directory:
>>> ls(dest)
d demo-0.2-py2.4.egg
d demoneeded-1.1-py2.4.egg
d extdemo-1.5-py2.4-linux-i686.egg
But we’ll get distributions in the cache directory:
>>> ls(cache)
- demo-0.2-py2.4.egg
- demoneeded-1.1.zip
- extdemo-1.5.zip
The cache directory contains uninstalled distributions, such as zipped eggs or source distributions.
Let’s recreate our destination directory and clear the index cache:
>>> remove(dest)
>>> dest = tmpdir('sample-install')
>>> zc.buildout.easy_install.clear_index_cache()
Now when we install the distributions:
>>> ws = zc.buildout.easy_install.install(
... ['demo==0.2'], dest,
... links=[link_server], index=link_server+'index/')
GET 200 /
GET 404 /index/demo/
GET 200 /index/
GET 404 /index/demoneeded/
>>> zc.buildout.easy_install.build(
... 'extdemo', dest,
... {'include-dirs': os.path.join(sample_buildout, 'include')},
... links=[link_server], index=link_server+'index/')
GET 404 /index/extdemo/
['/sample-install/extdemo-1.5-py2.4-linux-i686.egg']
>>> ls(dest)
d demo-0.2-py2.4.egg
d demoneeded-1.1-py2.4.egg
d extdemo-1.5-py2.4-linux-i686.egg
Note that we didn’t download the distributions from the link server.
If we remove the restriction on demo, we’ll download a newer version from the link server:
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest,
... links=[link_server], index=link_server+'index/')
GET 200 /demo-0.3-py2.4.egg
Normally, the download cache is the preferred source of downloads, but not the only one.
A download cache can be used as the basis of application source releases. In an application source release, we want to distribute an application that can be built without making any network accesses. In this case, we distribute a download cache and tell the easy_install module to install from the download cache only, without making network accesses. The install_from_cache function can be used to signal that packages should be installed only from the download cache. The function always returns the previous setting. Calling it with no arguments returns the current setting without changing it:
>>> zc.buildout.easy_install.install_from_cache()
False
Calling it with a boolean value changes the setting and returns the previous setting:
>>> zc.buildout.easy_install.install_from_cache(True)
False
Let’s remove demo-0.3-py2.4.egg from the cache, clear the index cache, recreate the destination directory, and reinstall demo:
>>> for f in os.listdir(cache):
... if f.startswith('demo-0.3-'):
... remove(cache, f)
>>> zc.buildout.easy_install.clear_index_cache()
>>> remove(dest)
>>> dest = tmpdir('sample-install')
>>> ws = zc.buildout.easy_install.install(
... ['demo'], dest,
... links=[link_server], index=link_server+'index/')
>>> ls(dest)
d demo-0.2-py2.4.egg
d demoneeded-1.1-py2.4.egg
This time, we didn’t download from or even query the link server.