
Caching extended configuration
******************************

As mentioned in the general buildout documentation, configuration
files can extend each other, including the ability to download
configuration being extended from a URL. If desired, zc.buildout
caches downloaded configuration in order to be able to use it when run
offline.

As we're going to talk about downloading things, let's start an HTTP
server. Also, all of the following will take place inside the sample
buildout.

>>> server_data = tmpdir('server_data')
>>> server_url = start_server(server_data)
>>> cd(sample_buildout)

We also use a fresh directory for temporary files in order to make
sure that all temporary files have been cleaned up in the end:

>>> import tempfile
>>> old_tempdir = tempfile.tempdir
>>> tempfile.tempdir = tmpdir('tmp')


Basic use of the extends cache
==============================

We put some base configuration on a server and reference it from a
sample buildout:

>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... foo = bar
... """)

>>> write('buildout.cfg', """\
... [buildout]
... extends = %sbase.cfg
... """ % server_url)

When trying to run this buildout offline, we'll find that we cannot
read all of the required configuration:

>>> print_(system(buildout + ' -o'))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

Trying the same online, we can:

>>> print_(system(buildout))
Unused options for buildout: 'foo'.

As long as we haven't said anything about caching downloaded
configuration, nothing gets cached. Offline mode will still cause the
buildout to fail:

>>> print_(system(buildout + ' -o'))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

Let's now specify a cache for base configuration files. This cache is
different from the download cache used by recipes for caching
distributions and other files; one might, however, use a namespace
subdirectory of the download cache for it. The configuration cache we
specify will be created when running buildout and the base.cfg file
will be put in it (with the file name being a hash of the complete
URL):

>>> mkdir('cache')
>>> write('buildout.cfg', """\
... [buildout]
... extends = %sbase.cfg
... extends-cache = cache
... """ % server_url)

>>> print_(system(buildout))
Unused options for buildout: 'foo'.

>>> cache = join(sample_buildout, 'cache')
>>> ls(cache)
-  5aedc98d7e769290a29d654a591a3a45

>>> import os
>>> cat(cache, os.listdir(cache)[0])
[buildout]
parts =
foo = bar

We can now run buildout offline as it will read base.cfg from the
cache:

>>> print_(system(buildout + ' -o'))
Unused options for buildout: 'foo'.

The cache is being used purely as a fall-back in case we are offline
or don't have access to a configuration file to be downloaded. As long
as we are online, buildout attempts to download a fresh copy of each
file even if a cached copy of the file exists. To see this, we put
different configuration in the same place on the server and run
buildout in offline mode so it takes base.cfg from the cache:

>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... bar = baz
... """)

>>> print_(system(buildout + ' -o'))
Unused options for buildout: 'foo'.

In online mode, buildout will download and use the modified version:

>>> print_(system(buildout))
Unused options for buildout: 'bar'.

Trying offline mode again, the new version will be used as it has been
put in the cache now:

>>> print_(system(buildout + ' -o'))
Unused options for buildout: 'bar'.

Clean up:

>>> rmdir(cache)


Specifying extends cache and offline mode
=========================================

Normally, the values of buildout options such as the location of a
download cache or whether to use offline mode are determined by first
reading the user's default configuration, updating it with the
project's configuration and finally applying command-line options.
User and project configuration are assembled by reading a file such as
"~/.buildout/default.cfg", "buildout.cfg" or a URL given on the
command line, recursively (depth-first) downloading any base
configuration specified by the "buildout:extends" option read from
each of those config files, and finally evaluating each config file to
provide default values for options not yet read.

This works fine for all options that do not influence how
configuration is downloaded in the first place. The "extends-cache"
and "offline" options, however, are treated differently from the
procedure described in order to make it simple and obvious to see
where a particular configuration file came from under any particular
circumstances.

* Offline and extends-cache settings are read from the two root config
  files exclusively. Otherwise one could construct configuration files
  that, when read, imply that they should have been read from a
  different source than they have. Also, specifying the extends cache
  within a file that might have to be taken from the cache before
  being read wouldn't make a lot of sense.

* Offline and extends-cache settings given by the user's defaults
  apply to the process of assembling the project's configuration. If
  no extends cache has been specified by the user's default
  configuration, the project's root config file must be available, be
  it from disk or from the net.

* Offline mode turned on by the "-o" command line option is honored
  from the beginning even though command line options are applied to
  the configuration last. If offline mode is not requested by the
  command line, it may be switched on by either the user's or the
  project's config root.


Extends cache
-------------

Let's see the above rules in action. We create a new home directory
for our user and write user and project configuration that recursively
extends online bases, using different caches:

>>> mkdir('home')
>>> mkdir('home', '.buildout')
>>> mkdir('cache')
>>> mkdir('user-cache')
>>> os.environ['HOME'] = join(sample_buildout, 'home')
>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... extends-cache = user-cache
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... """ % server_url)
>>> write(server_data, 'base_default.cfg', """\
... [buildout]
... foo = bar
... offline = false
... """)

>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... extends-cache = cache
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... """ % server_url)
>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... offline = false
... """)

Buildout will now assemble its configuration from all of these 6
files, defaults first. The online resources end up in the respective
extends caches:

>>> print_(system(buildout))
Unused options for buildout: 'foo'.

>>> ls('user-cache')
-  10e772cf422123ef6c64ae770f555740
>>> cat('user-cache', os.listdir('user-cache')[0])
[buildout]
foo = bar
offline = false

>>> ls('cache')
-  c72213127e6eb2208a3e1fc1dba771a7
>>> cat('cache', os.listdir('cache')[0])
[buildout]
parts =
offline = false

If, on the other hand, the extends caches are specified in files that
get extended themselves, they won't be used for assembling the
configuration they belong to (user's or project's, resp.). The extends
cache specified by the user's defaults does, however, apply to
downloading project configuration. Let's rewrite the config files,
clean out the caches and re-run buildout:

>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... extends-cache = user-cache
... """ % server_url)

>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... extends-cache = cache
... """ % server_url)

>>> remove('user-cache', os.listdir('user-cache')[0])
>>> remove('cache', os.listdir('cache')[0])

>>> print_(system(buildout))
Unused options for buildout: 'foo'.

>>> ls('user-cache')
-  0548bad6002359532de37385bb532e26
>>> cat('user-cache', os.listdir('user-cache')[0])
[buildout]
parts =
offline = false

>>> ls('cache')

Clean up:

>>> rmdir('user-cache')
>>> rmdir('cache')

Offline mode and installation from cache
----------------------------~~~~~~~~~~~~

If we run buildout in offline mode now, it will fail because it cannot
get at the remote configuration file needed by the user's defaults:

>>> print_(system(buildout + ' -o'))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

Let's now successively turn on offline mode by different parts of the
configuration and see when buildout applies this setting in each case:

>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... offline = true
... """)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... offline = true
... """ % server_url)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... """ % server_url)
>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... offline = true
... """)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... offline = true
... """ % server_url)
>>> print_(system(buildout))
Unused options for buildout: 'foo'.

The "install-from-cache" option is treated accordingly:

>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... install-from-cache = true
... """)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... install-from-cache = true
... """ % server_url)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... """ % server_url)
>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... install-from-cache = true
... """)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... install-from-cache = true
... """ % server_url)
>>> print_(system(buildout))
While:
  Installing.
  Checking for upgrades.
An internal error occurred ...
ValueError: install_from_cache set to true with no download cache

>>> rmdir('home', '.buildout')


Newest and non-newest behavior for extends cache
================================================

While offline mode forbids network access completely, 'newest' mode
determines whether to look for updated versions of a resource even if
some version of it is already present locally. If we run buildout in
newest mode ("newest = true"), the configuration files are updated
with each run:

>>> mkdir("cache")
>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... """)
>>> write('buildout.cfg', """\
... [buildout]
... extends-cache = cache
... extends = %sbase.cfg
... """ % server_url)
>>> print_(system(buildout))
>>> ls('cache')
-  5aedc98d7e769290a29d654a591a3a45
>>> cat('cache', os.listdir(cache)[0])
[buildout]
parts =

A change to "base.cfg" is picked up on the next buildout run:

>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... foo = bar
... """)
>>> print_(system(buildout + " -n"))
Unused options for buildout: 'foo'.
>>> cat('cache', os.listdir(cache)[0])
[buildout]
parts =
foo = bar

In contrast, when not using "newest" mode ("newest = false"), the
files already present in the extends cache will not be updated:

>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... """)
>>> print_(system(buildout + " -N"))
Unused options for buildout: 'foo'.
>>> cat('cache', os.listdir(cache)[0])
[buildout]
parts =
foo = bar

Even when updating base configuration files with a buildout run, any
given configuration file will be downloaded only once during that
particular run. If some base configuration file is extended more than
once, its cached copy is used:

>>> write(server_data, 'baseA.cfg', """\
... [buildout]
... extends = %sbase.cfg
... foo = bar
... """ % server_url)
>>> write(server_data, 'baseB.cfg', """\
... [buildout]
... extends-cache = cache
... extends = %sbase.cfg
... bar = foo
... """ % server_url)
>>> write('buildout.cfg', """\
... [buildout]
... extends-cache = cache
... newest = true
... extends = %sbaseA.cfg %sbaseB.cfg
... """ % (server_url, server_url))
>>> print_(system(buildout + " -n"))
Unused options for buildout: 'bar' 'foo'.

(XXX We patch download utility's API to produce readable output for
the test; a better solution would re-use the logging already done by
the utility.)

>>> import zc.buildout
>>> old_download = zc.buildout.download.Download.download
>>> def wrapper_download(self, url, md5sum=None, path=None):
...   print_("The URL %s was downloaded." % url)
...   return old_download(url, md5sum, path)
>>> zc.buildout.download.Download.download = wrapper_download

>>> zc.buildout.buildout.main([])
The URL http://localhost/baseA.cfg was downloaded.
The URL http://localhost/base.cfg was downloaded.
The URL http://localhost/baseB.cfg was downloaded.
Not upgrading because not running a local buildout command.
Unused options for buildout: 'bar' 'foo'.

>>> zc.buildout.download.Download.download = old_download


The deprecated "extended-by" option
===================================

The "buildout" section used to recognize an option named "extended-by"
that was deprecated at some point and removed in the 1.5 line. Since
ignoring this option silently was considered harmful as a matter of
principle, a UserError is raised if that option is encountered now:

>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... extended-by = foo.cfg
... """)
>>> print_(system(buildout))
While:
  Initializing.
Error: No-longer supported "extended-by" option found in http://localhost/base.cfg.


Clean up
========

We should have cleaned up all temporary files created by downloading
things:

>>> ls(tempfile.tempdir)

Reset the global temporary directory:

>>> tempfile.tempdir = old_tempdir
