# Caching extended configuration

As mentioned in the general buildout documentation, configuration files can extend each other, including the ability to download configuration being extended from a URL. If desired, zc.buildout caches downloaded configuration in order to be able to use it when run offline.

As we’re going to talk about downloading things, let’s start an HTTP server. Also, all of the following will take place inside the sample buildout.

    >>> server_data = tmpdir('server_data')
    >>> server_url = start_server(server_data)
    >>> cd(sample_buildout)

We also use a fresh directory for temporary files in order to make sure that all temporary files have been cleaned up in the end:

    >>> import tempfile
    >>> old_tempdir = tempfile.tempdir
    >>> tempfile.tempdir = tmpdir('tmp')

## Basic use of the extends cache

We put some base configuration on a server and reference it from a sample buildout:

    >>> write(server_data, 'base.cfg', """\
    ... [buildout]
    ... parts =
    ... foo = bar
    ... """)

    >>> write('buildout.cfg', """\
    ... [buildout]
    ... extends = %sbase.cfg
    ... """ % server_url)

When trying to run this buildout offline, we’ll find that we cannot read all of the required configuration:

    >>> print_(system(buildout + ' -o'))
    While:
      Initializing.
    Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

Trying the same online, we can:

    >>> print_(system(buildout))
    Unused options for buildout: 'foo'.

As long as we haven’t said anything about caching downloaded configuration, nothing gets cached. Offline mode will still cause the buildout to fail:

    >>> print_(system(buildout + ' -o'))
    While:
      Initializing.
    Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

Let’s now specify a cache for base configuration files. This cache is different from the download cache used by recipes for caching distributions and other files; one might, however, use a namespace subdirectory of the download cache for it. The configuration cache we specify will be created when running buildout and the base.cfg file will be put in it (with the file name being a hash of the complete URL):

    >>> mkdir('cache')
    >>> write('buildout.cfg', """\
    ... [buildout]
    ... extends = %sbase.cfg
    ... extends-cache = cache
    ... """ % server_url)

    >>> print_(system(buildout))
    Unused options for buildout: 'foo'.

    >>> cache = join(sample_buildout, 'cache')
    >>> ls(cache)
    -  5aedc98d7e769290a29d654a591a3a45

    >>> import os
    >>> cat(cache, os.listdir(cache)[0])
    [buildout]
    parts =
    foo = bar

We can now run buildout offline as it will read base.cfg from the cache:

    >>> print_(system(buildout + ' -o'))
    Unused options for buildout: 'foo'.

The cache is being used purely as a fall-back in case we are offline or don’t have access to a configuration file to be downloaded. As long as we are online, buildout attempts to download a fresh copy of each file even if a cached copy of the file exists. To see this, we put different configuration in the same place on the server and run buildout in offline mode so it takes base.cfg from the cache:

    >>> write(server_data, 'base.cfg', """\
    ... [buildout]
    ... parts =
    ... bar = baz
    ... """)

    >>> print_(system(buildout + ' -o'))
    Unused options for buildout: 'foo'.

In online mode, buildout will download and use the modified version:

    >>> print_(system(buildout))
    Unused options for buildout: 'bar'.

Trying offline mode again, the new version will be used as it has been put in the cache now:

    >>> print_(system(buildout + ' -o'))
    Unused options for buildout: 'bar'.

Clean up:

    >>> rmdir(cache)

## Specifying extends cache and offline mode

Normally, the values of buildout options such as the location of a download cache or whether to use offline mode are determined by first reading the user’s default configuration, updating it with the project’s configuration and finally applying command-line options. User and project configuration are assembled by reading a file such as `~/.buildout/default.cfg`, `buildout.cfg` or a URL given on the command line, recursively (depth-first) downloading any base configuration specified by the `buildout:extends` option read from each of those config files, and finally evaluating each config file to provide default values for options not yet read.

This works fine for all options that do not influence how configuration is downloaded in the first place. The `extends-cache` and `offline` options, however, are treated differently from the procedure described in order to make it simple and obvious to see where a particular configuration file came from under any particular circumstances.

- Offline and extends-cache settings are read from the two root config files exclusively. Otherwise one could construct configuration files that, when read, imply that they should have been read from a different source than they have. Also, specifying the extends cache within a file that might have to be taken from the cache before being read wouldn’t make a lot of sense.
- Offline and extends-cache settings given by the user’s defaults apply to the process of assembling the project’s configuration. If no extends cache has been specified by the user’s default configuration, the project’s root config file must be available, be it from disk or from the net.
- Offline mode turned on by the `-o` command line option is honored from the beginning even though command line options are applied to the configuration last. If offline mode is not requested by the command line, it may be switched on by either the user’s or the project’s config root.

### Extends cache

Let’s see the above rules in action. We create a new home directory for our user and write user and project configuration that recursively extends online bases, using different caches:

    >>> mkdir('home')
    >>> mkdir('home', '.buildout')
    >>> mkdir('cache')
    >>> mkdir('user-cache')
    >>> os.environ['HOME'] = join(sample_buildout, 'home')
    >>> write('home', '.buildout', 'default.cfg', """\
    ... [buildout]
    ... extends = fancy_default.cfg
    ... extends-cache = user-cache
    ... """)
    >>> write('home', '.buildout', 'fancy_default.cfg', """\
    ... [buildout]
    ... extends = %sbase_default.cfg
    ... """ % server_url)
    >>> write(server_data, 'base_default.cfg', """\
    ... [buildout]
    ... foo = bar
    ... offline = false
    ... """)

    >>> write('buildout.cfg', """\
    ... [buildout]
    ... extends = fancy.cfg
    ... extends-cache = cache
    ... """)
    >>> write('fancy.cfg', """\
    ... [buildout]
    ... extends = %sbase.cfg
    ... """ % server_url)
    >>> write(server_data, 'base.cfg', """\
    ... [buildout]
    ... parts =
    ... offline = false
    ... """)

Buildout will now assemble its configuration from all of these 6 files, defaults first. The online resources end up in the respective extends caches:

    >>> print_(system(buildout))
    Unused options for buildout: 'foo'.

    >>> ls('user-cache')
    -  10e772cf422123ef6c64ae770f555740
    >>> cat('user-cache', os.listdir('user-cache')[0])
    [buildout]
    foo = bar
    offline = false

    >>> ls('cache')
    -  c72213127e6eb2208a3e1fc1dba771a7
    >>> cat('cache', os.listdir('cache')[0])
    [buildout]
    parts =
    offline = false

If, on the other hand, the extends caches are specified in files that get extended themselves, they won’t be used for assembling the configuration they belong to (user’s or project’s, resp.). The extends cache specified by the user’s defaults does, however, apply to downloading project configuration. Let’s rewrite the config files, clean out the caches and re-run buildout:

    >>> write('home', '.buildout', 'default.cfg', """\
    ... [buildout]
    ... extends = fancy_default.cfg
    ... """)
    >>> write('home', '.buildout', 'fancy_default.cfg', """\
    ... [buildout]
    ... extends = %sbase_default.cfg
    ... extends-cache = user-cache
    ... """ % server_url)

    >>> write('buildout.cfg', """\
    ... [buildout]
    ... extends = fancy.cfg
    ... """)
    >>> write('fancy.cfg', """\
    ... [buildout]
    ... extends = %sbase.cfg
    ... extends-cache = cache
    ... """ % server_url)

    >>> remove('user-cache', os.listdir('user-cache')[0])
    >>> remove('cache', os.listdir('cache')[0])

    >>> print_(system(buildout))
    Unused options for buildout: 'foo'.

    >>> ls('user-cache')
    -  0548bad6002359532de37385bb532e26
    >>> cat('user-cache', os.listdir('user-cache')[0])
    [buildout]
    parts =
    offline = false

    >>> ls('cache')

Clean up:

    >>> rmdir('user-cache')
    >>> rmdir('cache')

Offline mode and installation from cache —————————-~~~~~~~~~~~~

If we run buildout in offline mode now, it will fail because it cannot get at the remote configuration file needed by the user’s defaults:

    >>> print_(system(buildout + ' -o'))
    While:
      Initializing.
    Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

Let’s now successively turn on offline mode by different parts of the configuration and see when buildout applies this setting in each case:

    >>> write('home', '.buildout', 'default.cfg', """\
    ... [buildout]
    ... extends = fancy_default.cfg
    ... offline = true
    ... """)
    >>> print_(system(buildout))
    While:
      Initializing.
    Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

    >>> write('home', '.buildout', 'default.cfg', """\
    ... [buildout]
    ... extends = fancy_default.cfg
    ... """)
    >>> write('home', '.buildout', 'fancy_default.cfg', """\
    ... [buildout]
    ... extends = %sbase_default.cfg
    ... offline = true
    ... """ % server_url)
    >>> print_(system(buildout))
    While:
      Initializing.
    Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

    >>> write('home', '.buildout', 'fancy_default.cfg', """\
    ... [buildout]
    ... extends = %sbase_default.cfg
    ... """ % server_url)
    >>> write('buildout.cfg', """\
    ... [buildout]
    ... extends = fancy.cfg
    ... offline = true
    ... """)
    >>> print_(system(buildout))
    While:
      Initializing.
    Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

    >>> write('buildout.cfg', """\
    ... [buildout]
    ... extends = fancy.cfg
    ... """)
    >>> write('fancy.cfg', """\
    ... [buildout]
    ... extends = %sbase.cfg
    ... offline = true
    ... """ % server_url)
    >>> print_(system(buildout))
    Unused options for buildout: 'foo'.

The `install-from-cache` option is treated accordingly:

    >>> write('home', '.buildout', 'default.cfg', """\
    ... [buildout]
    ... extends = fancy_default.cfg
    ... install-from-cache = true
    ... """)
    >>> print_(system(buildout))
    While:
      Initializing.
    Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

    >>> write('home', '.buildout', 'default.cfg', """\
    ... [buildout]
    ... extends = fancy_default.cfg
    ... """)
    >>> write('home', '.buildout', 'fancy_default.cfg', """\
    ... [buildout]
    ... extends = %sbase_default.cfg
    ... install-from-cache = true
    ... """ % server_url)
    >>> print_(system(buildout))
    While:
      Initializing.
    Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

    >>> write('home', '.buildout', 'fancy_default.cfg', """\
    ... [buildout]
    ... extends = %sbase_default.cfg
    ... """ % server_url)
    >>> write('buildout.cfg', """\
    ... [buildout]
    ... extends = fancy.cfg
    ... install-from-cache = true
    ... """)
    >>> print_(system(buildout))
    While:
      Initializing.
    Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

    >>> write('buildout.cfg', """\
    ... [buildout]
    ... extends = fancy.cfg
    ... """)
    >>> write('fancy.cfg', """\
    ... [buildout]
    ... extends = %sbase.cfg
    ... install-from-cache = true
    ... """ % server_url)
    >>> print_(system(buildout))
    While:
      Installing.
      Checking for upgrades.
    An internal error occurred ...
    ValueError: install_from_cache set to true with no download cache

    >>> rmdir('home', '.buildout')

## Newest and non-newest behavior for extends cache

While offline mode forbids network access completely, ‘newest’ mode determines whether to look for updated versions of a resource even if some version of it is already present locally. If we run buildout in newest mode (`newest = true`), the configuration files are updated with each run:

    >>> mkdir("cache")
    >>> write(server_data, 'base.cfg', """\
    ... [buildout]
    ... parts =
    ... """)
    >>> write('buildout.cfg', """\
    ... [buildout]
    ... extends-cache = cache
    ... extends = %sbase.cfg
    ... """ % server_url)
    >>> print_(system(buildout))
    >>> ls('cache')
    -  5aedc98d7e769290a29d654a591a3a45
    >>> cat('cache', os.listdir(cache)[0])
    [buildout]
    parts =

A change to `base.cfg` is picked up on the next buildout run:

    >>> write(server_data, 'base.cfg', """\
    ... [buildout]
    ... parts =
    ... foo = bar
    ... """)
    >>> print_(system(buildout + " -n"))
    Unused options for buildout: 'foo'.
    >>> cat('cache', os.listdir(cache)[0])
    [buildout]
    parts =
    foo = bar

In contrast, when not using `newest` mode (`newest = false`), the files already present in the extends cache will not be updated:

    >>> write(server_data, 'base.cfg', """\
    ... [buildout]
    ... parts =
    ... """)
    >>> print_(system(buildout + " -N"))
    Unused options for buildout: 'foo'.
    >>> cat('cache', os.listdir(cache)[0])
    [buildout]
    parts =
    foo = bar

Even when updating base configuration files with a buildout run, any given configuration file will be downloaded only once during that particular run. If some base configuration file is extended more than once, its cached copy is used:

    >>> write(server_data, 'baseA.cfg', """\
    ... [buildout]
    ... extends = %sbase.cfg
    ... foo = bar
    ... """ % server_url)
    >>> write(server_data, 'baseB.cfg', """\
    ... [buildout]
    ... extends-cache = cache
    ... extends = %sbase.cfg
    ... bar = foo
    ... """ % server_url)
    >>> write('buildout.cfg', """\
    ... [buildout]
    ... extends-cache = cache
    ... newest = true
    ... extends = %sbaseA.cfg %sbaseB.cfg
    ... """ % (server_url, server_url))
    >>> print_(system(buildout + " -n"))
    Unused options for buildout: 'bar' 'foo'.

(XXX We patch download utility’s API to produce readable output for the test; a better solution would re-use the logging already done by the utility.)

    >>> import zc.buildout
    >>> old_download = zc.buildout.download.Download.download
    >>> def wrapper_download(self, url, md5sum=None, path=None):
    ...   print_("The URL %s was downloaded." % url)
    ...   return old_download(url, md5sum, path)
    >>> zc.buildout.download.Download.download = wrapper_download

    >>> zc.buildout.buildout.main([])
    The URL http://localhost/baseA.cfg was downloaded.
    The URL http://localhost/base.cfg was downloaded.
    The URL http://localhost/baseB.cfg was downloaded.
    Not upgrading because not running a local buildout command.
    Unused options for buildout: 'bar' 'foo'.

    >>> zc.buildout.download.Download.download = old_download

## The deprecated `extended-by` option

The `buildout` section used to recognize an option named `extended-by` that was deprecated at some point and removed in the 1.5 line. Since ignoring this option silently was considered harmful as a matter of principle, a UserError is raised if that option is encountered now:

    >>> write(server_data, 'base.cfg', """\
    ... [buildout]
    ... parts =
    ... extended-by = foo.cfg
    ... """)
    >>> print_(system(buildout))
    While:
      Initializing.
    Error: No-longer supported "extended-by" option found in http://localhost/base.cfg.

## Clean up

We should have cleaned up all temporary files created by downloading things:

    >>> ls(tempfile.tempdir)

Reset the global temporary directory:

    >>> tempfile.tempdir = old_tempdir