Library (TODO)

This documents the standard library of components that batou provides.

Managing files and directories

File

The File component has been developed with Puppet‘s file type in mind. It accepts a very similar parameter set and has almost identical features.

You can use it to manage files, directories, and symlinks, and you can specify content (literally or as Jinja templates). You can also manage the Unix attributes and control whether leading directories should be managed or not.

The most basic usage is simply:

self += File('myfile')

This example creates a file at work/mycomponent/myfile, taking the contents from a file of the same name in the component’s directory (i.e. components/mycomponent/myfile). By default, the source file is run through Jinja, with the file’s parent component made available as component.

class batou.lib.File(path)

Creates a file. The main parameter for File is the target path. A File instance has an attribute path containing the full, absolute path to the resulting file.

File accepts the following additional parameters:

source

Filename of the source file to be used as the File’s content (absolute path or relative to the component’s directory). [Default: same as target path]

content

Literal file contents as a string.

is_template

Process file contents as Jinja template. [Default: True]

template_context

Object to make available as component to the Jinja template. [Default: File’s parent component]

template_args

Dict of additional arguments to make available to the Jinja template.

encoding

Encoding for the file contents [Default: utf-8]

owner

Unix owner username.

group

Unix group name.

mode

Unix permission mode (octal number, e.g. 0o755)

leading

Create leading directories that were given in the target path. [Default: False]

ensure

Type of object to be created: ‘file’, ‘directory’, or ‘symlink’. This is useful for complex situations (e.g. creating a symlink with special ownership), for simple situations it’s probably more readable to use Directory or Symlink.

Source of symlink (for ensure = ‘symlink’)

XXX explain detailed factoring of Presence and Content

class batou.lib.BinaryFile(path)

Subclass of batou.lib.File. Creates a non-template binary file.

Directory

class batou.lib.file.Directory(path)

Creates a directory. The main parameter is the target path.

source

Path to a source directory whose contents are to be synchronized to the target path (uses rsync internally).

exclude

List of file names or patterns that should not be synchronized to the target path (passed to rsync as --exclude argument, see the rsync documentation for details).

Creates a symlink at target by linking to source.

Removing files

Removal of obsolete things is a difficult topic in the convergence paradigm. If in the past we created a file foo, but now it is not used anymore, the code that originally said, “please manage foo”, will not be there anymore. This means that nobody knows that the file foo that is still lying around on the production system is not actually in use anymore. In most cases, a few stray files do not matter, but in case they do, the deployment code has to explicitly state that something should not be present anymore.

class batou.lib.file.Purge(pattern)

Ensures that a set of files (given as a glob pattern) does not exist.

Extracting archive files

batou can extract archive files in Tar, Zip, and DMG (on OSX target machines) format:

class batou.lib.archive.Extract(archive)

The main parameter is the archive filename (relative to the component’s directory). The archive format is determined according to the file name extension (‘.tar’, ‘.tar.gz’, ‘.tgz’, ‘.tar.bz2’, ‘.tar.xz’ use tar, ‘.zip’ uses unzip and ‘.dmg’ uses hdiutil). The following additional parameters are supported:

target

Target directory to extract the archive into. Directory is created if it does not exist (compare create_target_dir). [Default: base name of the archive file]

create_target_dir

Extract into the directory given in target. Set to False to extract directly into the work directory. [Default: True]

strip

Only for tar archives: number of directories contained in the archive to strip off (see the tar documentation for details) [Default: 0]

VFS mapping (TBD)

XXX writeme

Managing python installations

virtualenv

The basic building block for Python-based components is creation of virtualenvs (to separate package installations from each other):

self += VirtualEnv('2.7')
class batou.lib.python.VirtualEnv(version)

Creates a virtualenv for the given Python version in the working directory of the parent component. (Requires that pythonX.Y is in the PATH)

executable

Full path to the Python executable to create the virtualenv for (default: pythonX.Y based on the version attribute).

batou downloads a compatible version of virtualenv (depending on the Python version you need) to ensure everything works as expected and to avoid problems with incompatibilities or unexpected behaviours of whatever version might be installed already on the system. (virtualenv base installations are shared by all components for creating new virtualenvs, it is installed to work/.virtualenv).

Installing packages

Python packages are installed from a package index such as PyPI. batou uses pip or easy_install for this purpose (but that actually is an implementation detail and depends on the specifics of the Python and virtualenv version).

Packages must be added to a virtual environment.

venv = VirtualEnv('2.7')
self += venv
venv += Package('Sphinx', version='1.1.3')
class batou.lib.python.Package(package)

Install the Python package with the given name into the virtualenv of the parent component. Using Package requires that it is added to a VirtualEnv instance.

version

The version of the package to install (required).

install_options

List of options that are passed to pip/easy_install on the command line.

[Default: depends on the Python/virtualenv version in use]

check_package_is_module

Verify that the package is installed by trying to import it (more precisely, the first component of its dotted name). This is a stopgap against https://github.com/pypa/pip/issues/3, but should be pretty safe to disable if it causes trouble for specific packages (distribute is a notable example, since it installs a Python module named setuptools).

[Default: True]

timeout

A timeout (in seconds) that the installer should use to limit stalling network activity.

Only works when using pip.

[Default: equal to the environment’s timeout setting]

dependencies

Whether only the package itself or its dependencies should be installed.

[Default: True]

zc.buildout

batou has in-depth support for managing installations that use buildout. It automatically wraps them in a virtualenv, installs the appropriate buildout version, and takes care of running buildout whenever changes to configuration files makes it necessary. A typical usage example:

self += Buildout(python='2.7', version='2.2', setuptools='1.0',
                 additional_config=[Directory('profiles', source='profiles')])
class batou.lib.buildout.Buildout

Manage a buildout installation

python

Python version (required)

executable

Full path to the python executable to create the virtualenv for (used instead of pythonX.Y).

version

Version of zc.buildout to install (required)

setuptools

Version of setuptools to install into the virtualenv (must be appropriate to the buildout version, e.g. since 2.2 buildout requires setuptools, but some versions before that required distribute) (required)

distribute

Version of distribute to install into the virtualenv. Mutually exclusive with setuptools, of course.

config

If a different configuration file name than buildout.cfg should be used, pass in a File or Component instance.

additional_config

Optional list of component instances (e.g. File or Directory) that contain further configuration files (so Buildout knows when running buildout is needed).

Downloads and VCS checkouts

Downloading files

batou supports downloading files via HTTP(S) or FTP, for example:

self += Download(
    'http://python.org/ftp/python/3.3.2/Python-3.3.2.tar.bz2',
    checksum='md5:7dffe775f3bea68a44f762a3490e5e28')
class batou.lib.download.Download(url)

Download from the given URL (uses urllib or requests internally).

requests_kwargs

Keyword arguments to pass to requests get method, e.g. to support authentication.

checksum

Checksum of the file to be verified (required). Must be given in the format algorithm:value, where algorithm must be a function of the hashlib stdlib module.

target

Filename to save the download as. [Default: last component of the URL]

Mercurial

self += Clone('https://bitbucket.org/gocept/batou', revision='tip')
class batou.lib.mercurial.Clone(url)

Clone a Mercurial repository from the given URL.

revision

Which revision to clone. At least one of revision or branch is required. If both are given, revision will be used.

branch

The name of a branch to clone. At least one of revision or branch is required. If both are given, branch will be overridden by revision. A clone of a named branch will be updated to the most recent upstream revision of the branch whenever batou is run.

target

Path to clone into (Default: workdir of parent component)

vcs_update

Whether to update the clone with incoming changesets (Default: True). Leaving clones of source code unchanged is often desirable during development.

Git

self += Clone('https://github.com/Pylons/pyramid', revision='HEAD')
class batou.lib.git.Clone(url)

Clone a Git repository from the given URL.

target

Path to clone into (Default: workdir of parent component)

update_unpinned

Update the clone on each batou run. If False, the repository is cloned once and then never updated again. [Default: False]

Note

git.Clone does not support specifying a revision yet.

Subversion

self += Checkout('https://svn.zope.org/repos/main/zopetoolkit/trunk', revision='130345')
class batou.lib.svn.Checkout(url)

Check out a Subversion repository from the given URL.

revision

Which revision to check out (required)

target

Path to clone into (Default: workdir of parent component)

Building software

batou has some support for downloading and compiling software packages, aka the configure-make-make install (CMMI) dance. Example usage:

self += Build(
    'http://python.org/ftp/python/3.3.2/Python-3.3.2.tar.bz2',
    checksum='md5:7dffe775f3bea68a44f762a3490e5e28',
    configure_args='--with-libs=-lssl')
class batou.lib.cmmi.Build(url)

Download archive from url, extract it and run CMMI on it.

checksum
Checksum for download (see batou.lib.download.Download.checksum for details)
prefix

Path to use as prefix for the installation (passed to configure --prefix) [Default: workdir of parent component]

configure_args

String of additional arguments to pass to configure.

build_environment

Dict of variables to add to the environment during all CMMI invocations.

Supervisor

Cronjobs

MySQL

Nagios