batou¶

automating web application deployments

Release v2.3.1.

build-status badge-opensource badge-downloads

batou is a BSD-licensed utility, written in Python, to configure development and production environments for web applications.

Deployments are hard and complicated. Tools like Docker, Puppet, chef, and others exist that try to solve this problem once and for all. However, they usually need you to change your workflow or toolchain massively while still missing important steps.

batou makes deployments more bearable without requiring developers to change their applications. It provides a “one command” approach that should never need additional wrapper scripts.

As a developer all you ever have to run after cloning or updating your project is:

$ git clone https://github.com/myorg/myproject
$ cd myproject
$ ./batou deploy dev

To run a production deployment all you should ever have to run is:

$ cd my-project
$ git pull
$ ./batou deploy prod

Writing a deployment with batou is a two step process:

Step 1: Model your application’s configuration¶

With our component model you write a configuration specification in Python based on a simple API. Components make configuration convergent and idempotent and using Python lets you perform any computation you need. The component model is recursive, so you can refactor complicated components into simpler ones without breaking your setup.

Here is an example application model that installs a Python package into a VirtualEnv and asks Supervisor to run it:

from batou.component import Component
from batou.lib.python import VirtualEnv, Package
from batou.lib.supervisor import Program

class MyApp(Component):

  def configure(self):
    venv = VirtualEnv('3.5')
    self += venv
    venv += Package('myapp')
    self += Program('myapp', command='bin/myapp')

Step 2: Fit your model to your environments¶

Your model from step 1 is abstract: it does not mention the names of the servers you deploy to.

By describing an environment you tell batou how your abstract model should actually be applied: on your local development machine, to a vagrant setup, or on servers on the network.

Here’s an environment specification that sets up an application on multiple hosts and provides an override for the publicly visible address.

[environment]
host_domain = fcio.net

[host:host01]
components = nginx, haproxy, varnish

[host:host02]
components = myapp

[host:host03]
components = myapp

[host:host04]
components = postgresql

[component:nginx]
server_name = staging.example.com

Features¶

Run the same command to deploy locally, to Vagrant, or to remote clusters.
Use different versions of batou in different projects. batou automatically ensures everyone uses the correct version in each project and updates when needed.
Check before deploying whether your configuration is internally consistent and consistent with what has been deployed before.
Predict changes and predict what further changes will be triggered.
Convergent, idempotent components are fast to deploy.
Resume partial deployments where they were aborted.
Store database passwords, SSH keys, SSL certificates or other secret data with on the-fly decryption. Manage access to secrets per environment and user.
Use Jinja2 templates to easily create dynamic configuration.
Dynamically connect services during deployments and track their dependencies.
Few run-time requirements on your servers: only Python 2.7 and SSH are needed.
Use pre-defined components to manage files, python environments, supervisor, cronjobs, and more.
Writing your own components is easy and you can use additional Python package dependencies.

User guide¶

This part of the documentation, begins with some background information about Requests, then focuses on step-by-step instructions for getting the most out of batou.

Introduction¶

Philosophy¶

batou was developed with a number of ideas in mind:

Deploying should always be just a single command.
Python is the language batou uses. In case that you use Python to write your application the version of Python that batou runs on is independent of the version of Python your application runs with.
We expect little from the remote environments regarding software dependencies: Python 3.5+, OpenSSH, and rsync, Mercurial or git are sufficient.
batou does not become an active component during your application’s runtime. batou automates what a sysadmin would do – in a repeatable and documented manner.
It should be easy to switch between the declarative part of the model and the imperative implementation.
Deployment code that becomes too complicated should be easy to simplify by breaking it up into smaller pieces.
batou was not originally intended to perform provisioning or system configuration tasks. However, over time this is likely to evolve.
batou should be working with your existing applications without too many hassles.
No silver bullets: we want to make things simple for you but we do not insulate you from hurting yourself completely.

Name¶

The name “batou” is taken from the animated movie “Ghost in the Shell”.

Kudos¶

batou is built on the shoulders of giants. We’re extremely happy to be part of an active open source community:

Guido and the Python core developers – we could not have built this without such an awesome language.
Jim Fulton (@j1mfulton) and zc.buildout – we love using buildout to create Python application environments, but we started having our own ideas at some point.
Kenneth Reitz (@kennethreitz) and the Requests team – who built the awesome requests library. He and his team have brilliant documentation. When it came to finally document batou we used their works – quite literally and generously! Everything that is awesome about our documentation stems from them. Everything that sucks was caused by us (if you want to help us improve – you’re more than welcome!).
Jeff Forcier (@bitprophet) of Fabric and Paramiko fame – we’ve made huge advances when we switched from bash scripts to Fabric and kept refactoring until our Fabfile slowly turned into batou. We’ve used Paramiko before switching to execnet which gave us a nice way to slowly grow out of Fabric.
Holger Krekel (@hpk42) who gave us py.test and execnet – without those tools we would not have such a nice test suite and execnet is brilliant for lightweight remote Python processing.

Legal¶

The copyright holder for batou is:

Flying Circus Internet Operations GmbH
Leipziger Str. 70/71
06108 Halle (Saale)
GERMANY

The code of batou is licensed under the 2-clause BSD license:

Copyright (c) 2012-2014, gocept gmbh & co. kg
Copyright (c) 2015-2021, Flying Circus Internet Operations GmbH
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
   list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
   this list of conditions and the following disclaimer in the documentation
   and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

The views and conclusions contained in the software and documentation are those
of the authors and should not be interpreted as representing official policies,
either expressed or implied, of Flying Circus Internet Operations GmbH.

Installation¶

batou is installed in each of your projects to ensure that every project is run with the correct version of batou and possibly other dependencies. Typically this means you create a new deployment repository for every project.

Starting a new batou project¶

A new project is started by placing the batou master command into the project and adding that to your repository:

$ mkdir myproject
$ cd myproject
$ git init
$ curl -sL https://raw.githubusercontent.com/flyingcircusio/batou/2.3b3/bootstrap | sh
$ git commit -m "Start a batou project."

Local¶

To run the batou command on your machine you will need to have the following dependencies installed:

Python 3.5+
OpenSSH
GPG (optional, if you want to use encrypted secrets support)
Mercurial, git, or rsync (you only need to have the one installed you actually use to transfer your repository)

Remote¶

To deploy on a remote server, that server needs to have installed:

Python 3.5+
OpenSSH
Mercurial, git or rsync (you only need to have the one installed you actually use to transfer your repository)

Supported Platforms¶

batou is being tested to run on Linux and Mac OS X.

We do not support deploying from or to non-UNIX targets.

batou is written in Python and requires to be run with Python 3.5+.

Optional requirements¶

Depending on the actual components you are going to install, you may need to have those packages installed on your remote machines:

git if using the batou.lib.git.Clone component
make if using the batou.lib.cmmi.Build component
nrpe if using the batou.lib.nagios.NRPEHost component
rsync if using the batou.lib.file.Directory component
subversion if using the batou.lib.svn.Checkout component
unzip and tar if using the batou.lib.archive.Extract component

Distribution-specific installation instructions¶

Install batou’s requirements on Debian / Ubuntu / Mint¶

Depending on you specific distribution, software packages either need to be installed using apt-get or aptitude. apt-get is more specific to Ubuntu and its derivates, whereas aptitude is typically used on Debian. In our examples below, we will stick to apt-get.

Note

RPM-based environments are not native for any of the developers and not tested systematically. Let us know if any of this information is incorrect and we’ll gladly update this.

Python 3¶

In many distributions, Python 3 should ship as the default system Python and should therefore already be installed. If not, you can install it by executing:

sudo apt-get install python3-dev

Note

Although installing using sudo apt-get install python3 would be sufficient, we choose to python3-dev because having the additional development packages installed does not hurt and may help in situations that require functionality that is not included in the base package.

virtualenv¶

virtualenv creates isolated environments for Python, where you can install and upgrade libraries isolated from the system libraries and other environments on the system. You can install virtualenv by executing:

sudo apt-get install python3-virtualenv

Mercurial¶

Mercurial is a distributed source control management tool. You can install Mercurial by executing:

sudo apt-get install mercurial

SSH client¶

The OpenSSH client should ship with nearly any distribution and should already be installed. If not, you can install it by executing:

sudo apt-get install openssh-client

Git¶

Git is a distributed revision control and source code management system. You can install Git by executing:

sudo apt-get install git

In some cases you may want more functionality than the basic git package offers. To get a list of software packages related to git, execute:

sudo apt-cache search . | grep ^git

You can then install the respective package with apt-get like we did for git.

make¶

GNU Make is a tool which controls the generation of executables and other non-source files of a program from the program’s source files. It is typically available on nearly any Linux distribution.

Since it is heavily involved in self-compiling software, it is advisable to additionally install a useful set of packages that helps compiling software:

sudo apt-get install build-essential

NRPE¶

NRPE is an add-on for Nagios that allows you to execute plugins and commands on remote Linux/Unix machines. NRPE can be installed by executing:

sudo apt-get install nagios-nrpe-server

rsync¶

rsync is a file synchronization and file transfer program for Unix-like systems. You can install rsync by executing:

sudo apt-get install rsync

Subversion¶

Subversion is a software versioning and revision control system. To install it, execute:

sudo apt-get install subversion

Note

Additionally installing the package subversion-tools may be helpful when you need more functionality and helper tools for Subversion.

UnZip¶

UnZip is an extraction utility for archives compressed in .zip format. You can install it by executing:

sudo apt-get install unzip

Tar¶

GNU Tar provides the ability to create tar archives, as well as various other kinds of manipulation. It should already ship with nearly any Linux distribution. If not, you can install it by executing:

sudo apt-get install tar

Install batou’s requirements on Fedora / openSUSE / RHEL / CentOS¶

On rpm-based Linux distributions software packages are usually installed using yum.

Note

RPM-based environments are not native for any of the developers and not tested systematically. Let us know if any of this information is incorrect and we’ll gladly update this.

Python 3¶

In many distributions, Python 3 should ship as the default system Python and should therefore already be installed. If not, you can install it by executing:

sudo yum install python-devel

Note

Although installing using sudo yum install python would be sufficient, we choose to python-devel because having the additional development packages installed does not hurt and may help in situations that require functionality that is not included in the base package.

virtualenv¶

virtualenv creates isolated environments for Python, where you can install and upgrade libraries isolated from the system libraries and other environments on the system. You can install virtualenv by executing:

sudo yum install python-virtualenv

Mercurial¶

Mercurial is a distributed source control management tool. You can install Mercurial by executing:

sudo yum install mercurial

SSH client¶

The OpenSSH client should ship with nearly any distribution and should already be installed. If not, you can install it by executing:

sudo yum install openssh-clients

Git¶

Git is a distributed revision control and source code management system. You can install Git by executing:

sudo yum install git

In some cases you may want more functionality than the basic git package offers. To get a list of software packages related to git, execute:

sudo yum search git | grep ^git

You can then install the respective package with yum like we did for git.

make¶

GNU Make is a tool which controls the generation of executables and other non-source files of a program from the program’s source files. It is typically available on nearly any Linux distribution.

Since it is heavily involved in self-compiling software, it is advisable to additionally install a useful set of packages that helps compiling software:

sudo yum groupinstall "Development Tools"

Alternatively, you can install the most basic tools for compiling software by executing:

sudo yum install gcc gcc-c++ kernel-devel

NRPE¶

NRPE is an add-on for Nagios that allows you to execute plugins and commands on remote Linux/Unix machines. NRPE can be installed by executing:

sudo yum install nrpe

rsync¶

rsync is a file synchronization and file transfer program for Unix-like systems. You can install rsync by executing:

sudo yum install rsync

Subversion¶

Subversion is a software versioning and revision control system. To install it, execute:

sudo yum install subversion

Note

Additionally installing the package subversion-tools may be helpful when you need more functionality and helper tools for Subversion.

UnZip¶

UnZip is an extraction utility for archives compressed in .zip format. You can install it by executing:

sudo yum install unzip

Tar¶

GNU Tar provides the ability to create tar archives, as well as various other kinds of manipulation. It should already ship with nearly any Linux distribution. If not, you can install it by executing:

sudo yum install tar

Quickstart¶

Do you want to get started? We’ll go through the steps of developing a project with batou. The steps are built on top of each other, so if you have trouble with a specific step, it might help to review what happened earlier.

Create a new project¶

Deployments with batou are placed in a new directory. For this tutorial we will assume that you’re using git as your version control system. Feel feel free to follow along using Mercurial – batou can handle both.

$ mkdir myproject
$ cd myproject
$ git init
$ curl -sL https://raw.githubusercontent.com/flyingcircusio/batou/main/bootstrap | sh
$ git commit -m "Start a batou project."

The project is now initialized and batou is ready to be used.

Writing a component configuration¶

Once you bootstrapped your batou project you start modelling your configuration. This is done by creating a directory in the components directory and a component.py file in there. You can use those sub- directories to group together things that belong to each component:

$ cd myproject
$ mkdir -p components/myapp

In components/myapp/component.py put the following to manage a very simple application:

components/myapp/component.py (v1)¶

from batou.component import Component
from batou.lib.file import File


class Tick(Component):

    def configure(self):
        self += File(
            'tick.sh',
            mode="rwxr-xr-x", # note: you can also pass an octal number, e.g. 0o755
            content="""\
#!/bin/bash
while true; do
  date
  sleep 1
done
""")

The component has a configure method that is used to build a model of your configuration as a tree of components. By using the syntax self += File(...) you add a File component as a sub-component to your Tick component. Components can thus recursively combine configurations into larger, more complex setups.

The order of sub-components is given by the order they are added to their parent.

Local environments¶

Now, to deploy this “application” we can specify a local environment to deploy directly on the machine you are working on:

$ mkdir environments

Put the following in environments/local/environment.cfg to specify a local configuration that will deploy the “Tick” component:

environments/local/environment.cfg (v1)¶

[environment]
connect_method = local

[hosts]
localhost = tick

Your project now looks like this:

$ tree
.
├── batou
├── components
│   └── myapp
│       └── component.py
└── environments
    └── local
        └── environment.cfg

You can now deploy this environment:

$ ./batou deploy local
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
============================== Preparing ===============================
main: Loading environment `local`...
main: Verifying repository ...
main: Loading secrets ...
======================= Configuring first host =========================
localhost: Connecting via local (1/1)
===================== Connecting remaining hosts =======================
============================== Deploying ===============================
localhost: Deploying component tick ...
     Tick > File(work/tick/tick.sh) > Presence(work/tick/tick.sh)
     Tick > File(work/tick/tick.sh) > Mode(work/tick/tick.sh)
     Tick > File(work/tick/tick.sh) > Content(work/tick/tick.sh)
========================= DEPLOYMENT FINISHED ==========================

When deploying, batou creates a working directory for each component. Your project directory now looks like this:

$ tree
.
├── batou
├── components
│   └── myapp
│       └── component.py
├── environments
│   └── local
│       └── environment.cfg
└── work
    └── tick
        └── tick.sh

The application has been copied over to the work directory and the mode has been set. We can now use it:

$ ./work/tick/tick.sh
Thu Jan 28 21:47:58 CET 2016
Thu Jan 28 21:47:59 CET 2016
Thu Jan 28 21:48:00 CET 2016
Thu Jan 28 21:48:01 CET 2016
Thu Jan 28 21:48:02 CET 2016
Thu Jan 28 21:48:03 CET 2016
^C

When running the deployment again, you see that batou knows what has been deployed and that no action is necessary:

$ ./batou deploy local
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
============================== Preparing ===============================
main: Loading environment `local`...
main: Verifying repository ...
main: Loading secrets ...
======================= Configuring first host =========================
localhost: Connecting via local (1/1)
===================== Connecting remaining hosts =======================
============================== Deploying ===============================
localhost: Deploying component tick ...
========================= DEPLOYMENT FINISHED ==========================

Note

Things in the work directory are generated or data from your application – you should thus add the work directory to your version control systems’ ignore file.

Vagrant environments¶

If you would like to deploy a more complex application, that may involve a webserver, databases, and other auxiliary services, you may prefer to deploy into a virtual machine, instead of deploying those to your local work environment.

For this, batou supports Vagrant. Once you have Vagrant (and VirtualBox) installed, place a Vagrantfile directory in your batou project:

Vagrantfile¶

# -*- mode: ruby -*-
# vi: set ft=ruby :

required_plugins = %w( vagrant-nixos-plugin )
required_plugins.each do |plugin|
      abort("Plugin required: vagrant plugin install #{plugin}") unless Vagrant.has_plugin? plugin
end

Vagrant.configure("2") do |config|
  config.vm.box = "flyingcircus/nixos-19.03-dev-x86_64"
  config.vm.box_version = "= 471.427aac1"

  config.vm.network "private_network", ip: "192.168.50.4"
end

Now, we add a second environment that uses Vagrant to connect and rsync to ensure that our batou project gets synced. The user we want to deploy to in a vagrant box is vagrant and we specify that as the service user. The machine in our Vagrant file is “default”, so we use that as the hostname:

environments/vagrant/environment.cfg¶

[environment]
connect_method = vagrant
update_method = rsync
service_user = vagrant

[hosts]
default = tick

The deployment is invoked similar to the local deployment. Getting the vagrant machine up and running may take a while, though:

$ ./batou deploy vagrant
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `vagrant`...
main: Verifying repository ...
     You are using rsync. This is a non-verifying repository --
     continuing on your own risk!
main: Loading secrets ...
===================== Configuring first host =====================
vagrant: Ensuring machines are up ...
default: Connecting via vagrant (1/1)
=================== Connecting remaining hosts ===================
=========================== Deploying ============================
default: Deploying component tick ...
     Tick > File(work/tick/tick.sh) > Presence(work/tick/tick.sh)
     Tick > File(work/tick/tick.sh) > Mode(work/tick/tick.sh)
     Tick > File(work/tick/tick.sh) > Content(work/tick/tick.sh)
====================== DEPLOYMENT FINISHED =======================

Now, the tick component has been deployed on the virtual machine. We can connect there and see that the same structure has been deployed as previously for our local environment. batou places deployments in the service user’s home directory in a directory named deployment by default:

$ vagrant ssh
[vagrant@nixos:~]$ tree
.
└── deployment
    ├── batou
    ├── components
    │   └── myapp
    │       └── component.py
    ├── environments
    │   ├── local
    │   │   └── environment.cfg
    │   └── vagrant
    │       └── environment.cfg
    ├── Vagrantfile
    └── work
        └── tick
            └── tick.sh
[vagrant@nixos:~]$ ./deployment/work/tick/tick.sh
Thu Jan 28 21:02:31 UTC 2016
Thu Jan 28 21:02:32 UTC 2016
Thu Jan 28 21:02:33 UTC 2016
^C

Similarly, you can deploy into VMs that were set up by the test kitchen integration testing tool:

.kitchen.yml¶

---
driver:
  name: vagrant

platforms:
  - name: ubuntu-16.04

suites:
  - name: tick

environments/kitchen/environment.cfg¶

[environment]
connect_method = kitchen
update_method = rsync
service_user = vagrant

[hosts]
tick-ubuntu-1604 = tick

$ ./batou deploy kitchen

Remote environments¶

To deploy your application into a production environment you will typically use SSH to log in to the remote servers. This works similar to Vagrant environments. To try this out, you will have to replace the host name you see here, with a host that you have access to.

Note

Make sure that the few but important installation requirements requirements for remote hosts are satisfied!

Let’s add a third environment that uses SSH to connect and rsync to ensure that our batou project gets synced. We do not specify the user to deploy to, which means batou will use whatever your SSH configuration is set up to use. To save some typing (and for some other features) we specify a domain name that should be appended to all hosts.

Here’s the full environment configuration:

environments/production/environment.cfg (v2)¶

[environment]
connect_method = ssh
update_method = rsync
host_domain = fcio.net

[hosts]
test01 = tick

Now, to deploy to the remote host:

$ ./batou deploy production
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `production`...
main: Verifying repository ...
     You are using rsync. This is a non-verifying repository --
     continuing on your own risk!
main: Loading secrets ...
===================== Configuring first host =====================
test01: Connecting via ssh (1/1)
=================== Connecting remaining hosts ===================
=========================== Deploying ============================
test01.gocept.net: Deploying component tick ...
     Tick > File(work/tick/tick.sh) > Presence(work/tick/tick.sh)
     Tick > File(work/tick/tick.sh) > Mode(work/tick/tick.sh)
     Tick > File(work/tick/tick.sh) > Content(work/tick/tick.sh)
====================== DEPLOYMENT FINISHED =======================

Overriding configuration per environment¶

Every environment is currently deploying the same configuration of our application. It often is necessary to customize applications based on the environment: either your setup is larger or smaller, or you are using a different web address to access it, or …

To make a component configurable, we add an attribute to the component class:

components/myapp/component.py (v2)¶

from batou.component import Component, Attribute
from batou.lib.file import File


class Tick(Component):

    sleep = Attribute(int, 1)

    def configure(self):
        self += File(
            'tick.sh',
            mode="rwxr-xr-x",
            content="""\
#!/bin/bash
while true; do
  date
  sleep {}
done
""".format(self.sleep))

Attributes are specified with a conversion function or type, to help batou convert them from strings. The second argument given is the default that will be used when attribute is not specified explicitly otherwise. The attribute can then be accessed as usual during the configure method and include this in our application configuration.

To adjust the application for the development environment, we add a new section [component:tick] to the configuration:

environments/local/environment.cfg (v2)¶

[environment]
connect_method = local

[hosts]
localhost = tick

[component:tick]
sleep = 10

Now, let’s deploy this:

$ ./batou deploy local
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `local`...
main: Verifying repository ...
main: Loading secrets ...
===================== Configuring first host =====================
localhost: Connecting via local (1/1)
=================== Connecting remaining hosts ===================
=========================== Deploying ============================
localhost: Deploying component tick ...
     Tick > File(work/tick/tick.sh) > Content(work/tick/tick.sh)
====================== DEPLOYMENT FINISHED =======================

$ cat work/tick/tick.sh
#!/bin/bash
while true; do
  date
  sleep 10
done

$ ./work/tick/tick.sh
Fri Jan 29 09:44:18 CET 2016
Fri Jan 29 09:44:28 CET 2016
^C

Now, our model specific the default configuration and environment-specific overrides allow us to document variations between environments easily.

Templating from files¶

Currently our application configuration has been written directly in Python code. This quickly becomes unwieldy. File components can be used to pull their content from files by specifying the source parameter. The filename is relative to the directory that the component.py is placed in:

components/myapp/component.py (v3)¶

from batou.component import Component, Attribute
from batou.lib.file import File


class Tick(Component):

    sleep = Attribute(int, 1)

    def configure(self):
        self += File(
            'tick.sh',
            mode="rwxr-xr-x",
            source='tick.sh')

Also, templating with Python format strings is very limited. batou thus includes Jinja2 templating, which is enabled by default for all content (independent of whether you specify the content inline or in a separate file).

components/myapp/tick.sh (v1)¶

#!/bin/bash
while true; do
  date
  sleep {{component.sleep}}
done

As a simplification, you can leave out the source parameter if the filename of your template is identical to the name you want it to be in the work directory:

components/myapp/component.py (v4)¶

from batou.component import Component, Attribute
from batou.lib.file import File


class Tick(Component):

    sleep = Attribute(int, 1)

    def configure(self):
        self += File('tick.sh', mode="rwxr-xr-x")

To disable templating and encoding handling, you can use the BinaryFile component:

from batou.component import Component
from batou.lib.file import BinaryFile

class Example(Component):

    def configure(self):
        self += BinaryFile('something.zip')

Storing secrets as encrypted overrides¶

To deploy services we need to move secret data, like database passwords, third party API tokens, SSL certificates, SSH keys, etc. to the target environment. Simply adding those to your deployment scripts has a huge drawback:

it’s unsafe to pass the scripts to a third party like Github
you can’t let others review your code without accidentally revealing those
you can’t manage which of your colleagues can access development, staging, or production secrets.

batou has a built-in method to store secrets in a secure and flexible fashion: secrets are an encrypted version of the overrides we have already used: they are stored in one file per environment, in the same format as the environment configuration, and they control which of your users have access to them.

To start managing a secret, first add a new override attribute database_password to your component:

components/myapp/component.py (v5)¶

from batou.component import Component, Attribute
from batou.lib.file import File


class Tick(Component):

    database_password = Attribute(str, None)
    sleep = Attribute(int, 1)

    def configure(self):
        self += File('tick.sh', mode="rwxr-xr-x")

components/myapp/tick.sh (v2)¶

#!/bin/bash
while true; do
  date
  echo "The database password is {{component.database_password}}"
  sleep {{component.sleep}}
done

Now, to edit the secret, batou provides a set of commands. Let’s edit the secrets file for our production environment:

$ ./batou secrets edit production

This opens up your preferred editor and provides you with a template file:

secrets/production.cfg (decrypted) (v1)¶

[batou]
members =

The members option is used to control who the file will be encrypted for – and thus who can decrypt it in the future. You can specify any ID that GnuPG will accept as a key ID, which usually means you use your email address associated with your key.

To get started with GPG, check the GnuPG HOWTOs.

In addition to your own key, you will have to add the IDs of any of your colleagues that should be able to access this file.

Adding the database password works similar to the environment overrides. Our final file then looks like this:

secrets/production.cfg (decrypted) (v2)¶

[batou]
members = bob@example.com, alice@example.com

[component:tick]
database_password = AfMhV3EDznGbNnzVdxE8

To finish, save the file and exit your editor. batou will be careful not to leave any unencrypted copy of the file around so you do not accidentally check in the unencrypted version.

The encrypted version of the file is now stored in secrets/production.cfg and might look like this:

secrets/production.cfg (encrypted, hexdump -C)¶

00000000  85 02 0c 03 e4 fa c7 12  8f d9 8a 97 01 0f fe 32  |...............2|
00000010  d0 f7 f2 51 77 b5 89 9c  cb 3f 78 15 94 20 d9 dd  |...Qw....?x.. ..|
00000020  7d c3 52 93 e0 cc c5 09  c8 01 bc 32 11 fc 0c d0  |}.R........2....|
00000030  04 13 09 47 ab 2b e2 f0  12 51 fe 26 23 84 5d d6  |...G.+...Q.&#.].|
00000040  19 28 8f 6b f1 4b dc 39  cb 95 dd 31 52 09 b8 f0  |.(.k.K.9...1R...|
00000050  c8 99 0a 86 d3 f1 28 e6  6a 41 39 45 d3 ae 9a 01  |......(.jA9E....|
00000060  07 22 7b ce 7d b4 7c d5  22 16 11 8a 1d a5 9f cb  |."{.}.|.".......|
00000070  96 50 1e 30 16 ec 45 44  10 c0 73 40 e0 97 23 bf  |.P.0..ED..s@..#.|
00000080  ac b9 ea 46 df c4 67 a4  83 ae 4a 24 e4 6e 13 f9  |...F..g...J$.n..|
00000090  ad 9d 87 07 59 d4 46 0b  53 80 50 c1 e0 1d 79 be  |....Y.F.S.P...y.|
000000a0  53 e5 25 15 de 54 6d 65  be 37 35 81 4d 34 55 35  |S.%..Tme.75.M4U5|
000000b0  3c 08 13 db cf 0e e8 f5  9d fb f2 09 ca 22 f2 97  |<............"..|
000000c0  8d bb bb 7c 6e e4 b9 7a  92 eb 75 08 43 15 f9 07  |...|n..z..u.C...|
000000d0  40 24 a8 8e a1 4d 53 6f  7b fe df 07 d8 89 29 ad  |@$...MSo{.....).|
000000e0  a2 df 0d 40 d4 7e 25 b4  b7 cd e9 e8 71 de ff df  |...@.~%.....q...|
000000f0  b8 0d 4f bd 83 63 c0 02  88 d2 79 48 f6 05 76 66  |..O..c....yH..vf|
00000100  76 b3 44 34 36 74 16 b6  1d f1 c0 38 9a ac 33 e5  |v.D46t.....8..3.|
00000110  99 1c 69 10 45 72 28 8f  f8 b5 e1 71 71 fb 8e 8a  |..i.Er(....qq...|
00000120  e7 13 a4 0d dc 1e 42 f1  82 c6 83 cf a0 d8 ef e9  |......B.........|
00000130  f8 33 0c 8c 10 f8 5a 56  69 47 3f d4 65 57 10 1d  |.3....ZViG?.eW..|
00000140  cb 19 4d 51 68 d5 68 fe  82 c1 4f 7b e9 b9 23 12  |..MQh.h...O{..#.|
00000150  04 41 a0 88 14 85 27 23  86 92 77 62 7a 20 80 74  |.A....'#..wbz .t|
00000160  14 9e c7 e8 82 79 1c 10  04 f5 f4 67 94 b7 3e 8e  |.....y.....g..>.|
00000170  30 95 57 ab 0e 20 fb 4a  1f 10 c2 60 38 63 78 41  |0.W.. .J...`8cxA|
00000180  38 32 0d 48 35 3e b2 d1  19 9e 37 02 26 6f 11 c3  |82.H5>....7.&o..|
00000190  83 8f dd fe 11 12 8b c8  43 96 dd 49 b7 db f1 b7  |........C..I....|
000001a0  e9 09 44 8b 23 0d 71 3e  cc f8 a7 d2 e2 79 65 94  |..D.#.q>.....ye.|
000001b0  53 36 c6 43 19 df 7d 69  33 cb c0 ab 4c c3 db 7f  |S6.C..}i3...L...|
000001c0  b3 8f a9 35 a6 7d fb 94  6f df 04 37 88 44 a4 df  |...5.}..o..7.D..|
000001d0  66 39 2d 17 f2 a9 57 60  2f 11 10 ff 43 03 58 4c  |f9-...W`/...C.XL|
000001e0  5f bd f1 17 0c e8 c6 60  69 fe 6e 86 65 fa 12 2f  |_......`i.n.e../|
000001f0  12 91 80 9e 5d f7 da df  c0 6c 2a 90 40 94 f0 07  |....]....l*.@...|
00000200  3e a2 d6 07 83 71 28 e0  d3 26 76 51 d6 23 49 d2  |>....q(..&vQ.#I.|
00000210  95 01 24 3c 40 18 2c 05  65 4b c4 4a 86 3f 67 db  |..$<@.,.eK.J.?g.|
00000220  ac 5c e8 3e 49 90 5f f9  66 e5 0f 35 ad e8 99 57  |.\.>I._.f..5...W|
00000230  13 b5 4a b4 59 38 de a0  1c 89 67 e3 2f 3e b1 d8  |..J.Y8....g./>..|
00000240  0b 37 b4 d6 58 ee bf 47  f6 53 64 ed 70 ba 37 f5  |.7..X..G.Sd.p.7.|
00000250  be 56 e0 69 52 18 0e 04  ff e4 2d 05 43 c5 a5 4f  |.V.iR.....-.C..O|
00000260  04 a5 4e a1 d7 c4 5f 65  02 02 5c 29 fe 2c 34 c5  |..N..._e..\).,4.|
00000270  1b 65 4c 88 85 8c 6f ce  15 4e e7 43 2d db fb eb  |.eL...o..N.C-...|
00000280  28 b6 b2 2b b1 cc b2 04  1b bd 17 a5 89 5c fd 3f  |(..+.........\.?|
00000290  c7 bf 60 df 58 8d 41 35  2a 14 9f e5 99 83 a1 97  |..`.X.A5*.......|
000002a0  84 5a 5d ae 83 0e                                 |.Z]...|
000002a6

Now, deploying this will cause GPG to be invoked on your local machine (there is no need to install GPG on the remote hosts) and transfer the decrypted secrets securely through the SSH connection:

$ ./batou deploy production
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `production`...
main: Verifying repository ...
     You are using rsync. This is a non-verifying repository --
     continuing on your own risk!
main: Loading secrets ...

You need a passphrase to unlock the secret key for
user: "Bob Sample <bob@example.com>"
4096-bit RSA key, ID 4DE34B34, created 2013-10-05 (main key ID 4DE34B34)

===================== Configuring first host =====================
test01: Connecting via ssh (1/1)
=================== Connecting remaining hosts ===================
=========================== Deploying ============================
test01.gocept.net: Deploying component tick ...
     Tick > File(work/tick/tick.sh) > Content(work/tick/tick.sh)
====================== DEPLOYMENT FINISHED =======================

$ ssh test01.fcio.net
$ cat ./deployment/work/tick/tick.sh
while true; do
  date
  echo "The database password is AfMhV3EDznGbNnzVdxE8"
  sleep 1
done

If you deal with a growing number of environments and users, you can use the commands batou secrets add and batou secrets remove to manage access for users to multiple or all environments quickly.

Storing additional secrets as separate files¶

Sometimes the way to store secrets as overrides in an INI-style file is inconvenient: larger amounts of data (think SSL or SSH private keys) or structured data that is hard to embed syntactically correct in the INI format (think YAML).

For that you can split off this data into separate files (you still need to edit the main secrets file to tell batou which keys to encrypt for):

$ ./batou secrets edit production secretdata.yaml

This opens up your preferred editor and provides you with an empty file that you can edit.

secrets/production-secretdata.yaml (decrypted)¶

-
  a: 1
  list: 2
-
  with: 4
  multiple: 5
  dicts: 6

batou keeps the file suffix in the temporary file, so that your editor should be showing you appropriate syntax highlighting.

Whenever you edit one of the secret files of an environment all files will be reencrypted to be sure that they are encrypted with a consistent set of keys.

To use those secret files, you can simply retrieve them from the environment object in your deployment code via the secret_files dict:

components/myapp/component.py (v5)¶

from batou.component import Component
from batou.lib.file import File


class Tick(Component):

    def configure(self):
        self += File(
          'secrets.yaml',
          mode="rwxr-xr-x",
          content=self.environment.secret_files['secretdata.yaml'])

Using version control to ensure consistent deployments¶

Working in a team means that some of your colleagues may be deploying code independent of you. Deploying when a colleague may have deployed last Friday and forgot to push his code can lead to bad things happening …

batou supports integrating with Git or Mercurial to verify the repository integrity on the target systems. This consists of multiple steps:

Ensuring you do not have uncommitted changes and no unpushed commits in your repository.
Shipping changes to the remote servers (either via pulling from a central server or using an export/import mechanism if you server has no access to your central repository).
Switching to an environment-specific branch.
Ensuring the local working copy and the remote working copy are the same.

To leverage those features in batou, you have to select an update method in your environment that is not rsync. batou supports git-pull, git- bundle, hg-pull and hg-bundle.

Lets use git-bundle for this example:

environments/production/environment.cfg (v1)¶

[environment]
connect_method = ssh
update_method = git-bundle
branch = production
host_domain = fcio.net

[hosts]
test01 = tick

To deploy to production we now end up with the following workflow:

$ git checkout -tb production
$ git merge master
$ ./batou deploy production
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `production`...
main: Verifying repository ...
main: Loading secrets ...
===================== Configuring first host =====================
test01: Connecting via ssh (1/1)
=================== Connecting remaining hosts ===================
=========================== Deploying ============================
test01.gocept.net: Deploying component tick ...
====================== DEPLOYMENT FINISHED =======================

Now, if the remote server would have incompatible changes, batou would inform you and refuse to deploy until you clean up the situation:

$ ./batou deploy remote
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `production`...
main: Verifying repository ...
main: Loading secrets ...
===================== Configuring first host =====================
test01: Connecting via ssh (1/1)
ERROR: LANG=C LC_ALL=C LANGUAGE=C git bundle create /var/folders/
24/4w5jy6r532d0k5mgrrgkt0qm0000gn/T/tmp1ezxQj
36052b45d6cfd7ec7202cf34301fba76c1360c0e..production
     Return code: 1
STDOUT

STDERR
     fatal: Invalid revision range
      36052b45d6cfd7ec7202cf34301fba76c1360c0e..production
     error: rev-list died

======================= DEPLOYMENT FAILED ========================

Note

The error message may be cryptic at times, depending on the error situation of your version control system, but you’ll get the gist to look at the repository situation.

Downloading and building software¶

A typical action that you encounter when configuring software is to download and compile them. This is also known as the “CMMI” method: configure, make, make install.

batou provides a standard library component that subsumes downloading and building standard packages:

components/zlib/component.py¶

from batou.component import Component
from batou.lib.cmmi import Build

class Zlib(Component):

  def configure(self):
    self += Build(
      'http://zlib.net/zlib-1.2.8.tar.gz',
      checksum='md5:1142191120b845f4ed8c8c17455420ac')

Deploying this demonstrates how the build component includes sub-components, which you can also use to perform more fine-grained builds:

$ ./batou deploy local
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `local`...
main: Verifying repository ...
main: Loading secrets ...
===================== Configuring first host =====================
localhost: Connecting via local (1/1)
=================== Connecting remaining hosts ===================
=========================== Deploying ============================
localhost: Deploying component zlib ...
   Zlib > Build(zlib-1.2.8.tar.gz) > Extract(zlib-1.2.8.tar.gz) >
      Untar(zlib-1.2.8.tar.gz) > Directory(work/zlib/zlib-1.2.8)
   Zlib > Build(zlib-1.2.8.tar.gz) > Extract(zlib-1.2.8.tar.gz) >
      Untar(zlib-1.2.8.tar.gz)
   Zlib > Build(zlib-1.2.8.tar.gz) >
      Configure(/private/tmp/myproject/work/zlib/zlib-1.2.8)
   Zlib > Build(zlib-1.2.8.tar.gz) >
      Make(/private/tmp/myproject/work/zlib/zlib-1.2.8)
localhost: Deploying component tick ...
====================== DEPLOYMENT FINISHED =======================

Managing Python environments with VirtualEnv and Pip¶

Installing Python software has some best practices, that involve managing virtual environments, using Pip or zc.buildout. batou provides components that let you manage virtual environments (and automatically update them and keep them in order) for different Python versions.

batou includes pre-defined versions of virtualenv and Pip for the various supported Python versions.

Here is how a component installing Python packages looks like:

components/flask/component.py (v1)¶

from batou.component import Component
from batou.lib.python import VirtualEnv, Package


class Flask(Component):

    def configure(self):
        venv = VirtualEnv('3.5')
        self += venv

        venv += Package('Flask', version='0.10.1')

Add the “Flask” compoment to your local configuration:

environments/local/environment.cfg (v3)¶

[environment]
connect_method = local

[hosts]
localhost = tick, flask

[component:tick]
sleep = 10

$ ./batou deploy local
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `local`...
main: Verifying repository ...
main: Loading secrets ...
===================== Configuring first host =====================
localhost: Connecting via local (1/1)
=================== Connecting remaining hosts ===================
=========================== Deploying ============================
localhost: Deploying component tick ...
localhost: Deploying component zlib ...
localhost: Deploying component flask ...
  Flask > VirtualEnv(3.5) > VirtualEnvPy3_5 >
    VirtualEnvDownload(13.1.2) > Download(https://pypi.fcio.net/packages/source/v/virtualenv/virtualenv-13.1.2.tar.gz)
  Flask > VirtualEnv(3.5) > VirtualEnvPy3_5 >
    VirtualEnvDownload(13.1.2) > Extract(virtualenv-13.1.2.tar.gz) >
    Untar(virtualenv-13.1.2.tar.gz)
  Flask > VirtualEnv(3.5) > VirtualEnvPy3_5 > VirtualEnvDownload(13.1.2)
  Flask > VirtualEnv(3.5) > VirtualEnvPy3_5
  Flask > VirtualEnv(3.5) > Package(Flask==0.10.1)
====================== DEPLOYMENT FINISHED =======================

Note

Due to the work directory separation for each component you can easily manage many different virtual environments with different packages and different Python versions.

Managing Python environments with zc.buildout¶

If you manage your Python environment with zc.buildout, you can automate that easily as well. Place your buildout.cfg next to your component.py and use the following component:

components/flask/component.py (v2)¶

from batou.component import Component
from batou.lib.python import VirtualEnv, Package


class Flask(Component):

    def configure(self):
        venv = VirtualEnv('3.5')
        self += venv

        venv += Package('Flask', version='0.10.1')

components/flask/buildout.cfg¶

[buildout]
parts = flask
allow-picked-versions = false
versions = versions

[flask]
recipe = zc.recipe.egg
eggs = Flask

[versions]
zc.recipe.egg = 2.0.3
zc.buildout = 2.5.0
setuptools = 19.6.1
Flask = 0.10.1
itsdangerous = 0.24
Jinja2 = 2.8
Werkzeug = 0.11.3
MarkupSafe = 0.23

$ ./batou deploy local
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `local`...
main: Verifying repository ...
main: Loading secrets ...
===================== Configuring first host =====================
localhost: Connecting via local (1/1)
=================== Connecting remaining hosts ===================
=========================== Deploying ============================
localhost: Deploying component flask ...
   Flask > Buildout > File(work/flask/buildout.cfg) >
      Presence(work/flask/buildout.cfg)
   Flask > Buildout > File(work/flask/buildout.cfg) >
      Content(work/flask/buildout.cfg)
   Flask > Buildout > VirtualEnv(3.5) > VirtualEnvPy3_5
   Flask > Buildout > VirtualEnv(3.5) > Package(setuptools==19.6.1)
   Flask > Buildout > VirtualEnv(3.5) > Package(zc.buildout==2.5.0)
   Flask > Buildout
====================== DEPLOYMENT FINISHED =======================

Registering programs with supervisor¶

If you’re used to running your programs in supervisor, batou can help you with a pre-made Supervisor component. You can simply enable it by importing it into a component file and registering it with your environment:

components/supervisor/component.py¶

from batou.lib.supervisor import Supervisor

environments/local.cfg (v4)¶

[environment]
connect_method = local

[hosts]
localhost = tick, supervisor

To make your program run within supervisor, you register it by configuring a program component:

components/myapp/component.py (v6)¶

from batou.component import Component, Attribute
from batou.lib.file import File
from batou.lib.supervisor import Program

class Tick(Component):

    sleep = Attribute(int, 1)

    def configure(self):
        self += File('tick.sh', mode="rwxr-xr-x")
        self += Program('tick', command='tick.sh')

Deploying it enables supervisor in its own virtualenv and starts the registered program:

$ ./batou deploy local
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `local`...
main: Verifying repository ...
main: Loading secrets ...
===================== Configuring first host =====================
localhost: Connecting via local (1/1)
=================== Connecting remaining hosts ===================
=========================== Deploying ============================
localhost: Deploying component supervisor ...
  Supervisor > Buildout > File(work/supervisor/buildout.cfg) >
    Presence(work/supervisor/buildout.cfg)
  Supervisor > Buildout > File(work/supervisor/buildout.cfg) >
    Content(work/supervisor/buildout.cfg)
  Supervisor > Buildout > VirtualEnv(3.5) > VirtualEnvPy3_5
  Supervisor > Buildout > VirtualEnv(3.5) >
    Package(setuptools==19.2)
  Supervisor > Buildout > VirtualEnv(3.5) >
    Package(zc.buildout==2.5.0)
  Supervisor > Buildout
  Supervisor > Directory(work/supervisor/etc/supervisor.d)
  Supervisor > File(work/supervisor/etc/supervisord.conf) >
    Presence(work/supervisor/etc/supervisord.conf)
  Supervisor > File(work/supervisor/etc/supervisord.conf) >
    Content(work/supervisor/etc/supervisord.conf)
  Supervisor > Directory(work/supervisor/var/log)
  Supervisor > RunningSupervisor
localhost: Deploying component tick ...
  Tick > Program(tick) > File(work/supervisor/etc/supervisor.d/tick.conf)
    > Presence(work/supervisor/etc/supervisor.d/tick.conf)
  Tick > Program(tick) > File(work/supervisor/etc/supervisor.d/tick.conf)
    > Content(work/supervisor/etc/supervisor.d/tick.conf)
  Tick > Program(tick)
====================== DEPLOYMENT FINISHED =======================
$ ./work/supervisor/bin/supervisorctl
tick                             RUNNING   pid 30992, uptime 0:01:37
supervisor> ^D

Working with network addresses¶

Network addresses in batou typically appear in various config files: for clients to find their servers and for servers to configure their bind addresses.

A good practice is to use hostnames when configuring a client, so the IP of the server can be dynamically resolved when connecting. For servers, bind addresses should be configured with IPs because resolvers may not be reachable at the time the server is started and the server may not start reliably under that condition.

batou has a utility object Address that can be used for doing DNS lookups depending on what you’re doing:

>>> from batou.utils import Address
>>> address = Address('localhost', 8080)
>>> str(address.listen)
'127.0.0.1:8080'
>>> str(address.connect)
'localhost:8080'

You can also use the Address type for converting overrides automatically:

from batou.component import Component, Attribute
from batou.utils import Address

class MyApp(Component):

    address = Attribute(Address, 'localhost:8080')

To have services talk to each other on different machines, you can use the host attribute of a component to get the name of the host that the component is configured on. Specifically we recommend using host.fqdn, which can be used with the built-in Jinja2 templating:

from batou.component import Component, Attribute
from batou.utils import Address

class MyApp(Component):

    address = Attribute(Address, '{{host.fqdn}}:8080')

    # address is now environment- and host-specific, e.g.:
    # address == Address('test01.fcio.net', 8080)

To override this attribute in an environment configuration you simply do a regular override using templating:

[component:myapp]
address = {{host.fqdn}}:9000

or static addresses:

[component:myapp]
address = 192.168.0.1:8080

Registering and discovering services¶

If you deploy your application on multiple hosts, you need to communicate between components what is configured where and how often. For example, to configure a load balancer, you need to see where the application servers are installed.

batou provides an API that allows a component to provide a resource and other components to require a resource. This API has the following features:

handling scalars (single values) and list-oriented resources
filtering for resources by key and by host
warning if provided resources are never used
warning if required resources are never provided
warning if multiple resources are provided where exactly one is expected

Additionally, batou orders components based on their resource requirements: a component that provides a resource will be deployed before the component requiring it. This has component granularity, so batou may switch deploying components between hosts if that is what the dependencies require.

To provide a resource, you call provide with a key (that you can define as you like.)

from batou.component import Component, Attribute
from batou.utils import Address

class MyApp(Component):

    address = Attribute(Address, '{{host.fqdn}}:8080')

    def configure(self):
        self.provide('application', self.address)

If you call provide from multiple components, then batou will automatically maintain a list of those items.

To get the registered resources for a key, you call require with the key you are interested in:

from batou.component import Component
from batou.lib.file import File

class Loadbalancer(Component):

    def configure(self):
        application_servers = self.require('application')

        self += File('loadbalancer', content='''\
{% for server in component.application_servers %}
server connect={{server.connect}}
{% endfor %}
''')

To filter for resources from the same host as your component and you expect a single value, you can do this:

from batou.component import Component

class Nginx(Component):

    def configure(self):
        varnish = self.require_one('varnish', host=self.host)

Note

You can pick keys as you like, but if you re-used standard components then you may need to consider collisions.

Checking a deployment configuration before running it¶

If you are ready to deploy something but want to wait until a certain point in time, then you can use -c (aka --consistency-only) with the deployment to see whether the configuration for the target environment is consistent without performing any action. This lets you debug your deployments early in the release cycle:

$ ./batou deploy -c production
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `production`...
main: Verifying repository ...
main: Loading secrets ...
===================== Configuring first host =====================
test01: Connecting via ssh (1/1)
========================= CHECK FINISHED =========================

If the check fails you get the error message that you would otherwise have gotten when running the deployment for real:

$ ./batou deploy -c production
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
=========================== Preparing ============================
main: Loading environment `production`...
main: Verifying repository ...
main: Loading secrets ...
===================== Configuring first host =====================
test01: Connecting via ssh (1/1)
ERROR: Overrides for undefined attributes
           Host: test01
      Component: Tick
     Attributes: foobar
ERROR: Unused provided resources
     supervisor: [<Supervisor (test01) "Supervisor">]
ERROR: 1 remaining unconfigured component(s)
================ 3 ERRORS - CONFIGURATION FAILED =================
========================== CHECK FAILED ==========================

Predicting the changes a deployment will cause¶

In addition to performing a consistency check, you can also perform a prediction which changes would during a deployment. This is more expensive than a pure consistency check, because it connects to all hosts and runs all verify() commands. However, it gives you a rough estimate of the changes that would happen.

$ ./batou deploy tutorial -P
batou/2.0b13.dev0 (cpython 3.7.5-final0, Darwin 19.5.0 x86_64)
============================ Preparing ============================
main: Loading environment `tutorial`...
main: Verifying repository ...
main: Loading secrets ...
====================== Configuring first host =====================
localhost: Connecting via local (1/1)
==================== Connecting remaining hosts ===================
================== Predicting deployment actions ==================
localhost: Deploying component tick ...
     Tick > File(work/tick/tick.sh) > Content(work/tick/tick.sh)
localhost: Deploying component supervisor ...
     Supervisor
================== DEPLOYMENT PREDICTION FINISHED =================

The prediction assumes that an exception during verify() indicates that a previous component that had changes would have performed an action that would trigger the component with a failing verification to perform an update. Under very specific conditions this could also be a real error when deploying, but in most cases this is not true.

batou can not predict all possible changes, especially if you depend on timestamps of files on the disk that are updated due to performing an action (which doesn’t happen when predicting).

Remember, the prediction is intended to give you a rough estimate in which top- level components you can expect changes. There should always be an indicator for some change if a component is affected. After that you can consider the implications, e.g. to determine when would be a good time to run the deployment if you expect downtime.

Updating batou in an existing project¶

You can update batou to a specific version for your project with a single command:

$ ./batou update --version 1.2
Updating batou to 1.2
See the changelog at https://batou.readthedocs.io/en/latest/changes.html

$ git add batou
$ git commit -m 'Update batou to 1.2'

Now, check in the changed batou file and all your colleagues will get the new batou version automatically the next time the pull and run batou.

Advanced Usage¶

Writing a custom component (TODO)¶

Debugging batou runs¶

Using a debugger¶

batou comes with remote-pdb pre-installed. When running on Python 3.7+ 1 you can use breakpoint() to drop into the debugger. You need telnet or netcat to connect to the hostname and port displayed.

If you are using the default configuration, call:

$ nc -C 127.0.0.1 4444
or
$ telnet 127.0.0.1 4444

If you are debugging a remote deployment you should create a port forward beforehand like this:

$ ssh -L 4444:localhost:4444 my.remote.host.dev

You are able to configure hostname and port. For details see the documentation of remote-pdb. This works both for local and remote deployments. The environment variables for host and port are propagated to the remote host.

Example shell session for debugging using a custom port:

$ REMOTE_PDB_PORT=4445 ./batou deploy dev
batou/2.3b2.dev0 (cpython 3.6.15-final0, Darwin 20.6.0 x86_64)
============================= Preparing ============================
main: Loading environment `dev`...
main: Verifying repository ...
main: Loading secrets ...
============================= Connecting ... =======================
localhost: Connecting via local (1/1)
============================= Configuring model ... ================
RemotePdb session open at 127.0.0.1:4445, waiting for connection ...

Example for a second terminal where the opened port gets connected to:

$ nc -C 127.0.0.1 4445
> /.../components/supervisor/component.py(32)configure()
-> if self.host.platform in ('gocept.net', 'fcio.net'):
(Pdb)

Using “print debugging”¶

Inside a component you can use self.log("output") to print to the console.
You can call batou with the -d flag to enable debugging output during the deployment run.

Using 3rd party libraries within batou¶

Sometimes, when writing custom components, you may need additional Python packages, for example to configure databases by connecting directly to their SQL interface instead of using their command line clients.

You can use additional Python packages by adding a requirements.txt file to your batou project repository:

$ tree
.
├── batou
├── components
│   └── myapp
│       └── component.py
├── environments
│   └── local
│       └── environment.cfg
└── requirements.txt

requirements.txt¶

sqlalchemy

The next time when you call batou the dependencies will be automatically updated. When deploying then the requirements will also be installed on the remote hosts.

$ ./batou
Installing sqlalchemy
usage: batou [-h] [-d] {deploy,secrets,init,update} ...

Note

batou already provides a number of packages that it depends on. If you create contradicting requirements then this may lead to batou failing. You will see pip complaining in that case.

Multiple components in a single component.py (TODO)¶

Skipping individual hosts or components when deploying (TODO)¶

Events (TODO)¶

Using bundle transfers if the repository server is not reachable from your remote server (TODO)¶

Timeout (TODO)¶

VFS mapping for development (TODO)¶

VFS mapping with explicit rewrite rules (TODO)¶

Extended service discovery options (TODO)¶

Platform-specific components¶

New in version 1.4.

Platform-specific components allow to customize behavior depending on the system or “platform” the target system runs as. Examples:

Production system on Gentoo, local development on Ubuntu, or
All VMs on Ubuntu but Oracle is being run with RedHat.

To define a platform specific aspects, you use the platform class decorator. Example:

import batou.component
import batou.lib.file


class Test(batou.component.Component):

    def configure(self):
        self += batou.lib.file.File('base-component')


@batou.component.platform('nixos', Test)
class TestNixos(batou.component.Component):

    def configure(self):
        self += batou.lib.file.File('i-am-nixos')


@batou.component.platform('ubuntu', Test)
class TestUbuntu(batou.component.Component):

    def configure(self):
        self += batou.lib.file.File('i-am-ubuntu')

The platform is then defined in the environment:

[environment]
platform = default-platform

[host:nixos]
# Host specifc override:
platform = nixos
components = test

[host:ubuntu]
# Host specifc override:
platform = ubuntu
components = test

Host-specific data¶

New in version 1.5.

Host-specific data allows to set environment dependent data for a certain host. It looks like this in an environment configuration:

[host:myhost00]
components = test
data-alias = nice-alias.for.my.host.example.com

In a component you can access all data attributes via the host’s data dictionary:

def configure(self):
    alias = self.host.data['alias']

The data- prefix was chosen in resemblance of the HTML standard.

DNS overrides¶

New in version 1.6

When migrating services automatic DNS lookup of IP addresses to listen on can be cumbersome. You want to deploy the service before the DNS changes become active. This is where DNS overrides can help.

The DNS overrides short circuit the resolving completely for the given host names.

Example:

[environment]
...

[resolver]
www.example.com =
    3.2.1.4
    ::2

Whenever batou configuration (i.e. batou.utils.Address) looks up www.example.com it will result in the addresses 3.2.1.4 and ::2.

The overrides support IPv4 and IPv6. You should only set one IP address per type for each host name.

Note

You cannot override the addresses of the configured hosts. The SSH connection will always use genuine name resolving.

context manager (TODO)¶

last_updated (TODO)¶

prepare, `|=`, `component._` (TODO)¶

workdir overriding (TODO)¶

Importing components from a different component.py¶

The component configuration in the ./components folder is not a Python package: it has no __init__.py and should not have one. That’s why is not possible import a component into another one:

# This will not work
from components.nginx.component import MyAddress

In some rare circumstances it might be necessary to have this kind of import. There are two options:

Import from batou.c.
Create an extension module which can be imported.

Import from `batou.c`¶

There is a special module batou.c which is dynamically populated with all the classes in all component.py files.

Given the following tree:

$ tree
.
├── batou
├── components
│   └── myapp
│       └── component.py
│   └── myconfig
│       └── component.py
├── environments
│   └── local
│       └── environment.cfg
├── requirements.lock
└── requirements.txt

If myconfig/component.py looks like this:

from batou.component import Component

class MyAddress(Component):
    """Some custom address."""

    ipv4 = "0.0.0.0"

myapp/component.py could import MyAddress like this:

from batou.component import Component
import batou.c

class MyApp(Component):
    """Some custom address."""

    def configure(self):
        """Configure MyApp."""
        self.address = batou.c.myconfig.MyAddress(ipv4="127.0.0.1")
        self += self.address
        self.ip = self.address.ipv4

This way, importing is only possible and reasonable inside methods of the component, which are executed after initial configuration such as configure (). It cannot be used to create an attribute on a class or import a base class for the current component file.

Caution

The components are loaded alphabetically, which can be an issue for the import.

Create an extension module¶

Another option to share code between different component files is to create a custom extension module. This can either be a separate repository like batou_ext or batou_scm but for a light weight start it can be included in the deployment repository. Have a look at the setup.py and other files for inspiration.

$ tree
.
├── batou
├── batou_myapp
│   ├── setup.py
│   └── src
│    └── batou_myapp
│        └── utils.py
├── components
│   └── myapp
│       └── component.py
├── environments
│   └── local
│       └── environment.cfg
├── requirements.lock
└── requirements.txt

# batou_myapp/src/batou_myapp/utils.py
from batou.component import Component

class MyAddress(Component):
    """Some custom address."""

    ipv4 = "0.0.0.0"

class MyAppBase(Component):
    """Base component for all apps."""

# myapp/component.py
from batou.component import Component
from batou_myapp.utils import MyAppBase
from batou_myapp.utils import MyAddress

class MyApp(MyAppBase):
    """Some custom address."""

    address = MyAddress(ipv4="127.0.0.1")

    def configure(self):
        """Configure MyApp."""
        self.ip = self.address.ipv4

The requirements.txt has to be adapted to include the new extension module as requirement to be installed directly from source.

# requirements.txt
batou==2.3b2
-e ./batou_myapp

1: On Python 3.6 you have to use from remote_pdb import set_trace; set_trace().

Command line¶

If you are looking for information on what commands the batou CLI provides then this is for you.

Command Line Usage¶

General options¶

usage: batou [-h] [-d] {deploy,secrets,init} ...

batou v2.0b12: multi-(host|component|environment|version|platform) deployment

positional arguments:
  {deploy,secrets,init}
    deploy              Deploy an environment.
    secrets             Manage encrypted secret files. Relies on gpg being
                        installed and configured correctly.

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Enable debug mode. (default: False)

batou deploy¶

usage: batou deploy [-h] [-p PLATFORM] [-t TIMEOUT] [-D] [-c] [-P] [-j JOBS]
                    environment

positional arguments:
  environment           Environment to deploy.

optional arguments:
  -h, --help            show this help message and exit
  -p PLATFORM, --platform PLATFORM
                        Alternative platform to choose. Empty for no platform.
  -t TIMEOUT, --timeout TIMEOUT
                        Override the environment's timeout setting
  -D, --dirty           Allow deploying with dirty working copy or outgoing
                        changes.
  -c, --consistency-only
                        Only perform a deployment model and environment
                        consistency check. Only connects to a single host.
                        Does not touch anything.
  -P, --predict-only    Only predict what updates would happen. Do not change
                        anything.
  -j JOBS, --jobs JOBS  Defines number of jobs running parallel to deploy. The
                        default results in a serial deployment of components.
                        Will override the environment settings for operational
                        flexibility.

batou secrets edit¶

usage: batou secrets edit [-h] [--editor EDITOR] environment

positional arguments:
  environment           Environment to edit secrets for.

optional arguments:
  -h, --help            show this help message and exit
  --editor EDITOR, -e EDITOR
                        Invoke EDITOR to edit (default: $EDITOR or vi)

batou secrets summary¶

Show an overview of which users have access to what encrypted secrets.

usage: batou secrets summary [-h]

optional arguments:
  -h, --help  show this help message and exit

Example:

$ ./batou secrets summary

production
   members
    - alice@example.com
   secret files
    - secrets.yaml

tutorial
   members
    - alice@example.com
    - bob@example.com
   secret files
    (none)

batou secrets add¶

usage: batou secrets add [-h] [--environments ENVIRONMENTS] keyid

positional arguments:
  keyid                 The user's key ID or email address

optional arguments:
  -h, --help            show this help message and exit
  --environments ENVIRONMENTS
                        The environments to update. Update all if not
                        specified.

batou secrets remove¶

usage: batou secrets remove [-h] [--environments ENVIRONMENTS] keyid

positional arguments:
  keyid                 The user's key ID or email address

optional arguments:
  -h, --help            show this help message and exit
  --environments ENVIRONMENTS
                        The environments to update. Update all if not
                        specified.

Components¶

This is the list of components that batou provides – builtin and through the batou_ext package:

Managing files and directories¶

Files and Templates¶

The File component has been developed with Puppet’s file type in mind. It accepts a very similar parameter set and has almost identical features.

You can use it to manage files, directories, and symlinks, and you can specify content (literally or as Jinja templates). You can also manage the Unix attributes and control whether leading directories should be managed or not.

The most basic usage is simply:

self += File('myfile')

This example creates a file at work/mycomponent/myfile, taking the contents from a file of the same name in the component’s directory (i.e. components/mycomponent/myfile). By default, the source file is run through Jinja, with the file’s parent component made available as component.

class batou.lib.file.File(path)¶

Creates a file. The main parameter for File is the target path. A File instance has an attribute path containing the full, absolute path to the resulting file.

File accepts the following additional parameters:

source¶: Filename of the source file to be used as the File’s content (absolute path or relative to the component’s directory). [Default: same as target path]

content¶: Literal file contents as a string.

is_template¶: Process file contents as Jinja template. [Default: True]

template_context¶: Object to make available as component to the Jinja template. [Default: File’s parent component]

template_args¶: Dict of additional arguments to make available to the Jinja template.

encoding¶: Encoding for the file contents [Default: utf-8]

owner¶: Unix owner username.

group¶: Unix group name.

mode¶: Unix permission mode. Can be given as an integer value (0o755) or as an octal integer string (‘755’) or as a unix mode string similar to the output of ls -l (‘rwx–x–x’).

leading¶: Create leading directories that were given in the target path. [Default: False]

ensure¶: Type of object to be created: ‘file’, ‘directory’, or ‘symlink’. This is useful for complex situations (e.g. creating a symlink with special ownership), for simple situations it’s probably more readable to use Directory or Symlink.

link_to¶: Source of symlink (for ensure = ‘symlink’)

sensitive_data¶: Mark a file as sensitive so its content is not exposed by the (diff-)output of batou. This is useful in situations where the rendered file contains a password or other sensitive data. [Default: False]

class batou.lib.file.BinaryFile(path)¶

Subclass of batou.lib.file.File. Creates a non-template binary file.

Directories¶

class batou.lib.file.Directory(path)¶: Creates a directory. The main parameter is the target path.

source¶: Path to a source directory whose contents are to be synchronized to the target path (uses rsync internally).

exclude¶: List of file names or patterns that should not be synchronized to the target path (passed to rsync as --exclude argument, see the rsync documentation for details).

class Symlink(target, source)¶: Creates a symlink at target by linking to source.

Removing files¶

Removal of obsolete things is a difficult topic in the convergence paradigm. If in the past we created a file foo, but now it is not used anymore, the code that originally said, “please manage foo”, will not be there anymore. This means that nobody knows that the file foo that is still lying around on the production system is not actually in use anymore. In most cases, a few stray files do not matter, but in case they do, the deployment code has to explicitly state that something should not be present anymore.

class batou.lib.file.Purge(pattern)¶: Ensures that a set of files (given as a glob pattern) does not exist.

Extracting archive files¶

batou can extract archive files in Tar, Zip, and DMG (on OSX target machines) format:

class batou.lib.archive.Extract(archive)¶

The main parameter is the archive filename (relative to the component’s directory). The archive format is determined according to the file name extension (‘.tar’, ‘.tar.gz’, ‘.tgz’, ‘.tar.bz2’, ‘.tar.xz’ use tar, ‘.zip’ uses unzip and ‘.dmg’ uses hdiutil). The following additional parameters are supported:

target¶: Target directory to extract the archive into. Directory is created if it does not exist (compare create_target_dir). [Default: base name of the archive file]

create_target_dir¶: Extract into the directory given in target. Set to False to extract directly into the work directory. [Default: True]

strip¶: Only for tar archives: number of directories contained in the archive to strip off (see the tar documentation for details) [Default: 0]

VFS mapping (TODO)¶

XXX writeme

Downloads and VCS checkouts¶

Downloading files¶

batou supports downloading files via HTTP(S) or FTP, for example:

self += Download(
    'http://python.org/ftp/python/3.3.2/Python-3.3.2.tar.bz2',
    checksum='md5:7dffe775f3bea68a44f762a3490e5e28')

class batou.lib.download.Download(url)¶: Download from the given URL (uses urllib or requests internally).

requests_kwargs¶: Keyword arguments to pass to requests get method, e.g. to support authentication.

checksum¶: Checksum of the file to be verified (required). Must be given in the format algorithm:value, where algorithm must be a function of the hashlib stdlib module.

target¶: Filename to save the download as. [Default: last component of the URL]

Mercurial¶

self += Clone('https://bitbucket.org/gocept/batou', revision='tip')

class batou.lib.mercurial.Clone(url)¶: Clone a Mercurial repository from the given URL.

revision¶: Which revision to clone. At least one of revision or branch is required. If both are given, revision will be used.

branch¶: The name of a branch to clone. At least one of revision or branch is required. If both are given, branch will be overridden by revision. A clone of a named branch will be updated to the most recent upstream revision of the branch whenever batou is run.

target¶: Path to clone into (Default: workdir of parent component)

vcs_update¶: Whether to update the clone with incoming changesets (Default: True). Leaving clones of source code unchanged is often desirable during development.

Git¶

self += Clone('https://github.com/Pylons/pyramid', revision='HEAD')

class batou.lib.git.Clone(url)¶: Clone a Git repository from the given URL.

target¶: Path to clone into (Default: workdir of parent component)

update_unpinned¶: Update the clone on each batou run. If False, the repository is cloned once and then never updated again. [Default: False]

Note

git.Clone does not support specifying a revision yet.

Subversion¶

self += Checkout('https://svn.zope.org/repos/main/zopetoolkit/trunk', revision='130345')

class batou.lib.svn.Checkout(url)¶: Check out a Subversion repository from the given URL.

revision¶: Which revision to check out (required)

target¶: Path to clone into (Default: workdir of parent component)

Building software¶

batou has some support for downloading and compiling software packages, aka the configure-make-make install (CMMI) dance. Example usage:

self += Build(
    'http://python.org/ftp/python/3.3.2/Python-3.3.2.tar.bz2',
    checksum='md5:7dffe775f3bea68a44f762a3490e5e28',
    configure_args='--with-libs=-lssl')

class batou.lib.cmmi.Build(url)¶: Download archive from url, extract it and run CMMI on it.

checksum: Checksum for download (see batou.lib.download.Download.checksum for details)

prefix¶: Path to use as prefix for the installation (passed to configure --prefix) [Default: workdir of parent component]

configure_args¶: String of additional arguments to pass to configure.

build_environment¶: Dict of variables to add to the environment during all CMMI invocations.

Managing python installations¶

virtualenv¶

The basic building block for Python-based components is creation of virtualenvs (to separate package installations from each other):

self += VirtualEnv('2.7')

class batou.lib.python.VirtualEnv(version)¶: Creates a virtualenv for the given Python version in the working directory of the parent component. (Requires that pythonX.Y is in the PATH)

executable¶: Full path to the Python executable to create the virtualenv for (default: pythonX.Y based on the version attribute).

batou downloads a compatible version of virtualenv (depending on the Python version you need) to ensure everything works as expected and to avoid problems with incompatibilities or unexpected behaviours of whatever version might be installed already on the system. (virtualenv base installations are shared by all components for creating new virtualenvs, it is installed to work/.virtualenv).

Installing packages¶

Python packages are installed from a package index such as PyPI. batou uses pip or easy_install for this purpose (but that actually is an implementation detail and depends on the specifics of the Python and virtualenv version).

Packages must be added to a virtual environment.

venv = VirtualEnv('2.7')
self += venv
venv += Package('Sphinx', version='1.1.3')

class batou.lib.python.Package(package)¶: Install the Python package with the given name into the virtualenv of the parent component. Using Package requires that it is added to a VirtualEnv instance.

version¶: The version of the package to install (required).

install_options¶

List of options that are passed to pip/easy_install on the command line.

[Default: depends on the Python/virtualenv version in use]

check_package_is_module¶

Verify that the package is installed by trying to import it (more precisely, the first component of its dotted name). This is a stopgap against https://github.com/pypa/pip/issues/3, but should be pretty safe to disable if it causes trouble for specific packages (distribute is a notable example, since it installs a Python module named setuptools).

[Default: True]

timeout¶

A timeout (in seconds) that the installer should use to limit stalling network activity.

Only works when using pip.

[Default: equal to the environment’s timeout setting]

dependencies¶

Whether only the package itself or its dependencies should be installed.

[Default: True]

zc.buildout¶

batou has in-depth support for managing installations that use buildout. It automatically wraps them in a virtualenv, installs the appropriate buildout version, and takes care of running buildout whenever changes to configuration files makes it necessary. A typical usage example:

self += Buildout(python='3.7', version='2.2', setuptools='1.0',
                 pip='21.1',
                 additional_config=[Directory('profiles', source='profiles')])

class batou.lib.buildout.Buildout¶: Manage a buildout installation

python¶: Python version (required)

executable¶: Full path to the python executable to create the virtualenv for (used instead of pythonX.Y).

version¶: Version of zc.buildout to install (required)

setuptools¶: Version of setuptools to install into the virtualenv (must be appropriate to the buildout version, e.g. since 2.2 buildout requires setuptools, but some versions before that required distribute) (required)

pip¶: Version of pip to install into the virtualenv (must be appropriate to the buildout version).

distribute¶: Version of distribute to install into the virtualenv. Mutually exclusive with setuptools, of course.

config¶: If a different configuration file name than buildout.cfg should be used, pass in a File or Component instance.

additional_config¶: Optional list of component instances (e.g. File or Directory) that contain further configuration files (so Buildout knows when running buildout is needed).

Managing services¶

Aside from using batou’s general purpose functions for creating files and running commands we have a few ready-to-use abilities for higher level service management.

Supervisor (TODO)¶

Our built-in supervisor component allows you to run a supervisor process within your service user and has a simple API for declaring components that want to integrate with the supervisor config.

The supervisor itself will be integrated into the system’s startup automatically, depending on your platform.

SystemD¶

Note

SystemD is a non-core component provided through the batou_ext package.

Alternatively to using Supervisor you can register each program as a system-wide service managed by SystemD. You can also specify custom configuration in addition to (or overriding) the defaults:

from batou.component import Component
from batou.lib.file import File
from batou.lib.service import Service
import batou_ext.nix


class Tick(Component):

    def configure(self):
        self += File('tick.sh', mode="rwxr-xr-x")

        self += Service('tick.sh',
            systemd=dict(Type='simple',
                         Unit_After='cron.service memcached.service',
                         Service_RestartSec=11))

You should import the batou_ext.nix module to register the Platform specific Service component.

This will result in the following unit file:

/etc/local/systemd/tick.service¶

  [Service]
  Environment="LOCALE_ARCHIVE=/run/current-system/sw/lib/locale/locale-archive"
  Environment=PATH=/home/ctheune/bin:/var/setuid-wrappers:/home/ctheune/.nix-profile/bin:/home/ctheune/.nix-profile/sbin:/home/ctheune/.nix-profile/lib/kde4/libexec:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin:/nix/var/nix/profiles/default/lib/kde4/libexec:/run/current-system/sw/bin:/run/current-system/sw/sbin:/run/current-system/sw/lib/kde4/libexec
  Environment="TZDIR=/etc/zoneinfo"
  ExecStart=/home/ctheune/deployment/work/tick/foobar.sh start
  Group=service
  LimitNOFILE=64000
  LimitNPROC=64173
  LimitSIGPENDING=64173
  RestartSec=11
  Type=simple
  User=ctheune
  Restart=always

  [Unit]
  After=cron.service memcached.service

If you want to leverage SystemD’s ability to repeat a key in the configuration (like using multiple ExecStart statements) then you can simply pass that key as a list. This will be automatically expanded into multiple lines:

systemd=dict(Type='simple',
             ExecStart=['command1', 'command2'])

[Service]
...
ExecStart=/home/ctheune/deployment/work/tick/command1
ExecStart=/home/ctheune/deployment/work/tick/command2
...

Note

The SystemD support is currently geared towards the NixOS-based environment provided by us on our Flying Circus platform. We’re happy to extend and generalise this module upon request.

API¶

If you are looking for information on a specific function, class or method, this part of the documentation is for you.

Component Python API¶

This part of the documentation covers all the interfaces of batou you can use to develop your deployments.

Component¶

class batou.component.Component(namevar=None, **kw)¶

A component that models configuration and can apply it.

Use sub-classes of Component to create custom components.

The constructor takes one un-named argument which is assigned to the attribute set by the namevar class attribute.

The remaining keyword arguments are set as object attributes.

If a component is used as a sub-component (via +=), then the constructor arguments sets the object attributes. If a component is used directly from an environment (becoming a root component) then the constructor is called without arguments and overrides from the environment and the secrets are set through an internal mechanism.

namevar = None¶

The namevar attribute specifies the attribute name of the first unnamed argument passed to the constructor.

This helps making components more readable by providing one “natural” first argument:

class File(Component):

    namevar = 'filename'

    def configure(self):
       assert self.filename

class Something(Component):

    def configure(self):
        self += File('nginx.conf')

workdir = None¶

(readonly) The workdir attribute is set by batou when a component before a component is configured and defaults to <root>/work/<componentname>. Built-in components treat all relative destination paths as relative to the work directory.

During verify() and apply() batou automatically switches the current working directory to this.

property defdir¶

(readonly) The definition directory: (where the component.py lives).

Built-in components treat all path names of source files as relative to the definition directory.

property host¶: (readonly) The Host object this component is configured for.

property environment¶: (readonly) The Environment object this component is configured for.

property root¶: (readonly) The RootComponent object this component is configured for.

configure()¶

Configure the component by computing target state and declaring sub-components.

This is the “declarative” part of batou – consider a rather functional approach to implementing this.

Perform as much preparatory computation of target state as possible so that batou can perform as many checks as possible before starting to modify target systems.

Sub-components are added to this component by using the += syntax:

class MyComponent(Component):

    def configure(self):
        self += File('asdf')
        self += File('bsdf')
        self += File('csdf')

The order that sub-components will be worked on is given by the order of the += assignments.

Warning

configure must not change state on the target systems and should only interact with the outside system in certain situations. It is not guaranteed whether this method will be called on the host running the master command, or on any number of the target systems.

configure can be called by batou multiple times (with re-initialized attributes) for batou to automatically discover correct order.

Consequently, it is wise to keep computational overhead low to ensure fast deployments.

Warning

Using functions from :py:module:random can cause your configuration to be come non-convergent and thus cause unnecessary, repeated updates. If you like to use a random number, make sure you seed the random number generator with a predictable value, that may be stored in the environment overrides or secrets.

verify()¶

Verify whether this component has been deployed correctly or needs to be updated.

Raise the batou.UpdateNeeded exception if the desired target state is not reached. Use the assert_*() methods (see below) to check for typical conditions and raise this exception comfortably.

This method is run exactly once on the target system when batou has entered the deployment phase. The working directory is automatically switched to the workdir.

update()¶

Update the deployment of this component.

update is called when verify has raised the :py:class:UpdateNeeded exception.

When implementing update you can assume that the target state has not been reached but you are not guaranteed to find a clean environment. You need to take appropriate action to move whatever state you find to the state you want.

We recommend two best practices to have your components be reliable, convergent, and fast:

Create a clean temporary state before applying new state. (But be careful if you manage stateful things like database directories or running processes.)
If update and verify become too complicated then split your component into smaller components that can implement the verify/update cycle in a simpler fashion.

verify and update should usually not be longer than a few lines of code.

last_updated()¶

When this component was last updated, given as a timestamp ( seconds since epoch in local time on the system).

You can implement this, optionally, to help other components that depend on this component to determine whether they should update themselves or not.

__enter__()¶

Enter the component’s context.

Components are context managers: the context is entered before calling verify and left after calling update (or after verify if no update was needed).

This can be used to perform potentially expensive or stateful setup and teardown actions, like mounting volumes.

See Python’s context manager documentation if you want to know more about this mechanism.

__exit__(type, value, tb)¶: Exit the component’s context.

log(message, *args)¶

Log a message to console during deployment.

The message is %-substituted with *args, if it is put out, and prefixed with the hostname automatically.

Use this message to add additional status to the deployment output, i.e. “Deploying Version X”.

Note

During configure() log messages are not put out immediately but only after the configure phase is done, because configure() is called multiple times. Only the logs of the last call are put out.

In verify() and update() messages are put out immediately.

__add__(component)¶

Add a new sub-component.

This will also automatically prepare the added component if it hasn’t been prepared yet. Could have been prepared if it was configured in the context of a different component.

__or__(component)¶

Prepare a component in the context of this component but do not add it to the sub components.

This allows executing ‘configure’ in the context of this component

provide(key, value)¶

Provide a resource.

Parameters

key (str) – They key under which the resource is provided.
value (object) –
The value of the resource.

Resource values can be of any type. Typically you can pass around component objects or individual configuration values, like network addresses, or similar.

require(key, host=None, strict=True, reverse=False, dirty=False)¶

Require a resource.

Parameters

key (str) – The key under which the resource was provided.
host (object) – The host object that the provided resource belongs to.
strict (bool) – If true, then it is an error if no resources were provided given the required key.
reverse (bool) – By default a component that requires another one also depends on the one that provides a resource. If reverse is set to True then this dependency is reversed and the component that provides a resource depends on the component requiring it.
dirty (bool) –
When a component requires a resource then it will normally be configured again when another component is configured later that changes the list of resources that were required.

Under very special circumstances it may be necessary to not get reconfigured when the required resource changes to break cycles in dependencies. Use with highest caution as this can cause your components to have incomplete configuration.

Returns

The matching list of resources that were provided.

Return type

list

Note

Calling require may cause an internal exception to be raised that you must not catch: batou uses this as a signal that this component’s configuration is incomplete and keeps track of the desired resource key. If another component later provides this resource then this component’s configure will be run again, causing require to complete successfully.

require_one(key, host=None, strict=True, reverse=False, dirty=False)¶

Require a resource, returning a scalar.

For the parameters, see require().

Returns: The matching resource that was provided.
Return type: object

This version returns a single value instead of a list. Also, if the number of potential results is not exactly one, then an error will be raised (which you should not catch). batou will notify you of this as being an inconsistent configuration.

assert_cmd(*args, **kw)¶

Assert that given command returns successfully, raise UpdateNeeded otherwise.

For details about the command arguments and what a successful execution means, see batou.component.Component.cmd().

assert_file_is_current(reference, requirements=[], **kw)¶

Assert that the file given by the reference pathname has been created or updated after the given list of requirement file names, raise UpdateNeeded otherwise.

Parameters

reference (str) – The file path you want to check for being current.
requirements (list) – The list of filenames you want to check against.
kw (dict) – Arguments that are passed through to last_update which can be used to use different time stamps than st_mtime. See batou.lib.file.File.last_updated() for possible values.

Returns

None, if reference is as new or newer as all requirements.

Raises

UpdateNeeded – if the reference file is older than any of the requirements.

assert_component_is_current(requirements=[], **kw)¶

Assert that this component has been updated more recently than the components specified in the requirements, raise UpdateNeeded otherwise.

Parameters

requirements (list) – The list of components you want to check against.
kw (dict) – Arguments that are passed through to each last_update call. The semantics depend on the components’ implementations.

Returns

None, if this component is as new or newer as all requirements.

Raises

UpdateNeeded – if this component is older than any of the requirements.

The age of a component is determined by calling last_updated on this and each requirement component.

assert_no_subcomponent_changes()¶

Assert that, during this run of batou, non of this components’ sub-components have required an update.

Returns: None, if none if this components’ sub-components have required an update during this run of batou.
Raises: UpdateNeeded – if any of this components’ sub-components have required an update during this run of batou.

Note

Using this change indicator can be unreliable if you fail to perform your update correctly. It is likely that when later resuming an aborted deployment this change won’t be triggered again.

assert_no_changes()¶

Assert that, during this run of batou, neither this component nor any of its sub-components have required an update.

Returns: None, if neither this component nor any of its sub-components have required an update during this run of batou.
Raises: UpdateNeeded – if this component or any of its sub-components have required an update during this run of batou.

Note

Using this change indicator can be unreliable if you fail to perform your update correctly. It is likely that when later resuming an aborted deployment this change won’t be triggered again.

cmd(cmd, silent=False, ignore_returncode=False, communicate=True, env=None, expand=True)¶

Perform a (shell) command.

Use this to interact with the target system during verify, update, __enter__, or __exit__.

Warning

Do not use this during configure.

Parameters

cmd (str) – The command you want to execute including all arguments. This will be parsed by the system shell, so be careful of quoting.
silent (bool) – whether output should be shown in the case of errors.
ignore_returncode (bool) – If true, do not raise an exception if the return code of the command indicates failure.
communicate (bool) – If True, call communicate() and wait for the process to finish, and process the return code. If False start the process and return the Popen object after starting the process. You are then responsible for communicating, processing, and terminating the process yourself.
expand (bool) – Treat the cmd as a template and process it through Jinja2 in the context of this component.
env (dict) – Extends environment variables with given ones.

Returns

(stdout, stderr) if communicate is True, otherwise the Popen process is returned.

Raises

CmdExecutionError – if return code indicated failure and ignore_returncode was not set.

map(path)¶

Perform a VFS mapping on the given path.

If the environment has VFS mapping configured, compute the new path based on the mapping.

Whenever you get a path name from the outside (i.e. environment overrides or from the constructor) or use absolute paths in your configuration, you should call map as early as possible during configure. If you are using :py:class:batou.component.Attribute for constructor arguments or overrides, then you can specify map on the attribute to avoid having to map this yourself.

You should rely on other components to do the same, so if you pass a path to another component’s constructor, you do not have to call map yourself.

touch(filename)¶

Built-in equivalent of the touch UNIX command.

Use during verify, update, __enter__, or __exit__, to interact with the target system.

Warning

Do not use during configure.

expand(string, component=None, **kw)¶

Expand the given string in the context of this component.

When computing configuration data, you can perform inline template expansions of strings. This is an alternative to Python’s built-in string templates, to keep your inline configuration in sync with the external file templating based on Jinja2.

Parameters

string (unicode) – The string you want to be expanded as a Jinja2 template.
component (batou.component.Component) – By default this self. To perform the template expansion in the context of another component you can pass it through this argument (or call the other component’s expand).
kw (dict) – Additional keyword arguments are passed into the template’s context as global names.

Returns

the expanded template.

Return type

unicode

template(filename, component=None)¶

Expand the given file in the context of this component.

Instead of using the File component to expand templates, you can expand a file and receive a unicode string (instead of directly rendering the file to a target location).

Parameters

filename (str) – The file you want to expand. The filename is not mapped by this function. Map the filename before calling template if needed.
component (batou.component.Component) – By default this self. To perform the template expansion in the context of another component you can pass it through this argument (or call the other component’s expand).
kw (dict) – Additional keyword arguments are passed into the template’s context as global names.

Returns

the expanded template.

Return type

unicode

chdir(path)¶

Change the working directory.

Use this to interact with the target system during verify, update, __enter__, or __exit__.

Warning

Do not use this during configure.

The given path can be absolute or relative to the current working directory. No mapping is performed.

This is a context mapper, so you can change the path temporarily and automatically switch back:

def update(self):
    with self.chdir('/tmp'):
        self.touch('asdf')

class batou.component.HookComponent(namevar=None, **kw)¶

A component that provides itself as a resource.

configure()¶

Configure the component by computing target state and declaring sub-components.

This is the “declarative” part of batou – consider a rather functional approach to implementing this.

Perform as much preparatory computation of target state as possible so that batou can perform as many checks as possible before starting to modify target systems.

Sub-components are added to this component by using the += syntax:

class MyComponent(Component):

    def configure(self):
        self += File('asdf')
        self += File('bsdf')
        self += File('csdf')

The order that sub-components will be worked on is given by the order of the += assignments.

Warning

configure must not change state on the target systems and should only interact with the outside system in certain situations. It is not guaranteed whether this method will be called on the host running the master command, or on any number of the target systems.

configure can be called by batou multiple times (with re-initialized attributes) for batou to automatically discover correct order.

Consequently, it is wise to keep computational overhead low to ensure fast deployments.

Warning

Using functions from :py:module:random can cause your configuration to be come non-convergent and thus cause unnecessary, repeated updates. If you like to use a random number, make sure you seed the random number generator with a predictable value, that may be stored in the environment overrides or secrets.

class batou.component.RootComponent(name, environment, host, features, ignore, factory, defdir, workdir, overrides=None)¶

Wrapper to manage top-level components assigned to hosts in an environment.

Root components have a name and determine the initial working directory of the sub-components.

batou.component.platform(name, component)¶: Class decorator to register a component class as a platform-component for the given platform and component.

batou.component.handle_event(event, scope)¶

Attribute (TODO)¶

class batou.component.Attribute(conversion=<class 'str'>, default=<object object>, expand=True, map=False)¶

An attribute descriptor is used to provide:

declare overrideability for components
provide type-conversion from overrides that are strings
provide a default.

Conversion can be given as a string to indicate a built-in conversion:

literal - interprets the string as a Python literal list - interpretes the string as a comma separated list

If conversion is a callable the callable will be used for the conversion, when the value is read from config file. On a setattr the conversion is not applied.

The obj is expected to be a Component so that ‘expand’ can be accessed.

Parameters

conversion (str, callable) – A conversion callable which takes one parameter or a string for built-in conversion (literal or list). This function is used for strings from config files.
default – The default value for the Attribute. When a

ConfigString value is passed then it will expanded, mapped, and passed through the conversion function, depending on the other arguments. :type default: None

Parameters

expand (bool) – Expand the config string in the context of this component.
map (bool) – Perform a VFS mapping on the config string.

from_config_string(obj, value)¶: Perform expansion, mapping and conversion after another.

Host (TODO)¶

class batou.host.Host(name, environment, config={})¶

Environment (TODO)¶

class batou.environment.Environment(name, timeout=None, platform=None, basedir='.', provision_rebuild=False)¶: An environment assigns components to hosts and provides environment-specific configuration for components.

Environment configuration¶

Component assignment (TODO)¶

General parameters (TODO)¶

General environment parameters are set in the [environment] config section. Example:

[environment]
service_user = website
host_domain = gocept.net
platform = gocept
branch = production

service_user

The deployment is run as this user on remote machines. If this is not the same as the user connecting via ssh, a sudo to the service user is performed.

host_domain

All hosts in the [hosts] section are postfixed with this domain. This is handy do make the host/component assignment less verbose

update_method

hg-bundle|hg-pull|git-bundle|git-pull|rsync, sets how the remote deployment repository is updated.

pull, the default, uses hg/git clone and/or hg/git pull on the remote site.
bundle will copy the necessary changes as Mercurial/Git bundle, via the batou ssh link.
rsync will rsync the working copy. This is most useful in combination with the vagrant platform.

branch

For remote deployments, use this and only this branch. batou will complain if the local branch does not match the set branch in the environment.

platform

Set the platform for this environment.

timeout

Set the ssh connection timeout in seconds.

target_directory

Absolute path of the directory on remote machines where the remote deployment repository is stored. Supports tilde expansion. Default: ~/deployment.

vfs mapping (TODO)¶

Root-component attribute overrides (TODO)¶

Jinja2 templates (TODO)¶

Utilities¶

class batou.utils.Address(connect_address, port=None, require_v4=<object object>, require_v6=<object object>)¶

An internet service address that can be listened and connected to.

The constructor address is expected to be the address that can be connected to. The listen address will be computed automatically.

>>> x = Address('localhost', 80)
>>> str(x.connect)
'localhost:80'
>>> str(x.listen)
'127.0.0.1:80'

You can specify which IP versions are expected to be resolved for listen addresses in three ways with the require_v4/require_v6 flags:

False -> this version must not be used
True -> this version must be resolved properly
‘optional’ -> the listen/listen_v6 attribute will contain None if it does
not resolve.

connect = None¶: The connect address as it should be used when configuring clients. This is a batou.utils.NetLoc object.

property listen¶: The IPv4 listen (or bind) address as it should be used when configuring servers. This is a batou.utils.NetLoc object. It raises an batou.IPAddressConfigurationError if used unconfigured.

property listen_v6¶: The IPv6 listen (or bind) address as it should be used when configuring servers. This is a batou.utils.NetLoc object. It raises an batou.IPAddressConfigurationError if used unconfigured.

class batou.utils.BagOfAttributes¶

Provide a dict-like object that can also be accessed using attributes.

It’s sometimes more convenient to write a.x instead of a[‘x’]. However, namespaces may require being able to also use non-Python-identifier keys.

exception batou.utils.CmdExecutionError(cmd, returncode, stdout, stderr)¶

exception batou.utils.CycleError¶

class batou.utils.NetLoc(host, port=None)¶

A network location specified by host and port.

Network locations can automatically render an appropriate string representation:

>>> x = NetLoc('127.0.0.1')
>>> x.host
'127.0.0.1'
>>> x.port
None
>>> str(x)
'127.0.0.1'

>>> y = NetLoc('127.0.0.1', 80)
>>> str(y)
'127.0.0.1:80'

host = None¶: The host part of this network location. Can be a hostname or IP address.

port = None¶: The port of this network location. Can be None or an integer.

batou.utils.call_with_optional_args(func, **kw)¶: Provide a way to perform backwards-compatible call, passing only arguments that the function actually expects.

batou.utils.dict_merge(a, b)¶: recursively merges dict’s. not just simple a[‘key’] = b[‘key’], if both a and b have a key who’s value is a dict then dict_merge is called on both values and the result stored in the returned dictionary. https://www.xormedia.com/recursively-merge-dictionaries-in-python/

Exceptions¶

exception batou.ComponentLoadingError¶: The specified component file failed to load.

exception batou.ConfigurationError¶: Indicates that an environment could not be configured successfully.

exception batou.ConversionError¶: An override attribute could not be converted properly.

exception batou.CycleErrorDetected¶: We think we found a cycle in the component dependencies.

exception batou.DeploymentError¶: Indicates that a deployment failed..

exception batou.DuplicateComponent¶

exception batou.DuplicateHostError¶

exception batou.DuplicateHostMapping¶

exception batou.DuplicateOverride¶: An override for a component attribute was found both in the secrets and in the environment configuration.

exception batou.FileLockedError¶: A file is already locked and we do not want to block.

exception batou.GPGCallError¶: There was an error calling GPG on encrypted file.

exception batou.IPAddressConfigurationError¶: An IP address family was accessed but not configured.

exception batou.InvalidIPAddressError¶

exception batou.MissingComponent¶: The specified environment does not exist.

exception batou.MissingEnvironment¶: The specified environment does not exist.

exception batou.MissingOverrideAttributes¶

exception batou.NonConvergingWorkingSet¶: A working set did not converge.

exception batou.ReportingException¶

Exceptions that support user-readable reporting.

classmethod merge(selfs)¶: Merge multiple instances of this exception.

should_merge(other)¶: checks, wether two exceptions have the same type as well as data and as such, should be merged into one exception.

exception batou.RepositoryDifferentError¶: The repository on the remote side is different.

exception batou.SilentConfigurationError¶

These are exceptions that will be reported by other exceptions.

They basically only influence control flow during configuration and are manually placed to avoid double reporting.

exception batou.SuperfluousComponentSection¶: A component section was found in the environment but no associated component is known.

exception batou.SuperfluousSecretsSection¶: A component section was found in the secrets but no associated component is known.

exception batou.SuperfluousSection¶: A superfluous section was found in the environment configuration file.

exception batou.UnknownComponentConfigurationError¶: An unknown error occured while configuring a component.

exception batou.UnsatisfiedResources¶: Some required resources were never provided.

exception batou.UnusedResources¶: Some provided resources were never used.

exception batou.UpdateNeeded¶: A component requires an update.

Contributor guide¶

If you want to contribute to the project, this part of the documentation is for you.

Contributor’s guide¶

If you’re reading this you’re probably interested in contributing to batou. First, we’d like to say: thank you! Open source projects live-and-die based on the support they receive from others, and the fact that you’re even considering supporting batou is very generous of you.

This document lays out guidelines and advice for contributing to batou. If you’re thinking of contributing, start by reading this thoroughly and getting a feel for how contributing to the project works. If you have any questions, feel free to reach out to Christian Theune, the primary maintainer.

The guide is split into sections based on the type of contribution you’re thinking of making, with a section that covers general guidelines for all contributors.

All Contributions¶

Be Cordial¶

Be cordial or be on your way.

batou has adopted this very important rule from the Requests library. This rule governs all forms of contribution, including reporting bugs or requesting features. This golden rule is be cordial or be on your way. All contributions are welcome, as long as everyone involved is treated with respect.

Get Early Feedback¶

If you are contributing, do not feel the need to sit on your contribution until it is perfectly polished and complete. It helps everyone involved for you to seek feedback as early as you possibly can. Submitting an early, unfinished version of your contribution for feedback in no way prejudices your chances of getting that contribution accepted, and can save you from putting a lot of work into a contribution that is not suitable for the project.

Contribution Suitability¶

The project maintainer has the last word on whether or not a contribution is suitable for batou. All contributions will be considered, but from time to time contributions will be rejected because they do not suit the project.

If your contribution is rejected, don’t despair! So long as you followed these guidelines, you’ll have a much better chance of getting your next contribution accepted.

Code Contributions¶

When contributing code, you’ll want to follow this checklist:

Fork the repository on GitHub.
Run the tests to confirm they all pass on your system. If they don’t, you’ll need to investigate why they fail. If you’re unable to diagnose this yourself, raise it as a bug report by following the guidelines in this document.
Write tests that demonstrate your bug or feature. Ensure that they fail. Make your change.
Run the entire test suite again, confirming that all tests pass including the ones you just added.
Send a Pull Request to the main repository’s master branch. Pull Requests are the expected method of code collaboration on this project.

The following sub-sections go into more detail on some of the points above.

Code Review¶

Contributions will not be merged until they’ve been code reviewed. You should implement any code review feedback unless you strongly object to it. In the event that you object to the code review feedback, you should make your case clearly and calmly. If, after doing so, the feedback is judged to still apply, you must either apply the feedback or withdraw your contribution.

New Contributors¶

If you are new or relatively new to Open Source, welcome! batou aims to be a gentle introduction to the world of Open Source. If you’re concerned about how best to contribute, please consider mailing a maintainer (listed above) and asking for help.

Please also check the “Get Early Feedback” section.

Documentation Contributions¶

Documentation improvements are always welcome! The documentation files live in the doc/ directory of the codebase. They’re written in reStructuredText, and use Sphinx to generate the full suite of documentation.

When contributing documentation, please attempt to follow the style of the documentation files. This means a soft-limit of 79 characters wide in your text files and a semi-formal prose style.

Bug Reports¶

Bug reports are hugely important! Before you raise one, though, please check through the bugtracker issues, both open and closed, to confirm that the bug hasn’t been reported before. Duplicate bug reports are a huge drain on the time of other contributors, and should be avoided as much as possible.

When reporting a bug, make sure to include the batou version and platform identifier, e.g.:

$ ./batou
batou/2.3b2.dev0 (cpython 3.9.5-final0, Darwin 20.5.0 x86_64)
...

Also, include console output, relevant component code and maybe environment configuration if reporting errors.

Feature Requests¶

batou is under development. We have a strong idea about our architecture, though.

If you believe there is a feature missing, feel free to raise a feature request, but please demonstrate the issue you want to solve instead of only suggesting a certain function or feature. We want batou’s architecture to remain as small and clean as possible and thus we’re heavily interested in understanding the problem you are trying to solve.

Code of Conduct¶

Our Pledge¶

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.

Our Standards¶

Examples of behavior that contributes to creating a positive environment include:

Using welcoming and inclusive language
Being respectful of differing viewpoints and experiences
Gracefully accepting constructive criticism
Focusing on what is best for the community
Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

The use of sexualized language or imagery and unwelcome sexual attention or advances
Trolling, insulting/derogatory comments, and personal or political attacks
Public or private harassment
Publishing others’ private information, such as a physical or electronic address, without explicit permission
Other conduct which could reasonably be considered inappropriate in a professional setting

Our Responsibilities¶

Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.

Scope¶

This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.

Enforcement¶

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at mail@flyingcircus.io. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership.

Attribution¶

This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at http://contributor-covenant.org/version/1/4

Testing¶

Run the tests¶

bin/tox

Changing the examples¶

The batou configurations in the examples/ directory use a custom GPG key. To change the secrets files or to run the examples you have to set the environment variable GNUPGHOME to point to ./src/batou/secrets/tests/fixture/gnupg. (But the path has to be absolute!)

Example:

$ cd examples/errors
$ GNUPGHOME=/home/user1/clones/batou/src/batou/secrets/tests/fixture/gnupg ./batou deploy errors

Adding new migration steps¶

Migration steps are Python modules inside src/batou/migrate/migrations.
The name of a new migration step must be an integer number (plus .py extension) bigger than any existing one. The already existing ones suggest that the first digit corresponds to batou’s major version and the second one to the minor version, so migration steps for batou 2.3 should be named 23xx.py. Not each possible number has to be used.
The highest migration version already applied to a batou project is stored in .batou.json.
./batou migrate runs all migration with a higher version number than the one stored in .batou.json steps in ascending order.
Each migration step has to provide a migrate function which takes one positional argument.
The migration steps’ migrate function is called with a callable as argument which takes two arguments: title and text. The values there given are rendered in the output to inform the user about changes done by the migration step or changes which have to be applied manually. This function can be called multiple times per migration step.

How to Help¶

batou is under active development, and contributions are more than welcome!

Check for open issues or open a fresh issue to start a discussion around a bug.
Fork the repository on Bitbucket and start making your changes to a new branch.
Write a test which shows that the bug or feature works fine.
Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to AUTHORS.

Ideas¶

A continuous deployment server would be nice. I started experimenting with a new project that would be called “Aramaki”. Contact @theuni if you are interested.
Documentation improvements are always welcome. Especially if you’re missing something or would like to understand things. We’ll be happy to explain things in depth – a good deed would then be to improve our documentation.
Switching from the Component base class to an explicit API to avoid namespace collisions would be nice. The idea would be to create components using a class decorator:
```
from batou import component

@component
class MyApp(object):

    def configure(self, b):
        self += File(...)

    def verify(self, b):
        b.assert_no_subcomponent_changes('touch asdf')

    def verify(self, b):
        b.cmd('rm -rf /')
```
b is an API object bound to the component that provides the standard API. This way, you can use any names on your environment without the hassle of potential namespace collisions.
More pre-defined, reusable components are welcome. Check the batou_ext repository.
Supporting provisioning for platforms aside from Vagrant: e.g. Amazon, Flying Circus, …

Development Dependencies¶

We use zc.buildout to manage the batou build environment:

$ virtualenv --python=python2.7
$ bin/pip install zc.buildout
$ bin/buildout

Run py.test to verify that everything works fine:

$ bin/py.test
================= test session starts ==================
platform darwin -- Python 2.7.10 -- py-1.4.26 -- pytest-2.6.4
plugins: cache, capturelog, codecheckers, cov, timeout
collected 426 items

src/batou/__init__.py ..
src/batou/_output.py ..
src/batou/agent.py ..
src/batou/bootstrap.py ..
src/batou/buildout.py ..
src/batou/c.py ..
src/batou/component.py ..
...

Runtime Environments¶

We will likely not accept patches that are related to Windows. We are open to good arguments, though.

Downstream Repackaging¶

batou is not intended for downstream repackaging.

Authors¶

batou is written and maintained by Christian Theune and various contributors:

The Flying Circus Team¶

Christian Theune (ct@flyingcircus.io)
Christian Zagrodnick (cz@flyingcircus.io)
Christian Kauhaus (kc@flyingcircus.io)
Alexander Bittner (ab@flyingcircus.io)
Frank Lanitz (fl@flyingcircus.io)

Patches and Suggestions¶

Michael Howitz, gocept gmbh & co. kg
Thomas Lotze, gocept gmbh & co. kg
Steffen Allner, gocept gmbh & co. kg
Sebastian Wehrmann, gocept gmbh & co. kg
Wolfgang “wosc” Schnerring
Gil Forcada Codinachs
Stefan Walluhn
Cillian de Roiste
Florian Schulze
Hervé Coatanhay
Jan-Jaap Driessen
Marius Gedminas

Support¶

batou itself is released “as is”. We hang around #batou in the Freenode IRC network. You can also report bugs to our bugtracker.

Commercial support

We will be happy to give you commercial support for batou: feature implementation, bug fixing, consulting, or education.

To get in touch, send an email to mail@flyingcircus.io and we’ll be happy to discuss your requirements.

Resources¶

Changelog¶

2.3.1 (2023-05-22)¶

Error when accessing Attribute without an override or default now show the name of the attribute being accessed in the error message (#337)
a formatting bug in the error message for batou.lib.file.File has been fixed (#348)

2.3 (2023-01-24)¶

No further changes.

2.3b6 (2022-12-09)¶

Features¶

Allow mysql authentication via sudo.

Bugfixes¶

Secrets files created without proper prefix #333

2.3b5 (2022-09-21)¶

Features¶

Do not clobber git clones by default. Add new option clobber to allow specifying that a git clone may clobber the target directory. (#298)
Add checksum(value) method to Component to easily compute sha256 checksums. (#232)

Improvements¶

Clean up migrations and make the output easier to read.
Output usage strings for subcommands, instead of just the main command.
Revise the new Address require_v4/require_v6 to allow environment-based customization that can be used to revert back to the old settings. (#270)
Improve the error output by suppressing certain errors and grouping them properly to indicate which hosts see which errors in the model. (#272)

Bug fixes¶

Check python venvs for a functional pip and rebuild if necessary.
Fix use of File for syncing directories. (#270)
Fix incorrect handling of of missing Attribute defaults. (#318)

2.3b4 (2022-08-22)¶

Features¶

Add a migration framework. Automatic migrations can now be called using batou migrate. (#185)
Allow to pin the pip version used in the Buildout component. (#263)
Automatically migrate environments and secrets to the new structure using ./batou migrate. (#185)
Support creating users in MySQL ≥8. (#242)
Allow to check if an Address is configured for IPv4 resp. IPv6 using the attributes require_v4 resp. require_v6.
The behaviour to specify Attribute defaults as either Python native objects or strings that will be expanded, mapped and converted has been changed again to simplify the mechanics and make migration errors easier to understand.

2.3b3 (2021-11-30)¶

Action needed¶

Require defaults to be explicitly declared for Attribute. (#237)

Bug fixes¶

Ignore hostmap entries for hosts that have changed their dynamic hostname settings to false.

Features¶

Automatically pick up provision.sh and/or provision.nix.

You do not need to explicitly define a COPY command to copy the provision.nix to the container, but if you do then we avoid doing it twice.
Warn if neither provision.nix nor provision.sh are given as that seems more of an accident (like misspelling the filenames).
Continue deployments on failure when running fc-manage during provisioning but be more explicit about errors and warn the user that something maybe be fishy in the deployment subsequently.
Use different colors for success depending on whether you ran a real deployment, a consistency check, or a predition. (https://github.com/flyingcircusio/batou/issues/209)

2.3b2 (2021-10-05)¶

Action needed¶

Fail if an attribute is set both in environment and via secrets. (#28)
Avoid implicit conversion of Attribute defaults. In cases where the default value should be converted, use default_conf_string. This may result in some changes if your code relied on this implicit conversion. If you use batou_ext, update to a current commit.. (#89)
Raise an error if an internet protocol family is used but not configured. (#189)

Bug fixes¶

Fix Python 3 compatibility with some Mercurial-based batou repositories.
Adapt bootstrap.sh to the use of appenv.

Features¶

Integrate remote-pdb to debug batou runs. (#199)
NetLoc objects are now comparable. (#88)
Enhance file Mode objects to accept integers, octal mode strings and ‘rwx’ strings as the mode argument. This allows homogenous use in Python code and overrides through config files. (#61)
Do not render diffs for files which contain contents of secrets/*. (#91)
Assure that requirements.lock is build with the oldest supported Python version to keep it consistent – newer Python versions have included some packages in standard library which older ones need as dependencies. (#145)
Remove default option for installation via pip. (#212)
Implement dynamic, pluggable provisioning of hosts.

We provide a built-in plugin to support NixOS development containers that feel similar to the Flying Circus VM platform.

Other changes¶

Improve error message for DNS lookup semantics.
Render a better error message if gpg failed to decrypt a secrets file. (#123)
Raise exception when calling batou secrets add or batou secrets remove with an unknown environment name. (#143)
Render an error message if batou secrets summary fails during decryption. (#165)
Do not write secrets files without recipient. (#184)

2.3b1 (2021-05-21)¶

Drop support for Python 3.5. (#114)
batou.lib.buildout: Enable support for Buildout 3 by allowing to specify a wheel package version to install in the virtualenv. (#148)
Fix bootstrapping projects with the new appenv structure. Vendor an appenv version to ensure lockstep compatibility.
Fix consistency check semantics: we accidentally performed actual deployments during consistency checks.
Fix rsync repository mode to capture deleted top-level elements in the source.
Improve DNS lookup semantics.

We experienced two major problems with the current code:
1. IPv6 lookups were done opportunistically and thus if DNS issues happened during deployments we would suddenly drop IPv6 support instead of failing.
2. There was no logging to find out why the code was making specific decisions and to see what the underlying network APIs were returning. We now provide detailed debug logs for analyzing DNS issues.
There were slight adjustments in the internal API (resolve/resolve_v6) that should be backwards compatible.

The public API reflects a more strict stance now:
- by default we only look up IPv4
- you can explicitly set the require_v6 and require_v4 options for Address objects. batou will then perform the required lookups (or not) and it will be a hard failure if required lookups can not be performed.
We recommend to adjust those parameters on Address objects depending on your environment, e.g. if you want IPv6 in production but not in Vagrant.

2.2.4 (2021-02-11)¶

Repair File(group=), it now works just like File(owner=)
Remove debugging code from secrets editing which caused encryption errors to crash and loose unsaved edits. (#139)
Fix shipping of deployment code with git-bundle when using a branch. Before the entire branch history was uploaded with each deployment to each host (#131)
Allow specifying a custom pip version in AppEnv.

2.2.3 (2021-01-20)¶

Fix #124: notifications crashed when trying to display environment names but used environment objects.

2.2.2 (2020-12-14)¶

Another brownbag release - connecting to remote hosts was broken after refactoring due to missing test coverage. Fixed and added coverage.

2.2.1 (2020-12-14)¶

Fix error reporting that was partially broken in 2.2.

2.2 (2020-12-10)¶

Add secret files in addition to secret overrides. Using ./batou secrets edit {environment} {example.yaml} you can now create multiple files that are all encrypted using the environment’s keys.

To access those secrets you can read them from environment.secret_files['{example.yaml}'] in your deployment.

This feature is useful to embed longer data or formats that are hard to embed syntactically into the cfg/ini style of the main secrets file.
Fix bug preventing to use nagios=True in Supervisor components, introduced in batou 2.1. (#98)
Make batou compatible with Python 3.9, ie asyncio’s all_tasks has been moved to a new location. (#93)
Actually silence SilentConfiguration errors.
Consider unknown secret overrides (components and attributes) to be a configuration error.

2.1 (2020-09-09)¶

Bug 81: provide explicit support for JSON- and YAML-files with proper integration to the new diff support and the ability to update data through a “dict merge” approach.
Bug 77: use ConfigUpdater so comments are kept when editing secrets.
Bug 1: provide better error message if remote user does not exist.

This is also cleaning up the general error output and we’re now hiding full tracebacks unless batou is run with –debug. People keep complaining about traceback output and I agree that it makes things harder to read for someone not used to scanning through them quickly.
Bug 63: provide better error message if GPG is missing.
Bug 65: don’t allow passing undefined namevars or undefined attributes to the component constructor. Also prohibit (accidentally) overriding methods.
Bug: zsh compatibility on the remote host was broken with more complex sudo mechanism. Added a ZSH workaround.

2.0 (2020-07-15)¶

Ignore directories when verifying archive extractions to avoid false non-convergence.

2.0b14 (2020-06-25)¶

Make sudo properly conditional if we log in directly with the service user, but avoid adding a re-connect performance penalty.

2.0b13 (2020-06-25)¶

Fix git clone when cloning into the component work directory. #27
Fix binary file handling that broke during 2to3 migration and the test was doing the wrong thing.
Allow marking file content as sensitive, which - for now - will suppress diff generation/logging.
Allow specifying the service_user attribute per host.
Bugfixes for file components so that verify() is more robust in predictive runs.
Add argument ‘predicting’ to the verify() function signature. This argument can be accepted optionally (so we’re backwards compatible) and will indicate that we’re doing a predictive run so we can avoid failing when trying to rely on output from earlier components.
Allow the Content component to predict a change based on a not-yet-realized source file on the target system.
Limit parallel connection setup to 5 connections at once. Also, retry up to 3 times per connection and stagger retries according to a CSMA/CD schema. This helps make connection setup more reliable if using SSH jump hosts where many connections may cause sshd’s MaxStart to start rejecting new connections. (#55)
Allow adding data-* overrides to host sections in environments’ secrets files.
Reduce AppEnv component directory hashes to 8 byte to avoid the shebang (#!) 127 character path limit.
Improve verify() of archive handler so we predict a change if something goes wrong (like not having the archive downloaded yet)
Fix “is supervisor program running” check if it is stopped or exited

2.0b12 (2020-05-13)¶

Fix broken sort of configuration errors. (#52)

2.0b11 (2020-05-13)¶

Fix “is supervisord running” check in the Supervisor(enable=False) case

2.0b10 (2020-05-11)¶

Fix Python 3 compatibility for Debian logrotate component.
Improve output ordering and formatting. The diffs for predicted (or applied) changes now appear in proper order.
Provide better error messages when batou fails to lock a secret file.

2.0b9 (2020-05-09)¶

Refactor the appenv component into smaller components (and move it to batou.lib.appenv.
Always update pip when installing an appenv - this also fixes the Travis tests.

2.0b8 (2020-05-08)¶

Replace ‘Deploying ’ with ‘Scheduling’ as this is only the moment where we decide that a component is not blocked by another any longer and can done as soon as the worker pool is able to do it. Specifically this means that the following output isn’t necessarily from this component.
Mark the hostname for each deployed component in the breadcrumb output so that asynchronously deployed components can be visually identified correctly.
Show diff of file changes - both during predict and deployment - to better estimate whether template changes are as expected.

2.0b7 (2020-05-07)¶

Update embedded appenv to support Python 3.4+.
Add component AppEnv to manage virtualenvs similar to the appenv package superseding previous virtualenv and buildout components.
Allow using assert statements instead of batou.UpdateNeeded.
Ensure the working directory is the defdir during the configure phase to allow using relative path names.

2.0b6 (2020-04-24)¶

Various smaller fixes to get the remoting code working again.
Update supervisor to 4.1 and Python 3.
Allow specifying major versions for virtualenvs (i.e. ‘2’ and ‘3’) to get convergence for virtualenvs where we don’t control the minor version of the targets.
Add ability to disable supervisor programs.
Remove ‘–fast’ and ‘–reset’ mode as this isn’t needed/ supported by appenv at the moment.
Simplify SSH/sudo and try sudo first. Probably needs further attention once we’re along the release cycle.
Fix Python 2.7 virtualenvs - upgrade to latest old-style release of virtualenv.

2.0b5 (2020-04-15)¶

Switch to new bootstrapping/self-installing pattern “appenv”. See appenv.py for a work-in-progress schema.

2.0b4 (2020-01-13)¶

Fix incorrect long->int conversion from 2to3. (Mercurial support)
Fix brownbag release due to missing support files.

2.0b3 (2020-01-10)¶

Fix Python3 compatibility for logrotate and crontab
Add six as dependency, so deployments can be made compatible to both Python 2 and 3 more easy.

2.0b2 (2019-10-15)¶

General¶

Make encryption/decryption error prompt more readable.
Fix coding style issues.

Features¶

Add coarse grained parallelization of deployments.

Using asyncio we now schedule rooot components for deployment as soon as all their dependencies are completed. However, due to execnet being synchronous this only has a visible effect for components on multiple hosts. This will speed up long running multi-host deployments.

In the future this will become more fine-grained so that we can even deploy multiple independent components on the same host in parallel.

Bugs¶

Fix issue #5: allow using % in secrets similar to regular overrides.
Fix secrets editing, add test coverage for the affected code.

Testing¶

Relax global pytest timeouts to avoid slow network failures.
Switch from broken/abandoned pytest-codecheckers to pytest-flake8
Enable flake8, remove unused pytest markers.
Fix warnings
Make warnings errors.

Build system / Development environment¶

Update example batou scripts
Fix travis build environments. Support Python 3.5, 3.6, 3.7, 3.8-dev and nightly (also 3.8 currently).
Generalize development bootstrapping code for dual use in local and travis environments.

2.0b1 (2019-10-11)¶

Drop support for Python 2.
Move to Python 3.5+.

A smooth migration mechanism may become available in the future based on users’ needs.
The default hash function is now ‘sha512’, existing deployments need to be migrated manually.

Upgrading¶

Generally batou tries to be very friendly when upgrading from version to version so you don’t have to unnecessarily change your deployments. We try to introduce new features without breaking existing semantics.

Upgrading to the separated appenv bootstrap file¶

appenv is a small, separate utility that batou uses to bootstrap and update itself, introduced in batou 2.

Initially we placed this file directly as ./batou but this turned out to be hard to deal with due to its “meta argument parser”. New versions of batou will automatically bootstrap this into two separate files: ./appenv and a symlink batou -> appenv.

To upgrade your project to this schema, follow those steps:

$ curl -sL https://raw.githubusercontent.com/flyingcircusio/batou/main/appenv.py -o appenv
$ chmod +x appenv
$ rm batou
$ ln -sf appenv batou

You can now use the appenv meta commands by calling the appenv utility itself:

$ ./appenv --help
usage: appenv [-h] {update-lockfile,init,reset,python,run} ...

positional arguments:
  {update-lockfile,init,reset,python,run}
    update-lockfile     Update the lock file.
    init                Create a new appenv project.
    reset               Reset the environment.
    python              Spawn the embedded Python interpreter REPL
    run                 Run a script from the bin/ directory of the virtual env.

optional arguments:
  -h, --help            show this help message and exit

batou 1.x to batou 2.x¶

This upgrade mainly consists of a switch from Python 2 to Python 3 but it also does include a few details about how things are handled.

Major changes¶

batou requires at least Python 3.5, anything newer is fine, too.
File changes now show (expected) diffs.
You can deploy to multiple hosts at once using the -j option or the jobs option in the environment configuration.
Secrets can now also override data- sections for hosts.
You can now use assert instead of raise UpdateNeeded in the verify methods.
The embedded supervisor has been updated.

Updating your deployment¶

The following steps assume that you have cloned and checked out your deployment and it is your current working directory.

$ curl -sL https://raw.githubusercontent.com/flyingcircusio/batou/main/bootstrap | sh

The generated lockfile requirements.lock might not pick up all versions of all packages correctly (for example packages taken directly from source control will fail) so you might want to doublecheck those and massage the lockfile in case that this fails with something like:

ERROR: No matching distribution found for batou-ext==0.1.dev0 (from -r .batou/16f85f2d/requirements.lock (line 3))

Note

batou_ext will be updated automatically to a proper Python 3 version and syntax.

At this point you can now update your projects’ component code to Python 3:

$ 2to3 -w components

After this you need to run batou in your development environment to see whether any further changes may be needed.

Things you may have to change in your deployment¶

Add sensitive_data=True to files that may contain sensitive data and whose content you do not want to see in any logs. This suppresses diff generation.
The default hash function has been changed to sha512 and thus existing hashes relying on the default being md5 will fail.

Updating to a newer batou 2.x version¶

After having switched from batou 1.x to 2.x you might update to the latest batou release once in a while. This requires the following steps:

Edit requirements.txt and enter the desired version: batou==2.x.
- If you want to use an unreleased version use a git URL like this: -e git+https://github.com/flyingcircusio/batou.git#egg=batou instead of the batou==2.x requirement.
Run ./appenv update-lockfile to update requirements.lock.
Commit the changes and run batou so it can update itself to the new version.

Updating from 2.2 to 2.3¶

To define Attributes more explicitly the default value has to be passed in one of two ways.

default allows to pass a final python value or use a ConfigString value to trigger the conversion of the Attribute on a default value. This can be used for more concise code (e.g. Address) or for documentation of the format for environment.cfg. This way expand and map of the string will also be done.

Presentations from conferences: