Advanced Usage

Writing a custom component (TODO)

Debugging batou runs

Using a debugger

batou comes with remote-pdb pre-installed. When running on Python 3.7+ 1 you can use breakpoint() to drop into the debugger. You need telnet or netcat to connect to the hostname and port displayed.

If you are using the default configuration, call:

$ nc -C 127.0.0.1 4444
or
$ telnet 127.0.0.1 4444

If you are debugging a remote deployment you should create a port forward beforehand like this:

$ ssh -L 4444:localhost:4444 my.remote.host.dev

You are able to configure hostname and port. For details see the documentation of remote-pdb. This works both for local and remote deployments. The environment variables for host and port are propagated to the remote host.

Example shell session for debugging using a custom port:

$ REMOTE_PDB_PORT=4445 ./batou deploy dev
batou/2.3b2.dev0 (cpython 3.6.15-final0, Darwin 20.6.0 x86_64)
============================= Preparing ============================
main: Loading environment `dev`...
main: Verifying repository ...
main: Loading secrets ...
============================= Connecting ... =======================
localhost: Connecting via local (1/1)
============================= Configuring model ... ================
RemotePdb session open at 127.0.0.1:4445, waiting for connection ...

Example for a second terminal where the opened port gets connected to:

$ nc -C 127.0.0.1 4445
> /.../components/supervisor/component.py(32)configure()
-> if self.host.platform in ('gocept.net', 'fcio.net'):
(Pdb)

Using “print debugging”

  • Inside a component you can use self.log("output") to print to the console.

  • You can call batou with the -d flag to enable debugging output during the deployment run.

Using 3rd party libraries within batou

Sometimes, when writing custom components, you may need additional Python packages, for example to configure databases by connecting directly to their SQL interface instead of using their command line clients.

You can use additional Python packages by adding a requirements.txt file to your batou project repository:

$ tree
.
├── batou
├── components
│   └── myapp
│       └── component.py
├── environments
│   └── local.cfg
└── requirements.txt
requirements.txt
sqlalchemy

The next time when you call batou the dependencies will be automatically updated. When deploying then the requirements will also be installed on the remote hosts.

$ ./batou
Installing sqlalchemy
usage: batou [-h] [-d] {deploy,secrets,init,update} ...

Note

batou already provides a number of packages that it depends on. If you create contradicting requirements then this may lead to batou failing. You will see pip complaining in that case.

Multiple components in a single component.py (TODO)

Skipping individual hosts or components when deploying (TODO)

Events (TODO)

Using bundle transfers if the repository server is not reachable from your remote server (TODO)

Timeout (TODO)

VFS mapping for development (TODO)

VFS mapping with explicit rewrite rules (TODO)

Extended service discovery options (TODO)

Platform-specific components

New in version 1.4.

Platform-specific components allow to customize behavior depending on the system or “platform” the target system runs as. Examples:

  • Production system on Gentoo, local development on Ubuntu, or

  • All VMs on Ubuntu but Oracle is being run with RedHat.

To define a platform specific aspects, you use the platform class decorator. Example:

import batou.component
import batou.lib.file


class Test(batou.component.Component):

    def configure(self):
        self += batou.lib.file.File('base-component')


@batou.component.platform('nixos', Test)
class TestNixos(batou.component.Component):

    def configure(self):
        self += batou.lib.file.File('i-am-nixos')


@batou.component.platform('ubuntu', Test)
class TestUbuntu(batou.component.Component):

    def configure(self):
        self += batou.lib.file.File('i-am-ubuntu')

The platform is then defined in the environment:

[environment]
platform = default-platform

[host:nixos]
# Host specifc override:
platform = nixos
components = test

[host:ubuntu]
# Host specifc override:
platform = ubuntu
components = test

Host-specific data

New in version 1.5.

Host-specific data allows to set environment dependent data for a certain host. It looks like this in an environment configuration:

[host:myhost00]
components = test
data-alias = nice-alias.for.my.host.example.com

In a component you can access all data attributes via the host’s data dictionary:

def configure(self):
    alias = self.host.data['alias']

The data- prefix was chosen in resemblance of the HTML standard.

DNS overrides

New in version 1.6

When migrating services automatic DNS lookup of IP addresses to listen on can be cumbersome. You want to deploy the service before the DNS changes become active. This is where DNS overrides can help.

The DNS overrides short circuit the resolving completely for the given host names.

Example:

[environment]
...

[resolver]
www.example.com =
    3.2.1.4
    ::2

Whenever batou configuration (i.e. batou.utils.Address) looks up www.example.com it will result in the addresses 3.2.1.4 and ::2.

The overrides support IPv4 and IPv6. You should only set one IP address per type for each host name.

Note

You cannot override the addresses of the configured hosts. The SSH connection will always use genuine name resolving.

context manager (TODO)

last_updated (TODO)

prepare, |=, component._ (TODO)

workdir overriding (TODO)

Importing components from a different component.py

The component configuration in the ./components folder is not a Python package: it has no __init__.py and should not have one. That’s why is not possible import a component into another one:

# This will not work
from components.nginx.component import MyAddress

In some rare circumstances it might be necessary to have this kind of import. There are two options:

  1. Import from batou.c.

  2. Create an extension module which can be imported.

Import from batou.c

There is a special module batou.c which is dynamically populated with all the classes in all component.py files.

Given the following tree:

$ tree
.
├── batou
├── components
│   └── myapp
│       └── component.py
│   └── myconfig
│       └── component.py
├── environments
│   └── local.cfg
├── requirements.lock
└── requirements.txt

If myconfig/component.py looks like this:

from batou.component import Component

class MyAddress(Component):
    """Some custom address."""

    ipv4 = "0.0.0.0"

myapp/component.py could import MyAddress like this:

from batou.component import Component
import batou.c

class MyApp(Component):
    """Some custom address."""

    def configure(self):
        """Configure MyApp."""
        self.address = batou.c.myconfig.MyAddress(ipv4="127.0.0.1")
        self += self.address
        self.ip = self.address.ipv4

This way, importing is only possible and reasonable inside methods of the component, which are executed after initial configuration such as configure (). It cannot be used to create an attribute on a class or import a base class for the current component file.

Caution

The components are loaded alphabetically, which can be an issue for the import.

Create an extension module

Another option to share code between different component files is to create a custom extension module. This can either be a separate repository like batou_ext or batou_scm but for a light weight start it can be included in the deployment repository. Have a look at the setup.py and other files for inspiration.

$ tree
.
├── batou
├── batou_myapp
│   ├── setup.py
│   └── src
│    └── batou_myapp
│        └── utils.py
├── components
│   └── myapp
│       └── component.py
├── environments
│   └── local.cfg
├── requirements.lock
└── requirements.txt
# batou_myapp/src/batou_myapp/utils.py
from batou.component import Component

class MyAddress(Component):
    """Some custom address."""

    ipv4 = "0.0.0.0"

class MyAppBase(Component):
    """Base component for all apps."""
# myapp/component.py
from batou.component import Component
from batou_myapp.utils import MyAppBase
from batou_myapp.utils import MyAddress

class MyApp(MyAppBase):
    """Some custom address."""

    address = MyAddress(ipv4="127.0.0.1")

    def configure(self):
        """Configure MyApp."""
        self.ip = self.address.ipv4

The requirements.txt has to be adapted to include the new extension module as requirement to be installed directly from source.

# requirements.txt
batou==2.3b2
-e ./batou_myapp
1

On Python 3.6 you have to use from remote_pdb import set_trace; set_trace().