In-depth topics
===============


Writing a custom component (TODO)
---------------------------------

Debugging batou runs
--------------------

Using a debugger
++++++++++++++++

``batou`` comes with `remote-pdb <https://pypi.org/project/remote-pdb/>`_
pre-installed. When running on Python 3.7+ [#]_ you can use ``breakpoint()`` to
drop into the debugger. You need ``telnet`` or ``netcat`` to connect to the
hostname and port displayed.

If you are using the default configuration, call::

  $ nc -C 127.0.0.1 4444
  or
  $ telnet 127.0.0.1 4444

If you are debugging a remote deployment you should create a port forward
beforehand like this::

  $ ssh -L 4444:localhost:4444 my.remote.host.dev

You are able to configure hostname and port. For details see the documentation
of `remote-pdb <https://pypi.org/project/remote-pdb/>`_. This works both for
local and remote deployments. The environment variables for host and port are
propagated to the remote host.

Example shell session for debugging using a custom port::

  $ REMOTE_PDB_PORT=4445 ./batou deploy dev
  batou/2.3b2.dev0 (cpython 3.6.15-final0, Darwin 20.6.0 x86_64)
  ============================= Preparing ============================
  main: Loading environment `dev`...
  main: Verifying repository ...
  main: Loading secrets ...
  ============================= Connecting ... =======================
  localhost: Connecting via local (1/1)
  ============================= Configuring model ... ================
  RemotePdb session open at 127.0.0.1:4445, waiting for connection ...

Example for a second terminal where the opened port gets connected to::

  $ nc -C 127.0.0.1 4445
  > /.../components/supervisor/component.py(32)configure()
  -> if self.host.platform in ('gocept.net', 'fcio.net'):
  (Pdb)


Using "print debugging"
+++++++++++++++++++++++

* Inside a component you can use ``self.log("output")`` to print to the
  console.
* You can call ``batou`` with the ``-d`` flag to enable debugging output during
  the deployment run.


Using 3rd party libraries within batou
--------------------------------------

Sometimes, when writing custom components, you may need additional Python
packages, for example to configure databases by connecting directly to their
SQL interface instead of using their command line clients.

You can use additional Python packages by adding a  `requirements.txt` file to your batou project repository:

.. code-block:: console

  $ tree
  .
  ├── batou
  ├── components
  │   └── myapp
  │       └── component.py
  ├── environments
  │   └── local
  │       └── environment.cfg
  └── requirements.txt

.. code-block:: text
  :caption: requirements.txt

  sqlalchemy

The next time when you call batou the dependencies will be automatically
updated. When deploying then the requirements will also be installed
on the remote hosts.

.. code-block:: console

    $ ./batou
    Installing sqlalchemy
    usage: batou [-h] [-d] {deploy,secrets,init,update} ...

.. note::

   batou already provides a number of packages that it depends on.
   If you create contradicting requirements then this may lead to batou
   failing. You will see pip complaining in that case.


Multiple components in a single component.py (TODO)
---------------------------------------------------


Skipping individual hosts or components when deploying (TODO)
-------------------------------------------------------------


Events (TODO)
-------------


Using bundle transfers if the repository server is not reachable from your remote server (TODO)
-----------------------------------------------------------------------------------------------

Timeout (TODO)
--------------



VFS mapping for development (TODO)
----------------------------------

VFS mapping with explicit rewrite rules (TODO)
----------------------------------------------


Extended service discovery options (TODO)
-----------------------------------------


Platform-specific components
----------------------------

*New in version 1.4.*

Platform-specific components allow to customize behavior depending on the system or "platform" the target system runs as. Examples:

* Production system on Gentoo, local development on Ubuntu, or
* All VMs on Ubuntu but Oracle is being  run with RedHat.

To define a platform specific aspects, you use the `platform` class decorator. Example::

    import batou.component
    import batou.lib.file


    class Test(batou.component.Component):

        def configure(self):
            self += batou.lib.file.File('base-component')


    @batou.component.platform('nixos', Test)
    class TestNixos(batou.component.Component):

        def configure(self):
            self += batou.lib.file.File('i-am-nixos')


    @batou.component.platform('ubuntu', Test)
    class TestUbuntu(batou.component.Component):

        def configure(self):
            self += batou.lib.file.File('i-am-ubuntu')

The platform is then defined in the environment::

    [environment]
    platform = default-platform

    [host:nixos]
    # Host specifc override:
    platform = nixos
    components = test

    [host:ubuntu]
    # Host specifc override:
    platform = ubuntu
    components = test


Host-specific data
------------------

*New in version 1.5.*

Host-specific data allows to set environment dependent data for a certain *host*. It looks like this in an environment configuration::

    [host:myhost00]
    components = test
    data-alias = nice-alias.for.my.host.example.com


In a component you can access all data attributes via the host's `data` dictionary::

    def configure(self):
        alias = self.host.data['alias']

The ``data-`` prefix was chosen in resemblance of the `HTML standard <http://w3c.github.io/html/dom.html#dom-htmlelement-dataset>`_.



DNS overrides
-------------

*New in version 1.6*

When migrating services automatic DNS lookup of IP addresses to listen on can be cumbersome. You want to deploy the service before the DNS changes become active. This is where DNS overrides can help.

The DNS overrides short circuit the resolving completely for the given host names.

Example::

    [environment]
    ...

    [resolver]
    www.example.com =
        3.2.1.4
        ::2

Whenever batou *configuration* (i.e. ``batou.utils.Address``) looks up ``www.example.com`` it will result in the addresses ``3.2.1.4`` and ``::2``.


The overrides support IPv4 and IPv6. You should only set one IP address per type for each host name.

.. NOTE:: You *cannot* override the addresses of the configured hosts. The SSH connection will always use genuine name resolving.


context manager (TODO)
----------------------

last_updated (TODO)
-------------------

prepare, ``|=``, ``component._`` (TODO)
---------------------------------------

workdir overriding (TODO)
-------------------------

Importing components from a different component.py
--------------------------------------------------

The component configuration in the ``./components`` folder is not a Python
package: it has no ``__init__.py`` and should not have one. That's why is not
possible import a component into another one::

    # This will not work
    from components.nginx.component import MyAddress

In some rare circumstances it *might* be necessary to have this kind of import.
There are two options:

1. Import from ``batou.c``.
2. Create an extension module which can be imported.

Import from ``batou.c``
+++++++++++++++++++++++

There is a special module ``batou.c`` which is dynamically populated with all
the classes in all ``component.py`` files.

Given the following tree:

.. code-block:: console

  $ tree
  .
  ├── batou
  ├── components
  │   └── myapp
  │       └── component.py
  │   └── myconfig
  │       └── component.py
  ├── environments
  │   └── local
  │       └── environment.cfg
  ├── requirements.lock
  └── requirements.txt


If ``myconfig/component.py`` looks like this:

.. code-block:: python

    from batou.component import Component

    class MyAddress(Component):
        """Some custom address."""

        ipv4 = "0.0.0.0"


``myapp/component.py`` could import ``MyAddress`` like this:

.. code-block:: python

    from batou.component import Component
    import batou.c

    class MyApp(Component):
        """Some custom address."""

        def configure(self):
            """Configure MyApp."""
            self.address = batou.c.myconfig.MyAddress(ipv4="127.0.0.1")
            self += self.address
            self.ip = self.address.ipv4


This way, importing is only possible and reasonable inside methods of the
component, which are executed after initial configuration such as ``configure
()``. It cannot be used to create an attribute on a class or import a base
class for the current component file.

.. caution::

    The components are loaded alphabetically, which can be an issue for the
    import.


Create an extension module
++++++++++++++++++++++++++

Another option to share code between different component files is to create a
custom extension module. This can either be a separate repository like
`batou_ext <https://github.com/flyingcircusio/batou_ext>`_ or `batou_scm
<https://github.com/gocept/batou_scm>`_ but for a light weight start it can be
included in the deployment repository. Have a look at the ``setup.py`` and
other files for inspiration.

.. code-block:: console

  $ tree
  .
  ├── batou
  ├── batou_myapp
  │   ├── setup.py
  │   └── src
  │    └── batou_myapp
  │        └── utils.py
  ├── components
  │   └── myapp
  │       └── component.py
  ├── environments
  │   └── local
  │       └── environment.cfg
  ├── requirements.lock
  └── requirements.txt


.. code-block:: python

    # batou_myapp/src/batou_myapp/utils.py
    from batou.component import Component

    class MyAddress(Component):
        """Some custom address."""

        ipv4 = "0.0.0.0"

    class MyAppBase(Component):
        """Base component for all apps."""

.. code-block:: python

    # myapp/component.py
    from batou.component import Component
    from batou_myapp.utils import MyAppBase
    from batou_myapp.utils import MyAddress

    class MyApp(MyAppBase):
        """Some custom address."""

        address = MyAddress(ipv4="127.0.0.1")

        def configure(self):
            """Configure MyApp."""
            self.ip = self.address.ipv4

The ``requirements.txt`` has to be adapted to include the new extension module
as requirement to be installed directly from source.

.. code-block:: cfg

    # requirements.txt
    batou==2.3b2
    -e ./batou_myapp

.. [#] On Python 3.6 you have to use ``from remote_pdb import set_trace; set_trace()``.