Deploying debci to your own infrastructure

This document describes how to install debci for production usage. If you are trying to setup a development environment, see How to setup a development environment instead.

Architecture

The following picture represents the debci architecture:

  • The whole system communicates using an AMQP server, in this case rabbitmq, as a message queue.

  • Test jobs are injected into a queue by the debci-enqueue command line tool, or by the web UI/API.

  • Test jobs are picked up from the test queue by debci-worker daemons. They run the tests, and push the results back into a different queue.

  • debci-collector receives the test results, processes them, feeds the database with the results, and produces the HTML pages, atom feeds etc that in the web interface.

  • Each component (debci-collector, debci-worker, rabbitmq, and the command line tools) can live in different hosts if necessary, as long as all of the debci components can connect to the rabbitmq server.

  • You can have as many debci-worker nodes in the system as you want, but there must be only one debci-collector.

Deployment procedure

Install the debci-collector:

$ sudo apt install debci-collector

debci-collector recommends rabbitmq-server, and debci-collector will use a locally-installed rabbitmq server by default. If you want to use a remote rabbitmq, you need to add a configuration file with the .conf extension to /etc/debci/conf.d with something like this:

debci_amqp_server=amqp://MYRABBITMQSERVER

Note that if MYRABBITMQSERVER is network accessible, it has to have the proper ACLs configured. Check the rabbitmq documentation for details.

On each worker node, install apt-cacher-ng to cache package downloads, and debci-worker itself:

$ sudo apt install apt-cacher-ng debci-worker

As with debci-collector, debci-worker will connect to a local rabbitmq by default. To make it connect to a remote rabbitmq-server you can do the same as above.

Note that when first installed, debci-worker will first build a testbed (a container or a virtual machine image, depending on the selected backend), and only after that is finished the worker will be able to start processing test jobs.

Submitting test jobs

On any host that can connect to rabbitmq-server, first install debci:

$ sudo apt install debci

As usual, you will prompted for the address of the AMQP server. If the rabbitmq-server is on the same host, just leave it blank.

Say you want to run the tests for the ruby-defauts package. It is as easy as

$ debci enqueue ruby-defaults

Scheduling job submission

By default, debci will not submit any jobs. You need to decide how, and how often, you want to submit jobs. A simple way of doing that is using cron.

Example: schedule tests for a set of packages once a day

# /etc/cron.d/debci
0 7 * * * debci debci enqueue rake rubygems-integration ruby-defaults

You can also automate calls to debci enqueue in any other way you want.

Multiple debci-worker processes per node

If you have a system which has a lot of CPUs and a large amount of RAM available you can run multiple debci-worker daemons at once to maximise the resources being used.

By default there is only a single instance running, however it is possible to increase this. Example to have 4 simultaneous instances:

$ systemctl enable --now debci-worker@{1..4}.service

If we want to check on the status of them:

$ systemctl status debci-worker@{1..4}.service
$ journalctl -u debci-worker@2.service

However, the above only supports the same default backend. As a starting point to be able to support multiple backends, you could create a systemd override /etc/systemd/system/debci-worker@.service.d/override.conf:

[Service]
ExecStart=
ExecStart=/bin/bash -c 'v=%i; b="$${v%%%%_*}"; exec debci worker --backend "$$b"'

Then disable any previously created debci-workers, before creating any new ones:

$ systemctl --type=service \
  | awk '/debci-worker@/ {print $1}' \
  | while read x; do
    systemctl disable --now $x
  done
$ systemctl enable --now debci-worker@lxc_{1..4}.service
$ systemctl enable --now debci-worker@qemu_{1..2}.service

It is possible to start/stop all debci-workers at once rather than individually by:

$ systemctl enable --now debci-worker

Recap of services:

  • debci-worker.service which is all debci-worker’s, allowing them all to be controlled at once

  • debci-worker.service@X.service is the individual daemons (replacing X with the instant)

Setting up backends

debci supports various different backends such as lxc and qemu. As the default is lxc, it will be recommend with debci-worker:

$ sudo apt install debci-worker

Upon first install, debci-worker will first build a testbed based on the selected backend (environment variable: debci_backend). It is possible to create additional testbeds powered by different backends by doing:

$ sudo -u debci debci setup --backend qemu

For another example see MAINTAINERS.md and for more information about fake backend see HACKING.md.

Setting up PostgreSQL database

debci requires a database, and supports either: PostgreSQL and SQLite. The default is SQLite which is stored in data/debci.sqlite3. However the database can be changed using the environment variable, database_url.

First, make sure PostgreSQL is installed by:

$ sudo apt install postgresql

Afterwards, write a file to debci’s configuration directory, such as /etc/debci/conf.d/database.conf, with the following contents to use debci as the database name:

debci_database_url='postgresql:///debci'

After switching, you may wish to initialize the database:

$ sudo -u debci debci migrate

Note, this does not copy any data, but creates fresh.

Check the PostgreSQL documentation for securing the service.

Accessing Self Service (Web UI)

It is possible to access debci-collector web interface to perform certain actions. The authentication uses Debian Salsa (which is a self-hosted GitLab instance) for OAuth.

First, create a new application with the necessary values:

  • Name: Example Project

  • Redirect URI: https://ci.example.org/user/auth/gitlab/callback (Replace: ci.example.org with your public domain)

  • Scope: read_user

Afterwards, write a file to debci’s configuration directory, such as /etc/debci/conf.d/oauth.conf, with the following contents:

  • debci’s environment variable salsa_client_id: GitLab’s Application ID

  • debci’s environment variable salsa_client_secret: GitLab’s Secret

Below is an example format:

debci_salsa_client_id="0000000000000000000000000000000000000000000000000000000000000000"
debci_salsa_client_secret="0000000000000000000000000000000000000000000000000000000000000000"

You will then need to restart the debci-collector service to reload the configuration changes.

Creating API tokens

It is possible to self-create a single API token from debci-collector‘s web UI. Alternatively, the command line can be used, which also gives more options such as being able to specify the username:

$ sudo -u debci debci api
[...]
$ sudo -u debci debci api setkey USER1
00000000-0000-0000-0000-000000000000

Note, it is not possible at this stage to revoke tokens via debci api without manually altering the database.