FAQ for package maintainers

Why should I use autopkgtest when my build runs the test suite?

autopkgtest is intended to test packages in their installed state, and that is not the same as testing the code that is in the source tree. During the build, the tests run against the code in the source tree. Depending on the package, that could be enough, but also running those same tests against the installed package can catch a host of problems that can’t be caught when running against the source tree, e.g.:

  • ABI breakages and other types of problems that are “magically” fixed with a rebuild.

  • Missing dependencies (as opposed to build dependencies)

  • Some files that are essential for the software to function correctly were not included in the binary package.

Another practical reason to have autopkgtest tests in your package is that debci runs autopkgtest more frequently than packages are built. For example, when any of your dependencies is updated, your package is tested again against that new version.

Now, besides running tests that normally run during the package build under autopkgtest as well, it is also a good idea to add tests that are specific to the package as installed and are more “black box”, i.e. tests that don’t assume or have access to implementation details of the package (those are usually called integration tests). For example:

  • if a package provides a service, add tests that exercises that service as a client, and check the expected results of that interaction.

  • if a package provides a program, add tests that check that the invocations of that program with different parameters and inputs produce the corresponding expected results.

This type of test can also, and most often that not should, be contributed upstream.

If autopkgtest can rebuild the package, why won’t debci do that?

The primary reason is because we want to test the packages that are actually in the archive, because that is what our users get from us.

Weaker, but practical reasons, are:

  • Rebuilding every package would add significant load to the system.

  • The buildd network already builds packages and we don’t want to duplicate that function.

How can I reproduce the test run locally?

NOTE: if you intend to run tests frequently, you should consider installing apt-cacher-ng before anything else. debci will notice the running proxy and will setup the testbed to use it, so you won’t have to wait for the download of each package more than once.

Using no virtualization backend

This is the fastest way to run tests, but does not reproduce the continuous integration environment faithfully and your local environment may make the results unreliable. It’s useful nevertheless when initially writing the tests, or for quick runs against your main host system.

To run the test suite from the root of a source package against the currently installed packages, run:

$ autopkgtest --output-dir /tmp/output-dir -B  . -- null

For more details, see the documentation for the autopkgtest package.

Using the lxc backend

The first step is installing debci and autopkgtest:

$ sudo apt install debci autopkgtest

You will also need permissions to run the lxc-* commands as root, preserving your environment. An easy way to do that is to add yourself to the debci group.

$ sudo adduser YOUR_USERNAME debci

You’re now ready to create the lxc container:

$ sudo debci setup

This might take a few minutes since it will create a fresh container from scratch.

You can use an alternative mirror using, for example:

$ sudo env debci_mirror=http://my.local.mirror/debian debci setup

Now to actually run tests, we’ll use autopkgtest directly. The following examples assume your architecture is amd64, replace it by your actual architecture if that is not the case.

To run the test suite from a source package in the archive, just pass the source package name to autopkgtest:

$ autopkgtest --user debci --output-dir /tmp/output-dir SOURCEPACKAGE \
  -- lxc --sudo autopkgtest-unstable-amd64

To run the test suite against a locally-built source package, using the test suite from that source package and the binary packages you just built, you can pass the .changes file to autopkgtest:

$ autopkgtest --user debci --output-dir /tmp/output-dir \
  /path/to/PACKAGE_x.y-z_amd64.changes \
  -- lxc --sudo autopkgtest-unstable-amd64

For more details, see the documentation for the autopkgtest package.

How do I get my package to have its test suite executed?

Test suites must be included in source packages as defined in the DEP-8 specification. In short.

  • The fact that the package has a test suite must be declared in the source control file. With dpkg-source version 1.17.11 or later this is added automatically when debian/tests/control is present.

  • if the package is built with dpkg earlier than 1.17.11, you need to add an Testsuite: autopkgtest entry to the source stanza in debian/control.

  • if the package is built with dpkg earlier than 1.17.6, you need to use XS-Testsuite: autopkgtest instead.

  • tests are declared in debian/tests/control.

Please refer to the DEP-8 spec for details on how to declare your tests.

How exactly is the test suite executed?

Test suites are executed by autopkgtest. The version of autopkgtest used to execute the tests is shown in the log file for each test run.

How often are test suites executed?

The test suite for a source package will be executed:

  • when any package in the dependency chain of its binary packages changes;

  • when the package itself changes;

  • when 1 month is passed since the test suite was run for the last time.

edit this page