autopkgtest #1

Supports: xenial bionic eoan
Add to new model

Description

This charm will deploy a basic environment to run DEP8 tests (with autopkgtest) on the target machine, VM or container. Basic idea is to reuse the tests already available in the packages and run them, using Juju to automate target selection and deployment. It is assumed the target environment is "disposable", i.e. it will released or recreated once the tests are finished.


Overview

This charm is a thin automation layer to run DEP8 tests in a target environment.

This charm will deploy a basic environment to run DEP8 tests (with autopkgtest) on the target MAAS-provisioned machine, VM or container. Packages will be build and run in the target itself to improve test coverage.

Basic idea is to reuse the tests already available in the Debian packages to validate new hardware configurations and run these tests in existing infrastructure while using Juju to abstract the nature of the target environment. It will not matter, for the charm, if the test is running in a VM or in bare metal, but the intention is run them on metal to minimize the interference of virtualization providers on test results.

The test run directly on the target environment using the "null" autopkgtest virtualization provider and it is assumed the environment is "disposable", i.e. it will released or recreated once the tests are finished.

Usage

This charm is intended to be deployed and used through test automation but, in development enviroments, the deployment procedure is:

  • Install the charm with $ juju deploy cs:~ce-hyperscale/autopkgtest testbed1. Notice that this example assumes a testbed named "testbed1".

  • Run a test with the Juju action run-test, passing the Debian package name in argument pkgname:

    $ juju run-action testbed1/0 run-test pkgname=nginx
    Action queued with id: d13d1822-724f-4719-8ed1-35dd0d124474
    
  • Use the action-id to check if the execution is already finished:

    $ juju show-action-status d13d1822-724f-4719-8ed1-35dd0d124474
    actions:
    - action: run-test
      completed at: n/a
      id: d13d1822-724f-4719-8ed1-35dd0d124474
      status: running
      unit: testbed1/0
    
  • Use juju debug-log to see the outputs of the test (stdout and stderr) in realtime;

  • Once the test is finished, grab the output with:

    $ juju show-action-output d13d1822-724f-4719-8ed1-35dd0d124474
      results:
        output-file: /tmp/test-nginx-kz6TT/output.txt
        retcode: "0"
        summary-file: /tmp/test-nginx-kz6TT/summary.txt
      status: completed
      timing:
        completed: 2019-10-22 21:54:33 +0000 UTC
        enqueued: 2019-10-22 21:48:01 +0000 UTC
        started: 2019-10-22 21:48:06 +0000 UTC
    

Optionally, run the action with the "--wait" command line switch. This spares the user from polling the test state, but blocks the Juju client process.

Interpreting outputs

The exit code returned by autopkgtest will be saved in variable retcode (see autpkgtest manpage for more information on it meaning). This only happens if autopkgtest gets a chance to run and won't be created if the test script fails before this step.

Variables summary and output will have paths to files containing the summary of the test results and a copy of stdout and stderr of the test execution. Use juju ssh to see their contents or copy them to the host machine.

    $ juju ssh testbed1/0 cat /tmp/test-nginx-kz6TT/summary.txt
    light-simple         PASS
    core-simple          PASS
    lua                  PASS
    full-simple          PASS
    extras-simple        PASS
    full-module-deps     PASS
    light-module-deps    PASS
    core-module-deps     PASS
    extras-module-deps   PASS
    Connection to 10.214.19.195 closed.

Notice that file output.txt can be pretty large.

Known Limitations and Issues

  • Some tests may refuse to run in containers;

  • Autopkgtest has a protection against running tests that can break the testbed (restriction breaks-testbed). It does not matter for us as the target will be recreated anyway, but these tests will require patching to force them to run.