Overview

This charm is a thin automation layer to run DEP8 tests in a target environment.

This charm will deploy a basic environment to run DEP8 tests (with autopkgtest) on the target MAAS-provisioned machine, VM or container. Packages will be build and run in the target itself to improve test coverage.

Basic idea is to reuse the tests already available in the Debian packages to validate new hardware configurations and run these tests in existing infrastructure while using Juju to abstract the nature of the target environment. It will not matter, for the charm, if the test is running in a VM or in bare metal, but the intention is run them on metal to minimize the interference of virtualization providers on test results.

The test run directly on the target environment using the "null" autopkgtest virtualization provider and it is assumed the environment is "disposable", i.e. it will released or recreated once the tests are finished.

Usage

This charm is intended to be deployed and used through test automation but, in development enviroments, the deployment procedure is:

• Install the charm with
  $ juju deploy cs:~ce-hyperscale/autopkgtest testbed1.
  Notice that this example assumes a testbed named "testbed1".

• Run a test with the Juju action run-test, passing the Debian package name in argument pkgname:

  $ juju run-action testbed1/0 run-test pkgname=nginx
  Action queued with id: d13d1822-724f-4719-8ed1-35dd0d124474
  

• Use the action-id to check if the execution is already finished:

  $ juju show-action-status d13d1822-724f-4719-8ed1-35dd0d124474
  actions:
  - action: run-test
    completed at: n/a
    id: d13d1822-724f-4719-8ed1-35dd0d124474
    status: running
    unit: testbed1/0
  

• Use juju debug-log to see the outputs of the test (stdout and stderr) in realtime;

• Once the test is finished, grab the output with:

  $ juju show-action-output d13d1822-724f-4719-8ed1-35dd0d124474
    results:
      output-file: /tmp/test-nginx-kz6TT/output.txt
      retcode: "0"
      summary-file: /tmp/test-nginx-kz6TT/summary.txt
    status: completed
    timing:
      completed: 2019-10-22 21:54:33 +0000 UTC
      enqueued: 2019-10-22 21:48:01 +0000 UTC
      started: 2019-10-22 21:48:06 +0000 UTC
  

Optionally, run the action with the "--wait" command line switch. This spares the user from polling the test state, but blocks the Juju client process.

Interpreting outputs

The exit code returned by autopkgtest will be saved in variable retcode (see autpkgtest manpage for more information on it meaning). This only happens if autopkgtest gets a chance to run and won't be created if the test script fails before this step.

Variables summary and output will have paths to files containing the summary of the test results and a copy of stdout and stderr of the test execution. Use juju ssh to see their contents or copy them to the host machine.

    $ juju ssh testbed1/0 cat /tmp/test-nginx-kz6TT/summary.txt
    light-simple         PASS
    core-simple          PASS
    lua                  PASS
    full-simple          PASS
    extras-simple        PASS
    full-module-deps     PASS
    light-module-deps    PASS
    core-module-deps     PASS
    extras-module-deps   PASS
    Connection to 10.214.19.195 closed.

Notice that file output.txt can be pretty large.

Known Limitations and Issues

• Some tests may refuse to run in containers;

• Autopkgtest has a protection against running tests that can break the testbed (restriction breaks-testbed). It does not matter for us as the target will be recreated anyway, but these tests will require patching to force them to run.