How to manage controllers

See also: Controller

To be able to manage a controller, a user must have a controller superuser access level.

This document demonstrates various ways in which you can interact with a controller.

Contents:

Bootstrap a controller

To create a juju controller in a cloud, use the bootstrap command:

On Kubernetes: The Juju controller needs two container images (one for the controller agent container and one for the database container). These are by default downloaded from Docker Hub, but can also be downloaded from public.ecr.aws/juju or https://ghcr.io/juju if you pass them to the caas-image-repo bootstrap configuration key. We currently recommend you get them from public.ecr.aws/juju: juju bootstrap mycloud --config caas-image-repo="public.ecr.aws/juju".

See more: List of controller configuration keys > caas-image-repo
While this key can technically be changed after bootstrap, that is only for a very specific use case (adjusting credentials used for a custom registry). For most cases it is safe to assume you can only set it during bootstrap.

In a production setting:

- On machines: Make sure to bootstrap with no less than 50 GB disk, 2 CPUs, and 4 GB RAM (e.g., juju bootstrap aws/us-east-1 mymachinecontroller --bootstrap-constraints "root-disk=50G cores=2 mem=4G"). Bootstrapping a controller like this allows you to manage a few hundred units. However, if your needs go beyond this, consider making the controller highly available.

See more: How to manage machine constraints for a controller, How to make the controller highly available

- On Kubernetes: Juju does not currently support high-availability and backup and restore for Kubernetes controllers. Consider bootstrapping your controller on a machine cloud and then adding your Kubernetes cloud(s) to it, in a multi-cloud controller setup (juju add-k8s myk8scloud --controller mymachinecontroller).

See more: How to add a cloud

juju bootstrap 

This will start an interactive session where you will be asked for the name of the cloud and the name you want to give the controller.

Alternatively, you can specify these things directly by adding the name of the cloud and of the controller right after the bootstrap command. For example, below we bootstrap a controller with the name aws-controller into our aws cloud:

juju bootstrap aws aws-controller

When you use the bootstrap command in this way (non-interactively), you can also add many different options, to specify the cloud credentials to be used, to select a specific cloud region, to specify a storage pool, to constrain the controller or workload machines, to configure the deployment in various ways, to pass a cloud-specific setting, to choose a specific juju agent version, etc.

See more: juju bootstrap

The terraform juju client does not support this. Please use the juju client.

With the terraform juju client, in your Terraform plan, you can only connect to a pre-existing controller. You do that by configuring the juju provider to point to the pre-existing controller.

See more: How to manage the client

The python-libjuju client does not support this. Please use the juju client.

With the python-libjuju client, you can only connect to a pre-existing controller. See the section Switch to a different controller for more details.

See more: How to manage the client

View all the known controllers

To see a list of all the controllers known to the juju client, run the controllers command:

juju controllers

Sample output for a case where there is just a single controller boostrapped into the localhost cloud:

Use --refresh option with this command to see the latest information.

Controller             Model       User   Access     Cloud/Region         Models  Nodes    HA  Version
localhost-controller*  controller  admin  superuser  localhost/localhost       1      1  none  3.0.0  

By specifying various options you can also choose a specific output format, an output file, etc.

See more: juju controllers

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not support this. Please use the juju client.

View details about a controller

To view detailed information about a controller, use the show-controller command, optionally followed by one or more controller names. For example, below we examine a controller called localhost-controller:

juju show-controller localhost-controller

By specifying various options you can also choose an output format, an output file, or get an output that includes the password for the logged in user.

See more: juju show-controller

The terraform juju client does not support this. Please use the juju client.

To view details about a controller in python-libjuju, with a connected controller object (below, controller), you can call the Controller.info() function to retrieve information about the connected controller:

await controller.info()

See more: Controller.info()

Switch to a different controller

To switch from one controller to another, use the switch command followed by the name of the controller. For example, below we switch to a controller called localhost-controller-prod:

juju switch localhost-controller-prod

The switch command can also be used to switch to a different model. To remove any ambiguity, in some cases it may be safer to specify the model name explicitly on the template <controller-name>:<model-name>

See more: juju switch

The terraform juju client does not support this. Please use the juju client.

To switch to a different controller with python-libjuju, simply connect to the controller you want to work with, which is done by calling connect on the Controller object (below, controller):

from juju.model import Controller

controller = Controller()
await controller.connect() # will connect to the "current" controller

await controller.connect('mycontroller') # will connect to the controller named "mycontroller"

Note that if the controller object is already connected to a controller, then that connection will be closed before making the new connection.

See more: Controller.connect(), Connect with Authentication, Connect with explicit endpoints

Configure a controller

See also: Configuration, List of controller configuration keys

See related: How to configure a model

Set values. A controller configuration key can be assigned a value during controller-creation time or post-creation time. The vast majority of keys are set in the former way.

  • To set a controller’s configuration at controller-creation time, use the bootstrap command with the --config followed by the relevant <key>=<value> pair(s). For example, the code below creates a controller localhost on a cloud lxd and at the same time configures the controller such that the bootstrap-timeout key is 700 seconds:
juju bootstrap --config bootstrap-timeout=700 localhost lxd
  • To set a controller’s configuration once it’s already been created, use the controller-config command followed by the relevant <key>=<value> pair(s). For example, the code below configures an existing controller named aws so as to record auditing information, with the number of old audit log files to keep being set at 5.
juju controller-config -c aws auditing-enabled=true audit-log-max-backups=5

See more: juju bootstrap --config, juju controller-config

Get values. To get a controller’s current configuration, run:

juju controller-config

This will output a list of configuration keys and their values. This will include those that were set during controller creation, inherited as a default value, or dynamically set by Juju.

See more: juju controller-config

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not support this. Please use the juju client.

Manage constraints for a controller

See also: Constraint

Set values. You can set constraint values for either the entire controller model or just the controller application.

This distinction helps you address the fact that, while the controller model always contains the controller application, you may also deploy to it other applications (e.g., the juju-dashboard application), and their hardware needs may be different.

  • To apply a constraint to the entire controller model, run the bootstrap command with the --constraints flag, followed by the constraint key-value pairs that you want to set. For example, the code below ensures that every applicable resource in the controller model has 4GiB memory.
juju bootstrap --constraints mem=4G aws

See more: juju bootstrap --constraints, How to manage constraints for a model

  • To set constraints for just the controller application in the controller model, run the bootstrap command with the --bootstrap-constraints flag followed by the constraint key-value pairs that you want to set. For example, the code below ensures that any applicable resource associated with the controller application will have at least 2 CPUs:
juju bootstrap --bootstrap-constraints cores=2 google

See more: juju bootstrap --bootstrap-constraints

If you want to set both types of constraints at the same time, and they are different: Use --constraints to set values for the entire model, and --bootstrap-constraints to override them for the controller application.

Get values. You can get constraint values for either the entire controller model or just for the controller application.

  • To get constraint values for the entire controller model, use the usual procedure for getting model constraints (juju model-constraints -m controller).

See more: juju model-constraints

  • To get constraint values for just the controller application, inspect the controller machine (usually, juju show-machine 0).

See more: juju show-machine

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not support this. Please use the juju client.

Share a controller with other users

See also: User

The procedure for how to share a controller with other users depends on whether your controller is private or public.

Share a private controller. To share a private controller with other users:

  1. Create the users.

See more: How to add a user

  1. Send the users the information they need to register your controller with their client and to set up their login information for the controller.

See more: How to register a private controller

Share a public controller.

[TO BE ADDED]

Manage a controller’s connection to the client

To add / remove details of a controller to / from your Juju client, you need to register / unregister the controller.

Register a controller

If you are the creator of the controller: You can skip this step. It only applies for cases where you are trying to connect to an external controller.

The procedure for how to register a controller with the local system varies slightly depending on whether the controller is private or public.

Register a private controller. To register a private controller, use the register command followed by your unique registration key – that is, copy-paste and run the line of code provided to you by the person who has added you to the controller via the juju add-user command. For example:

juju register MFATA3JvZDAnExMxMDQuMTU0LjQyLjQ0OjE3MDcwExAxMC4xMjguMC4yOjE3MDcwBCBEFCaXerhNImkKKabuX5ULWf2Bp4AzPNJEbXVWgraLrAA=

This will start an interactive session prompting you to supply a local name for the controller as well as a username and a password for you as a new juju user on the controller.


Example session

Admin adding a new user ‘alex’ to the controller:

# Add a user named `alex`:
$ juju add-user alex
User "alex" added
Please send this command to alex:
    juju register MFUTBGFsZXgwFRMTMTAuMTM2LjEzNi4xOToxNzA3MAQghBj6RLW5VgmCSWsAesRm5unETluNu1-FczN9oVfNGuYTFGxvY2FsaG9zdC1jb250cm9sbGVy

"alex" has not been granted access to any models. You can use "juju grant" to grant access.

New user ‘alex’ accessing the controller:

$ juju register MFUTBGFsZXgwFRMTMTAuMTM2LjEzNi4xOToxNzA3MAQghBj6RLW5VgmCSWsAesRm5unETluNu1-FczN9oVfNGuYTFGxvY2FsaG9zdC1jb250cm9sbGVy
Enter a new password: ********
Confirm password: ********
Enter a name for this controller [localhost-controller]: localhost-controller
Initial password successfully set for alex.

Welcome, alex. You are now logged into "localhost-controller".

There are no models available. You can add models with
"juju add-model", or you can ask an administrator or owner
of a model to grant access to that model with "juju grant".


The command also has a flag that allows you to overwrite existing information, for cases where you need to reregister a controller.

See more: juju register, How to add a user

Register a public controller.

Network requirements: The client must be able to connect to the controller API over port 17070. Juju takes care of everything else. (And in most cases it takes care of this requirement too: for all clouds except for OpenStack Juju defaults to provisioning the controller with a public IP, and even for OpenStack you can choose to bootstrap with a floating IP as well.)

To register a public controller, use the register command followed by the DNS host name of the public controller. For example:

juju register public-controller.example.com

This will open a login window in your browser.

By specifying various flags you can also use this to reregister a controller or to type in your login information in your terminal rather than the browser.

See more: juju register

Unregister a controller

To remove knowledge of the controller from the juju client, run the unregister command followed by the name of the controller. For example:

juju unregister localhost-controller-prod

Note that this does not destroy the controller (though, to regain access to it, you will have to re-register it).

See more: juju unregister

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not support this. Please use the juju client.

Make a controller highly available

See also: High availability

To make a controller highly available, use the enable-ha command:

Currently only supported for controllers on a machine cloud.

juju enable-ha

This will make sure that the number of controllers increases to the default minimum of 3. Sample output:

maintaining machines: 0
adding machines: 1, 2

Expand to see other ways of checking this

We can also query for machines in the ‘controller’ model:

juju machines -m controller

The output should show two new machines being provisioned:

Machine  State    DNS            Inst id              Series  AZ          Message
0        started  54.166.164.0   i-04790c2414e4c8e80  xenial  us-east-1a  running
1        pending  54.145.192.13  i-071660e9ce3c3cee5  xenial  us-east-1c  running
2        pending  54.80.176.66   i-0b36284d1ebb816cf  xenial  us-east-1a  running

Invoking juju enable-ha again would have no effect since 3 controllers are already present.

Refreshing the list of controllers with juju controllers --refresh displays an HA level of 3:

Controller  Model    User   Access     Cloud/Region         Models  Machines    HA  Version
aws-ha*     default  admin  superuser  aws/us-east-1             2         3     3  2.4-beta2

Optionally, you can also mention a specific controller and also the number of controller machines you want to use for HA, among other things (e.g., constraints).

The number of controllers must be an odd number in order for a master to be “voted in” amongst its peers. A cluster with an even number of members will cause a random member to become inactive. This latter system will become a “hot standby” and automatically become active should some other member fail. Furthermore, due to limitations of the underlying database in an HA context, that number cannot exceed seven. All this means that a cluster can only have three, five, or seven active members.


If a controller is misbehaving, or if you’ve decided that you don’t need as many controllers for HA after all, you can remove them. To remove a controller, remove its machine from the controller model via the remove-machine command.

The enable-ha command cannot be used to remove machines from the cluster.

For example, below we remove controller 1 by removing machine 1 from the controller model:

juju remove-machine -m controller 1

If the removal of a controller will result in an even number of systems then one will act as a “hot standby”.
If the removal of a controller will result in an odd number of systems then each one will actively participate in the cluster.

The output to juju controllers --refresh now becomes as below, where the HA column says 1/2 – that is, there is now a single active controller, with the remaining controller on standby.

Controller  Model    User   Access     Cloud/Region         Models  Machines    HA  Version
aws-ha*     default  admin  superuser  aws/us-east-1             2         2   1/2  2.4-beta2

See more: juju enable-ha

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not support this. Please use the juju client.

Collect metrics about a controller

To collect metrics about a controller, take advantage of the controller application’s metrics endpoint to set up a relation between the controller and Prometheus, then collect metrics about the controller using Prometheus:

  1. On a Kubernetes cloud, deploy Prometheus / the Canonical Observability Stack, then offer Prometheus / Grafana Agent for cross-model relations.

This charm / bundle is offered only for Kubernetes, so you’ll need a Kubernetes cloud.

If your controller is already on a Kubernetes cloud, you can technically also deploy directly to the controller model and set up a same-model relation. However, it is much better practice to deploy your observability applications to a separate model and then set up cross-model relations.

  1. On your controller cloud, switch to the controller model and integrate the controller application with the Prometheus / Grafana Agent offer.

See more:

Back up a controller

This section demonstrates the various steps involved in backing up a controller.

The procedure documented below is currently supported only for machine (non-Kubernetes) controllers.

Create a controller backup

To create a backup of a controller configuration / metadata, use the create-backup followed by the -m flag and the name of the target controller model. For example, assuming a controller called localhost-controller, and the standard controller model name (controller), we will do:

juju create-backup -m localhost-controller:controller

Alternatively, you can switch to the controller model and use this command without any arguments or use the -m flag followed by just controller. However, due to the delicate nature of data backups, the verbose but explicit method demonstrated above is highly recommended.

Sample output:

backup format version: 1 
juju version:          3.0.0 
base:                  ubuntu@22.04 

controller UUID:       ca60f7e9-647b-4460-8232-fe75749e17c7
model UUID:            a04d7604-3073-45b7-871f-030ac0360fb4 
machine ID:            0 
created on host:       juju-360fb4-0 

checksum:              BrOGsXIK375529xlXJHX7m23Amk= 
checksum format:       SHA-1, base64 encoded 
size (B):              114919198 
stored:                0001-01-01 00:00:00 +0000 UTC 
started:               2022-11-09 09:06:46.800165238 +0000 UTC 
finished:              2022-11-09 09:07:05.133077079 +0000 UTC 

notes:                  

Downloaded to juju-backup-20221109-090646.tar.gz

The backup is downloaded to a default location on your computer (e.g., /home/user). A backup of a fresh (empty) environment, regardless of cloud type, is approximately 75 MiB in size.

The create-backup command also allows you to specify a custom filename for the backup file (--filename <custom-filename>). Note: You can technically also choose to save the backup on the controller (--no-download), but starting with juju v.3.0 this flag is deprecated.

See more: juju create-backup

Download a controller backup

Suppose you’ve created a backup with the --no-download option, as shown below (where controller is the name of the controller model).

Starting with juju v.3.0, this flag is deprecated.

$ juju create-backup -m controller --no-download
WARNING --no-download flag is DEPRECATED.

backup format version: 1 
juju version:          3.0.0 
base:                  ubuntu@22.04 

controller UUID:       ca60f7e9-647b-4460-8232-fe75749e17c7
model UUID:            a04d7604-3073-45b7-871f-030ac0360fb4 
machine ID:            0 
created on host:       juju-360fb4-0 

checksum:              tjqEvlspc88mYQmjV9u/m4i+prg= 
checksum format:       SHA-1, base64 encoded 
size (B):              114919131 
stored:                0001-01-01 00:00:00 +0000 UTC 
started:               2022-11-09 09:08:51.314128218 +0000 UTC 
finished:              2022-11-09 09:09:10.296320799 +0000 UTC 

notes:                  

Remote backup stored on the controller as /tmp/juju-backup-20221109-090851.tar.gz

As you can see from the output, this has resulted in the backup being saved remotely on the controller as /tmp/juju-backup-20221109-090851.tar.gz.

To download the backup, use the download-backup command followed by the remote location of the backup. In our case:

juju download-backup /tmp/juju-backup-20221109-090851.tar.gz

This will output the name of the downloaded backup file. In our case:

juju-backup-20221109-090851.tar.gz

This file will have been downloaded to a temporary location (in our case, /home/user).

See more: juju download-backup

Restore a controller from a backup

To restore a controller from a backup, you can use the stand-alone juju-restore tool.

First, download the juju-restore tool and copy it to the primary controller machine in the MongoDB replica set (typically, machine 0). To identify the primary controller machine, you can use the juju show-controller – its output will list all the machines and the primary will contain ha-primary: true:

juju show-controller
...
  controller-machines:
    "0":
      instance-id: i-073443a840f1a3626
      ha-status: ha-enabled
      ha-primary: true
    "1":
      instance-id: i-0be2c1b818e54a2ba
      ha-status: ha-enabled
    "2":
      instance-id: i-0b4705ede7d3c0faa
      ha-status: ha-enabled
...

Then you can copy the restore tool:

# Download the latest release binary (Linux, AMD64):
wget https://github.com/juju/juju-restore/releases/latest/download/juju-restore
chmod +x juju-restore

# Switch to the controller model:
juju switch controller

# Copy juju-restore to the primary controller machine:  
juju scp juju-restore 0:

Second, assuming that during the create-backup step you chose to save a local copy (the default option), use scp to copy the file to the same controller machine, as shown below.

juju scp <path-to-backup> 0:

If you’ve used create-download with the --no-download option, you can skip this step – the backup is already on the primary controller machine.

Now, SSH into this machine and run ./juju-restore followed by the path to the backup file, as shown below. All replica set nodes need to be healthy and in PRIMARY or SECONDARY state.

# SSH into the controller machine
juju ssh 0

# Start the restore! (it will ask for confirmation)
./juju-restore <path-to-backup>

The juju-restore tool also provides several options, among which:

  • --yes: answer “yes” to confirmation prompts (for non-interactive mode)
  • --include-status-history: restore the status history collection for machines and units (which can be large, and usually isn’t needed)
  • --username, --password, and related options: override the defaults for connecting to MongoDB
  • --allow-downgrade: restore from a backup created with an earlier juju version
  • --manual-agent-control: (in the case of restoring backups to high availability controllers) stop and restart juju agents and Mongo daemons on the secondary controller machines manually
  • --copy-controller: clone the configuration of an old controller into a new controller (download the latest juju-restore to see this option).
    Note: This feature is useful in upgrading to 3.0. See How to upgrade your deployment from 2.9 to 3.0.

For the full list of options, type: ./juju-restore --help

See more: juju-restore

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not support this. Please use the juju client.

Upgrade a controller

The procedure for upgrading a controller depends on whether you’re upgrading it across patch versions or rather minor or major versions.

Upgrade a controller’s patch version

To upgrade a controller’s patch version (e.g., v2.9.25v2.9.26), use the upgrade-controller command (or, equivalently, upgrade-model -m controller):

juju upgrade-controller

By using various options you can make this into just a simulation of an upgrade, or choose a specific agent version to upgrade to, or choose a timeout, etc.

When you want to upgrade the controller but the controller is without internet access, you can copy agent binaries from the official agent store into the controller via an internet-connected client by using the juju sync-agent-binary command: juju sync-agent-binary. By passing various options you can also specify a local destination directory, a version, etc.

See more: juju upgrade-controller, `juju upgrade-model -m controller

Upgrade a controller’s minor or major version

If you want to upgrade a controller’s minor (e.g., v3.0.xv3.1.x ) or major (v2.x.xv3.x.x ) version, you cannot do this in place via the upgrade-controller command. Instead, you must create a new controller (juju bootstrap), migrate the models from your old controller to it (juju migrate), and then remove the old controller.

Before juju v3.0.0, it is possible to use juju upgrade-controller to upgrade a minor version (e.g., v2.7.xv2.8.x, v2.8.xv2.9.x). But this will not work past v2.9.x (e.g., v2.9.xv3.0.x or v3.0.xv3.1.x will not proceeed).

See more: How to create a controller, How to migrate a workload model to another controller

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not support this. Please use the juju client.

Remove a controller

See also: Removing things

There are two ways to remove a controller. Below we demonstrate each, in order of severity.

For how to remove knowledge about a controller from a juju client, see Unregister a controller.

Destroy a controller

A controller can be destroyed with:

juju destroy-controller <controller-name>

You will always be prompted to confirm this action. Use the -y option to override this.

As a safety measure, if there are any models (besides the ‘controller’ model) associated with the controller you will need to pass the --destroy-all-models option.

Additionally, if there is persistent storage in any of the controller’s models you will be prompted to either destroy or release the storage, using the --destroy-storage or --release-storage options respectively.

For example:

juju destroy-controller -y --destroy-all-models --destroy-storage aws

Any model in the controller that has disabled commands will block a controller from being destroyed. A controller administrator is able to enable all the commands across all the models in a Juju controller so that the controller can be destroyed if desired. This can be done via the enable-destroy-controller command: juju enable-destroy-controller.

See more: juju destroy-controller

Use the kill-controller command as a last resort if the controller is not accessible for some reason.

Kill a controller

The kill-controller command deserves some attention as it is very destructive and also has exceptional behaviour modes. This command will first attempt to remove a controller and its models in an orderly fashion. That is, it will behave like destroy-controller. If this fails, usually due the controller itself being unreachable, then the controller machine and the workload machines will be destroyed by having the client contact the backing cloud’s API directly.

See more: juju kill-controller

The terraform juju client does not support this. Please use the juju client.

The python-libjuju client does not support this. Please use the juju client.


Contributors: @cderici, @hmlanigan, @pmatulis, @tmihoc

Last updated 3 days ago. Help improve this document in the forum.