ubuntu mirror #3

Supports: bionic xenial trusty focal
Add to new model

Description

Provides an Ubuntu archive mirror and keeps it up-to-date using rsync


Overview

This charm provides a complete Ubuntu mirror service, handling fetching the data via rsync from a master server (updates either triggered on-demand using ssh or by cron), and making it available through http, ftp and rsync.

It can work alongside a log-archive subordinate charm to transfer logs to a suitable storage host.

Usage

Assuming you already have a bootstrapped Juju environment, the charm can be deployed using:

juju deploy ubuntu-mirror

Once deployed, you will need to configure it for your local environment. An Ubuntu archive mirror can be created using the following commands:

juju set ubuntu-mirror mirror_ubuntu_name="myhost.mydomain.com"
juju set ubuntu-mirror mirror_ubuntu_sync_time="0 1 * * *"
juju set ubuntu-mirror role_map='{"myhost":["ubuntu"]}'

Replace "myhost" with the hostname of your server and "myhost.mydomain.com" with its fully qualified domain name (FQDN).

The "role_map" variable tells the charm what type of mirror you want to create (currently, this can be any of "ubuntu" (a mirror of archive.ubuntu.com), "releases" (a mirror of releases.ubuntu.com), "cdimage" (a mirror of cdimage.ubuntu.com" or "ports" (a mirror of ports.ubuntu.com) If you have a sufficiently large server, multiple roles can be assigned to one server (role_map='{"myhost":["ubuntu","cdimage"]}' for example)

If you have more than one server, you should list each one (e.g. role_map='{"host1":["cdimage","ubuntu"],"host2":["releases"]}')

The "mirror_ubuntu_name" variable is the name the webserver will use for the Ubuntu mirror. In the above example, it would answer requests for http://myhost.mydomain.com

The "mirror_ubuntu_sync_time" variable tells the charm what time it should update. In the above example, the update would happen every day at 1am.

There are a number of variables that can be set on a per-role basis. They all follow the pattern mirror__, so if we were setting up a cdimage mirror, we would use "mirror_cdimage_name" and "mirror_cdimage_sync_time".

To cater for rsync servers that require authentication, there is a per-role variable, mirror__rsync_auth. If you don't need to authenticate against your upstream mirror, leave this alone, the defaults will be fine. If you do need to authenticate, this variable should be used like this:

juju set ubuntu-mirror mirror_cdimage_rsync_auth='{"myhost":{"user":"eric", "password":"YXBwbGVzCg=="}}'

The password value is just a Base64-encoded string, just in case it needs to contain any special characters (it also helps avoid people casually reading passwords over your shoulder when you run juju get).

As with role_map (above), if you have multiple servers they should all be listed (e.g. mirror_cdimage_rsync_auth='{"host1":{"user":"eric","password":"YXBwbGVzCg=="},"host2":{"user":"frank","password":"b3Jhbmdlcwo="}}')

For simplicity, if all of your servers authenticate with the same username and password, you can use: '{"<>":{"user":"eric","password":"YXBwbGVzCg=="}}'.


Configuration

apache_delete_logs_after
(int) Delete apache logs after n days.
14
apache_logdir
(string) Where Apache logfiles should be written
/var/log/apache2
apache_maxclients
(int) Maximum number of simultaneous client connections
4096
apache_maxrequestsperchild
(int) Maximum number of requests a server process serves
apache_maxsparethreads
(int) Maximum number of Apache worker threads which are kept spare
100
apache_minsparethreads
(int) Minimum number of Apache worker threads which are kept spare
50
apache_modules
(string) A space separated list of extra apache modules to enable
apache_serverlimit
(int) Maximum number of Apache server processes allowed
256
apache_startservers
(int) Initial number of Apache server processes to start
2
apache_status_allowed_from
(string) A list of IP addresses allowed to access the Apache server-status page
[]
apache_threadlimit
(int) Upper limit of ThreadsPerChild allowed in a graceful restart
64
apache_threadsperchild
(int) Constant number of worker threads in each Apache server process
64
application_name
(string) The name of this charm
ubuntu-mirror
archive_logs_after
(int) Archive the logs after n days. This is only used when a suitable log-archive subordinate charm is provided
1
archive_logs_apache
(boolean) Enable log archiving for Apache logs
True
archive_logs_ftp
(boolean) Enable log archiving for FTP logs
archive_logs_rsync
(boolean) Enable log archiving for Rsync logs
disk_inode_crit
(string) Disk inode critical threshold
5%
disk_space_crit
(string) Disk space critical threshold
20%
disk_space_warn
(string) Disk space warning threshold
25%
ftp_delete_logs_after
(int) Delete FTP logs after n days.
28
ftp_logdir
(string) Where ftpd logfiles should be written
/var/log/ftp
ftp_max_instances
(int) Maximum number of FTP connections allowed
200
ftp_max_per_source
(int) Maximum number of concurrent FTP connections allowed
10
ftp_welcome_banner
(string) FTP welcome message
FTP server (vsftpd)
mirror_cdimage_aliases
(string) List of Apache aliases for the cdimage mirror
[]
mirror_cdimage_apache_early_extra
(string) A base64 string containing apache configuration options to be included early in the config file
mirror_cdimage_apache_late_extra
(string) A base64 string containing apache configuration options to be included late in the config file
mirror_cdimage_command
(string) The command to use to sync the cdimage mirror
mirror-1stage.sh
mirror_cdimage_description
(string) A brief welcome message for the cdimage mirror
Ubuntu CD Images
mirror_cdimage_name
(string) FQDN of the cdimage mirror
cdimage.ubuntu.com
mirror_cdimage_path
(string) Root location of mirrored files for the cdimage mirror
/srv/ftp.root/cdimage
mirror_cdimage_rsync_auth
(string) Optional rsync authentication details for mirror_cdimage_source_url
{}
mirror_cdimage_rsync_log
(boolean) Whether to log rsync requests for the cdimage mirror
True
mirror_cdimage_rsync_module
(string) The name of the rsync module for this mirror role
cdimage
mirror_cdimage_source_url
(string) The URL the cdimage mirror will be fetched from
rsync://cdimage.ubuntu.com/cdimage
mirror_cdimage_sync_time
(string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering
ondemand
mirror_cdimage_trigger
(string) A base64 string containing the ssh trigger public key
mirror_cloud-images_aliases
(string) List of Apache aliases for the cloud-images mirror
[]
mirror_cloud-images_apache_early_extra
(string) A base64 string containing apache configuration options to be included early in the config file
mirror_cloud-images_apache_late_extra
(string) A base64 string containing apache configuration options to be included late in the config file
mirror_cloud-images_command
(string) The command to use to sync the cloud-images mirror
mirror-1stage.sh
mirror_cloud-images_description
(string) A brief welcome message for the cloud-images mirror
Ubuntu Cloud Images
mirror_cloud-images_name
(string) FQDN of the cloud-images mirror
cloud-images.ubuntu.com
mirror_cloud-images_path
(string) Root location of mirrored files for the cloud-images mirror
/srv/ftp.root/cloud-images
mirror_cloud-images_rsync_auth
(string) Optional rsync authentication details for mirror_cloud-images_source_url
{}
mirror_cloud-images_rsync_log
(boolean) Whether to log rsync requests for the cloud-images mirror
True
mirror_cloud-images_rsync_module
(string) The name of the rsync module for this mirror role
cloud-images
mirror_cloud-images_source_url
(string) The URL the cloud-images mirror will be fetched from
rsync://strix.canonical.com/cloud-images
mirror_cloud-images_sync_time
(string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering
ondemand
mirror_cloud-images_trigger
(string) A base64 string containing the ssh trigger public key
mirror_old-releases_aliases
(string) List of Apache aliases for the old Ubuntu releases mirror
[]
mirror_old-releases_apache_early_extra
(string) A base64 string containing apache configuration options to be included early in the config file
mirror_old-releases_apache_late_extra
(string) A base64 string containing apache configuration options to be included late in the config file
mirror_old-releases_command
(string) The command to use to sync the old Ubuntu releases mirror
mirror-2stage.sh
mirror_old-releases_description
(string) A brief welcome message for the old Ubuntu releases mirror
Ubuntu Old Releases
mirror_old-releases_name
(string) FQDN of the old Ubuntu releases mirror
old-releases.ubuntu.com
mirror_old-releases_path
(string) Root location of mirrored files for the old Ubuntu releases mirror
/srv/ftp.root/old-images
mirror_old-releases_rsync_auth
(string) Optional rsync authentication details for mirror_old-releases_source_url
{}
mirror_old-releases_rsync_log
(boolean) Whether to log rsync requests for the old Ubuntu releases mirror
mirror_old-releases_rsync_module
(string) The name of the rsync module for this mirror role
old-releases
mirror_old-releases_source_url
(string) The URL the old Ubuntu releases mirror will be fetched from
rsync://old-releases.ubuntu.com/old-releases
mirror_old-releases_sync_time
(string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering
ondemand
mirror_old-releases_trigger
(string) A base64 string containing the ssh trigger public key
mirror_ports_aliases
(string) List of Apache aliases for the ports mirror
[]
mirror_ports_apache_early_extra
(string) A base64 string containing apache configuration options to be included early in the config file
mirror_ports_apache_late_extra
(string) A base64 string containing apache configuration options to be included late in the config file
mirror_ports_command
(string) The command to use to sync the ports mirror
mirror-2stage.sh
mirror_ports_description
(string) A brief welcome message for the ports mirror
Ubuntu Ports Archive
mirror_ports_name
(string) FQDN of the ports mirror
ports.ubuntu.com
mirror_ports_path
(string) Root location of mirrored files for the ports mirror
/srv/ftp.root/ubuntu-ports
mirror_ports_rsync_auth
(string) Optional rsync authentication details for mirror_ports_source_url
{}
mirror_ports_rsync_log
(boolean) Whether to log rsync requests for the ports mirror
mirror_ports_rsync_module
(string) The name of the rsync module for this mirror role
ubuntu-ports
mirror_ports_source_url
(string) The URL the ports mirror will be fetched from
rsync://ports.ubuntu.com/ubuntu-ports
mirror_ports_sync_time
(string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering
ondemand
mirror_ports_trigger
(string) A base64 string containing the ssh trigger public key
mirror_releases_aliases
(string) List of Apache aliases for the releases mirror
[]
mirror_releases_apache_early_extra
(string) A base64 string containing apache configuration options to be included early in the config file
mirror_releases_apache_late_extra
(string) A base64 string containing apache configuration options to be included late in the config file
mirror_releases_command
(string) The command to use to sync the releases mirror
mirror-1stage.sh
mirror_releases_description
(string) A brief welcome message for the releases mirror
Ubuntu Releases CD Images
mirror_releases_name
(string) FQDN of the releases mirror
releases.ubuntu.com
mirror_releases_path
(string) Root location of mirrored files for the releases mirror
/srv/ftp.root/releases
mirror_releases_rsync_auth
(string) Optional rsync authentication details for mirror_releases_source_url
{}
mirror_releases_rsync_log
(boolean) Whether to log rsync requests for the releases mirror
True
mirror_releases_rsync_module
(string) The name of the rsync module for this mirror role
releases
mirror_releases_source_url
(string) The URL the releases mirror will be fetched from
rsync://rsync.releases.ubuntu.com/releases
mirror_releases_sync_time
(string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering
ondemand
mirror_releases_trigger
(string) A base64 string containing the ssh trigger public key
mirror_simple-streams_aliases
(string) List of Apache aliases for the simple-streams mirror
[]
mirror_simple-streams_apache_early_extra
(string) A base64 string containing apache configuration options to be included early in the config file
mirror_simple-streams_apache_late_extra
(string) A base64 string containing apache configuration options to be included late in the config file
mirror_simple-streams_command
(string) The command to use to sync the simple-streams mirror
mirror-1stage.sh
mirror_simple-streams_description
(string) A brief welcome message for the simple-streams mirror
Juju Simple Streams Mirror
mirror_simple-streams_name
(string) FQDN of the simple-streams mirror
streams.canonical.com
mirror_simple-streams_path
(string) Root location of mirrored files for the simple-streams mirror
/srv/ftp.root/simple-streams
mirror_simple-streams_rsync_auth
(string) Optional rsync authentication details for mirror_simple-streams_source_url
{}
mirror_simple-streams_rsync_log
(boolean) Whether to log rsync requests for the simple-streams mirror
True
mirror_simple-streams_rsync_module
(string) The name of the rsync module for this mirror role
simple-streams
mirror_simple-streams_source_url
(string) The URL the simple-streams mirror will be fetched from
rsync://strix.canonical.com/simple-streams
mirror_simple-streams_sync_time
(string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering
ondemand
mirror_simple-streams_trigger
(string) A base64 string containing the ssh trigger public key
mirror_ubuntu_aliases
(string) List of Apache aliases for the Ubuntu mirror
[]
mirror_ubuntu_apache_early_extra
(string) A base64 string containing apache configuration options to be included early in the config file
mirror_ubuntu_apache_late_extra
(string) A base64 string containing apache configuration options to be included late in the config file
mirror_ubuntu_command
(string) The command to use to sync the Ubuntu mirror
mirror-2stage.sh
mirror_ubuntu_description
(string) A brief welcome message for the Ubuntu mirror
Ubuntu Archive
mirror_ubuntu_name
(string) FQDN of the Ubuntu mirror
archive.ubuntu.com
mirror_ubuntu_path
(string) Root location of mirrored files for the Ubuntu mirror
/srv/ftp.root/ubuntu
mirror_ubuntu_rsync_auth
(string) Optional rsync authentication details for mirror_ubuntu_source_url
{}
mirror_ubuntu_rsync_log
(boolean) Whether to log rsync requests for the Ubuntu mirror
mirror_ubuntu_rsync_module
(string) The name of the rsync module for this mirror role
ubuntu
mirror_ubuntu_source_url
(string) The URL the Ubuntu mirror will be fetched from
rsync://archive.ubuntu.com/ubuntu
mirror_ubuntu_sync_time
(string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering
ondemand
mirror_ubuntu_trigger
(string) A base64 string containing the ssh trigger public key
mirror_user
(string) Mirror system user id
archvsync
nagios_servicegroup
(string) If set, this is the Nagios servicegroup for alerts. If unset, an appropriate one will be chosen
package_status
(string) The status of service-affecting packages will be set to this value in the dpkg database. Useful valid values are "install" and "hold".
install
role_map
(string) JSON document describing which mirror is enabled on which machine. There are two formats. The first format is simple: {"hostname1": ["cdimage", "ubuntu"], "hostname2": ["releases"]} In the second more complex format each hostname's value is a dictionary rather than a list of roles. For example: { "hostname1": { "releases": { "addresses": ["1.2.3.4"], "https": true }, "ubuntu": null }, "hostname2": [ "releases", "ubuntu" ] } Here, hostname1's ubuntu role will result in an HTTP VirtualHost that listens on *:80 and serves the Ubuntu archive. The releases role will result in HTTP and HTTPS VirtualHosts that listen on 1.2.3.4:80 and 1.2.3.4:443 respectively.
rsync_logdir
(string) Rsync log directory
/var/log/rsyncd
rsync_max_connections
(int) Maximum number of rsync connections allowed
65
rsync_max_connections_per_source
(int) Maximum number of rsync connections allowed per source/IP
5
rsync_motd
(string) Rsync server welcome message text (base64 encoded)
VGhpcyBpcyBhbiBVYnVudHUgbWlycm9yIC0gdHJlYXQgaXQga2luZGx5Cg==
rsync_motdfile
(string) Rsync server welcome message file
/etc/rsyncd/motd
script_dir
(string) Install location for any mirror-related scripts
/srv/ubuntu-mirror/bin