postgresql #3

  • By stub
  • stable, edge
Supports: trusty


PostgreSQL is a powerful, open source object-relational database system. It has more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness. It is fully ACID compliant, has full support for foreign keys, joins, views, triggers, and stored procedures (in multiple languages). It includes most SQL:2008 data types, including INTEGER, NUMERIC, BOOLEAN, CHAR, VARCHAR, DATE, INTERVAL, and TIMESTAMP. It also supports storage of binary large objects, including pictures, sounds, or video. It has native programming interfaces for C/C++, Java, .Net, Perl, Python, Ruby, Tcl, ODBC, among others, and exceptional documentation (


excerpt from

PostgreSQL is a powerful, open source object-relational database system. It has more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness. It is fully ACID compliant, has full support for foreign keys, joins, views, triggers, and stored procedures (in multiple languages). It includes most SQL:2008 data types, including INTEGER, NUMERIC, BOOLEAN, CHAR, VARCHAR, DATE, INTERVAL, and TIMESTAMP. It also supports storage of binary large objects, including pictures, sounds, or video. It has native programming interfaces for C/C++, Java, .Net, Perl, Python, Ruby, Tcl, ODBC, among others, and exceptional documentation.

An enterprise class database, PostgreSQL boasts sophisticated features such as Multi-Version Concurrency Control (MVCC), point in time recovery, tablespaces, asynchronous replication, nested transactions (savepoints), online/hot backups, a sophisticated query planner/optimizer, and write ahead logging for fault tolerance. It supports international character sets, multibyte character encodings, Unicode, and it is locale-aware for sorting, case-sensitivity, and formatting. It is highly scalable both in the sheer quantity of data it can manage and in the number of concurrent users it can accommodate. There are active PostgreSQL systems in production environments that manage in excess of 4 terabytes of data.


This charm supports several deployment models:

  • A single service containing one unit. This provides a 'standalone' environment.

  • A service containing multiple units. One unit will be a 'master', and every other unit is a 'hot standby'. The charm sets up and maintains replication for you, using standard PostgreSQL streaming replication.

To setup a single 'standalone' service::

juju deploy postgresql pg-a

Scale Out Usage

To replicate this 'standalone' database to a 'hot standby', turning the existing unit into a 'master'::

juju add-unit pg-a

To deploy a new service containing a 'master' and two 'hot standbys'::

juju deploy -n 2 postgresql pg-b
[ ... wait until units are stable ... ]
juju add-unit pg-b

You can remove units as normal. If the master unit is removed, failover occurs and the most up to date 'hot standby' is promoted to 'master'. The 'db-relation-changed' and 'db-admin-relation-changed' hooks are fired, letting clients adjust::

juju remove-unit pg-b/0

To setup a client using a PostgreSQL database, in this case a vanilla Django installation listening on port 8080::

juju deploy postgresql
juju deploy python-django
juju deploy gunicorn
juju add-relation python-django postgresql:db
juju add-relation python-django gunicorn
juju expose python-django

Known Limitations and Issues

⚠ Due to current limitations with juju, you cannot reliably create a service initially containing more than 2 units (eg. juju deploy -n 3 postgresql). Instead, you must first create a service with 2 units. Once the environment is stable and all the hooks have finished running, you may add more units.

⚠ Do not attempt to relate client charms to a PostgreSQL service containing multiple units unless you know the charm supports a replicated service.

⚠ To host multiple units on a single server, you must use an lxc container.

Interacting with the Postgresql Service

At a minimum, you just need to join a the db relation, and a user and database will be created for you. For more complex environments, you can provide the database name allowing multiple services to share the same database. A client may also wish to defer its setup until the unit name is listed in allowed-units, to avoid attempting to connect to a database before it has been authorized.

The db-admin relation may be used similarly to the db relation. The automatically generated user for db-admin relations is a PostgreSQL superuser.

Database Permissions and Disaster Recovery

⚠ These two topics are entwined, because failing to follow best practice with your database permissions will make your life difficult when you need to recover after failure.

Always set the 'roles' relationship setting when joining a relationship. Always grant permissions to database roles for all database objects your charm creates. Never rely on access permissions given directly to a user, either explicitly or implicitly (such as being the user who created a table). Consider the users you are provided by the PostgreSQL charm as ephemeral. Any rights granted directly to them will be lost if relations are recreated, as the generated usernames will be different. If you don't follow this advice, you will need to manually repair permissions on all your database objects after any of the available recovery mechanisms.

Learn about the SQL GRANT statement in the excellect PostgreSQL reference guide.


If you are using external storage provided by the block storage broker, recovery or a failed unit is simply a matter of ensuring the old unit is fully shut down, and then bringing up a fresh unit with the old external storage mounted. The charm will see the old database there and use it.

If you are unable or do not wish to to simply remount the same filesystem, you can of course copy all the data from the old filesystem to the new one before bringing up the new unit.


PostgreSQL dumps, such as those that can be scheduled in the charm, can be recovered on a new unit by using 'juju ssh' to connect to the new unit and using the standard PostgreSQL pg_restore(1) tool. This new unit must be standalone, or the master unit. Any hot standbys will replicate the recovered data from the master.

You will need to use pg_restore(1) with the --no-owner option, as users that existed in the old service will not exist in the new service.


If you had configured WAL-E, you can recover a WAL-E backup and replay to a point in time of your choosing using the wal-e tool. This will recover the whole database cluster, so all databases will be replaced.

If there are any hot standby units, they will need to be destroyed and recreated after the PITR recovery.

During db-relation-joined

the client service provides:

  • database: Optional. The name of the database to use. The postgresql service will create it if necessary. If your charm sets this, then it must wait until a matching database value is presented on the PostgreSQL side of the relation (ie. relation-get database returns the value you set).
  • roles: Optional. A comma separated list of database roles to grant the database user. Typically these roles will have been granted permissions to access the tables and other database objects. Do not grant permissions directly to juju generated database users, as the charm may revoke them.
  • extensions: Optional. A comma separated list of required postgresql extensions.

During db-relation-changed

the postgresql service provides:

  • host: the host to contact.
  • database: a regular database.
  • port: the port PostgreSQL is listening on.
  • user: a regular user authorized to read the database.
  • password: the password for user.
  • state: 'standalone', 'master' or 'hot standby'.
  • allowed-units: space separated list of allowed clients (unit name). You should check this to determine if you can connect to the database yet.

During db-admin-relation-changed

the postgresql service provides:

  • host: the host to contact
  • port: the port PostgreSQL is listening on
  • user: a created super user
  • password: the password for user
  • state: 'standalone', 'master' or 'hot standby'
  • allowed-units: space separated list of allowed clients (unit name). You should check this to determine if you can connect to the database yet.

During syslog-relation-changed

the postgresql service provides:

  • programname: the syslog 'programname' identifying this unit's PostgreSQL logs.
  • log_line_prefix: the log_line_prefix setting for the PostgreSQL service.

For replicated database support

A PostgreSQL service may contain multiple units (a single master, and optionally one or more hot standbys). The client charm can tell which unit in a relation is the master and which are hot standbys by inspecting the 'state' property on the relation, and it needs to be aware of how many units are in the relation by using the 'relation-list' hook tool.

If there is a single PostgreSQL unit related, the state will be 'standalone'. All database connections of course go to this unit.

If there is more than one PostgreSQL unit related, the client charm must only use units with state set to 'master' or 'hot standby'. The unit with 'master' state can accept read and write connections. The units with 'hot standby' state can accept read-only connections, and any attempted writes will fail. Units with any other state must not be used and should be ignored ('standalone' units are new units joining the service that are not yet setup, and 'failover' state will occur when the master unit is being shutdown and a new master is being elected).

The client charm needs to watch for state changes in its relation-changed hook. New units may be added to a single unit service, and the client charm must stop using existing 'standalone' unit and wait for 'master' and 'hot standby' units to appear. Units may be removed, possibly causing a 'hot standby' unit to be promoted to a master, or even having the service revert to a single 'standalone' unit.

Example client hooks


import sys
from charmhelpers.core.hookenv import (
    Hooks, config, relation_set, relation_get,
    local_unit, related_units, remote_unit)

hooks = Hooks()
hook = hooks.hook

def db_relation_joined():
    relation_set('database', config('database'))  # Explicit database name
    relation_set('roles', 'reporting,standard')  # DB roles required

@hook('db-relation-changed', 'db-relation-departed')
def db_relation_changed():
    # Rather than try to merge in just this particular database
    # connection that triggered the hook into our existing connections,
    # it is easier to iterate over all active related databases and
    # reset the entire list of connections.
    conn_str_tmpl = "dbname={dbname} user={user} host={host} port={port}"
    master_conn_str = None
    slave_conn_strs = []
    for db_unit in related_units():
        if relation_get('database', db_unit) != config('database'):
            continue  # Not yet acknowledged requested database name.

        allowed_units = relation_get('allowed-units') or ''  # May be None
        if local_unit() not in allowed_units.split():
            continue  # Not yet authorized.

        conn_str = conn_str_tmpl.format(**relation_get(unit=db_unit)
        remote_state = relation_get('state', db_unit)

        if remote_state == 'standalone' and len(active_db_units) == 1:
            master_conn_str = conn_str
        elif relation_state == 'master':
            master_conn_str = conn_str
        elif relation_state == 'hot standby':

    update_my_db_config(master=master_conn_str, slaves=slave_conn_strs)

if __name__ == '__main__':

Upgrade-charm hook notes

The PostgreSQL charm has deprecated volume-map and volume-ephemeral-storage configuration options in favor of using the storage subordinate charm for general external storage management. If the installation being upgraded is using these deprecated options, there are a couple of manual steps necessary to finish migration and continue using the current external volumes. Even though all data will remain intact, and PostgreSQL service will continue running, the upgrade-charm hook will intentionally fail and exit 1 as well to raise awareness of the manual procedure which will also be documented in the juju logs on the PostgreSQL units.

The following steps must be additionally performed to continue using external volume maps for the PostgreSQL units once juju upgrade-charm is run from the command line: 1. cat > storage.cfg <<EOF storage: provider:block-storage-broker root: /srv/data volume_map: "{postgresql/0: your-vol-id, postgresql/1: your-2nd-vol-id}" EOF 2. juju deploy --config storage.cfg storage 3. juju deploy block-storage-broker 4. juju add-relation block-storage-broker storage 5. juju resolved --retry postgresql/0 # for each postgresql unit running 6. juju add-relation postgresql storage

Point In Time Recovery

The PostgreSQL charm has experimental support for log shipping and point in time recovery. This feature uses the wal-e2 tool, and requires access to Amazon S3, Microsoft Azure Block Storage or Swift. This feature is flagged as experimental because it has only been tested with Swift, and not yet been tested under load. It also may require some API changes, particularly on how authentication credentials are accessed when a standard emerges. The charm can be configured to perform regular filesystem backups and ship WAL files to the object store. Hot standbys will make use of the archived WAL files, allowing them to resync after extended netsplits or even let you turn off streaming replication entirely.

With a base backup and the WAL archive you can perform point in time recovery, but this is still a manual process and the charm does not yet help you do it. The simplest approach would be to create a new PostgreSQL service containing a single unit, 'juju ssh' in and use wal-e to replace the database after shutting it down, create a recovery.conf to replay the archived WAL files using wal-e, restart the database and wait for it to recover. Once recovered, new hot standby units can be added and client services related to the new database service.

To enable the experimental wal-e support with Swift, you will need to use Ubuntu 14.04 (Trusty), and set the service configuration settings similar to the following::

    wal_e_storage_uri: swift://mycontainer
    os_username: my_swift_username
    os_password: my_swift_password
    os_tenant_name: my_tenant_name
    install_sources: |
        - ppa:stub/pgcharm
        - cloud:icehouse

Contact Information



(string) A comma-separated list of IP Addresses (or single IP) admin tools like pgAdmin3 will connect from, this is most useful for developers running juju in local mode who need to connect tools like pgAdmin to a postgres. The IP addresses added here will be included in the pg_hba.conf file allowing ip connections to all databases on the server from the given using md5 password encryption.
(int) An advisory lock key used internally by the charm. You do not need to change it unless it happens to conflict with an advisory lock key being used by your applications.
(string) Command used to archive WAL files when archive_mode is set and wal_level > minimal.
(boolean) Enable archiving of WAL files using the command specified by archive_command. If archive_mode is enabled and archive_command not set, then archiving is deferred until archive_command is set and the WAL files will accumulate.
(boolean) Autovacuum should almost always be running. If you want to turn this off, you are probably following out of date documentation.
(float) Fraction of table size before analyze
(int) min number of row updates before analyze
(string) Default vacuum cost delay for autovacuum, in milliseconds; -1 means use vacuum_cost_delay
(float) Fraction of table size before vacuum
(string) EXPERIMENTAL. Amazon AWS access key id.
(string) EXPERIMENTAL. Amazon AWS secret access key.
(string) Directory to place backups in
(int) Number of recent backups to retain.
(string) Cron-formatted schedule for database backups.
13 4 * * *
(float) checkpoint target duration time, as a fraction of checkpoint_timeout. Range [0.0, 1.0].
(int) in logfile segments, min 1, 16MB each. Ignored unless 'performance_tuning' is set to 'manual'.
(string) Maximum time between automatic WAL checkpoints. range '30s-1h'. If left empty, the default postgresql value will be used.
(string) Name of the cluster we want to install the DBs into
(int) Sets the from_collapse_limit and join_collapse_limit query planner options, controlling the maximum number of tables that can be joined before the turns off the table collapse query optimization.
(int) Sets the default statistics target for table columns without a column-specific target set via ALTER TABLE SET STATISTICS. Leave unchanged to use the server default, which in recent releases is 100. Ignored unless 'performance_tuning' is 'manual'. Larger values increase the time needed to do ANALYZE, but might improve the quality of the planner's estimates.
(string) Path to a dumpfile to load into DB when service is initiated.
(string) Effective cache size is an estimate of how much memory is available for disk caching within the database. (50% to 75% of system memory). This string should be of the format '###MB'. Ignored unless 'performance_tuning' is set to 'manual'.
(string) Default encoding used to store text in this service. Can only be set when deploying the first unit of a service.
(string) Extra packages to install on the postgresql service units.
(string) DEPRECATED & IGNORED. Use install_sources and install_keys.
(string) A comma separated extra pg_hba.conf auth rules. This will be written to the pg_hba.conf file, one line per rule. Note that this should not be needed as db relations already create those rules the right way. Only use this if you really need too (e.g. on a development environment), or are connecting juju managed databases to external managed systems, or configuring replication between unrelated PostgreSQL services using the manual_replication option.
(boolean) Turns forced synchronization on/off. If fsync is turned off, database failures are likely to involve database corruption and require recreating the unit
(boolean) Recover from partial page writes.
(boolean) Hot standby or warm standby. When True, queries can be run against the database when in recovery or standby mode (ie. replicated). Overridden when service contains multiple units.
(boolean) Hot standby feedback, informing a master about in progress transactions on a streaming hot standby and allowing the master to defer cleanup and avoid query cancelations on the hot standby.
(string) List of signing keys for install_sources package sources, per charmhelpers standard. YAML format.
(string) List of extra package sources, per charm-helpers standard. YAML format.
(int) Total amount of shared memory available, in bytes.
(int) The maximum size, in bytes, of a shared memory segment.
(string) IP to listen on
(int) Port to listen on. Default is automatically assigned.
(string) Locale of service, defining language, default collation order, and default formatting of numbers, currency, dates & times. Can only be set when deploying the first unit of a service.
(int) -1 disables, 0 logs all actions and their durations, > 0 logs only actions running at least this number of milliseconds.
(boolean) Log checkpoints
(boolean) Log connections
(boolean) Log disconnections
(string) special values: %a = application name %u = user name %d = database name %r = remote host and port %h = remote host %p = process ID %t = timestamp without milliseconds %m = timestamp with milliseconds %i = command tag %e = SQL state %c = session ID %l = session line number %s = session start timestamp %v = virtual transaction ID %x = transaction ID (0 if none) %q = stop here in non-session processes %% = '%' e.g. '<%u%%%d> '
%t [%p]: [%l-1] db=%d,user=%u
(boolean) log lock waits >= deadlock_timeout
(int) -1 is disabled, 0 logs all statements and their durations, > 0 logs only statements running at least this number of milliseconds
(string) Log creation of temporary files larger than the threshold. -1 disables the feature, 0 logs all temporary files, or specify the threshold size with an optional unit (eg. "512KB", default unit is kilobytes).
(string) Log timezone
(string) Maintenance working memory. Ignored unless 'performance_tuning' is set to 'manual'.
(boolean) Enable or disable charm managed replication. When manual_replication is True, the operator is responsible for maintaining recovery.conf and performing any necessary database mirroring. The charm will still advertise the unit as standalone, master or hot standby to relations based on whether the system is in recovery mode or not. Note that this option makes it possible to create a PostgreSQL service with multiple master units, which is probably a very silly thing to do.
(int) Maximum number of connections to allow to the PG database
(int) Maximum number of prepared two phase commit transactions, waiting to be committed. Defaults to 0. as using two phase commit without a process to monitor and resolve lost transactions is dangerous.
(int) Maximum number of hot standbys that can connect using streaming replication. Set this to the expected maximum number of hot standby units to avoid unnecessary blocking and database restarts. Overridden by juju if necessary.
(string) Prefix for metrics. Special value $UNIT can be used to include the name of the unit in the prefix.
(int) Period for metrics cron job to run in minutes
(string) Destination for statsd-format metrics, format "host:port". If not present and valid, metrics disabled.
(string) Used by the nrpe-external-master subordinate charm. A comma-separated list of servicegroups to include along with nagios_context when generating nagios service check configs. This is useful for nagios installations where servicegroups are used to apply special treatment to particular checks.
(string) Used by the nrpe-external-master subordinate charm. A string that will be prepended to instance name to set the host name in nagios. So for instance the hostname would be something like: juju-postgresql-0 If you're running multiple environments with the same services in them this allows you to differentiate between them.
(string) EXPERIMENTAL. OpenStack Swift authentication URL.
(string) EXPERIMENTAL. OpenStack Swift password.
(string) EXPERIMENTAL. OpenStack Swift tenant name.
(string) EXPERIMENTAL. OpenStack Swift username.
(string) The status of service-affecting packages will be set to this value in the dpkg database. Useful valid values are "install" and "hold".
(string) Possible values here are "manual", "DW" (data warehouse), "OLTP" (online transaction processing), "Web" (web application), "Desktop" or "Mixed". When this is set to a value other than "manual", the charm invokes the pgtune tool to tune a number of performance parameters based on the specified load type. pgtune gathers information about the node on which you are deployed and tries to make intelligent guesses about what tuning parameters to set based on available RAM and CPU under the assumption that it's the only significant service running on this node.
(boolean) Enable the PostgreSQL Global Development Group APT repository ( This package source provides official PostgreSQL packages for Ubuntu LTS releases beyond those provided by the main Ubuntu archive.
(float) Random page cost
(int) Value of wal_keep_segments used when this service is replicated. This setting only exists to provide a sane default when replication is requested (so it doesn't fail) and nobody bothered to change the wal_keep_segments setting.
(string) Comma separated list of schema names for the default SQL search path.
(string) The amount of memory the database server uses for shared memory buffers. This string should be of the format '###MB'. Ignored unless 'performance_tuning' is set to 'manual'.
(string) Whether PostgreSQL should talk SSL
(boolean) Standard conforming strings
(boolean) Enable streaming replication. Normally, streaming replication is always used, and any log shipping configured is used as a fallback. Turning this off without configuring log shipping is an error.
(int) EXPERIMENTAL. Number of recent base backups to retain. You need enough space in Swift for this many backups plus one more, as an old backup will only be removed after a new one has been successfully made to replace it.
(string) EXPERIMENTAL. Cron-formatted schedule for SwiftWAL database backups.
(string) EXPERIMENTAL. Swift container prefix for SwiftWAL to use. Must be set if any SwiftWAL features are enabled. This will become a simple swiftwal_container config item when proper leader election is implemented in juju.
(boolean) EXPERIMENTAL. Archive WAL files into Swift. If swiftwal_backup_schedule is set, allows point-in-time recovery and WAL files are removed automatically with old backups. If swiftwal_backup_schedule is not set then WAL files are never removed. Enabling this option will override the archive_mode and archive_command settings.
(boolean) Immediate fsync after commit.
(string) The maximum number of temporary buffers used by each database session.
(string) Version of PostgreSQL that we want to install. Supported versions are "9.1", "9.2", "9.3". The default version for the deployed Ubuntu release is used when the version is not specified.
(string) EXPERIMENTAL. Windows Azure access key.
(string) EXPERIMENTAL. Windows Azure account name.
(string) min 32kB, -1 sets based on shared_buffers (change requires restart). Ignored unless 'performance_tuning' is set to 'manual'.
(int) EXPERIMENTAL. Number of recent base backups and WAL files to retain. You need enough space for this many backups plus one more, as an old backup will only be removed after a new one has been successfully made to replace it.
(string) EXPERIMENTAL. Cron-formatted schedule for WAL-E database backups. If wal_e_backup_schedule is unset, WAL files will never be removed from WAL-E storage.
13 0 * * *
(string) EXPERIMENTAL. Specify storage to be used by WAL-E. Every PostgreSQL service must use a unique URI. Backups will be unrecoverable if it is not unique. The URI's scheme must be one of 'swift' (OpenStack Swift), 's3' (Amazon AWS) or 'wabs' (Windows Azure). For example: 'swift://some-container/directory/or/whatever' 's3://some-bucket/directory/or/whatever' 'wabs://some-bucket/directory/or/whatever' Setting the wal_e_storage_uri enables regular WAL-E filesystem level backups (per wal_e_backup_schedule), and log shipping to the configured storage. Point-in-time recovery becomes possible, as is disabling the streaming_replication configuration item and relying solely on log shipping for replication.
(int) Number of old WAL files to keep, providing a larger buffer for streaming hot standbys to catch up from when lagged. Each WAL file is 16MB in size. The WAL files are the buffer of how far a hot standby can lag behind the master, and replication fails if this buffer is overrun. When this service is replicated, the larger value of wal_keep_segments and replicated_wal_keep_segments is used.
(string) 'minimal', 'archive' or 'hot_standby'. Defines how much information is written to the WAL. Set to 'minimal' for stand alone databases and 'hot_standby' for replicated setups. Overridden by juju when replication is used.
(string) Working Memory. Ignored unless 'performance_tuning' is set to 'manual'.