Upgrading RabbitMQ
Overview
This guide covers topics related to RabbitMQ installation upgrades.
- An overview of several common approaches to upgrading RabbitMQ
- RabbitMQ version upgradability, version upgrading from and version upgrading to
- Erlang version requirement
- Plugin compatibility between versions
- Features that do not support in-place upgrade
- Changes in system resource usage and reporting in the new version
- How upgrades of multi-node clusters is different from those with only a single node
- Marking nodes for maintenance
- Recommended upgrade steps
- Caveats
- Handling node restarts in applications
See Release Information to find out what RabbitMQ release series are supported. Release notes of individual releases are published on GitHub.
Important Note on Upgrading to 3.12 and 3.13
RabbitMQ 3.12 requires all previously existing feature flags to be enabled before the upgrade.
The upgrade will fail if you miss this step.
Basics
There are two major upgrade scenarios that are covered in this guide: a single node and a cluster, as well as several most commonly used strategies:
- In-place upgrade where each node is upgraded with its existing on disk data
- Blue-green deployment where a new cluster is created and existing data is migrated to it
- A grow-then-shrink approach where one or more new nodes are added to the cluster, then the old nodes are eventually removed
Below is a brief overview of the common strategies. The rest of the guide covers each strategy in more detail.
The RabbitMQ version upgradability section explains what versions or series can be upgraded to what later series.
In-place Upgrades
This upgrade strategy is recommended
An in-place upgrade usually involves the following steps performed by a deployment tool or manually by an operator. Each step is covered in more detail later in this guide. An intentionally oversimplified list of steps would include:
- Investigate if the current and target versions have an in-place upgrade path: check version upgradability, Erlang version requirements, release notes, features that do not support in-place upgrades, and known caveats
- Check that the node or cluster is in a good state in order to be upgraded: no alarms are in effect, no ongoing queue or stream replica sync operations and the system is otherwise under a reasonable load
- Stop the node
- Upgrade RabbitMQ and, if applicable, Erlang
- Start the node
- Watch monitoring and health check data to assess the health and recovery of the upgraded node or cluster
Rolling upgrades between certain versions are not supported. Full Stop Upgrades and the Blue/Green deployment upgrade strategy cover the two options available for those cases.
Blue-Green Deployment Upgrades
This upgrade strategy is the safest option. It is recommended for environments where a rolling in-place upgrade is not an option for any reason, or extra safety is particularly important
The Blue/Green deployment strategy offers the benefit of making the upgrade process safer at the cost of temporary increasing infrastructure footprint. The safety aspect comes from the fact that the operator can abort an upgrade by switching applications back to the existing cluster.
Grow-then-Shrink Upgrades
This upgrade strategy changes replica identities, can result in massive unnecessary data transfers between nodes, and is only safe with important precautions. Therefore, it is highly recommended against.
A grow-and-shrink upgrade usually involves the following steps. Consider a three node cluster with nodes A, B, and C:
- Add a new node, node D, to the cluster
- Place a new replica of every quorum queue and every stream to the new node using commands such as
rabbitmq-queues grow
- Check that the cluster is in a good state: no alarms are in effect, no ongoing queue or stream replica sync operations and the system is otherwise under a reasonable load
- Remove node A from the cluster using
rabbitmqctl forget_cluster_node
- Add a new node, node E, to the cluster
- Place a new replica of every quorum queue and every stream to the new node using commands such as
rabbitmq-queues grow
- Check that the cluster is in a good state
- Remove node B from the cluster using
rabbitmqctl forget_cluster_node
- and so on
Multiple nodes can be added and removed at a time.
Similarly to rolling upgrades, grow-and-shrink upgrades between certain versions are not supported. Full Stop Upgrades and the Blue/Green deployment upgrade strategy cover the two options available for those cases.
RabbitMQ Version Upgradability
All versions starting with 3.7.27
support rolling upgrades to compatible
later versions using feature flags.
A full cluster stop may be required for feature version upgrades but such cases are rare in modern release series thanks to feature flags.
As a rule of thumb, upgrade to the latest patch release in the target series, and then enable all stable feature flags after all cluster nodes have been upgraded.
When an upgrade jumps multiple release series (e.g. goes from 3.9.x
to 3.13.x
), it may be necessary to perform
one or more intermediate upgrades first.
For each intermediary upgrade, upgrade to the latest patch release in the target series, and then enable all stable feature flags after all cluster nodes have been upgraded.
When an upgrade jumps multiple release series (e.g. goes from 3.9.x
to 3.13.x
), it may be necessary to perform
one or more intermediate upgrades first. For each intermediary upgrade, upgrade to the latest patch release in the target series,
and then enable all stable feature flags
after all cluster nodes have been upgraded.
For example, when upgrading from 3.9.x
to 3.13.x
, it would be necessary to
- First upgrade to the latest 3.10.x patch release
- Then to the latest patch release of 3.11.x
- Then to the latest patch release of 3.12.x
- Finally, upgrade to the latest release in the 3.13.x
Don't forget to enable all stable feature flags every step of the process or the follow step may fail because some feature flags have graduated (became mandatory).
Or consider a The Blue/Green deployment upgrade in one go.
Release Series Upgradeability with Rolling Upgrades
Current release series upgrade compatibility with rolling upgrade:
From | To | Notes |
---|---|---|
3.12.x | 3.13.x | |
3.11.18 | 3.12.x | All feature flags must be enabled before the upgrade |
3.10.x | 3.11.x | Some feature flags must be enabled before the upgrade |
3.9.x | 3.10.x | |
3.8.x | 3.9.x | |
3.7.18 | 3.8.x |
Release Series Upgradeability with Full Stop Upgrades
Current release series upgrade compatibility with full stop upgrade:
From | To | Notes |
---|---|---|
3.12.x | 3.13.x | |
3.11.18 | 3.12.x | All feature flags should be enabled before this upgrade |
3.10.x | 3.11.x | Some feature flags should be enabled before this upgrade |
3.9.x | 3.10.x | |
3.8.x | 3.9.x | |
3.7.27 | 3.9.x | |
3.6.x | 3.8.x | |
3.6.x | 3.7.x | |
3.5.x | 3.7.x | |
=< 3.4.x | 3.6.16 |
Erlang Version Requirements
We recommend that you upgrade Erlang together with RabbitMQ. Please refer to the Erlang Version Requirements guide.
Features that Do Not Support In-place Upgrades
Priority queue on disk data currently cannot be migrated in place between 3.6 and any later series. If an upgrade is performed in place, such queues would start empty (without any messages) after node restart.
To migrate an environment with priority queues and preserve their content (messages), a blue-green upgrade strategy should be used.
Plugin Compatibility Between Versions
Unless otherwise specified in release notes, RabbitMQ plugin API
introduces no breaking changes within a release series (e.g. between
3.12.5
and 3.12.13
). If upgrading to a new minor or major version
(e.g. 3.13.2
), plugin must be upgraded to their versions that support
the new RabbitMQ version series.
In rare cases patch versions of RabbitMQ can break some plugin APIs. Such cases will be documented in the breaking changes section of the release notes document.
Community plugins page contains information on RabbitMQ version support for plugins not included into the RabbitMQ distribution.
Management Plugin Upgrades
RabbitMQ management plugin comes with a Web application that runs in the browser.
After upgrading a cluster, it is highly recommended to clear browser cache, local storage, session storage and cookies for the domain(s) used to access the management UI.
Discontinued Plugins
Sometimes a new feature release drops a plugin or multiple plugins from the distribution.
For example, rabbitmq_management_visualiser
no longer ships with RabbitMQ as of
3.7.0. Such plugins must be disabled before the upgrade.
A node that has a missing plugin enabled will fail to start.
Changes in System Resource Usage and Reporting
Different versions of RabbitMQ can have different resource usage. That should be taken into account before upgrading: make sure there's enough capacity to run the workload with the new version. Always consult with the release notes of all versions between the one currently deployed and the target one in order to find out about changes which could impact your workload and resource usage.
Single Node and Cluster Upgrades
Upgrading a Single Node Installation
Upgrading single node installation is similar to upgrading clusters. Feature flags should be enabled after each upgrade (it's always a good idea to double-check by enabling them before the next upgrade as well - if they are already enabled, it will just do nothing). You should also follow the upgrade compatibility matrix.
Client (application) connections will be dropped when the node stops. Applications need to be prepared to handle this and reconnect.
With some distributions (e.g. the generic binary UNIX) you can install a newer version of RabbitMQ without removing or replacing the old one, which can make upgrade faster. You should make sure the new version uses the same data directory.
RabbitMQ does not support downgrades; it's strongly advised to back node's data directory up before upgrading.
Single node deployments are often local development or test environments. In such cases, if
the upgrade involves jumping multiple release series (eg. from 3.8.15
to 3.13.2
),
it's easier to simply delete everything in the data directory and go directly
to the desired version. Effectively, it's no longer an upgrade but a fresh installation of the new version.
Please note that this process will delete all data in your RabbitMQ (definitions and messages), but this is usually
not a problem in a developement/test environment. The definitions can be preserved using export/import.