Maintaining OpenStack Software Updates
Introduction
Software in the OpenStack ecosystem evolves over time, either through new feature additions, bug fixes, or when vulnerabilities are patched. Part of operating an OpenStack cloud involves maintaining its software through updates. In this guide, we point out the sections in an OpenMetal cloud where software updates occur and explain best practices when performing updates.
Prerequisites
Before getting started, it is beneficial to have experience with Docker.
To follow this guide, ensure you have:
- Root access to your cloud's control plane
Software that can be Updated
The software of an OpenMetal cloud that can be updated include each hardware node's package manager and the Kolla Ansible Docker images. Ceph updates are handled through the node's package manager.
Updating Kolla Ansible Docker Images
Refer to Kolla Ansible's documentation for more information regarding updating images.
Performing Package Manager Updates
Migrate Workload
Package manager updates requiring a server reboot to an OpenMetal control plane node can be disruptive to any workload running on it. Prior to performing disruptive actions, it may be possible to migrate instances another node running the Compute service. For information on how to migrate instances, see OpenStack Nova's documentation.
Update One Node at a Time
While performing package manager updates, ensure updates occur successfully for one hardware node before updating another node.
Disable Docker
Before updating the package manager, ensure the Docker socket and service within SystemD are stopped and disabled. For example:
systemctl disable docker.socket
systemctl stop docker.socket
systemctl disable docker.service
systemctl stop docker.service
Upgrade Host OS Packages
After verifying the Docker socket and service are stopped, perform the package manager updates.:
dnf upgrade
Determine Reboot Need
Once package manager completes, check if a reboot is required with
dnf-utils needs-restarting
and the reboot hint flag (-r):
$ needs-restarting -r
Core libraries or services have been updated since boot-up:
* kernel
* systemd
Reboot is required to fully utilize these updates.
More information: https://access.redhat.com/solutions/27943
$
Ceph Maintenance
Warning! This step is optional and only required if the node needs to be rebooted.
Prior to reboot, if the node is part of the Ceph cluster automatic OSD removal and data rebalance should be temporarily suspended. To do so, perform:
ceph osd set noout
ceph osd set norebalance
This will reduce rebuild time and help ensure the node rejoins the cluster automatically.
Once the node reboots and a healthy Ceph cluster is confirmed, these parameters must be unset. To unset this configuration, perform:
ceph osd unset noout
ceph osd unset norebalance
Reboot if Required
Reboot the node if required:
shutdown -r now
Verify Successful Reboot
When the node comes back online, SSH into it to verify the OpenStack Docker containers have started. Additionally, if this node was part of the Ceph cluster, check Ceph's cluster status.
To verify the Docker containers have started, use docker ps
. You
should see a number of Docker containers running. Under the STATUS
column, each container should reflect the status Up
.
For example:
[root@smiling-pelican ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6f7590bc2191 harbor.imhadmin.net/kolla/centos-binary-telegraf:victoria "dumb-init --single-…" 20 hours ago Restarting (1) 14 seconds ago telegraf
67a4d47e8c78 harbor.imhadmin.net/kolla/centos-binary-watcher-api:victoria "dumb-init --single-…" 3 days ago Up 6 minutes watcher_api
af815b1dcb5d harbor.imhadmin.net/kolla/centos-binary-watcher-engine:victoria "dumb-init --single-…" 3 days ago Up 6 minutes watcher_engine
a52ab61933ac harbor.imhadmin.net/kolla/centos-binary-watcher-applier:victoria "dumb-init --single-…" 3 days ago Up 6 minutes watcher_applier
[...output truncated...]
Next, if this node is part of a Ceph cluster, check Ceph's status using
ceph status
.
For example:
[root@smiling-pelican ~]# ceph status
cluster:
id: 06bf4555-7c0c-4b96-a3b7-502bf8f6f213
health: HEALTH_OK
[...output truncated...]
The above output shows the status as HEALTH_OK
, indicating the Ceph
cluster is healthy. Ceph is naturally resilient and should recover from
a node being rebooted.