Page MenuHomePhabricator
Create Task
Maniphest T428445

Upgrade Data Platform ceph clusters to version 19 - squid
Open, HighPublic
Actions

Description

Currently the Ceph clusters supporting the Data Platform are running major version 18, which is referred to as reef.
https://docs.ceph.com/en/latest/releases/reef/

btullis@cephosd1001:~$ ceph --version
ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)

As per the release schedule, reef has already had its final point release and is now considered EoL:
https://docs.ceph.com/en/latest/releases/

The next major release is version 19. which is referred to as squid
https://docs.ceph.com/en/latest/releases/squid/

We should upgrade our clusters to reef.

One new feature is particularly relevant to us, which is the new support of the IAM API in the Rados Gateway component.
https://docs.ceph.com/en/latest/radosgw/iam/#ceph-object-gateway-iam-api

We do not use it yet, but we expect it to be part of the security model being defined in:
T428255: Define a comprehensive security model for the Data Platform S3/Ceph services

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
ceph: Use the squid package repository on cephosd2003Btullisoperations/puppetproduction+2 -0
ceph: Use the squid package repository on cephosd2002Btullisoperations/puppetproduction+2 -0
ceph: Use the squid package repository on cephosd2001Btullisoperations/puppetproduction+2 -0
ceph: Stop managing CephX key material in codfwBtullisoperations/puppetproduction+3 -0
ceph: Allow CephX key material to be unmanagedBtullisoperations/puppetproduction+88 -7
Add the upstream ceph-squid repository bookworm-wikimediaBtullisoperations/puppetproduction+12 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

BTullis created this task.Jun 8 2026, 12:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 8 2026, 12:55 PM
BTullis added a parent task: T428255: Define a comprehensive security model for the Data Platform S3/Ceph services.Jun 8 2026, 12:55 PM
BTullis moved this task from Incoming to Software Upgrades on the Data-Platform-SRE board.Jun 8 2026, 1:22 PM
BTullis added a project: Ceph.Jun 9 2026, 10:15 AM
BTullis mentioned this in T435470: Enable OIDC authentication and IAM authorization on the Data Platform Ceph clusters.Thu, Aug 20, 11:59 AM
BTullis added a parent task: T435470: Enable OIDC authentication and IAM authorization on the Data Platform Ceph clusters.Thu, Aug 20, 3:36 PM
BTullis removed a parent task: T428255: Define a comprehensive security model for the Data Platform S3/Ceph services.Thu, Aug 20, 4:03 PM
BTullis mentioned this in T435590: Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers.Fri, Aug 21, 9:58 AM
BTullis edited projects, added: Data-Platform-SRE (2026-08-07 - 2026-08-28); removed: Data-Platform-SRE.Fri, Aug 21, 12:48 PM
Gehel edited projects, added: Data-Platform-SRE (2026-08-28 - 2026-09-18); removed: Data-Platform-SRE (2026-08-07 - 2026-08-28).Fri, Aug 28, 1:01 PM
BTullis claimed this task.Thu, Sep 3, 11:34 AM
BTullis triaged this task as High priority.
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-08-28 - 2026-09-18) board.
gerritbot added a comment.Fri, Sep 4, 12:36 PM

Change #1335835 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add the upstream ceph-squid repository bookworm-wikimedia

https://gerrit.wikimedia.org/r/1335835

gerritbot added a project: Patch-For-Review.Fri, Sep 4, 12:36 PM
bking subscribed.Fri, Sep 4, 4:27 PM
gerritbot added a comment.Mon, Sep 7, 9:24 AM

Change #1335835 merged by Btullis:

[operations/puppet@production] Add the upstream ceph-squid repository bookworm-wikimedia

https://gerrit.wikimedia.org/r/1335835

Maintenance_bot removed a project: Patch-For-Review.Mon, Sep 7, 9:30 AM
BTullis added a comment.EditedMon, Sep 7, 10:01 AM

I'm reviewing the notable changes and critical upgrade steps for this upgrade.

In particular, two seem relevant:

The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. This is the first time Ceph has introduced a new key type for CephX credentials. Therefore, a new procedure exists for upgrading and rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph cluster, the operator should familiarize themselves with the procedure documented at Upgrading and Rotating CephX Keys.

As part of upgrading a Ceph cluster, six new health warnings and errors will be generated. This is normal. As part of the process to upgrade/rotate entity keys, you will address the warnings and errors as you go.

This is highly relevant to us, as we currently have known issues around how the puppet management of our CephX key provision works:
See: T399594: Proposed improvement: Manage CephX users via exported/collected Puppet resources for more details.

I think that we just need to work out the procedure for this one as we go along.

The second critical upgrade step is this one:

Patched RGW now checks any new x-amz-* headers not included in the signature are present in the SigV4 request. If this is the case then the request is rejected.

I think that we just need to be aware of this in terms of Airflow logs and PostgreSQL backups/WAL writes, but I am not aware of any changes we will need to make.

BTullis added a comment.Mon, Sep 7, 10:23 AM

Oh, this is also interesting. From here: https://docs.ceph.com/en/latest/releases/squid/#upgrading-non-cephadm-clusters

systemd unit file names have changed to include the cluster fsid.

The example they give is:

ceph-6ce0347c-314a-11ee-9b52-000af7995d6c@mon.f28-h21-000-r630.service

This might well be at-odds with the service management that we have in place.
e.g. in mon.pp we manage a systemd service.

service { "ceph-mon@${facts['networking']['hostname']}":
    ensure  => running,
    enable  => true,
    require => [Exec['ceph-mon-mkfs'], File['/etc/ceph/ceph.conf']],
}

In mgr.pp we manage a systemd service.

service { "ceph-mgr@${facts['networking']['hostname']}":
    ensure    => running,
    enable    => true,
    require   => Ceph::Auth::Keyring[$client],
    subscribe => File['/etc/ceph/ceph.conf'],
}

In mds.pp we manage a systemd service.

service { "ceph-mds@${facts['networking']['hostname']}":
    ensure    => running,
    enable    => true,
    require   => Ceph::Auth::Keyring[$client],
    subscribe => File['/etc/ceph/ceph.conf'],
}

In radosgw.pp we manage a systemd service.

service { 'ceph-radosgw@radosgw':
    ensure    => running,
    enable    => true,
    subscribe => File['/etc/ceph/ceph.conf'],
}

We don't directly manage the services for OSDs in osd.pp or osds.pp,

We should be very careful not to duplicate the systemd services. i.e. if we have one that is managed by the ceph packages themselves, plus another that is managed by puppet, this may be problematic.

Perhaps we need some conditionals within the puppet code to allow puppet to manage the correctly named services. We need to be careful because this puppet module is also applied to the WMCS clusters, which are not being upgraded at the same time.

gerritbot added a comment.Mon, Sep 7, 1:18 PM

Change #1337592 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] ceph: Allow CephX key material to be unmanaged

https://gerrit.wikimedia.org/r/1337592

gerritbot added a project: Patch-For-Review.Mon, Sep 7, 1:18 PM

Change #1337593 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] ceph: Stop managing CephX key material in codfw

https://gerrit.wikimedia.org/r/1337593

gerritbot added a comment.Mon, Sep 7, 1:18 PM

Change #1337594 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] ceph: Use the squid package repository on cephosd2001

https://gerrit.wikimedia.org/r/1337594

gerritbot added a comment.Mon, Sep 7, 1:18 PM

Change #1337595 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] ceph: Use the squid package repository on cephosd2002

https://gerrit.wikimedia.org/r/1337595

gerritbot added a comment.Mon, Sep 7, 1:18 PM

Change #1337596 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] ceph: Use the squid package repository on cephosd2003

https://gerrit.wikimedia.org/r/1337596

gerritbot added a comment.Mon, Sep 7, 1:45 PM

Change #1337592 merged by Btullis:

[operations/puppet@production] ceph: Allow CephX key material to be unmanaged

https://gerrit.wikimedia.org/r/1337592

gerritbot added a comment.Mon, Sep 7, 1:54 PM

Change #1337593 merged by Btullis:

[operations/puppet@production] ceph: Stop managing CephX key material in codfw

https://gerrit.wikimedia.org/r/1337593

gerritbot added a comment.Mon, Sep 7, 2:05 PM

Change #1337594 merged by Btullis:

[operations/puppet@production] ceph: Use the squid package repository on cephosd2001

https://gerrit.wikimedia.org/r/1337594

BTullis mentioned this in T399594: Proposed improvement: Manage CephX users via exported/collected Puppet resources.Mon, Sep 7, 2:24 PM
gerritbot added a comment.Mon, Sep 7, 2:24 PM

Change #1337595 merged by Btullis:

[operations/puppet@production] ceph: Use the squid package repository on cephosd2002

https://gerrit.wikimedia.org/r/1337595

gerritbot added a comment.Mon, Sep 7, 2:25 PM

Change #1337596 merged by Btullis:

[operations/puppet@production] ceph: Use the squid package repository on cephosd2003

https://gerrit.wikimedia.org/r/1337596

Maintenance_bot removed a project: Patch-For-Review.Mon, Sep 7, 2:30 PM
BTullis added a comment.Mon, Sep 7, 2:47 PM

I don't think that the systemd unit name change is going to bite us, after all.
I still see the existing names of systemd units in https://github.com/ceph/ceph/tree/v19.2.0/systemd so I think that this actually affects cephadm clusters. Maybe it's a documentation bug.

For the password rotation issue, I think that this is a bit bigger and will force us to have a look at a couple of outstanding authentication issues.

I added a switch to allow us to disable puppet's distribution of keys, at a cluster level, host level, and at a keyring level. This means that we will be able to disable the keyfile handling for codfw without disabling puppet. Then when we have done the upgrade, we will be able to update puppet's private repo in one go and distribute the keys then restart the services on a host-by-host basis.

I think that I have a fix for that, but it made me look again at T399594: Proposed improvement: Manage CephX users via exported/collected Puppet resources and eventually decline the ticket.
So I'm now ready to go with pulling the updated packages on the three cephosd200[1-3] hosts.

btullis@cephosd2001:~$ apt list --upgradable
Listing... Done
bird2/bookworm-wikimedia 2.18.2-1~wmf12u1 amd64 [upgradable from: 2.18-1~wmf12u2]
ca-certificates/oldstable-security 20250419~deb12u1 all [upgradable from: 20230311+deb12u1]
ceph-base/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
ceph-common/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
ceph-mds/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
ceph-mgr-modules-core/bookworm-wikimedia 19.2.6-1bookworm all [upgradable from: 18.2.8-1~bpo12+1]
ceph-mgr/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
ceph-mon/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
ceph-osd/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
ceph-volume/bookworm-wikimedia 19.2.6-1bookworm all [upgradable from: 18.2.8-1~bpo12+1]
libcephfs2/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
libdbi-perl/oldstable-security 1.643-4+deb12u2 amd64 [upgradable from: 1.643-4+deb12u1]
libexpat1/oldstable-security 2.5.0-1+deb12u3 amd64 [upgradable from: 2.5.0-1+deb12u2]
liblzma5/oldstable 5.4.1-1+deb12u1 amd64 [upgradable from: 5.4.1-1]
libnet-dns-perl/oldstable-security 1.36-1+deb12u1 all [upgradable from: 1.36-1]
libpcre2-16-0/oldstable-security 10.42-1+deb12u1 amd64 [upgradable from: 10.42-1]
libpcre2-8-0/oldstable-security 10.42-1+deb12u1 amd64 [upgradable from: 10.42-1]
librabbitmq4/oldstable-security 0.11.0-1+deb12u3 amd64 [upgradable from: 0.11.0-1+deb12u2]
librados2/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
libradosstriper1/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
librbd1/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
librgw2/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
libsqlite3-mod-ceph/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
libssh2-1/oldstable-security 1.10.0-3+deb12u1 amd64 [upgradable from: 1.10.0-3+b1]
linux-libc-dev/oldstable-security 6.1.180-1 amd64 [upgradable from: 6.1.174-1]
linux-perf/oldstable-security 6.1.180-1 amd64 [upgradable from: 6.1.174-1]
python3-ceph-argparse/bookworm-wikimedia 19.2.6-1bookworm all [upgradable from: 18.2.8-1~bpo12+1]
python3-ceph-common/bookworm-wikimedia 19.2.6-1bookworm all [upgradable from: 18.2.8-1~bpo12+1]
python3-cephfs/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
python3-httplib2/oldstable-security 0.20.4-3+deb12u1 all [upgradable from: 0.20.4-3]
python3-rados/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
python3-rbd/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
python3-rgw/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
python3-urllib3/oldstable,oldstable-security 1.26.12-1+deb12u4 all [upgradable from: 1.26.12-1+deb12u3]
radosgw/bookworm-wikimedia 19.2.6-1bookworm amd64 [upgradable from: 18.2.8-1~bpo12+1]
xz-utils/oldstable 5.4.1-1+deb12u1 amd64 [upgradable from: 5.4.1-1]
BTullis added a comment.Mon, Sep 7, 3:01 PM

I have created T437233: ceph: Fix the CephX key guard so that Puppet can correctly update the key material to fix the puppet management of CephX key data.

BTullis added a comment.Tue, Sep 8, 11:15 AM

The procedure set out here https://docs.ceph.com/en/latest/releases/squid/#upgrading-non-cephadm-clusters is broadly as follows:

  • Set the cluster into noout mode with ceph osd set noout
  • Upgrade monitors by installing the new packages and restarting the monitor daemons.
  • Once all monitors are up, verify that the monitor upgrade is complete by looking for the squid string in the mon map.
  • Upgrade ceph-mgr daemons by installing the new packages and restarting all manager daemons.
  • Verify the ceph-mgr daemons are running by checking ceph -s.
  • Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD hosts.
  • Upgrade all CephFS MDS daemons. For each CephFS file system,
    • Disable standby_replay: ceph fs set <fs_name> allow_standby_replay false
    • Reduce the number of ranks to 1. (Make note of the original number of MDS daemons first if you plan to restore it later.) n.b. I think that we only have one rank each, anyway
    • Take all standby MDS daemons offline on the appropriate hosts with; systemctl stop ceph-mds@<daemon_name>
    • Confirm that only one MDS is online and is rank 0 for your FS ceph fs status
    • Upgrade the last remaining MDS daemon by installing the new packages and restarting the daemon
    • Restart all standby MDS daemons that were taken offline
    • Restore the original value of max_mds for the volume n.b. Again, not necessary for us in our current config
  • Upgrade all radosgw daemons by upgrading packages and restarting daemons on all hosts
  • Complete the upgrade by disallowing pre-Squid OSDs and enabling all new Squid-only functionality
    • ceph osd require-osd-release squid
  • If you set noout at the beginning, be sure to clear it with ceph osd unset noout
BTullis added a comment.Tue, Sep 8, 11:18 AM

Since ours is a co-located installation and all daemons are on the same hosts, it is not possible to defer the installation of the packages. When I try to install the ceph-mon package, all of the other relevant daemons are upgraded, too.

btullis@cephosd2001:~$ sudo apt-get install --dry-run ceph-mon
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfmt9
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  ceph-base ceph-common ceph-mds ceph-mgr ceph-mgr-modules-core ceph-osd ceph-volume libcephfs2 liblua5.3-dev libncurses-dev librados2 libradosstriper1 librbd1 libreadline-dev librgw2 libsqlite3-mod-ceph
  lua-any lua5.1 luarocks python3-ceph-argparse python3-ceph-common python3-cephfs python3-rados python3-rbd python3-rgw radosgw unzip zip
Suggested packages:
  python3-influxdb ncurses-doc readline-doc
Recommended packages:
  btrfs-tools ceph-fuse ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-cephadm pkg-config libtool-bin lua-sec
The following NEW packages will be installed:
  liblua5.3-dev libncurses-dev libreadline-dev lua-any lua5.1 luarocks unzip zip
The following packages will be upgraded:
  ceph-base ceph-common ceph-mds ceph-mgr ceph-mgr-modules-core ceph-mon ceph-osd ceph-volume libcephfs2 librados2 libradosstriper1 librbd1 librgw2 libsqlite3-mod-ceph python3-ceph-argparse python3-ceph-common
  python3-cephfs python3-rados python3-rbd python3-rgw radosgw
21 upgraded, 8 newly installed, 0 to remove and 15 not upgraded.
Inst ceph-mgr [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst ceph-mgr-modules-core [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [all]) []
Inst ceph-volume [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [all]) []
Inst ceph-osd [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst ceph-mon [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst ceph-mds [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst ceph-base [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst radosgw [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst ceph-common [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst python3-cephfs [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst libcephfs2 [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst python3-rgw [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst python3-rados [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst python3-ceph-argparse [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [all]) []
Inst libsqlite3-mod-ceph [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst librbd1 [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst libradosstriper1 [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst librgw2 [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst librados2 [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst libncurses-dev (6.4-4 Debian:12.15/oldstable [amd64]) []
Inst libreadline-dev (8.2-1.3 Debian:12.15/oldstable [amd64]) []
Inst liblua5.3-dev (5.3.6-2 Debian:12.15/oldstable [amd64]) []
Inst lua5.1 (5.1.5-9 Debian:12.15/oldstable [amd64]) []
Inst lua-any (29 Debian:12.15/oldstable [all]) []
Inst zip (3.0-13 Debian:12.15/oldstable [amd64]) []
Inst unzip (6.0-28+deb12u1 Debian-Security:12/oldstable-security [amd64]) []
Inst luarocks (3.8.0+dfsg1-1 Debian:12.15/oldstable [all]) []
Inst python3-rbd [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64]) []
Inst python3-ceph-common [18.2.8-1~bpo12+1] (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [all])
Conf ceph-mgr (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf ceph-mgr-modules-core (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [all])
Conf ceph-volume (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [all])
Conf ceph-osd (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf ceph-mon (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf ceph-mds (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf ceph-base (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf radosgw (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf ceph-common (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf python3-cephfs (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf libcephfs2 (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf python3-rgw (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf python3-rados (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf python3-ceph-argparse (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [all])
Conf libsqlite3-mod-ceph (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf librbd1 (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf libradosstriper1 (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf librgw2 (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf librados2 (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf libncurses-dev (6.4-4 Debian:12.15/oldstable [amd64])
Conf libreadline-dev (8.2-1.3 Debian:12.15/oldstable [amd64])
Conf liblua5.3-dev (5.3.6-2 Debian:12.15/oldstable [amd64])
Conf lua5.1 (5.1.5-9 Debian:12.15/oldstable [amd64])
Conf lua-any (29 Debian:12.15/oldstable [all])
Conf zip (3.0-13 Debian:12.15/oldstable [amd64])
Conf unzip (6.0-28+deb12u1 Debian-Security:12/oldstable-security [amd64])
Conf luarocks (3.8.0+dfsg1-1 Debian:12.15/oldstable [all])
Conf python3-rbd (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [amd64])
Conf python3-ceph-common (19.2.6-1bookworm Wikimedia:12/bookworm-wikimedia [all])
BTullis added a comment.Tue, Sep 8, 11:35 AM

Going for it now.

Setting the cluster to noout mode.

btullis@cephosd2001:~$ sudo ceph osd set noout
noout is set

I then ran the following on each of the three nodes.

btullis@cephosd2001:~$ sudo apt-get install ceph-mon

I then restarted the first daemon, which was the mon daemon running on cephosd2001.

btullis@cephosd2001:~$ sudo systemctl restart ceph-mon@cephosd2001.service

As expected, we started to see issues with the cluster health, relating to the use of insecure keys.

btullis@cephosd2001:~$ sudo ceph status
  cluster:
    id:     8e69717a-518b-4c00-9f96-0635d9b913c6
    health: HEALTH_ERR
            14 auth client entities with insecure key types
            Monitors are configured to allow auth using insecure key types
            Monitors are configured to allow creation of insecure key types
            4 rotating auth service keys using insecure key types
            75 auth service entities with insecure key types
            Monitors are configured to issue insecure service tickets
            no active mgr
            noout flag(s) set
 
  services:
    mon: 3 daemons, quorum cephosd2001,cephosd2002,cephosd2003 (age 7m)
    mgr: no daemons active (since 7m)
    mds: 1/1 daemons up, 2 standby
    osd: 66 osds: 66 up (since 7w), 66 in (since 14M)
         flags noout
    rgw: 3 daemons active (3 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   14 pools, 2497 pgs
    objects: 725.89k objects, 2.8 TiB
    usage:   25 TiB used, 687 TiB / 712 TiB avail
    pgs:     2497 active+clean
 
  io:
    client:   65 KiB/s rd, 121 KiB/s wr, 7 op/s rd, 13 op/s wr

And for more detail:

btullis@cephosd2001:~$ sudo ceph health detail
HEALTH_ERR 14 auth client entities with insecure key types; Monitors are configured to allow auth using insecure key types; Monitors are configured to allow creation of insecure key types; 4 rotating auth service keys using insecure key types; 75 auth service entities with insecure key types; Monitors are configured to issue insecure service tickets; no active mgr; noout flag(s) set
[WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 14 auth client entities with insecure key types
    entity client.admin using insecure key type: aes
    entity client.aux-k8s-csi-rbd using insecure key type: aes
    entity client.bootstrap-mds using insecure key type: aes
    entity client.bootstrap-mgr using insecure key type: aes
    entity client.bootstrap-osd using insecure key type: aes
    entity client.bootstrap-rbd using insecure key type: aes
    entity client.bootstrap-rbd-mirror using insecure key type: aes
    entity client.bootstrap-rgw using insecure key type: aes
    entity client.dse-k8s-csi using insecure key type: aes
    entity client.dse-k8s-csi-cephfs using insecure key type: aes
    entity client.radosgw using insecure key type: aes
    entity client.rgw.cephosd2001 using insecure key type: aes
    entity client.rgw.cephosd2002 using insecure key type: aes
    entity client.rgw.cephosd2003 using insecure key type: aes
[WRN] AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow auth using insecure key types
    insecure cipher aes allowed for auth
[WRN] AUTH_INSECURE_KEYS_CREATABLE: Monitors are configured to allow creation of insecure key types
[WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types
    rotating service keys for mon using insecure key type: aes
    rotating service keys for mds using insecure key type: aes
    rotating service keys for osd using insecure key type: aes
    rotating service keys for mgr using insecure key type: aes
[ERR] AUTH_INSECURE_SERVICE_KEY_TYPE: 75 auth service entities with insecure key types
    entity mds.cephosd2001 using insecure key type: aes
    entity mds.cephosd2002 using insecure key type: aes
    entity mds.cephosd2003 using insecure key type: aes
    entity osd.0 using insecure key type: aes
    entity osd.1 using insecure key type: aes
    entity osd.10 using insecure key type: aes
    entity osd.11 using insecure key type: aes
    entity osd.12 using insecure key type: aes
    entity osd.13 using insecure key type: aes
    entity osd.14 using insecure key type: aes
    entity osd.15 using insecure key type: aes
    entity osd.16 using insecure key type: aes
    entity osd.17 using insecure key type: aes
    entity osd.18 using insecure key type: aes
    entity osd.19 using insecure key type: aes
    entity osd.2 using insecure key type: aes
    entity osd.20 using insecure key type: aes
    entity osd.21 using insecure key type: aes
    entity osd.22 using insecure key type: aes
    entity osd.23 using insecure key type: aes
    entity osd.24 using insecure key type: aes
    entity osd.25 using insecure key type: aes
    entity osd.26 using insecure key type: aes
    entity osd.27 using insecure key type: aes
    entity osd.28 using insecure key type: aes
    entity osd.29 using insecure key type: aes
    entity osd.3 using insecure key type: aes
    entity osd.30 using insecure key type: aes
    entity osd.31 using insecure key type: aes
    entity osd.32 using insecure key type: aes
    entity osd.33 using insecure key type: aes
    entity osd.34 using insecure key type: aes
    entity osd.35 using insecure key type: aes
    entity osd.36 using insecure key type: aes
    entity osd.37 using insecure key type: aes
    entity osd.38 using insecure key type: aes
    entity osd.39 using insecure key type: aes
    entity osd.4 using insecure key type: aes
    entity osd.40 using insecure key type: aes
    entity osd.41 using insecure key type: aes
    entity osd.42 using insecure key type: aes
    entity osd.43 using insecure key type: aes
    entity osd.44 using insecure key type: aes
    entity osd.45 using insecure key type: aes
    entity osd.46 using insecure key type: aes
    entity osd.47 using insecure key type: aes
    entity osd.48 using insecure key type: aes
    entity osd.49 using insecure key type: aes
    entity osd.5 using insecure key type: aes
    entity osd.50 using insecure key type: aes
    entity osd.51 using insecure key type: aes
    entity osd.52 using insecure key type: aes
    entity osd.53 using insecure key type: aes
    entity osd.54 using insecure key type: aes
    entity osd.55 using insecure key type: aes
    entity osd.56 using insecure key type: aes
    entity osd.57 using insecure key type: aes
    entity osd.58 using insecure key type: aes
    entity osd.59 using insecure key type: aes
    entity osd.6 using insecure key type: aes
    entity osd.60 using insecure key type: aes
    entity osd.61 using insecure key type: aes
    entity osd.62 using insecure key type: aes
    entity osd.63 using insecure key type: aes
    entity osd.64 using insecure key type: aes
    entity osd.65 using insecure key type: aes
    entity osd.7 using insecure key type: aes
    entity osd.8 using insecure key type: aes
    entity osd.9 using insecure key type: aes
    entity osd.cephosd2001 using insecure key type: aes
    entity osd.cephosd2002 using insecure key type: aes
    entity osd.cephosd2003 using insecure key type: aes
    entity mgr.cephosd2001 using insecure key type: aes
    entity mgr.cephosd2002 using insecure key type: aes
    entity mgr.cephosd2003 using insecure key type: aes
[ERR] AUTH_INSECURE_SERVICE_TICKETS: Monitors are configured to issue insecure service tickets
[WRN] MGR_DOWN: no active mgr
[WRN] OSDMAP_FLAGS: noout flag(s) set

However, the new mon service is running correctly.

We can see that the new key type is also available.

btullis@cephosd2001:~$ sudo ceph --format=json mon dump | jq -r '.auth_allowed_ciphers | map(.name) | join (",")'
dumped monmap epoch 1
aes,aes256k

Proceeding to restart the other two mon daemons.

btullis@cephosd2002:~$ sudo systemctl restart ceph-mon@cephosd2002.service 
btullis@cephosd2002:~$ systemctl status ceph-mon@cephosd2002.service 
● ceph-mon@cephosd2002.service - Ceph cluster monitor daemon
     Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-09-08 11:32:56 UTC; 9s ago
   Main PID: 4064657 (ceph-mon)
      Tasks: 24
     Memory: 184.1M
        CPU: 1.887s
     CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@cephosd2002.service
             └─4064657 /usr/bin/ceph-mon -f --cluster ceph --id cephosd2002 --setuser ceph --setgroup ceph

btullis@cephosd2003:~$ sudo systemctl restart ceph-mon@cephosd2003.service 
btullis@cephosd2003:~$ systemctl status ceph-mon@cephosd2003.service 
● ceph-mon@cephosd2003.service - Ceph cluster monitor daemon
     Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-09-08 11:33:38 UTC; 8s ago
   Main PID: 4023827 (ceph-mon)
      Tasks: 24
     Memory: 231.3M
        CPU: 1.767s
     CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@cephosd2003.service
             └─4023827 /usr/bin/ceph-mon -f --cluster ceph --id cephosd2003 --setuser ceph --setgroup ceph

Checking the minimum mon version number:

btullis@cephosd2003:~$ sudo ceph mon dump | grep min_mon_release
dumped monmap epoch 2
min_mon_release 19 (squid)
BTullis added a comment.Tue, Sep 8, 11:39 AM

Restarting the mgr daemons on each host.

btullis@cephosd2001:~$ sudo systemctl restart ceph-mgr.target

btullis@cephosd2001:~$ systemctl status ceph-mgr@cephosd2001.service 
● ceph-mgr@cephosd2001.service - Ceph cluster manager daemon
     Loaded: loaded (/lib/systemd/system/ceph-mgr@.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-09-08 11:36:06 UTC; 1min 16s ago
   Main PID: 4045817 (ceph-mgr)
      Tasks: 124 (limit: 308990)
     Memory: 411.6M
        CPU: 6.158s
     CGroup: /system.slice/system-ceph\x2dmgr.slice/ceph-mgr@cephosd2001.service
             └─4045817 /usr/bin/ceph-mgr -f --cluster ceph --id cephosd2001 --setuser ceph --setgroup ceph

btullis@cephosd2002:~$ sudo systemctl restart ceph-mgr.target
btullis@cephosd2002:~$ systemctl status ceph-mgr@cephosd2002.service 
● ceph-mgr@cephosd2002.service - Ceph cluster manager daemon
     Loaded: loaded (/lib/systemd/system/ceph-mgr@.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-09-08 11:37:56 UTC; 14s ago
   Main PID: 4066575 (ceph-mgr)
      Tasks: 29 (limit: 308990)
     Memory: 291.9M
        CPU: 3.509s
     CGroup: /system.slice/system-ceph\x2dmgr.slice/ceph-mgr@cephosd2002.service
             └─4066575 /usr/bin/ceph-mgr -f --cluster ceph --id cephosd2002 --setuser ceph --setgroup ceph

btullis@cephosd2003:~$ sudo systemctl restart ceph-mgr.target
btullis@cephosd2003:~$ systemctl status ceph-mgr@cephosd2003.service 
● ceph-mgr@cephosd2003.service - Ceph cluster manager daemon
     Loaded: loaded (/lib/systemd/system/ceph-mgr@.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-09-08 11:38:31 UTC; 8s ago
   Main PID: 4025716 (ceph-mgr)
      Tasks: 29 (limit: 308990)
     Memory: 293.1M
        CPU: 3.431s
     CGroup: /system.slice/system-ceph\x2dmgr.slice/ceph-mgr@cephosd2003.service
             └─4025716 /usr/bin/ceph-mgr -f --cluster ceph --id cephosd2003 --setuser ceph --setgroup ceph

The health still looks OK, bearing in mind the key rotation issue that we know about.

btullis@cephosd2001:~$ sudo ceph status
  cluster:
    id:     8e69717a-518b-4c00-9f96-0635d9b913c6
    health: HEALTH_ERR
            14 auth client entities with insecure key types
            Monitors are configured to allow auth using insecure key types
            Monitors are configured to allow creation of insecure key types
            4 rotating auth service keys using insecure key types
            75 auth service entities with insecure key types
            Monitors are configured to issue insecure service tickets
            noout flag(s) set
 
  services:
    mon: 3 daemons, quorum cephosd2001,cephosd2002,cephosd2003 (age 5m)
    mgr: cephosd2001(active, since 2m), standbys: cephosd2002, cephosd2003
    mds: 1/1 daemons up, 2 standby
    osd: 66 osds: 66 up (since 7w), 66 in (since 14M)
         flags noout
    rgw: 3 daemons active (3 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   14 pools, 2497 pgs
    objects: 725.89k objects, 2.8 TiB
    usage:   25 TiB used, 687 TiB / 712 TiB avail
    pgs:     2497 active+clean
 
  io:
    client:   2.0 KiB/s rd, 250 KiB/s wr, 0 op/s rd, 15 op/s wr
BTullis added a comment.Tue, Sep 8, 12:52 PM

I have restarted all OSD daemons by running the following an all three hosts, several minutes apart.

sudo systemctl restart ceph-osd.target

We can see the new versions being reported here: https://grafana-rw.wikimedia.org/goto/s4mw5r?orgId=default

image.png (931×374 px, 37 KB)

Now proceeding to upgrade the metadata servers.

BTullis added a comment.Tue, Sep 8, 1:00 PM

Disabling standby replay.

btullis@cephosd2001:~$ sudo ceph fs ls
name: dpe, metadata pool: cephfs.dpe.meta, data pools: [cephfs.dpe.data-ssd ]

btullis@cephosd2001:~$ sudo ceph fs set dpe allow_standby_replay false

We only have one filesystem on this cluster. Currently cephosd2001 is the master, cephosd200[2-3] are both standby MDS servers for this.

btullis@cephosd2001:~$ sudo ceph fs dump
e106
btime 2026-09-08T12:50:57:070360+0000
enable_multiple, ever_enabled_multiple: 1,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
 
Filesystem 'dpe' (1)
fs_name	dpe
epoch	106
flags	12 joinable allow_snaps allow_multimds_snaps
created	2025-07-09T16:19:10.593758+0000
modified	2026-09-08T12:50:57.070356+0000
tableserver	0
root	0
session_timeout	60
session_autoclose	300
max_file_size	1099511627776
max_xattr_size	65536
required_client_features	{}
last_failure	0
last_failure_osd_epoch	30227
compat	compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds	1
in	0
up	{0=4720308}
failed	
damaged	
stopped	
data_pools	[9]
metadata_pool	8
inline_data	disabled
balancer	
bal_rank_mask	-1
standby_count_wanted	1
qdb_cluster	leader: 4720308 members: 4720308
[mds.cephosd2001{0:4720308} state up:active seq 261 addr [v2:10.192.9.17:6800/787382144,v1:10.192.9.17:6801/787382144] compat {c=[1],r=[1],i=[7ff]}]
 
 
Standby daemons:
 
[mds.cephosd2003{-1:4702097} state up:standby seq 1 addr [v2:10.192.37.16:6800/864763936,v1:10.192.37.16:6801/864763936] compat {c=[1],r=[1],i=[7ff]}]
[mds.cephosd2002{-1:4779985} state up:standby seq 1 addr [v2:10.192.26.19:6800/1751560837,v1:10.192.26.19:6801/1751560837] compat {c=[1],r=[1],i=[7ff]}]
dumped fsmap epoch 106

Proceeding to shut down 2002 and 2003, then restart 2001.

btullis@cephosd2002:~$ sudo systemctl stop ceph-mds@cephosd2002.service 

btullis@cephosd2003:~$ sudo systemctl stop ceph-mds@cephosd2003.service 

btullis@cephosd2001:~$ sudo systemctl restart ceph-mds@cephosd2001.service

Checked the logs of the MDS

btullis@cephosd2001:~$ journalctl -f -u ceph-mds@cephosd2001.service 
Sep 08 12:56:04 cephosd2001 ceph-mds[4095644]: 2026-09-08T12:56:04.204+0000 7f10345b86c0  1 mds.0.110 rejoin_joint_start
Sep 08 12:56:04 cephosd2001 ceph-mds[4095644]: 2026-09-08T12:56:04.204+0000 7f10345b86c0  1 mds.0.110 rejoin_done
Sep 08 12:56:05 cephosd2001 ceph-mds[4095644]: 2026-09-08T12:56:05.216+0000 7f10345b86c0  1 mds.cephosd2001 Updating MDS map to version 113 from mon.0
Sep 08 12:56:05 cephosd2001 ceph-mds[4095644]: 2026-09-08T12:56:05.216+0000 7f10345b86c0  1 mds.0.110 handle_mds_map I am now mds.0.110
Sep 08 12:56:05 cephosd2001 ceph-mds[4095644]: 2026-09-08T12:56:05.216+0000 7f10345b86c0  1 mds.0.110 handle_mds_map state change up:rejoin --> up:active
Sep 08 12:56:05 cephosd2001 ceph-mds[4095644]: 2026-09-08T12:56:05.216+0000 7f10345b86c0  1 mds.0.110 recovery_done -- successful recovery!
Sep 08 12:56:05 cephosd2001 ceph-mds[4095644]: 2026-09-08T12:56:05.216+0000 7f10345b86c0  1 mds.0.110 active_start
Sep 08 12:56:05 cephosd2001 ceph-mds[4095644]: 2026-09-08T12:56:05.216+0000 7f10345b86c0  1 mds.0.110 cluster recovered.

Started the services on 2002 and 2003.

btullis@cephosd2002:~$ sudo systemctl start ceph-mds@cephosd2002.service 

btullis@cephosd2003:~$ sudo systemctl start ceph-mds@cephosd2003.service

Re-enabled standby replay.

btullis@cephosd2001:~$ sudo ceph fs set dpe allow_standby_replay true

Checked health status.

btullis@cephosd2001:~$ sudo ceph status
  cluster:
    id:     8e69717a-518b-4c00-9f96-0635d9b913c6
    health: HEALTH_ERR
            14 auth client entities with insecure key types
            Monitors are configured to allow auth using insecure key types
            Monitors are configured to allow creation of insecure key types
            4 rotating auth service keys using insecure key types
            75 auth service entities with insecure key types
            Monitors are configured to issue insecure service tickets
            noout flag(s) set
            all OSDs are running squid or later but require_osd_release < squid
 
  services:
    mon: 3 daemons, quorum cephosd2001,cephosd2002,cephosd2003 (age 84m)
    mgr: cephosd2001(active, since 82m), standbys: cephosd2002, cephosd2003
    mds: 1/1 daemons up, 1 standby, 1 hot standby
    osd: 66 osds: 66 up (since 9m), 66 in (since 14M)
         flags noout
    rgw: 3 daemons active (3 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   14 pools, 2497 pgs
    objects: 725.90k objects, 2.8 TiB
    usage:   25 TiB used, 687 TiB / 712 TiB avail
    pgs:     2497 active+clean
 
  io:
    client:   851 B/s rd, 298 KiB/s wr, 1 op/s rd, 17 op/s wr

Interesting point to note;

mds: 1/1 daemons up, 1 standby, 1 hot standby

I think that hot standby is new. I will investigate.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits