| BTullis | |
| Apr 19 2024, 3:59 PM |
| F48592976: image.png | |
| Apr 25 2024, 9:34 AM |
| F48440466: image.png | |
| Apr 24 2024, 12:01 PM |
We have not updated cephosd100[1-5] in a little over a year, but we are now looking to make use of it as part of the Airflow migration to dse-k8s.
As such, we will likely want to upgrade the Ceph packages to make sure that we have the latest release.
This might entail upgrading to the latest point release of quincy , or we could look at upgrading to the cluster to reef.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T362788 Migrate Airflow to the dse-k8s cluster | |||
| Resolved | BTullis | T364386 Validate postgres operator and Ceph integration | |||
| Resolved | BTullis | T362993 Update cephosd100[1-5] with the most recent stable version of Ceph |
For reference, the latest point release for Quincy is 17.2.7 and we could sync these packages to our apt server at any time.
btullis@apt1002:~$ sudo -i reprepro --component thirdparty/ceph-quincy checkupdate bullseye-wikimedia Calculating packages to get... Updates needed for 'bullseye-wikimedia|thirdparty/ceph-quincy|amd64': 'ceph': '17.2.5-1~bpo11+1' will be upgraded to '17.2.7-1~bpo11+1' (from 'thirdparty/ceph-quincy'): <snip snip>
The reef packages, which have been added recently by @MatthewVernon as part of T279621: Set up new S3-level replicated storage cluster "apus" are at version 18.2.2, which is the latest point release.
btullis@apt1002:~$ sudo -i reprepro -C thirdparty/ceph-reef list bookworm-wikimedia ceph bookworm-wikimedia|thirdparty/ceph-reef|amd64: ceph 18.2.2-1~bpo12+1
The release notes for these two versions are here:
I have pulled in the latest quincy packages for bullseye with:
btullis@apt1002:~$ sudo -i reprepro --component thirdparty/ceph-quincy --noskipold update bullseye-wikimedia Calculating packages to get... Getting packages... Installing (and possibly deleting) packages... Exporting indices... Deleting files no longer referenced...
I'm using a debdeploy spec to roll out the new packages to cephosd1001 first. I want to check if services are restarted as part of the upgrade.
btullis@cumin1002:~$ sudo debdeploy deploy -u 2024-04-24-ceph.yaml -Q cephosd1001.eqiad.wmnet Rolling out ceph: Library update, several services might need to be restarted librgw2 was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) ceph-mgr-modules-core was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) ceph-osd was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) python3-rgw was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) libcephfs2 was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) ceph-volume was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) libradosstriper1 was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) ceph-mgr was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) librados2 was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) ceph-mon was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) libsqlite3-mod-ceph was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) python3-ceph-argparse was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) python3-rbd was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) python3-ceph-common was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) ceph-common was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) ceph-base was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) python3-cephfs was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) librbd1 was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts) python3-rados was updated: 17.2.5-1~bpo11+1 -> 17.2.7-1~bpo11+1 cephosd1001.eqiad.wmnet (1 hosts)
No, services are not restarted, which means that I can roll out the packages to the remaining hosts.
btullis@cephosd1001:~$ systemctl status ceph-mon@cephosd1001
● ceph-mon@cephosd1001.service - Ceph cluster monitor daemon
Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-08-21 12:08:25 UTC; 8 months 3 days ago
Main PID: 1347 (ceph-mon)
Tasks: 25
Memory: 1.3G
CPU: 4d 18h 23min 50.191s
CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@cephosd1001.service
└─1347 /usr/bin/ceph-mon -f --cluster ceph --id cephosd1001 --setuser ceph --setgroup cephOur Ceph cluster isn't managed by cephadm (yet) so we do not have ready access to the ceph orch upgrade command, which would make the upgrade simpler.
In this case, we will have to perform the upgrade ourselves by restarting services manually.
I will be following the staggered upgrade sequence that would be carried out by cephadm, which is described here:https://docs.ceph.com/en/quincy/cephadm/upgrade/#staggered-upgrade
The order I will be using is as set out on that page: mgr -> mon -> crash -> osd -> mds -> rgw -> rbd-mirror -> cephfs-mirror -> iscsi -> nfs
Our cluster looks like this:
btullis@cephosd1001:~$ sudo ceph -s
cluster:
id: 6d4278e1-ea45-4d29-86fe-85b44c150813
health: HEALTH_WARN
4 pool(s) have non-power-of-two pg_num
services:
mon: 5 daemons, quorum cephosd1001,cephosd1002,cephosd1003,cephosd1004,cephosd1005 (age 6M)
mgr: cephosd1002(active, since 4w), standbys: cephosd1004, cephosd1005, cephosd1001, cephosd1003
osd: 100 osds: 100 up (since 6M), 100 in (since 9M)
data:
pools: 5 pools, 4001 pgs
objects: 444.98k objects, 132 GiB
usage: 28 TiB used, 1.1 PiB / 1.1 PiB avail
pgs: 4001 active+cleanSo the upgrade sequence will be as follows:
We do not have instances of any of the other services mentioned: mds -> rgw -> rbd-mirror -> cephfs-mirror -> iscsi -> nfs so we do not need to consider these.
First standby mgr upgrade is fine.
btullis@cephosd1001:~$ sudo systemctl status ceph-mgr.target
● ceph-mgr.target - ceph target allowing to start/stop all ceph-mgr@.service instances at once
Loaded: loaded (/lib/systemd/system/ceph-mgr.target; enabled; vendor preset: enabled)
Active: active since Wed 2024-04-24 11:48:13 UTC; 8s ago
Apr 24 11:48:13 cephosd1001 systemd[1]: Reached target ceph target allowing to start/stop all ceph-mgr@.service instances at once.
btullis@cephosd1001:~$ sudo systemctl status ceph-mgr@cephosd1001.service
● ceph-mgr@cephosd1001.service - Ceph cluster manager daemon
Loaded: loaded (/lib/systemd/system/ceph-mgr@.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2024-04-24 11:48:13 UTC; 16s ago
Main PID: 3877180 (ceph-mgr)
Tasks: 21 (limit: 309007)
Memory: 212.3M
CPU: 2.992s
CGroup: /system.slice/system-ceph\x2dmgr.slice/ceph-mgr@cephosd1001.service
└─3877180 /usr/bin/ceph-mgr -f --cluster ceph --id cephosd1001 --setuser ceph --setgroup cephProceeding to upgrade the remaining standby servers.
sudo cumin -b 1 -s 30 'cephosd100[3-5].eqiad.wmnet' 'systemctl restart ceph-mgr.target'
Interesting. Restarting the active mgr service on cephosd1002 didn't cause it to failover automatically to a new one. It came up still active.
From sudo ceph health -w we see the following logs:
2024-04-24T12:06:41.529193+0000 mon.cephosd1001 [INF] Active manager daemon cephosd1002 restarted 2024-04-24T12:06:41.537124+0000 mon.cephosd1001 [INF] Activating manager daemon cephosd1002 2024-04-24T12:06:41.652453+0000 mon.cephosd1001 [INF] Manager daemon cephosd1002 is now available
ceph -s shows that cephosd1002 is still the active mgr, which is unexpected, but totally fine.
btullis@cephosd1001:~$ sudo ceph -s
cluster:
id: 6d4278e1-ea45-4d29-86fe-85b44c150813
health: HEALTH_WARN
4 pool(s) have non-power-of-two pg_num
services:
mon: 5 daemons, quorum cephosd1001,cephosd1002,cephosd1003,cephosd1004,cephosd1005 (age 6M)
mgr: cephosd1002(active, since 33s), standbys: cephosd1001, cephosd1004, cephosd1005, cephosd1003
osd: 100 osds: 100 up (since 6M), 100 in (since 9M)
data:
pools: 5 pools, 4001 pgs
objects: 444.98k objects, 132 GiB
usage: 28 TiB used, 1.1 PiB / 1.1 PiB avail
pgs: 4001 active+cleanThe active monitor has been upgraded.
btullis@cephosd1001:~$ sudo ceph tell mgr version
{
"version": "17.2.7",
"release": "quincy",
"release_type": "stable"
}Proceeding to restart the mon services in sequence.
I have upgraded all of the mons successfully.
btullis@cephosd1001:~$ sudo ceph tell mon.* version
mon.cephosd1001: {
"version": "17.2.7",
"release": "quincy",
"release_type": "stable"
}
mon.cephosd1002: {
"version": "17.2.7",
"release": "quincy",
"release_type": "stable"
}
mon.cephosd1003: {
"version": "17.2.7",
"release": "quincy",
"release_type": "stable"
}
mon.cephosd1004: {
"version": "17.2.7",
"release": "quincy",
"release_type": "stable"
}
mon.cephosd1005: {
"version": "17.2.7",
"release": "quincy",
"release_type": "stable"
}Also all of the crash services.
Before doing the osd services, I have set the osds into noout mode, which will cause them not to do any rebalancing whilst the osd services restart.
btullis@cephosd1001:~$ sudo ceph osd set noout noout is set
Now proceeding to restart all of the osds with:
sudo cumin -b 1 -s 90 A:cephosd 'systemctl restart ceph-osd.target'
Change #1023902 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Add thirdparty/ceph-reef to wikimedia-bullseye
I think that it would be a good idea to make sure that a reimage of an existing cephosd server works as expected. before proceeding to upgrade to Ceph reef, then bookworm.
I'll start with cephosd1001.
I have set the noout flag so that it won't try to rebalance data while cephosd1001 is being reimaged.
btullis@cephosd1001:~$ sudo ceph osd set noout noout is set
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1001.eqiad.wmnet with OS bullseye
The first puppet run failed with an error relating to the ceph::osds class.
Oh, I see. It's because we don't yet have the ceph_disks custom fact yet.
root@cephosd1001:~# facter ceph_disks root@cephosd1001:~# facter -p ceph_disks root@cephosd1001:~#
That custom fact is required during the first puppet run.
The file that generates the fact is present:
root@cephosd1001:~# ls -l /var/lib/puppet/lib/facter/ceph_disks.rb -rw-r--r-- 1 root root 3003 Apr 25 09:21 /var/lib/puppet/lib/facter/ceph_disks.rb
...but the perccli64 binary that is required to generate the output isn't available, because the wikimedia-private apt repo isn't configured yet.
root@cephosd1001:~# apt-cache policy perccli N: Unable to locate package perccli
There is a second problem, in that the ceph_disks.rb file also checks to see whether the ceph-osd package is installed before generating the fact.
Of course, this isn't installed before the first puppet run.
root@cephosd1001:~# dpkg-query -W --showformat='${Status}' ceph-osd
dpkg-query: no packages found matching ceph-osdThe reimage wasn't as successful as I had hoped.
The 20 original OSDs (12 hdd and 8 ssd) in cephosd1001 are shown as down.
These were numbered 0-19
Puppet has created 12 new osds, numbered 100-111, which represent the 12 hdds in cephosd1001.
I'll try to fix this up manually.
Also the ceph-mon@cephosd1001 service did not start successfully. The following error is shown in the journal:
Apr 25 10:13:08 cephosd1001 ceph-mon[43798]: 2024-04-25T10:13:08.181+0000 7fde3b572a00 -1 auth: error reading file: /var/lib/ceph/mon/ceph-cephosd1001/keyring: can't open /var/lib/ceph/mon/ceph-cephosd1001/keyr> Apr 25 10:13:08 cephosd1001 ceph-mon[43798]: 2024-04-25T10:13:08.181+0000 7fde3b572a00 -1 auth: error reading file: /var/lib/ceph/mon/ceph-cephosd1001/keyring: can't open /var/lib/ceph/mon/ceph-cephosd1001/keyr> Apr 25 10:13:08 cephosd1001 ceph-mon[43798]: 2024-04-25T10:13:08.181+0000 7fde3b572a00 -1 mon.cephosd1001@-1(???) e0 unable to load initial keyring /var/lib/ceph/mon/ceph-cephosd1001/keyring Apr 25 10:13:08 cephosd1001 ceph-mon[43798]: 2024-04-25T10:13:08.181+0000 7fde3b572a00 -1 failed to initialize Apr 25 10:13:08 cephosd1001 ceph-mon[43798]: 2024-04-25T10:13:08.181+0000 7fde3b572a00 -1 mon.cephosd1001@-1(???) e0 unable to load initial keyring /var/lib/ceph/mon/ceph-cephosd1001/keyring Apr 25 10:13:08 cephosd1001 ceph-mon[43798]: 2024-04-25T10:13:08.181+0000 7fde3b572a00 -1 failed to initialize
This reminds me very much of the issue that was described in T332987: Review and fix any bugs found in the automated bootstrap process for a ceph mon/mgr server. It should be an easy manual fix, but getting puppet to do the right thing in the right order might be more tricky.
The cluster is all back to a healthy state, but it's pretty disappointing that the reimage took so much manual work.
I don't think that I had considered this part of the process carefully enough as part of T330151: Deploy ceph osd processes to data-engineering cluster.
Maybe I should have alook again at the option of Converting an existing cluster to cephadm, or even just bootstrapping this cluster afresh with cephadm to begin with.
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1001.eqiad.wmnet with OS bullseye completed:
Change #1023902 merged by Btullis:
[operations/puppet@production] Add thirdparty/ceph-reef to wikimedia-bullseye
Change #1024742 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Update the DPE ceph cluster to reef
Change #1024773 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Fix the cephosd server reimages
Change #1024773 merged by Btullis:
[operations/puppet@production] Fix the cephosd server reimages
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1001.eqiad.wmnet with OS bullseye
Change #1025292 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Fix the cephosd reimage process
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1001.eqiad.wmnet with OS bullseye completed:
Change #1025292 merged by Btullis:
[operations/puppet@production] Fix the cephosd reimage process
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1002.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1002.eqiad.wmnet with OS bullseye completed:
Change #1025345 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Avoid divide by zero errors if ceph_disks fact is not yet available
Change #1025345 merged by Btullis:
[operations/puppet@production] Avoid divide by zero errors if ceph_disks fact is not yet available
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1003.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1003.eqiad.wmnet with OS bullseye completed:
Change #1025365 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Fix the ceph osd activate command for hdd devices
Change #1025365 merged by Btullis:
[operations/puppet@production] Fix the ceph osd activate command for hdd devices
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1004.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1004.eqiad.wmnet with OS bullseye completed:
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1005.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1005.eqiad.wmnet with OS bullseye completed:
Change #1025428 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Allow the ceph-common package to create the ceph user/group
Change #1025428 merged by Btullis:
[operations/puppet@production] Allow the ceph-common package to create the ceph user/group
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1001.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1001.eqiad.wmnet with OS bullseye completed:
Change #1025724 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Test a fix for the bootstrapping of mon daemons on cephosd*
Change #1025724 merged by Btullis:
[operations/puppet@production] Test a fix for the bootstrapping of mon daemons on cephosd*
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1002.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1002.eqiad.wmnet with OS bullseye completed:
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1003.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1003.eqiad.wmnet with OS bullseye completed:
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1004.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1004.eqiad.wmnet with OS bullseye completed:
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1005.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1005.eqiad.wmnet with OS bullseye completed:
Change #1024742 merged by Btullis:
[operations/puppet@production] Update the DPE ceph cluster to reef
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1001.eqiad.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1001.eqiad.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1002.eqiad.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1002.eqiad.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1003.eqiad.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1003.eqiad.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1003.eqiad.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1004.eqiad.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1004.eqiad.wmnet with OS bookworm completed:
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1005.eqiad.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1002 for host cephosd1005.eqiad.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1002 for host cephosd1005.eqiad.wmnet with OS bookworm completed:
After the rolling upgrade, all versions are now up-to-date, but I see the following heath warning.
btullis@cephosd1002:~$ sudo ceph health detail
HEALTH_WARN 13 mgr modules have failed dependencies
[WRN] MGR_MODULE_DEPENDENCY: 13 mgr modules have failed dependencies
Module 'balancer' has failed dependency: No module named 'bcrypt'
Module 'crash' has failed dependency: No module named 'bcrypt'
Module 'devicehealth' has failed dependency: No module named 'bcrypt'
Module 'iostat' has failed dependency: No module named 'bcrypt'
Module 'nfs' has failed dependency: No module named 'bcrypt'
Module 'orchestrator' has failed dependency: No module named 'bcrypt'
Module 'pg_autoscaler' has failed dependency: No module named 'bcrypt'
Module 'progress' has failed dependency: No module named 'bcrypt'
Module 'rbd_support' has failed dependency: No module named 'bcrypt'
Module 'restful' has failed dependency: No module named 'bcrypt'
Module 'status' has failed dependency: No module named 'bcrypt'
Module 'telemetry' has failed dependency: No module named 'bcrypt'
Module 'volumes' has failed dependency: No module named 'bcrypt'I believe that I can fix this by installing the python3-bcrypt package, but I will have to check that this does not have any adverse impact on the cloudceph clusters.
Is that package not pulled in by dependencies? The Debian/unstable version (which is 18.x) has a Depends: python3-bcrypt. Hm, no, looking at the packages we get from upstream, ceph-mgr does not. That's probably worth reporting upstream?
Ah, this seems to have been tracked as an upstream bug here:
https://github.com/ceph/ceph/pull/54662/commits/387b840c1786e77afab5c88b90682d1df3d42811
They seems to have fixed it here, but I'll check if has been released yet.
There's a new reef point release in QA, but currently blocked on an issue there, I think.
I think I can fix this one by hand now and pull in the new point release in the near future.
I was going to add the package to ceph::mon with a note about its being temporary, but in this case I don't think it's worth it.
I think it is worth fixing in the docker images, though - would you mind a review/+1 on https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1026509 please?
Yep, good thinking.
With that manual install...
sudo cumin A:cephosd 'apt install python3-bcrypt'
...and a rolling restart of the mgr daemons...
sudo cumin -b 1 -s 10 A:cephosd 'systemctl restart ceph-mgr.target'
The cluster health is all OK.
btullis@cephosd1005:~$ sudo ceph health detail HEALTH_OK
All daemons are on version 18.2.2.
btullis@cephosd1005:~$ sudo ceph tell osd.* version|grep version|uniq -c
100 "version": "18.2.2",
btullis@cephosd1005:~$ sudo ceph tell mon.* version|grep version|uniq -c
5 "version": "18.2.2",
btullis@cephosd1005:~$ sudo ceph tell mgr.* version|grep version|uniq -c
1 "version": "18.2.2",So I think I can resolve this ticket and move on.
I notice that we haven't yet enabled the prometheus metrics collection for this cluster, so I will create a new ticket for doing that.
btullis@cephosd1005:~$ sudo ceph mgr module ls MODULE balancer on (always on) crash on (always on) devicehealth on (always on) orchestrator on (always on) pg_autoscaler on (always on) progress on (always on) rbd_support on (always on) status on (always on) telemetry on (always on) volumes on (always on) iostat on nfs on restful on alerts - influx - insights - localpool - mirroring - osd_perf_query - osd_support - prometheus - selftest - snap_schedule - stats - telegraf - test_orchestrator - zabbix -