Page MenuHomePhabricator

ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288
Closed, ResolvedPublic

Description

Write the description below

There was a security update for ceph released today to fix https://docs.ceph.com/en/latest/security/CVE-2021-20288/:

We have to upgrade to either of:

  • v14.2.20 (Nautilus)
  • v15.2.11 (Octopus)
  • v16.2.1 (Pacific)

Recommended upgrade process:

  • Users should upgrade to a patched version of Ceph at their earliest convenience.
  • Users should upgrade any unpatched clients at their earliest convenience. By default, these clients can be easily identified by checking the ceph health detail output for the AUTH_INSECURE_GLOBAL_ID_RECLAIM alert.
  • If all clients cannot be upgraded immediately, the health alerts can be temporarily muted with:
ceph health mute AUTH_INSECURE_GLOBAL_ID_RECLAIM 1w  # 1 week
ceph health mute AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED 1w  # 1 week
  • After all clients have been updated and the AUTH_INSECURE_GLOBAL_ID_RECLAIM alert is no longer present, the cluster should be set to prevent insecure global_id reclaim with:
ceph config set mon auth_allow_insecure_global_id_reclaim false

Details

SubjectRepoBranchLines +/-
operations/cookbookswmcs+122 -9
operations/cookbookswmcs+89 -0
operations/cookbookswmcs+113 -5
operations/cookbookswmcs+114 -1
operations/cookbooksmaster+89 -0
operations/cookbooksmaster+113 -5
operations/cookbooksmaster+114 -1
operations/cookbooksmaster+114 -1
operations/cookbooksmaster+113 -5
operations/cookbooksmaster+89 -0
operations/cookbooksmaster+114 -1
operations/cookbooksmaster+113 -5
operations/cookbooksmaster+89 -0
operations/cookbooksmaster+122 -9
operations/puppetproduction+10 -2
operations/cookbooksmaster+272 -8
operations/cookbooksmaster+3 -1
operations/software/spicerackmaster+2 -2
operations/software/spicerackmaster+13 -5
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-cloud) [2021-04-28T10:57:21Z] <dcaro> Got a PG getting stuck on 'remapping' after the OSD came up, had to unset the norebalance and then set it again to get it unstuck (T280641)

Mentioned in SAL (#wikimedia-cloud) [2021-04-28T11:06:01Z] <dcaro> All ceph server side upgraded to Octopus! \o/ (T280641)

Change 683370 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@master] wmcs: add cloudvirt drain cookbook

https://gerrit.wikimedia.org/r/683370

Mentioned in SAL (#wikimedia-cloud) [2021-04-30T09:47:59Z] <dcaro> draining coludvirt1013 for reboot (T280641)

Mentioned in SAL (#wikimedia-cloud) [2021-04-30T10:37:41Z] <dcaro> draining coludvirt1016 for reboot (T280641)

Mentioned in SAL (#wikimedia-cloud) [2021-04-30T11:16:54Z] <dcaro> draining and rebooting coludvirt1017, last one today (T280641)

Change 683857 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs.drain_hypervisor: skip all VMs in the canary project

https://gerrit.wikimedia.org/r/683857

Change 683371 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@master] wmcs.openstack: add live_upgrade cloudvirt cookbook

https://gerrit.wikimedia.org/r/683371

Change 683888 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@master] wmcs.openstack: add safe_reboot cloudvirt cookbook

https://gerrit.wikimedia.org/r/683888

Mentioned in SAL (#wikimedia-cloud) [2021-05-03T08:26:11Z] <dcaro> draining and rebooting coludvirt1018 (T280641)

Mentioned in SAL (#wikimedia-cloud) [2021-05-03T09:12:39Z] <dcaro> draining and rebooting coludvirt1021 (T280641)

Mentioned in SAL (#wikimedia-cloud) [2021-05-03T10:31:29Z] <wm-bot> Safe rebooting 'cloudvirt1021.eqiad.wmnet'. (T280641 - cookbook ran by dcaro@vulcanus)

Mentioned in SAL (#wikimedia-cloud) [2021-05-03T14:07:25Z] <dcaro> depooling tols-sgeexec-0908/7 to be able to restart the VMs as they got stuck during migration (T280641)

Mentioned in SAL (#wikimedia-cloud) [2021-05-03T15:13:12Z] <wm-bot> Safe rebooting 'cloudvirt1022.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-03T15:41:23Z] <wm-bot> Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-03T15:41:50Z] <wm-bot> Safe rebooting 'cloudvirt1023.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-03T16:23:57Z] <dcaro> started tools-sgeexec-0907, was stuck on initramfs due to an unclean fs (/dev/vda3, root), ran fsck manually fixing all the errors and booted up correctly after (T280641)

Mentioned in SAL (#wikimedia-cloud) [2021-05-03T16:29:08Z] <wm-bot> Safe rebooting 'cloudvirt1023.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-03T16:34:03Z] <wm-bot> Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T08:03:25Z] <wm-bot> Safe rebooting 'cloudvirt1024.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T08:20:34Z] <wm-bot> Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T08:34:31Z] <wm-bot> Safe rebooting 'cloudvirt1025.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T09:10:28Z] <wm-bot> Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T09:10:31Z] <wm-bot> Safe rebooting 'cloudvirt1026.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T10:04:05Z] <wm-bot> Safe rebooting 'cloudvirt1026.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T15:15:35Z] <wm-bot> Safe rebooting 'cloudvirt1026.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T15:19:10Z] <wm-bot> Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T15:22:50Z] <wm-bot> Safe rebooting 'cloudvirt1027.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T15:44:46Z] <wm-bot> Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T15:45:57Z] <wm-bot> Safe rebooting 'cloudvirt1028.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-04T16:05:57Z] <wm-bot> Safe reboot of 'cloudvirt1028.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T08:02:07Z] <wm-bot> Safe rebooting 'cloudvirt1029.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T08:19:02Z] <wm-bot> Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T08:19:06Z] <wm-bot> Safe rebooting 'cloudvirt1030.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T08:45:42Z] <wm-bot> Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T08:45:45Z] <wm-bot> Safe rebooting 'cloudvirt1031.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T09:21:32Z] <wm-bot> Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T09:21:35Z] <wm-bot> Safe rebooting 'cloudvirt1032.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T09:47:25Z] <wm-bot> Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T09:47:28Z] <wm-bot> Safe rebooting 'cloudvirt1033.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T10:13:15Z] <wm-bot> Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T10:13:19Z] <wm-bot> Safe rebooting 'cloudvirt1034.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Change 683857 merged by David Caro:

[operations/puppet@production] wmcs.drain_hypervisor: use canary project instead of VM name

https://gerrit.wikimedia.org/r/683857

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T10:39:42Z] <wm-bot> Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T10:39:46Z] <wm-bot> Safe rebooting 'cloudvirt1035.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T11:08:08Z] <wm-bot> Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T11:08:11Z] <wm-bot> Safe rebooting 'cloudvirt1036.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T11:31:17Z] <wm-bot> Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T11:31:21Z] <wm-bot> Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T11:56:23Z] <wm-bot> Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T11:56:26Z] <wm-bot> Safe rebooting 'cloudvirt1038.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T12:35:14Z] <wm-bot> Safe rebooting 'cloudvirt1039.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T14:10:07Z] <wm-bot> Safe rebooting 'cloudvirt1039.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T14:14:42Z] <wm-bot> Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T14:14:46Z] <wm-bot> Safe rebooting 'cloudvirt1041.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T14:39:57Z] <wm-bot> Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T14:40:00Z] <wm-bot> Safe rebooting 'cloudvirt1042.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T14:59:44Z] <wm-bot> Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T14:59:48Z] <wm-bot> Safe rebooting 'cloudvirt1043.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T15:03:39Z] <wm-bot> Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T15:03:41Z] <wm-bot> Safe rebooting 'cloudvirt1044.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T15:07:34Z] <wm-bot> Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T15:07:37Z] <wm-bot> Safe rebooting 'cloudvirt1045.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T15:11:27Z] <wm-bot> Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T15:11:29Z] <wm-bot> Safe rebooting 'cloudvirt1046.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T15:15:22Z] <wm-bot> Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-05-05T16:07:25Z] <dcaro> disallowing insecure global ids on the eqiad ceph cluster (T280641)

Change 695202 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] wmcs.ceph: add cookbook to upgrade all osds

https://gerrit.wikimedia.org/r/695202

Change 682106 abandoned by David Caro:

[operations/cookbooks@master] wmcs.ceph: add cookbook to upgrade all osds

Reason:

Moved to another branch

https://gerrit.wikimedia.org/r/682106

Change 695205 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@master] wmcs: add cloudvirt drain cookbook

https://gerrit.wikimedia.org/r/695205

Change 695206 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@master] wmcs.openstack: add safe_reboot cloudvirt cookbook

https://gerrit.wikimedia.org/r/695206

Change 695207 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@master] wmcs.openstack: add live_upgrade cloudvirt cookbook

https://gerrit.wikimedia.org/r/695207

Change 683371 abandoned by David Caro:

[operations/cookbooks@master] wmcs.openstack: add live_upgrade cloudvirt cookbook

Reason:

https://gerrit.wikimedia.org/r/683371

Change 683888 abandoned by David Caro:

[operations/cookbooks@master] wmcs.openstack: add safe_reboot cloudvirt cookbook

Reason:

https://gerrit.wikimedia.org/r/683888

Change 683370 abandoned by David Caro:

[operations/cookbooks@master] wmcs: add cloudvirt drain cookbook

Reason:

https://gerrit.wikimedia.org/r/683370

Change 695207 abandoned by David Caro:

[operations/cookbooks@master] wmcs.openstack: add live_upgrade cloudvirt cookbook

Reason:

wrong branch

https://gerrit.wikimedia.org/r/695207

Change 695206 abandoned by David Caro:

[operations/cookbooks@master] wmcs.openstack: add safe_reboot cloudvirt cookbook

Reason:

wrong branch

https://gerrit.wikimedia.org/r/695206

Change 695205 abandoned by David Caro:

[operations/cookbooks@master] wmcs: add cloudvirt drain cookbook

Reason:

wrong branch

https://gerrit.wikimedia.org/r/695205

Change 695210 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@master] wmcs: add cloudvirt drain cookbook

https://gerrit.wikimedia.org/r/695210

Change 695211 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@master] wmcs.openstack: add safe_reboot cloudvirt cookbook

https://gerrit.wikimedia.org/r/695211

Change 695212 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@master] wmcs.openstack: add live_upgrade cloudvirt cookbook

https://gerrit.wikimedia.org/r/695212

Change 695210 abandoned by David Caro:

[operations/cookbooks@master] wmcs: add cloudvirt drain cookbook

Reason:

wrong branch

https://gerrit.wikimedia.org/r/695210

Change 695211 abandoned by David Caro:

[operations/cookbooks@master] wmcs.openstack: add safe_reboot cloudvirt cookbook

Reason:

wrong branch

https://gerrit.wikimedia.org/r/695211

Change 695212 abandoned by David Caro:

[operations/cookbooks@master] wmcs.openstack: add live_upgrade cloudvirt cookbook

Reason:

wrong branch

https://gerrit.wikimedia.org/r/695212

Change 695220 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] wmcs: add cloudvirt drain cookbook

https://gerrit.wikimedia.org/r/695220

Change 695221 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] wmcs.openstack: add safe_reboot cloudvirt cookbook

https://gerrit.wikimedia.org/r/695221

Change 695222 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] wmcs.openstack: add live_upgrade cloudvirt cookbook

https://gerrit.wikimedia.org/r/695222

Change 695220 merged by jenkins-bot:

[operations/cookbooks@wmcs] wmcs: add cloudvirt drain cookbook

https://gerrit.wikimedia.org/r/695220

Change 695221 merged by jenkins-bot:

[operations/cookbooks@wmcs] wmcs.openstack: add safe_reboot cloudvirt cookbook

https://gerrit.wikimedia.org/r/695221

Change 695222 merged by jenkins-bot:

[operations/cookbooks@wmcs] wmcs.openstack: add live_upgrade cloudvirt cookbook

https://gerrit.wikimedia.org/r/695222

Change 695202 merged by jenkins-bot:

[operations/cookbooks@wmcs] wmcs.ceph: add cookbook to upgrade all osds

https://gerrit.wikimedia.org/r/695202