Page MenuHomePhabricator
Create Task
Maniphest T272303

[ceph] Upgrade to 14.2.16 from 14.2.5
Closed, ResolvedPublic
Actions

Description

We have a new version available, that fixes lots of bugs and CVEs.
This task is to upgrade all our clusters to this new version.

The release notes:
https://docs.ceph.com/en/latest/releases/nautilus/#v14-2-6-nautilus

Process

  • Add the new packages to the repo (they will not be upgraded automatically): T272296
  • Upgrade and test the upgrade on codfw1 cluster (official docs: https://docs.ceph.com/en/nautilus/install/upgrading-ceph/)
    • Upgrade mons (one by one, upgrade packages, then restart daemons, then wait for rejoin)
    • Upgrade osds (one by one, upgrade packages, then restart daemons, then wait for rejoin)
  • Setup a time and upgrade equiad1 (same procedure)

Related Objects
Search...

Event Timeline

dcaro triaged this task as High priority.Jan 18 2021, 2:26 PM
dcaro created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 18 2021, 2:26 PM
dcaro added a subtask: T272296: ceph: nautilus: decide on pending package upgrades.Jan 18 2021, 2:27 PM
Stashbot added a comment.Jan 18 2021, 3:21 PM

Mentioned in SAL (#wikimedia-cloud) [2021-01-18T15:21:42Z] <dcaro> Starting upgrade of ceph mon nodes on codfw (T272303)

Stashbot added a comment.Jan 18 2021, 3:35 PM

Mentioned in SAL (#wikimedia-cloud) [2021-01-18T15:35:00Z] <dcaro> Upgraded mon sevices on codfw ceph cluster, starting with mgr ones (T272303)

Stashbot added a comment.Jan 18 2021, 3:38 PM

Mentioned in SAL (#wikimedia-cloud) [2021-01-18T15:38:04Z] <dcaro> Upgraded mgr sevices on codfw ceph cluster, starting with osd ones (T272303)

dcaro mentioned this in T271782: cloudvirt1024 reports backup job failures.Jan 18 2021, 3:56 PM
Stashbot added a comment.Jan 18 2021, 4:00 PM

Mentioned in SAL (#wikimedia-cloud) [2021-01-18T16:00:34Z] <dcaro> Codfw1 ceph cluster uprgaded, will wait until tomorrow to see if there's any instability, but everything looks fine (T272303)

dcaro moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Jan 19 2021, 12:18 PM
• Bstorm awarded a token.Jan 19 2021, 3:10 PM
Stashbot added a comment.Jan 20 2021, 9:01 AM

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:01:13Z] <dcaro> Will start the ceph upgrade in 15 min, no downtime nor performance impact is expected (T272303)

Stashbot added a comment.Jan 20 2021, 9:16 AM

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:16:02Z] <dcaro> Starting eqiad ceph upgrade, upgrading the mon servers cloudcephmon1* (T272303)

Stashbot added a comment.Jan 20 2021, 9:22 AM

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:22:13Z] <dcaro> Mon daemons upgraded and running, upgrading mgr daemons on servers cloudcephmon1* (T272303)

Stashbot added a comment.Jan 20 2021, 9:24 AM

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:24:39Z] <dcaro> Mgr daemons upgraded and running, upgrading osd daemons on servers cloudcephosd1*, this make take a bit longer (T272303)

Stashbot added a comment.Jan 20 2021, 9:37 AM

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:37:55Z] <dcaro> 25% of the eqiad cluster upgraded... continuing (T272303)

Stashbot added a comment.Jan 20 2021, 9:46 AM

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:46:30Z] <dcaro> 75% of the eqiad cluster upgraded... continuing (T272303)

Stashbot added a comment.Jan 20 2021, 9:55 AM

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T09:55:08Z] <dcaro> Eqiad ceph cluster uprgaded, doing sanity checks (T272303)

dcaro added a comment.Jan 20 2021, 9:59 AM

Will also do an all-round reboot, tracked on T272458

Stashbot added a comment.Jan 20 2021, 10:05 AM

Mentioned in SAL (#wikimedia-cloud) [2021-01-20T10:05:10Z] <dcaro> Everything looks ok, created a new vm with a volume in ceph without issues, and on warnings/errors on ceph status, closing (T272303)

dcaro closed this task as Resolved.Jan 20 2021, 10:05 AM
aborrero awarded a token.Jan 20 2021, 10:18 AM
aborrero closed subtask T272296: ceph: nautilus: decide on pending package upgrades as Resolved.
dcaro mentioned this in T272993: [ceph] investigate osd crash in eqiad.Jan 26 2021, 5:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL