Page MenuHomePhabricator

[ceph] Test and upgrade to Octopus
Closed, ResolvedPublic

Description

In octopus therels several improvements on the OSDs performance and librbd that can help getting the performance we
need.

Upgrade process:
https://docs.ceph.com/en/latest/releases/octopus/#instructions

Event Timeline

dcaro triaged this task as High priority.Feb 11 2021, 4:47 PM
dcaro created this task.

Change 675812 had a related patch set uploaded (by David Caro; author: David Caro):
[operations/puppet@production] ceph: Add octopus repo entry

https://gerrit.wikimedia.org/r/675812

Change 675812 merged by David Caro:

[operations/puppet@production] ceph: Add octopus repo entry

https://gerrit.wikimedia.org/r/675812

Mentioned in SAL (#wikimedia-operations) [2021-04-06T14:29:21Z] <dcaro> populated thirdparty/ceph-octopus buster repo with reprepro (T274566)

Change 677306 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph.mon: parametrize the repository to pull the packages from

https://gerrit.wikimedia.org/r/677306

Change 677306 merged by David Caro:

[operations/puppet@production] ceph.mon: parametrize the repository to pull the packages from

https://gerrit.wikimedia.org/r/677306

Change 677583 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph.codfw1: enable ceph octopus repo

https://gerrit.wikimedia.org/r/677583

It seems we are not using the ceph repos for the clients, will have to address that too:

dcaro@cloudcontrol1003:~$ sudo apt-cache policy ceph-common
ceph-common:
  Installed: 14.2.16-2~bpo10+1
  Candidate: 14.2.16-2~bpo10+1
  Version table:
 *** 14.2.16-2~bpo10+1 1002
        100 http://mirrors.wikimedia.org/debian buster-backports/main amd64 Packages
        100 /var/lib/dpkg/status
     12.2.11+dfsg1-2.1+b1 500
        500 http://mirrors.wikimedia.org/debian buster/main amd64 Packages

Change 677595 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph: use ensure_packages instead of package directly

https://gerrit.wikimedia.org/r/677595

Change 677911 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph: add ceph repo parameter to all client modules

https://gerrit.wikimedia.org/r/677911

Change 677938 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph.common: pin any package from ceph repo to prio 1003

https://gerrit.wikimedia.org/r/677938

Change 677595 merged by David Caro:

[operations/puppet@production] ceph: use ensure_packages instead of package directly

https://gerrit.wikimedia.org/r/677595

Change 677583 merged by David Caro:

[operations/puppet@production] ceph.codfw1: enable ceph octopus repo

https://gerrit.wikimedia.org/r/677583

Mentioned in SAL (#wikimedia-cloud) [2021-04-14T10:07:37Z] <dcaro> Merged the ceph 15 (Octopus) repo deployment to codfw, only the repo, not the packages (T274566)

Mentioned in SAL (#wikimedia-cloud) [2021-04-14T10:15:24Z] <dcaro> starting the upgrade of codfw ceph to octopus 15.2.20 (T274566)

Icinga downtime set by dcaro@cumin1001 for 1 day, 0:00:00 6 host(s) and their services with reason: Upgrading ceph to octopus

cloudcephmon[2002-2004]-dev.codfw.wmnet,cloudcephosd[2001-2003]-dev.codfw.wmnet

Mentioned in SAL (#wikimedia-cloud) [2021-04-14T10:37:08Z] <dcaro> Upgrade of codfw ceph to octopus 15.2.20, mons upgraded, mgrs next (T274566)

Mentioned in SAL (#wikimedia-cloud) [2021-04-14T10:41:53Z] <dcaro> Upgrade of codfw ceph to octopus 15.2.20, mgrs upgraded, osds next (T274566)

Mentioned in SAL (#wikimedia-cloud) [2021-04-14T10:48:30Z] <dcaro> Upgrade of codfw ceph to octopus 15.2.20 done, will run some performance tests now (T274566)

Performance did not seem impacted notably.

Change 677911 merged by David Caro:

[operations/puppet@production] ceph: add ceph repo and parameter to all client modules

https://gerrit.wikimedia.org/r/677911

Change 677938 abandoned by David Caro:

[operations/puppet@production] ceph.common: pin any package from ceph repo to prio 1003

Reason:

This is not needed anymore, replaced by the previous patch it's based on.

https://gerrit.wikimedia.org/r/677938

Change 681296 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph.eqiad: enable octopus repositories

https://gerrit.wikimedia.org/r/681296

Change 681296 merged by David Caro:

[operations/puppet@production] ceph.eqiad: enable octopus repositories

https://gerrit.wikimedia.org/r/681296

Mentioned in SAL (#wikimedia-cloud) [2021-04-29T07:50:42Z] <dcaro> Upgrading ceph libraries on cloudcontrol1004 to octopus (T274566)

Mentioned in SAL (#wikimedia-cloud) [2021-04-29T07:51:20Z] <dcaro> Upgrading ceph libraries on cloudcontrol1003 to octopus (T274566)

Mentioned in SAL (#wikimedia-cloud) [2021-04-29T07:53:15Z] <dcaro> Upgrading ceph libraries on cloudcontrol1005 to octopus (T274566)