Page MenuHomePhabricator

cloud ceph: refactor rbd client puppet profiles
Closed, ResolvedPublic

Description

To better support diversity of use cases, we decided a new layout for the puppet code that supports ceph rbd clients.

See https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/ceph_client_refactor for details.

deploymentceph identityidentity consumertypestatepatches
codfw1devcomputenova-compute @ cloudvirtsnormal keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/737936
eqiad1computenova-compute @ cloudvirtsnormal keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/737936
codfw1devglance-imagesglance-api @ cloudcontrolnormal keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/737925
eqiad1glance-imagesglance-api @ cloudcontrolnormal keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/737936
codfw1devcindercinder-api @ cloudcontrolnormal keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/738411 https://gerrit.wikimedia.org/r/c/operations/puppet/+/738412
eqiad1cindercinder-api @ cloudcontrolnormal keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/738413 https://gerrit.wikimedia.org/r/c/operations/puppet/+/738414
codfw1devcinderlibvirt @ cloudvirtlibvirt secret[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/739474 https://gerrit.wikimedia.org/r/c/operations/puppet/+/739516
eqiad1cinderlibvirt @ cloudvirtlibvirt secret[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/739474 https://gerrit.wikimedia.org/r/c/operations/puppet/+/739516
codfw1devcinder-backupscinder-backups @ cloudbackupnormal keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/736201
eqiad1cinder-backupscinder-backups @ cloudbackupnormal keyring[x] done(not in use, no patch)
codfw1devadminceph @ codfw (mons, osds), backy2 @ cloudvirt, glance-backup @ cloudcontrolnormal keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/738904 https://gerrit.wikimedia.org/r/c/operations/puppet/+/738963
eqiad1adminceph @ eqiad (mons, osds) backy2 @ cloudvirt, glance-backup @ cloudcontrolnormal keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/738904 https://gerrit.wikimedia.org/r/c/operations/puppet/+/738963
codfw1devradosgwradosgw @ cloudcontrolnormal keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/739522
eqiad1radosgwnot in use yetnormal keyring[x] done(not in use, no patch)
codfw1devbootstrap.*ceph @ codfwnormal keyrings[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/742132
eqiad1bootstrap.*ceph @ eqiadnormal keyrings[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/742132
codfw1devmon.*cephmon @ codfwtemp keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/742176
eqiad1mon.*cephmon @ eqiadtemp keyring[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/742176
codfw1devmgr.*cephmon @ codfwnormal keyrings ?[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/744808
eqiad1mgr.*cephmon @ eqiadnormal keyrings ?[x] donehttps://gerrit.wikimedia.org/r/c/operations/puppet/+/744808

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -39
operations/puppetproduction+1 -1
operations/puppetproduction+2 -1
operations/puppetproduction+1 -1
operations/puppetproduction+72 -130
labs/privatemaster+36 -0
labs/privatemaster+12 -0
operations/puppetproduction+1 -1
operations/puppetproduction+92 -52
operations/puppetproduction+10 -1
operations/puppetproduction+9 -1
labs/privatemaster+12 -0
operations/puppetproduction+7 -11
labs/privatemaster+4 -2
operations/puppetproduction+8 -24
labs/privatemaster+2 -0
operations/puppetproduction+0 -5
operations/puppetproduction+39 -7
operations/puppetproduction+53 -49
operations/puppetproduction+28 -90
operations/puppetproduction+6 -5
operations/puppetproduction+3 -0
operations/puppetproduction+35 -51
operations/puppetproduction+12 -1
operations/puppetproduction+0 -12
operations/puppetproduction+1 -1
operations/puppetproduction+20 -30
labs/privatemaster+4 -0
labs/privatemaster+4 -0
operations/puppetproduction+55 -105
operations/puppetproduction+64 -11
operations/puppetproduction+1 -0
operations/puppetproduction+0 -12
operations/puppetproduction+1 -0
operations/puppetproduction+0 -12
operations/puppetproduction+3 -1
operations/puppetproduction+45 -56
operations/puppetproduction+41 -54
operations/puppetproduction+6 -24
operations/puppetproduction+2 -4
operations/puppetproduction+62 -11
operations/puppetproduction+47 -1
operations/puppetproduction+4 -3
operations/puppetproduction+22 -3
operations/puppetproduction+3 -0
operations/puppetproduction+0 -3
operations/puppetproduction+17 -26
operations/puppetproduction+9 -8
operations/puppetproduction+19 -2
operations/puppetproduction+6 -1
operations/puppetproduction+148 -6
operations/puppetproduction+216 -20
operations/puppetproduction+64 -109
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 738383 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: eqiad1: deploy cinder ceph auth creds

https://gerrit.wikimedia.org/r/738383

Change 738411 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] openstakc:codfw1: remove cinder keyring

https://gerrit.wikimedia.org/r/738411

Change 738412 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] openstack::codfw: enable cinder key generation

https://gerrit.wikimedia.org/r/738412

Change 738413 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] openstack::eqiad: Remove cinder key generation from cloudcontrols

https://gerrit.wikimedia.org/r/738413

Change 738414 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] openstack::eqiad: enable cinder keyring generation on control nodes

https://gerrit.wikimedia.org/r/738414

dcaro changed the task status from In Progress to Open.Nov 15 2021, 8:58 AM
dcaro moved this task from Doing to Today on the User-dcaro board.
dcaro moved this task from Today to Doing on the User-dcaro board.
dcaro moved this task from Doing to Today on the User-dcaro board.

Change 738411 merged by David Caro:

[operations/puppet@production] openstack: codfw1dev: remove cinder keyring

https://gerrit.wikimedia.org/r/738411

Change 738412 merged by David Caro:

[operations/puppet@production] openstack: codfw1dev: enable cinder key generation

https://gerrit.wikimedia.org/r/738412

dcaro changed the task status from Open to In Progress.Nov 15 2021, 12:45 PM
dcaro moved this task from Today to Doing on the User-dcaro board.

Change 738413 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: eqiad1: Remove cinder key generation from cloudcontrols

https://gerrit.wikimedia.org/r/738413

Change 738414 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: eqiad1: enable cinder keyring generation on control nodes

https://gerrit.wikimedia.org/r/738414

Change 738903 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph::auth::keyring: allow passing the full client name

https://gerrit.wikimedia.org/r/738903

Change 738904 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: ceph: migrate admin keyring to new abstraction

https://gerrit.wikimedia.org/r/738904

Change 738908 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph::auth::keyring: Generate keyring_path if not passed

https://gerrit.wikimedia.org/r/738908

Change 738904 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: ceph: migrate admin keyring to new abstraction

https://gerrit.wikimedia.org/r/738904

Change 738963 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: ceph: auth: enable admin client

https://gerrit.wikimedia.org/r/738963

Change 738964 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hieradata: ceph: auth: add dummy keydata for the admin client

https://gerrit.wikimedia.org/r/738964

Change 738964 merged by Arturo Borrero Gonzalez:

[labs/private@master] hieradata: ceph: auth: add dummy keydata for the admin client

https://gerrit.wikimedia.org/r/738964

Change 738969 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hieradata: ceph: auth: add dummy keydata for the admin client

https://gerrit.wikimedia.org/r/738969

Change 738969 merged by Arturo Borrero Gonzalez:

[labs/private@master] hieradata: ceph: auth: add dummy keydata for the admin client

https://gerrit.wikimedia.org/r/738969

Change 738963 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: ceph: auth: enable admin client

https://gerrit.wikimedia.org/r/738963

Change 738982 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: ceph: reorder osd/auth profile declaration

https://gerrit.wikimedia.org/r/738982

Change 738982 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: ceph: reorder osd/auth profile declaration

https://gerrit.wikimedia.org/r/738982

dcaro changed the task status from In Progress to Open.Nov 16 2021, 8:30 AM
dcaro moved this task from Doing to Today on the User-dcaro board.

Change 737401 abandoned by David Caro:

[operations/puppet@production] ceph:mon/osd: remove admin class

Reason:

https://gerrit.wikimedia.org/r/737401

Change 738903 merged by David Caro:

[operations/puppet@production] ceph::auth::keyring: allow passing the full client name

https://gerrit.wikimedia.org/r/738903

Change 738908 merged by David Caro:

[operations/puppet@production] ceph::auth::keyring: Generate keyring_path if not passed

https://gerrit.wikimedia.org/r/738908

Change 739127 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph::auth: require load_all when checking keyrings

https://gerrit.wikimedia.org/r/739127

Change 739127 merged by David Caro:

[operations/puppet@production] p:{osd,b_g_images,backy2}: require c::a::deploy when checking keyrings

https://gerrit.wikimedia.org/r/739127

Change 739223 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: nova: factorize libvirt secrets management

https://gerrit.wikimedia.org/r/739223

Change 739228 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] ceph: auth: introduce datatype for configuration hash

https://gerrit.wikimedia.org/r/739228

Change 739235 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: ceph: libvirt: migrate to new ceph auth abstraction

https://gerrit.wikimedia.org/r/739235

Change 739228 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] ceph: auth: introduce datatype for configuration hash

https://gerrit.wikimedia.org/r/739228

Change 739474 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: ceph: client: rbd_libvirt: enable ceph::auth::conf

https://gerrit.wikimedia.org/r/739474

Change 739235 abandoned by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: ceph: libvirt: migrate to new ceph auth abstraction

Reason:

trying with https://gerrit.wikimedia.org/r/c/operations/puppet/+/739474 instead

https://gerrit.wikimedia.org/r/739235

Change 739223 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: nova: factorize libvirt secrets management

https://gerrit.wikimedia.org/r/739223

Change 739474 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: ceph: client: rbd_libvirt: enable ceph::auth::conf

https://gerrit.wikimedia.org/r/739474

Change 739516 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: don't deploy cinder keyring in cloudvirts

https://gerrit.wikimedia.org/r/739516

Change 739516 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: don't deploy cinder keyring in cloudvirts

https://gerrit.wikimedia.org/r/739516

Change 739522 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: radosgw: migrate to new ceph auth abstraction

https://gerrit.wikimedia.org/r/739522

Change 739523 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hieradata: codfw: ceph: add dummy keydata for radosgw

https://gerrit.wikimedia.org/r/739523

Change 739523 merged by Arturo Borrero Gonzalez:

[labs/private@master] hieradata: codfw: ceph: add dummy keydata for radosgw

https://gerrit.wikimedia.org/r/739523

Change 739522 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: radosgw: migrate to new ceph auth abstraction

https://gerrit.wikimedia.org/r/739522

Change 742132 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] ceph: move bootstrap keyring into new auth abstraction

https://gerrit.wikimedia.org/r/742132

Change 742133 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hieradata: ceph: refresh bootstrap auth

https://gerrit.wikimedia.org/r/742133

Change 742175 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] ceph: auth: introduce new parameter 'import_to_ceph'

https://gerrit.wikimedia.org/r/742175

Change 742176 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] ceph: migrate mon auth to the new abstraction

https://gerrit.wikimedia.org/r/742176

Change 742133 merged by Arturo Borrero Gonzalez:

[labs/private@master] hieradata: ceph: refresh bootstrap auth

https://gerrit.wikimedia.org/r/742133

Change 742132 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] ceph: move bootstrap keyring into new auth abstraction

https://gerrit.wikimedia.org/r/742132

Change 742699 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] ceph: auth: introduce keydata for mon.xxxx entries

https://gerrit.wikimedia.org/r/742699

Change 742709 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] ceph: auth: introduce function to calculate keyring_path

https://gerrit.wikimedia.org/r/742709

Change 742699 merged by Arturo Borrero Gonzalez:

[labs/private@master] ceph: auth: introduce keydata for mon.xxxx entries

https://gerrit.wikimedia.org/r/742699

Change 742175 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] ceph: auth: introduce new parameter 'import_to_ceph'

https://gerrit.wikimedia.org/r/742175

Change 742709 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] ceph: auth: introduce function to calculate keyring_path

https://gerrit.wikimedia.org/r/742709

Change 742176 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] ceph: migrate mon auth to the new abstraction

https://gerrit.wikimedia.org/r/742176

Change 744784 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] ceph: mgr: fix typo in relationship

https://gerrit.wikimedia.org/r/744784

Change 744808 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] ceph: mgr: migrate keyring to new auth abstraction

https://gerrit.wikimedia.org/r/744808

Change 744784 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] ceph: mgr: fix typo in relationship

https://gerrit.wikimedia.org/r/744784

Change 745477 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hiera: ceph: add mgr keyrings placeholders

https://gerrit.wikimedia.org/r/745477

Change 745477 merged by Arturo Borrero Gonzalez:

[labs/private@master] hiera: ceph: add mgr keyrings placeholders

https://gerrit.wikimedia.org/r/745477

Change 745478 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hiera: ceph: add dummy caps for mgr auth entries

https://gerrit.wikimedia.org/r/745478

Change 745478 merged by Arturo Borrero Gonzalez:

[labs/private@master] hiera: ceph: add dummy caps for mgr auth entries

https://gerrit.wikimedia.org/r/745478

Change 744808 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] ceph: mgr: migrate keyring to new auth abstraction

https://gerrit.wikimedia.org/r/744808

Change 745767 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] ceph: auth: load_all: fix input datatype

https://gerrit.wikimedia.org/r/745767

Change 745768 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] ceph: auth: load_all: don't fail if only keydata is defined

https://gerrit.wikimedia.org/r/745768

Change 745767 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] ceph: auth: load_all: fix input datatype

https://gerrit.wikimedia.org/r/745767

Change 745768 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] ceph: auth: load_all: don't fail if only keydata is defined

https://gerrit.wikimedia.org/r/745768

Change 745779 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] ceph: auth: eqiad: fix rgw client name

https://gerrit.wikimedia.org/r/745779

Change 745779 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] ceph: auth: eqiad: fix rgw client name

https://gerrit.wikimedia.org/r/745779

After two months, a deep rabbit hole to climb, some frictions within the WMCS team. more than 50 patches, and a severe sidetrack in the WMCS roadmap, I think we could consider this refactor completed.
On the other side, we are (or at least I am) wiser now, gained a lot of knowledge on how ceph works, a couple of interesting code patterns in puppet, and in general, a more robust ceph abstraction. Worth it :-)

Pretty sure there are things that can be improved, but let's do them in a future iteration. Also, the day we bootstrap again a ceph cluster from scratch we may or may not discover a few potential bugs, or race conditions with how keyrings are deployed and how soon ceph needs them in the bootstrap process.

For the future: figure out a way to cleanup old unused keyrings from the ceph internal DB.

Change 737869 abandoned by David Caro:

[operations/puppet@production] p:ceph::client::rbd_cloudcontrol: remove keyring generation

Reason:

obsolete

https://gerrit.wikimedia.org/r/737869