Page MenuHomePhabricator

refactor openstack puppet code to use lists of servers
Closed, ResolvedPublic

Description

Most of our current puppet code expects our servers to be in pairs, a primary and secondary. Now that we have proper HA support in nearly all cases, there's no reason to think of things this way -- instead, it should all be list-based. So, for example, instead of

profile::openstack::eqiad1::nova_controller: 'cloudcontrol1003.wikimedia.org'
profile::openstack::eqiad1::nova_controller_standby: 'cloudcontrol1004.wikimedia.org'

we should have something like

profile::openstack::eqiad1::nova_controllers:
- 'cloudcontrol1003.wikimedia.org'
- 'cloudcontrol1004.wikimedia.org'

This will be a big refactor but should be mostly no-ops that we can validate with the PCC.

The only exception will be glance, which still has a primary/secondary image store; we should probably add special-case code for that so it can be easily extracted once the backend moves to ceph.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+94 -89
operations/puppetproduction+1 -1
operations/puppetproduction+0 -4
operations/puppetproduction+6 -5
operations/puppetproduction+6 -6
operations/puppetproduction+6 -8
operations/puppetproduction+45 -43
operations/puppetproduction+18 -20
operations/puppetproduction+0 -2
operations/puppetproduction+0 -8
operations/puppetproduction+1 -9
operations/puppetproduction+0 -2
operations/puppetproduction+14 -9
operations/puppetproduction+0 -16
operations/puppetproduction+10 -10
operations/puppetproduction+66 -76
operations/puppetproduction+5 -8
operations/puppetproduction+15 -15
operations/puppetproduction+10 -17
operations/puppetproduction+11 -18
operations/puppetproduction+10 -19
operations/puppetproduction+0 -2
operations/puppetproduction+33 -37
operations/puppetproduction+8 -14
operations/puppetproduction+5 -6
operations/puppetproduction+1 -15
operations/puppetproduction+9 -22
operations/puppetproduction+7 -7
operations/puppetproduction+2 -7
operations/puppetproduction+5 -6
operations/puppetproduction+0 -1
operations/puppetproduction+72 -80
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+9 -0
operations/puppetproduction+1 -1
operations/puppetproduction+2 -2
operations/puppetproduction+13 -10
operations/puppetproduction+1 -0
operations/puppetproduction+69 -28
operations/puppetproduction+91 -4
operations/puppetproduction+1 -1
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+17 -17
operations/puppetproduction+139 -177
operations/puppetproduction+1 -1
operations/puppetproduction+217 -301
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 588752 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: replace standalone memcached with a mcrouter cluster

https://gerrit.wikimedia.org/r/588752

Change 589095 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps hiera: introduce openstack_controllers and keystone_api_fqdn

https://gerrit.wikimedia.org/r/589095

Change 589096 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] glance image_sync: use primary_glance_image_store to choose the image store

https://gerrit.wikimedia.org/r/589096

Change 589112 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] glance profiles: remove use of $nova_controller param

https://gerrit.wikimedia.org/r/589112

Change 589143 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Glance: use keystone_api_fqdn for endpoints

https://gerrit.wikimedia.org/r/589143

Change 589095 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps hiera: introduce openstack_controllers and keystone_api_fqdn

https://gerrit.wikimedia.org/r/589095

Change 588752 merged by Andrew Bogott:
[operations/puppet@production] Designate: replace standalone memcached with a mcrouter cluster

https://gerrit.wikimedia.org/r/588752

Change 589630 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate mcrouter: add a default_policy to the route

https://gerrit.wikimedia.org/r/589630

Change 589630 merged by Andrew Bogott:
[operations/puppet@production] designate mcrouter: add a default_policy to the route

https://gerrit.wikimedia.org/r/589630

Change 589652 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate/mcrouter: another attempt at mcrouter config

https://gerrit.wikimedia.org/r/589652

Change 589652 merged by Andrew Bogott:
[operations/puppet@production] Designate/mcrouter: another attempt at mcrouter config

https://gerrit.wikimedia.org/r/589652

Change 589657 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: further (largely random) mcrouter config tweak

https://gerrit.wikimedia.org/r/589657

Change 589657 merged by Andrew Bogott:
[operations/puppet@production] designate: further (largely random) mcrouter config tweak

https://gerrit.wikimedia.org/r/589657

Change 589664 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate mcrouter: talk to memcached on port 11000

https://gerrit.wikimedia.org/r/589664

Change 589664 merged by Andrew Bogott:
[operations/puppet@production] Designate mcrouter: talk to memcached on port 11000

https://gerrit.wikimedia.org/r/589664

Change 589676 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate/mcrouter: all each mcrouter to talk to each memcached

https://gerrit.wikimedia.org/r/589676

Change 589676 merged by Andrew Bogott:
[operations/puppet@production] designate/mcrouter: all each mcrouter to talk to each memcached

https://gerrit.wikimedia.org/r/589676

Change 589678 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate/mcrouter: typo fix

https://gerrit.wikimedia.org/r/589678

Change 589678 merged by Andrew Bogott:
[operations/puppet@production] Designate/mcrouter: typo fix

https://gerrit.wikimedia.org/r/589678

Change 589694 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate/mcrouter: use ipv4 address for servers rather than fqdn

https://gerrit.wikimedia.org/r/589694

Change 589694 merged by Andrew Bogott:
[operations/puppet@production] designate/mcrouter: use ipv4 address for servers rather than fqdn

https://gerrit.wikimedia.org/r/589694

Change 589096 merged by Andrew Bogott:
[operations/puppet@production] glance image_sync: use primary_glance_image_store to choose the image store

https://gerrit.wikimedia.org/r/589096

Change 589797 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] glance image_sync: don't make a file depend on itself!

https://gerrit.wikimedia.org/r/589797

Change 589797 merged by Andrew Bogott:
[operations/puppet@production] glance image_sync: don't make a file depend on itself!

https://gerrit.wikimedia.org/r/589797

Change 589814 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] glance image sync: simplify secondary image handling

https://gerrit.wikimedia.org/r/589814

Change 589814 merged by Andrew Bogott:
[operations/puppet@production] glance image sync: simplify secondary image handling

https://gerrit.wikimedia.org/r/589814

Change 589112 merged by Andrew Bogott:
[operations/puppet@production] glance profiles: remove use of $nova_controller param

https://gerrit.wikimedia.org/r/589112

Change 589143 merged by Andrew Bogott:
[operations/puppet@production] Glance: use keystone_api_fqdn for endpoints

https://gerrit.wikimedia.org/r/589143

Change 589858 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova common: replace nova_controller and nova_controller_standby with openstack_controllers

https://gerrit.wikimedia.org/r/589858

Change 589858 merged by Andrew Bogott:
[operations/puppet@production] nova common: replace nova_controller and nova_controller_standby

https://gerrit.wikimedia.org/r/589858

Change 590587 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova project spreadcheck: run on all openstack_controllers

https://gerrit.wikimedia.org/r/590587

Change 590588 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone::fernet_keys profile: use openstack_controllers from hiera

https://gerrit.wikimedia.org/r/590588

Change 590589 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone service: stop using nova_controller and nova_controller_standby

https://gerrit.wikimedia.org/r/590589

Change 590587 merged by Andrew Bogott:
[operations/puppet@production] Nova project spreadcheck: run on all openstack_controllers

https://gerrit.wikimedia.org/r/590587

Change 590588 merged by Andrew Bogott:
[operations/puppet@production] keystone::fernet_keys profile: use openstack_controllers from hiera

https://gerrit.wikimedia.org/r/590588

Change 590589 merged by Andrew Bogott:
[operations/puppet@production] Keystone service: stop using nova_controller and nova_controller_standby

https://gerrit.wikimedia.org/r/590589

Change 590813 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] rabbitmq: use openstack_controllers instead of nova_controller

https://gerrit.wikimedia.org/r/590813

Change 590814 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] openstack::designate::service: remove some unused internal variables

https://gerrit.wikimedia.org/r/590814

Change 590815 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: remove use of nova_controller and nova_controller_standby

https://gerrit.wikimedia.org/r/590815

Change 590822 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate firewall: use openstack_controllers instead of nova-specific things

https://gerrit.wikimedia.org/r/590822

Change 590823 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack env scripts: use keystone_api_fqdn

https://gerrit.wikimedia.org/r/590823

Change 590824 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova common: use keystone_api_fqdn instead of keystone_host

https://gerrit.wikimedia.org/r/590824

Change 590813 merged by Andrew Bogott:
[operations/puppet@production] rabbitmq: use openstack_controllers instead of nova_controller

https://gerrit.wikimedia.org/r/590813

Change 590814 merged by Andrew Bogott:
[operations/puppet@production] openstack::designate::service: remove some unused internal variables

https://gerrit.wikimedia.org/r/590814

Change 590815 merged by Andrew Bogott:
[operations/puppet@production] Designate: remove use of nova_controller and nova_controller_standby

https://gerrit.wikimedia.org/r/590815

Change 590822 merged by Andrew Bogott:
[operations/puppet@production] designate firewall: use openstack_controllers instead of nova-specific things

https://gerrit.wikimedia.org/r/590822

Change 590823 merged by Andrew Bogott:
[operations/puppet@production] Openstack env scripts: use keystone_api_fqdn

https://gerrit.wikimedia.org/r/590823

Change 590824 merged by Andrew Bogott:
[operations/puppet@production] nova common: use keystone_api_fqdn instead of keystone_host

https://gerrit.wikimedia.org/r/590824

WOW this is turning out to be a real mountain of patches. @JHedden can I interest you in doing the HA and/or Neutron pieces of this? (I can keep at it if you're still engrossed in monitoring things.)

Change 591062 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] mariadb wmcs ferm: use openstack_controllers instead of nova_controller etc.

https://gerrit.wikimedia.org/r/591062

Change 591063 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] mariadb wmcs ferm: update ::wmcs profiles to use types and lookup()

https://gerrit.wikimedia.org/r/591063

Change 591082 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] profile::openstack::codfw1dev::db: tidy up firewall rules

https://gerrit.wikimedia.org/r/591082

Change 591086 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps puppetmasters: use openstack_controllers instead of nova_controller

https://gerrit.wikimedia.org/r/591086

Change 591082 merged by Andrew Bogott:
[operations/puppet@production] profile::openstack::codfw1dev::db: tidy up firewall rules

https://gerrit.wikimedia.org/r/591082

Change 591102 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron common: use openstack_controllers instead of nova_controller params

https://gerrit.wikimedia.org/r/591102

Change 591125 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron metadata agent: use keystone_api_fqdn for the metadata api

https://gerrit.wikimedia.org/r/591125

Change 591102 merged by Andrew Bogott:
[operations/puppet@production] Neutron common: use openstack_controllers instead of nova_controller params

https://gerrit.wikimedia.org/r/591102

Change 591133 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs openstack haproxy: use openstack_controllers instead of nova_controller

https://gerrit.wikimedia.org/r/591133

Change 591125 merged by Andrew Bogott:
[operations/puppet@production] Neutron metadata agent: use keystone_api_fqdn for the metadata api

https://gerrit.wikimedia.org/r/591125

Change 591155 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Remove nova_api_host params from places where it wasn't doing anything

https://gerrit.wikimedia.org/r/591155

Change 591156 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova fullstack: use openstack_controllers instead of nova_api_host

https://gerrit.wikimedia.org/r/591156

Change 591158 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs: remove nova_api_host

https://gerrit.wikimedia.org/r/591158

Change 591155 merged by Andrew Bogott:
[operations/puppet@production] Remove nova_api_host params from places where it wasn't doing anything

https://gerrit.wikimedia.org/r/591155

Change 591156 merged by Andrew Bogott:
[operations/puppet@production] nova fullstack: use openstack_controllers instead of nova_api_host

https://gerrit.wikimedia.org/r/591156

Change 591158 merged by Andrew Bogott:
[operations/puppet@production] wmcs: remove nova_api_host

https://gerrit.wikimedia.org/r/591158

Change 591167 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova.conf: remove [glance] settings

https://gerrit.wikimedia.org/r/591167

Change 591167 merged by Andrew Bogott:
[operations/puppet@production] nova.conf: remove [glance] settings

https://gerrit.wikimedia.org/r/591167

Change 591173 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: remove glance_api firewall rule

https://gerrit.wikimedia.org/r/591173

Change 591174 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs hiera: remove glance_host values

https://gerrit.wikimedia.org/r/591174

Change 591173 merged by Andrew Bogott:
[operations/puppet@production] keystone: remove glance_api firewall rule

https://gerrit.wikimedia.org/r/591173

Change 591174 merged by Andrew Bogott:
[operations/puppet@production] wmcs hiera: remove glance_host values

https://gerrit.wikimedia.org/r/591174

JHedden triaged this task as Medium priority.Apr 21 2020, 4:30 PM

Change 591133 merged by Andrew Bogott:
[operations/puppet@production] wmcs openstack haproxy: use openstack_controllers instead of nova_controller

https://gerrit.wikimedia.org/r/591133

Change 591086 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps puppetmasters: use openstack_controllers instead of nova_controller

https://gerrit.wikimedia.org/r/591086

Change 591062 merged by Andrew Bogott:
[operations/puppet@production] mariadb wmcs ferm: use openstack_controllers instead of nova_controller etc.

https://gerrit.wikimedia.org/r/591062

Change 591063 merged by Andrew Bogott:
[operations/puppet@production] mariadb wmcs ferm: update ::wmcs profiles to use types and lookup()

https://gerrit.wikimedia.org/r/591063

Change 591410 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] glance_seed.sh.erb: update to use openstack_controllers

https://gerrit.wikimedia.org/r/591410

Change 591411 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: remove nova_controller and nova_controller_standby from hiera

https://gerrit.wikimedia.org/r/591411

Change 591410 merged by Andrew Bogott:
[operations/puppet@production] glance_seed.sh.erb: update to use openstack_controllers

https://gerrit.wikimedia.org/r/591410

Change 591411 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: remove nova_controller and nova_controller_standby from hiera

https://gerrit.wikimedia.org/r/591411

Change 592184 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps puppetmaster: add a missing )

https://gerrit.wikimedia.org/r/592184

Change 592184 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps puppetmaster: add a missing )

https://gerrit.wikimedia.org/r/592184

Andrew closed this task as Resolved.Apr 23 2020, 4:54 PM
Andrew claimed this task.

This is as done. There are still a few ways that cloudcontrol1003 is different from 1004:

  • Glance only runs on 1003. This is pending having a ceph backend for the image store
  • Some monitoring things only run on 1003. Those are handled via openstack_controllers[0] so when we replace cloudcontrol1003 the monitoring should shift over gracefully.

Change 593035 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs pdns: use a 'hosts' list for auth resolvers and recursors

https://gerrit.wikimedia.org/r/593035

Change 593035 merged by Andrew Bogott:
[operations/puppet@production] wmcs pdns: stop using primary/secondary language for resolvers and recursors

https://gerrit.wikimedia.org/r/593035