Page MenuHomePhabricator

Refactor OpenStack Puppet to account for Neutron
Closed, ResolvedPublic

Description

Right now we assume Nova everywhere, and we take a a liberal approach mixing up hierarchy and settings. We need to untangle to a certain point to have a sane application of Neutron things, and to refactor for moving nova-api (along with neutron-server) to the labcontrol* servers. Since this will mean some deep refactoring it seems like the time to approach the module/profile/role layout.

We have agreed to:

  • use an openstack2 module to hold module level things that are rewired for new form
  • use a role directory of wmcs to match the current data.yaml designation `wmcs-[roots|admins]'
  • use a base profile mechanism to create several clusters with distinct hiera key trees

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+290 -123
operations/puppetproduction+17 -15
operations/puppetproduction+14 -11
operations/puppetproduction+0 -8
operations/puppetproduction+33 -2
operations/puppetproduction+0 -6
operations/puppetproduction+45 -25
operations/puppetproduction+11 -1
operations/puppetproduction+28 -5
operations/puppetproduction+4 -1
operations/puppetproduction+50 -28
operations/puppetproduction+0 -37
operations/puppetproduction+0 -1
operations/puppetproduction+24 -35
labs/privatemaster+0 -72
operations/puppetproduction+189 -189
operations/puppetproduction+1 -0
operations/puppetproduction+53 -295
operations/puppetproduction+4 -2
operations/puppetproduction+42 -29
operations/puppetproduction+8 -16
operations/puppetproduction+2 -4
operations/puppetproduction+47 -39
operations/puppetproduction+4 -1
operations/puppetproduction+424 -338
operations/puppetproduction+5 -6
operations/puppetproduction+0 -150
operations/puppetproduction+0 -16
operations/puppetproduction+8 -4
operations/puppetproduction+6 -13
operations/puppetproduction+15 -2
operations/puppetproduction+31 -19
operations/puppetproduction+195 -231
operations/puppetproduction+3 -3
operations/puppetproduction+1 K -114
operations/puppetproduction+0 -77
operations/puppetproduction+1 -1
operations/puppetproduction+1 K -378
operations/puppetproduction+0 -3
operations/puppetproduction+279 -18
operations/puppetproduction+3 -2
operations/puppetproduction+534 -305
operations/puppetproduction+33 -46
operations/puppetproduction+4 -1 K
operations/puppetproduction+0 -11 K
operations/puppetproduction+5 -0
operations/puppetproduction+33 -0
operations/puppetproduction+7 -2
operations/puppetproduction+5 -1
operations/puppetproduction+4 -4
operations/puppetproduction+18 -0
operations/puppetproduction+0 -0
operations/puppetproduction+6 -0
operations/puppetproduction+2 -6
operations/puppetproduction+1 -1
operations/puppetproduction+24 -24
operations/puppetproduction+3 -3
operations/puppetproduction+11 -1
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+9 -3
operations/puppetproduction+15 -15
operations/puppetproduction+17 -17
operations/puppetproduction+2 K -824
operations/puppetproduction+2 -5
operations/puppetproduction+0 -6
operations/puppetproduction+0 -3
operations/puppetproduction+3 -3
operations/puppetproduction+167 -473
operations/puppetproduction+48 -29
operations/puppetproduction+25 -119
operations/puppetproduction+3 -6
operations/puppetproduction+4 -4
operations/puppetproduction+302 -165
operations/puppetproduction+2 -1
operations/puppetproduction+0 -1
operations/puppetproduction+3 K -10
operations/puppetproduction+2 -43
operations/puppetproduction+2 -1
operations/puppetproduction+1 -2
operations/puppetproduction+7 -35
operations/puppetproduction+233 -35
operations/puppetproduction+120 -39
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+2 -2
operations/puppetproduction+77 -71
operations/puppetproduction+190 -57
operations/puppetproduction+191 -57
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 376531 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: cleanup keystone references in old module

https://gerrit.wikimedia.org/r/376531

Change 376531 merged by Rush:
[operations/puppet@production] openstack: cleanup keystone references in old module

https://gerrit.wikimedia.org/r/376531

Change 376708 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: re-enable notify and subscribe for nova

https://gerrit.wikimedia.org/r/376708

Change 376848 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: designate as module/profile/role

https://gerrit.wikimedia.org/r/376848

Change 376708 merged by Rush:
[operations/puppet@production] openstack: re-enable notify and subscribe for nova

https://gerrit.wikimedia.org/r/376708

Change 376848 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: designate as module/profile/role

https://gerrit.wikimedia.org/r/376848

Change 376848 merged by Rush:
[operations/puppet@production] openstack: designate as module/profile/role

https://gerrit.wikimedia.org/r/376848

Change 381295 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: pdns auth module/role/profile

https://gerrit.wikimedia.org/r/381295

Change 381445 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: pdns fixup SOA default answer

https://gerrit.wikimedia.org/r/381445

Change 381445 merged by Rush:
[operations/puppet@production] openstack: pdns fixup SOA default answer

https://gerrit.wikimedia.org/r/381445

Change 381295 merged by Rush:
[operations/puppet@production] openstack: pdns auth module/role/profile

https://gerrit.wikimedia.org/r/381295

Change 383710 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: remove pdns auth things for labtestservices200[23]

https://gerrit.wikimedia.org/r/383710

Change 383710 merged by Rush:
[operations/puppet@production] openstack: remove pdns auth things for labtestservices200[23]

https://gerrit.wikimedia.org/r/383710

Change 383909 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: pdns recursor module/profile/role

https://gerrit.wikimedia.org/r/383909

Change 383946 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: cleanup ceilometer files and roles

https://gerrit.wikimedia.org/r/383946

Change 383909 merged by Rush:
[operations/puppet@production] openstack: pdns recursor module/profile/role

https://gerrit.wikimedia.org/r/383909

Change 384038 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: net_standby vs net_secondary for cold spare labnet

https://gerrit.wikimedia.org/r/384038

Change 384038 merged by Rush:
[operations/puppet@production] openstack: net_standby vs net_secondary for cold spare labnet

https://gerrit.wikimedia.org/r/384038

Change 383946 merged by Rush:
[operations/puppet@production] openstack: cleanup ceilometer files and roles

https://gerrit.wikimedia.org/r/383946

Change 384069 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: horizon to module/profile/role

https://gerrit.wikimedia.org/r/384069

Change 384069 merged by Rush:
[operations/puppet@production] openstack: horizon to module/profile/role

https://gerrit.wikimedia.org/r/384069

Change 384517 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: horizon correct wmflabsdotorg_admin lookup path

https://gerrit.wikimedia.org/r/384517

Change 384517 merged by Rush:
[operations/puppet@production] openstack: horizon correct wmflabsdotorg_admin lookup path

https://gerrit.wikimedia.org/r/384517

Change 384566 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: nova manager/osm/wikitech to module/profile/role

https://gerrit.wikimedia.org/r/384566

Change 384566 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: nova manager/osm/wikitech to module/profile/role

https://gerrit.wikimedia.org/r/384566

Change 384566 merged by Rush:
[operations/puppet@production] openstack: nova manager/osm/wikitech to module/profile/role

https://gerrit.wikimedia.org/r/384566

Change 384582 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: nodepool to module/profile/role

https://gerrit.wikimedia.org/r/384582

Change 384582 merged by Rush:
[operations/puppet@production] openstack: nodepool to module/profile/role

https://gerrit.wikimedia.org/r/384582

Change 384585 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: ensure system::role on all role/wmcs

https://gerrit.wikimedia.org/r/384585

Change 384585 merged by Rush:
[operations/puppet@production] openstack: ensure system::role on all role/wmcs

https://gerrit.wikimedia.org/r/384585

Change 384587 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: reuse canonical pdns values for nova network

https://gerrit.wikimedia.org/r/384587

Change 384587 merged by Rush:
[operations/puppet@production] openstack: reuse canonical pdns values for nova network

https://gerrit.wikimedia.org/r/384587

Change 384708 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] labstore: use profile param for observer_pass

https://gerrit.wikimedia.org/r/384708

Change 384708 merged by Rush:
[operations/puppet@production] labstore: use profile param for observer_pass

https://gerrit.wikimedia.org/r/384708

Change 384710 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] mariadb: remove cruft class for old wikitech setup

https://gerrit.wikimedia.org/r/384710

Change 384710 merged by Rush:
[operations/puppet@production] mariadb: remove cruft class for old wikitech setup

https://gerrit.wikimedia.org/r/384710

Change 384712 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: move unused role/labs/dns

https://gerrit.wikimedia.org/r/384712

Change 384712 merged by Rush:
[operations/puppet@production] openstack: remove unused role/labs/dns

https://gerrit.wikimedia.org/r/384712

Change 386386 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] puppetmaster: make hiera lookups class params

https://gerrit.wikimedia.org/r/386386

Change 386386 merged by Rush:
[operations/puppet@production] puppetmaster: make hiera lookups class params

https://gerrit.wikimedia.org/r/386386

Change 386612 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: refactor deployment specific puppetmaster code

https://gerrit.wikimedia.org/r/386612

Change 386612 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: refactor deployment specific puppetmaster code

https://gerrit.wikimedia.org/r/386612

Change 386612 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: refactor deployment specific puppetmaster code

https://gerrit.wikimedia.org/r/386612

Change 386612 merged by Rush:
[operations/puppet@production] openstack: refactor deployment specific puppetmaster code

https://gerrit.wikimedia.org/r/386612

Change 387366 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: labtestpuppetmaster params adjustments

https://gerrit.wikimedia.org/r/387366

Change 387371 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] puppetmaster standalone: pull puppetmaster from hiera

https://gerrit.wikimedia.org/r/387371

Change 387371 merged by Rush:
[operations/puppet@production] puppetmaster standalone: pull puppetmaster from hiera

https://gerrit.wikimedia.org/r/387371

Change 387366 merged by Rush:
[operations/puppet@production] openstack: labtestpuppetmaster params adjustments

https://gerrit.wikimedia.org/r/387366

Change 387433 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] admin: wmcs-roots got lost on labpuppetmaster1002

https://gerrit.wikimedia.org/r/387433

Change 387433 merged by Rush:
[operations/puppet@production] admin: wmcs-roots got lost on labpuppetmaster1002

https://gerrit.wikimedia.org/r/387433

Change 387625 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: move main hiera deployment config to common

https://gerrit.wikimedia.org/r/387625

Change 387625 merged by Rush:
[operations/puppet@production] openstack: move main hiera deployment config to common

https://gerrit.wikimedia.org/r/387625

Change 392062 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: move ferm rules out of site.pp

https://gerrit.wikimedia.org/r/392062

Change 392062 merged by Rush:
[operations/puppet@production] openstack: move ferm rules out of site.pp

https://gerrit.wikimedia.org/r/392062

Change 392087 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: labsaliases extra records move to module/profile/role

https://gerrit.wikimedia.org/r/392087

Change 392087 merged by Rush:
[operations/puppet@production] openstack: labsaliases extra records move to module/profile/role

https://gerrit.wikimedia.org/r/392087

Change 392091 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: cleanup hiera tree for cloud/labs things

https://gerrit.wikimedia.org/r/392091

Change 392091 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: cleanup hiera tree for cloud/labs things

https://gerrit.wikimedia.org/r/392091

Change 392168 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: openstack2 => openstack

https://gerrit.wikimedia.org/r/392168

Change 392091 merged by Rush:
[operations/puppet@production] openstack: cleanup hiera tree for cloud/labs things

https://gerrit.wikimedia.org/r/392091

Change 392440 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: labs_tld is still used by instances

https://gerrit.wikimedia.org/r/392440

Change 392440 merged by Rush:
[operations/puppet@production] openstack: labs_tld is still used by instances

https://gerrit.wikimedia.org/r/392440

Change 392168 merged by Rush:
[operations/puppet@production] openstack: openstack2 => openstack

https://gerrit.wikimedia.org/r/392168

Change 392711 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: dualing regex matches for labtest

https://gerrit.wikimedia.org/r/392711

Change 392711 merged by Rush:
[operations/puppet@production] openstack: consolidate labtest regex matches

https://gerrit.wikimedia.org/r/392711

Change 392859 had a related patch set uploaded (by Rush; owner: cpettet):
[labs/private@master] openstack: remove labtest per host values

https://gerrit.wikimedia.org/r/392859

Change 392859 merged by Rush:
[labs/private@master] openstack: remove labtest per host values

https://gerrit.wikimedia.org/r/392859

Change 392861 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: remove todo for horizon

https://gerrit.wikimedia.org/r/392861

Change 392861 abandoned by Rush:
openstack: remove todo for horizon

https://gerrit.wikimedia.org/r/392861

Change 393600 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: disable notify temporarily

https://gerrit.wikimedia.org/r/393600

Change 393600 merged by Rush:
[operations/puppet@production] openstack: disable notify temporarily

https://gerrit.wikimedia.org/r/393600

This is mostly at a stopping point. There is more hiera cleanup left behind (for instance labstore* and ldap things) but that is not blocking the further work at the moment it feels more urgent to move forward. A big remaining piece here is to test failover for our three scenarios (labcontrol, labservices, labnet) and make sure that functions as we expect and to update docs https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Fail-over. I will sync up with @Andrew for this.

Change 397835 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: first control node dependency issues

https://gerrit.wikimedia.org/r/397835

Change 397835 merged by Rush:
[operations/puppet@production] openstack: first install control node dependency issues

https://gerrit.wikimedia.org/r/397835

Change 397874 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: contain relationship for needed classes

https://gerrit.wikimedia.org/r/397874

Change 397874 merged by Rush:
[operations/puppet@production] openstack: contain relationship for needed classes

https://gerrit.wikimedia.org/r/397874

Change 397903 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: contain classes for dependency handling

https://gerrit.wikimedia.org/r/397903

Change 397903 merged by Rush:
[operations/puppet@production] openstack: contain classes for dependency handling

https://gerrit.wikimedia.org/r/397903

Change 398088 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: cloud repo explicit apt-key update and apt-get update

https://gerrit.wikimedia.org/r/398088

Change 398088 merged by Rush:
[operations/puppet@production] openstack: cloud repo explicit apt-key update and apt-get update

https://gerrit.wikimedia.org/r/398088

Change 398118 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: dedupe packages and reduce require_package

https://gerrit.wikimedia.org/r/398118

Change 398118 merged by Rush:
[operations/puppet@production] openstack: dedupe packages and reduce require_package

https://gerrit.wikimedia.org/r/398118

Change 398121 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: dependency changes for require_package

https://gerrit.wikimedia.org/r/398121

Change 398121 merged by Rush:
[operations/puppet@production] openstack: dependency changes for require_package

https://gerrit.wikimedia.org/r/398121

Change 398169 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: sane class dependency handling for labtest[n]

https://gerrit.wikimedia.org/r/398169

Change 398169 merged by Rush:
[operations/puppet@production] openstack: sane class dependency handling for labtest[n]

https://gerrit.wikimedia.org/r/398169

Change 398266 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: nova::common dependency handled higher up

https://gerrit.wikimedia.org/r/398266

Change 398266 merged by Rush:
[operations/puppet@production] openstack: nova::common dependency handled higher up

https://gerrit.wikimedia.org/r/398266

Change 401742 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] nova: fix dependency order on dir and mount for instances

https://gerrit.wikimedia.org/r/401742

Change 401742 merged by Rush:
[operations/puppet@production] nova: fix dependency order on dir and mount for instances

https://gerrit.wikimedia.org/r/401742

Change 405366 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: nova-network and neutron nova::common split

https://gerrit.wikimedia.org/r/405366

Change 407503 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: move log permissions handling out of nova::common

https://gerrit.wikimedia.org/r/407503

Change 407503 merged by Rush:
[operations/puppet@production] openstack: move log permissions handling out of nova::common

https://gerrit.wikimedia.org/r/407503

Change 405366 merged by Rush:
[operations/puppet@production] openstack: nova-network and neutron nova::common split

https://gerrit.wikimedia.org/r/405366

There is a lot more cleanup here, especially on the edges with LDAP and actual tenant puppet coe organization but the goal of feeling like the openstack deployment provider setup is ready for htis work is really done.