Page MenuHomePhabricator

upgrade cloud-vps openstack to Openstack version 'Stein'
Closed, ResolvedPublic

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+7 -5
operations/puppetproduction+2 -2
operations/puppetproduction+0 -1
operations/puppetproduction+0 -5
operations/puppetproduction+13 -0
operations/puppetproduction+2 -2
operations/puppetproduction+10 -1
operations/puppetproduction+1 -5
operations/puppetproduction+1 -1
operations/puppetproduction+252 -5
operations/puppetproduction+17 -30
operations/puppetproduction+4 -2
operations/puppetproduction+0 -865
operations/puppetproduction+4 -4
operations/puppetproduction+0 -2
operations/puppetproduction+0 -1
operations/puppetproduction+1 -69
operations/puppetproduction+31 -21
operations/puppetproduction+255 -9
operations/puppetproduction+240 -0
operations/puppetproduction+5 -103
operations/puppetproduction+38 -0
operations/puppetproduction+8 -3
operations/puppetproduction+0 -0
operations/puppetproduction+18 -0
operations/puppetproduction+41 -1
operations/puppetproduction+263 -0
operations/puppetproduction+180 -1 K
operations/puppetproduction+341 -0
operations/puppetproduction+70 -0
operations/puppetproduction+56 -24
operations/puppetproduction+118 -8 K
operations/puppetproduction+116 -148
operations/puppetproduction+7 -5
operations/puppetproduction+4 -0
operations/puppetproduction+2 -2
operations/puppetproduction+33 -1 K
operations/puppetproduction+107 -0
operations/puppetproduction+78 -0
operations/puppetproduction+25 K -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
ResolvedMajavah
ResolvedAndrew
ResolvedAndrew
Resolveddcaro
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedCmjohnson
Resolvedayounsi
Resolvedaborrero
ResolvedCmjohnson
ResolvedJclark-ctr
ResolvedJclark-ctr
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
Resolveddcaro
Resolvedaborrero
Declineddcaro
Resolveddcaro
OpenNone
OpenNone
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedBstorm
Resolvedaborrero
Resolvedaborrero

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 641230 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add initial config files for version Stein

https://gerrit.wikimedia.org/r/641230

Change 641231 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add server packages for version Stein

https://gerrit.wikimedia.org/r/641231

Change 641232 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Designate: updates for version Stein

https://gerrit.wikimedia.org/r/641232

Change 641233 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add client packages for Stein

https://gerrit.wikimedia.org/r/641233

Change 641230 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add initial config files for version Stein

https://gerrit.wikimedia.org/r/641230

Change 641231 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add server packages for version Stein

https://gerrit.wikimedia.org/r/641231

Change 641233 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add client packages for Stein

https://gerrit.wikimedia.org/r/641233

Change 641232 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Designate: updates for version Stein

https://gerrit.wikimedia.org/r/641232

Change 642889 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Codfw1dev designate: move to version 'Stein'

https://gerrit.wikimedia.org/r/642889

Change 642889 merged by Andrew Bogott:
[operations/puppet@production] Codfw1dev designate: move to version 'Stein'

https://gerrit.wikimedia.org/r/642889

Change 642898 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] eqiad1 designate: move to version 'Stein'

https://gerrit.wikimedia.org/r/642898

Change 642898 merged by Andrew Bogott:
[operations/puppet@production] eqiad1 designate: move to version 'Stein'

https://gerrit.wikimedia.org/r/642898

Since the cloudservices nodes are already running Buster, I upgraded them to Stein just now. All seems well for the moment.

Change 644534 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova config upgrades for Stein

https://gerrit.wikimedia.org/r/644534

Change 644534 merged by Andrew Bogott:
[operations/puppet@production] Nova config upgrades for Stein

https://gerrit.wikimedia.org/r/644534

Change 651668 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: update stein l3 packages

https://gerrit.wikimedia.org/r/651668

Change 651669 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: apply our local dmz hacks for Stein

https://gerrit.wikimedia.org/r/651669

Change 651683 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: add Stein manifests

https://gerrit.wikimedia.org/r/651683

Change 651684 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Glance: add Stein versions of manifests

https://gerrit.wikimedia.org/r/651684

Change 651685 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cinder: add Stein version of service manifest

https://gerrit.wikimedia.org/r/651685

Change 651686 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: Add Stein service manifests

https://gerrit.wikimedia.org/r/651686

Change 651687 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron service manifests for Stein

https://gerrit.wikimedia.org/r/651687

Change 651688 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Barbican: add service manifest for Stein

https://gerrit.wikimedia.org/r/651688

Change 651689 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add OpenStack client package manifests for Stein

https://gerrit.wikimedia.org/r/651689

Change 651691 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack codfw1dev -> Stein

https://gerrit.wikimedia.org/r/651691

Change 651668 merged by Andrew Bogott:
[operations/puppet@production] Neutron: update stein l3 python files

https://gerrit.wikimedia.org/r/651668

Change 651683 merged by Andrew Bogott:
[operations/puppet@production] Nova: add Stein manifests

https://gerrit.wikimedia.org/r/651683

Change 651669 merged by Andrew Bogott:
[operations/puppet@production] Neutron: apply our local dmz hacks for Stein

https://gerrit.wikimedia.org/r/651669

Change 651684 merged by Andrew Bogott:
[operations/puppet@production] Glance: add Stein versions of manifests

https://gerrit.wikimedia.org/r/651684

Change 651685 merged by Andrew Bogott:
[operations/puppet@production] Cinder: add Stein version of service manifest

https://gerrit.wikimedia.org/r/651685

Change 651686 merged by Andrew Bogott:
[operations/puppet@production] Keystone: Add Stein service manifests

https://gerrit.wikimedia.org/r/651686

Change 651687 merged by Andrew Bogott:
[operations/puppet@production] Neutron service manifests for Stein

https://gerrit.wikimedia.org/r/651687

Change 651688 merged by Andrew Bogott:
[operations/puppet@production] Barbican: add service manifest for Stein

https://gerrit.wikimedia.org/r/651688

Change 651689 merged by Andrew Bogott:
[operations/puppet@production] Add OpenStack client package manifests for Stein

https://gerrit.wikimedia.org/r/651689

Change 651812 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Keystone: move ldap-common-rocky-fixed.py into files/rocky

https://gerrit.wikimedia.org/r/651812

Change 651812 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Keystone: move ldap-common-rocky-fixed.py into files/rocky

https://gerrit.wikimedia.org/r/651812

Change 651691 merged by Andrew Bogott:
[operations/puppet@production] OpenStack codfw1dev -> Stein

https://gerrit.wikimedia.org/r/651691

Change 651820 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add some resource files for Cinder/Stein

https://gerrit.wikimedia.org/r/651820

Change 651820 merged by Andrew Bogott:
[operations/puppet@production] Add some resource files for Cinder/Stein

https://gerrit.wikimedia.org/r/651820

Change 651822 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Glance: remove glance-registry service from Stein deploys

https://gerrit.wikimedia.org/r/651822

Change 651822 merged by Andrew Bogott:
[operations/puppet@production] Glance: remove glance-registry service from Stein deploys

https://gerrit.wikimedia.org/r/651822

Change 651825 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cinder: add a puppetized api init script in Stein

https://gerrit.wikimedia.org/r/651825

Change 651825 merged by Andrew Bogott:
[operations/puppet@production] Cinder: add a puppetized api init script in Stein

https://gerrit.wikimedia.org/r/651825

Change 651829 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova-api: add a puppetized api init script in Stein

https://gerrit.wikimedia.org/r/651829

Change 651829 merged by Andrew Bogott:
[operations/puppet@production] Nova-api: add a puppetized api init script in Stein

https://gerrit.wikimedia.org/r/651829

Change 651840 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfkeystonehooks: Update the monkeypatch that renames project IDs for Stein

https://gerrit.wikimedia.org/r/651840

Change 651840 merged by Andrew Bogott:
[operations/puppet@production] wmfkeystonehooks: Update the monkeypatch that renames project IDs for Stein

https://gerrit.wikimedia.org/r/651840

Change 651875 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: use the keystone-wsgi-admin script installed by the package

https://gerrit.wikimedia.org/r/651875

Change 651876 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: update wmtotp plugin to match upstream Password auth code

https://gerrit.wikimedia.org/r/651876

Change 651875 merged by Andrew Bogott:
[operations/puppet@production] Keystone: use the keystone-wsgi-admin script installed by the package

https://gerrit.wikimedia.org/r/651875

Change 651876 merged by Andrew Bogott:
[operations/puppet@production] Keystone: update wmtotp plugin to match upstream Password auth code

https://gerrit.wikimedia.org/r/651876

Change 651877 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove [cors] sections that produce deprecation warnings

https://gerrit.wikimedia.org/r/651877

Change 651878 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: update password_safelist to match upstream code

https://gerrit.wikimedia.org/r/651878

Change 651877 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove [cors] sections that produce deprecation warnings

https://gerrit.wikimedia.org/r/651877

Change 651878 merged by Andrew Bogott:
[operations/puppet@production] Keystone: update password_safelist to match upstream code

https://gerrit.wikimedia.org/r/651878

Change 651880 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove token_formatters-fixed.py backport

https://gerrit.wikimedia.org/r/651880

Change 651880 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove token_formatters-fixed.py backport

https://gerrit.wikimedia.org/r/651880

Change 652832 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: update otp auth code for Stein

https://gerrit.wikimedia.org/r/652832

Change 652832 merged by Andrew Bogott:
[operations/puppet@production] Keystone: update otp auth code for Stein

https://gerrit.wikimedia.org/r/652832

Change 653667 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: modernize api-paste.ini.erb

https://gerrit.wikimedia.org/r/653667

Change 653667 merged by Andrew Bogott:
[operations/puppet@production] Nova: modernize api-paste.ini.erb

https://gerrit.wikimedia.org/r/653667

Change 654044 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova: puppetize the nova-api-metadata service

https://gerrit.wikimedia.org/r/654044

Change 654044 merged by Andrew Bogott:
[operations/puppet@production] nova: puppetize the nova-api-metadata service

https://gerrit.wikimedia.org/r/654044

Change 655977 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode for Stein upgrade

https://gerrit.wikimedia.org/r/655977

Change 655979 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'rocky' to 'stein'

https://gerrit.wikimedia.org/r/655979

Change 655977 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode for Stein upgrade

https://gerrit.wikimedia.org/r/655977

Change 655979 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'rocky' to 'stein'

https://gerrit.wikimedia.org/r/655979

Here is the record of the upgrade:

  • update IRC topic
  • downtime everything in icinga through 14:00CDT
    • aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab*
    • aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
  • downtime Horizon with https://gerrit.wikimedia.org/r/c/operations/puppet/+/655977
  • disable puppet on all cloud* hosts

$ sudo cumin 'cloud*' "disable-puppet 'Upgrading to openstack Stein - T261134 - ${USER}'"

  • merge puppet patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/655979
  • dump databases on cloudcontrol1003: nova_eqiad1 , nova_api_eqiad1, nova_cell0_eqiad1, neutron, glance, keystone:
    1. mysqldump -u root nova_eqiad1 > /root/steindbbackups/nova_eqiad1.sql
    2. mysqldump -u root nova_api_eqiad1 > /root/steindbbackups/nova_api_eqiad1.sql
    3. mysqldump -u root nova_cell0_eqiad1 > /root/steindbbackups/nova_cell0_eqiad1.sql
    4. mysqldump -u root neutron > /root/steindbbackups/neutron.sql
    5. mysqldump -u root glance > /root/steindbbackups/glance.sql
    6. mysqldump -u root keystone > /root/steindbbackups/keystone.sql

Cloudcontrols:

Cloudcontrol1003:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • nova-manage api_db sync
  • nova-manage db sync
  • glance-manage db_sync
  • keystone-manage db_sync
  • puppet agent -tv
  • nova-manage db online_data_migrations

Cloudcontrol1004:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • puppet agent -tv

Cloudcontrol1005:

  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • puppet agent -tv

cloudnets (one at a time please):

Begin with the standby node, as determined with:

$ neutron l3-agent-list-hosting-router cloudinstances2b-gw

Standby node:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • chgrp -R neutron /var/lib/neutron/* && chown -R neutron /var/lib/neutron/*
  • service neutron-l3-agent restart
  • neutron-db-manage upgrade heads on cloudcontrol1003

Active node:

  • service neutron-l3-agent stop # should cause an immediate failover to the already-upgraded standby node
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • chgrp -R neutron /var/lib/neutron/* && chown -R neutron /var/lib/neutron/*
  • service neutron-l3-agent restart

Break/meeting, whatever

Cloudvirts (start with one test host first, cloudvirt1039. Don't forget about cloudvirtwdqs ):

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-linuxbridge-agent restart
  • service nova-compute restart
  • service libvirtd restart
  • update IRC topic

Mentioned in SAL (#wikimedia-cloud) [2021-01-14T17:58:24Z] <bstorm> hard rebooting tools-sgecron-01 following network issues during upgrade to stein T261134

Change 656225 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova monitoring: update the egrep string for the nova-api process

https://gerrit.wikimedia.org/r/656225

Change 656225 merged by Andrew Bogott:
[operations/puppet@production] Nova monitoring: update the egrep string for the nova-api process

https://gerrit.wikimedia.org/r/656225

Change 656231 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-api monitoring: simplify grep for detecting processes

https://gerrit.wikimedia.org/r/656231

Change 656231 merged by Andrew Bogott:
[operations/puppet@production] nova-api monitoring: simplify grep for detecting processes

https://gerrit.wikimedia.org/r/656231

Change 656237 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone/stein: disable the packaged keystone init script

https://gerrit.wikimedia.org/r/656237

Change 656237 merged by Andrew Bogott:
[operations/puppet@production] Keystone/stein: disable the packaged keystone init script

https://gerrit.wikimedia.org/r/656237

Change 656240 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: remove obsolete service definition

https://gerrit.wikimedia.org/r/656240

Change 656240 merged by Andrew Bogott:
[operations/puppet@production] keystone: remove obsolete service definition

https://gerrit.wikimedia.org/r/656240

Change 656514 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] openstack: remove the queens hiera hiding out in places

https://gerrit.wikimedia.org/r/656514

Change 656514 merged by Bstorm:
[operations/puppet@production] openstack: remove the queens hiera hiding out in places

https://gerrit.wikimedia.org/r/656514

Change 657870 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Move eqiad1/horizon to openstack train

https://gerrit.wikimedia.org/r/657870

Change 657870 merged by Andrew Bogott:
[operations/puppet@production] Move eqiad1/horizon to openstack train

https://gerrit.wikimedia.org/r/657870

Change 660085 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron-linuxbridge hack: restart service after hack is applied

https://gerrit.wikimedia.org/r/660085

Change 660085 merged by Andrew Bogott:
[operations/puppet@production] neutron-linuxbridge hack: restart service after hack is applied

https://gerrit.wikimedia.org/r/660085