Details
Event Timeline
Change 641230 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add initial config files for version Stein
Change 641231 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add server packages for version Stein
Change 641232 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Designate: updates for version Stein
Change 641233 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add client packages for Stein
Change 641230 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add initial config files for version Stein
Change 641231 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add server packages for version Stein
Change 641233 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add client packages for Stein
Change 641232 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Designate: updates for version Stein
Change 642889 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Codfw1dev designate: move to version 'Stein'
Change 642889 merged by Andrew Bogott:
[operations/puppet@production] Codfw1dev designate: move to version 'Stein'
Change 642898 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] eqiad1 designate: move to version 'Stein'
Change 642898 merged by Andrew Bogott:
[operations/puppet@production] eqiad1 designate: move to version 'Stein'
Since the cloudservices nodes are already running Buster, I upgraded them to Stein just now. All seems well for the moment.
Change 644534 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova config upgrades for Stein
Change 644534 merged by Andrew Bogott:
[operations/puppet@production] Nova config upgrades for Stein
Change 651668 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: update stein l3 packages
Change 651669 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: apply our local dmz hacks for Stein
Change 651683 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: add Stein manifests
Change 651684 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Glance: add Stein versions of manifests
Change 651685 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cinder: add Stein version of service manifest
Change 651686 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: Add Stein service manifests
Change 651687 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron service manifests for Stein
Change 651688 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Barbican: add service manifest for Stein
Change 651689 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add OpenStack client package manifests for Stein
Change 651691 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack codfw1dev -> Stein
Change 651668 merged by Andrew Bogott:
[operations/puppet@production] Neutron: update stein l3 python files
Change 651683 merged by Andrew Bogott:
[operations/puppet@production] Nova: add Stein manifests
Change 651669 merged by Andrew Bogott:
[operations/puppet@production] Neutron: apply our local dmz hacks for Stein
Change 651684 merged by Andrew Bogott:
[operations/puppet@production] Glance: add Stein versions of manifests
Change 651685 merged by Andrew Bogott:
[operations/puppet@production] Cinder: add Stein version of service manifest
Change 651686 merged by Andrew Bogott:
[operations/puppet@production] Keystone: Add Stein service manifests
Change 651687 merged by Andrew Bogott:
[operations/puppet@production] Neutron service manifests for Stein
Change 651688 merged by Andrew Bogott:
[operations/puppet@production] Barbican: add service manifest for Stein
Change 651689 merged by Andrew Bogott:
[operations/puppet@production] Add OpenStack client package manifests for Stein
Change 651812 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Keystone: move ldap-common-rocky-fixed.py into files/rocky
Change 651812 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Keystone: move ldap-common-rocky-fixed.py into files/rocky
Change 651691 merged by Andrew Bogott:
[operations/puppet@production] OpenStack codfw1dev -> Stein
Change 651820 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add some resource files for Cinder/Stein
Change 651820 merged by Andrew Bogott:
[operations/puppet@production] Add some resource files for Cinder/Stein
Change 651822 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Glance: remove glance-registry service from Stein deploys
Change 651822 merged by Andrew Bogott:
[operations/puppet@production] Glance: remove glance-registry service from Stein deploys
Change 651825 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cinder: add a puppetized api init script in Stein
Change 651825 merged by Andrew Bogott:
[operations/puppet@production] Cinder: add a puppetized api init script in Stein
Change 651829 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova-api: add a puppetized api init script in Stein
Change 651829 merged by Andrew Bogott:
[operations/puppet@production] Nova-api: add a puppetized api init script in Stein
Change 651840 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfkeystonehooks: Update the monkeypatch that renames project IDs for Stein
Change 651840 merged by Andrew Bogott:
[operations/puppet@production] wmfkeystonehooks: Update the monkeypatch that renames project IDs for Stein
Change 651875 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: use the keystone-wsgi-admin script installed by the package
Change 651876 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: update wmtotp plugin to match upstream Password auth code
Change 651875 merged by Andrew Bogott:
[operations/puppet@production] Keystone: use the keystone-wsgi-admin script installed by the package
Change 651876 merged by Andrew Bogott:
[operations/puppet@production] Keystone: update wmtotp plugin to match upstream Password auth code
Change 651877 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove [cors] sections that produce deprecation warnings
Change 651878 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: update password_safelist to match upstream code
Change 651877 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove [cors] sections that produce deprecation warnings
Change 651878 merged by Andrew Bogott:
[operations/puppet@production] Keystone: update password_safelist to match upstream code
Change 651880 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove token_formatters-fixed.py backport
Change 651880 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove token_formatters-fixed.py backport
Change 652832 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: update otp auth code for Stein
Change 652832 merged by Andrew Bogott:
[operations/puppet@production] Keystone: update otp auth code for Stein
Change 653667 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: modernize api-paste.ini.erb
Change 653667 merged by Andrew Bogott:
[operations/puppet@production] Nova: modernize api-paste.ini.erb
Change 654044 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova: puppetize the nova-api-metadata service
Change 654044 merged by Andrew Bogott:
[operations/puppet@production] nova: puppetize the nova-api-metadata service
Change 655977 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode for Stein upgrade
Change 655979 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'rocky' to 'stein'
Change 655977 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode for Stein upgrade
Change 655979 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'rocky' to 'stein'
Here is the record of the upgrade:
- update IRC topic
- downtime everything in icinga through 14:00CDT
- aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab*
- aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
- downtime Horizon with https://gerrit.wikimedia.org/r/c/operations/puppet/+/655977
- disable puppet on all cloud* hosts
$ sudo cumin 'cloud*' "disable-puppet 'Upgrading to openstack Stein - T261134 - ${USER}'"
- merge puppet patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/655979
- dump databases on cloudcontrol1003: nova_eqiad1 , nova_api_eqiad1, nova_cell0_eqiad1, neutron, glance, keystone:
- mysqldump -u root nova_eqiad1 > /root/steindbbackups/nova_eqiad1.sql
- mysqldump -u root nova_api_eqiad1 > /root/steindbbackups/nova_api_eqiad1.sql
- mysqldump -u root nova_cell0_eqiad1 > /root/steindbbackups/nova_cell0_eqiad1.sql
- mysqldump -u root neutron > /root/steindbbackups/neutron.sql
- mysqldump -u root glance > /root/steindbbackups/glance.sql
- mysqldump -u root keystone > /root/steindbbackups/keystone.sql
Cloudcontrols:
- All open database connections post-upgrade https://phabricator.wikimedia.org/P10999
- Checking haproxy status echo "show stat" | socat /var/run/haproxy/haproxy.sock stdio | grep DOWN
Cloudcontrol1003:
- puppet agent --enable && puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- puppet agent -tv
- nova-manage api_db sync
- nova-manage db sync
- glance-manage db_sync
- keystone-manage db_sync
- puppet agent -tv
- nova-manage db online_data_migrations
Cloudcontrol1004:
- puppet agent --enable && puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- puppet agent -tv
- puppet agent -tv
Cloudcontrol1005:
- puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- puppet agent -tv
- puppet agent -tv
cloudnets (one at a time please):
Begin with the standby node, as determined with:
$ neutron l3-agent-list-hosting-router cloudinstances2b-gw
Standby node:
- puppet agent --enable && puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- puppet agent -tv
- chgrp -R neutron /var/lib/neutron/* && chown -R neutron /var/lib/neutron/*
- service neutron-l3-agent restart
- neutron-db-manage upgrade heads on cloudcontrol1003
Active node:
- service neutron-l3-agent stop # should cause an immediate failover to the already-upgraded standby node
- puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- puppet agent -tv
- chgrp -R neutron /var/lib/neutron/* && chown -R neutron /var/lib/neutron/*
- service neutron-l3-agent restart
- restore Horizon with https://gerrit.wikimedia.org/r/c/operations/puppet/+/655980
Break/meeting, whatever
Cloudvirts (start with one test host first, cloudvirt1039. Don't forget about cloudvirtwdqs ):
- puppet agent --enable && puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- puppet agent -tv
- service neutron-linuxbridge-agent restart
- service nova-compute restart
- service libvirtd restart
- update IRC topic
Mentioned in SAL (#wikimedia-cloud) [2021-01-14T17:58:24Z] <bstorm> hard rebooting tools-sgecron-01 following network issues during upgrade to stein T261134
Change 656225 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova monitoring: update the egrep string for the nova-api process
Change 656225 merged by Andrew Bogott:
[operations/puppet@production] Nova monitoring: update the egrep string for the nova-api process
Change 656231 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-api monitoring: simplify grep for detecting processes
Change 656231 merged by Andrew Bogott:
[operations/puppet@production] nova-api monitoring: simplify grep for detecting processes
Change 656237 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone/stein: disable the packaged keystone init script
Change 656237 merged by Andrew Bogott:
[operations/puppet@production] Keystone/stein: disable the packaged keystone init script
Change 656240 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: remove obsolete service definition
Change 656240 merged by Andrew Bogott:
[operations/puppet@production] keystone: remove obsolete service definition
Change 656514 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] openstack: remove the queens hiera hiding out in places
Change 656514 merged by Bstorm:
[operations/puppet@production] openstack: remove the queens hiera hiding out in places
Change 657870 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Move eqiad1/horizon to openstack train
Change 657870 merged by Andrew Bogott:
[operations/puppet@production] Move eqiad1/horizon to openstack train
Change 660085 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron-linuxbridge hack: restart service after hack is applied
Change 660085 merged by Andrew Bogott:
[operations/puppet@production] neutron-linuxbridge hack: restart service after hack is applied