Page MenuHomePhabricator
Create Task
Maniphest T261134

upgrade cloud-vps openstack to Openstack version 'Stein'
Closed, ResolvedPublic
Actions

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+7 -5neutron-linuxbridge hack: restart service after hack is applied
operations/puppetproduction+2 -2Move eqiad1/horizon to openstack train
operations/puppetproduction+0 -1openstack: remove the queens hiera hiding out in places
operations/puppetproduction+0 -5keystone: remove obsolete service definition
operations/puppetproduction+13 -0Keystone/stein: disable the packaged keystone init script
operations/puppetproduction+2 -2nova-api monitoring: simplify grep for detecting processes
operations/puppetproduction+10 -1Nova monitoring: update the egrep string for the nova-api process
operations/puppetproduction+1 -5cloud-vps: move eqiad1 from openstack 'rocky' to 'stein'
operations/puppetproduction+1 -1Horizon: put into maintenance mode for Stein upgrade
operations/puppetproduction+252 -5nova: puppetize the nova-api-metadata service
operations/puppetproduction+17 -30Nova: modernize api-paste.ini.erb
operations/puppetproduction+4 -2Keystone: update otp auth code for Stein
operations/puppetproduction+0 -865Keystone: remove token_formatters-fixed.py backport
operations/puppetproduction+4 -4Keystone: update password_safelist to match upstream code
operations/puppetproduction+0 -2Keystone: remove [cors] sections that produce deprecation warnings
operations/puppetproduction+0 -1Keystone: update wmtotp plugin to match upstream Password auth code
operations/puppetproduction+1 -69Keystone: use the keystone-wsgi-admin script installed by the package
operations/puppetproduction+31 -21wmfkeystonehooks: Update the monkeypatch that renames project IDs for Stein
operations/puppetproduction+255 -9Nova-api: add a puppetized api init script in Stein
operations/puppetproduction+240 -0Cinder: add a puppetized api init script in Stein
operations/puppetproduction+5 -103Glance: remove glance-registry service from Stein deploys
operations/puppetproduction+38 -0Add some resource files for Cinder/Stein
operations/puppetproduction+8 -3OpenStack codfw1dev -> Stein
operations/puppetproduction+0 -0OpenStack Keystone: move ldap-common-rocky-fixed.py into files/rocky
operations/puppetproduction+18 -0Add OpenStack client package manifests for Stein
operations/puppetproduction+41 -1Barbican: add service manifest for Stein
operations/puppetproduction+263 -0Neutron service manifests for Stein
operations/puppetproduction+180 -1 KKeystone: Add Stein service manifests
operations/puppetproduction+341 -0Cinder: add Stein version of service manifest
operations/puppetproduction+70 -0Glance: add Stein versions of manifests
operations/puppetproduction+56 -24Neutron: apply our local dmz hacks for Stein
operations/puppetproduction+118 -8 KNova: add Stein manifests
operations/puppetproduction+116 -148Neutron: update stein l3 python files
operations/puppetproduction+7 -5Nova config upgrades for Stein
operations/puppetproduction+4 -0eqiad1 designate: move to version 'Stein'
operations/puppetproduction+2 -2Codfw1dev designate: move to version 'Stein'
operations/puppetproduction+33 -1 KOpenStack Designate: updates for version Stein
operations/puppetproduction+107 -0OpenStack: add client packages for Stein
operations/puppetproduction+78 -0OpenStack: add server packages for version Stein
operations/puppetproduction+25 K -0OpenStack: add initial config files for version Stein
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedAndrewT194334 [Epic] Modern Cloud VPS storage layer
Unknown Object (Task)
Unknown Object (Task)
 ResolvedtaaviT211393 openstack-browser and horizon: Security group and floating IP quota information being pulled from Nova instead of Neutron for eqiad1-r
 ResolvedAndrewT211777 Can't get quota information from Neutron API
 ResolvedAndrewT261137 upgrade cloud-vps openstack to Openstack version 'Victoria'
 ResolveddcaroT261136 upgrade cloud-vps openstack to Openstack version 'Ussuri'
 ResolvedAndrewT261138 Upgrade Horizon to latest OpenStack release
 ResolvedAndrewT261135 upgrade cloud-vps openstack to Openstack version 'Train'
 ResolvedAndrewT261134 upgrade cloud-vps openstack to Openstack version 'Stein'
 ResolvedAndrewT259399 Upgrade cloudvirts to Debian Buster
 ResolvedAndrewT261132 Move all cloud-vps VMs to Ceph
 ResolvedAndrewT253365 Complete build out of Ceph cluster and attach "diskless" cloudvirts
 ResolvedCmjohnsonT251619 (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org
 ResolvedayounsiT258764 Error: 'cloudsw1-c8-eqiad.mgmt.eqiad.wmnet' is not a valid parent for host 'cloudcephosd1004'
 ResolvedaborreroT266749 cloudcephosd1006: down, possible memory error
 ResolvedCmjohnsonT251627 (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet
 ResolvedJclark-ctrT251632 (Need By: 2020-06-12) rack/setup/install WMCS 10G switches
 ResolvedJclark-ctrT276922 cloudvirt1038: PCIe error
 ResolvedAndrewT258619 Decide on network setup for WMCS ceph cluster
 ResolvedAndrewT258826 Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan
 ResolvedAndrewT258968 Migrate existing ceph workload off of cloudcephosd100[1-3] and rebuild with new vlan config
 ResolvedAndrewT259057 Move cloudcephosd100[1-3] to new vlans
 ResolvedAndrewT259192 Investigate/prototype ceph backup options
 ResolveddcaroT260692 Ceph VM image backups
 ResolvedaborreroT260941 Practice restoring ceph backups
 DeclineddcaroT267195 CloudVPS: improve VM backups to make them redundant and discoverable
 ResolveddcaroT271094 [dumps] Review backup strategy
 OpenNoneT273720 [ceph][rbd] Periodically cleanup dangling snapshots
 OpenNoneT273723 [backups] Periodically cleanup non-handled backups
 ResolvedAndrewT261252 Create system for migrating single VMs to ceph
 ResolvedAndrewT264380 Rebuild tools-prometheus-04
 ResolvedAndrewT270305 Ceph performance tuning
 ResolvedAndrewT263677 Upgrade cloudvirt1019 and cloudvirt1020 to Buster
 ResolvedBstormT263679 Plan and manage the required downtime for the clouddb-services VMs on cloudvirt1019 and cloudvirt1020
 ResolvedaborreroT267967 set up bpo repos for OpenStack Stein and Debian Buster
 ResolvedaborreroT271517 codfw1dev: drop cloudgw setup briefly for testing openstack stein

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Nov 16 2020, 7:51 PM

Change 641230 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add initial config files for version Stein

https://gerrit.wikimedia.org/r/641230

gerritbot added a project: Patch-For-Review.Nov 16 2020, 7:51 PM

Change 641231 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add server packages for version Stein

https://gerrit.wikimedia.org/r/641231

gerritbot added a comment.Nov 16 2020, 7:51 PM

Change 641232 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Designate: updates for version Stein

https://gerrit.wikimedia.org/r/641232

gerritbot added a comment.Nov 16 2020, 7:51 PM

Change 641233 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add client packages for Stein

https://gerrit.wikimedia.org/r/641233

aborrero closed subtask T267967: set up bpo repos for OpenStack Stein and Debian Buster as Resolved.Nov 19 2020, 10:23 AM
gerritbot added a comment.Nov 22 2020, 4:13 PM

Change 641230 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add initial config files for version Stein

https://gerrit.wikimedia.org/r/641230

gerritbot added a comment.Nov 22 2020, 4:14 PM

Change 641231 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add server packages for version Stein

https://gerrit.wikimedia.org/r/641231

gerritbot added a comment.Nov 22 2020, 4:18 PM

Change 641233 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add client packages for Stein

https://gerrit.wikimedia.org/r/641233

gerritbot added a comment.Nov 22 2020, 4:20 PM

Change 641232 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Designate: updates for version Stein

https://gerrit.wikimedia.org/r/641232

gerritbot added a comment.Nov 22 2020, 4:48 PM

Change 642889 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Codfw1dev designate: move to version 'Stein'

https://gerrit.wikimedia.org/r/642889

gerritbot added a comment.Nov 22 2020, 4:50 PM

Change 642889 merged by Andrew Bogott:
[operations/puppet@production] Codfw1dev designate: move to version 'Stein'

https://gerrit.wikimedia.org/r/642889

Maintenance_bot removed a project: Patch-For-Review.Nov 22 2020, 5:10 PM
gerritbot added a comment.Nov 22 2020, 5:26 PM

Change 642898 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] eqiad1 designate: move to version 'Stein'

https://gerrit.wikimedia.org/r/642898

gerritbot added a project: Patch-For-Review.Nov 22 2020, 5:26 PM

Change 642898 merged by Andrew Bogott:
[operations/puppet@production] eqiad1 designate: move to version 'Stein'

https://gerrit.wikimedia.org/r/642898

Andrew added a comment.EditedNov 22 2020, 6:01 PM

Since the cloudservices nodes are already running Buster, I upgraded them to Stein just now. All seems well for the moment.

Maintenance_bot removed a project: Patch-For-Review.Nov 22 2020, 6:10 PM
gerritbot added a comment.Dec 1 2020, 2:34 PM

Change 644534 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova config upgrades for Stein

https://gerrit.wikimedia.org/r/644534

gerritbot added a project: Patch-For-Review.Dec 1 2020, 2:34 PM
gerritbot added a comment.Dec 1 2020, 2:44 PM

Change 644534 merged by Andrew Bogott:
[operations/puppet@production] Nova config upgrades for Stein

https://gerrit.wikimedia.org/r/644534

Maintenance_bot removed a project: Patch-For-Review.Dec 1 2020, 3:10 PM
Andrew closed subtask T259399: Upgrade cloudvirts to Debian Buster as Resolved.Dec 17 2020, 3:18 PM
gerritbot added a comment.Dec 23 2020, 2:15 AM

Change 651668 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: update stein l3 packages

https://gerrit.wikimedia.org/r/651668

gerritbot added a project: Patch-For-Review.Dec 23 2020, 2:15 AM

Change 651669 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: apply our local dmz hacks for Stein

https://gerrit.wikimedia.org/r/651669

gerritbot added a comment.Dec 23 2020, 4:59 AM

Change 651683 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: add Stein manifests

https://gerrit.wikimedia.org/r/651683

gerritbot added a comment.Dec 23 2020, 5:00 AM

Change 651684 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Glance: add Stein versions of manifests

https://gerrit.wikimedia.org/r/651684

gerritbot added a comment.Dec 23 2020, 5:00 AM

Change 651685 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cinder: add Stein version of service manifest

https://gerrit.wikimedia.org/r/651685

gerritbot added a comment.Dec 23 2020, 5:00 AM

Change 651686 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: Add Stein service manifests

https://gerrit.wikimedia.org/r/651686

gerritbot added a comment.Dec 23 2020, 5:00 AM

Change 651687 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron service manifests for Stein

https://gerrit.wikimedia.org/r/651687

gerritbot added a comment.Dec 23 2020, 5:00 AM

Change 651688 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Barbican: add service manifest for Stein

https://gerrit.wikimedia.org/r/651688

gerritbot added a comment.Dec 23 2020, 5:00 AM

Change 651689 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add OpenStack client package manifests for Stein

https://gerrit.wikimedia.org/r/651689

gerritbot added a comment.Dec 23 2020, 5:03 AM

Change 651691 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack codfw1dev -> Stein

https://gerrit.wikimedia.org/r/651691

gerritbot added a comment.Dec 23 2020, 5:48 PM

Change 651668 merged by Andrew Bogott:
[operations/puppet@production] Neutron: update stein l3 python files

https://gerrit.wikimedia.org/r/651668

gerritbot added a comment.Dec 23 2020, 5:50 PM

Change 651683 merged by Andrew Bogott:
[operations/puppet@production] Nova: add Stein manifests

https://gerrit.wikimedia.org/r/651683

gerritbot added a comment.Dec 23 2020, 5:52 PM

Change 651669 merged by Andrew Bogott:
[operations/puppet@production] Neutron: apply our local dmz hacks for Stein

https://gerrit.wikimedia.org/r/651669

gerritbot added a comment.Dec 23 2020, 6:06 PM

Change 651684 merged by Andrew Bogott:
[operations/puppet@production] Glance: add Stein versions of manifests

https://gerrit.wikimedia.org/r/651684

gerritbot added a comment.Dec 23 2020, 6:06 PM

Change 651685 merged by Andrew Bogott:
[operations/puppet@production] Cinder: add Stein version of service manifest

https://gerrit.wikimedia.org/r/651685

gerritbot added a comment.Dec 23 2020, 6:06 PM

Change 651686 merged by Andrew Bogott:
[operations/puppet@production] Keystone: Add Stein service manifests

https://gerrit.wikimedia.org/r/651686

gerritbot added a comment.Dec 23 2020, 6:06 PM

Change 651687 merged by Andrew Bogott:
[operations/puppet@production] Neutron service manifests for Stein

https://gerrit.wikimedia.org/r/651687

gerritbot added a comment.Dec 23 2020, 6:06 PM

Change 651688 merged by Andrew Bogott:
[operations/puppet@production] Barbican: add service manifest for Stein

https://gerrit.wikimedia.org/r/651688

gerritbot added a comment.Dec 23 2020, 6:07 PM

Change 651689 merged by Andrew Bogott:
[operations/puppet@production] Add OpenStack client package manifests for Stein

https://gerrit.wikimedia.org/r/651689

gerritbot added a comment.Dec 23 2020, 6:10 PM

Change 651812 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Keystone: move ldap-common-rocky-fixed.py into files/rocky

https://gerrit.wikimedia.org/r/651812

gerritbot added a comment.Dec 23 2020, 6:11 PM

Change 651812 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Keystone: move ldap-common-rocky-fixed.py into files/rocky

https://gerrit.wikimedia.org/r/651812

gerritbot added a comment.Dec 23 2020, 7:04 PM

Change 651691 merged by Andrew Bogott:
[operations/puppet@production] OpenStack codfw1dev -> Stein

https://gerrit.wikimedia.org/r/651691

gerritbot added a comment.Dec 23 2020, 7:08 PM

Change 651820 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add some resource files for Cinder/Stein

https://gerrit.wikimedia.org/r/651820

gerritbot added a comment.Dec 23 2020, 7:11 PM

Change 651820 merged by Andrew Bogott:
[operations/puppet@production] Add some resource files for Cinder/Stein

https://gerrit.wikimedia.org/r/651820

gerritbot added a comment.Dec 23 2020, 7:16 PM

Change 651822 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Glance: remove glance-registry service from Stein deploys

https://gerrit.wikimedia.org/r/651822

gerritbot added a comment.Dec 23 2020, 7:23 PM

Change 651822 merged by Andrew Bogott:
[operations/puppet@production] Glance: remove glance-registry service from Stein deploys

https://gerrit.wikimedia.org/r/651822

gerritbot added a comment.Dec 23 2020, 7:39 PM

Change 651825 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cinder: add a puppetized api init script in Stein

https://gerrit.wikimedia.org/r/651825

gerritbot added a comment.Dec 23 2020, 7:40 PM

Change 651825 merged by Andrew Bogott:
[operations/puppet@production] Cinder: add a puppetized api init script in Stein

https://gerrit.wikimedia.org/r/651825

gerritbot added a comment.Dec 23 2020, 7:53 PM

Change 651829 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova-api: add a puppetized api init script in Stein

https://gerrit.wikimedia.org/r/651829

gerritbot added a comment.Dec 23 2020, 8:07 PM

Change 651829 merged by Andrew Bogott:
[operations/puppet@production] Nova-api: add a puppetized api init script in Stein

https://gerrit.wikimedia.org/r/651829

gerritbot added a comment.Dec 23 2020, 9:15 PM

Change 651840 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfkeystonehooks: Update the monkeypatch that renames project IDs for Stein

https://gerrit.wikimedia.org/r/651840

gerritbot added a comment.Dec 23 2020, 9:21 PM

Change 651840 merged by Andrew Bogott:
[operations/puppet@production] wmfkeystonehooks: Update the monkeypatch that renames project IDs for Stein

https://gerrit.wikimedia.org/r/651840

gerritbot added a comment.Dec 24 2020, 3:23 AM

Change 651875 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: use the keystone-wsgi-admin script installed by the package

https://gerrit.wikimedia.org/r/651875

gerritbot added a comment.Dec 24 2020, 3:23 AM

Change 651876 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: update wmtotp plugin to match upstream Password auth code

https://gerrit.wikimedia.org/r/651876

gerritbot added a comment.Dec 24 2020, 3:25 AM

Change 651875 merged by Andrew Bogott:
[operations/puppet@production] Keystone: use the keystone-wsgi-admin script installed by the package

https://gerrit.wikimedia.org/r/651875

gerritbot added a comment.Dec 24 2020, 3:26 AM

Change 651876 merged by Andrew Bogott:
[operations/puppet@production] Keystone: update wmtotp plugin to match upstream Password auth code

https://gerrit.wikimedia.org/r/651876

gerritbot added a comment.Dec 24 2020, 3:35 AM

Change 651877 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove [cors] sections that produce deprecation warnings

https://gerrit.wikimedia.org/r/651877

gerritbot added a comment.Dec 24 2020, 3:35 AM

Change 651878 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: update password_safelist to match upstream code

https://gerrit.wikimedia.org/r/651878

gerritbot added a comment.Dec 24 2020, 3:50 AM

Change 651877 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove [cors] sections that produce deprecation warnings

https://gerrit.wikimedia.org/r/651877

gerritbot added a comment.Dec 24 2020, 3:58 AM

Change 651878 merged by Andrew Bogott:
[operations/puppet@production] Keystone: update password_safelist to match upstream code

https://gerrit.wikimedia.org/r/651878

gerritbot added a comment.Dec 24 2020, 4:02 AM

Change 651880 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove token_formatters-fixed.py backport

https://gerrit.wikimedia.org/r/651880

gerritbot added a comment.Dec 24 2020, 4:02 AM

Change 651880 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove token_formatters-fixed.py backport

https://gerrit.wikimedia.org/r/651880

gerritbot added a comment.Dec 31 2020, 9:44 PM

Change 652832 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: update otp auth code for Stein

https://gerrit.wikimedia.org/r/652832

gerritbot added a comment.Dec 31 2020, 9:47 PM

Change 652832 merged by Andrew Bogott:
[operations/puppet@production] Keystone: update otp auth code for Stein

https://gerrit.wikimedia.org/r/652832

gerritbot added a comment.Jan 3 2021, 4:14 AM

Change 653667 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: modernize api-paste.ini.erb

https://gerrit.wikimedia.org/r/653667

gerritbot added a comment.Jan 3 2021, 4:16 AM

Change 653667 merged by Andrew Bogott:
[operations/puppet@production] Nova: modernize api-paste.ini.erb

https://gerrit.wikimedia.org/r/653667

gerritbot added a comment.Jan 4 2021, 6:38 AM

Change 654044 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova: puppetize the nova-api-metadata service

https://gerrit.wikimedia.org/r/654044

gerritbot added a comment.Jan 4 2021, 6:48 AM

Change 654044 merged by Andrew Bogott:
[operations/puppet@production] nova: puppetize the nova-api-metadata service

https://gerrit.wikimedia.org/r/654044

Andrew claimed this task.Jan 4 2021, 5:30 PM
aborrero mentioned this in T271518: cron spam from cloudcontrol2004-dev.wikimedia.org.Jan 8 2021, 10:24 AM
aborrero closed subtask T271517: codfw1dev: drop cloudgw setup briefly for testing openstack stein as Resolved.Jan 8 2021, 11:58 AM
gerritbot added a comment.Jan 13 2021, 8:24 PM

Change 655977 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode for Stein upgrade

https://gerrit.wikimedia.org/r/655977

gerritbot added a comment.Jan 13 2021, 8:28 PM

Change 655979 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'rocky' to 'stein'

https://gerrit.wikimedia.org/r/655979

Andrew added a comment.Jan 13 2021, 8:51 PM

Upgrade plan at https://etherpad.wikimedia.org/p/stein

gerritbot added a comment.Jan 14 2021, 2:57 PM

Change 655977 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode for Stein upgrade

https://gerrit.wikimedia.org/r/655977

gerritbot added a comment.Jan 14 2021, 3:02 PM

Change 655979 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'rocky' to 'stein'

https://gerrit.wikimedia.org/r/655979

Andrew added a comment.Jan 14 2021, 5:49 PM

Here is the record of the upgrade:

  • update IRC topic
  • downtime everything in icinga through 14:00CDT
    • aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab*
    • aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
  • downtime Horizon with https://gerrit.wikimedia.org/r/c/operations/puppet/+/655977
  • disable puppet on all cloud* hosts

$ sudo cumin 'cloud*' "disable-puppet 'Upgrading to openstack Stein - T261134 - ${USER}'"

  • merge puppet patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/655979
  • dump databases on cloudcontrol1003: nova_eqiad1 , nova_api_eqiad1, nova_cell0_eqiad1, neutron, glance, keystone:
    1. mysqldump -u root nova_eqiad1 > /root/steindbbackups/nova_eqiad1.sql
    2. mysqldump -u root nova_api_eqiad1 > /root/steindbbackups/nova_api_eqiad1.sql
    3. mysqldump -u root nova_cell0_eqiad1 > /root/steindbbackups/nova_cell0_eqiad1.sql
    4. mysqldump -u root neutron > /root/steindbbackups/neutron.sql
    5. mysqldump -u root glance > /root/steindbbackups/glance.sql
    6. mysqldump -u root keystone > /root/steindbbackups/keystone.sql

Cloudcontrols:

Cloudcontrol1003:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • nova-manage api_db sync
  • nova-manage db sync
  • glance-manage db_sync
  • keystone-manage db_sync
  • puppet agent -tv
  • nova-manage db online_data_migrations

Cloudcontrol1004:




    

  •  puppet agent --enable && puppet agent -tv
    • 

  •  apt-get update
    • 

  •  DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common      keystone nova-api nova-conductor nova-placement-api nova-scheduler      nova-common glance neutron-server python3-requests python3-urllib3      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
    • 

  •  DEBIAN_FRONTEND=noninteractive  apt-get upgrade      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
    • 

  •  puppet agent -tv
    • 

  •  puppet agent -tv
    • 




Cloudcontrol1005:






    

  •  puppet agent -tv
    • 

  •  apt-get update
    • 

  •  DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common      keystone nova-api nova-conductor nova-placement-api nova-scheduler      nova-common glance neutron-server python3-requests python3-urllib3      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
    • 

  •  DEBIAN_FRONTEND=noninteractive  apt-get upgrade      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
    • 

  •  puppet agent -tv
    • 

  •  puppet agent -tv
    • 




cloudnets (one at a time please):






Begin with the standby node, as determined with:



$ neutron l3-agent-list-hosting-router cloudinstances2b-gw



Standby node:



    

  •  puppet agent --enable && puppet agent -tv
    • 

  •  apt-get update
    • 

  •  DEBIAN_FRONTEND=noninteractive apt-get install   -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold" neutron-l3-agent      python3-neutronclient  python3-glanceclient
    • 

  •  DEBIAN_FRONTEND=noninteractive apt-get upgrade      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
    • 

  •  puppet agent -tv
    • 

  •  chgrp -R neutron /var/lib/neutron/* && chown -R neutron /var/lib/neutron/*
    • 

  •  service neutron-l3-agent restart
    • 

  •  neutron-db-manage upgrade heads on cloudcontrol1003
    • 




Active node:



    

  •  service neutron-l3-agent stop  # should cause an immediate failover to the already-upgraded standby node
    • 

  •  puppet agent -tv
    • 

  •  apt-get update
    • 

  •  DEBIAN_FRONTEND=noninteractive apt-get install   -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold" neutron-l3-agent      python3-neutronclient  python3-glanceclient
    • 

  •  DEBIAN_FRONTEND=noninteractive apt-get upgrade      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
    • 

  •  puppet agent -tv
    • 

  •  chgrp -R neutron /var/lib/neutron/* && chown -R neutron /var/lib/neutron/*
    • 

  •  service neutron-l3-agent restart
    • 








Break/meeting, whatever



Cloudvirts (start with one test host first, cloudvirt1039. Don't forget about cloudvirtwdqs ):



    

  •  puppet agent --enable && puppet agent -tv
    • 

  •  apt-get update
    • 

  •  DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common  neutron-linuxbridge-agent python3-neutron python3-keystoneauth1      python3-positional python3-requests python3-urllib3  -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
    • 

  •  DEBIAN_FRONTEND=noninteractive  apt-get upgrade  -y --allow-downgrades    -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
    • 

  •  puppet agent -tv
    • 

  •  service neutron-linuxbridge-agent restart
    • 

  •  service nova-compute restart
    • 

  •  service libvirtd restart
    • 






    

  •  update IRC topic
    • 




Stashbot added a comment.Jan 14 2021, 5:58 PM
Mentioned in SAL (#wikimedia-cloud) [2021-01-14T17:58:24Z] <bstorm> hard rebooting tools-sgecron-01 following network issues during upgrade to stein T261134
gerritbot added a comment.Jan 14 2021, 6:08 PM
Change 656225 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):

[operations/puppet@production] Nova monitoring: update the egrep string for the nova-api process



https://gerrit.wikimedia.org/r/656225
gerritbot added a comment.Jan 14 2021, 6:09 PM
Change 656225 merged by Andrew Bogott:

[operations/puppet@production] Nova monitoring: update the egrep string for the nova-api process



https://gerrit.wikimedia.org/r/656225
gerritbot added a comment.Jan 14 2021, 6:17 PM
Change 656231 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):

[operations/puppet@production] nova-api monitoring: simplify grep for detecting processes



https://gerrit.wikimedia.org/r/656231
gerritbot added a comment.Jan 14 2021, 6:20 PM
Change 656231 merged by Andrew Bogott:

[operations/puppet@production] nova-api monitoring: simplify grep for detecting processes



https://gerrit.wikimedia.org/r/656231
gerritbot added a comment.Jan 14 2021, 6:26 PM
Change 656237 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):

[operations/puppet@production] Keystone/stein: disable the packaged keystone init script



https://gerrit.wikimedia.org/r/656237
gerritbot added a comment.Jan 14 2021, 6:31 PM
Change 656237 merged by Andrew Bogott:

[operations/puppet@production] Keystone/stein: disable the packaged keystone init script



https://gerrit.wikimedia.org/r/656237
gerritbot added a comment.Jan 14 2021, 6:36 PM
Change 656240 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):

[operations/puppet@production] keystone: remove obsolete service definition



https://gerrit.wikimedia.org/r/656240
gerritbot added a comment.Jan 14 2021, 6:37 PM
Change 656240 merged by Andrew Bogott:

[operations/puppet@production] keystone: remove obsolete service definition



https://gerrit.wikimedia.org/r/656240
Andrew closed this task as Resolved.Jan 14 2021, 7:17 PM
Bstorm mentioned this in T272161: Toolforge: reFill has not processed two days.Jan 15 2021, 4:43 PM
gerritbot added a comment.Jan 15 2021, 9:24 PM
Change 656514 had a related patch set uploaded (by Bstorm; owner: Bstorm):

[operations/puppet@production] openstack: remove the queens hiera hiding out in places



https://gerrit.wikimedia.org/r/656514
gerritbot added a comment.Jan 15 2021, 9:34 PM
Change 656514 merged by Bstorm:

[operations/puppet@production] openstack: remove the queens hiera hiding out in places



https://gerrit.wikimedia.org/r/656514
gerritbot added a comment.Jan 22 2021, 4:31 PM
Change 657870 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):

[operations/puppet@production] Move eqiad1/horizon to openstack train



https://gerrit.wikimedia.org/r/657870
gerritbot added a comment.Jan 22 2021, 4:31 PM
Change 657870 merged by Andrew Bogott:

[operations/puppet@production] Move eqiad1/horizon to openstack train



https://gerrit.wikimedia.org/r/657870
gerritbot added a comment.Jan 30 2021, 3:00 AM
Change 660085 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):

[operations/puppet@production] neutron-linuxbridge hack: restart service after hack is applied



https://gerrit.wikimedia.org/r/660085
gerritbot added a comment.Jan 30 2021, 3:01 AM
Change 660085 merged by Andrew Bogott:

[operations/puppet@production] neutron-linuxbridge hack: restart service after hack is applied



https://gerrit.wikimedia.org/r/660085
Bstorm mentioned this in T274144: cloudvirt1023/Check the last execution of backup_vms is CRITICAL.Feb 8 2021, 5:35 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL