Page MenuHomePhabricator
Create Task
Maniphest T248635

upgrade cloud-vps openstack to Openstack version 'Rocky'
Closed, ResolvedPublic
Actions

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+2 -6cloud-vps: move eqiad1 from openstack 'queens' to 'rocky'
operations/puppetproduction+1 -1Horizon: put into maintenance mode for Rocky upgrade
operations/puppetproduction+5 -7Openstack tooz hack: don't explicitly depend on python3-tooz
operations/puppetproduction+1 K -0Openstack Designate: fix an encoding issue in the memcached tooz driver
operations/puppetproduction+2 -2OpenStack designate: upgrade eqiad1 to version Rocky
operations/puppetproduction+32 -0cloud-vps: add vm client packages for Rocky
operations/puppetproduction+865 -0Keystone: Drop in a backported fix to token_formatters.py
operations/puppetproduction+25 -20Keystone hacks: update keystone api handling
operations/puppetproduction+74 -1 KNeutron: replace l3 hacks for Rocky
operations/puppetproduction+12 -8neutron-api: create /var/run/neutron
operations/puppetproduction+1 -4Neutron: move uwsgi socket file to /var/run/neutron
operations/puppetproduction+82 -1Neutron: add management of neutron-api-uwsgi.ini and api-paste.ini
operations/puppetproduction+14 -0OpenStack Rocky: pin uwsgi-plugin-python3 uwsgi-core for Rocky
operations/puppetproduction+2 -2codfw1dev: switch nova, keystone, etc. from queens to rocky
operations/puppetproduction+10 K -135OpenStack: add config and manifests for version Rocky
operations/puppetproduction+2 K -0Openstack Neutron: add neutron l3 hacks for Rocky
operations/puppetproduction+1 K -0Openstack: add Designate manifest and config for version Rocky
operations/puppetproduction+5 -0Designate: set enabled = True in the [service:worker] section
operations/puppetproduction+7 -3Openstack configs: comment out service_token_roles_required = True
operations/puppetproduction+2 -2neutron.conf: fix an earlier typo with auth_uri vs auth_url
operations/puppetproduction+2 -3designate.conf: update a few deprecated settings
operations/puppetproduction+2 -1neutron.conf: update a couple of settings in response to deprecation warnings
operations/puppetproduction+3 -5nova.conf: update conf to suppress deprecation warnings
operations/puppetproduction+0 -1Keystone: remove member_role_name from keystone.conf
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.Mar 26 2020, 10:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 26 2020, 10:24 PM
Andrew triaged this task as Medium priority.Mar 26 2020, 10:24 PM
gerritbot added a comment.Mar 27 2020, 1:04 PM

Change 583940 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove member_role_name from keystone.conf

https://gerrit.wikimedia.org/r/583940

gerritbot added a project: Patch-For-Review.Mar 27 2020, 1:04 PM
gerritbot added a comment.Mar 27 2020, 1:37 PM

Change 583940 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove member_role_name from keystone.conf

https://gerrit.wikimedia.org/r/583940

Maintenance_bot removed a project: Patch-For-Review.Mar 27 2020, 2:10 PM
gerritbot added a comment.Mar 27 2020, 3:45 PM

Change 583989 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova.conf: update conf to suppress deprecation warnings

https://gerrit.wikimedia.org/r/583989

gerritbot added a project: Patch-For-Review.Mar 27 2020, 3:45 PM
gerritbot added a comment.Mar 27 2020, 3:49 PM

Change 583990 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron.conf: update a couple of settings in response to deprecation warnings

https://gerrit.wikimedia.org/r/583990

gerritbot added a comment.Mar 27 2020, 3:55 PM

Change 583994 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate.conf: update a few deprecated settings

https://gerrit.wikimedia.org/r/583994

gerritbot added a comment.Mar 27 2020, 3:57 PM

Change 583989 merged by Andrew Bogott:
[operations/puppet@production] nova.conf: update conf to suppress deprecation warnings

https://gerrit.wikimedia.org/r/583989

gerritbot added a comment.Mar 27 2020, 4:14 PM

Change 583990 merged by Andrew Bogott:
[operations/puppet@production] neutron.conf: update a couple of settings in response to deprecation warnings

https://gerrit.wikimedia.org/r/583990

gerritbot added a comment.Mar 27 2020, 4:14 PM

Change 583994 merged by Andrew Bogott:
[operations/puppet@production] designate.conf: update a few deprecated settings

https://gerrit.wikimedia.org/r/583994

gerritbot added a comment.Mar 27 2020, 4:20 PM

Change 584000 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron.conf: fix an earlier typo with auth_uri vs auth_url

https://gerrit.wikimedia.org/r/584000

gerritbot added a comment.Mar 27 2020, 4:21 PM

Change 584000 merged by Andrew Bogott:
[operations/puppet@production] neutron.conf: fix an earlier typo with auth_uri vs auth_url

https://gerrit.wikimedia.org/r/584000

gerritbot added a comment.Mar 27 2020, 4:32 PM

Change 584007 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack configs: comment out service_token_roles_required = True

https://gerrit.wikimedia.org/r/584007

gerritbot added a comment.Mar 27 2020, 4:33 PM

Change 584007 merged by Andrew Bogott:
[operations/puppet@production] Openstack configs: comment out service_token_roles_required = True

https://gerrit.wikimedia.org/r/584007

gerritbot added a comment.Mar 27 2020, 4:38 PM

Change 584011 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: set enabled = True in the [service:worker] section

https://gerrit.wikimedia.org/r/584011

gerritbot added a comment.Mar 27 2020, 4:40 PM

Change 584011 merged by Andrew Bogott:
[operations/puppet@production] Designate: set enabled = True in the [service:worker] section

https://gerrit.wikimedia.org/r/584011

Maintenance_bot removed a project: Patch-For-Review.Mar 27 2020, 5:10 PM
gerritbot added a comment.Mar 27 2020, 9:06 PM

Change 584058 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack: add Designate manifest and config for version Rocky

https://gerrit.wikimedia.org/r/584058

gerritbot added a project: Patch-For-Review.Mar 27 2020, 9:06 PM
gerritbot added a comment.Mar 27 2020, 9:08 PM

Change 584058 merged by Andrew Bogott:
[operations/puppet@production] Openstack: add Designate manifest and config for version Rocky

https://gerrit.wikimedia.org/r/584058

Maintenance_bot removed a project: Patch-For-Review.Mar 27 2020, 9:10 PM
gerritbot added a comment.Mar 31 2020, 8:53 PM

Change 585034 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Neutron: add neutron l3 hacks for Rocky

https://gerrit.wikimedia.org/r/585034

gerritbot added a project: Patch-For-Review.Mar 31 2020, 8:53 PM
gerritbot added a comment.Apr 3 2020, 11:18 PM

Change 585838 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add config and manifests for version Rocky

https://gerrit.wikimedia.org/r/585838

gerritbot added a comment.Apr 4 2020, 12:10 AM

Change 585034 merged by Andrew Bogott:
[operations/puppet@production] Openstack Neutron: add neutron l3 hacks for Rocky

https://gerrit.wikimedia.org/r/585034

gerritbot added a comment.Apr 4 2020, 12:10 AM

Change 585838 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add config and manifests for version Rocky

https://gerrit.wikimedia.org/r/585838

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2020, 12:10 AM
gerritbot added a comment.Apr 4 2020, 12:14 AM

Change 585844 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] codfw1dev: switch nova, keystone, etc. from queens to rocky

https://gerrit.wikimedia.org/r/585844

gerritbot added a project: Patch-For-Review.Apr 4 2020, 12:14 AM
gerritbot added a comment.Apr 4 2020, 1:46 AM

Change 585844 merged by Andrew Bogott:
[operations/puppet@production] codfw1dev: switch nova, keystone, etc. from queens to rocky

https://gerrit.wikimedia.org/r/585844

Andrew added a comment.EditedApr 4 2020, 1:50 AM

Dallas checklist:

The final checklist (with amendments) is:

Cloudcontrols:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • dump databases: nova, nova-api, neutron, glance, keystone
  • puppet agent -tv
  • nova-manage db sync
  • nova-manage api_db sync
  • glance-manage db_sync
  • keystone-manage db_sync
  • neutron-db-manage upgrade heads (( neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head `))
  • puppet agent -tv
  • nova-manage db online_data_migrations

Cloudvirts:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-linuxbridge-agent restart
  • service nova-compute restart
  • service libvirtd restart

cloudnets:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient

    SERVICE INTERRUPTION
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-l3-agent restart

    SERVICE RESTORED
Maintenance_bot removed a project: Patch-For-Review.Apr 4 2020, 2:10 AM
gerritbot added a comment.Apr 4 2020, 2:50 AM

Change 585850 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Rocky: pin uwsgi-plugin-python3 uwsgi-core for Rocky

https://gerrit.wikimedia.org/r/585850

gerritbot added a project: Patch-For-Review.Apr 4 2020, 2:50 AM
gerritbot added a comment.Apr 4 2020, 2:52 AM

Change 585850 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Rocky: pin uwsgi-plugin-python3 uwsgi-core for Rocky

https://gerrit.wikimedia.org/r/585850

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2020, 3:10 AM
gerritbot added a comment.Apr 4 2020, 4:04 AM

Change 585854 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: add management of neutron-api-uwsgi.ini and api-paste.ini

https://gerrit.wikimedia.org/r/585854

gerritbot added a project: Patch-For-Review.Apr 4 2020, 4:04 AM
gerritbot added a comment.Apr 4 2020, 4:11 AM

Change 585854 merged by Andrew Bogott:
[operations/puppet@production] Neutron: add management of neutron-api-uwsgi.ini and api-paste.ini

https://gerrit.wikimedia.org/r/585854

gerritbot added a comment.Apr 4 2020, 4:17 AM

Change 585855 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: move uwsgi socket file to /var/run/neutron

https://gerrit.wikimedia.org/r/585855

gerritbot added a comment.Apr 4 2020, 4:18 AM

Change 585855 merged by Andrew Bogott:
[operations/puppet@production] Neutron: move uwsgi socket file to /var/run/neutron

https://gerrit.wikimedia.org/r/585855

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2020, 5:10 AM
gerritbot added a comment.Apr 4 2020, 3:05 PM

Change 585895 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron-api: create /var/run/neutron

https://gerrit.wikimedia.org/r/585895

gerritbot added a project: Patch-For-Review.Apr 4 2020, 3:05 PM
gerritbot added a comment.Apr 4 2020, 3:12 PM

Change 585895 merged by Andrew Bogott:
[operations/puppet@production] neutron-api: create /var/run/neutron

https://gerrit.wikimedia.org/r/585895

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2020, 4:10 PM
gerritbot added a comment.Apr 4 2020, 4:28 PM

Change 585917 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: replace l3 hacks for Rocky

https://gerrit.wikimedia.org/r/585917

gerritbot added a project: Patch-For-Review.Apr 4 2020, 4:28 PM

Change 585917 merged by Andrew Bogott:
[operations/puppet@production] Neutron: replace l3 hacks for Rocky

https://gerrit.wikimedia.org/r/585917

gerritbot added a comment.Apr 4 2020, 4:56 PM

Change 585934 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone hacks: update keystone api handling

https://gerrit.wikimedia.org/r/585934

gerritbot added a comment.Apr 4 2020, 5:02 PM

Change 585934 merged by Andrew Bogott:
[operations/puppet@production] Keystone hacks: update keystone api handling

https://gerrit.wikimedia.org/r/585934

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2020, 5:10 PM
gerritbot added a comment.Apr 5 2020, 3:36 PM

Change 586105 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: Drop in a backported fix to token_formatters.py

https://gerrit.wikimedia.org/r/586105

gerritbot added a project: Patch-For-Review.Apr 5 2020, 3:36 PM
gerritbot added a comment.Apr 5 2020, 3:46 PM

Change 586105 merged by Andrew Bogott:
[operations/puppet@production] Keystone: Drop in a backported fix to token_formatters.py

https://gerrit.wikimedia.org/r/586105

Maintenance_bot removed a project: Patch-For-Review.Apr 5 2020, 4:10 PM
gerritbot added a comment.Apr 7 2020, 4:40 PM

Change 587292 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: add vm client packages for Rocky

https://gerrit.wikimedia.org/r/587292

gerritbot added a project: Patch-For-Review.Apr 7 2020, 4:40 PM

Change 587292 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: add vm client packages for Rocky

https://gerrit.wikimedia.org/r/587292

Maintenance_bot removed a project: Patch-For-Review.Apr 7 2020, 5:10 PM
gerritbot added a comment.Apr 8 2020, 1:51 PM

Change 587522 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack designate: upgrade eqiad1 to version Rocky

https://gerrit.wikimedia.org/r/587522

gerritbot added a project: Patch-For-Review.Apr 8 2020, 1:51 PM
gerritbot added a comment.Apr 9 2020, 7:57 PM

Change 587522 merged by Andrew Bogott:
[operations/puppet@production] OpenStack designate: upgrade eqiad1 to version Rocky

https://gerrit.wikimedia.org/r/587522

Maintenance_bot removed a project: Patch-For-Review.Apr 9 2020, 8:10 PM
gerritbot added a comment.Apr 9 2020, 8:47 PM

Change 587864 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Designate: fix an encoding issue in the memcached tooz driver

https://gerrit.wikimedia.org/r/587864

gerritbot added a project: Patch-For-Review.Apr 9 2020, 8:47 PM
gerritbot added a comment.Apr 9 2020, 8:55 PM

Change 587864 merged by Andrew Bogott:
[operations/puppet@production] Openstack Designate: fix an encoding issue in the memcached tooz driver

https://gerrit.wikimedia.org/r/587864

gerritbot added a comment.Apr 9 2020, 8:58 PM

Change 587866 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack tooz hack: don't explicitly depend on python3-tooz

https://gerrit.wikimedia.org/r/587866

gerritbot added a comment.Apr 9 2020, 9:04 PM

Change 587866 merged by Andrew Bogott:
[operations/puppet@production] Openstack tooz hack: don't explicitly depend on python3-tooz

https://gerrit.wikimedia.org/r/587866

Maintenance_bot removed a project: Patch-For-Review.Apr 9 2020, 9:10 PM
gerritbot added a comment.Apr 16 2020, 1:28 PM

Change 589299 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode for Rocky upgrade

https://gerrit.wikimedia.org/r/589299

gerritbot added a project: Patch-For-Review.Apr 16 2020, 1:28 PM

Change 589300 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'queens' to 'rocky'

https://gerrit.wikimedia.org/r/589300

gerritbot added a comment.Apr 16 2020, 3:01 PM

Change 589299 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode for Rocky upgrade

https://gerrit.wikimedia.org/r/589299

gerritbot added a comment.Apr 16 2020, 3:03 PM

Change 589300 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'queens' to 'rocky'

https://gerrit.wikimedia.org/r/589300

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2020, 3:12 PM
Andrew added a comment.Apr 16 2020, 9:24 PM

This is done. For posterity, here's the etherpad from the upgrade window:

[x] update IRC topic
[x] downtime everything in icinga through 14:00CDT
** aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab*
** aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
[x] downtime Horizon with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589299/
[x] merge puppet patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589300/

Cloudcontrols:

* All open database connections post-upgrade https://phabricator.wikimedia.org/P10999
* Checking haproxy `status echo "show stat" | socat /var/run/haproxy/haproxy.sock stdio | grep  DOWN`
Cloudcontrol1003:
    
[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common      keystone nova-api nova-conductor nova-placement-api nova-scheduler      nova-common glance neutron-server python3-requests python3-urllib3      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[] DEBIAN_FRONTEND=noninteractive  apt-get upgrade      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[x] dump databases: nova, nova-api, neutron, glance, keystone 
[x] puppet agent -tv
[x] nova-manage api_db sync
[x] nova-manage db sync
[x] glance-manage db_sync
[x] keystone-manage db_sync
[x] puppet agent -tv
[x] nova-manage db online_data_migrations

Encoding issues:
* https://phabricator.wikimedia.org/T234830#5802394
* https://phabricator.wikimedia.org/T192156#6002730

Cloudcontrol1004:
    
[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common      keystone nova-api nova-conductor nova-placement-api nova-scheduler      nova-common glance neutron-server python3-requests python3-urllib3      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[x] DEBIAN_FRONTEND=noninteractive  apt-get upgrade      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold" 
[x] puppet agent -tv
[x] puppet agent -tv

cloudnets (one at a time please):

[x] neutron-db-manage upgrade heads on cloudcontrol1003 (( neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head `))
[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install   -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold" neutron-l3-agent      python3-neutronclient  python3-glanceclient

    SERVICE INTERRUPTION?

[x] DEBIAN_FRONTEND=noninteractive apt-get upgrade      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[x] puppet agent -tv
[x] service neutron-l3-agent restart

    SERVICE RESTORED




Cloudvirts (start with one test host first, cloudvirt1023. Don't forget about cloudvirtwdqs ):

[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common  neutron-linuxbridge-agent python3-neutron python3-keystoneauth1      python3-positional python3-requests python3-urllib3  -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[x] DEBIAN_FRONTEND=noninteractive  apt-get upgrade  -y --allow-downgrades    -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[x] puppet agent -tv
[x] service neutron-linuxbridge-agent restart
[x] service nova-compute restart
[x] service libvirtd restart




[x] restore Horizon with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589301/
[x] update IRC topic


/usr/lib/python3/dist-packages/neutron/db/migration/alembic_migrations/versions/rocky/expand/867d39095bf4_port_forwarding.py

CREATE TABLE portforwardings (id VARCHAR(36) NOT NULL, floatingip_id VARCHAR(36) NOT NULL, external_port INTEGER NOT NULL, internal_neutron_port_id VARCHAR(36) NOT NULL, protocol VARCHAR(40) NOT NULL, socket VARCHAR(36) NOT NULL, PRIMARY KEY (id), FOREIGN KEY(floatingip_id) REFERENCES floatingips (id) ON DELETE CASCADE, FOREIGN KEY(internal_neutron_port_id) REFERENCES ports (id) ON DELETE CASCADE, CONSTRAINT uniq_port_forwardings0floatingip_id0external_port UNIQUE (floatingip_id, external_port), CONSTRAINT uniq_port_forwardings0internal_neutron_port_id0socket UNIQUE (internal_neutron_port_id, socket)) ENGINE=InnoDB DEFAULT CHARSET=latin1;

root@cloudcontrol1003:~# neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
  Running upgrade for neutron ...
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 61663558142c -> 867d39095bf4, port forwarding
  OK
• JHedden awarded a token.Apr 16 2020, 9:25 PM
Andrew closed this task as Resolved.Apr 27 2020, 2:02 PM
• JHedden closed subtask T250428: Investigate nova metadata issues that appeared during the Rocky upgrade as Resolved.May 5 2020, 4:44 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL