Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T248635 upgrade cloud-vps openstack to Openstack version 'Rocky' | |||
| Resolved | Andrew | T250428 Investigate nova metadata issues that appeared during the Rocky upgrade |
Event Timeline
Comment Actions
Change 583940 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove member_role_name from keystone.conf
Comment Actions
Change 583940 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove member_role_name from keystone.conf
Comment Actions
Change 583989 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova.conf: update conf to suppress deprecation warnings
Comment Actions
Change 583990 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron.conf: update a couple of settings in response to deprecation warnings
Comment Actions
Change 583994 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate.conf: update a few deprecated settings
Comment Actions
Change 583989 merged by Andrew Bogott:
[operations/puppet@production] nova.conf: update conf to suppress deprecation warnings
Comment Actions
Change 583990 merged by Andrew Bogott:
[operations/puppet@production] neutron.conf: update a couple of settings in response to deprecation warnings
Comment Actions
Change 583994 merged by Andrew Bogott:
[operations/puppet@production] designate.conf: update a few deprecated settings
Comment Actions
Change 584000 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron.conf: fix an earlier typo with auth_uri vs auth_url
Comment Actions
Change 584000 merged by Andrew Bogott:
[operations/puppet@production] neutron.conf: fix an earlier typo with auth_uri vs auth_url
Comment Actions
Change 584007 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack configs: comment out service_token_roles_required = True
Comment Actions
Change 584007 merged by Andrew Bogott:
[operations/puppet@production] Openstack configs: comment out service_token_roles_required = True
Comment Actions
Change 584011 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: set enabled = True in the [service:worker] section
Comment Actions
Change 584011 merged by Andrew Bogott:
[operations/puppet@production] Designate: set enabled = True in the [service:worker] section
Comment Actions
Change 584058 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack: add Designate manifest and config for version Rocky
Comment Actions
Change 584058 merged by Andrew Bogott:
[operations/puppet@production] Openstack: add Designate manifest and config for version Rocky
Comment Actions
Change 585034 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Neutron: add neutron l3 hacks for Rocky
Comment Actions
Change 585838 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add config and manifests for version Rocky
Comment Actions
Change 585034 merged by Andrew Bogott:
[operations/puppet@production] Openstack Neutron: add neutron l3 hacks for Rocky
Comment Actions
Change 585838 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add config and manifests for version Rocky
Comment Actions
Change 585844 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] codfw1dev: switch nova, keystone, etc. from queens to rocky
Comment Actions
Change 585844 merged by Andrew Bogott:
[operations/puppet@production] codfw1dev: switch nova, keystone, etc. from queens to rocky
Comment Actions
Dallas checklist:
The final checklist (with amendments) is:
Cloudcontrols:
- merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
- puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- dump databases: nova, nova-api, neutron, glance, keystone
- puppet agent -tv
- nova-manage db sync
- nova-manage api_db sync
- glance-manage db_sync
- keystone-manage db_sync
- neutron-db-manage upgrade heads (( neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head `))
- puppet agent -tv
- nova-manage db online_data_migrations
Cloudvirts:
- merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
- puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- puppet agent -tv
- service neutron-linuxbridge-agent restart
- service nova-compute restart
- service libvirtd restart
cloudnets:
- merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
- puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient
SERVICE INTERRUPTION
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- puppet agent -tv
- service neutron-l3-agent restart
SERVICE RESTORED
Comment Actions
Change 585850 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Rocky: pin uwsgi-plugin-python3 uwsgi-core for Rocky
Comment Actions
Change 585850 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Rocky: pin uwsgi-plugin-python3 uwsgi-core for Rocky
Comment Actions
Change 585854 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: add management of neutron-api-uwsgi.ini and api-paste.ini
Comment Actions
Change 585854 merged by Andrew Bogott:
[operations/puppet@production] Neutron: add management of neutron-api-uwsgi.ini and api-paste.ini
Comment Actions
Change 585855 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: move uwsgi socket file to /var/run/neutron
Comment Actions
Change 585855 merged by Andrew Bogott:
[operations/puppet@production] Neutron: move uwsgi socket file to /var/run/neutron
Comment Actions
Change 585895 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron-api: create /var/run/neutron
Comment Actions
Change 585895 merged by Andrew Bogott:
[operations/puppet@production] neutron-api: create /var/run/neutron
Comment Actions
Change 585917 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: replace l3 hacks for Rocky
Comment Actions
Change 585917 merged by Andrew Bogott:
[operations/puppet@production] Neutron: replace l3 hacks for Rocky
Comment Actions
Change 585934 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone hacks: update keystone api handling
Comment Actions
Change 585934 merged by Andrew Bogott:
[operations/puppet@production] Keystone hacks: update keystone api handling
Comment Actions
Change 586105 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: Drop in a backported fix to token_formatters.py
Comment Actions
Change 586105 merged by Andrew Bogott:
[operations/puppet@production] Keystone: Drop in a backported fix to token_formatters.py
Comment Actions
Change 587292 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: add vm client packages for Rocky
Comment Actions
Change 587292 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: add vm client packages for Rocky
Comment Actions
Change 587522 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack designate: upgrade eqiad1 to version Rocky
Comment Actions
Change 587522 merged by Andrew Bogott:
[operations/puppet@production] OpenStack designate: upgrade eqiad1 to version Rocky
Comment Actions
Change 587864 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Designate: fix an encoding issue in the memcached tooz driver
Comment Actions
Change 587864 merged by Andrew Bogott:
[operations/puppet@production] Openstack Designate: fix an encoding issue in the memcached tooz driver
Comment Actions
Change 587866 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack tooz hack: don't explicitly depend on python3-tooz
Comment Actions
Change 587866 merged by Andrew Bogott:
[operations/puppet@production] Openstack tooz hack: don't explicitly depend on python3-tooz
Comment Actions
Change 589299 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode for Rocky upgrade
Comment Actions
Change 589300 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'queens' to 'rocky'
Comment Actions
Change 589299 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode for Rocky upgrade
Comment Actions
Change 589300 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'queens' to 'rocky'
Comment Actions
This is done. For posterity, here's the etherpad from the upgrade window:
[x] update IRC topic
[x] downtime everything in icinga through 14:00CDT
** aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab*
** aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
[x] downtime Horizon with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589299/
[x] merge puppet patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589300/
Cloudcontrols:
* All open database connections post-upgrade https://phabricator.wikimedia.org/P10999
* Checking haproxy `status echo "show stat" | socat /var/run/haproxy/haproxy.sock stdio | grep DOWN`
Cloudcontrol1003:
[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
[] DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
[x] dump databases: nova, nova-api, neutron, glance, keystone
[x] puppet agent -tv
[x] nova-manage api_db sync
[x] nova-manage db sync
[x] glance-manage db_sync
[x] keystone-manage db_sync
[x] puppet agent -tv
[x] nova-manage db online_data_migrations
Encoding issues:
* https://phabricator.wikimedia.org/T234830#5802394
* https://phabricator.wikimedia.org/T192156#6002730
Cloudcontrol1004:
[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
[x] DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
[x] puppet agent -tv
[x] puppet agent -tv
cloudnets (one at a time please):
[x] neutron-db-manage upgrade heads on cloudcontrol1003 (( neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head `))
[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient
SERVICE INTERRUPTION?
[x] DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
[x] puppet agent -tv
[x] service neutron-l3-agent restart
SERVICE RESTORED
Cloudvirts (start with one test host first, cloudvirt1023. Don't forget about cloudvirtwdqs ):
[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
[x] DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
[x] puppet agent -tv
[x] service neutron-linuxbridge-agent restart
[x] service nova-compute restart
[x] service libvirtd restart
[x] restore Horizon with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589301/
[x] update IRC topic
/usr/lib/python3/dist-packages/neutron/db/migration/alembic_migrations/versions/rocky/expand/867d39095bf4_port_forwarding.py
CREATE TABLE portforwardings (id VARCHAR(36) NOT NULL, floatingip_id VARCHAR(36) NOT NULL, external_port INTEGER NOT NULL, internal_neutron_port_id VARCHAR(36) NOT NULL, protocol VARCHAR(40) NOT NULL, socket VARCHAR(36) NOT NULL, PRIMARY KEY (id), FOREIGN KEY(floatingip_id) REFERENCES floatingips (id) ON DELETE CASCADE, FOREIGN KEY(internal_neutron_port_id) REFERENCES ports (id) ON DELETE CASCADE, CONSTRAINT uniq_port_forwardings0floatingip_id0external_port UNIQUE (floatingip_id, external_port), CONSTRAINT uniq_port_forwardings0internal_neutron_port_id0socket UNIQUE (internal_neutron_port_id, socket)) ENGINE=InnoDB DEFAULT CHARSET=latin1;
root@cloudcontrol1003:~# neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Running upgrade for neutron ...
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 61663558142c -> 867d39095bf4, port forwarding
OK