Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | Andrew | T248635 upgrade cloud-vps openstack to Openstack version 'Rocky' | |||
Resolved | Andrew | T250428 Investigate nova metadata issues that appeared during the Rocky upgrade |
Event Timeline
Change 583940 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove member_role_name from keystone.conf
Change 583940 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove member_role_name from keystone.conf
Change 583989 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova.conf: update conf to suppress deprecation warnings
Change 583990 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron.conf: update a couple of settings in response to deprecation warnings
Change 583994 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate.conf: update a few deprecated settings
Change 583989 merged by Andrew Bogott:
[operations/puppet@production] nova.conf: update conf to suppress deprecation warnings
Change 583990 merged by Andrew Bogott:
[operations/puppet@production] neutron.conf: update a couple of settings in response to deprecation warnings
Change 583994 merged by Andrew Bogott:
[operations/puppet@production] designate.conf: update a few deprecated settings
Change 584000 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron.conf: fix an earlier typo with auth_uri vs auth_url
Change 584000 merged by Andrew Bogott:
[operations/puppet@production] neutron.conf: fix an earlier typo with auth_uri vs auth_url
Change 584007 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack configs: comment out service_token_roles_required = True
Change 584007 merged by Andrew Bogott:
[operations/puppet@production] Openstack configs: comment out service_token_roles_required = True
Change 584011 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: set enabled = True in the [service:worker] section
Change 584011 merged by Andrew Bogott:
[operations/puppet@production] Designate: set enabled = True in the [service:worker] section
Change 584058 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack: add Designate manifest and config for version Rocky
Change 584058 merged by Andrew Bogott:
[operations/puppet@production] Openstack: add Designate manifest and config for version Rocky
Change 585034 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Neutron: add neutron l3 hacks for Rocky
Change 585838 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add config and manifests for version Rocky
Change 585034 merged by Andrew Bogott:
[operations/puppet@production] Openstack Neutron: add neutron l3 hacks for Rocky
Change 585838 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add config and manifests for version Rocky
Change 585844 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] codfw1dev: switch nova, keystone, etc. from queens to rocky
Change 585844 merged by Andrew Bogott:
[operations/puppet@production] codfw1dev: switch nova, keystone, etc. from queens to rocky
Dallas checklist:
The final checklist (with amendments) is:
Cloudcontrols:
- merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
- puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- dump databases: nova, nova-api, neutron, glance, keystone
- puppet agent -tv
- nova-manage db sync
- nova-manage api_db sync
- glance-manage db_sync
- keystone-manage db_sync
- neutron-db-manage upgrade heads (( neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head `))
- puppet agent -tv
- nova-manage db online_data_migrations
Cloudvirts:
- merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
- puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- puppet agent -tv
- service neutron-linuxbridge-agent restart
- service nova-compute restart
- service libvirtd restart
cloudnets:
- merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
- puppet agent -tv
- apt-get update
- DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient
SERVICE INTERRUPTION
- DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
- puppet agent -tv
- service neutron-l3-agent restart
SERVICE RESTORED
Change 585850 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Rocky: pin uwsgi-plugin-python3 uwsgi-core for Rocky
Change 585850 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Rocky: pin uwsgi-plugin-python3 uwsgi-core for Rocky
Change 585854 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: add management of neutron-api-uwsgi.ini and api-paste.ini
Change 585854 merged by Andrew Bogott:
[operations/puppet@production] Neutron: add management of neutron-api-uwsgi.ini and api-paste.ini
Change 585855 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: move uwsgi socket file to /var/run/neutron
Change 585855 merged by Andrew Bogott:
[operations/puppet@production] Neutron: move uwsgi socket file to /var/run/neutron
Change 585895 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron-api: create /var/run/neutron
Change 585895 merged by Andrew Bogott:
[operations/puppet@production] neutron-api: create /var/run/neutron
Change 585917 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: replace l3 hacks for Rocky
Change 585917 merged by Andrew Bogott:
[operations/puppet@production] Neutron: replace l3 hacks for Rocky
Change 585934 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone hacks: update keystone api handling
Change 585934 merged by Andrew Bogott:
[operations/puppet@production] Keystone hacks: update keystone api handling
Change 586105 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: Drop in a backported fix to token_formatters.py
Change 586105 merged by Andrew Bogott:
[operations/puppet@production] Keystone: Drop in a backported fix to token_formatters.py
Change 587292 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: add vm client packages for Rocky
Change 587292 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: add vm client packages for Rocky
Change 587522 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack designate: upgrade eqiad1 to version Rocky
Change 587522 merged by Andrew Bogott:
[operations/puppet@production] OpenStack designate: upgrade eqiad1 to version Rocky
Change 587864 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Designate: fix an encoding issue in the memcached tooz driver
Change 587864 merged by Andrew Bogott:
[operations/puppet@production] Openstack Designate: fix an encoding issue in the memcached tooz driver
Change 587866 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack tooz hack: don't explicitly depend on python3-tooz
Change 587866 merged by Andrew Bogott:
[operations/puppet@production] Openstack tooz hack: don't explicitly depend on python3-tooz
Change 589299 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode for Rocky upgrade
Change 589300 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'queens' to 'rocky'
Change 589299 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode for Rocky upgrade
Change 589300 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'queens' to 'rocky'
This is done. For posterity, here's the etherpad from the upgrade window:
[x] update IRC topic [x] downtime everything in icinga through 14:00CDT ** aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab* ** aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud* [x] downtime Horizon with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589299/ [x] merge puppet patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589300/ Cloudcontrols: * All open database connections post-upgrade https://phabricator.wikimedia.org/P10999 * Checking haproxy `status echo "show stat" | socat /var/run/haproxy/haproxy.sock stdio | grep DOWN` Cloudcontrol1003: [x] puppet agent -tv [x] apt-get update [x] DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" [] DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" [x] dump databases: nova, nova-api, neutron, glance, keystone [x] puppet agent -tv [x] nova-manage api_db sync [x] nova-manage db sync [x] glance-manage db_sync [x] keystone-manage db_sync [x] puppet agent -tv [x] nova-manage db online_data_migrations Encoding issues: * https://phabricator.wikimedia.org/T234830#5802394 * https://phabricator.wikimedia.org/T192156#6002730 Cloudcontrol1004: [x] puppet agent -tv [x] apt-get update [x] DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" [x] DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" [x] puppet agent -tv [x] puppet agent -tv cloudnets (one at a time please): [x] neutron-db-manage upgrade heads on cloudcontrol1003 (( neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head `)) [x] puppet agent -tv [x] apt-get update [x] DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient SERVICE INTERRUPTION? [x] DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" [x] puppet agent -tv [x] service neutron-l3-agent restart SERVICE RESTORED Cloudvirts (start with one test host first, cloudvirt1023. Don't forget about cloudvirtwdqs ): [x] puppet agent -tv [x] apt-get update [x] DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" [x] DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" [x] puppet agent -tv [x] service neutron-linuxbridge-agent restart [x] service nova-compute restart [x] service libvirtd restart [x] restore Horizon with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589301/ [x] update IRC topic /usr/lib/python3/dist-packages/neutron/db/migration/alembic_migrations/versions/rocky/expand/867d39095bf4_port_forwarding.py CREATE TABLE portforwardings (id VARCHAR(36) NOT NULL, floatingip_id VARCHAR(36) NOT NULL, external_port INTEGER NOT NULL, internal_neutron_port_id VARCHAR(36) NOT NULL, protocol VARCHAR(40) NOT NULL, socket VARCHAR(36) NOT NULL, PRIMARY KEY (id), FOREIGN KEY(floatingip_id) REFERENCES floatingips (id) ON DELETE CASCADE, FOREIGN KEY(internal_neutron_port_id) REFERENCES ports (id) ON DELETE CASCADE, CONSTRAINT uniq_port_forwardings0floatingip_id0external_port UNIQUE (floatingip_id, external_port), CONSTRAINT uniq_port_forwardings0internal_neutron_port_id0socket UNIQUE (internal_neutron_port_id, socket)) ENGINE=InnoDB DEFAULT CHARSET=latin1; root@cloudcontrol1003:~# neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Running upgrade for neutron ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade 61663558142c -> 867d39095bf4, port forwarding OK