Page MenuHomePhabricator

upgrade cloud-vps openstack to Openstack version 'Rocky'
Closed, ResolvedPublic

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+2 -6
operations/puppetproduction+1 -1
operations/puppetproduction+5 -7
operations/puppetproduction+1 K -0
operations/puppetproduction+2 -2
operations/puppetproduction+32 -0
operations/puppetproduction+865 -0
operations/puppetproduction+25 -20
operations/puppetproduction+74 -1 K
operations/puppetproduction+12 -8
operations/puppetproduction+1 -4
operations/puppetproduction+82 -1
operations/puppetproduction+14 -0
operations/puppetproduction+2 -2
operations/puppetproduction+10 K -135
operations/puppetproduction+2 K -0
operations/puppetproduction+1 K -0
operations/puppetproduction+5 -0
operations/puppetproduction+7 -3
operations/puppetproduction+2 -2
operations/puppetproduction+2 -3
operations/puppetproduction+2 -1
operations/puppetproduction+3 -5
operations/puppetproduction+0 -1
Show related patches Customize query in gerrit

Event Timeline

Andrew created this task.Mar 26 2020, 10:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 26 2020, 10:24 PM
Andrew triaged this task as Medium priority.Mar 26 2020, 10:24 PM

Change 583940 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove member_role_name from keystone.conf

https://gerrit.wikimedia.org/r/583940

Change 583940 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove member_role_name from keystone.conf

https://gerrit.wikimedia.org/r/583940

Change 583989 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova.conf: update conf to suppress deprecation warnings

https://gerrit.wikimedia.org/r/583989

Change 583990 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron.conf: update a couple of settings in response to deprecation warnings

https://gerrit.wikimedia.org/r/583990

Change 583994 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate.conf: update a few deprecated settings

https://gerrit.wikimedia.org/r/583994

Change 583989 merged by Andrew Bogott:
[operations/puppet@production] nova.conf: update conf to suppress deprecation warnings

https://gerrit.wikimedia.org/r/583989

Change 583990 merged by Andrew Bogott:
[operations/puppet@production] neutron.conf: update a couple of settings in response to deprecation warnings

https://gerrit.wikimedia.org/r/583990

Change 583994 merged by Andrew Bogott:
[operations/puppet@production] designate.conf: update a few deprecated settings

https://gerrit.wikimedia.org/r/583994

Change 584000 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron.conf: fix an earlier typo with auth_uri vs auth_url

https://gerrit.wikimedia.org/r/584000

Change 584000 merged by Andrew Bogott:
[operations/puppet@production] neutron.conf: fix an earlier typo with auth_uri vs auth_url

https://gerrit.wikimedia.org/r/584000

Change 584007 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack configs: comment out service_token_roles_required = True

https://gerrit.wikimedia.org/r/584007

Change 584007 merged by Andrew Bogott:
[operations/puppet@production] Openstack configs: comment out service_token_roles_required = True

https://gerrit.wikimedia.org/r/584007

Change 584011 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: set enabled = True in the [service:worker] section

https://gerrit.wikimedia.org/r/584011

Change 584011 merged by Andrew Bogott:
[operations/puppet@production] Designate: set enabled = True in the [service:worker] section

https://gerrit.wikimedia.org/r/584011

Change 584058 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack: add Designate manifest and config for version Rocky

https://gerrit.wikimedia.org/r/584058

Change 584058 merged by Andrew Bogott:
[operations/puppet@production] Openstack: add Designate manifest and config for version Rocky

https://gerrit.wikimedia.org/r/584058

Change 585034 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Neutron: add neutron l3 hacks for Rocky

https://gerrit.wikimedia.org/r/585034

Change 585838 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add config and manifests for version Rocky

https://gerrit.wikimedia.org/r/585838

Change 585034 merged by Andrew Bogott:
[operations/puppet@production] Openstack Neutron: add neutron l3 hacks for Rocky

https://gerrit.wikimedia.org/r/585034

Change 585838 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add config and manifests for version Rocky

https://gerrit.wikimedia.org/r/585838

Change 585844 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] codfw1dev: switch nova, keystone, etc. from queens to rocky

https://gerrit.wikimedia.org/r/585844

Change 585844 merged by Andrew Bogott:
[operations/puppet@production] codfw1dev: switch nova, keystone, etc. from queens to rocky

https://gerrit.wikimedia.org/r/585844

Andrew added a comment.EditedApr 4 2020, 1:50 AM

Dallas checklist:

The final checklist (with amendments) is:

Cloudcontrols:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • dump databases: nova, nova-api, neutron, glance, keystone
  • puppet agent -tv
  • nova-manage db sync
  • nova-manage api_db sync
  • glance-manage db_sync
  • keystone-manage db_sync
  • neutron-db-manage upgrade heads (( neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head `))
  • puppet agent -tv
  • nova-manage db online_data_migrations

Cloudvirts:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-linuxbridge-agent restart
  • service nova-compute restart
  • service libvirtd restart

cloudnets:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient

    SERVICE INTERRUPTION
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-l3-agent restart

    SERVICE RESTORED

Change 585850 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Rocky: pin uwsgi-plugin-python3 uwsgi-core for Rocky

https://gerrit.wikimedia.org/r/585850

Change 585850 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Rocky: pin uwsgi-plugin-python3 uwsgi-core for Rocky

https://gerrit.wikimedia.org/r/585850

Change 585854 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: add management of neutron-api-uwsgi.ini and api-paste.ini

https://gerrit.wikimedia.org/r/585854

Change 585854 merged by Andrew Bogott:
[operations/puppet@production] Neutron: add management of neutron-api-uwsgi.ini and api-paste.ini

https://gerrit.wikimedia.org/r/585854

Change 585855 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: move uwsgi socket file to /var/run/neutron

https://gerrit.wikimedia.org/r/585855

Change 585855 merged by Andrew Bogott:
[operations/puppet@production] Neutron: move uwsgi socket file to /var/run/neutron

https://gerrit.wikimedia.org/r/585855

Change 585895 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron-api: create /var/run/neutron

https://gerrit.wikimedia.org/r/585895

Change 585895 merged by Andrew Bogott:
[operations/puppet@production] neutron-api: create /var/run/neutron

https://gerrit.wikimedia.org/r/585895

Change 585917 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: replace l3 hacks for Rocky

https://gerrit.wikimedia.org/r/585917

Change 585917 merged by Andrew Bogott:
[operations/puppet@production] Neutron: replace l3 hacks for Rocky

https://gerrit.wikimedia.org/r/585917

Change 585934 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone hacks: update keystone api handling

https://gerrit.wikimedia.org/r/585934

Change 585934 merged by Andrew Bogott:
[operations/puppet@production] Keystone hacks: update keystone api handling

https://gerrit.wikimedia.org/r/585934

Change 586105 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: Drop in a backported fix to token_formatters.py

https://gerrit.wikimedia.org/r/586105

Change 586105 merged by Andrew Bogott:
[operations/puppet@production] Keystone: Drop in a backported fix to token_formatters.py

https://gerrit.wikimedia.org/r/586105

Change 587292 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: add vm client packages for Rocky

https://gerrit.wikimedia.org/r/587292

Change 587292 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: add vm client packages for Rocky

https://gerrit.wikimedia.org/r/587292

Change 587522 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack designate: upgrade eqiad1 to version Rocky

https://gerrit.wikimedia.org/r/587522

Change 587522 merged by Andrew Bogott:
[operations/puppet@production] OpenStack designate: upgrade eqiad1 to version Rocky

https://gerrit.wikimedia.org/r/587522

Change 587864 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Designate: fix an encoding issue in the memcached tooz driver

https://gerrit.wikimedia.org/r/587864

Change 587864 merged by Andrew Bogott:
[operations/puppet@production] Openstack Designate: fix an encoding issue in the memcached tooz driver

https://gerrit.wikimedia.org/r/587864

Change 587866 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack tooz hack: don't explicitly depend on python3-tooz

https://gerrit.wikimedia.org/r/587866

Change 587866 merged by Andrew Bogott:
[operations/puppet@production] Openstack tooz hack: don't explicitly depend on python3-tooz

https://gerrit.wikimedia.org/r/587866

Change 589299 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode for Rocky upgrade

https://gerrit.wikimedia.org/r/589299

Change 589300 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'queens' to 'rocky'

https://gerrit.wikimedia.org/r/589300

Change 589299 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode for Rocky upgrade

https://gerrit.wikimedia.org/r/589299

Change 589300 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'queens' to 'rocky'

https://gerrit.wikimedia.org/r/589300

This is done. For posterity, here's the etherpad from the upgrade window:

[x] update IRC topic
[x] downtime everything in icinga through 14:00CDT
** aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab*
** aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
[x] downtime Horizon with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589299/
[x] merge puppet patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589300/

Cloudcontrols:

* All open database connections post-upgrade https://phabricator.wikimedia.org/P10999
* Checking haproxy `status echo "show stat" | socat /var/run/haproxy/haproxy.sock stdio | grep  DOWN`
Cloudcontrol1003:
    
[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common      keystone nova-api nova-conductor nova-placement-api nova-scheduler      nova-common glance neutron-server python3-requests python3-urllib3      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[] DEBIAN_FRONTEND=noninteractive  apt-get upgrade      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[x] dump databases: nova, nova-api, neutron, glance, keystone 
[x] puppet agent -tv
[x] nova-manage api_db sync
[x] nova-manage db sync
[x] glance-manage db_sync
[x] keystone-manage db_sync
[x] puppet agent -tv
[x] nova-manage db online_data_migrations

Encoding issues:
* https://phabricator.wikimedia.org/T234830#5802394
* https://phabricator.wikimedia.org/T192156#6002730

Cloudcontrol1004:
    
[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common      keystone nova-api nova-conductor nova-placement-api nova-scheduler      nova-common glance neutron-server python3-requests python3-urllib3      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[x] DEBIAN_FRONTEND=noninteractive  apt-get upgrade      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold" 
[x] puppet agent -tv
[x] puppet agent -tv

cloudnets (one at a time please):

[x] neutron-db-manage upgrade heads on cloudcontrol1003 (( neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head `))
[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install   -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold" neutron-l3-agent      python3-neutronclient  python3-glanceclient

    SERVICE INTERRUPTION?

[x] DEBIAN_FRONTEND=noninteractive apt-get upgrade      -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[x] puppet agent -tv
[x] service neutron-l3-agent restart

    SERVICE RESTORED




Cloudvirts (start with one test host first, cloudvirt1023. Don't forget about cloudvirtwdqs ):

[x] puppet agent -tv
[x] apt-get update
[x] DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common  neutron-linuxbridge-agent python3-neutron python3-keystoneauth1      python3-positional python3-requests python3-urllib3  -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[x] DEBIAN_FRONTEND=noninteractive  apt-get upgrade  -y --allow-downgrades    -o "Dpkg::Options::=--force-confdef" -o      "Dpkg::Options::=--force-confold"
[x] puppet agent -tv
[x] service neutron-linuxbridge-agent restart
[x] service nova-compute restart
[x] service libvirtd restart




[x] restore Horizon with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/589301/
[x] update IRC topic


/usr/lib/python3/dist-packages/neutron/db/migration/alembic_migrations/versions/rocky/expand/867d39095bf4_port_forwarding.py

CREATE TABLE portforwardings (id VARCHAR(36) NOT NULL, floatingip_id VARCHAR(36) NOT NULL, external_port INTEGER NOT NULL, internal_neutron_port_id VARCHAR(36) NOT NULL, protocol VARCHAR(40) NOT NULL, socket VARCHAR(36) NOT NULL, PRIMARY KEY (id), FOREIGN KEY(floatingip_id) REFERENCES floatingips (id) ON DELETE CASCADE, FOREIGN KEY(internal_neutron_port_id) REFERENCES ports (id) ON DELETE CASCADE, CONSTRAINT uniq_port_forwardings0floatingip_id0external_port UNIQUE (floatingip_id, external_port), CONSTRAINT uniq_port_forwardings0internal_neutron_port_id0socket UNIQUE (internal_neutron_port_id, socket)) ENGINE=InnoDB DEFAULT CHARSET=latin1;

root@cloudcontrol1003:~# neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
  Running upgrade for neutron ...
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 61663558142c -> 867d39095bf4, port forwarding
  OK
Andrew closed this task as Resolved.Apr 27 2020, 2:02 PM