Page MenuHomePhabricator

upgrade cloud-vps openstack to Openstack version 'Queens'
Closed, ResolvedPublic

Description

As far as I know there's nothing preventing us from speeding ahead to Queens.

Horizon is already running version 'train' so this task does not apply to Horizon.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+14 -6
operations/puppetproduction+1 -44
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+3 -3
operations/puppetproduction+3 -3
operations/puppetproduction+1 -1
operations/puppetproduction+16 -0
operations/puppetproduction+2 -3
operations/puppetproduction+3 -3
operations/puppetproduction+10 -8
operations/puppetproduction+112 -42
operations/puppetproduction+235 -0
operations/puppetproduction+2 -1
operations/puppetproduction+4 -0
operations/puppetproduction+2 -2
operations/puppetproduction+1 -0
operations/puppetproduction+32 -0
Show related patches Customize query in gerrit

Event Timeline

Here are the steps from the O -> P upgrade, rearranged based on what we've learned.

  • downtime everything
  • on cloudcontrol1003 and cloudcontrol1004:
    • apply puppet
    • DEBIAN_FRONTEND=noninteractive apt-get install nova-common glance keystone nova-placement-api nova-api nova-conductor neutron-server -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
    • apt-get upgrade
    • apply puppet again
    • disable puppet
    • stop nova-conductor, nova-api, neutron-server, nova-scheduler sudo systemctl stop nova-conductor nova-api neutron-server nova-scheduler apache2
  • on cloudcontrol1003:
    • keystone-manage db_sync
    • nova-manage db sync

      root@cloudcontrol1003:~# nova-manage db sync

Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
Option "notification_topics" from group "DEFAULT" is deprecated. Use option "topics" from group "oslo_messaging_notifications".

  • glance-manage db sync
  • nova-manage api_db sync

Neutron rolling upgrade order:

  1. stop and disable neutron-server
  2. upgrade neutron packages on cloudcontrol100[34]
  3. ensure neutron-server is stopped and run neutron-db-manage upgrade head
  4. restart neutron-server
  5. upgrade and restart agents on cloudnet and cloudvirt
  • on cloudcontrol1003 and cloudcontrol1004:
    • re-enable puppet
    • re-run puppet agent
  • on cloudnet1003 and cloudnet1004: (Move to after neutron-db-manage)
    • DEBIAN_FRONTEND=noninteractive apt-get install python-openstackclient python-openstacksdk neutron-common neutron-dhcp-agent neutron-l3-agent neutron-linuxbridge-agent neutron-metadata-agent python-neutron python-neutron-lib python-neutronclient -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
    • DEBIAN_FRONTEND=noninteractive apt-get upgrade (needed to resolve neutron linuxbridge dependencies)
    • puppet agent -tv
  • on cloudcontrol1003:
    • neutron-db-manage upgrade head (ensure neutron-server is stopped before running)
  • on cloudnet1003 and cloudnet1004: (only after neutron-db-manage)
    • puppet agent -tv
    • restart all neutron services sudo systemctl restart neutron*
  • on cloudvirts (via cumin): (only after neutron-db-manage)
    • puppet agent -tv
    • DEBIAN_FRONTEND=noninteractive apt-get install nova-compute nova-common nova-compute-kvm python-nova python-novaclient neutron-common neutron-linuxbridge-agent python-neutron python-neutron-lib python-neutronclient -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
    • DEBIAN_FRONTEND=noninteractive apt-get upgrade (needed to resolve neutron linuxbridge dependencies)
    • puppet agent -tv
    • service neutron-linuxbridge-agent restart
    • service nova-compute restart
  • merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/564658/, apply on labweb hosts
Bstorm triaged this task as Medium priority.Feb 11 2020, 4:13 PM

Change 576399 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: install python3 versions of sink handlers

https://gerrit.wikimedia.org/r/576399

Change 576399 merged by Andrew Bogott:
[operations/puppet@production] designate: install python3 versions of sink handlers

https://gerrit.wikimedia.org/r/576399

Change 576403 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: include python3-git -- we need this for wmfsink

https://gerrit.wikimedia.org/r/576403

Change 576403 merged by Andrew Bogott:
[operations/puppet@production] designate: include python3-git -- we need this for wmfsink

https://gerrit.wikimedia.org/r/576403

Change 576410 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmf_sink: remove utf8 encoding

https://gerrit.wikimedia.org/r/576410

Change 576410 merged by Andrew Bogott:
[operations/puppet@production] wmf_sink: remove utf8 encoding

https://gerrit.wikimedia.org/r/576410

Change 576972 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudservices: update designate to openstack Queens

https://gerrit.wikimedia.org/r/576972

Change 576972 merged by Andrew Bogott:
[operations/puppet@production] cloudservices: update designate to openstack Queens

https://gerrit.wikimedia.org/r/576972

Change 577347 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: use python2 mod_wsgi for pike but python3 for queens

https://gerrit.wikimedia.org/r/577347

Change 577347 merged by Andrew Bogott:
[operations/puppet@production] keystone: use python2 mod_wsgi for pike but python3 for queens

https://gerrit.wikimedia.org/r/577347

Change 577358 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack glance queens: override package-installed init script

https://gerrit.wikimedia.org/r/577358

Change 577358 merged by Andrew Bogott:
[operations/puppet@production] Openstack glance queens: override package-installed init script

https://gerrit.wikimedia.org/r/577358

Change 577620 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] openstack haproxy: change glance-api health check

https://gerrit.wikimedia.org/r/577620

Change 577625 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-placement: update the usgi init script for Queens

https://gerrit.wikimedia.org/r/577625

Change 577625 merged by Andrew Bogott:
[operations/puppet@production] nova-placement: update the uwsgi init script for Queens

https://gerrit.wikimedia.org/r/577625

Change 577630 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-placement haproxy: update the health check route

https://gerrit.wikimedia.org/r/577630

Change 577620 merged by Andrew Bogott:
[operations/puppet@production] openstack haproxy: change glance-api health check

https://gerrit.wikimedia.org/r/577620

Change 577630 merged by Andrew Bogott:
[operations/puppet@production] nova-placement haproxy: update the health check route

https://gerrit.wikimedia.org/r/577630

Change 578359 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfnovamiddleware: adjust string encoding for python3

https://gerrit.wikimedia.org/r/578359

Change 578359 merged by Andrew Bogott:
[operations/puppet@production] wmfnovamiddleware: adjust string encoding for python3

https://gerrit.wikimedia.org/r/578359

Change 578378 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack queens packages: absent some python2 packages

https://gerrit.wikimedia.org/r/578378

Change 578378 merged by Andrew Bogott:
[operations/puppet@production] Openstack queens packages: absent some python2 packages

https://gerrit.wikimedia.org/r/578378

Change 581104 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron metadata_agent.ini: use nova_metadata_host instead of nova_metadata_ip

https://gerrit.wikimedia.org/r/581104

Change 581104 merged by Andrew Bogott:
[operations/puppet@production] Neutron metadata_agent.ini: use nova_metadata_host instead of nova_metadata_ip

https://gerrit.wikimedia.org/r/581104

Change 581210 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfnovamiddleware: adjust encoding, again

https://gerrit.wikimedia.org/r/581210

Change 581210 merged by Andrew Bogott:
[operations/puppet@production] wmfnovamiddleware: adjust encoding, again

https://gerrit.wikimedia.org/r/581210

Change 581666 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-compute: change virt_type to qemu

https://gerrit.wikimedia.org/r/581666

Change 581666 merged by Andrew Bogott:
[operations/puppet@production] nova-compute: change virt_type to qemu

https://gerrit.wikimedia.org/r/581666

Change 581682 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-compute: install version-specific config

https://gerrit.wikimedia.org/r/581682

Change 581682 merged by Andrew Bogott:
[operations/puppet@production] nova-compute: install version-specific config

https://gerrit.wikimedia.org/r/581682

Change 583599 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: queens: don't install libpam-systemd from bpo

https://gerrit.wikimedia.org/r/583599

Change 583599 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: queens: don't install libpam-systemd from bpo

https://gerrit.wikimedia.org/r/583599

Change 583600 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put in maintenance mode for the pike=>queens upgrade

https://gerrit.wikimedia.org/r/583600

Change 583601 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack: move eqiad1 to version 'queens'

https://gerrit.wikimedia.org/r/583601

Change 583600 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put in maintenance mode for the pike=>queens upgrade

https://gerrit.wikimedia.org/r/583600

Mentioned in SAL (#wikimedia-cloud) [2020-03-26T15:01:09Z] <andrewbogott> beginning openstack upgrade window for T242766

Change 583601 merged by Andrew Bogott:
[operations/puppet@production] Openstack: move eqiad1 to version 'queens'

https://gerrit.wikimedia.org/r/583601

Change 583680 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: queens: drop python2 packages

https://gerrit.wikimedia.org/r/583680

Andrew closed this task as Resolved.EditedMar 26 2020, 9:12 PM

The final checklist (with amendments) is:

Cloudcontrols:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • dump databases: nova, nova-api, neutron, glance, keystone
  • puppet agent -tv
  • nova-manage db sync
  • nova-manage api_db sync
  • glance-manage db_sync
  • keystone-manage db_sync
  • neutron-db-manage upgrade heads (( neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head `))
  • Check to see if ceph user exists ('groups ceph'). If not, 'dpkg-reconfigure ceph-common'

Cloudvirts:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • touch /usr/share/nova-common/policy.json
  • DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • service virtlogd restart
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • service virtlogd restart
  • puppet agent -tv
  • service neutron-linuxbridge-agent restart
  • service nova-compute restart
  • service libvirtd restart
  • dpkg-reconfigure ceph-common

cloudnets:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-l3-agent restart

There was also a lot of encoding nonsense -- I changed all of the glance tables to have a default encoding of utf8, and we also had to change the specific alembic version column to utf8. For Neutron we didn't re-encode anything but I did hack the upgrade code to skip table creation and then created a table by hand.

Soon we'll rebuild and re-import these databases and then hopefully they'll have more modern encodings.

For cumin runs on cloudvirts:

cumin 'P{R:Class = role::wmcs::openstack::eqiad1::virt or R:Class = role::wmcs::openstack::eqiad1::virt_ceph}'

Change 583680 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: queens: drop python2 packages

https://gerrit.wikimedia.org/r/583680

Change 587741 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmf_sink: update conf settings

https://gerrit.wikimedia.org/r/587741

Change 587741 merged by Andrew Bogott:
[operations/puppet@production] wmf_sink: update conf settings

https://gerrit.wikimedia.org/r/587741