Page MenuHomePhabricator

upgrade cloud-vps openstack to Openstack version 'Ussuri'
Closed, ResolvedPublic

Description

The Designate hosts (cloudservices2003/2004) are already running Ussuri. The current Horizon deploy is forward-compatible with U. So that leaves the cloudcontrol, cloudnet, and cloudvirt nodes to upgrade.

  • update IRC topic
  • downtime everything in icinga through 14:00CDT

    aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab*

    aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
  • dump databases on cloudcontrol1003: nova_eqiad1, nova_api_eqiad1, nova_cell0_eqiad1, neutron, glance, keystone, cinder:
    1. mysqldump -u root nova_eqiad1 > /root/traindbbackups/nova_eqiad1.sql
    2. mysqldump -u root nova_api_eqiad1 > /root/traindbbackups/nova_api_eqiad1.sql
    3. mysqldump -u root nova_cell0_eqiad1 > /root/traindbbackups/nova_cell0_eqiad1.sql
    4. mysqldump -u root neutron > /root/traindbbackups/neutron.sql
    5. mysqldump -u root glance > /root/traindbbackups/glance.sql
    6. mysqldump -u root placement > /root/traindbbackups/placement.sql
    7. mysqldump -u root keystone > /root/traindbbackups/keystone.sql

Cloudcontrols:

All open database connections post-upgrade https://phabricator.wikimedia.org/P10999
Checking haproxy status echo "show stat" | socat /var/run/haproxy/haproxy.sock stdio | grep DOWN

Cloudcontrol1003:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • nova-manage api_db sync
  • nova-manage db sync
  • placement-manage db sync
  • glance-manage db_sync
  • keystone-manage db_sync
  • cinder-manage db online_data_migrations
  • cinder-manage db sync
  • puppet agent -tv
  • nova-manage db online_data_migrations

Cloudcontrol1004:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api placement-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • puppet agent -tv

Cloudcontrol1005:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api placement-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • puppet agent -tv

cloudnets (one at a time please):

Begin with the standby node, as determined with:

$ neutron l3-agent-list-hosting-router cloudinstances2b-gw

Standby node:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-oslo.messaging python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • neutron-db-manage upgrade heads on cloudcontrol1003

Active node:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-oslo.messaging python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv

Break Time

Cloudvirts (start with one test host first, cloudvirt1039. Don't forget about cloudvirtwdqs ):

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-eventlet python3-oslo.messaging python3-taskflow python3-tooz python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-linuxbridge-agent restart
  • service libvirtd restart
  • service nova-compute restart
  • update IRC topic
  • enable puppet on all cloud* hosts

    $ sudo cumin 'cloud*' "enable-puppet 'Upgrading to openstack Train - T261135 - ${USER}'"

Things to check

  • Check 'openstack region list'. There should be exactly one region, eqiad1-r. If there is a second region named 'RegionOne' (this happened in codfw1dev), delete it; otherwise scripts that enumerate regions will be confused.
  • Clean up VMs in the admin-monitoring project that leaked during upgrade; delete them.
  • Create a new VM and confirm that DNS and ssh work properly
  • Logs will be extremely noisy about policy deprecations and value checks; this is expected because OpenStack is poised between two different policy systems; our existing policies are still (noisily) supported in U.

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+0 -16 K
operations/puppetproduction+1 -1
operations/puppetproduction+54 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+4 -0
operations/puppetproduction+0 -121
operations/puppetproduction+0 -3
operations/puppetproduction+13 -13
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+0 -3
operations/puppetproduction+0 -1 K
operations/puppetproduction+18 -0
operations/puppetproduction+3 -6
operations/puppetproduction+17 K -0
operations/puppetproduction+133 -0
operations/puppetproduction+2 -2
operations/puppetproduction+1 K -0
operations/puppetproduction+144 -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
ResolvedAndrew
Resolvedtaavi
ResolvedAndrew
ResolvedAndrew
Resolveddcaro
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
Resolved Cmjohnson
Resolvedayounsi
Resolvedaborrero
Resolved Cmjohnson
ResolvedJclark-ctr
ResolvedJclark-ctr
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
Resolveddcaro
Resolvedaborrero
Declineddcaro
Resolveddcaro
OpenNone
OpenNone
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedBstorm
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
InvalidNone

Event Timeline

Change 676172 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] OpenStack: add package manifests for Ussuri

https://gerrit.wikimedia.org/r/676172

Change 676173 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Add Designate files and manifests for version Ussuri

https://gerrit.wikimedia.org/r/676173

Change 676174 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Codfw1dev designate -> OpenStack version Ussuri

https://gerrit.wikimedia.org/r/676174

Change 676172 merged by Andrew Bogott:

[operations/puppet@production] OpenStack: add package manifests for Ussuri

https://gerrit.wikimedia.org/r/676172

Change 676173 merged by Andrew Bogott:

[operations/puppet@production] Add Designate files and manifests for version Ussuri

https://gerrit.wikimedia.org/r/676173

Change 676174 merged by Andrew Bogott:

[operations/puppet@production] Codfw1dev designate -> OpenStack version Ussuri

https://gerrit.wikimedia.org/r/676174

Change 676453 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] OpenStack: add manifests, files and templates for version Ussuri

https://gerrit.wikimedia.org/r/676453

Change 676461 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] OpenStack Designate: forward the domain deletion fix to Ussuri

https://gerrit.wikimedia.org/r/676461

Change 676461 merged by Andrew Bogott:

[operations/puppet@production] OpenStack Designate: forward the domain deletion fix to Ussuri

https://gerrit.wikimedia.org/r/676461

Change 676464 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Openstack Nova: update config for Ussuri

https://gerrit.wikimedia.org/r/676464

Change 676466 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Openstack Neutron: update our l3 agent hacks for Ussuri

https://gerrit.wikimedia.org/r/676466

Change 676467 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Neutron: Remove ip_lib.py hack for Ussuri

https://gerrit.wikimedia.org/r/676467

Change 676468 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] OpenStack Glance: update config for Ussuri

https://gerrit.wikimedia.org/r/676468

Change 676584 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] codfw1dev OpenStack to version Ussuri

https://gerrit.wikimedia.org/r/676584

Change 676453 merged by Andrew Bogott:

[operations/puppet@production] OpenStack: add manifests, files and templates for version Ussuri

https://gerrit.wikimedia.org/r/676453

Change 676464 merged by Andrew Bogott:

[operations/puppet@production] Openstack Nova: update config for Ussuri

https://gerrit.wikimedia.org/r/676464

Change 676466 merged by Andrew Bogott:

[operations/puppet@production] Openstack Neutron: update our l3 agent hacks for Ussuri

https://gerrit.wikimedia.org/r/676466

Change 676467 merged by Andrew Bogott:

[operations/puppet@production] Neutron: Remove ip_lib.py hack for Ussuri

https://gerrit.wikimedia.org/r/676467

Change 676468 merged by Andrew Bogott:

[operations/puppet@production] OpenStack Glance: update config for Ussuri

https://gerrit.wikimedia.org/r/676468

Change 676584 merged by Andrew Bogott:

[operations/puppet@production] codfw1dev OpenStack to version Ussuri

https://gerrit.wikimedia.org/r/676584

Change 676621 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Openstack Glance: fix init-script to be ipv4 only

https://gerrit.wikimedia.org/r/676621

Change 676621 merged by Andrew Bogott:

[operations/puppet@production] Openstack Glance: fix init-script to be ipv4 only

https://gerrit.wikimedia.org/r/676621

Change 676628 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Openstack Keystone: update our hacked projects.py for Ussuri

https://gerrit.wikimedia.org/r/676628

Change 676628 merged by Andrew Bogott:

[operations/puppet@production] Openstack Keystone: update our hacked projects.py for Ussuri

https://gerrit.wikimedia.org/r/676628

Change 676644 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Openstack Nova policy.yaml: remove all keypair-related policies

https://gerrit.wikimedia.org/r/676644

Change 676644 merged by Andrew Bogott:

[operations/puppet@production] Openstack Nova policy.yaml: remove all keypair-related policies

https://gerrit.wikimedia.org/r/676644

Change 676778 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Ussury policy.yaml files: remove redundant rules

https://gerrit.wikimedia.org/r/676778

Change 676778 merged by Andrew Bogott:

[operations/puppet@production] Ussury policy.yaml files: remove redundant rules

https://gerrit.wikimedia.org/r/676778

Change 676815 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] eqiad1 designate -> Ussuri

https://gerrit.wikimedia.org/r/676815

Change 676815 merged by Andrew Bogott:

[operations/puppet@production] eqiad1 designate -> Ussuri

https://gerrit.wikimedia.org/r/676815

Change 676847 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Horizon: put into maintenance mode for Ussuri upgrade

https://gerrit.wikimedia.org/r/676847

Change 676848 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Openstack eqiad1 -> Ussuri

https://gerrit.wikimedia.org/r/676848

Andrew updated the task description. (Show Details)
Andrew updated the task description. (Show Details)
Andrew updated the task description. (Show Details)
Andrew added a subscriber: aborrero.

heads up: the codfw1dev openstack deployment no longer uses our neutron hacks, it uses cloudgw instead. I'm planning on migrating eqiad1 to cloudgw "soon" (within the month or the next) see T270704: cloud: introduce new edge network architecture for eqiad1 and codfw1dev

Change 676847 merged by David Caro:

[operations/puppet@production] Horizon: put into maintenance mode for Ussuri upgrade

https://gerrit.wikimedia.org/r/676847

Change 676848 merged by David Caro:

[operations/puppet@production] Openstack eqiad1 -> Ussuri

https://gerrit.wikimedia.org/r/676848

Change 677323 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Added openstack::clientpackages::ussuri::stretch

https://gerrit.wikimedia.org/r/677323

Change 677323 merged by Andrew Bogott:

[operations/puppet@production] Added openstack::clientpackages::ussuri::stretch

https://gerrit.wikimedia.org/r/677323

Change 677335 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Pontoon: update openstack version to Ussuri

https://gerrit.wikimedia.org/r/677335

Change 677337 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] OpenStack: remove config and manifests for version 'Stein'

https://gerrit.wikimedia.org/r/677337

Change 677335 merged by Andrew Bogott:

[operations/puppet@production] Pontoon: update openstack version to Ussuri

https://gerrit.wikimedia.org/r/677335

Change 677337 merged by Andrew Bogott:

[operations/puppet@production] OpenStack: remove config and manifests for version 'Stein'

https://gerrit.wikimedia.org/r/677337