Page MenuHomePhabricator

upgrade cloud-vps openstack to Openstack version 'Train'
Closed, ResolvedPublic

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
operations/puppetproduction+0 -1
operations/puppetproduction+1 -1
operations/puppetproduction+2 -6
operations/puppetproduction+1 K -0
operations/puppetproduction+1 -0
operations/puppetproduction+313 -0
operations/puppetproduction+2 -9
operations/puppetproduction+2 -9
operations/puppetproduction+218 -190
operations/puppetproduction+57 -25
operations/puppetproduction+263 -0
operations/puppetproduction+9 -1
operations/puppetproduction+172 -0
operations/puppetproduction+63 -0
operations/puppetproduction+59 -0
operations/puppetproduction+169 -112
operations/puppetproduction+18 -0
operations/puppetproduction+40 -0
operations/puppetproduction+149 -0
operations/puppetproduction+12 K -0
operations/puppetproduction+4 -0
operations/puppetproduction+52 -71
operations/puppetproduction+1 K -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
ResolvedAndrew
Resolvedtaavi
ResolvedAndrew
ResolvedAndrew
Resolveddcaro
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
Resolved Cmjohnson
Resolvedayounsi
Resolved aborrero
Resolved Cmjohnson
ResolvedJclark-ctr
ResolvedJclark-ctr
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
Resolveddcaro
Resolved aborrero
Declineddcaro
Resolveddcaro
OpenNone
OpenNone
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
Resolved Bstorm
Resolved aborrero
Resolved aborrero
Resolved aborrero
InvalidNone

Event Timeline

Change 656502 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add designate packages and manifests for openstack/train

https://gerrit.wikimedia.org/r/656502

Change 656502 merged by Andrew Bogott:
[operations/puppet@production] Add designate packages and manifests for openstack/train

https://gerrit.wikimedia.org/r/656502

Change 656533 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate nova_fixed_multi: update to catch up with upstream changes

https://gerrit.wikimedia.org/r/656533

Change 656533 merged by Andrew Bogott:
[operations/puppet@production] designate nova_fixed_multi: update to catch up with upstream changes

https://gerrit.wikimedia.org/r/656533

Change 657866 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Update eqiad1 designate to version 'train'

https://gerrit.wikimedia.org/r/657866

Change 657866 merged by Andrew Bogott:
[operations/puppet@production] Update eqiad1 designate to version 'train'

https://gerrit.wikimedia.org/r/657866

Change 658652 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add config files for openstack Train

https://gerrit.wikimedia.org/r/658652

Change 658653 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: forward our dmz hacks from version Stein to Train

https://gerrit.wikimedia.org/r/658653

Change 658654 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: forward our server name regex hack to version Train

https://gerrit.wikimedia.org/r/658654

Change 658655 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: Very minor config update for version Train

https://gerrit.wikimedia.org/r/658655

Change 658656 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifests for Neutron version 'Train'

https://gerrit.wikimedia.org/r/658656

Change 658657 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifests for Cinder version 'Train'

https://gerrit.wikimedia.org/r/658657

Change 658658 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifest for Glance version Train

https://gerrit.wikimedia.org/r/658658

Change 658659 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifests for Keystone version Train

https://gerrit.wikimedia.org/r/658659

Change 658660 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifests for Nova version Train

https://gerrit.wikimedia.org/r/658660

Change 658661 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifest for Barbican version Train

https://gerrit.wikimedia.org/r/658661

Change 658662 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add OpenStack client package manifests for version Train

https://gerrit.wikimedia.org/r/658662

Change 658652 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add config files for openstack Train

https://gerrit.wikimedia.org/r/658652

Change 658660 merged by Andrew Bogott:
[operations/puppet@production] Add manifests for Nova version Train

https://gerrit.wikimedia.org/r/658660

Change 658661 merged by Andrew Bogott:
[operations/puppet@production] Add manifest for Barbican version Train

https://gerrit.wikimedia.org/r/658661

Change 658662 merged by Andrew Bogott:
[operations/puppet@production] Add OpenStack client package manifests for version Train

https://gerrit.wikimedia.org/r/658662

Change 658654 merged by Andrew Bogott:
[operations/puppet@production] Nova: forward our server name regex hack to version Train

https://gerrit.wikimedia.org/r/658654

Change 658657 merged by Andrew Bogott:
[operations/puppet@production] Add manifests for Cinder version 'Train'

https://gerrit.wikimedia.org/r/658657

Change 658658 merged by Andrew Bogott:
[operations/puppet@production] Add manifest for Glance version Train

https://gerrit.wikimedia.org/r/658658

Change 658659 merged by Andrew Bogott:
[operations/puppet@production] Add manifests for Keystone version Train

https://gerrit.wikimedia.org/r/658659

Change 658655 merged by Andrew Bogott:
[operations/puppet@production] Nova: Very minor config update for version Train

https://gerrit.wikimedia.org/r/658655

Change 658656 merged by Andrew Bogott:
[operations/puppet@production] Add manifests for Neutron version 'Train'

https://gerrit.wikimedia.org/r/658656

Change 658999 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: update the l3 files with Neutron Train upstream

https://gerrit.wikimedia.org/r/658999

Change 658999 merged by Andrew Bogott:
[operations/puppet@production] Neutron: update the l3 files with Neutron Train upstream

https://gerrit.wikimedia.org/r/658999

Change 658653 merged by Andrew Bogott:
[operations/puppet@production] Neutron: forward our dmz hacks from version Stein to Train

https://gerrit.wikimedia.org/r/658653

Change 659311 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Update codfw1dev to OpenStack version 'train'

https://gerrit.wikimedia.org/r/659311

Change 659311 merged by Andrew Bogott:
[operations/puppet@production] Update codfw1dev to OpenStack version 'train'

https://gerrit.wikimedia.org/r/659311

Change 659929 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Update codfw1dev to OpenStack version 'train'

https://gerrit.wikimedia.org/r/659929

Change 659929 merged by Andrew Bogott:
[operations/puppet@production] Update codfw1dev to OpenStack version 'train'

https://gerrit.wikimedia.org/r/659929

Change 659942 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: Forward placement files and manifests from stein to train

https://gerrit.wikimedia.org/r/659942

Change 659942 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: Forward placement files and manifests from stein to train

https://gerrit.wikimedia.org/r/659942

Change 659964 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova hostname regex hack: import re

https://gerrit.wikimedia.org/r/659964

Change 659964 merged by Andrew Bogott:
[operations/puppet@production] Nova hostname regex hack: import re

https://gerrit.wikimedia.org/r/659964

Change 660084 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: Hack in a bugfix for the linux-bridge agent

https://gerrit.wikimedia.org/r/660084

Change 660084 merged by Andrew Bogott:
[operations/puppet@production] Neutron: Hack in a bugfix for the linux-bridge agent

https://gerrit.wikimedia.org/r/660084

Change 662978 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode for Train upgrade

https://gerrit.wikimedia.org/r/662978

Change 662980 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'stein' to 'train'

https://gerrit.wikimedia.org/r/662980

Change 662978 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode for Train upgrade

https://gerrit.wikimedia.org/r/662978

Change 662980 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'stein' to 'train'

https://gerrit.wikimedia.org/r/662980

Change 662997 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Train/buster: don't install python-pycadf

https://gerrit.wikimedia.org/r/662997

Change 662997 merged by Andrew Bogott:
[operations/puppet@production] Train/buster: don't install python-pycadf

https://gerrit.wikimedia.org/r/662997

Upgrade notes:

  • update IRC topic
  • downtime everything in icinga through 14:00CDT

    aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab*

    aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
  • merge puppet patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/662980
  • dump databases on cloudcontrol1003: nova_eqiad1, nova_api_eqiad1, nova_cell0_eqiad1, neutron, glance, keystone, cinder:
    1. mysqldump -u root nova_eqiad1 > /root/traindbbackups/nova_eqiad1.sql
    2. mysqldump -u root nova_api_eqiad1 > /root/traindbbackups/nova_api_eqiad1.sql
    3. mysqldump -u root nova_cell0_eqiad1 > /root/traindbbackups/nova_cell0_eqiad1.sql
    4. mysqldump -u root neutron > /root/traindbbackups/neutron.sql
    5. mysqldump -u root glance > /root/traindbbackups/glance.sql
    6. mysqldump -u root placement > /root/traindbbackups/placement.sql
    7. mysqldump -u root keystone > /root/traindbbackups/keystone.sql

Cloudcontrols:

All open database connections post-upgrade https://phabricator.wikimedia.org/P10999
Checking haproxy status echo "show stat" | socat /var/run/haproxy/haproxy.sock stdio | grep DOWN

Cloudcontrol1003:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • nova-manage api_db sync
  • nova-manage db sync
  • placement-manage db sync
  • glance-manage db_sync
  • keystone-manage db_sync
  • cinder-manage db sync
  • puppet agent -tv
  • nova-manage db online_data_migrations

Cloudcontrol1004:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api placement-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • puppet agent -tv

Cloudcontrol1005:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api placement-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • puppet agent -tv

cloudnets (one at a time please):

Begin with the standby node, as determined with:

$ neutron l3-agent-list-hosting-router cloudinstances2b-gw

Standby node:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-oslo.messaging python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • neutron-db-manage upgrade heads on cloudcontrol1003

Active node:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-oslo.messaging python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • restore Horizon

Break/meeting, whatever

Cloudvirts (start with one test host first, cloudvirt1039. Don't forget about cloudvirtwdqs ):

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-eventlet python3-oslo.messaging python3-taskflow python3-tooz python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-linuxbridge-agent restart
  • service libvirtd restart
  • service nova-compute restart
  • possibly mysql -u root placement; delete from resource_provider_traits;
  • update IRC topic
  • enable puppet on all cloud* hosts

    $ sudo cumin 'cloud*' "enable-puppet 'Upgrading to openstack Train - T261135 - ${USER}'"