Page MenuHomePhabricator

upgrade cloud-vps openstack to Openstack version 'Train'
Closed, ResolvedPublic

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -1
operations/puppetproduction+1 -1
operations/puppetproduction+2 -6
operations/puppetproduction+1 K -0
operations/puppetproduction+1 -0
operations/puppetproduction+313 -0
operations/puppetproduction+2 -9
operations/puppetproduction+2 -9
operations/puppetproduction+218 -190
operations/puppetproduction+57 -25
operations/puppetproduction+263 -0
operations/puppetproduction+9 -1
operations/puppetproduction+172 -0
operations/puppetproduction+63 -0
operations/puppetproduction+59 -0
operations/puppetproduction+169 -112
operations/puppetproduction+18 -0
operations/puppetproduction+40 -0
operations/puppetproduction+149 -0
operations/puppetproduction+12 K -0
operations/puppetproduction+4 -0
operations/puppetproduction+52 -71
operations/puppetproduction+1 K -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
ResolvedMajavah
ResolvedAndrew
ResolvedAndrew
Resolveddcaro
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedCmjohnson
Resolvedayounsi
Resolvedaborrero
ResolvedCmjohnson
ResolvedJclark-ctr
OpenCmjohnson
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
Opendcaro
Resolvedaborrero
Opendcaro
Resolveddcaro
Opendcaro
Opendcaro
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedBstorm
Resolvedaborrero
Resolvedaborrero
Resolvedaborrero
OpenNone

Event Timeline

Change 656502 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add designate packages and manifests for openstack/train

https://gerrit.wikimedia.org/r/656502

Change 656502 merged by Andrew Bogott:
[operations/puppet@production] Add designate packages and manifests for openstack/train

https://gerrit.wikimedia.org/r/656502

Change 656533 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate nova_fixed_multi: update to catch up with upstream changes

https://gerrit.wikimedia.org/r/656533

Change 656533 merged by Andrew Bogott:
[operations/puppet@production] designate nova_fixed_multi: update to catch up with upstream changes

https://gerrit.wikimedia.org/r/656533

Change 657866 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Update eqiad1 designate to version 'train'

https://gerrit.wikimedia.org/r/657866

Change 657866 merged by Andrew Bogott:
[operations/puppet@production] Update eqiad1 designate to version 'train'

https://gerrit.wikimedia.org/r/657866

Change 658652 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: add config files for openstack Train

https://gerrit.wikimedia.org/r/658652

Change 658653 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: forward our dmz hacks from version Stein to Train

https://gerrit.wikimedia.org/r/658653

Change 658654 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: forward our server name regex hack to version Train

https://gerrit.wikimedia.org/r/658654

Change 658655 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova: Very minor config update for version Train

https://gerrit.wikimedia.org/r/658655

Change 658656 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifests for Neutron version 'Train'

https://gerrit.wikimedia.org/r/658656

Change 658657 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifests for Cinder version 'Train'

https://gerrit.wikimedia.org/r/658657

Change 658658 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifest for Glance version Train

https://gerrit.wikimedia.org/r/658658

Change 658659 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifests for Keystone version Train

https://gerrit.wikimedia.org/r/658659

Change 658660 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifests for Nova version Train

https://gerrit.wikimedia.org/r/658660

Change 658661 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add manifest for Barbican version Train

https://gerrit.wikimedia.org/r/658661

Change 658662 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add OpenStack client package manifests for version Train

https://gerrit.wikimedia.org/r/658662

Change 658652 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: add config files for openstack Train

https://gerrit.wikimedia.org/r/658652

Change 658660 merged by Andrew Bogott:
[operations/puppet@production] Add manifests for Nova version Train

https://gerrit.wikimedia.org/r/658660

Change 658661 merged by Andrew Bogott:
[operations/puppet@production] Add manifest for Barbican version Train

https://gerrit.wikimedia.org/r/658661

Change 658662 merged by Andrew Bogott:
[operations/puppet@production] Add OpenStack client package manifests for version Train

https://gerrit.wikimedia.org/r/658662

Change 658654 merged by Andrew Bogott:
[operations/puppet@production] Nova: forward our server name regex hack to version Train

https://gerrit.wikimedia.org/r/658654

Change 658657 merged by Andrew Bogott:
[operations/puppet@production] Add manifests for Cinder version 'Train'

https://gerrit.wikimedia.org/r/658657

Change 658658 merged by Andrew Bogott:
[operations/puppet@production] Add manifest for Glance version Train

https://gerrit.wikimedia.org/r/658658

Change 658659 merged by Andrew Bogott:
[operations/puppet@production] Add manifests for Keystone version Train

https://gerrit.wikimedia.org/r/658659

Change 658655 merged by Andrew Bogott:
[operations/puppet@production] Nova: Very minor config update for version Train

https://gerrit.wikimedia.org/r/658655

Change 658656 merged by Andrew Bogott:
[operations/puppet@production] Add manifests for Neutron version 'Train'

https://gerrit.wikimedia.org/r/658656

Change 658999 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: update the l3 files with Neutron Train upstream

https://gerrit.wikimedia.org/r/658999

Change 658999 merged by Andrew Bogott:
[operations/puppet@production] Neutron: update the l3 files with Neutron Train upstream

https://gerrit.wikimedia.org/r/658999

Change 658653 merged by Andrew Bogott:
[operations/puppet@production] Neutron: forward our dmz hacks from version Stein to Train

https://gerrit.wikimedia.org/r/658653

Change 659311 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Update codfw1dev to OpenStack version 'train'

https://gerrit.wikimedia.org/r/659311

Change 659311 merged by Andrew Bogott:
[operations/puppet@production] Update codfw1dev to OpenStack version 'train'

https://gerrit.wikimedia.org/r/659311

Change 659929 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Update codfw1dev to OpenStack version 'train'

https://gerrit.wikimedia.org/r/659929

Change 659929 merged by Andrew Bogott:
[operations/puppet@production] Update codfw1dev to OpenStack version 'train'

https://gerrit.wikimedia.org/r/659929

Change 659942 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: Forward placement files and manifests from stein to train

https://gerrit.wikimedia.org/r/659942

Change 659942 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: Forward placement files and manifests from stein to train

https://gerrit.wikimedia.org/r/659942

Change 659964 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Nova hostname regex hack: import re

https://gerrit.wikimedia.org/r/659964

Change 659964 merged by Andrew Bogott:
[operations/puppet@production] Nova hostname regex hack: import re

https://gerrit.wikimedia.org/r/659964

Change 660084 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: Hack in a bugfix for the linux-bridge agent

https://gerrit.wikimedia.org/r/660084

Change 660084 merged by Andrew Bogott:
[operations/puppet@production] Neutron: Hack in a bugfix for the linux-bridge agent

https://gerrit.wikimedia.org/r/660084

Change 662978 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode for Train upgrade

https://gerrit.wikimedia.org/r/662978

Change 662980 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'stein' to 'train'

https://gerrit.wikimedia.org/r/662980

Change 662978 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode for Train upgrade

https://gerrit.wikimedia.org/r/662978

Change 662980 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps: move eqiad1 from openstack 'stein' to 'train'

https://gerrit.wikimedia.org/r/662980

Change 662997 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Train/buster: don't install python-pycadf

https://gerrit.wikimedia.org/r/662997

Change 662997 merged by Andrew Bogott:
[operations/puppet@production] Train/buster: don't install python-pycadf

https://gerrit.wikimedia.org/r/662997

Upgrade notes:

  • update IRC topic
  • downtime everything in icinga through 14:00CDT

    aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab*

    aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
  • merge puppet patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/662980
  • dump databases on cloudcontrol1003: nova_eqiad1, nova_api_eqiad1, nova_cell0_eqiad1, neutron, glance, keystone, cinder:
    1. mysqldump -u root nova_eqiad1 > /root/traindbbackups/nova_eqiad1.sql
    2. mysqldump -u root nova_api_eqiad1 > /root/traindbbackups/nova_api_eqiad1.sql
    3. mysqldump -u root nova_cell0_eqiad1 > /root/traindbbackups/nova_cell0_eqiad1.sql
    4. mysqldump -u root neutron > /root/traindbbackups/neutron.sql
    5. mysqldump -u root glance > /root/traindbbackups/glance.sql
    6. mysqldump -u root placement > /root/traindbbackups/placement.sql
    7. mysqldump -u root keystone > /root/traindbbackups/keystone.sql

Cloudcontrols:

All open database connections post-upgrade https://phabricator.wikimedia.org/P10999
Checking haproxy status echo "show stat" | socat /var/run/haproxy/haproxy.sock stdio | grep DOWN

Cloudcontrol1003:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • nova-manage api_db sync
  • nova-manage db sync
  • placement-manage db sync
  • glance-manage db_sync
  • keystone-manage db_sync
  • cinder-manage db sync
  • puppet agent -tv
  • nova-manage db online_data_migrations

Cloudcontrol1004:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api placement-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • puppet agent -tv

Cloudcontrol1005:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api placement-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • puppet agent -tv

cloudnets (one at a time please):

Begin with the standby node, as determined with:

$ neutron l3-agent-list-hosting-router cloudinstances2b-gw

Standby node:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-oslo.messaging python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • neutron-db-manage upgrade heads on cloudcontrol1003

Active node:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-oslo.messaging python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • restore Horizon

Break/meeting, whatever

Cloudvirts (start with one test host first, cloudvirt1039. Don't forget about cloudvirtwdqs ):

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-eventlet python3-oslo.messaging python3-taskflow python3-tooz python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-linuxbridge-agent restart
  • service libvirtd restart
  • service nova-compute restart
  • possibly mysql -u root placement; delete from resource_provider_traits;
  • update IRC topic
  • enable puppet on all cloud* hosts

    $ sudo cumin 'cloud*' "enable-puppet 'Upgrading to openstack Train - T261135 - ${USER}'"