Page MenuHomePhabricator
Create Task
Maniphest T261137

upgrade cloud-vps openstack to Openstack version 'Victoria'
Closed, ResolvedPublic
Actions

Description

The Designate hosts (cloudservices1003/1004) are already running Victoria. The current Horizon deploy is backwards-compatible with V. So that leaves the cloudcontrol, cloudnet, and cloudvirt nodes to upgrade.

  • update IRC topic
  • downtime everything in icinga through 14:00CDT

    aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab*

    aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
  • dump databases on cloudcontrol1003: nova_eqiad1, nova_api_eqiad1, nova_cell0_eqiad1, neutron, glance, keystone, cinder:
    1. mysqldump -u root nova_eqiad1 > /root/victoriadbbackups/nova_eqiad1.sql
    2. mysqldump -u root nova_api_eqiad1 > /root/victoriadbbackups/nova_api_eqiad1.sql
    3. mysqldump -u root nova_cell0_eqiad1 > /root/victoriadbbackups/nova_cell0_eqiad1.sql
    4. mysqldump -u root neutron > /root/victoriadbbackups/neutron.sql
    5. mysqldump -u root glance > /root/victoriadbbackups/glance.sql
    6. mysqldump -u root placement > /root/victoriadbbackups/placement.sql
    7. mysqldump -u root keystone > /root/victoriadbbackups/keystone.sql

Cloudcontrols:

All open database connections post-upgrade https://phabricator.wikimedia.org/P10999
Checking haproxy status echo "show stat" | socat /var/run/haproxy/haproxy.sock stdio | grep DOWN

Cloudcontrol1003:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • nova-manage api_db sync
  • nova-manage db sync
  • placement-manage db sync
  • glance-manage db_sync
  • keystone-manage db_sync
  • cinder-manage db online_data_migrations
  • cinder-manage db sync
  • puppet agent -tv
  • nova-manage db online_data_migrations
  • systemctl list-units --failed (should show nothing failed, or just keystone. If keystone is failed just reset with systemctl reset-failed)

Cloudcontrol1004:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api placement-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • puppet agent -tv
  • systemctl list-units --failed (should show nothing failed, or just keystone. If keystone is failed just reset with systemctl reset-failed)

Cloudcontrol1005:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api placement-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • puppet agent -tv
  • systemctl list-units --failed (should show nothing failed, or just keystone. If keystone is failed just reset with systemctl reset-failed)

cloudnets (one at a time please):

Begin with the standby node, as determined with:

$ neutron l3-agent-list-hosting-router cloudinstances2b-gw

Standby node:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-oslo.messaging python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • neutron-db-manage upgrade heads on cloudcontrol1003

Active node:

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-oslo.messaging python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv

Break Time

Cloudvirts (start with one test host first, cloudvirt1039. Don't forget about cloudvirtwdqs ):

  • puppet agent --enable && puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common nova-compute-kvm neutron-linuxbridge-agent python3-neutron python3-eventlet python3-oslo.messaging python3-taskflow python3-tooz python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-linuxbridge-agent restart
  • service libvirtd restart
  • service nova-compute restart
  • update IRC topic
  • enable puppet on all cloud* hosts

    $ sudo cumin 'cloud*' "enable-puppet 'Upgrading to openstack Train - T261135 - ${USER}'"

Things to check

  • Check 'openstack region list'. There should be exactly one region, eqiad1-r. If there is a second region named 'RegionOne' (this happened in codfw1dev), delete it; otherwise scripts that enumerate regions will be confused.
  • Clean up VMs in the admin-monitoring project that leaked during upgrade; delete them.
  • Create a new VM and confirm that DNS and ssh work properly
  • Logs will be extremely noisy about policy deprecations and value checks; this is expected because OpenStack is poised between two different policy systems; our existing policies are still (noisily) supported in U.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedAndrewT194334 [Epic] Modern Cloud VPS storage layer
Unknown Object (Task)
Unknown Object (Task)
 ResolvedtaaviT211393 openstack-browser and horizon: Security group and floating IP quota information being pulled from Nova instead of Neutron for eqiad1-r
 ResolvedAndrewT211777 Can't get quota information from Neutron API
 ResolvedAndrewT261137 upgrade cloud-vps openstack to Openstack version 'Victoria'
 ResolveddcaroT261136 upgrade cloud-vps openstack to Openstack version 'Ussuri'
 ResolvedAndrewT261138 Upgrade Horizon to latest OpenStack release
 ResolvedAndrewT261135 upgrade cloud-vps openstack to Openstack version 'Train'
 ResolvedAndrewT261134 upgrade cloud-vps openstack to Openstack version 'Stein'
 ResolvedAndrewT259399 Upgrade cloudvirts to Debian Buster
 ResolvedAndrewT261132 Move all cloud-vps VMs to Ceph
 ResolvedAndrewT253365 Complete build out of Ceph cluster and attach "diskless" cloudvirts
 Resolved• CmjohnsonT251619 (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org
 ResolvedayounsiT258764 Error: 'cloudsw1-c8-eqiad.mgmt.eqiad.wmnet' is not a valid parent for host 'cloudcephosd1004'
 Resolved• aborreroT266749 cloudcephosd1006: down, possible memory error
 Resolved• CmjohnsonT251627 (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet
 ResolvedJclark-ctrT251632 (Need By: 2020-06-12) rack/setup/install WMCS 10G switches
 ResolvedJclark-ctrT276922 cloudvirt1038: PCIe error
 ResolvedAndrewT258619 Decide on network setup for WMCS ceph cluster
 ResolvedAndrewT258826 Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan
 ResolvedAndrewT258968 Migrate existing ceph workload off of cloudcephosd100[1-3] and rebuild with new vlan config
 ResolvedAndrewT259057 Move cloudcephosd100[1-3] to new vlans
 ResolvedAndrewT259192 Investigate/prototype ceph backup options
 ResolveddcaroT260692 Ceph VM image backups
 Resolved• aborreroT260941 Practice restoring ceph backups
 DeclineddcaroT267195 CloudVPS: improve VM backups to make them redundant and discoverable
 ResolveddcaroT271094 [dumps] Review backup strategy
 OpenNoneT273720 [ceph][rbd] Periodically cleanup dangling snapshots
 OpenNoneT273723 [backups] Periodically cleanup non-handled backups
 ResolvedAndrewT261252 Create system for migrating single VMs to ceph
 ResolvedAndrewT264380 Rebuild tools-prometheus-04
 ResolvedAndrewT270305 Ceph performance tuning
 ResolvedAndrewT263677 Upgrade cloudvirt1019 and cloudvirt1020 to Buster
 Resolved• BstormT263679 Plan and manage the required downtime for the clouddb-services VMs on cloudvirt1019 and cloudvirt1020
 Resolved• aborreroT267967 set up bpo repos for OpenStack Stein and Debian Buster
 Resolved• aborreroT271517 codfw1dev: drop cloudgw setup briefly for testing openstack stein
 Resolved• aborreroT273200 buster-train-backports bpo is missing package nova-placement-api
 InvalidNoneT273598 openstack train: permission bug in neutron-linuxbridge-agent
 Resolved• aborreroT279470 Dependency failures for Designate version Victoria

Event Timeline

Andrew created this task.Aug 24 2020, 3:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 24 2020, 3:36 PM
Andrew triaged this task as Medium priority.Aug 24 2020, 3:36 PM
Andrew added a subtask: T261136: upgrade cloud-vps openstack to Openstack version 'Ussuri'.
Andrew added a parent task: T211777: Can't get quota information from Neutron API.
Andrew added a subtask: T261138: Upgrade Horizon to latest OpenStack release.Apr 5 2021, 3:12 AM
Andrew closed subtask T261138: Upgrade Horizon to latest OpenStack release as Resolved.Apr 6 2021, 6:51 PM
Andrew claimed this task.Apr 6 2021, 7:04 PM
gerritbot added a comment.Apr 6 2021, 7:06 PM

Change 677340 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] OpenStack: Add package and config for Designate/Victoria

https://gerrit.wikimedia.org/r/677340

gerritbot added a project: Patch-For-Review.Apr 6 2021, 7:06 PM

Change 677340 merged by Andrew Bogott:

[operations/puppet@production] OpenStack: Add package and config for Designate/Victoria

https://gerrit.wikimedia.org/r/677340

Maintenance_bot removed a project: Patch-For-Review.Apr 6 2021, 7:11 PM
gerritbot added a comment.Apr 6 2021, 7:40 PM

Change 677350 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Designate/Victoria: remove a hacked file

https://gerrit.wikimedia.org/r/677350

gerritbot added a project: Patch-For-Review.Apr 6 2021, 7:40 PM

Change 677351 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Add config and manifests for Openstack version Victoria

https://gerrit.wikimedia.org/r/677351

Andrew added a comment.Apr 6 2021, 8:00 PM

I've reviewed and applied release note changes for:

  • designate (none)
  • nova (one trivial)
  • glance (none)
  • keystone (??? no notes)
  • cinder (none)
gerritbot added a comment.Apr 6 2021, 8:40 PM

Change 677360 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Removed an uneeded setting from nova.conf

https://gerrit.wikimedia.org/r/677360

Andrew closed subtask T279470: Dependency failures for Designate version Victoria as Resolved.Apr 7 2021, 7:21 PM
gerritbot added a comment.Apr 7 2021, 7:34 PM

Change 677350 merged by Andrew Bogott:

[operations/puppet@production] Designate/Victoria: remove a hacked file

https://gerrit.wikimedia.org/r/677350

gerritbot added a comment.Apr 7 2021, 7:39 PM

Change 677647 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Neutron: forward our dmz hacks to Victoria

https://gerrit.wikimedia.org/r/677647

gerritbot added a comment.Apr 7 2021, 9:16 PM

Change 677658 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] codfw1dev designate -> OpenStack Victoria

https://gerrit.wikimedia.org/r/677658

gerritbot added a comment.Apr 7 2021, 9:31 PM

Change 677658 merged by Andrew Bogott:

[operations/puppet@production] codfw1dev designate -> OpenStack Victoria

https://gerrit.wikimedia.org/r/677658

gerritbot added a comment.Apr 7 2021, 9:54 PM

Change 677351 merged by Andrew Bogott:

[operations/puppet@production] Add config and manifests for Openstack version Victoria

https://gerrit.wikimedia.org/r/677351

gerritbot added a comment.Apr 7 2021, 9:54 PM

Change 677360 merged by Andrew Bogott:

[operations/puppet@production] Removed an unneeded setting from nova.conf

https://gerrit.wikimedia.org/r/677360

gerritbot added a comment.Apr 13 2021, 1:04 PM

Change 678833 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] eqiad1 designate -> Victoria

https://gerrit.wikimedia.org/r/678833

gerritbot added a comment.Apr 13 2021, 1:08 PM

Change 678833 merged by Andrew Bogott:

[operations/puppet@production] eqiad1 designate -> Victoria

https://gerrit.wikimedia.org/r/678833

Stashbot added a comment.Apr 13 2021, 1:11 PM

Mentioned in SAL (#wikimedia-cloud) [2021-04-13T13:11:06Z] <andrewbogott> upgrading eqiad1 designate to version Victoria, T261137

gerritbot added a comment.Apr 13 2021, 1:49 PM

Change 678840 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps codfw1dev -> OpenStack Victoria

https://gerrit.wikimedia.org/r/678840

Andrew updated the task description. (Show Details)Apr 13 2021, 2:16 PM
Andrew closed subtask T261136: upgrade cloud-vps openstack to Openstack version 'Ussuri' as Resolved.
gerritbot added a comment.Apr 13 2021, 2:18 PM

Change 677647 merged by Andrew Bogott:

[operations/puppet@production] Neutron: forward our dmz hacks to Victoria

https://gerrit.wikimedia.org/r/677647

gerritbot added a comment.Apr 13 2021, 2:20 PM

Change 678840 merged by Andrew Bogott:

[operations/puppet@production] cloud-vps codfw1dev -> OpenStack Victoria

https://gerrit.wikimedia.org/r/678840

Andrew updated the task description. (Show Details)Apr 13 2021, 2:23 PM
Stashbot added a comment.Apr 13 2021, 2:36 PM

Mentioned in SAL (#wikimedia-cloud) [2021-04-13T14:36:46Z] <andrewbogott> upgrading codfw1dev to version Victoria, T261137

Andrew updated the task description. (Show Details)Apr 13 2021, 3:24 PM
Andrew updated the task description. (Show Details)Apr 13 2021, 3:29 PM
Andrew updated the task description. (Show Details)Apr 23 2021, 2:43 PM
gerritbot added a comment.Apr 27 2021, 1:54 PM

Change 682948 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Horizon: put into maintenance mode for Victoria upgrade

https://gerrit.wikimedia.org/r/682948

gerritbot added a comment.Apr 27 2021, 1:54 PM

Change 682949 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps eqiad1 -> version Victoria

https://gerrit.wikimedia.org/r/682949

gerritbot added a comment.Apr 27 2021, 1:54 PM

Change 682950 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Revert "Horizon: put into maintenance mode for Victoria upgrade"

https://gerrit.wikimedia.org/r/682950

Andrew updated the task description. (Show Details)Apr 27 2021, 1:55 PM

https://etherpad.wikimedia.org/p/eqiad1_victoria_upgrade

gerritbot added a comment.Apr 27 2021, 1:58 PM

Change 682948 merged by Andrew Bogott:

[operations/puppet@production] Horizon: put into maintenance mode for Victoria upgrade

https://gerrit.wikimedia.org/r/682948

gerritbot added a comment.Apr 27 2021, 2:11 PM

Change 682949 merged by Andrew Bogott:

[operations/puppet@production] cloud-vps eqiad1 -> version Victoria

https://gerrit.wikimedia.org/r/682949

gerritbot added a comment.Apr 27 2021, 4:10 PM

Change 682950 merged by David Caro:

[operations/puppet@production] Revert "Horizon: put into maintenance mode for Victoria upgrade"

https://gerrit.wikimedia.org/r/682950

gerritbot added a comment.Apr 27 2021, 5:11 PM

Change 683002 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps: set cloudvirt nodes to OpenStack U

https://gerrit.wikimedia.org/r/683002

gerritbot added a comment.Apr 27 2021, 5:25 PM

Change 683002 merged by Andrew Bogott:

[operations/puppet@production] cloud-vps: set cloudvirt nodes to OpenStack U

https://gerrit.wikimedia.org/r/683002

gerritbot added a comment.Apr 28 2021, 1:09 PM

Change 683274 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs.openstack: unpin cloudvirt1039

https://gerrit.wikimedia.org/r/683274

gerritbot added a comment.Apr 28 2021, 1:11 PM

Change 683274 merged by David Caro:

[operations/puppet@production] wmcs.openstack: unpin cloudvirt1039

https://gerrit.wikimedia.org/r/683274

gerritbot added a comment.Apr 28 2021, 1:28 PM

Change 683278 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs.openstack: unpin cloudvirts to continue upgrade to victoria

https://gerrit.wikimedia.org/r/683278

gerritbot added a comment.Apr 28 2021, 1:31 PM

Change 683278 merged by David Caro:

[operations/puppet@production] wmcs.openstack: unpin cloudvirts to continue upgrade to victoria

https://gerrit.wikimedia.org/r/683278

Andrew closed this task as Resolved.Apr 28 2021, 9:23 PM
  • update IRC topic
  • downtime everything in icinga through 14:00CDT

    aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 lab* aborrero@cumin1001:~ $ sudo cookbook sre.hosts.downtime -r "upgrading openstack" --min 120 cloud*
  • dump databases on cloudcontrol1003: nova_eqiad1, nova_api_eqiad1, nova_cell0_eqiad1, neutron, glance, keystone, cinder:
    1. mysqldump -u root nova_eqiad1 > /root/victoriadbbackups/nova_eqiad1.sql
    2. mysqldump -u root nova_api_eqiad1 > /root/victoriadbbackups/nova_api_eqiad1.sql
    3. mysqldump -u root nova_cell0_eqiad1 > /root/victoriadbbackups/nova_cell0_eqiad1.sql
    4. mysqldump -u root neutron > /root/victoriadbbackups/neutron.sql
    5. mysqldump -u root glance > /root/victoriadbbackups/glance.sql
    6. mysqldump -u root placement > /root/victoriadbbackups/placement.sql
    7. mysqldump -u root keystone > /root/victoriadbbackups/keystone.sql
    8. mysqldump -u root cinder > /root/victoriadbbackups/keystone.sql

Cloudcontrols:

All open database connections post-upgrade https://phabricator.wikimedia.org/P10999
Checking haproxy status echo "show stat" | socat /var/run/haproxy/haproxy.sock stdio | grep DOWN

Cloudcontrol1003:

  • puppet agent --enable && puppet agent -tv
  • apt update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • nova-manage api_db sync
  • nova-manage db sync
  • placement-manage db sync
  • glance-manage db_sync

2021-04-27 14:31:31.766 45009 WARNING oslo_config.cfg [-] Deprecated: Option "sql_idle_timeout" from group "DEFAULT" is deprecated. Use option "connection_recycle_time" from group "database".
2021-04-27 14:31:31.768 45009 WARNING oslo_db.sqlalchemy.engines [-] URL mysql://glance:***@openstack.eqiad1.wikimediacloud.org/glance does not contain a '+drivername' portion, and will make use of a default driver. A full dbname+drivername:// protocol is recommended. For MySQL, it is strongly recommended that mysql+pymysql:// be specified for maximum service compatibility

  • keystone-manage db_sync
  • cinder-manage db online_data_migrations
  • cinder-manage db sync
  • puppet agent -tv
  • puppet agent -tv # again to check config convergence, no changes should happen
  • nova-manage db online_data_migrations
  • systemctl list-units --failed (should show nothing failed, or just keystone. If keystone is failed just reset with systemctl reset-failed)

Cloudcontrol1004:

  • puppet agent --enable && puppet agent -tv
  • apt update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api placement-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • systemctl list-units --failed (should show nothing failed, or just keystone. If keystone is failed just reset with systemctl reset-failed)

Cloudcontrol1005:

  • puppet agent --enable && puppet agent -tv
  • apt update
  • systemctl unmask keystone
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-common keystone nova-api nova-conductor nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 placement-api cinder-volume cinder-scheduler cinder-api placement-api python3-oslo.messaging python3-tooz -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • systemctl mask keystone
  • puppet agent -tv
  • puppet agent -tv # again to check config convergence, no changes should happen
  • systemctl list-units --failed (should show nothing failed, or just keystone. If keystone is failed just reset with systemctl reset-failed)

cloudnets (one at a time please):

Begin with the standby node, as determined with:

cloudcontrol1003$ source novaenv.sh && neutron l3-agent-list-hosting-router cloudinstances2b-gw
ex:
+--------------------------------------+--------------+----------------+-------+----------+

idhostadmin_state_upaliveha_state

+--------------------------------------+--------------+----------------+-------+----------+

4be214c8-76ef-40f8-9d5d-4c344d213311cloudnet1003True:-)active
970df1d1-505d-47a4-8d35-1b13c0dfe098cloudnet1004True:-)standby

+--------------------------------------+--------------+----------------+-------+----------+

Standby node:

  • puppet agent --enable && puppet agent -tv
  • apt update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-oslo.messaging python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • neutron-db-manage upgrade heads on cloudcontrol1003

Active node:

  • puppet agent --enable && puppet agent -tv
  • apt update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-oslo.messaging python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv

Break Time

Cloudvirts (start with one test host first, cloudvirt1039. Don't forget about cloudvirtwdqs ):

  • puppet agent --enable && run-puppet-agent # expected that it might fail
  • apt update
  • DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common nova-compute-kvm neutron-linuxbridge-agent python3-neutron python3-eventlet python3-oslo.messaging python3-taskflow python3-tooz python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • run-puppet-agent # expected that it might fail
  • systemctl restart neutron-linuxbridge-agent
  • for some reason libvirtd-tls wants libvirtd to be stopped before starting
  • systemctl stop libvirtd && systemctl start libvirtd-tls.socket && systemctl start libvirtd
  • run-puppet-agent
  • systemctl restart nova-compute
  • cloudvirt-wdqs1001
  • cloudvirt-wdqs1002
  • cloudvirt-wdqs1003
  • cloudvirt1012
  • cloudvirt1013
  • cloudvirt1014
  • cloudvirt1016
  • cloudvirt1017
  • cloudvirt1018
  • cloudvirt1019 (toolsdb)
  • cloudvirt1020 (toolsdb)
  • cloudvirt1021
  • cloudvirt1022
  • cloudvirt1023
  • cloudvirt1024
  • cloudvirt1025
  • cloudvirt1026
  • cloudvirt1027
  • cloudvirt1028
  • cloudvirt1029
  • cloudvirt1030
  • cloudvirt1031
  • cloudvirt1032
  • cloudvirt1033
  • cloudvirt1034
  • cloudvirt1035
  • cloudvirt1036
  • cloudvirt1037
  • cloudvirt1038
  • cloudvirt1039
  • update IRC topic
  • enable puppet on all cloud* hosts

    $ sudo cumin 'cloud*' "enable-puppet 'Upgrading to openstack Victoria - T261137 - ${USER}'"

Things to check

  • Check 'openstack region list'. There should be exactly one region, eqiad1-r. If there is a second region named 'RegionOne' (this happened in codfw1dev), delete it; otherwise scripts that enumerate regions will be confused.
  • Clean up VMs in the admin-monitoring project that leaked during upgrade; delete them.
  • Create a new VM and confirm that DNS and ssh work properly
  • Logs will be extremely noisy about policy deprecations and value checks; this is expected because OpenStack is poised between two different policy systems; our existing policies are still (noisily) supported in U.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits