Page MenuHomePhabricator
Create Task
Maniphest T237749

Upgrade wmcs OpenStack version to Ocata
Closed, ResolvedPublic
Actions

Description

Currently Horizon is running Ocata; all over services are running Newton.

Designate can be upgraded first; the other services are coupled and need to be upgraded together.

Details

SubjectRepoBranchLines +/-
Revert "Horizon: put in maintenance mode for the newton=>ocata upgrade"operations/puppetproduction+1 -1
Openstack: move eqiad1 to version 'ocata'operations/puppetproduction+3 -7
Horizon: put in maintenance mode for the newton=>ocata upgradeoperations/puppetproduction+1 -1
Openstack codfw1dev: everything is ocata nowoperations/puppetproduction+1 -16
neutron dhcp: reconcile dhcp_domain and dns_domainoperations/puppetproduction+11 -3
Neutron/ocata: update our l3 hacks using the latest ocata base filesoperations/puppetproduction+1 K -64
keystone ocata: include custom wsgi scriptoperations/puppetproduction+66 -1
wmfkeystonehooks: reconcile some differences with upstream keystoneoperations/puppetproduction+5 -4
Keystone wmtotp: remove unused argoperations/puppetproduction+1 -1
keystone wmtotp: update to conform with new ocata parent classoperations/puppetproduction+5 -3
Openstack Keystone: update the password_whitelist auth pluginoperations/puppetproduction+2 -3
Openstack Neutron: add missing files for Openstack Ocataoperations/puppetproduction+23 -0
Openstack keystone: add more missing manifests for Ocataoperations/puppetproduction+99 -0
eqiad1: move designate to 'Ocata'operations/puppetproduction+2 -2
codfw1dev: move designate to 'Ocata'operations/puppetproduction+2 -0
OpenStack Nova: Update config to work with Ocataoperations/puppetproduction+24 -18
openstack: bootstrap ocata puppet code for serversoperations/puppetproduction+426 -1
Openstack: Add config files for version 'ocata'operations/puppetproduction+8 K -12
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedAndrewT237749 Upgrade wmcs OpenStack version to Ocata
 ResolvedAndrewT210715 cloudvps: PDNS 3.x vs 4.x
 ResolvedaborreroT212302 CloudVPS: upgrade: jessie -> stretch & mitaka -> newton
 ResolvedaborreroT214299 cloudvps: neutron: upgrade jessie -> stretch
 ResolvedaborreroT214167 labtestneutron2001: reimage to stretch & rename to cloudnet2001-dev
 ResolvedPapaulT214181 codfw: rename/relabel labtestneutron2001 to cloudnet2001-dev
 Resolved• JHeddenT214297 cloudvps: neutron: investigate keepalived warning
 ResolvedaborreroT214303 labtestneutron2002: reimage to stretch & rename to cloudnet2002-dev
 ResolvedaborreroT214322 cloudnet2002-dev: ACPI error
 ResolvedPapaulT214370 labtestneutron2002: refresh/rename to cloudnet2002-dev
 ResolvedaborreroT215407 cloudvps: cloudcontrol support for mitaka/stretch
 ResolvedaborreroT215605 cloudvps: missing packages in stretch for cloudcontrol servers
 ResolvedaborreroT223832 puppet vs. stretch vs. keystone
 ResolvedaborreroT216497 CloudVPS: workaround archival of jessie-backports repo
 ResolvedAndrewT221769 Upgrade cloudservices1003/1004 to stretch/mitaka
 ResolvedaborreroT224354 backport pdns-server version 3.x to Stretch
 ResolvedaborreroT224877 prometheus-pdns-exporter: add stretch support
 ResolvedAndrewT221770 Upgrade cloucontrol1003/1004 to stretch/mitaka
 ResolvedaborreroT224345 stretch build of prometheus-openstack-exporter incompatible with our mitaka apt repo
 ResolvedaborreroT224424 cloudservices1003: gateway timeout error
 ResolvedAndrewT233258 designate: switch from designate-pool-manager to designate-producer/designate-worker
 ResolvedMarosteguiT233978 Drop 'designate_pool_manager' database from m5 and remove associated grants
 ResolvedaborreroT233665 Forward our neutron-l3-agent routing hacks to Openstack Newton
 ResolvedAndrewT234834 Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07
 DuplicateNoneT234830 CloudVPS: m5-master databases for openstack may require re-encoding
 ResolvedAndrewT234876 nova-conductor running out of mysql connections
 ResolvedAndrewT234836 CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress
Invalid• JHeddenT235070 Resolve keystone schema errors
 ResolvedaborreroT238338 Import packages for Openstack Ocata
 ResolvedAndrewT239160 nova: set up cell and host mappings
 ResolvedMarosteguiT239170 Create a new nova database on m5 named 'nova_cell0'
 ResolvedAndrewT239161 nova: start using the placement service

Event Timeline

Andrew created this task.Nov 8 2019, 5:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 8 2019, 5:15 PM
Andrew added a project: cloud-services-team (Kanban).Nov 8 2019, 5:15 PM
Andrew added a subtask: T210715: cloudvps: PDNS 3.x vs 4.x.Nov 8 2019, 8:58 PM
gerritbot added a comment.Nov 12 2019, 4:08 PM

Change 550502 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack: Add config files for version 'ocata'

https://gerrit.wikimedia.org/r/550502

gerritbot added a project: Patch-For-Review.Nov 12 2019, 4:08 PM

Change 550503 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Nova: Update config to work with Ocata

https://gerrit.wikimedia.org/r/550503

Andrew moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Nov 19 2019, 4:04 PM
gerritbot added a comment.Nov 20 2019, 12:06 PM

Change 552059 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: bootstrap ocata puppet code for servers

https://gerrit.wikimedia.org/r/552059

gerritbot added a comment.Nov 20 2019, 5:46 PM

Change 550502 merged by Andrew Bogott:
[operations/puppet@production] Openstack: Add config files for version 'ocata'

https://gerrit.wikimedia.org/r/550502

Andrew closed subtask T210715: cloudvps: PDNS 3.x vs 4.x as Resolved.Nov 21 2019, 3:00 PM
gerritbot added a comment.Nov 21 2019, 3:18 PM

Change 552059 merged by Andrew Bogott:
[operations/puppet@production] openstack: bootstrap ocata puppet code for servers

https://gerrit.wikimedia.org/r/552059

gerritbot added a comment.Nov 21 2019, 3:30 PM

Change 550503 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Nova: Update config to work with Ocata

https://gerrit.wikimedia.org/r/550503

gerritbot added a comment.Nov 21 2019, 3:36 PM

Change 552287 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] codfw1dev: move designate to 'Ocata'

https://gerrit.wikimedia.org/r/552287

gerritbot added a comment.Nov 21 2019, 3:39 PM

Change 552287 merged by Andrew Bogott:
[operations/puppet@production] codfw1dev: move designate to 'Ocata'

https://gerrit.wikimedia.org/r/552287

Maintenance_bot removed a project: Patch-For-Review.Nov 21 2019, 4:10 PM
Andrew closed subtask T238338: Import packages for Openstack Ocata as Resolved.Nov 21 2019, 6:58 PM
gerritbot added a comment.Nov 21 2019, 6:59 PM

Change 552322 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] codfw1dev: move designate to 'Ocata'

https://gerrit.wikimedia.org/r/552322

gerritbot added a project: Patch-For-Review.Nov 21 2019, 6:59 PM
Paladox subscribed.Nov 21 2019, 7:00 PM
gerritbot added a comment.Nov 21 2019, 7:04 PM

Change 552322 merged by Andrew Bogott:
[operations/puppet@production] eqiad1: move designate to 'Ocata'

https://gerrit.wikimedia.org/r/552322

Maintenance_bot removed a project: Patch-For-Review.Nov 21 2019, 7:10 PM
Andrew added a comment.Nov 21 2019, 8:04 PM

eqiad1 (cloudservices1003/1004) now running ocata Designate.

Andrew closed subtask T239161: nova: start using the placement service as Resolved.Nov 27 2019, 5:37 PM
gerritbot added a comment.Dec 2 2019, 3:04 AM

Change 553877 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack keystone: add more missing manifests for Ocata

https://gerrit.wikimedia.org/r/553877

gerritbot added a project: Patch-For-Review.Dec 2 2019, 3:05 AM

Change 553878 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Neutron: add missing files for Openstack Ocata

https://gerrit.wikimedia.org/r/553878

gerritbot added a comment.Dec 2 2019, 3:08 AM

Change 553877 merged by Andrew Bogott:
[operations/puppet@production] Openstack keystone: add more missing manifests for Ocata

https://gerrit.wikimedia.org/r/553877

gerritbot added a comment.Dec 2 2019, 3:08 AM

Change 553878 merged by Andrew Bogott:
[operations/puppet@production] Openstack Neutron: add missing files for Openstack Ocata

https://gerrit.wikimedia.org/r/553878

Maintenance_bot removed a project: Patch-For-Review.Dec 2 2019, 3:10 AM
Andrew closed subtask T239160: nova: set up cell and host mappings as Resolved.Dec 2 2019, 6:35 AM
gerritbot added a comment.Dec 3 2019, 1:01 AM

Change 554199 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack Keystone: update the password_whitelist auth plugin

https://gerrit.wikimedia.org/r/554199

gerritbot added a project: Patch-For-Review.Dec 3 2019, 1:01 AM

Change 554199 merged by Andrew Bogott:
[operations/puppet@production] Openstack Keystone: update the password_whitelist auth plugin

https://gerrit.wikimedia.org/r/554199

Maintenance_bot removed a project: Patch-For-Review.Dec 3 2019, 1:10 AM
gerritbot added a comment.Dec 3 2019, 5:34 AM

Change 554232 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone wmtotp: update to conform with new ocata parent class

https://gerrit.wikimedia.org/r/554232

gerritbot added a project: Patch-For-Review.Dec 3 2019, 5:34 AM

Change 554232 merged by Andrew Bogott:
[operations/puppet@production] keystone wmtotp: update to conform with new ocata parent class

https://gerrit.wikimedia.org/r/554232

gerritbot added a comment.Dec 3 2019, 5:43 AM

Change 554233 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone wmtotp: remove unused arg

https://gerrit.wikimedia.org/r/554233

gerritbot added a comment.Dec 3 2019, 5:49 AM

Change 554233 merged by Andrew Bogott:
[operations/puppet@production] Keystone wmtotp: remove unused arg

https://gerrit.wikimedia.org/r/554233

Maintenance_bot removed a project: Patch-For-Review.Dec 3 2019, 6:10 AM
gerritbot added a comment.Dec 3 2019, 6:56 AM

Change 554239 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfkeystonehooks: reconcile some differences with upstream keystone

https://gerrit.wikimedia.org/r/554239

gerritbot added a project: Patch-For-Review.Dec 3 2019, 6:56 AM
gerritbot added a comment.Dec 3 2019, 7:00 AM

Change 554239 merged by Andrew Bogott:
[operations/puppet@production] wmfkeystonehooks: reconcile some differences with upstream keystone

https://gerrit.wikimedia.org/r/554239

Maintenance_bot removed a project: Patch-For-Review.Dec 3 2019, 7:10 AM
gerritbot added a comment.Dec 3 2019, 8:18 AM

Change 554241 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone ocata: include custom wsgi script

https://gerrit.wikimedia.org/r/554241

gerritbot added a project: Patch-For-Review.Dec 3 2019, 8:18 AM
gerritbot added a comment.Dec 3 2019, 8:24 AM

Change 554241 merged by Andrew Bogott:
[operations/puppet@production] keystone ocata: include custom wsgi script

https://gerrit.wikimedia.org/r/554241

Maintenance_bot removed a project: Patch-For-Review.Dec 3 2019, 9:10 AM
gerritbot added a comment.Dec 4 2019, 8:09 AM

Change 554423 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron/ocata: update our l3 hacks using the latest ocata base files

https://gerrit.wikimedia.org/r/554423

gerritbot added a project: Patch-For-Review.Dec 4 2019, 8:09 AM
gerritbot added a comment.Dec 4 2019, 8:34 AM

Change 554423 merged by Andrew Bogott:
[operations/puppet@production] Neutron/ocata: update our l3 hacks using the latest ocata base files

https://gerrit.wikimedia.org/r/554423

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2019, 9:10 AM
gerritbot added a comment.Dec 4 2019, 9:22 AM

Change 554474 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron dhcp: reconcile dhcp_domain and dns_domain

https://gerrit.wikimedia.org/r/554474

gerritbot added a project: Patch-For-Review.Dec 4 2019, 9:22 AM
gerritbot added a comment.Dec 4 2019, 9:43 AM

Change 554474 merged by Andrew Bogott:
[operations/puppet@production] neutron dhcp: reconcile dhcp_domain and dns_domain

https://gerrit.wikimedia.org/r/554474

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2019, 10:10 AM
gerritbot added a comment.Dec 5 2019, 12:09 PM

Change 554829 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack: move eqiad1 to version 'ocata'

https://gerrit.wikimedia.org/r/554829

gerritbot added a project: Patch-For-Review.Dec 5 2019, 12:09 PM
Andrew added a comment.EditedDec 5 2019, 12:21 PM

codfw1-dev is running Ocata, and I've scheduled an upgrade window for eqiad1. Approximate steps are...

  • merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/556515/, apply on labweb hosts
  • on cloudcontrol1003:
    • nova-manage db online_migrations
  • merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/554829/
  • on cloudcontrol1003 and cloudcontrol1004:
    • apply puppet
    • apt-get install nova glance keystone nova-placement-api nova-api nova-conductor
    • apt-get update
    • Clean out keystone files in /etc/apache2/sites-available
    • apply puppet again
    • stop nova-conductor, nova-api, neutron-server
  • on cloudcontrol1003:
    • keystone-manage db_sync
    • nova-manage db sync
    • glance-manage db sync
    • nova-manage api_db sync
    • # nova-manage db online_data_migrations (?)
  • on cloudnet1003 and cloudnet1004:
    • apt-get install python-openstackclient python-openstacksdk neutron-common neutron-dhcp-agent neutron-l3-agent neutron-linuxbridge-agent neutron-metadata-agent python-neutron python-neutron-lib python-neutronclient -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
    • apt-get upgrade (needed to resolve neutron linuxbridge dependencies)
    • puppet agent -tv
  • on cloudcontrol1003:
    • neutron-db-manage upgrade head
  • on cloudnet1003 and cloudnet1004:
    • puppet agent -tv
    • restart all neutron services
  • on cloudvirts (via cumin):
    • puppet agent -tv
    • apt-get install nova-compute nova-common nova-compute-kvm python-nova python-novaclient neutron-common neutron-linuxbridge-agent python-neutron python-neutron-lib python-neutronclient -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
    • apt-get upgrade (needed to resolve neutron linuxbridge dependencies)
    • puppet agent -tv
    • service neutron-linuxbridge-agent restart
    • service nova-compute restart
  • merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/556516/, apply on labweb hosts
aborrero subscribed.Dec 5 2019, 12:45 PM
In T237749#5715108, @Andrew wrote:

codfw1-dev is running Ocata, and I've scheduled an upgrade window for eqiad1. Approximate steps are...

This sounds right to me. Thanks!

One question, a few new nova services were introduced but I don't see any reference to them in the steps. No steps are required? I'm thinking on the placement and cell stuff. Just double-checking.

gerritbot added a comment.Dec 5 2019, 12:49 PM

Change 554842 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack codfw1dev: everything is ocata now

https://gerrit.wikimedia.org/r/554842

Andrew added a comment.Dec 5 2019, 12:51 PM
In T237749#5715147, @aborrero wrote:
In T237749#5715108, @Andrew wrote:

codfw1-dev is running Ocata, and I've scheduled an upgrade window for eqiad1. Approximate steps are...

This sounds right to me. Thanks!

One question, a few new nova services were introduced but I don't see any reference to them in the steps. No steps are required? I'm thinking on the placement and cell stuff. Just double-checking.

you're right, I will add a step about the placement api. Nothing needed for the cells change, that's handled by the existing db sync steps.

gerritbot added a comment.Dec 6 2019, 3:10 AM

Change 554842 merged by Andrew Bogott:
[operations/puppet@production] Openstack codfw1dev: everything is ocata now

https://gerrit.wikimedia.org/r/554842

gerritbot added a comment.Dec 12 2019, 3:21 AM

Change 556515 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put in maintenance mode for the newton=>ocata upgrade

https://gerrit.wikimedia.org/r/556515

gerritbot added a comment.Dec 12 2019, 3:31 AM

Change 556516 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Revert "Horizon: put in maintenance mode for the newton=>ocata upgrade"

https://gerrit.wikimedia.org/r/556516

gerritbot added a comment.Dec 12 2019, 5:57 AM

Change 556515 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put in maintenance mode for the newton=>ocata upgrade

https://gerrit.wikimedia.org/r/556515

gerritbot added a comment.Dec 12 2019, 6:00 AM

Change 554829 merged by Andrew Bogott:
[operations/puppet@production] Openstack: move eqiad1 to version 'ocata'

https://gerrit.wikimedia.org/r/554829

gerritbot added a comment.Dec 12 2019, 8:29 AM

Change 556516 merged by Andrew Bogott:
[operations/puppet@production] Revert "Horizon: put in maintenance mode for the newton=>ocata upgrade"

https://gerrit.wikimedia.org/r/556516

Maintenance_bot removed a project: Patch-For-Review.Dec 12 2019, 9:10 AM
Andrew closed this task as Resolved.Dec 12 2019, 12:05 PM

Everything is upgraded and all major functions seem fine.

aborrero mentioned this in T240851: CloudVPS: stretch base images fails to boot.Dec 16 2019, 12:12 PM
aborrero mentioned this in T234830: CloudVPS: m5-master databases for openstack may require re-encoding.Dec 19 2019, 10:16 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits