Page MenuHomePhabricator
Create Task
Maniphest T223905

HA for openstack services
Closed, ResolvedPublic
Actions

Description

Right now we have cold spares for most of our openstack hosts. It should be fairly straightforward to set up active/active support for most or all of these. The (incomplete) upstream docs for this are here:

https://docs.openstack.org/ha-guide/

  • rabbitmq
  • active/active backend services (e.g. conductor).
  • nova
  • neuron
  • keystone
  • [] glance (requires shared storage)
  • HAproxy layer7 load balancing
  • Nova API and Metadata
  • Neutron server
  • Glance API and Registry
  • Keystone API
  • Run some failover tests
  • Adjust monitoring/paging to reflect that these are now redundant

Details

SubjectRepoBranchLines +/-
keystone: make the api service active on both controller nodesoperations/puppetproduction+1 -8
keystone: add firewall rules to acess from the nova controlleroperations/puppetproduction+3 -1
neutron: make the neutron api server ('neutron-server') active/activeoperations/puppetproduction+1 -6
designate: make designate nodes active/activeoperations/puppetproduction+2 -2
nova: make all services active/activeoperations/puppetproduction+3 -18
clouddb2001-dev: adjust firewall rules to allow access to both cloudcontrols.operations/puppetproduction+3 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.May 20 2019, 2:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 20 2019, 2:10 PM
Andrew added a parent task: T221770: Upgrade cloucontrol1003/1004 to stretch/mitaka.May 20 2019, 2:10 PM
Andrew closed subtask T223906: Active/active rabbitMQ servers on wmcs controller nodes as Resolved.May 22 2019, 3:06 AM
Andrew updated the task description. (Show Details)May 22 2019, 3:22 AM
Andrew added a comment.May 22 2019, 3:28 AM

The docs say:

"To make nova-conductor highly available and fault tolerant, just launch more instances of the nova-conductor process, either on the same server or across multiple servers."

so that's good!

I also found "If you running two nova-scheduler processes they race each other, they don’t find out about each others choices until the DB gets updated by the nova-compute resource tracker." that's a little bit bad, but only really bad in terms of overrunning quotas as I understand it.

Andrew added a comment.EditedMay 22 2019, 3:30 AM

https://docs.openstack.org/designate/latest/admin/ha.html for designate HA. Looks pretty trivial, we can just run multiples of each service except for the api which needs a load balancer.

Krenair subscribed.May 22 2019, 7:37 AM
gerritbot added a comment.May 22 2019, 2:50 PM

Change 511901 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] clouddb2001-dev: adjust firewall rules to allow access to both cloudcontrols.

https://gerrit.wikimedia.org/r/511901

gerritbot added a project: Patch-For-Review.May 22 2019, 2:50 PM
gerritbot added a comment.May 22 2019, 2:53 PM

Change 511901 merged by Andrew Bogott:
[operations/puppet@production] clouddb2001-dev: adjust firewall rules to allow access to both cloudcontrols.

https://gerrit.wikimedia.org/r/511901

Maintenance_bot removed a project: Patch-For-Review.May 22 2019, 4:04 PM
gerritbot added a comment.May 22 2019, 7:03 PM

Change 511950 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova: make nova-conductor and nova-scheduler active/active

https://gerrit.wikimedia.org/r/511950

gerritbot added a project: Patch-For-Review.May 22 2019, 7:03 PM
Andrew updated the task description. (Show Details)May 23 2019, 3:22 AM
Andrew updated the task description. (Show Details)
gerritbot added a comment.May 23 2019, 3:44 AM

Change 512081 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: make designate nodes active/active

https://gerrit.wikimedia.org/r/512081

Paladox subscribed.May 23 2019, 3:53 AM
gerritbot added a comment.May 23 2019, 5:03 PM

Change 512192 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] neutron: make the neutron api server ('neutron-server') active/active

https://gerrit.wikimedia.org/r/512192

gerritbot added a comment.May 23 2019, 7:55 PM

Change 511950 merged by Andrew Bogott:
[operations/puppet@production] nova: make all services active/active

https://gerrit.wikimedia.org/r/511950

gerritbot added a comment.May 23 2019, 8:24 PM

Change 512081 merged by Andrew Bogott:
[operations/puppet@production] designate: make designate nodes active/active

https://gerrit.wikimedia.org/r/512081

gerritbot added a comment.May 23 2019, 8:47 PM

Change 512192 merged by Andrew Bogott:
[operations/puppet@production] neutron: make the neutron api server ('neutron-server') active/active

https://gerrit.wikimedia.org/r/512192

Andrew updated the task description. (Show Details)May 24 2019, 2:50 PM

The rabbit based services are now active/active and seem to be working OK. I'm setting the API issues aside because we're blocked on naming (and, certain naming decisions may mean that I have to build my own load balancer)

Andrew removed a parent task: T221770: Upgrade cloucontrol1003/1004 to stretch/mitaka.May 24 2019, 4:33 PM
gerritbot added a comment.May 28 2019, 12:16 AM

Change 512788 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: add firewall rules to acess from the nova controller

https://gerrit.wikimedia.org/r/512788

gerritbot added a comment.May 28 2019, 12:18 AM

Change 512788 merged by Andrew Bogott:
[operations/puppet@production] keystone: add firewall rules to acess from the nova controller

https://gerrit.wikimedia.org/r/512788

gerritbot added a comment.May 28 2019, 12:21 AM

Change 512789 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: make the api service active on both controller nodes

https://gerrit.wikimedia.org/r/512789

gerritbot added a comment.May 28 2019, 3:39 PM

Change 512789 merged by Andrew Bogott:
[operations/puppet@production] keystone: make the api service active on both controller nodes

https://gerrit.wikimedia.org/r/512789

Maintenance_bot removed a project: Patch-For-Review.May 28 2019, 3:41 PM
• JHedden closed subtask T224740: Puppet broken on cloudcontrol2001-dev.wikimedia.org and cloudcontrol2003-dev.wikimedia.org "Invalid value "True"" as Resolved.May 31 2019, 7:13 PM
Andrew mentioned this in T224743: cloudservices: unify puppet roles/profiles.Jun 4 2019, 2:59 PM
Andrew updated the task description. (Show Details)Jun 17 2019, 12:27 PM
Andrew updated the task description. (Show Details)
Andrew removed Andrew as the assignee of this task.Jun 27 2019, 5:14 PM
bd808 assigned this task to • JHedden.Sep 11 2019, 3:19 PM
bd808 triaged this task as Medium priority.
bd808 added a project: Goal.
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
• JHedden updated the task description. (Show Details)Sep 11 2019, 3:46 PM
• JHedden closed subtask T223907: Set up HA endpoints for keystone, glance, nova, designate apis as Resolved.Oct 17 2019, 3:43 PM
• JHedden updated the task description. (Show Details)
Andrew added a comment.Oct 17 2019, 4:46 PM

nice work!

• JHedden updated the task description. (Show Details)Oct 23 2019, 3:59 PM
• JHedden updated the task description. (Show Details)
• JHedden closed this task as Resolved.Oct 23 2019, 4:02 PM

Grafana dashboard for HAproxy metrics: https://grafana.wikimedia.org/d/tanisM2Zz/wmcs-openstack-eqiad1-api-stats

dcaro subscribed.Dec 8 2020, 4:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits