Page MenuHomePhabricator

Set up HA endpoints for keystone, glance, nova, designate apis
Closed, ResolvedPublic

Description

The openstack REST endpoints are stateless, so it should be simple to set up load-balanced pairs. As per T223902, we'll name these endpoints openstack.eqiad1.wikimediacloud.org. HAproxy should probably run on cloudcontrol1003/1004.

Details

Related Gerrit Patches:
operations/mediawiki-config : masterwikitech: Update hostnames for OpenStack endpoints
operations/puppet : productionopenstack: update eqiad1 clients to wikimediacloud auth url
operations/puppet : productionopenstack: update codfw1 keystone clients for wikimediacloud domain
operations/dns : masterwikimediacloud.org: add initial zone file
operations/puppet : productionopenstack: haproxy fix stats url for exporter
operations/puppet : productionopenstack: move haproxy exporter to controller role
operations/puppet : productionopenstack: add designate to eqiad1 haproxy
operations/puppet : productionopenstack: update designate config for newton release
operations/puppet : productionopenstack: add desginate API ferm rules to haproxy
operations/puppet : productionopenstack: add designate ferm rules to haproxy
operations/puppet : productionopenstack: add designate API ferm rules to haproxy
operations/puppet : productionopenstack: add designate to haproxy
operations/puppet : productionopenstack: update haproxy ferm rules for remote services
operations/puppet : productionopenldap: remove some extra newlines in labs-acls
operations/puppet : productioncloud ldap: support acls for n keystone hosts (in this case, n=2)
operations/puppet : productionopenstack: Use WMF style apache logs
operations/puppet : productionopenstack: Update backend API ports for haproxy
operations/puppet : productionopenstack: configure haproxy for eqiad1 APIs
operations/puppet : productionopenstack: configure eqiad1 keystone for apache wsgi
operations/puppet : productionopenstack: add newton keystone apache config
operations/puppet : productionopenstack: add codfw1dev keystone APIs to haproxy
operations/puppet : productionopenstack: configure apache wsgi for keystone api
operations/puppet : productionopenstack: add haproxy health check path support
operations/puppet : productionopenstack: add haproxy health check path support
operations/puppet : productionopenstack: Add codfw1dev glance API to haproxy
operations/puppet : productionopenstack: Add codfw1dev neutron server to haproxy
operations/puppet : productionopenstack: Add codfw1dev nova API and metadata to haproxy
operations/puppet : productionopenstack: add glance image sync to codfw
operations/puppet : productionopenstack: initial haproxy profile

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 533552 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: Add codfw1dev neutron server to haproxy

https://gerrit.wikimedia.org/r/533552

Change 533552 merged by Jhedden:
[operations/puppet@production] openstack: Add codfw1dev neutron server to haproxy

https://gerrit.wikimedia.org/r/533552

Change 534680 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: Add codfw1dev glance API to haproxy

https://gerrit.wikimedia.org/r/534680

Change 534680 merged by Jhedden:
[operations/puppet@production] openstack: Add codfw1dev glance API to haproxy

https://gerrit.wikimedia.org/r/534680

Change 534832 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: add haproxy health check path support

https://gerrit.wikimedia.org/r/534832

Change 534832 merged by Jhedden:
[operations/puppet@production] openstack: add haproxy health check path support

https://gerrit.wikimedia.org/r/534832

Change 534839 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: add haproxy health check path support

https://gerrit.wikimedia.org/r/534839

Change 534839 merged by Jhedden:
[operations/puppet@production] openstack: add haproxy health check path support

https://gerrit.wikimedia.org/r/534839

Change 536664 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: configure apache wsgi for keystone api

https://gerrit.wikimedia.org/r/536664

Change 536664 merged by Jhedden:
[operations/puppet@production] openstack: configure apache wsgi for keystone api

https://gerrit.wikimedia.org/r/536664

Change 537738 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: add codfw1dev keystone APIs to haproxy

https://gerrit.wikimedia.org/r/537738

Change 537738 merged by Jhedden:
[operations/puppet@production] openstack: add codfw1dev keystone APIs to haproxy

https://gerrit.wikimedia.org/r/537738

Change 538267 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: configure eqiad1 keystone for apache wsgi

https://gerrit.wikimedia.org/r/538267

Change 538269 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: add newton keystone apache config

https://gerrit.wikimedia.org/r/538269

Change 538269 merged by Jhedden:
[operations/puppet@production] openstack: add newton keystone apache config

https://gerrit.wikimedia.org/r/538269

Mentioned in SAL (#wikimedia-cloud) [2019-09-23T19:07:59Z] <jeh> reconfigure keystone in eqiad1 with apache and mod_wsgi T223907

Change 538267 merged by Jhedden:
[operations/puppet@production] openstack: configure eqiad1 keystone for apache wsgi

https://gerrit.wikimedia.org/r/538267

Change 538901 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: configure haproxy for eqiad1 APIs

https://gerrit.wikimedia.org/r/538901

Change 538901 merged by Jhedden:
[operations/puppet@production] openstack: configure haproxy for eqiad1 APIs

https://gerrit.wikimedia.org/r/538901

Change 538971 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: Update backend API ports for haproxy

https://gerrit.wikimedia.org/r/538971

Change 538971 merged by Jhedden:
[operations/puppet@production] openstack: Update backend API ports for haproxy

https://gerrit.wikimedia.org/r/538971

Mentioned in SAL (#wikimedia-cloud) [2019-09-24T22:32:02Z] <jeh> disable puppet agent and keystone on cloudcontrol1004 (standby) T223907

Mentioned in SAL (#wikimedia-cloud) [2019-09-24T22:32:25Z] <jeh> add icingia downtime for puppet and systemd state on cloudcontrol1004 T223907

Change 539212 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: Use WMF style apache logs

https://gerrit.wikimedia.org/r/539212

Change 539212 merged by Jhedden:
[operations/puppet@production] openstack: Use WMF style apache logs

https://gerrit.wikimedia.org/r/539212

Change 539354 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud ldap: support acls for n keystone hosts (in this case, n=2)

https://gerrit.wikimedia.org/r/539354

Change 539354 merged by Andrew Bogott:
[operations/puppet@production] cloud ldap: support acls for n keystone hosts (in this case, n=2)

https://gerrit.wikimedia.org/r/539354

Change 539365 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] openldap: remove some extra newlines in labs-acls

https://gerrit.wikimedia.org/r/539365

Change 539365 merged by Andrew Bogott:
[operations/puppet@production] openldap: remove some extra newlines in labs-acls

https://gerrit.wikimedia.org/r/539365

Change 539398 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: update haproxy ferm rules for remote services

https://gerrit.wikimedia.org/r/539398

Change 539398 merged by Jhedden:
[operations/puppet@production] openstack: update haproxy ferm rules for remote services

https://gerrit.wikimedia.org/r/539398

Change 539406 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: add designate to haproxy

https://gerrit.wikimedia.org/r/539406

Change 539406 merged by Jhedden:
[operations/puppet@production] openstack: add designate to haproxy

https://gerrit.wikimedia.org/r/539406

Change 539421 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: add designate API ferm rules to haproxy

https://gerrit.wikimedia.org/r/539421

Change 539533 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: add designate ferm rules to haproxy

https://gerrit.wikimedia.org/r/539533

Change 539421 abandoned by Jhedden:
openstack: add designate API ferm rules to haproxy

Reason:
abandoned for a solution that doesn't refactor

https://gerrit.wikimedia.org/r/539421

Change 539533 abandoned by Jhedden:
openstack: add designate ferm rules to haproxy

https://gerrit.wikimedia.org/r/539533

Change 539542 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: add desginate API ferm rules to haproxy

https://gerrit.wikimedia.org/r/539542

Change 539542 merged by Jhedden:
[operations/puppet@production] openstack: add desginate API ferm rules to haproxy

https://gerrit.wikimedia.org/r/539542

Change 539594 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: update designate config for newton release

https://gerrit.wikimedia.org/r/539594

Change 539594 merged by Jhedden:
[operations/puppet@production] openstack: update designate config for newton release

https://gerrit.wikimedia.org/r/539594

Change 539889 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: add designate to eqiad1 haproxy

https://gerrit.wikimedia.org/r/539889

Change 539889 merged by Jhedden:
[operations/puppet@production] openstack: add designate to eqiad1 haproxy

https://gerrit.wikimedia.org/r/539889

Change 539936 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: move haproxy exporter to controller role

https://gerrit.wikimedia.org/r/539936

Change 539936 merged by Jhedden:
[operations/puppet@production] openstack: move haproxy exporter to controller role

https://gerrit.wikimedia.org/r/539936

Change 539959 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: haproxy fix stats url for exporter

https://gerrit.wikimedia.org/r/539959

Change 539959 merged by Jhedden:
[operations/puppet@production] openstack: haproxy fix stats url for exporter

https://gerrit.wikimedia.org/r/539959

Change 540148 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/dns@master] wikimediacloud.org: add initial zone file

https://gerrit.wikimedia.org/r/540148

Change 540148 merged by Jhedden:
[operations/dns@master] wikimediacloud.org: add initial zone file

https://gerrit.wikimedia.org/r/540148

Change 540685 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: use wikimediacloud for keystone API requests

https://gerrit.wikimedia.org/r/540685

Change 540685 merged by Jhedden:
[operations/puppet@production] openstack: update codfw1 keystone clients for wikimediacloud domain

https://gerrit.wikimedia.org/r/540685

Change 542452 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: update eqiad1 clients to wikimediacloud auth url

https://gerrit.wikimedia.org/r/542452

Change 542506 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/mediawiki-config@master] wikitech: Update hostnames for OpenStack endpoints

https://gerrit.wikimedia.org/r/542506

Mentioned in SAL (#wikimedia-cloud) [2019-10-17T14:54:58Z] <jeh> update eqiad1's hiera keystone_host to new wikimediacloud.org domain T223907

Mentioned in SAL (#wikimedia-cloud) [2019-10-17T14:56:25Z] <jeh> added icingia downtime for cloudcontrol100[34] and checker.tools.wmflabs.org for service restarts T223907

Change 542452 merged by Jhedden:
[operations/puppet@production] openstack: update eqiad1 clients to wikimediacloud auth url

https://gerrit.wikimedia.org/r/542452

Updating the OpenStack endpoints in eqiad1 to the new domain.

current configuration
+----------------------------------+----------+--------------+--------------+---------+-----------+------------------------------------------------------------------------+
| ID                               | Region   | Service Name | Service Type | Enabled | Interface | URL                                                                    |
+----------------------------------+----------+--------------+--------------+---------+-----------+------------------------------------------------------------------------+
| 0ed1b6c32ec9433f842cd31ba1f11a48 | eqiad1-r | glance       | image        | True    | public    | http://cloudcontrol1003.wikimedia.org:9292                             |
| 3d1ce837946444e3b90c11138a448fa0 | eqiad1-r | proxy        | proxy        | True    | admin     | http://proxy-eqiad1.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| 3ed2c8aba1e648e2be01423a00b3b6b2 | eqiad1-r | glance       | image        | True    | admin     | http://cloudcontrol1003.wikimedia.org:9292                             |
| 43d19fccac134e868c308673c7b2de7e | eqiad1-r | keystone     | identity     | True    | internal  | http://cloudcontrol1003.wikimedia.org:5000/v3                          |
| 465f331e03de4bfcbef1f8a7dbb4de15 | eqiad1-r | designate    | dns          | True    | public    | http://cloudservices1003.wikimedia.org:9001                            |
| 4ea7fdb8da904b0290465cea0a74d545 | eqiad1-r | neutron      | network      | True    | admin     | http://cloudcontrol1003.wikimedia.org:9696                             |
| 53e93fccd882487396f8ae6208d09f3d | eqiad1-r | neutron      | network      | True    | public    | http://cloudcontrol1003.wikimedia.org:9696                             |
| 6e808288d5814ff390b1638d1d58ab13 | eqiad1-r | designate    | dns          | True    | internal  | http://cloudservices1003.wikimedia.org:9001                            |
| 7ea7c2c9fc104edabb0b928680594f41 | eqiad1-r | keystone     | identity     | True    | public    | http://cloudcontrol1003.wikimedia.org:5000/v3                          |
| 88c39643619a4244b39861e2ee309999 | eqiad1-r | glance       | image        | True    | internal  | http://cloudcontrol1003.wikimedia.org:9292                             |
| a3f184beff584a4bb1dca6fe54545f8b | eqiad1-r | keystone     | identity     | True    | admin     | http://cloudcontrol1003.wikimedia.org:35357/v3                         |
| c48d1abaf5544b348e782bf376fa056a | eqiad1-r | nova         | compute      | True    | public    | http://cloudcontrol1003.wikimedia.org:8774/v2.1                        |
| ca509a8114af454d9b0621fdc60d6c20 | eqiad1-r | proxy        | proxy        | True    | internal  | http://proxy-eqiad1.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| d900008185c3476d8a969c1212df5967 | eqiad1-r | nova         | compute      | True    | internal  | http://cloudcontrol1003.wikimedia.org:8774/v2.1                        |
| df820b6526dd40cbae48b83a9856903c | eqiad1-r | nova         | compute      | True    | admin     | http://cloudcontrol1003.wikimedia.org:8774/v2.1                        |
| e136f6b43be84aebb182e7e634512a2c | eqiad1-r | neutron      | network      | True    | internal  | http://cloudcontrol1003.wikimedia.org:9696                             |
| ebf523fce03a4ff4b859277f0a3d2477 | eqiad1-r | proxy        | proxy        | True    | public    | http://proxy-eqiad1.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| f22af870a67547fc9b2c965f25d7d4f9 | eqiad1-r | designate    | dns          | True    | admin     | http://cloudservices1003.wikimedia.org:9001                            |
+----------------------------------+----------+--------------+--------------+---------+-----------+------------------------------------------------------------------------+
update commands
openstack endpoint create --region eqiad1-r compute admin http://openstack.eqiad1.wikimediacloud.org:8774/v2.1
openstack endpoint delete df820b6526dd40cbae48b83a9856903c
openstack endpoint create --region eqiad1-r compute public http://openstack.eqiad1.wikimediacloud.org:8774/v2.1
openstack endpoint delete c48d1abaf5544b348e782bf376fa056a
openstack endpoint create --region eqiad1-r compute internal http://openstack.eqiad1.wikimediacloud.org:8774/v2.1
openstack endpoint delete d900008185c3476d8a969c1212df5967

openstack endpoint create --region eqiad1-r image admin http://openstack.eqiad1.wikimediacloud.org:9292
openstack endpoint delete 3ed2c8aba1e648e2be01423a00b3b6b2
openstack endpoint create --region eqiad1-r image public http://openstack.eqiad1.wikimediacloud.org:9292
openstack endpoint delete 0ed1b6c32ec9433f842cd31ba1f11a48
openstack endpoint create --region eqiad1-r image internal http://openstack.eqiad1.wikimediacloud.org:9292
openstack endpoint delete 88c39643619a4244b39861e2ee309999

openstack endpoint create --region eqiad1-r network admin http://openstack.eqiad1.wikimediacloud.org:9696
openstack endpoint delete 4ea7fdb8da904b0290465cea0a74d545
openstack endpoint create --region eqiad1-r network public http://openstack.eqiad1.wikimediacloud.org:9696
openstack endpoint delete 53e93fccd882487396f8ae6208d09f3d
openstack endpoint create --region eqiad1-r network internal http://openstack.eqiad1.wikimediacloud.org:9696
openstack endpoint delete e136f6b43be84aebb182e7e634512a2c

openstack endpoint create --region eqiad1-r identity admin http://openstack.eqiad1.wikimediacloud.org:35357/v3
openstack endpoint delete a3f184beff584a4bb1dca6fe54545f8b
openstack endpoint create --region eqiad1-r identity public http://openstack.eqiad1.wikimediacloud.org:5000/v3
openstack endpoint delete 7ea7c2c9fc104edabb0b928680594f41
openstack endpoint create --region eqiad1-r identity internal http://openstack.eqiad1.wikimediacloud.org:5000/v3
openstack endpoint delete 43d19fccac134e868c308673c7b2de7e

openstack endpoint create --region eqiad1-r dns admin http://openstack.eqiad1.wikimediacloud.org:9001
openstack endpoint delete f22af870a67547fc9b2c965f25d7d4f9
openstack endpoint create --region eqiad1-r dns public http://openstack.eqiad1.wikimediacloud.org:9001
openstack endpoint delete 465f331e03de4bfcbef1f8a7dbb4de15
openstack endpoint create --region eqiad1-r dns internal http://openstack.eqiad1.wikimediacloud.org:9001
openstack endpoint delete 6e808288d5814ff390b1638d1d58ab13

Mentioned in SAL (#wikimedia-cloud) [2019-10-17T15:27:56Z] <jeh> update eqiad1's endpoint catalog with the new wikimediacloud.org domain T223907

New configuration

+----------------------------------+----------+--------------+--------------+---------+-----------+------------------------------------------------------------------------+
| ID                               | Region   | Service Name | Service Type | Enabled | Interface | URL                                                                    |
+----------------------------------+----------+--------------+--------------+---------+-----------+------------------------------------------------------------------------+
| 2c359c04443e4dc98fd60550bc3e37d5 | eqiad1-r | neutron      | network      | True    | admin     | http://openstack.eqiad1.wikimediacloud.org:9696                        |
| 2fde125392234fa79d3affc5af3cf1e6 | eqiad1-r | keystone     | identity     | True    | internal  | http://openstack.eqiad1.wikimediacloud.org:5000/v3                     |
| 31d67bc468f64eabb5a808d43f8565e1 | eqiad1-r | nova         | compute      | True    | admin     | http://openstack.eqiad1.wikimediacloud.org:8774/v2.1                   |
| 3a15bcbea365490f8c3b04c63d3ec35d | eqiad1-r | glance       | image        | True    | public    | http://openstack.eqiad1.wikimediacloud.org:9292                        |
| 3d001add3d1f4ef8a89217bb6a842dc5 | eqiad1-r | glance       | image        | True    | internal  | http://openstack.eqiad1.wikimediacloud.org:9292                        |
| 3d1ce837946444e3b90c11138a448fa0 | eqiad1-r | proxy        | proxy        | True    | admin     | http://proxy-eqiad1.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| 5662f4ecfd1942728e015bd9aff8c23a | eqiad1-r | designate    | dns          | True    | internal  | http://openstack.eqiad1.wikimediacloud.org:9001                        |
| 75d15f0fe25246ca8fe8328a549d3d5f | eqiad1-r | keystone     | identity     | True    | public    | http://openstack.eqiad1.wikimediacloud.org:5000/v3                     |
| 7dbe9002a1174ba2b4c4b62f935ab520 | eqiad1-r | glance       | image        | True    | admin     | http://openstack.eqiad1.wikimediacloud.org:9292                        |
| a1182f6b8e8d4f868b5d8dbe5c403386 | eqiad1-r | designate    | dns          | True    | public    | http://openstack.eqiad1.wikimediacloud.org:9001                        |
| a39310abf6e74ff187fc54658a679d2a | eqiad1-r | nova         | compute      | True    | public    | http://openstack.eqiad1.wikimediacloud.org:8774/v2.1                   |
| af3e3becab9846bf82b29d6e7ca66679 | eqiad1-r | neutron      | network      | True    | public    | http://openstack.eqiad1.wikimediacloud.org:9696                        |
| b9993a52c42b4ef694de853996f6e27a | eqiad1-r | keystone     | identity     | True    | admin     | http://openstack.eqiad1.wikimediacloud.org:35357/v3                    |
| ca509a8114af454d9b0621fdc60d6c20 | eqiad1-r | proxy        | proxy        | True    | internal  | http://proxy-eqiad1.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| cffd8ff7351549719d292cec1ba8abc9 | eqiad1-r | nova         | compute      | True    | internal  | http://openstack.eqiad1.wikimediacloud.org:8774/v2.1                   |
| e22f941762c64695a56f1650516b6c97 | eqiad1-r | neutron      | network      | True    | internal  | http://openstack.eqiad1.wikimediacloud.org:9696                        |
| ea013ac239db49e0b6c58cc63aac8352 | eqiad1-r | designate    | dns          | True    | admin     | http://openstack.eqiad1.wikimediacloud.org:9001                        |
| ebf523fce03a4ff4b859277f0a3d2477 | eqiad1-r | proxy        | proxy        | True    | public    | http://proxy-eqiad1.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
+----------------------------------+----------+--------------+--------------+---------+-----------+------------------------------------------------------------------------+
JHedden closed this task as Resolved.Thu, Oct 17, 3:43 PM

All HA changes are now implemented in both regions codfw1dev and eqiad1.

Change 542506 abandoned by BryanDavis:
wikitech: Update hostnames for OpenStack endpoints

Reason:
Obsoleted by Ia9bbc1ad8b4d4ab4c7850738747268ab7890d957

https://gerrit.wikimedia.org/r/542506