Page MenuHomePhabricator
Create Task
Maniphest T267194

CloudVPS: enable TLS in openstack API endpoints
Closed, ResolvedPublic
Actions

Description

Our current openstack deployment uses the following API endpoints:

aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack endpoint list -c URL -c 'Service Name' -c 'Service Type' -c Interface
+--------------+--------------+-----------+------------------------------------------------------------------------+
| Service Name | Service Type | Interface | URL                                                                    |
+--------------+--------------+-----------+------------------------------------------------------------------------+
| placement    | placement    | internal  | http://openstack.eqiad1.wikimediacloud.org:8778                        |
| neutron      | network      | admin     | http://openstack.eqiad1.wikimediacloud.org:9696                        |
| keystone     | identity     | internal  | http://openstack.eqiad1.wikimediacloud.org:5000/v3                     |
| nova         | compute      | admin     | http://openstack.eqiad1.wikimediacloud.org:8774/v2.1                   |
| placement    | placement    | public    | http://openstack.eqiad1.wikimediacloud.org:8778                        |
| glance       | image        | public    | http://openstack.eqiad1.wikimediacloud.org:9292                        |
| glance       | image        | internal  | http://openstack.eqiad1.wikimediacloud.org:9292                        |
| proxy        | proxy        | admin     | http://proxy-eqiad1.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| designate    | dns          | internal  | http://openstack.eqiad1.wikimediacloud.org:9001                        |
| keystone     | identity     | public    | http://openstack.eqiad1.wikimediacloud.org:5000/v3                     |
| glance       | image        | admin     | http://openstack.eqiad1.wikimediacloud.org:9292                        |
| placement    | placement    | admin     | http://openstack.eqiad1.wikimediacloud.org:8778                        |
| designate    | dns          | public    | http://openstack.eqiad1.wikimediacloud.org:9001                        |
| nova         | compute      | public    | http://openstack.eqiad1.wikimediacloud.org:8774/v2.1                   |
| neutron      | network      | public    | http://openstack.eqiad1.wikimediacloud.org:9696                        |
| keystone     | identity     | admin     | http://openstack.eqiad1.wikimediacloud.org:35357/v3                    |
| proxy        | proxy        | internal  | http://proxy-eqiad1.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| nova         | compute      | internal  | http://openstack.eqiad1.wikimediacloud.org:8774/v2.1                   |
| neutron      | network      | internal  | http://openstack.eqiad1.wikimediacloud.org:9696                        |
| designate    | dns          | admin     | http://openstack.eqiad1.wikimediacloud.org:9001                        |
| proxy        | proxy        | public    | http://proxy-eqiad1.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
+--------------+--------------+-----------+------------------------------------------------------------------------+

The openstack.eqiad1.wikimediacloud.org FQDN points to one of the cloudcontrol nodes, which runs HAproxy to load-balance request between the 3 cloudcontrol servers.

One of the things that we have to decide is where to do TLS termination. In HAproxy? in the openstack API service itself? In any case we should implement TLS by using acme-chief.

NOTE: this change should happen first in codfw1dev for the following endpoints:
aborrero@cloudcontrol2001-dev:~ $ sudo wmcs-openstack endpoint list -c URL -c 'Service Name' -c 'Service Type' -c Interface
+--------------+--------------+-----------+-------------------------------------------------------------------------------+
| Service Name | Service Type | Interface | URL                                                                           |
+--------------+--------------+-----------+-------------------------------------------------------------------------------+
| keystone     | identity     | internal  | http://openstack.codfw1dev.wikimediacloud.org:5000/v3                         |
| proxy        | proxy        | admin     | http://novaproxy.codfw1dev.wmcloud.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| designate    | dns          | admin     | http://openstack.codfw1dev.wikimediacloud.org:9001                            |
| neutron      | network      | internal  | http://openstack.codfw1dev.wikimediacloud.org:9696                            |
| nova         | compute      | admin     | http://openstack.codfw1dev.wikimediacloud.org:8774/v2.1                       |
| barbican     | key-manager  | admin     | http://openstack.codfw1dev.wikimediacloud.org:9311                            |
| designate    | dns          | public    | http://openstack.codfw1dev.wikimediacloud.org:9001                            |
| placement    | placement    | public    | http://openstack.codfw1dev.wikimediacloud.org:8778                            |
| keystone     | identity     | admin     | http://openstack.codfw1dev.wikimediacloud.org:35357/v3                        |
| glance       | image        | admin     | http://openstack.codfw1dev.wikimediacloud.org:9292                            |
| barbican     | key-manager  | internal  | http://openstack.codfw1dev.wikimediacloud.org:9311                            |
| nova         | compute      | internal  | http://openstack.codfw1dev.wikimediacloud.org:8774/v2.1                       |
| placement    | placement    | internal  | http://openstack.codfw1dev.wikimediacloud.org:8778                            |
| neutron      | network      | admin     | http://openstack.codfw1dev.wikimediacloud.org:9696                            |
| nova         | compute      | public    | http://openstack.codfw1dev.wikimediacloud.org:8774/v2.1                       |
| proxy        | proxy        | internal  | http://novaproxy.codfw1dev.wmcloud.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| neutron      | network      | public    | http://openstack.codfw1dev.wikimediacloud.org:9696                            |
| barbican     | key-manager  | public    | http://openstack.codfw1dev.wikimediacloud.org:9311                            |
| glance       | image        | public    | http://openstack.codfw1dev.wikimediacloud.org:9292                            |
| proxy        | proxy        | public    | http://novaproxy.codfw1dev.wmcloud.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| keystone     | identity     | public    | http://openstack.codfw1dev.wikimediacloud.org:5000/v3                         |
| designate    | dns          | internal  | http://openstack.codfw1dev.wikimediacloud.org:9001                            |
| glance       | image        | internal  | http://openstack.codfw1dev.wikimediacloud.org:9292                            |
| placement    | placement    | admin     | http://openstack.codfw1dev.wikimediacloud.org:8778                            |
+--------------+--------------+-----------+-------------------------------------------------------------------------------+

Related documentation:

Details

SubjectRepoBranchLines +/-
profile::openstack::base::designate::firewall::api: add a missing )operations/puppetproduction+1 -1
P:openstack: misc cleanup for non-tls portsoperations/puppetproduction+22 -22
P:openstack::designate::firewall: cleanupoperations/puppetproduction+3 -33
P:openstack::haproxy: eqiad1: remove non-tls portsoperations/puppetproduction+0 -24
P:openstack::haproxy: codfw1dev: remove non-tls portsoperations/puppetproduction+0 -39
P:openstack::designate: set base_url to use the https portoperations/puppetproduction+2 -2
openstack:haproxy add tls for nova metadata serviceoperations/puppetproduction+16 -12
openstack: fix remaining http keystone urlsoperations/puppetproduction+15 -15
openstack: haproxy: add tls termination supportoperations/puppetproduction+85 -0
acme_chief: add wildcard to openstack certsoperations/puppetproduction+2 -0
codfw1dev.wikimediacloud.org: Add new hostnames for tls openstack endpointsoperations/dnsmaster+11 -0
keystone: use https check rather than http check for endpoint checksoperations/puppetproduction+2 -2
Striker: use https endpoint for OpenStackoperations/puppetproduction+3 -3
Designate: specify https for keystone authtoken urloperations/puppetproduction+2 -2
OpenStack: Always use the tls ports (25000, 25357) for openstack servicesoperations/puppetproduction+61 -61
Designate: add tls firewall rules to the api firewall classoperations/puppetproduction+13 -1
openStack:haproxy add tls termination for openstack in eqiad1operations/puppetproduction+70 -45
cr-cloud: add tls ports for openstack servicesoperations/homer/publicmaster+22 -2
openstack::haproxy: add tls for the barbican api in codfw1devoperations/puppetproduction+8 -6
openstack::haproxy: add tls for the swift api in codfw1devoperations/puppetproduction+8 -6
openstack::haproxy: add tls for the placement api in codfw1devoperations/puppetproduction+8 -6
openstack::haproxy: add tls for the trove api in codfw1devoperations/puppetproduction+8 -6
openstack::haproxy: add tls for the neutron api in codfw1devoperations/puppetproduction+8 -6
openstack::haproxy: add tls for the cinder api in codfw1devoperations/puppetproduction+8 -6
openstack::haproxy: add tls for the keystone-admin api in codfw1devoperations/puppetproduction+16 -7
openstack::haproxy: Add tls frontend for designate api in codfw1devoperations/puppetproduction+9 -6
openstack::haproxy: tls-ify nova and glance on codfw1devoperations/puppetproduction+16 -12
openstack:haproxy: set x-forwarded-protooperations/puppetproduction+2 -1
openstack:haproxy: port-based tls termination supportoperations/puppetproduction+34 -6
acme_chief: add openstack certsoperations/puppetproduction+14 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
aborrero triaged this task as Medium priority.Nov 4 2020, 10:57 AM
aborrero moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.
Nintendofan885 subscribed.Nov 4 2020, 12:41 PM
• Bstorm added a subtask: T256144: CloudVPS: cleanup hiera values using old openstack services names.Feb 11 2021, 5:46 PM
• Bstorm mentioned this in T256144: CloudVPS: cleanup hiera values using old openstack services names.
• Bstorm merged a task: T225932: support ssl for openstack REST endpoints.Feb 11 2021, 5:51 PM
• Bstorm added subscribers: • JHedden, Krenair, Paladox.
dcaro edited projects, added cloud-services-team (zz-archived1); removed cloud-services-team (Kanban).Sep 28 2021, 4:32 PM
dcaro edited projects, added User-dcaro; removed cloud-services-team (zz-archived1).Oct 5 2021, 8:58 AM
taavi subscribed.Oct 5 2021, 12:09 PM
gerritbot added a comment.Oct 5 2021, 12:29 PM

Change 726585 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] acme_chief: add openstack certs

https://gerrit.wikimedia.org/r/726585

gerritbot added a project: Patch-For-Review.Oct 5 2021, 12:29 PM
taavi added a project: Cloud-VPS.Oct 5 2021, 3:00 PM
taavi added a comment.Oct 6 2021, 2:55 PM

One of the things that we have to decide is where to do TLS termination. In HAproxy? in the openstack API service itself?

Ideally we would terminate TLS on the api service itself, to prevent unencrypted requests between haproxy and the api service on another cloudcontrol node.

Essentially we can either

  • Reserve two more ports for each service (one internal and one external). There are no "official" alternative TLS ports for each service (per https://docs.openstack.org/install-guide/firewalls-default-ports.html), but we can likely pick our own
  • Perform vhost-based routing and use one common public port (so likely 443) but multiple hostnames (like trove.openstack.eqiad1.wikimediacloud.org and cinder.openstack.eqiad1.wikimediacloud.org).

I slightly favour the latter, it gives us cleaner urls (in my opinion), makes it easier to split a service to separate from cloudcontrol if needed and it only needs one extra internal port (so we can just prepend 2, like 1 for current internal ports). The downsides are extra complexity, mostly. We can either terminate TLS on the HAProxy layer (and re-encrypt the traffic to the backend) or do SNI-based routing.

gerritbot added a comment.Oct 7 2021, 8:03 AM

Change 726585 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] acme_chief: add openstack certs

https://gerrit.wikimedia.org/r/726585

Maintenance_bot removed a project: Patch-For-Review.Oct 7 2021, 8:10 AM
dcaro added a comment.Oct 8 2021, 8:09 AM

Perform vhost-based routing and use one common public port (so likely 443) but multiple hostnames (like trove.openstack.eqiad1.wikimediacloud.org and cinder.openstack.eqiad1.wikimediacloud.org).

+1 for this, I also see it as way easier to understand and flexible, probably SNI based routing might be simpler (only have to maintain one cert). This might mean though that we have split DNS, so other internal hosts resolve the names to the internal IPs, or all the traffic has to pass through HAProxy that it's probably nicer, so we get HA also internally, but might slow down requests and increase traffic, we can try and see if that becomes a problem though.

gerritbot added a comment.Oct 8 2021, 8:21 AM

Change 728246 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] acme_chief: add wildcard to openstack certs

https://gerrit.wikimedia.org/r/728246

gerritbot added a project: Patch-For-Review.Oct 8 2021, 8:21 AM
gerritbot added a comment.Oct 8 2021, 9:06 AM

Change 728260 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: haproxy: add tls termination support

https://gerrit.wikimedia.org/r/728260

gerritbot added a comment.Oct 14 2021, 6:48 PM

Change 730879 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/dns@master] codfw1dev.wikimediacloud.org: Add new hostnames for tls openstack endpoints

https://gerrit.wikimedia.org/r/730879

aborrero added a comment.EditedOct 15 2021, 10:14 AM

I have some doubts about the approach being taken (different FQDNs, all share the same port), for the following reasons:

  • it was mentioned that it simplifies x509 certificate handling. Nowadays x509 certificate management is fully automated by acme-chief, so I don't think the number of certificates at play should be of any concern.
  • some services may use different ports for different interfaces (admin, internal, public). This can be seen today in at least keystone, which uses a different TCP port for admin interface. As far as I know, this is the standard way upstream recommends separating the 3 interfaces. But I may be wrong here (didn't check in deep).
  • There is a mention to the TLS gap between haproxy <-> API service. The gap only happens if we use haproxy in HTTP mode, no? I don't see any problem with using haproxy in TCP mode. We would need the certificate to be installed in each service individually anyway. In TCP mode, there is no "gap" in the TLS connection between the end client and the final openstack API service.
  • We currently use a lot of firewalling to allow/disallow specific ports for specific clients. If all the endpoints share the same port (443/TCP for example) this would eliminate our ability to do access control like we have been doing so far.
  • It was mentioned that we want to have the flexibility to create API endpoints that doesn't live on cloudcontrol servers. I think this is not related to the TLS thing. We could still create whatever.openstack.xxx.wikimediacloud.org and point that to another host anytime, in case we had such case (swift is the first and only projected case of this).

In summary: I'm failing to see any benefits in having multiple FQDNs, specially if 90% requests will end in the same system anyway.

I'd suggest we take this direction:

  • select one individual openstack API endpoint, and add TLS to its service, on a new TCP port.
  • make HAproxy work on TCP mode (currently configured on haproxy.conf)
  • register the new API endpoint to the openstack catalog, and update client configs to see if that works.
  • if everything works as expected, move to the next service.
NOTE: (edit) I'm open to be convinced that the multiple FQDN approach is better
aborrero added a comment.Oct 15 2021, 11:50 AM

Per our IRC chat, it was surfaced that the HAproxy TCP mode is not desirable in this situation because some API services need to know the original client source IP address (example: keystone password safelist mechanism).

However, I believe we can still use a single FQDN + multiple ports, with HAproxy in HTTP mode. We would just need to load the same x509 cert in both HAproxy and each individual openstack service.

• Bstorm added a comment.Oct 15 2021, 3:07 PM

The sort of vhost routing this does is the common use of haproxy and where they've done the most work on the software itself. It has an advanced and extensive ACL interface so that you can share one port with many FQDNs that allows extensive and fine grained access control if you want it as well since people use haproxy to handle edge traffic. It will probably be how LBaaS works in Openstack and is how I've used haproxy elsewhere to keep entire businesses behind a cluster of them (behind a caching layer). I might suggest the ultimate goal probably ought to be allowing access to the APIs at least inside the cloud instead of firewalling quite so much. That would allow for automation and more standard capabilities.

That's not to speak to your specific concerns around the admin interface and such because I haven't dug in at all, but you could easily replace the firewalling with acls in haproxy in a way that might even be more immediately readable. Just food for thought.

aborrero added a comment.Oct 19 2021, 4:58 PM

My personal opinion is that we should try first with the single FQDN approach. I believe is the option demanding less changes, easier to implement, and easier to understand.

However, I'm open to be convinced otherwise.

gerritbot added a comment.Oct 19 2021, 5:14 PM

Change 732012 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack:haproxy: port-based tls termination support

https://gerrit.wikimedia.org/r/732012

gerritbot added a comment.Oct 19 2021, 5:25 PM

Change 732012 merged by Andrew Bogott:

[operations/puppet@production] openstack:haproxy: port-based tls termination support

https://gerrit.wikimedia.org/r/732012

gerritbot added a comment.Oct 19 2021, 5:32 PM

Change 732017 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack:haproxy: set x-forwarded-proto

https://gerrit.wikimedia.org/r/732017

gerritbot added a comment.Oct 19 2021, 5:37 PM

Change 732017 merged by Andrew Bogott:

[operations/puppet@production] openstack:haproxy: set x-forwarded-proto

https://gerrit.wikimedia.org/r/732017

gerritbot added a comment.Oct 19 2021, 5:44 PM

Change 732021 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack::haproxy: tls-ify nova and glance on codfw1dev

https://gerrit.wikimedia.org/r/732021

gerritbot added a comment.Oct 19 2021, 5:47 PM

Change 732021 merged by Andrew Bogott:

[operations/puppet@production] openstack::haproxy: tls-ify nova and glance on codfw1dev

https://gerrit.wikimedia.org/r/732021

gerritbot added a comment.Oct 19 2021, 7:24 PM

Change 732039 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack::haproxy: Add tls frontend for designate api in codfw1dev

https://gerrit.wikimedia.org/r/732039

gerritbot added a comment.Oct 19 2021, 7:24 PM

Change 732040 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack::haproxy: add tls for the keystone-admin api in codfw1dev

https://gerrit.wikimedia.org/r/732040

gerritbot added a comment.Oct 19 2021, 7:24 PM

Change 732041 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack::haproxy: add tls for the placement api in codfw1dev

https://gerrit.wikimedia.org/r/732041

gerritbot added a comment.Oct 19 2021, 7:24 PM

Change 732042 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack::haproxy: add tls for the trove api in codfw1dev

https://gerrit.wikimedia.org/r/732042

gerritbot added a comment.Oct 19 2021, 7:24 PM

Change 732043 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack::haproxy: add tls for the cinder api in codfw1dev

https://gerrit.wikimedia.org/r/732043

gerritbot added a comment.Oct 19 2021, 7:24 PM

Change 732044 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack::haproxy: add tls for the neutron api in codfw1dev

https://gerrit.wikimedia.org/r/732044

gerritbot added a comment.Oct 19 2021, 7:24 PM

Change 732045 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack::haproxy: add tls for the swift api in codfw1dev

https://gerrit.wikimedia.org/r/732045

gerritbot added a comment.Oct 19 2021, 7:24 PM

Change 732046 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack::haproxy: add tls for the barbican api in codfw1dev

https://gerrit.wikimedia.org/r/732046

gerritbot added a comment.Oct 19 2021, 9:32 PM

Change 732039 merged by Andrew Bogott:

[operations/puppet@production] openstack::haproxy: Add tls frontend for designate api in codfw1dev

https://gerrit.wikimedia.org/r/732039

gerritbot added a comment.Oct 19 2021, 9:34 PM

Change 732040 merged by Andrew Bogott:

[operations/puppet@production] openstack::haproxy: add tls for the keystone-admin api in codfw1dev

https://gerrit.wikimedia.org/r/732040

gerritbot added a comment.Oct 19 2021, 9:40 PM

Change 732043 merged by Andrew Bogott:

[operations/puppet@production] openstack::haproxy: add tls for the cinder api in codfw1dev

https://gerrit.wikimedia.org/r/732043

gerritbot added a comment.Oct 19 2021, 9:43 PM

Change 732044 merged by Andrew Bogott:

[operations/puppet@production] openstack::haproxy: add tls for the neutron api in codfw1dev

https://gerrit.wikimedia.org/r/732044

gerritbot added a comment.Oct 19 2021, 9:47 PM

Change 732042 merged by Andrew Bogott:

[operations/puppet@production] openstack::haproxy: add tls for the trove api in codfw1dev

https://gerrit.wikimedia.org/r/732042

gerritbot added a comment.Oct 19 2021, 9:49 PM

Change 732041 merged by Andrew Bogott:

[operations/puppet@production] openstack::haproxy: add tls for the placement api in codfw1dev

https://gerrit.wikimedia.org/r/732041

gerritbot added a comment.Oct 19 2021, 9:51 PM

Change 732045 merged by Andrew Bogott:

[operations/puppet@production] openstack::haproxy: add tls for the swift api in codfw1dev

https://gerrit.wikimedia.org/r/732045

gerritbot added a comment.Oct 19 2021, 9:54 PM

Change 732046 merged by Andrew Bogott:

[operations/puppet@production] openstack::haproxy: add tls for the barbican api in codfw1dev

https://gerrit.wikimedia.org/r/732046

Andrew added a comment.Oct 19 2021, 10:25 PM

Now we have all openstack service in codfw1dev working with tls:

andrew@cloudcontrol2001-dev:~$ sudo wmcs-openstack endpoint list -c URL -c 'Service Name' -c 'Service Type' -c Interface
+--------------+--------------+-----------+-----------------------------------------------------------------------------------+
| Service Name | Service Type | Interface | URL                                                                               |
+--------------+--------------+-----------+-----------------------------------------------------------------------------------+
| glance       | image        | admin     | https://openstack.codfw1dev.wikimediacloud.org:29292                              |
| proxy        | proxy        | admin     | http://novaproxy.codfw1dev.wmcloud.org:5668/dynamicproxy-api/v1/$(tenant_id)s     |
| placement    | placement    | admin     | https://openstack.codfw1dev.wikimediacloud.org:28778                              |
| nova         | compute      | internal  | https://openstack.codfw1dev.wikimediacloud.org:28774/v2.1                         |
| nova         | compute      | public    | https://openstack.codfw1dev.wikimediacloud.org:28774/v2.1                         |
| neutron      | network      | public    | https://openstack.codfw1dev.wikimediacloud.org:29696                              |
| designate    | dns          | admin     | https://openstack.codfw1dev.wikimediacloud.org:29001                              |
| neutron      | network      | internal  | https://openstack.codfw1dev.wikimediacloud.org:29696                              |
| trove        | database     | public    | https://openstack.codfw1dev.wikimediacloud.org:28779/v1.0/%(tenant_id)s           |
| keystone     | identity     | internal  | https://openstack.codfw1dev.wikimediacloud.org:25000/v3                           |
| cinderv2     | volumev2     | admin     | https://openstack.codfw1dev.wikimediacloud.org:28776/v2/%(project_id)s            |
| neutron      | network      | admin     | https://openstack.codfw1dev.wikimediacloud.org:29696                              |
| swift        | object-store | internal  | https://openstack.codfw1dev.wikimediacloud.org:28080/swift/v1/AUTH_$(project_id)s |
| swift        | object-store | public    | https://openstack.codfw1dev.wikimediacloud.org:28080/swift/v1/AUTH_$(project_id)s |
| keystone     | identity     | internal  | http://openstack.codfw1dev.wikimediacloud.org:5000/v3                             |
| proxy        | proxy        | internal  | http://novaproxy.codfw1dev.wmcloud.org:5668/dynamicproxy-api/v1/$(tenant_id)s     |
| trove        | database     | internal  | https://openstack.codfw1dev.wikimediacloud.org:28779/v1.0/%(tenant_id)s           |
| barbican     | key-manager  | public    | https://openstack.codfw1dev.wikimediacloud.org:29311                              |
| designate    | dns          | internal  | https://openstack.codfw1dev.wikimediacloud.org:29001                              |
| barbican     | key-manager  | admin     | https://openstack.codfw1dev.wikimediacloud.org:29311                              |
| keystone     | identity     | public    | https://openstack.codfw1dev.wikimediacloud.org:25000/v3                           |
| glance       | image        | internal  | https://openstack.codfw1dev.wikimediacloud.org:29292                              |
| keystone     | identity     | admin     | https://openstack.codfw1dev.wikimediacloud.org:25357/v3                           |
| cinderv3     | volumev3     | admin     | https://openstack.codfw1dev.wikimediacloud.org:28776/v3/%(project_id)s            |
| barbican     | key-manager  | internal  | https://openstack.codfw1dev.wikimediacloud.org:29311                              |
| swift        | object-store | admin     | https://openstack.codfw1dev.wikimediacloud.org:28080/swift/v1/AUTH_$(project_id)s |
| cinderv2     | volumev2     | public    | https://openstack.codfw1dev.wikimediacloud.org:28776/v2/%(project_id)s            |
| proxy        | proxy        | public    | http://novaproxy.codfw1dev.wmcloud.org:5668/dynamicproxy-api/v1/$(tenant_id)s     |
| cinderv2     | volumev2     | internal  | https://openstack.codfw1dev.wikimediacloud.org:28776/v2/%(project_id)s            |
| trove        | database     | admin     | https://openstack.codfw1dev.wikimediacloud.org:28779/v1.0/%(tenant_id)s           |
| placement    | placement    | public    | https://openstack.codfw1dev.wikimediacloud.org:28778                              |
| cinderv3     | volumev3     | internal  | https://openstack.codfw1dev.wikimediacloud.org:28776/v3/%(project_id)s            |
| designate    | dns          | public    | https://openstack.codfw1dev.wikimediacloud.org:29001                              |
| nova         | compute      | admin     | https://openstack.codfw1dev.wikimediacloud.org:28774/v2.1                         |
| placement    | placement    | internal  | https://openstack.codfw1dev.wikimediacloud.org:28778                              |
| keystone     | identity     | admin     | http://openstack.codfw1dev.wikimediacloud.org:35357/v3                            |
| glance       | image        | public    | https://openstack.codfw1dev.wikimediacloud.org:29292                              |
| cinderv3     | volumev3     | public    | https://openstack.codfw1dev.wikimediacloud.org:28776/v3/%(project_id)s            |
+--------------+--------------+-----------+-----------------------------------------------------------------------------------+

I've left a few http keystone proxies in place to support existing users; those cases share config between eqiad1 and codfw1dev so it's much simpler to cut them both over at once. I've done a lot of by-hand testing with those endpoints deleted and things seem fine. I also hacked Horizon to use the tls endpoint and it continued to work.

So as far as I can tell we're ready to apply this same change in eqiad1.

aborrero added a comment.Oct 20 2021, 10:49 AM

Awesome! thanks you both @Majavah and @Andrew

gerritbot added a comment.Oct 20 2021, 1:49 PM

Change 732321 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/homer/public@master] cr-cloud: add tls ports for openstack services

https://gerrit.wikimedia.org/r/732321

gerritbot added a comment.Oct 20 2021, 6:20 PM

Change 732397 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openStack:haproxy add tls termination for openstack

https://gerrit.wikimedia.org/r/732397

gerritbot added a comment.Oct 20 2021, 6:28 PM

Change 732398 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack:haproxy add tls for nova metadata service

https://gerrit.wikimedia.org/r/732398

gerritbot added a comment.Oct 21 2021, 9:42 AM

Change 732321 merged by Arturo Borrero Gonzalez:

[operations/homer/public@master] cr-cloud: add tls ports for openstack services

https://gerrit.wikimedia.org/r/732321

taavi mentioned this in rOHPUc5f92831d207: cr-cloud: add tls ports for openstack services.Oct 21 2021, 9:43 AM
gerritbot added a comment.Oct 21 2021, 2:36 PM

Change 732397 merged by Andrew Bogott:

[operations/puppet@production] openStack:haproxy add tls termination for openstack in eqiad1

https://gerrit.wikimedia.org/r/732397

gerritbot added a comment.Oct 21 2021, 3:18 PM

Change 732720 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Designate: add tls firewall rules to the api firewall class

https://gerrit.wikimedia.org/r/732720

gerritbot added a comment.Oct 21 2021, 3:20 PM

Change 732720 merged by Andrew Bogott:

[operations/puppet@production] Designate: add tls firewall rules to the api firewall class

https://gerrit.wikimedia.org/r/732720

Andrew added a comment.EditedOct 21 2021, 3:33 PM

We now have tls endpoints in eqiad1 as well; everything seems to be working.

Some remaining tasks are:

  • Move existing uses of the non-tls keystone endoint to tls (mostly puppet search/replace)
  • Make a plan to open/announce these endpoints to the greated internet
  • Figure out about moving the novaproxy behind tls (and also add auth)
  • Figure out about moving the puppet enc behind tls (and also add auth)

Those last two are somewhat out-of-band for this task but they're needed for the larger goal of allowing people to run orchestration tools somewhere other than cloudcontrol.

gerritbot added a comment.Oct 21 2021, 4:37 PM

Change 732738 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] OpenStack: Always use the tls port (25357) for keystone admin endpoints

https://gerrit.wikimedia.org/r/732738

gerritbot added a comment.Oct 24 2021, 12:46 AM

Change 732738 merged by Andrew Bogott:

[operations/puppet@production] OpenStack: Always use the tls ports (25000, 25357) for openstack services

https://gerrit.wikimedia.org/r/732738

gerritbot added a comment.Oct 24 2021, 12:58 AM

Change 733353 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Designate: specify https for keystone authtoken url

https://gerrit.wikimedia.org/r/733353

gerritbot added a comment.Oct 24 2021, 12:59 AM

Change 733353 merged by Andrew Bogott:

[operations/puppet@production] Designate: specify https for keystone authtoken url

https://gerrit.wikimedia.org/r/733353

gerritbot added a comment.Oct 24 2021, 1:14 AM

Change 733362 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Striker: use https endpoint for OpenStack

https://gerrit.wikimedia.org/r/733362

gerritbot added a comment.Oct 24 2021, 1:15 AM

Change 733362 merged by Andrew Bogott:

[operations/puppet@production] Striker: use https endpoint for OpenStack

https://gerrit.wikimedia.org/r/733362

gerritbot added a comment.Oct 24 2021, 1:21 AM

Change 733363 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] keystone: use https check rather than http check for endpoint checks

https://gerrit.wikimedia.org/r/733363

gerritbot added a comment.Oct 24 2021, 1:22 AM

Change 733363 merged by Andrew Bogott:

[operations/puppet@production] keystone: use https check rather than http check for endpoint checks

https://gerrit.wikimedia.org/r/733363

gerritbot added a comment.Oct 26 2021, 3:02 PM

Change 730879 abandoned by Andrew Bogott:

[operations/dns@master] codfw1dev.wikimediacloud.org: Add new hostnames for tls openstack endpoints

Reason:

dropped in favor of per-port tls

https://gerrit.wikimedia.org/r/730879

dcaro reassigned this task from dcaro to Andrew.Nov 22 2021, 1:42 PM
taavi mentioned this in T296192: Requesting access to wmcs-roots, labtest-roots for Taavi Väänänen (Majavah).Nov 22 2021, 2:40 PM
dcaro removed a project: User-dcaro.Dec 2 2021, 4:34 PM
gerritbot added a comment.Jan 22 2022, 4:46 PM

Change 728246 abandoned by Majavah:

[operations/puppet@production] acme_chief: add wildcard to openstack certs

Reason:

we ended up not needing this

https://gerrit.wikimedia.org/r/728246

gerritbot added a comment.Jan 22 2022, 4:46 PM

Change 728260 abandoned by Majavah:

[operations/puppet@production] openstack: haproxy: add tls termination support

Reason:

ended up going with a different route

https://gerrit.wikimedia.org/r/728260

gerritbot added a comment.Mar 6 2022, 11:16 AM

Change 768293 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: fix remaining http keystone urls

https://gerrit.wikimedia.org/r/768293

gerritbot added a comment.Mar 7 2022, 9:18 AM

Change 768293 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: fix remaining http keystone urls

https://gerrit.wikimedia.org/r/768293

Andrew added a comment.Mar 29 2022, 4:27 PM

This looks done to me! @Majavah, @aborrero any leftover pieces you're aware of?

taavi added a comment.Mar 29 2022, 4:37 PM

We'll want to eventually disable the non-tls endpoints, otherwise this is pretty much done

Andrew closed subtask T256144: CloudVPS: cleanup hiera values using old openstack services names as Resolved.Mar 29 2022, 5:05 PM
gerritbot added a comment.Apr 11 2022, 1:55 PM

Change 732398 abandoned by Andrew Bogott:

[operations/puppet@production] openstack:haproxy add tls for nova metadata service

Reason:

https://gerrit.wikimedia.org/r/732398

Andrew mentioned this in T305414: Q4:(Need By: TBD) rack/setup/install cloudweb100[34].Apr 11 2022, 6:55 PM
gerritbot added a comment.May 29 2022, 4:55 PM

Change 800948 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:openstack::designate: set base_url to use the https port

https://gerrit.wikimedia.org/r/800948

gerritbot added a comment.May 29 2022, 4:55 PM

Change 800952 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:openstack::haproxy: codfw1dev: remove non-tls ports

https://gerrit.wikimedia.org/r/800952

gerritbot added a comment.May 29 2022, 4:55 PM

Change 800953 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:openstack::haproxy: eqiad1: remove non-tls ports

https://gerrit.wikimedia.org/r/800953

gerritbot added a comment.May 29 2022, 4:55 PM

Change 800954 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:openstack::designate::firewall: cleanup

https://gerrit.wikimedia.org/r/800954

gerritbot added a comment.May 29 2022, 4:55 PM

Change 800955 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:openstack: misc cleanup for non-tls ports

https://gerrit.wikimedia.org/r/800955

gerritbot added a comment.Jun 27 2022, 7:29 PM

Change 800948 merged by Andrew Bogott:

[operations/puppet@production] P:openstack::designate: set base_url to use the https port

https://gerrit.wikimedia.org/r/800948

bd808 added a project: cloud-services-team (Kanban).Jul 6 2022, 5:25 PM
bd808 moved this task from Soon! to Doing on the cloud-services-team (Kanban) board.
gerritbot added a comment.Aug 14 2022, 7:16 PM

Change 800952 merged by Andrew Bogott:

[operations/puppet@production] P:openstack::haproxy: codfw1dev: remove non-tls ports

https://gerrit.wikimedia.org/r/800952

gerritbot added a comment.Aug 14 2022, 7:26 PM

Change 800953 merged by Andrew Bogott:

[operations/puppet@production] P:openstack::haproxy: eqiad1: remove non-tls ports

https://gerrit.wikimedia.org/r/800953

gerritbot added a comment.Aug 14 2022, 7:40 PM

Change 800954 merged by Andrew Bogott:

[operations/puppet@production] P:openstack::designate::firewall: cleanup

https://gerrit.wikimedia.org/r/800954

gerritbot added a comment.Aug 14 2022, 7:49 PM

Change 800955 merged by Andrew Bogott:

[operations/puppet@production] P:openstack: misc cleanup for non-tls ports

https://gerrit.wikimedia.org/r/800955

gerritbot added a comment.Aug 14 2022, 8:01 PM

Change 822738 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] profile::openstack::base::designate::firewall::api: add a missing )

https://gerrit.wikimedia.org/r/822738

gerritbot added a comment.Aug 14 2022, 8:04 PM

Change 822738 merged by Andrew Bogott:

[operations/puppet@production] profile::openstack::base::designate::firewall::api: add a missing )

https://gerrit.wikimedia.org/r/822738

Andrew added a comment.Aug 31 2022, 9:00 PM

Anything left?

Andrew added a parent task: T319312: Open Openstack APIs to the public internet.Oct 4 2022, 3:31 PM
taavi removed a subtask: T319313: Prepare to block and throttle Openstack APIs.Oct 4 2022, 3:36 PM
taavi closed this task as Resolved.Oct 4 2022, 7:26 PM
taavi claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL