Page MenuHomePhabricator

debmonitor TLS termination
Closed, ResolvedPublic

Description

The list of services without TLS that ATS needs to connect to (T210411) was missing debmonitor, given that debmonitor.discovery.wmnet does listen on 443.

The TLS certificate for debmonitor.discovery.wmnet did not list debmonitor.wikimedia.org in the SAN field, causing HTTP requests going via ATS to result in 502 errors. I've re-created the certificate for debmonitor on the puppetmaster by using cergen and including both debmonitor.wikimedia.org and debmonitor.discovery.wmnet in SAN.

This is however not enough to fix ATS request routing to debmonitor, as the origins expect a client certificate to be supplied for TLS connections to go through.

# Internal-only HTTPS listener for debmonitor clients to POST their package lists.
# Clients will be authenticating with the Puppet certificate to ensure that each
# host can update only its own packages.

All debmonitor requests going via ATS now result in 400 errors.

Details

Related Gerrit Patches:
operations/puppet : productionATS: use port 7443 for debmonitor
operations/puppet : productiondebmonitor: expect 302 on successful TLS termination
operations/puppet : productiondebmonitor: terminate TLS on port 7443

Related Objects

Event Timeline

ema created this task.Nov 13 2019, 11:08 AM
Restricted Application added a project: Operations. · View Herald TranscriptNov 13 2019, 11:08 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as Medium priority.Nov 13 2019, 11:08 AM
ema added a subscriber: MoritzMuehlenhoff.

As we discussed a while ago about this, the easiest solution is to pick another port for the public TLS server on the debmonitor servers as the 443 is already taken for the internal clients to report the package list to it and it's used to perform authz/n with the client certificate.

So let's add say 8443 on https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/templates/debmonitor/server/nginx.conf.erb similar to the 80 configuration for the public TLS one, keeping the clearing of the HTTP headers used for authn/z.

I've two questions:

  1. should we add the clear of those HTTP headers at the ATS level too?
  2. I'm wondering if we should keep the certificates separate too (creating another discovery endpoint at that point maybe) and the reasoning for it would be to avoid that by mistake the internal endpoint to push data might be exposed to the public.

Also keep in mind that the debmonitor discovery record is now hardcoded in the DNS and not managed via geoip:

templates/wmnet:5335:debmonitor            300 IN CNAME debmonitor1001.eqiad.wmnet.

Change 550670 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] debmonitor: terminate TLS on port 7443

https://gerrit.wikimedia.org/r/550670

Change 550670 merged by Ema:
[operations/puppet@production] debmonitor: terminate TLS on port 7443

https://gerrit.wikimedia.org/r/550670

Change 550696 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] debmonitor: expect 302 on successful TLS termination

https://gerrit.wikimedia.org/r/550696

Change 550696 merged by Ema:
[operations/puppet@production] debmonitor: expect 302 on successful TLS termination

https://gerrit.wikimedia.org/r/550696

Change 550697 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: use port 7443 for debmonitor

https://gerrit.wikimedia.org/r/550697

Change 550697 merged by Ema:
[operations/puppet@production] ATS: use port 7443 for debmonitor

https://gerrit.wikimedia.org/r/550697

ema closed this task as Resolved.Nov 14 2019, 10:19 AM
ema claimed this task.

TLS termination configured on port 7443:

$ curl -v https://debmonitor.wikimedia.org:7443/login/ --resolve debmonitor.wikimedia.org:7443:10.64.32.62 2>&1 | grep '< HTTP'
< HTTP/2 200