Page MenuHomePhabricator

Applayer services without TLS
Open, Stalled, MediumPublic

Description

The following application layer services have been defined in ATS as accessible only via plain HTTP. We should figure out whether they currently support TLS. If not, they should, to allow us to perform cross-DC HTTPS requests with ATS.

The list is in the format $origin_server - $websites and was generated with P7842.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -1
labs/privatemaster+3 -0
operations/puppetproduction+24 -0
operations/puppetproduction+9 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+22 -22
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+10 -0
operations/puppetproduction+25 -0
operations/dnsmaster+2 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+9 -1
operations/puppetproduction+20 -20
operations/puppetproduction+24 -0
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+6 -3
operations/puppetproduction+28 -0
operations/puppetproduction+9 -0
labs/privatemaster+3 -0
operations/dnsmaster+1 -0
operations/dnsmaster+1 -0
operations/dnsmaster+1 -0
labs/privatemaster+3 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -2
operations/puppetproduction+21 -20
operations/dnsmaster+1 -0
operations/puppetproduction+10 -0
operations/puppetproduction+9 -0
operations/dnsmaster+1 -1
operations/puppetproduction+27 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+46 -1
operations/puppetproduction+22 -24
operations/puppetproduction+2 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -2
operations/puppetproduction+6 -3
operations/puppetproduction+0 -1
operations/puppetproduction+3 -3
operations/puppetproduction+21 -2
operations/deployment-chartsmaster+193 -98
operations/puppetproduction+8 -0
labs/privatemaster+3 -0
operations/puppetproduction+28 -0
operations/puppetproduction+1 -2
operations/puppetproduction+7 -0
operations/dnsmaster+2 -0
operations/puppetproduction+27 -0
labs/privatemaster+3 -0
operations/puppetproduction+1 -2
operations/dnsmaster+2 -0
operations/puppetproduction+7 -0
operations/puppetproduction+29 -0
operations/puppetproduction+1 -2
labs/privatemaster+3 -0
operations/puppetproduction+36 -6
operations/puppetproduction+8 -0
operations/puppetproduction+24 -23
operations/puppetproduction+2 -2
operations/puppetproduction+32 -6
operations/puppetproduction+7 -0
operations/puppetproduction+7 -0
operations/puppetproduction+25 -0
labs/privatemaster+3 -0
operations/dnsmaster+1 -0
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+26 -0
operations/puppetproduction+20 -20
operations/puppetproduction+23 -22
operations/puppetproduction+2 -2
operations/puppetproduction+1 -0
operations/puppetproduction+7 -0
operations/puppetproduction+26 -0
labs/privatemaster+3 -0
operations/puppetproduction+1 -1
operations/puppetproduction+2 -0
operations/puppetproduction+55 -24
operations/puppetproduction+7 -0
operations/puppetproduction+2 -3
operations/puppetproduction+7 -0
operations/puppetproduction+8 -0
operations/puppetproduction+1 -1
operations/puppetproduction+34 -2
operations/dnsmaster+1 -0
operations/puppetproduction+2 -3
operations/puppetproduction+8 -0
operations/puppetproduction+25 -0
operations/puppetproduction+2 -0
operations/puppetproduction+3 -3
operations/puppetproduction+3 -1
operations/puppetproduction+3 -3
operations/puppetproduction+24 -24
operations/dnsmaster+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+7 -0
operations/puppetproduction+28 -0
labs/privatemaster+3 -0
operations/puppetproduction+8 -0
operations/puppetproduction+7 -0
labs/privatemaster+3 -0
operations/puppetproduction+28 -0
operations/puppetproduction+5 -10
operations/puppetproduction+8 -0
operations/puppetproduction+7 -0
operations/dnsmaster+1 -0
operations/puppetproduction+33 -1
operations/puppetproduction+27 -0
labs/privatemaster+3 -0
operations/puppetproduction+10 -20
operations/puppetproduction+13 -0
operations/puppetproduction+202 -0
operations/dnsmaster+1 -0
labs/privatemaster+3 -0
operations/puppetproduction+31 -0
operations/puppetproduction+7 -7
operations/puppetproduction+1 -1
operations/puppetproduction+6 -6
operations/puppetproduction+5 -0
operations/puppetproduction+25 -0
labs/privatemaster+3 -0
operations/puppetproduction+40 -0
operations/puppetproduction+27 -0
labs/privatemaster+3 -0
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 552944 merged by Dzahn:
[operations/dns@master] rename maintenance.discovery to mwmaint.discovery

https://gerrit.wikimedia.org/r/552944

Change 539633 merged by Dzahn:
[operations/puppet@production] mediawiki::maintenance: add envoy for TLS termination for noc.wm.org

https://gerrit.wikimedia.org/r/539633

Change 553199 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: use TLS to noc.wikimedia.org backend

https://gerrit.wikimedia.org/r/553199

Change 552947 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] otrs: add envoy for TLS termination behind ATS

https://gerrit.wikimedia.org/r/552947

Change 553424 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch OTRS to use TLS and discovery record

https://gerrit.wikimedia.org/r/553424

Change 552947 merged by Alexandros Kosiaris:
[operations/puppet@production] otrs: add envoy for TLS termination behind ATS

https://gerrit.wikimedia.org/r/552947

Change 554125 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add ticket.discovery.wmnet, point to mendelevium

https://gerrit.wikimedia.org/r/554125

Change 554125 merged by Dzahn:
[operations/dns@master] add ticket.discovery.wmnet, point to mendelevium

https://gerrit.wikimedia.org/r/554125

Change 554177 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ssl: add noc.wikimedia.org to mwmaint puppet TLS cert

https://gerrit.wikimedia.org/r/554177

Change 554177 merged by Dzahn:
[operations/puppet@production] ssl: add noc.wikimedia.org to mwmaint puppet TLS cert

https://gerrit.wikimedia.org/r/554177

Change 553199 merged by Dzahn:
[operations/puppet@production] ATS: use TLS to noc.wikimedia.org backend

https://gerrit.wikimedia.org/r/553199

Dzahn updated the task description. (Show Details)Dec 2 2019, 10:57 PM

https://noc.wikimedia.org has been switched to use https://mwmaint.discovery.wmnet (envoy on mwmaint1002).

Change 553424 merged by Alexandros Kosiaris:
[operations/puppet@production] ATS: switch OTRS to use TLS and discovery record

https://gerrit.wikimedia.org/r/553424

Dzahn updated the task description. (Show Details)Dec 4 2019, 5:15 AM

https://ticket.wikimedia.org (OTRS) has been switched to use https://ticket.discovery.wmnet (envoy on mendelevium).

PSA!

I've noticed that usages of envoyproxy for service TLS termination uses unencrypted private key files, but the cergen certificate manifests for these are configured with a password, meaning the key files cergen outputs will be encrypted.

If you need distributable unencrypted private key files, you can just omit specifying a key password in the certificate manifest.  E.g.

eventgate-analytics.discovery.wmnet:
  authority: puppet_ca
  expiry: null
  alt_names: ['eventgate-analytics.discovery.wmnet', 'eventgate-analytics.svc.codfw.wmnet', 'eventgate-analytics.svc.eqiad.wmnet']
  key:
    algorithm: ec

Notice there is no key.password.  Cergen will output the <name>.key.private.pem file unencrypted.

This should eliminate the extra step of manually generating an unencrypted file using openssl CLI.

Also, I see that the unencrypted key file is stored in the private repo at secrets/ssl/<name>.key, which is then distributed by the sslcert::certificate define.  I suggest we symlink this file to the cergen managed one to reduce duplication and manual steps if we need to change the key.

I'm going to do this for schema.discovery.wmnet now.

Ah hm, I also just realized the public cert is manually committed to public puppet in files/ssl.

Should we maybe just change sslcert::certificate to be smart(er) about where it gets its stuff? Sure the certificate can be public, but it is already in puppet private, so maybe we should just grab it from there instead of duplicating it?

Change 572378 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] doc: add envoy for TLS termination on doc1001

https://gerrit.wikimedia.org/r/572378

Change 572380 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add doc.discovery.wmnet for use in envoy config

https://gerrit.wikimedia.org/r/572380

Change 572381 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wmcs::monitoring: add envoy for TLS termination for grafana-labs

https://gerrit.wikimedia.org/r/572381

Change 572382 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backend URL to https for grafana-labs

https://gerrit.wikimedia.org/r/572382

Change 572385 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add grafana-labs.discovery.wmnet

https://gerrit.wikimedia.org/r/572385

Change 572387 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add graphite-labs.discovery.wmnet

https://gerrit.wikimedia.org/r/572387

Change 572391 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backend URL to https/discovery for graphite-labs

https://gerrit.wikimedia.org/r/572391

Dzahn updated the task description. (Show Details)Feb 15 2020, 3:25 AM

meanwhile there is another one in ATS backend.yaml.

added [ ] cloudweb2001-dev.wikimedia.org - http://labtesthorizon.wikimedia.org , http://labtestwikitech.wikimedia.org

Change 572937 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] add fake key for doc.discovery.wmnet

https://gerrit.wikimedia.org/r/572937

Change 572937 merged by Dzahn:
[labs/private@master] add fake key for doc.discovery.wmnet

https://gerrit.wikimedia.org/r/572937

Change 572380 merged by Dzahn:
[operations/dns@master] add doc.discovery.wmnet for use in envoy config

https://gerrit.wikimedia.org/r/572380

Change 572353 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] releases: remove port 80 firewall hole

https://gerrit.wikimedia.org/r/572353

Dzahn added a comment.Feb 19 2020, 6:47 PM

In this topic branch i am also switching monitoring of these services from HTTP to HTTPS:

https://gerrit.wikimedia.org/r/q/topic:%22icinga-http-https%22+(status:open%20OR%20status:merged)

Change 572385 merged by Dzahn:
[operations/dns@master] add grafana-labs.discovery.wmnet

https://gerrit.wikimedia.org/r/572385

Change 572387 merged by Dzahn:
[operations/dns@master] add graphite-labs.discovery.wmnet

https://gerrit.wikimedia.org/r/572387

Dzahn updated the task description. (Show Details)Mar 3 2020, 6:33 PM

grafana-labs-admin.wikimedia.org has been removed from DNS in https://gerrit.wikimedia.org/r/c/operations/dns/+/576408 therefore also removed here

Dzahn updated the task description. (Show Details)Mar 3 2020, 6:37 PM

labmon1001 has been replaced by cloudmetrics1002 and is still hosting grafana-labs and graphite-labs.

Change 576417 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] add fake key for grafana-labs.discovery.wmnet cert

https://gerrit.wikimedia.org/r/576417

Change 576417 merged by Dzahn:
[labs/private@master] add fake key for grafana-labs.discovery.wmnet cert

https://gerrit.wikimedia.org/r/576417

Change 572381 merged by Dzahn:
[operations/puppet@production] wmcs::monitoring: add envoy for TLS termination for grafana-labs

https://gerrit.wikimedia.org/r/572381

Change 576428 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ssl: add certificate for grafana-labs.discovery.wmnet

https://gerrit.wikimedia.org/r/576428

Change 576428 merged by Dzahn:
[operations/puppet@production] ssl: add certificate for grafana-labs.discovery.wmnet

https://gerrit.wikimedia.org/r/576428

Change 576434 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch grafana-labs backends from http to https

https://gerrit.wikimedia.org/r/576434

Change 576434 merged by Dzahn:
[operations/puppet@production] ATS: switch grafana-labs backends from http to https

https://gerrit.wikimedia.org/r/576434

Dzahn updated the task description. (Show Details)Mar 3 2020, 8:56 PM

grafana-labs and graphite-labs have switched to TLS now.

Change 572391 abandoned by Dzahn:
ATS: switch backend URL to https/discovery for graphite-labs

Reason:
duplicate of https://gerrit.wikimedia.org/r/c/operations/puppet/ /576434

https://gerrit.wikimedia.org/r/572391

Change 572382 abandoned by Dzahn:
ATS: switch backend URL to https for grafana-labs

Reason:
grafana-labs-admin does not exist anymore. done in https://gerrit.wikimedia.org/r/c/operations/puppet/ /576434

https://gerrit.wikimedia.org/r/572382

Change 572378 merged by Dzahn:
[operations/puppet@production] doc: add envoy for TLS termination on doc1001

https://gerrit.wikimedia.org/r/572378

Change 579360 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add certificate for envoy TLS termination on doc1001

https://gerrit.wikimedia.org/r/579360

Change 579360 merged by Dzahn:
[operations/puppet@production] add certificate for envoy TLS termination on doc1001

https://gerrit.wikimedia.org/r/579360

Dzahn updated the task description. (Show Details)Mar 12 2020, 6:51 PM
Dzahn updated the task description. (Show Details)

Change 579390 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ssl: update cert for doc.discovery.wmnet to include doc.wikimedia.org

https://gerrit.wikimedia.org/r/579390

Change 579390 merged by Dzahn:
[operations/puppet@production] ssl: update cert for doc.discovery.wmnet to include doc.wikimedia.org

https://gerrit.wikimedia.org/r/579390

Change 579407 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch doc.wikimedia.org to https to backend

https://gerrit.wikimedia.org/r/579407

Mentioned in SAL (#wikimedia-operations) [2020-03-12T21:47:28Z] <mutante> doc1001 - had to manually run "/usr/local/sbin/build-envoy-config -c /etc/envoy/" to get envoy tls_terminator_443 listener into the config or envoy would not listen on 443 (T210411)

Change 579407 merged by Dzahn:
[operations/puppet@production] ATS: switch doc.wikimedia.org to https to backend

https://gerrit.wikimedia.org/r/579407

Dzahn updated the task description. (Show Details)Mar 12 2020, 10:04 PM

Change 572353 merged by Dzahn:
[operations/puppet@production] releases: close port 80 for caching servers.

https://gerrit.wikimedia.org/r/572353

Mentioned in SAL (#wikimedia-operations) [2020-03-25T14:46:06Z] <mutante> closed port 80 for caching servers on misc backends https://gerrit.wikimedia.org/r/q/topic:%22applayer-tls%22+(status:open%20OR%20status:merged) as final step per service on T210411

Change 588980 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ci::master: add envoy for TLS termination for integration

https://gerrit.wikimedia.org/r/588980

Change 589285 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add contint.wikimedia.org service alias for contint machines

https://gerrit.wikimedia.org/r/589285

Change 589285 merged by Dzahn:
[operations/dns@master] add contint.wikimedia.org service alias for contint machines

https://gerrit.wikimedia.org/r/589285

Change 588980 merged by Dzahn:
[operations/puppet@production] ci::master: add envoy for TLS termination for integration

https://gerrit.wikimedia.org/r/588980

Change 589556 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add certificate for contint/integration.wikimedia.org

https://gerrit.wikimedia.org/r/589556

Change 589556 merged by Dzahn:
[operations/puppet@production] add certificate for contint/integration.wikimedia.org

https://gerrit.wikimedia.org/r/589556

Change 589565 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch contint backend to use TLS

https://gerrit.wikimedia.org/r/589565

Change 589565 merged by Dzahn:
[operations/puppet@production] ATS: switch contint backend to use TLS

https://gerrit.wikimedia.org/r/589565

Mentioned in SAL (#wikimedia-operations) [2020-04-21T11:06:12Z] <mutante> https://integration.wikimedia.org now also using TLS between ATS and contint1001 using envoy (T210411)

Change 591321 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ssl: add integration.mediawiki.org to contint cert

https://gerrit.wikimedia.org/r/591321

Change 591321 merged by Dzahn:
[operations/puppet@production] ssl: add integration.mediawiki.org to contint cert

https://gerrit.wikimedia.org/r/591321

Dzahn updated the task description. (Show Details)Apr 21 2020, 11:22 AM

Change 591325 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] Revert "ATS: switch contint backend to use TLS"

https://gerrit.wikimedia.org/r/591325

Change 591325 merged by Dzahn:
[operations/puppet@production] Revert "ATS: switch contint backend to use TLS"

https://gerrit.wikimedia.org/r/591325

Dzahn updated the task description. (Show Details)Apr 21 2020, 12:16 PM

The change for contint (integration.wikimedia.org, integration.mediawiki.org) had to be reverted because https://integration.wikimedia.org/ci/ returned 502s.

Change 597757 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: fix never_cache rule to apply to peopleweb discovery name

https://gerrit.wikimedia.org/r/597757

Change 597757 merged by Dzahn:
[operations/puppet@production] ATS: fix never_cache rule to apply to peopleweb discovery name

https://gerrit.wikimedia.org/r/597757

Change 605578 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: stop caching noc.wm.org responses

https://gerrit.wikimedia.org/r/605578

Change 605578 merged by Ema:
[operations/puppet@production] ATS: stop caching noc.wm.org responses

https://gerrit.wikimedia.org/r/605578

ema updated the task description. (Show Details)Jul 22 2020, 2:47 PM
ema changed the task status from Open to Stalled.Jul 22 2020, 2:49 PM

Change 615569 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ores: add envoy-proxy for TLS termination behind ATS

https://gerrit.wikimedia.org/r/615569

Change 615569 merged by Dzahn:
[operations/puppet@production] ores: add envoy-proxy for TLS termination behind ATS

https://gerrit.wikimedia.org/r/615569

Change 618367 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ssl: add TLS certificate for ORES

https://gerrit.wikimedia.org/r/618367

Change 618368 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] add fake key for ORES TLS cert

https://gerrit.wikimedia.org/r/618368

Change 618367 merged by Dzahn:
[operations/puppet@production] ssl: add TLS certificate for ORES

https://gerrit.wikimedia.org/r/618367

Change 618368 merged by Dzahn:
[labs/private@master] add fake key for ORES TLS cert

https://gerrit.wikimedia.org/r/618368

Change 618379 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch ORES to TLS to backends

https://gerrit.wikimedia.org/r/618379

jijiki moved this task from Incoming 🐫 to Unsorted on the serviceops board.Aug 17 2020, 11:48 PM