Page MenuHomePhabricator

Use envoy for TLS termination on the appservers
Closed, ResolvedPublic

Description

Given we're using envoy as a service proxy, we should start using it also to proxy TLS requests to the appservers.

Things to look out for:

  • We would like to have a timeout of 201 seconds, like we had in nginx
  • There is no way to set a maximum number of keepalive requests in envoy, which could represent a problem during the transition.

Event Timeline

Joe triaged this task as Medium priority.Mar 26 2020, 9:19 AM

Change 583563 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: move debug servers to use envoy for TLS termination

https://gerrit.wikimedia.org/r/583563

Change 583563 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki: move debug servers to use envoy for TLS termination

https://gerrit.wikimedia.org/r/583563

Change 584904 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: switch TLS termination to envoy on appserver canaries

https://gerrit.wikimedia.org/r/584904

Change 584904 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki: switch TLS termination to envoy on appserver canaries

https://gerrit.wikimedia.org/r/584904

Change 584921 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: convert API canaries to use envoy

https://gerrit.wikimedia.org/r/584921

Change 584921 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki: convert API canaries to use envoy

https://gerrit.wikimedia.org/r/584921

Both appserver and api canaries now use envoy for TLS termination.

Change 586204 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: convert all API servers to use envoy for TLS termination.

https://gerrit.wikimedia.org/r/586204

Change 586205 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: convert all appserver to use envoy for TLS termination

https://gerrit.wikimedia.org/r/586205

Change 586204 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki: convert all API servers to use envoy for TLS termination.

https://gerrit.wikimedia.org/r/586204

Change 586354 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] parsoid: switch to envoy for TLS termination

https://gerrit.wikimedia.org/r/586354

Change 586354 merged by Giuseppe Lavagetto:
[operations/puppet@production] parsoid: switch to envoy for TLS termination

https://gerrit.wikimedia.org/r/586354

This might have broken officewiki: T249535

Change 587490 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] parsoid: switch to envoy, take 2

https://gerrit.wikimedia.org/r/587490

Change 587490 merged by Giuseppe Lavagetto:
[operations/puppet@production] parsoid: switch to envoy, take 2

https://gerrit.wikimedia.org/r/587490

Change 597242 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] appservers: switch to envoy in all of codfw

https://gerrit.wikimedia.org/r/597242

Change 597243 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] appservers: convert mw1265-1275 to use envoy for TLS termination

https://gerrit.wikimedia.org/r/597243

Change 597244 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] appserver: use envoy everywhere

https://gerrit.wikimedia.org/r/597244

Change 597242 merged by Giuseppe Lavagetto:
[operations/puppet@production] appservers: switch to envoy in all of codfw

https://gerrit.wikimedia.org/r/597242

Change 597243 merged by Giuseppe Lavagetto:
[operations/puppet@production] appservers: convert mw1265-1275 to use envoy for TLS termination

https://gerrit.wikimedia.org/r/597243

Mentioned in SAL (#wikimedia-operations) [2020-05-20T08:49:33Z] <_joe_> converting mw1266-1275 to use envoy T247389

Status update: we've deployed envoy on all mediawiki servers with the exception of:

  • jobrunners (where we still have to reproduce what nginx was doing)
  • all servers in the appserver cluster in eqiad with a sequence number above mw1275.

I will wait monday before completing the appserver pool.

Change 597513 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] jobrunner: add code to switch to envoy, switch codfw

https://gerrit.wikimedia.org/r/597513

Change 597513 merged by Giuseppe Lavagetto:
[operations/puppet@production] jobrunner: add code to switch to envoy, switch codfw

https://gerrit.wikimedia.org/r/597513

Change 598418 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] jobrunner: switch mw1337 to envoy

https://gerrit.wikimedia.org/r/598418

Change 598418 merged by Giuseppe Lavagetto:
[operations/puppet@production] jobrunner: switch mw1337 to envoy

https://gerrit.wikimedia.org/r/598418

Mentioned in SAL (#wikimedia-operations) [2020-05-25T09:17:16Z] <_joe_> migrated mw1337 to use envoy for TLS termination T247389

Change 597244 merged by Giuseppe Lavagetto:
[operations/puppet@production] appserver: use envoy everywhere

https://gerrit.wikimedia.org/r/597244

As of today, all appservers use envoy too.

Change 598689 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] jobrunner: switch all to envoy for TLS termination

https://gerrit.wikimedia.org/r/598689

Change 598689 merged by Giuseppe Lavagetto:
[operations/puppet@production] jobrunner: switch all to envoy for TLS termination

https://gerrit.wikimedia.org/r/598689

Change 586205 abandoned by Giuseppe Lavagetto:
mediawiki: convert all appserver to use envoy for TLS termination

https://gerrit.wikimedia.org/r/586205