Page MenuHomePhabricator
Create Task
Maniphest T247389

Use envoy for TLS termination on the appservers
Closed, ResolvedPublic
Actions

Description

Given we're using envoy as a service proxy, we should start using it also to proxy TLS requests to the appservers.

Things to look out for:

  • We would like to have a timeout of 201 seconds, like we had in nginx
  • There is no way to set a maximum number of keepalive requests in envoy, which could represent a problem during the transition.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Joe created this task.Mar 11 2020, 8:19 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMar 11 2020, 8:19 AM
Joe triaged this task as Medium priority.Mar 26 2020, 9:19 AM
gerritbot added a comment.Mar 26 2020, 9:19 AM

Change 583563 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: move debug servers to use envoy for TLS termination

https://gerrit.wikimedia.org/r/583563

gerritbot added a project: Patch-For-Review.Mar 26 2020, 9:19 AM
gerritbot added a comment.Mar 26 2020, 9:35 AM

Change 583563 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki: move debug servers to use envoy for TLS termination

https://gerrit.wikimedia.org/r/583563

gerritbot added a comment.Mar 31 2020, 10:34 AM

Change 584904 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: switch TLS termination to envoy on appserver canaries

https://gerrit.wikimedia.org/r/584904

gerritbot added a comment.Mar 31 2020, 12:32 PM

Change 584904 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki: switch TLS termination to envoy on appserver canaries

https://gerrit.wikimedia.org/r/584904

gerritbot added a comment.Mar 31 2020, 1:07 PM

Change 584921 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: convert API canaries to use envoy

https://gerrit.wikimedia.org/r/584921

gerritbot added a comment.Mar 31 2020, 1:09 PM

Change 584921 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki: convert API canaries to use envoy

https://gerrit.wikimedia.org/r/584921

Joe added a comment.EditedMar 31 2020, 1:46 PM

Both appserver and api canaries now use envoy for TLS termination.

gerritbot added a comment.Apr 6 2020, 6:23 AM

Change 586204 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: convert all API servers to use envoy for TLS termination.

https://gerrit.wikimedia.org/r/586204

gerritbot added a comment.Apr 6 2020, 6:23 AM

Change 586205 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: convert all appserver to use envoy for TLS termination

https://gerrit.wikimedia.org/r/586205

gerritbot added a comment.Apr 6 2020, 6:32 AM

Change 586204 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki: convert all API servers to use envoy for TLS termination.

https://gerrit.wikimedia.org/r/586204

gerritbot added a comment.Apr 6 2020, 2:05 PM

Change 586354 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] parsoid: switch to envoy for TLS termination

https://gerrit.wikimedia.org/r/586354

gerritbot added a comment.Apr 6 2020, 4:18 PM

Change 586354 merged by Giuseppe Lavagetto:
[operations/puppet@production] parsoid: switch to envoy for TLS termination

https://gerrit.wikimedia.org/r/586354

cscott subscribed.Apr 6 2020, 7:14 PM

This might have broken officewiki: T249535

gerritbot added a comment.Apr 8 2020, 7:45 AM

Change 587490 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] parsoid: switch to envoy, take 2

https://gerrit.wikimedia.org/r/587490

gerritbot added a comment.Apr 8 2020, 7:57 AM

Change 587490 merged by Giuseppe Lavagetto:
[operations/puppet@production] parsoid: switch to envoy, take 2

https://gerrit.wikimedia.org/r/587490

gerritbot added a comment.May 19 2020, 9:49 AM

Change 597242 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] appservers: switch to envoy in all of codfw

https://gerrit.wikimedia.org/r/597242

gerritbot added a comment.May 19 2020, 9:49 AM

Change 597243 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] appservers: convert mw1265-1275 to use envoy for TLS termination

https://gerrit.wikimedia.org/r/597243

gerritbot added a comment.May 19 2020, 9:49 AM

Change 597244 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] appserver: use envoy everywhere

https://gerrit.wikimedia.org/r/597244

gerritbot added a comment.May 19 2020, 1:15 PM

Change 597242 merged by Giuseppe Lavagetto:
[operations/puppet@production] appservers: switch to envoy in all of codfw

https://gerrit.wikimedia.org/r/597242

gerritbot added a comment.May 20 2020, 8:45 AM

Change 597243 merged by Giuseppe Lavagetto:
[operations/puppet@production] appservers: convert mw1265-1275 to use envoy for TLS termination

https://gerrit.wikimedia.org/r/597243

Stashbot added a comment.May 20 2020, 8:49 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-20T08:49:33Z] <_joe_> converting mw1266-1275 to use envoy T247389

Joe added a comment.May 20 2020, 9:29 AM

Status update: we've deployed envoy on all mediawiki servers with the exception of:

  • jobrunners (where we still have to reproduce what nginx was doing)
  • all servers in the appserver cluster in eqiad with a sequence number above mw1275.

I will wait monday before completing the appserver pool.

gerritbot added a comment.May 20 2020, 11:25 AM

Change 597513 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] jobrunner: add code to switch to envoy, switch codfw

https://gerrit.wikimedia.org/r/597513

gerritbot added a comment.May 21 2020, 7:57 AM

Change 597513 merged by Giuseppe Lavagetto:
[operations/puppet@production] jobrunner: add code to switch to envoy, switch codfw

https://gerrit.wikimedia.org/r/597513

gerritbot added a comment.May 25 2020, 8:50 AM

Change 598418 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] jobrunner: switch mw1337 to envoy

https://gerrit.wikimedia.org/r/598418

gerritbot added a comment.May 25 2020, 9:07 AM

Change 598418 merged by Giuseppe Lavagetto:
[operations/puppet@production] jobrunner: switch mw1337 to envoy

https://gerrit.wikimedia.org/r/598418

Stashbot added a comment.May 25 2020, 9:17 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-25T09:17:16Z] <_joe_> migrated mw1337 to use envoy for TLS termination T247389

gerritbot added a comment.May 25 2020, 9:39 AM

Change 597244 merged by Giuseppe Lavagetto:
[operations/puppet@production] appserver: use envoy everywhere

https://gerrit.wikimedia.org/r/597244

Joe added a comment.May 25 2020, 1:07 PM

As of today, all appservers use envoy too.

gerritbot added a comment.May 26 2020, 8:45 AM

Change 598689 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] jobrunner: switch all to envoy for TLS termination

https://gerrit.wikimedia.org/r/598689

gerritbot added a comment.May 26 2020, 8:51 AM

Change 598689 merged by Giuseppe Lavagetto:
[operations/puppet@production] jobrunner: switch all to envoy for TLS termination

https://gerrit.wikimedia.org/r/598689

gerritbot added a comment.May 26 2020, 12:34 PM

Change 586205 abandoned by Giuseppe Lavagetto:
mediawiki: convert all appserver to use envoy for TLS termination

https://gerrit.wikimedia.org/r/586205

Joe closed this task as Resolved.May 28 2020, 6:47 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits