Page MenuHomePhabricator
Create Task
Maniphest T368950

Consider migrating our Elastic TLS termination from nginx to envoy
Closed, DeclinedPublic
Actions

Description

Our Elastic Puppet roles are the only set of roles besides maps that still use nginx to terminate TLS, as illustrated by this CR . As seen in T360439 and other tasks, this is slowly building up technical debt as almost no one uses the same Puppet code for TLS termination. Moving to envoy as TLS terminator will align us with the rest of WMF.

Creating this ticket to:

  • Discuss proposed changes with stakeholders
  • Assess our current nginx config and see what can be migrated to envoy (besides TLS, there could be headers inserted, rewrites etc)
  • Migrate our TLS termination to envoy.
  • If possible, completely remove nginx in favor of envoy.

Details

Other Assignee
RKemper
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
relforge: remove non-functional TLS termination changesoperations/puppetproduction+12 -34
relforge: Attempt to use envoyproxy instead of nginx for TLSoperations/puppetproduction+3 -8
relforge: test envoyproxyoperations/puppetproduction+27 -20
relforge: test envoyproxyoperations/puppetproduction+15 -32
elastic: test envoy TLS terminator in relforgeoperations/puppetproduction+33 -1
Customize query in gerrit

Related Objects

Event Timeline

bking created this task.Jul 1 2024, 4:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 1 2024, 4:53 PM
Gehel triaged this task as Medium priority.Jul 1 2024, 6:31 PM
Gehel moved this task from Incoming to Toil / Automation on the Data-Platform-SRE board.
Gehel moved this task from needs triage to Ops / SRE on the Discovery-Search board.
bking added a subscriber: Gehel.EditedJul 8 2024, 8:40 PM

I checked out the current Elastic nginx config and as @Gehel said last week, there is nothing elastic-specific in said nginx config: nginx is purely terminating TLS.

As such, it should be possible to treat the envoy TLS terminator (and its puppet role) as a drop-in replacement. We'll start writing the puppet code and testing this out in relforge.

gerritbot added a comment.Jul 8 2024, 9:51 PM

Change #1052819 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: test envoy TLS terminator in relforge

https://gerrit.wikimedia.org/r/1052819

gerritbot added a project: Patch-For-Review.Jul 8 2024, 9:51 PM
Gehel merged a task: T352872: Consider implementing Envoy for Elastic hosts.Jul 9 2024, 12:57 PM
Gehel added a subscriber: pfischer.
bking changed the task status from Open to In Progress.Jul 9 2024, 12:58 PM
bking claimed this task.
bking lowered the priority of this task from Medium to Low.
bking updated Other Assignee, added: RKemper.
bking edited projects, added: Data-Platform-SRE (2024.07.08 - 2024.07.28); removed: Data-Platform-SRE.
bking raised the priority of this task from Low to Medium.Jul 9 2024, 1:04 PM
bking updated the task description. (Show Details)
gerritbot added a comment.Jul 9 2024, 1:29 PM

Change #1052819 merged by Bking:

[operations/puppet@production] elastic: test envoy TLS terminator in relforge

https://gerrit.wikimedia.org/r/1052819

bking moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.07.08 - 2024.07.28) board.Jul 9 2024, 3:37 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 9 2024, 3:47 PM
bking updated the task description. (Show Details)Jul 9 2024, 5:01 PM
Stashbot added a comment.Jul 9 2024, 5:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-09T17:03:40Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on relforge[1003-1004].eqiad.wmnet with reason: T368950

Stashbot added a comment.Jul 9 2024, 5:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-09T17:03:57Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on relforge[1003-1004].eqiad.wmnet with reason: T368950

gerritbot added a comment.Jul 9 2024, 7:07 PM

Change #1053041 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] relforge: test envoyproxy

https://gerrit.wikimedia.org/r/1053041

gerritbot added a project: Patch-For-Review.Jul 9 2024, 7:07 PM
Stashbot added a comment.Jul 11 2024, 1:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-11T13:03:50Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 6 days, 0:00:00 on relforge[1003-1004].eqiad.wmnet with reason: T368950

Stashbot added a comment.Jul 11 2024, 1:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-11T13:04:07Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on relforge[1003-1004].eqiad.wmnet with reason: T368950

bking added a comment.Jul 11 2024, 1:20 PM

Per conversation with @EBernhardson , we need to verify that Envoy doesn't timeout or reject large payloads, as data sent to the Elastic bulk API can be around ~100MB in a single request.

gerritbot added a comment.Jul 11 2024, 5:38 PM

Change #1053041 merged by Bking:

[operations/puppet@production] relforge: test envoyproxy

https://gerrit.wikimedia.org/r/1053041

gerritbot added a comment.Jul 11 2024, 5:51 PM

Change #1053751 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] relforge: test envoyproxy with multiple instances

https://gerrit.wikimedia.org/r/1053751

gerritbot added a comment.Jul 11 2024, 8:20 PM

Change #1053751 merged by Bking:

[operations/puppet@production] relforge: test envoyproxy

https://gerrit.wikimedia.org/r/1053751

Maintenance_bot removed a project: Patch-For-Review.Jul 11 2024, 8:30 PM
gerritbot added a comment.Jul 11 2024, 9:14 PM

Change #1053789 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] relforge: Attempt to use envoyproxy instead of nginx for TLS

https://gerrit.wikimedia.org/r/1053789

gerritbot added a project: Patch-For-Review.Jul 11 2024, 9:14 PM
gerritbot added a comment.Jul 16 2024, 2:18 PM

Change #1054578 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] relforge: remove non-functional TLS termination changes

https://gerrit.wikimedia.org/r/1054578

gerritbot added a comment.Jul 16 2024, 2:47 PM

Change #1054578 merged by Bking:

[operations/puppet@production] relforge: remove non-functional TLS termination changes

https://gerrit.wikimedia.org/r/1054578

bking added a comment.EditedJul 22 2024, 9:29 PM

Upon further review, I think this is a tough climb for a couple of reasons:

  • the defined type envoyproxy::tls_terminator doesn't generate valid envoy config when using multiple Elastic clusters on the same hosts (at least, I failed with multiple different approaches, see CRs attached to this ticket). If we decided to use Envoy, we'd likely have to start with envoy.pp and work from there.
  • nginx is used to enforce read-only access for Cloudelastic via the nginx tlsproxy module, so we'd have to find/implement something similar in Envoy.

Despite this, I'm going to book some time with @Gehel to discuss this before we completely give up on the idea.

bking closed this task as Declined.Jul 23 2024, 2:14 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2024.07.08 - 2024.07.28) board.

As we're already planning on migrating to Opensearch , and Opensearch supports TLS connections natively , I think we should forget about this migration for now and address it as part of the Opensearch migration.

Gehel edited projects, added: Discovery-Search (Current work); removed: Discovery-Search.Feb 11 2025, 3:50 PM
Gehel moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.
bking mentioned this in T386983: Explore migrating wdqs' nginx config into envoy.Feb 20 2025, 10:53 PM
bking mentioned this in T422860: Migrate Cloudelastic to OpenSearch 2.x.Apr 10 2026, 9:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits