Page MenuHomePhabricator

Configure LVS endpoints for new elasticsearch clusters
Closed, ResolvedPublic

Description

With multiple elasticsearch instances per server and multiple clusters spreading to subsets of elasticsearch nodes, we need to have dedicated LVS endpoints to expose the different clusters. For details about the context, see T193654.

There are some related questions about how to route requests to those different clusters (see T198352#4671179 as well):

  1. differentiate on TCP port only (first instance on 9243, second on 9443)
  2. differentiate on server names
  3. differentiate on IP
  4. a combination of some of the above
  1. seems the simpler solution, it matches the expectations of the clients and does not have any significant drawback that we could find
  2. SAN / SNI support in HTTP libraries is often broken if supported at all. Our current clients might be OK (unchecked), but if we can avoid the pain, we should
  3. it would work fine, but since we want (at least at some point) to have elasticsearch listening only on localhost, elasticsearch will be on different ports already. Exposing this mapping to the TLS endpoint as well seems simpler and less surprising. (yes, we could use different lo:aliases, but that seems even more confusing for not much gain).
  4. why not ?

It looks like 1) is the simplest solution, but feedback is welcomed.

Details

Related Gerrit Patches:

Event Timeline

Gehel triaged this task as Normal priority.Oct 16 2018, 5:11 PM
Gehel created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 16 2018, 5:11 PM
Gehel added a comment.Oct 19 2018, 1:36 PM

After some discussion (P7699), the idea is:

  • each elasticsearch instance listen on a different local port
  • nginx is used for SSL termination, exposing the different instances on different ports
  • LVS endpoint are created for each instance, routing to the correct subset of nodes, on the correct port
  • we keep using a single fqdn (search.svc.{eqiad|codfw}.wmnet) for all instances
EBjune moved this task from needs triage to Up Next on the Discovery-Search board.Oct 25 2018, 5:20 PM
EBjune moved this task from Up Next to This Quarter on the Discovery-Search board.

Change 475753 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: configure LVS endpoint for new codfw clusters

https://gerrit.wikimedia.org/r/475753

Change 475753 merged by Giuseppe Lavagetto:
[operations/puppet@production] elasticsearch: configure LVS endpoint for new codfw clusters

https://gerrit.wikimedia.org/r/475753

Change 479184 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/puppet@production] elasticsearch: configure LVS endpoint for new eqiad clusters

https://gerrit.wikimedia.org/r/479184

Change 479184 merged by Filippo Giunchedi:
[operations/puppet@production] elasticsearch: configure LVS endpoint for new eqiad clusters

https://gerrit.wikimedia.org/r/479184

debt closed this task as Resolved.Dec 13 2018, 6:02 PM