Page MenuHomePhabricator
Create Task
Maniphest T224324

LB for cloudelastic
Closed, ResolvedPublic
Actions

Description

cloudelastic refers to cloudelastic100[1234].wikimedia.org.

There are two separate streams of requests that cloudelastic cares about:

prod mediawiki -> cloudelastic:9[246]43. https, tls terminated by nginx. Receives production document updates

internet -> cloudelastic:8[246]43. tls terminated by nginx. Only accepts GET requests. Receives search queries. Limited by ferm to wmf cloud ip ranges not for privacy, but for service stability.

Ideally both of these should have a load balancer (likely LVS) in front of them so nodes can be cleanly added/removed/etc. without any clients knowing or caring.

Details

SubjectRepoBranchLines +/-
cloudelastic: Fix LVS IPv6 addressoperations/puppetproduction+1 -1
cloudelastic: Fix LVS IPv6 addressoperations/dnsmaster+2 -2
LVS for cloudelasticoperations/puppetproduction+198 -4
Add cloudelastic LVS to DNSoperations/dnsmaster+6 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 DeclinedFeatureNoneT71489 Expose mwgrep functionality on-wiki
 ResolvedNoneT109715 Replicate production elasticsearch indices to labs
 ResolvedEBernhardsonT220625 Initialize CirrusSearch on cloudelastic
 ResolveddebtT224324 LB for cloudelastic

Event Timeline

EBernhardson created this task.May 24 2019, 11:11 PM
Restricted Application edited projects, added Discovery-Search; removed Discovery-Search (Current work). · View Herald TranscriptMay 24 2019, 11:11 PM
Krenair subscribed.May 24 2019, 11:15 PM
bd808 mentioned this in T223902: cloudcontrol: decide on FQDN for service endpoints.May 24 2019, 11:16 PM
EBernhardson moved this task from needs triage to Current work on the Discovery-Search board.May 28 2019, 3:56 PM
EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.
gerritbot added a comment.May 28 2019, 4:23 PM

Change 512924 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/dns@master] Add cloudelastic LVS to DNS

https://gerrit.wikimedia.org/r/512924

gerritbot added a project: Patch-For-Review.May 28 2019, 4:23 PM

Change 512925 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] LVS for cloudelastic

https://gerrit.wikimedia.org/r/512925

EBernhardson mentioned this in T224358: 20c of thoughts for globalsearch tool.May 28 2019, 4:36 PM
MusikAnimal subscribed.May 28 2019, 5:09 PM
EBernhardson moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.May 28 2019, 5:18 PM
debt subscribed.Jun 11 2019, 5:26 PM

Will bring up in Scrums of Scrums :)

EBernhardson added a subscriber: Andrew.Jul 22 2019, 8:54 PM

@Andrew Talked to @BBlack today and he said you two had talked about taking a different direction in terms of networking to expose bare metal services to cloud. The situation today is:

  • cloudelastic100[1-4] all have public ip addresses. The access provided by the service is acceptable for full public release, but for operational reasons we are limiting via ferm to only cloud ip ranges.
  • Production mediawiki job runners need to write to this service. Because we don't have a LB to handle pool/depool operations we are currently only sending writes for group0. Once an LB is in place we will be sending writes from all wikis.
  • The servers each run 3 services on 6 ports. 3 ports accept read and write requests and are only accessible from prod hosts. The other 3 ports only accept read requests and are publicly accessible.
  • The plan initially was to expose these 6 ports via LVS on cloudelastic.wikimedia.org, in the attached patch.

Would this fit into the plans Brandon mentioned for some sort of haproxy and cloud specific domain names and address spaces? In particular possibly our need to push writes from production job runners makes this different from other services you've been considering (or maybe not, mysql is similar). If so, what kind of timeline is this on?

Andrew added a subscriber: • JHedden.Jul 23 2019, 4:45 PM

@EBernhardson Jason is in the process of building out an HA solution for our internal services, so I'm cc'ing him on this task. I think he'll have a lot more to contribute than I have.

EBernhardson moved this task from Needs review to Waiting on the Discovery-Search (Current work) board.Jul 23 2019, 5:20 PM
• JHedden added a comment.Jul 24 2019, 2:07 PM

@EBernhardson Unfortunately we don't have a good solution for this today or in the near future. We've discussed future load balancing as a service options, but this requires a lot of effort on backend upgrades, automation and configuration.

The HA architecture that we discussed with @BBlack was focused on the backend OpenStack services. We've decided to not go that route and opt for a less complex solution that is dedicated to the OpenStack controllers.

EBernhardson moved this task from Waiting to Incoming on the Discovery-Search (Current work) board.Jul 30 2019, 5:21 PM
gerritbot added a comment.Jul 31 2019, 4:30 PM

Change 512924 merged by BBlack:
[operations/dns@master] Add cloudelastic LVS to DNS

https://gerrit.wikimedia.org/r/512924

Stashbot added a comment.Aug 1 2019, 5:42 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-01T17:42:45Z] <bblack> disable puppet on lvs1014 + lvs1016 for cloudelastic LVS merge - T224324

gerritbot added a comment.Aug 1 2019, 5:44 PM

Change 512925 merged by BBlack:
[operations/puppet@production] LVS for cloudelastic

https://gerrit.wikimedia.org/r/512925

Maintenance_bot removed a project: Patch-For-Review.Aug 1 2019, 6:10 PM
Stashbot added a comment.Aug 1 2019, 6:30 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-01T18:30:01Z] <bblack> lvs1016: puppet re-enabled, pybal restarted, cloudelastic deploy - T224324

Stashbot added a comment.Aug 1 2019, 7:34 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-01T19:34:00Z] <bblack> lvs1014 - puppetize and restart pybal for cloudelastic LVS - T224324

Stashbot added a comment.Aug 1 2019, 7:57 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-01T19:57:32Z] <bblack> lvs1016 - restart pybal for slight LVS config change for cloudelastic - T224324

gerritbot added a comment.Aug 5 2019, 6:41 PM

Change 528215 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] cloudelastic: Fix LVS IPv6 address

https://gerrit.wikimedia.org/r/528215

gerritbot added a project: Patch-For-Review.Aug 5 2019, 6:41 PM

Change 528216 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] cloudelastic: Fix LVS IPv6 address

https://gerrit.wikimedia.org/r/528216

gerritbot added a comment.Aug 5 2019, 7:31 PM

Change 528216 merged by BBlack:
[operations/dns@master] cloudelastic: Fix LVS IPv6 address

https://gerrit.wikimedia.org/r/528216

gerritbot added a comment.Aug 5 2019, 7:32 PM

Change 528215 merged by BBlack:
[operations/puppet@production] cloudelastic: Fix LVS IPv6 address

https://gerrit.wikimedia.org/r/528215

Maintenance_bot removed a project: Patch-For-Review.Aug 5 2019, 8:10 PM
EBernhardson moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Aug 6 2019, 3:53 PM
debt closed this task as Resolved.Aug 9 2019, 5:09 PM
debt claimed this task.
gerritbot added a comment.Sep 28 2021, 10:33 PM

Change 724520 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/dns@master] wcqs: add discovery record

https://gerrit.wikimedia.org/r/724520

gerritbot added a project: Patch-For-Review.Sep 28 2021, 10:33 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits