Page MenuHomePhabricator
Create Task
Maniphest T351475

Reduce impact of Elastic snapshots
Closed, ResolvedPublic
Actions

Description

Per today's Wikimedia-Search IRC conversation with @BCornwall , we inadvertently triggered an LVS alert for high network bandwidth by running an Elastic snapshot/restore (See Grafana). Creating this ticket to:

  • Investigate Discuss ways to reduce strain on external services (LVS, Swift) during snapshot operations with owning teams.
  • Implement changes, if possible necessary.

Updated task description above to reflect IRC discussions re: alert detuning (Traffic/Infrastructure Foundations) vs. outbound traffic rate-limiting from ES hosts (Data Platform SRE).

Related Objects

Event Timeline

bking created this task.Nov 16 2023, 10:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2023, 10:17 PM
bking updated the task description. (Show Details)Nov 16 2023, 10:18 PM
bking added a subscriber: EBernhardson.Nov 16 2023, 10:22 PM
bking moved this task from Incoming to Blocked / Waiting on the Data-Platform-SRE board.Nov 17 2023, 4:07 PM
bking added a subscriber: CDanis.

Per today's IRC discussion in the security channel, @CDanis mentioned detuning or removing the LVS alerts for internal hosts. So I'll set this one to blocked at the moment. Chris and/or Brett, let us know what your teams decide.

bking renamed this task from Reduce network impact of Elastic snapshots to Reduce impact of Elastic snapshots.Nov 17 2023, 4:09 PM
bking updated the task description. (Show Details)
bking updated the task description. (Show Details)Nov 17 2023, 6:40 PM
Gehel removed a project: Discovery-Search.Nov 20 2023, 3:12 PM
BCornwall added a subscriber: BBlack.Nov 20 2023, 8:05 PM
bking added a subscriber: MatthewVernon.Nov 27 2023, 6:00 PM

Just wanted to add that Envoy is deployed for Swift frontends per today's SRE meeting.

That being said, we (Search Platform/Data Platform SRE) would prefer not to implement Envoy across our Elastic fleet without confirmation from Traffic and/or Data Persistence that this internal LVS bandwidth alert actually represents a problem. Tagging @MatthewVernon for awareness.

MatthewVernon added a comment.Nov 28 2023, 10:23 AM

I don't expect the change to make difference to how anyone is using swift - moving from nginx to envoy for TLS termination was more about bringing swift more up-to-date in terms of TLS termination, and getting better observability and reliability.

Gehel edited projects, added Data-Platform-SRE (2023.12.01 - 2023.12.31); removed Data-Platform-SRE.Dec 6 2023, 9:23 AM
Gehel moved this task from Backlog to Blocked / Waiting on the Data-Platform-SRE (2023.12.01 - 2023.12.31) board.Dec 6 2023, 9:25 AM
bking added a comment.Dec 6 2023, 2:49 PM

Update: Traffic team merged a patch that makes these LVS high RX alerts non-paging . Thus, I believe we don't be inadvertently paging on-call SREs every time we take a snapshot. As such, I'm closing out this ticket.

Please feel free to reopen if Search Platform/Data Platform SRE does need to take further action.

bking closed this task as Resolved.Dec 6 2023, 2:50 PM
bking claimed this task.
bking moved this task from Blocked / Waiting to Done on the Data-Platform-SRE (2023.12.01 - 2023.12.31) board.
bking mentioned this in T352872: Consider implementing Envoy for Elastic hosts.Dec 6 2023, 3:03 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL