Page MenuHomePhabricator
Create Task
Maniphest T303036

Incident: 2022-03-4 Banner sampling leading to a relatively wide site outage (mostly esams)
Closed, ResolvedPublic
Actions

Description

The setup of a particular banner for all wikis/all users with 100% sampling caused traffic layer instabilities due to the large amount of traffic received- causing wikis to be unreachable in the impacted region (initially esams datacenter users - mostly Europe, Africa and middle East), with some temporary issues on other datacenters (eqiad). The reasons for the issues was connections piled up all the way up to the applayer (eventgate-analytics-external), and varnish wasn't able to handle it.

This ticket has been created to track the followups and make sure the post mortem is documented on wikitech in the usual places and the incident is scored appropriately.

Related Objects
Search...

Event Timeline

• jcrespo created this task.Mar 4 2022, 11:16 AM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptMar 4 2022, 11:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore subscribed.Mar 4 2022, 11:22 AM
• jcrespo mentioned this in T303035: ReferenceError: xpatrolAddBannerLink is not defined.Mar 4 2022, 11:24 AM
LSobanski subscribed.Mar 4 2022, 12:56 PM
• toan subscribed.Mar 4 2022, 1:49 PM
Peachey88 subscribed.Mar 4 2022, 10:05 PM
RhinosF1 subscribed.Mar 5 2022, 8:26 AM
Ladsgroup added a subtask: Restricted Task.Mar 7 2022, 9:46 AM
BTullis subscribed.Mar 7 2022, 9:47 AM
Krinkle subscribed.Mar 7 2022, 11:54 AM
odimitrijevic moved this task from Incoming (new tickets) to Apache Iceberg Migration on the Data-Engineering board.Mar 10 2022, 6:16 PM
JMeybohm closed subtask T303042: eventgate-* tls telemetry is disabled as Resolved.Mar 10 2022, 7:11 PM
jbond triaged this task as Medium priority.Mar 21 2022, 11:35 AM
Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 21 2022, 4:29 PM
Krinkle updated the task description. (Show Details)
lmata renamed this task from Banner sampling leading to a relatively wide site outage (mostly esams) to Incident: Banner sampling leading to a relatively wide site outage (mostly esams).Apr 28 2022, 10:20 PM
lmata renamed this task from Incident: Banner sampling leading to a relatively wide site outage (mostly esams) to Incident: 2022-03-4 Banner sampling leading to a relatively wide site outage (mostly esams).Apr 28 2022, 10:28 PM
lmata moved this task from Backlog to Scorecard Done on the SRE-OnFire (FY2021/2022-Q3) board.Apr 29 2022, 1:00 AM
Ottomata subscribed.Aug 11 2022, 5:57 PM

Are there actionables on this task? I'm considering removing the Event Platform tag?

• jcrespo added a comment.EditedAug 11 2022, 6:18 PM

@Ottomata: The actionables of the task pending is to understand what the actionables are for the incident mentioned on the header (https://docs.google.com/document/d/1xYYzFlJcAP9pckqBWyiXUbs7HN5iThg85lkjv_RUh_o/edit) as that is pending documenting: https://wikitech.wikimedia.org/wiki/Incidents/2022-03-04_esams_availability_banner_sampling . After that, this task can be closed (no need to wait for the actionables to be closed, only documented).

I believe there is T303155 already, but unsure if there are others. Do you know which teams or individual people could help us with that? I believe it wouldn't be DE, but as your layer's availability was affected, maybe your team knows or could help routing the task?

Marostegui added subscribers: lmata, Marostegui.Nov 28 2022, 9:04 AM

@lmata what should we do with this follow up task?

BTullis added a comment.Nov 28 2022, 9:40 AM

I'm not sure that there's much more to do, is there? From a technical perspective, this has now been prevented from happening again accidentially by {T303326}, so we can no longer configure sampling from CN at rate of more than 1%

Do we just need to update https://wikitech.wikimedia.org/wiki/Incidents/2022-03-04_esams_availability_banner_sampling#Actionables and add a link to T303326, before closing this ticket and T303155?

Apologies if I'm missing something about the process.

lmata closed this task as Resolved.Nov 28 2022, 12:35 PM
lmata claimed this task.

Thank you @BTullis for T303036#8423773. I think that note should be enough documentation to close this task; however, if you could tie the update back to the incident report, that would prove most helpful. I'll be resolving this task for completion.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits