Enabling api-request eventgate to group1 caused minor service disruptions
Closed, ResolvedPublic
Actions

Description

Influx of service errors

Mar 13: 19:12-19:14 UTC
Mar 13: 20:12-20:15 UTC

https://etherpad.wikimedia.org/p/incident13Mar2019

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	eventgate-analytics - Fix liveness and readiness probes	operations/deployment-charts	master	+55 -46
	Disable api-request eventgate logging for group0 wikis	operations/mediawiki-config	master	+0 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Ottomata	T218255 Enabling api-request eventgate to group1 caused minor service disruptions
Resolved	• Pchelolo	T218254 EventBus extension should never log unserialized events
Resolved	• Pchelolo	T218260 Decrease timeout for EventBus extension for analytics events
Resolved	Ottomata	T218268 eventgate-analytics k8s pods occasionally can't produce to kafka

Event Timeline

jijiki created this task.Mar 13 2019, 8:38 PM

jijiki triaged this task as High priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2019, 8:38 PM

jijiki updated the task description. (Show Details)Mar 13 2019, 8:38 PM

• Pchelolo added a subtask: T218254: EventBus extension should never log unserialized events.Mar 13 2019, 9:05 PM

• Pchelolo added a subtask: T218260: Decrease timeout for EventBus extension for analytics events.

jijiki added a project: serviceops.Mar 13 2019, 9:06 PM

• mobrovac added projects: Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Platform Team Workboards (Doing), Services (doing).Mar 13 2019, 9:10 PM

Change 496329 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Disable api-request eventgate logging for group0 wikis

https://gerrit.wikimedia.org/r/496329

Change 496329 merged by Ottomata:
[operations/mediawiki-config@master] Disable api-request eventgate logging for group0 wikis

https://gerrit.wikimedia.org/r/496329

jijiki renamed this task from Influx of service errors Mar 13: 19:12-19:14 UTC and 20:12-20:15 to Enabling api-request eventgate to group1 caused minor service disruptions .Mar 14 2019, 1:03 PM

jijiki updated the task description. (Show Details)

Change 496454 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Fix liveness and readiness probes

https://gerrit.wikimedia.org/r/496454

Ottomata mentioned this in rDEPLOYCHARTSb2617c22ef0a: eventgate-analytics - Fix liveness and readiness probes.Mar 14 2019, 2:48 PM

Change 496454 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Fix liveness and readiness probes

https://gerrit.wikimedia.org/r/496454

• Pchelolo closed subtask T218260: Decrease timeout for EventBus extension for analytics events as Resolved.Mar 25 2019, 7:00 PM

Ottomata closed subtask T218254: EventBus extension should never log unserialized events as Resolved.Mar 25 2019, 7:03 PM

akosiaris closed subtask T218268: eventgate-analytics k8s pods occasionally can't produce to kafka as Resolved.Mar 26 2019, 9:39 PM

greg moved this task from Active investigation to Active Situation on the Wikimedia-Incident board.Mar 28 2019, 5:04 PM