Maniphest T279893

High load on jobrunners (12 Apr 2021)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jijiki
	Apr 12 2021, 9:32 AM

Tags

Referenced Files

	F34358641: image.png
	Apr 12 2021, 9:32 AM

Subscribers

Description

A user uploaded a large number of videos, starving the videoscalers/jobrunners of resources. We killed ffmpeg from the top busiest servers, we depooled 3 servers from the videoscaler cluster, and increased their weight on the jobrunner cluster, so to give priority to normal jobs.

https://phabricator.wikimedia.org/P15272 (tx to @Urbanecm)

image.png (1×3 px, 284 KB)

Impact was rather small, we had a few alerts for failed monitoring checks, and increases in jobqueue's job backlog.

Event Timeline

jijiki triaged this task as Medium priority.Apr 12 2021, 9:32 AM

jijiki created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 12 2021, 9:32 AM

jijiki renamed this task from High load on jobrunners to High load on jobrunners (12 Apr 2021).Apr 12 2021, 9:55 AM

jijiki updated the task description. (Show Details)

Closing this task, no further issues were observed

jijiki added a project: Wikimedia-Incident.Apr 13 2021, 10:27 AM