Page MenuHomePhabricator

High load on jobrunners (12 Apr 2021)
Closed, ResolvedPublic

Description

A user uploaded a large number of videos, starving the videoscalers/jobrunners of resources. We killed ffmpeg from the top busiest servers, we depooled 3 servers from the videoscaler cluster, and increased their weight on the jobrunner cluster, so to give priority to normal jobs.

https://phabricator.wikimedia.org/P15272 (tx to @Urbanecm)

image.png (1×3 px, 284 KB)

Impact was rather small, we had a few alerts for failed monitoring checks, and increases in jobqueue's job backlog.

Event Timeline

jijiki triaged this task as Medium priority.Apr 12 2021, 9:32 AM
jijiki created this task.
jijiki renamed this task from High load on jobrunners to High load on jobrunners (12 Apr 2021).Apr 12 2021, 9:55 AM
jijiki updated the task description. (Show Details)

Closing this task, no further issues were observed