Page MenuHomePhabricator

Job queue is growing and growing
Closed, ResolvedPublic

Description

The Job queue grows and grows. This causes a lot of issues. For example the candidates of speedy deletion are not at the category for that anymore.

Event Timeline

Luke081515 raised the priority of this task from to Needs Triage.
Luke081515 updated the task description. (Show Details)
Luke081515 added a subscriber: Luke081515.
Luke081515 triaged this task as Unbreak Now! priority.Jan 20 2016, 5:52 PM

Looks like it is growing since 1st january.

Change 265438 had a related patch set uploaded (by Ori.livneh):
Job runners: Add a dedicated htmlCacheUpdate runner

https://gerrit.wikimedia.org/r/265438

Change 265438 merged by Ori.livneh:
Job runners: Add a dedicated htmlCacheUpdate runner

https://gerrit.wikimedia.org/r/265438

The bulk of the jobs in the queue are htmlCacheUpdate jobs, and @aaron suspects that the runners are slow to clear the backlog because the job back-off throttling configuration is not having the effect that we expect it to have (thus T123815: Rewrite/modify $wgJobBackoffThrottling handling code to be less pessimistic). As an interim measure, we added (in I180856917b) a dedicated runner for htmlCacheUpdate jobs, and we're monitoring the situation.

Luke081515 claimed this task.

Seems like it is fixed now, (the queue needs still some time to make the backlog smaller) but categorysation etc. works now.

Currently we have 4 million jobs, and the queue is still growing...