Page MenuHomePhabricator

Job queue is growing and growing
Closed, ResolvedPublic

Description

The Job queue grows and grows. This causes a lot of issues. For example the candidates of speedy deletion are not at the category for that anymore.

Event Timeline

Luke081515 raised the priority of this task from to Needs Triage.
Luke081515 updated the task description. (Show Details)
Luke081515 added a subscriber: Luke081515.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 20 2016, 5:52 PM
Luke081515 triaged this task as Unbreak Now! priority.Jan 20 2016, 5:52 PM

Looks like it is growing since 1st january.

Change 265438 had a related patch set uploaded (by Ori.livneh):
Job runners: Add a dedicated htmlCacheUpdate runner

https://gerrit.wikimedia.org/r/265438

Change 265438 merged by Ori.livneh:
Job runners: Add a dedicated htmlCacheUpdate runner

https://gerrit.wikimedia.org/r/265438

ori added subscribers: aaron, ori.Jan 21 2016, 2:30 AM

The bulk of the jobs in the queue are htmlCacheUpdate jobs, and @aaron suspects that the runners are slow to clear the backlog because the job back-off throttling configuration is not having the effect that we expect it to have (thus T123815: Rewrite/modify $wgJobBackoffThrottling handling code to be less pessimistic). As an interim measure, we added (in I180856917b) a dedicated runner for htmlCacheUpdate jobs, and we're monitoring the situation.

Looks good now

Luke081515 closed this task as Resolved.Jan 21 2016, 12:33 PM
Luke081515 claimed this task.

Seems like it is fixed now, (the queue needs still some time to make the backlog smaller) but categorysation etc. works now.

Luke081515 reassigned this task from Luke081515 to ori.Jan 21 2016, 12:33 PM
Luke081515 removed a project: Patch-For-Review.
Luke081515 reopened this task as Open.Mar 10 2016, 5:47 PM

Currently we have 4 million jobs, and the queue is still growing...

Luke081515 removed ori as the assignee of this task.Mar 10 2016, 5:47 PM
Luke081515 closed this task as Resolved.Mar 10 2016, 5:50 PM
Luke081515 assigned this task to ori.

(created T129517 for the new case)

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptApr 5 2017, 7:40 PM