Maniphest T124194

Job queue is growing and growing
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Luke081515
	Jan 20 2016, 5:52 PM

Project Tags

Referenced Files

None

Subscribers

View All 16 Subscribers

Description

The Job queue grows and grows. This causes a lot of issues. For example the candidates of speedy deletion are not at the category for that anymore.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Job runners: Add a dedicated htmlCacheUpdate runner	Ori	operations/puppet	production	+8 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		ori	T124194 Job queue is growing and growing
		Resolved		None	T122069 jobrunner memory leaks
		Resolved		aaron	T123284 Record per-job-type memory usage statistics
		Declined		None	T123815 Rewrite/modify $wgJobBackoffThrottling handling code to be less pessimistic

Event Timeline

Luke081515 created this task.Jan 20 2016, 5:52 PM

Luke081515 raised the priority of this task from to Needs Triage.

Luke081515 updated the task description. (Show Details)

Luke081515 added projects: MediaWiki-Core-JobQueue, MediaWiki-extensions-General.

Luke081515 subscribed.

Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 20 2016, 5:52 PM

Luke081515 triaged this task as Unbreak Now! priority.Jan 20 2016, 5:52 PM

Looks like it is growing since 1st january.

• matmarex edited projects, added: WMF-General-or-Unknown; removed: MediaWiki-extensions-General, MediaWiki-Core-JobQueue.Jan 20 2016, 5:59 PM

• matmarex set Security to None.

Steinsplitter subscribed.Jan 20 2016, 7:24 PM

Raymond subscribed.Jan 20 2016, 7:24 PM

Agabi10 subscribed.Jan 20 2016, 7:38 PM

Luke081515 awarded a token.Jan 20 2016, 8:23 PM

Schulhofpassage subscribed.Jan 20 2016, 8:35 PM

Yellowcard subscribed.Jan 20 2016, 8:42 PM

Boshomi subscribed.Jan 20 2016, 10:05 PM

Peachey88 added a project: SRE.Jan 21 2016, 1:40 AM

ori added subtasks: T122069: jobrunner memory leaks, T123815: Rewrite/modify $wgJobBackoffThrottling handling code to be less pessimistic.Jan 21 2016, 1:43 AM

Krenair subscribed.Jan 21 2016, 1:45 AM

Change 265438 had a related patch set uploaded (by Ori.livneh):
Job runners: Add a dedicated htmlCacheUpdate runner

https://gerrit.wikimedia.org/r/265438

gerritbot added a project: Patch-For-Review.Jan 21 2016, 2:06 AM

Change 265438 merged by Ori.livneh:
Job runners: Add a dedicated htmlCacheUpdate runner

https://gerrit.wikimedia.org/r/265438

The bulk of the jobs in the queue are htmlCacheUpdate jobs, and @aaron suspects that the runners are slow to clear the backlog because the job back-off throttling configuration is not having the effect that we expect it to have (thus T123815: Rewrite/modify $wgJobBackoffThrottling handling code to be less pessimistic). As an interim measure, we added (in I180856917b) a dedicated runner for htmlCacheUpdate jobs, and we're monitoring the situation.

• MZMcBride subscribed.Jan 21 2016, 2:53 AM

Looks good now

Seems like it is fixed now, (the queue needs still some time to make the backlog smaller) but categorysation etc. works now.

Luke081515 reassigned this task from Luke081515 to ori.Jan 21 2016, 12:33 PM

Luke081515 removed a project: Patch-For-Review.

Luke081515 mentioned this in T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan.Jan 23 2016, 5:10 PM

Currently we have 4 million jobs, and the queue is still growing...

Luke081515 removed ori as the assignee of this task.Mar 10 2016, 5:47 PM

(created T129517 for the new case)

aaron closed subtask T123815: Rewrite/modify $wgJobBackoffThrottling handling code to be less pessimistic as Declined.Apr 5 2017, 7:40 PM

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptApr 5 2017, 7:40 PM

Joe mentioned this in T163418: jobqueue is full of refreshlinks duplicates after the switchover..Apr 20 2017, 8:38 AM

Joe closed subtask T122069: jobrunner memory leaks as Resolved.Sep 5 2017, 8:23 AM