Maniphest T197003

Dismantle most of the old jobqueue infrastructure
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Joe
	Jun 12 2018, 2:27 PM

Description

Given we're managing just 0.04 jobs/second, it's time for a final blow to the supporting infrastructure.

Specifically:

Reduce the number of jobqueue workers to 2 per server, one "basic", one for gwt". They're more than enough
Remove servers from the redis cluster - specifically let's remove all but rdb1005/6 in eqiad and rdb2005/6 in codfw; the other servers in eqiad will be decommissioned, while the ones in codfw will be returned to spares.

Details

Subject	Repo	Branch	Lines +/-
Reduce the jobqueue redis to use just one server per dc	operations/mediawiki-config	master	+2 -25
jobrunner: reduce to one redis server per datacenter	operations/puppet	production	+0 -23
jobrunner: reduce the number of old runners	operations/puppet	production	+1 -1

Customize query in gerrit

Event Timeline

Joe created this task.Jun 12 2018, 2:27 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 12 2018, 2:27 PM

Change 439943 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] jobrunner: reduce the number of old runners

https://gerrit.wikimedia.org/r/439943

Change 439944 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] jobrunner: reduce to one redis server per datacenter

https://gerrit.wikimedia.org/r/439944

Change 439945 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Reduce the jobqueue redis to use just one server per dc

https://gerrit.wikimedia.org/r/439945

Change 439943 merged by Giuseppe Lavagetto:
[operations/puppet@production] jobrunner: reduce the number of old runners

https://gerrit.wikimedia.org/r/439943

Change 439944 merged by Giuseppe Lavagetto:
[operations/puppet@production] jobrunner: reduce to one redis server per datacenter

https://gerrit.wikimedia.org/r/439944

Change 439945 merged by jenkins-bot:
[operations/mediawiki-config@master] Reduce the jobqueue redis to use just one server per dc

https://gerrit.wikimedia.org/r/439945

Mentioned in SAL (#wikimedia-operations) [2018-06-13T07:59:34Z] <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: Remove unused redis shards from the jobqueue T197003 (duration: 00m 58s)