Services Q2 2017/18 goal: Migrate a subset of jobs to multi-DC enabled event processing infrastructure.
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• GWicke
	Sep 6 2017, 8:04 PM

Description

Tracking task for Services work towards reliable, multi-datacenter enabled job processing in Q2.

Annual plan:
Program 8: Multi-datacenter support
Outcome 2: Backend infrastructure works reliably across data centers
Objective 1: Reliable, multi-DC job processing

Quarterly objectives for Q2 2017/18 (October - December 2017)

Migrate a subset of jobs to the new job queue implementation.
Identify needs for the migration of the remaining jobs.
Document metrics and common operational tasks.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
[JobQueue] Only produce wikibase-addUsagesForPage to EventBus	operations/mediawiki-config	master	+1 -0
JobQueue: Switch RecordLintJob to EventBus	operations/mediawiki-config	master	+1 -0
Config: Process RecordLintJob via CPJQ	mediawiki/services/change-propagation/jobqueue-deploy	master	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• Gilles	T175206 2017/18 Annual Plan Program 8: Multi-datacenter support
Resolved	• Gilles	T175213 2017/18 Annual Plan Program 8: Multi-datacenter support, Q2 goals
Resolved	• Pchelolo	T157088 [EPIC] Develop a JobQueue backend based on EventBus
Resolved	• Pchelolo	T175212 Services Q2 2017/18 goal: Migrate a subset of jobs to multi-DC enabled event processing infrastructure.

Event Timeline

• GWicke created this task.Sep 6 2017, 8:04 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 6 2017, 8:04 PM

• GWicke renamed this task from Services Q1 2017/18 goal: Migrate most jobs to multi-DC enabled event processing infrastructure. to Services Q1 2017/18 goal: Migrate a subset of jobs to multi-DC enabled event processing infrastructure..Sep 6 2017, 8:04 PM

• GWicke renamed this task from Services Q1 2017/18 goal: Migrate a subset of jobs to multi-DC enabled event processing infrastructure. to Services Q2 2017/18 goal: Migrate a subset of jobs to multi-DC enabled event processing infrastructure..

• GWicke mentioned this in T175213: 2017/18 Annual Plan Program 8: Multi-datacenter support, Q2 goals.Sep 6 2017, 8:06 PM

• GWicke added a parent task: T175213: 2017/18 Annual Plan Program 8: Multi-datacenter support, Q2 goals.Sep 6 2017, 8:19 PM

• mobrovac added projects: Event-Platform, ChangeProp, MediaWiki-Core-JobQueue.Sep 7 2017, 9:53 AM

Restricted Application added a project: Analytics. · View Herald TranscriptSep 7 2017, 9:53 AM

mforns moved this task from Incoming to Radar on the Analytics board.Sep 28 2017, 3:41 PM

Change 391166 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/mediawiki-config@master] JobQueue: Switch RecordLintJob to EventBus

https://gerrit.wikimedia.org/r/391166

gerritbot added a project: Patch-For-Review.Nov 14 2017, 8:45 AM

Change 391167 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/change-propagation/jobqueue-deploy@master] Config: Process RecordLintJob via CPJQ

https://gerrit.wikimedia.org/r/391167

• mobrovac edited projects, added Services (doing); removed Services (next).Nov 14 2017, 8:54 AM

Change 391167 merged by Ppchelko:
[mediawiki/services/change-propagation/jobqueue-deploy@master] Config: Process RecordLintJob via CPJQ

https://gerrit.wikimedia.org/r/391167

Change 391166 merged by jenkins-bot:
[operations/mediawiki-config@master] JobQueue: Switch RecordLintJob to EventBus

https://gerrit.wikimedia.org/r/391166

Mentioned in SAL (#wikimedia-operations) [2017-11-14T09:59:28Z] <mobrovac@tin> Synchronized wmf-config/jobqueue.php: JobQueue: Migrate RecordLintJob to EventBus - T175212 (duration: 00m 46s)

• mobrovac added a subtask: T175210: Select candidate jobs for transferring to the new infrastucture.Nov 28 2017, 10:33 AM

• mobrovac added a parent task: T157088: [EPIC] Develop a JobQueue backend based on EventBus.

Change 393734 had a related patch set uploaded (by Mobrovac; owner: Ppchelko):
[operations/mediawiki-config@master] [JobQueue] Only produce wikibase-addUsagesForPage to EventBus

https://gerrit.wikimedia.org/r/393734

Change 393734 merged by jenkins-bot:
[operations/mediawiki-config@master] [JobQueue] Only produce wikibase-addUsagesForPage to EventBus

https://gerrit.wikimedia.org/r/393734

Mentioned in SAL (#wikimedia-operations) [2017-11-28T10:44:08Z] <mobrovac@tin> Synchronized wmf-config/jobqueue.php: Process wikibase-addUsagesForPage only via EventBus - T175212 (duration: 00m 44s)

• mobrovac added a subtask: T182023: Migrate htmlCacheUpdate job to Kafka.Dec 5 2017, 9:42 AM

• mobrovac removed a subtask: T182023: Migrate htmlCacheUpdate job to Kafka.Dec 28 2017, 12:46 PM