Maniphest T192005

Disable MirrorMaker for job queue events
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Pchelolo
	Apr 11 2018, 5:25 PM

Description

Until we have TLS support for MirrorMaker for cross-dc kafka job replication in order to enable new JobQueue support for private wikis we need to temporarily disable MirrorMaker for jobs.

We don't really use the mirrored jobs right now and the only reason to mirror them is to avoid losing the jobs in case of an actual data center disaster, and as things stand now if an disastrous event occurs losing jobs will not be the biggest of our problems, so right now it's acceptable to just disable mirroring jobs.

Mirroring should be enabled back when we get TLS for kafka and mirror maker (Q1)

Details

	Subject	Repo	Branch	Lines +/-
	Blacklist mediawiki.job and change-prop from cross DC main <-> main mirror	operations/puppet	production	+6 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• Pchelolo	T157088 [EPIC] Develop a JobQueue backend based on EventBus
Resolved	• Pchelolo	T190327 FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus
Resolved	• Pchelolo	T191464 Enable CP4JQ support for private wikis
Resolved	Ottomata	T192005 Disable MirrorMaker for job queue events

Event Timeline

• Pchelolo triaged this task as Medium priority.Apr 11 2018, 5:25 PM

• Pchelolo created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 11 2018, 5:25 PM

This goes for change-prop too, right? So blacklist .+\.(job|change-prop)\..+?

@Ottomata I think we can disable anything that have /mediawiki\.job/ in there.

The retry topics will contain the original topic names too.

Ok, but we should keep (other?) change-prop topics mirrored?

@Ottomata yes. We're not enabling normal mediawiki events for private wikis because RESTBase or EventStreams are not ready or that.

Change 425824 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Blacklist mediawiki.job topics from cross DC main <-> main Kafka mirroring

https://gerrit.wikimedia.org/r/425824

gerritbot added a project: Patch-For-Review.Apr 12 2018, 2:05 PM

• fdans added a project: Analytics-Kanban.Apr 12 2018, 4:48 PM

• fdans moved this task from Incoming to Operational Excellence on the Analytics board.

• fdans moved this task from Next Up to Ready to Deploy on the Analytics-Kanban board.

Change 425824 merged by Ottomata:
[operations/puppet@production] Blacklist mediawiki.job and change-prop from cross DC main <-> main mirror

https://gerrit.wikimedia.org/r/425824