Page MenuHomePhabricator

Create partitioned CirrusSearchElasticaWrite topic
Closed, ResolvedPublic

Description

We need to have the following topics with 3 partitions in the main Kafka cluster in both eqiad and codfw.

eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite
codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite

Event Timeline

Pchelolo created this task.Mon, Nov 25, 5:39 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptMon, Nov 25, 5:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I believe these should do it?

On eqiad:

kafka topics --create --partitions 3 --replication-factor 3 eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite

On codfw:

kafka topics --create --partitions 3 --replication-factor 3 codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite

Will the mirrored topics be created by mirror maker automagically?

elukey added a subscriber: elukey.EditedMon, Nov 25, 5:58 PM

I believe these should do it?
On eqiad:

kafka topics --create --partitions 3 --replication-factor 3 eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite

On codfw:

kafka topics --create --partitions 3 --replication-factor 3 codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite

Will the mirrored topics be created by mirror maker automagically?

Commands looks good, but better to create the topics before MM so we'll be sure that they'll have the right partitions.

DISCLAIMER: I assume that the .partitioned. name was discussed and decided, don't want to get into the naming bikeshedding in here :D

DISCLAIMER: I assume that the .partitioned. name was discussed and decided, don't want to get into the naming bikeshedding in here :D

That is indeed the case and is a standard for partitioned topics in the job queue... so yeah, the bikeshedding ship has sailed.

Mentioned in SAL (#wikimedia-operations) [2019-11-26T19:59:01Z] <Pchelolo> create partitioned topics for cirrusSearchElasticaWrite on kafka-main T239135

Pchelolo closed this task as Resolved.Tue, Nov 26, 8:02 PM

Done.