We need to have the following topics with 3 partitions in the main Kafka cluster in both eqiad and codfw.
eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite
codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | Feature | None | T71489 Expose mwgrep functionality on-wiki | ||
| Resolved | None | T109715 Replicate production elasticsearch indices to labs | |||
| Resolved | EBernhardson | T220625 Initialize CirrusSearch on cloudelastic | |||
| Resolved | TJones | T235831 [Objective Fiscal 19-20/Q2] (4) Isolate failures between different search components to maintain a resilient service | |||
| Resolved | EBernhardson | T235832 CirrusSearch writes are split into per cluster kafka partitions to isolate clusters from each others by end of Q2 | |||
| Resolved | EBernhardson | T230495 Partition CirrusSearch mediawiki jobs by cluster | |||
| Resolved | Ottomata | T239135 Create partitioned CirrusSearchElasticaWrite topic |
Event Timeline
Comment Actions
I believe these should do it?
On eqiad:
kafka topics --create --partitions 3 --replication-factor 3 eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite
On codfw:
kafka topics --create --partitions 3 --replication-factor 3 codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite
Will the mirrored topics be created by mirror maker automagically?
Comment Actions
Commands looks good, but better to create the topics before MM so we'll be sure that they'll have the right partitions.
DISCLAIMER: I assume that the .partitioned. name was discussed and decided, don't want to get into the naming bikeshedding in here :D
Comment Actions
DISCLAIMER: I assume that the .partitioned. name was discussed and decided, don't want to get into the naming bikeshedding in here :D
That is indeed the case and is a standard for partitioned topics in the job queue... so yeah, the bikeshedding ship has sailed.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-11-26T19:59:01Z] <Pchelolo> create partitioned topics for cirrusSearchElasticaWrite on kafka-main T239135