Page MenuHomePhabricator
Create Task
Maniphest T387865

Repartition [eqiad|codfw].mediawiki.cirrussearch.page_weighted_tags_change.v1 topics in kafka-main@[eqiad|codfw]
Closed, ResolvedPublic
Actions

Description

We are going to move away from the *.mediawiki.cirrussearch.page_weighted_tags_change.rc0 topics that were useful as part of the development of the Search Update Pipeline.
Both schemas, stream & topic names should refer to a stable version now that the pipeline is running in production for all the wikis.

For this we need the two new v1 topics in each kafka-main clusters to be properly partitioned:

  • eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1: 5 partitions
  • codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1: 5 partitions

Related Objects
Search...

Event Timeline

dcausse created this task.Mar 4 2025, 2:30 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptMar 4 2025, 2:30 PM
Restricted Application added a subscriber: • Aklapper. · View Herald Transcript
dcausse added a parent task: T375821: Migrate streaming updater event schema to the standard schema repository.Mar 4 2025, 2:30 PM
Gehel triaged this task as High priority.Mar 4 2025, 4:35 PM
Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.
Gehel edited projects, added: Data-Platform-SRE (2025.03.01 - 2025.03.21); removed: Data-Platform-SRE.
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.
Gehel moved this task from needs triage to 2025.03.01 - 2025.03.21 on the Discovery-Search board.Mar 7 2025, 9:08 AM
Gehel edited projects, added: Discovery-Search (2025.03.01 - 2025.03.21); removed: Discovery-Search.
dcausse added a comment.Mar 10 2025, 9:30 AM

In the interest of time I'd suggest to keep 5 partitions for now

dcausse updated the task description. (Show Details)Mar 10 2025, 9:31 AM
Gehel moved this task from Incoming to Blocked / Waiting on the Discovery-Search (2025.03.01 - 2025.03.21) board.Mar 10 2025, 4:44 PM
brouberol subscribed.Mar 18 2025, 10:33 AM

done for kafka-main-eqiad:

brouberol@kafka-main1006:~$ kafka topics --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/main-eqiad --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 1005	Replicas: 1005,1001,1002	Isr: 1005,1001,1002
brouberol@kafka-main1006:~$ kafka topics --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/main-eqiad --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
brouberol@kafka-main1006:~$ kafka topics --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/main-eqiad --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 1001	Replicas: 1001,1002,1003	Isr: 1001,1002,1003
brouberol@kafka-main1006:~$ kafka topics --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/main-eqiad --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
brouberol@kafka-main1006:~$ kafka topics --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/main-eqiad --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:5	ReplicationFactor:3	Configs:
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 1005	Replicas: 1005,1001,1002	Isr: 1005,1001,1002
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 1	Leader: 1001	Replicas: 1001,1002,1003	Isr: 1001,1002,1003
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 2	Leader: 1002	Replicas: 1002,1003,1004	Isr: 1002,1003,1004
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 3	Leader: 1003	Replicas: 1003,1004,1001	Isr: 1003,1004,1001
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 4	Leader: 1004	Replicas: 1004,1001,1002	Isr: 1004,1001,1002
brouberol@kafka-main1006:~$ kafka topics --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/main-eqiad --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:5	ReplicationFactor:3	Configs:
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 1001	Replicas: 1001,1002,1003	Isr: 1001,1002,1003
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 1	Leader: 1002	Replicas: 1002,1003,1004	Isr: 1002,1003,1004
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 2	Leader: 1003	Replicas: 1003,1004,1001	Isr: 1003,1004,1001
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 3	Leader: 1004	Replicas: 1004,1001,1002	Isr: 1004,1001,1002
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 4	Leader: 1005	Replicas: 1005,1001,1002	Isr: 1005,1001,1002
brouberol added a comment.Mar 18 2025, 10:33 AM

done for kafka-main-codfw:

brouberol@kafka-main2006:~$ kafka topics --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf2004.codfw.wmnet,conf2005.codfw.wmnet,conf2006.codfw.wmnet/kafka/main-codfw --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 2001	Replicas: 2001,2002,2003	Isr: 2001,2002,2003
brouberol@kafka-main2006:~$  kafka topics --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
kafka-topics --zookeeper conf2004.codfw.wmnet,conf2005.codfw.wmnet,conf2006.codfw.wmnet/kafka/main-codfw --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
brouberol@kafka-main2006:~$ kafka topics --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf2004.codfw.wmnet,conf2005.codfw.wmnet,conf2006.codfw.wmnet/kafka/main-codfw --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 2004	Replicas: 2004,2001,2002	Isr: 2004,2001,2002
brouberol@kafka-main2006:~$ kafka topics --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
kafka-topics --zookeeper conf2004.codfw.wmnet,conf2005.codfw.wmnet,conf2006.codfw.wmnet/kafka/main-codfw --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
brouberol@kafka-main2006:~$ kafka topics --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf2004.codfw.wmnet,conf2005.codfw.wmnet,conf2006.codfw.wmnet/kafka/main-codfw --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:5	ReplicationFactor:3	Configs:
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 2001	Replicas: 2001,2002,2003	Isr: 2001,2002,2003
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 1	Leader: 2002	Replicas: 2002,2003,2004	Isr: 2002,2003,2004
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 2	Leader: 2003	Replicas: 2003,2004,2001	Isr: 2003,2004,2001
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 3	Leader: 2004	Replicas: 2004,2001,2002	Isr: 2004,2001,2002
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 4	Leader: 2005	Replicas: 2005,2001,2002	Isr: 2005,2001,2002
brouberol@kafka-main2006:~$ kafka topics --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf2004.codfw.wmnet,conf2005.codfw.wmnet,conf2006.codfw.wmnet/kafka/main-codfw --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:5	ReplicationFactor:3	Configs:
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 2004	Replicas: 2004,2001,2002	Isr: 2004,2001,2002
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 1	Leader: 2005	Replicas: 2005,2001,2002	Isr: 2005,2001,2002
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 2	Leader: 2001	Replicas: 2001,2002,2003	Isr: 2001,2002,2003
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 3	Leader: 2002	Replicas: 2002,2003,2004	Isr: 2002,2003,2004
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 4	Leader: 2003	Replicas: 2003,2004,2001	Isr: 2003,2004,2001
brouberol closed this task as Resolved.Mar 18 2025, 10:35 AM
brouberol claimed this task.
brouberol moved this task from Backlog - operations to Done on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.

done for kafka-jumbo-eqiad (mirror):

brouberol@kafka-jumbo1014:~$ kafka topics --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 1011	Replicas: 1011,1008,1013	Isr: 1011,1008,1013
brouberol@kafka-jumbo1014:~$ kafka topics --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
brouberol@kafka-jumbo1014:~$ kafka topics --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 1015	Replicas: 1015,1009,1011	Isr: 1015,1009,1011
brouberol@kafka-jumbo1014:~$ kafka topics --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --alter --partitions 5
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
brouberol@kafka-jumbo1014:~$ kafka topics --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --topic eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:5	ReplicationFactor:3	Configs:
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 1011	Replicas: 1011,1008,1013	Isr: 1011,1008,1013
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 1	Leader: 1011	Replicas: 1011,1014,1007	Isr: 1011,1014,1007
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 2	Leader: 1014	Replicas: 1014,1012,1007	Isr: 1014,1012,1007
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 3	Leader: 1012	Replicas: 1012,1015,1007	Isr: 1012,1015,1007
	Topic: eqiad.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 4	Leader: 1015	Replicas: 1015,1007,1008	Isr: 1015,1007,1008
brouberol@kafka-jumbo1014:~$ kafka topics --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --topic codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1 --describe
Topic:codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	PartitionCount:5	ReplicationFactor:3	Configs:
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 0	Leader: 1015	Replicas: 1015,1009,1011	Isr: 1015,1009,1011
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 1	Leader: 1007	Replicas: 1007,1008,1010	Isr: 1007,1008,1010
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 2	Leader: 1008	Replicas: 1008,1010,1013	Isr: 1008,1010,1013
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 3	Leader: 1010	Replicas: 1010,1013,1009	Isr: 1010,1013,1009
	Topic: codfw.mediawiki.cirrussearch.page_weighted_tags_change.v1	Partition: 4	Leader: 1013	Replicas: 1013,1009,1011	Isr: 1013,1009,1011
dcausse added a comment.Mar 18 2025, 10:36 AM

@brouberol thanks!!

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.Mar 21 2025, 9:39 AM
Gehel moved this task from Blocked / Waiting to Done on the Discovery-Search (2025.03.01 - 2025.03.21) board.Mar 21 2025, 1:55 PM
Gehel moved this task from Done to Reported on the Discovery-Search (2025.03.01 - 2025.03.21) board.Mar 21 2025, 2:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits