SUP: Partition update_pipeline kafka topic
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	pfischer
	Dec 27 2023, 10:07 PM

Description

Processing updates cannot be handled by a single flink operator slot. To support and hint parallel processing inside flink, the topic should be partitioned.

AC:

SRE/ops: kafka-test is configured to use 5 partitions for topic (eqiad|codfw).cirrussearch.update_pipeline.update.rc0
stream configuration for cirrussearch.update_pipeline.update.rc0 should define message_key_fields (see 983719)
producer writes ~~explicitly~~ keyed records to the kafka sink
consumer reads ~~keyed records and reinterpretes them as keyed stream~~ partitioned topic

Details

	Subject	Repo	Branch	Lines +/-
	Search update pipeline: bump version	operations/deployment-charts	master	+1 -1
	Search update pipeline: enable kafka partition discovery	operations/deployment-charts	master	+1 -0

Customize query in gerrit

	Title	Reference	Author	Source Branch	Dest Branch
	Rely on kafka partitioning to aid parallel execution	repos/search-platform/cirrus-streaming-updater!88	pfischer	parallelism	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T317045 [Epic] Re-architect the Search Update Pipeline
		Resolved		pfischer	T354064 SUP: Partition update_pipeline kafka topic

Event Timeline

pfischer created this task.Dec 27 2023, 10:07 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 27 2023, 10:07 PM

pfischer updated the task description. (Show Details)Dec 27 2023, 10:10 PM

pfischer opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/88

Interpret update source as keyed stream

pfischer claimed this task.Jan 2 2024, 8:15 AM

pfischer set the point value for this task to 3.Jan 2 2024, 8:40 AM

pfischer updated the task description. (Show Details)Jan 2 2024, 9:42 AM

pfischer renamed this task from SUP: Partition update topic to SUP: Partition update_pipeline kafka topic.Jan 2 2024, 9:46 AM

pfischer changed the task status from Open to In Progress.

pfischer triaged this task as High priority.

pfischer added projects: serviceops, SRE.

pfischer updated the task description. (Show Details)

pfischer moved this task from needs triage to Current work on the Discovery-Search board.

pfischer edited projects, added Discovery-Search (Current work); removed Discovery-Search.

pfischer removed pfischer as the assignee of this task.Jan 2 2024, 10:49 AM

pfischer updated Other Assignee, added: pfischer.

pfischer added a project: Data-Platform-SRE.

Change 987160 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[operations/deployment-charts@master] Search update pipeline: enable kafka partition discovery

https://gerrit.wikimedia.org/r/987160

Change 987160 merged by jenkins-bot:

[operations/deployment-charts@master] Search update pipeline: enable kafka partition discovery

https://gerrit.wikimedia.org/r/987160

pfischer merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/88

Rely on kafka partitioning to aid parallel execution

Maintenance_bot removed a project: Patch-For-Review.Jan 3 2024, 8:30 AM

Change 987395 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[operations/deployment-charts@master] Search update pipeline: bump version

https://gerrit.wikimedia.org/r/987395

Change 987395 merged by jenkins-bot:

[operations/deployment-charts@master] Search update pipeline: bump version

https://gerrit.wikimedia.org/r/987395

Maintenance_bot removed a project: Patch-For-Review.Jan 3 2024, 9:30 AM

brouberol@kafka-test1010:~$ kafka topics --topic codfw.cirrussearch.update_pipeline.update.rc0 --alter --partitions 5
kafka-topics --zookeeper zookeeper-test1002.eqiad.wmnet/kafka/test-eqiad --topic codfw.cirrussearch.update_pipeline.update.rc0 --alter --partitions 5
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
brouberol@kafka-test1010:~$ kafka topics --topic eqiad.cirrussearch.update_pipeline.update.rc0 --alter --partitions 5
kafka-topics --zookeeper zookeeper-test1002.eqiad.wmnet/kafka/test-eqiad --topic eqiad.cirrussearch.update_pipeline.update.rc0 --alter --partitions 5
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

brouberol updated the task description. (Show Details)Jan 3 2024, 1:26 PM

bking subscribed.Jan 3 2024, 2:15 PM

pfischer claimed this task.Jan 3 2024, 10:23 PM

pfischer updated the task description. (Show Details)

pfischer moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.

pfischer updated Other Assignee, removed: pfischer.

pfischer moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.

Gehel added a parent task: T317045: [Epic] Re-architect the Search Update Pipeline.Jan 8 2024, 10:25 AM

Gehel moved this task from Incoming to 2024.01.01 - 2024.01.21 on the Data-Platform-SRE board.Jan 8 2024, 12:20 PM

Gehel edited projects, added Data-Platform-SRE (2024.01.01 - 2024.01.21); removed Data-Platform-SRE.

Gehel moved this task from Backlog to Done on the Data-Platform-SRE (2024.01.01 - 2024.01.21) board.

pfischer mentioned this in T353473: The cirrussearch.update_pipeline.update stream should be keyed by wiki and page_id.Jan 9 2024, 8:20 AM