⚓ T345806 [Event Platform] mediawiki.page_content_change.v1 topic should be partitioned.

	Subject	Repo	Branch	Lines +/-
	mw-page-content-enrich: version bump.	operations/deployment-charts	master	+1 -1

	Title	Reference	Author	Source Branch	Dest Branch
	requirements: version bump eventutilities.	repos/data-engineering/mediawiki-event-enrichment!77	gmodena	version-bump-eventutilities	main
	make: version bump mediawiki-event-utilities.	repos/data-engineering/eventutilities-python!84	gmodena	mediawiki-event-utilities-version-bump	main

Status	Assigned	Task
Resolved	gmodena	T345657 [SPIKE] Should we enable compression on kafka jumbo?
Resolved	gmodena	T345806 [Event Platform] mediawiki.page_content_change.v1 topic should be partitioned.
Resolved	gmodena	T338231 [Event Platform] mw-page-content-change-enrich should (re)produce kafka keys

gmodena renamed this task from [NEEDS GROOMING] mediawiki.page_content_change.v1 topic should be partinioned. to [NEEDS GROOMING] mediawiki.page_content_change.v1 topic should be partitioned..Sep 7 2023, 8:40 AM

gmodena created this task.

gmodena mentioned this in T345657: [SPIKE] Should we enable compression on kafka jumbo? .Sep 7 2023, 8:44 AM

gmodena mentioned this in T307959: [Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content.Sep 8 2023, 10:39 AM

gmodena renamed this task from [NEEDS GROOMING] mediawiki.page_content_change.v1 topic should be partitioned. to mediawiki.page_content_change.v1 topic should be partitioned..Sep 8 2023, 10:49 AM

gmodena removed gmodena as the assignee of this task.

gmodena updated the task description. (Show Details)

lbowmaker moved this task from Sprint 1 to Sprint 2 on the Data Engineering and Event Platform Team board.Sep 8 2023, 1:23 PM

lbowmaker edited projects, added Data Engineering and Event Platform Team (Sprint 2); removed Data Engineering and Event Platform Team (Sprint 1).

gmodena added a subscriber: brouberol.Sep 11 2023, 2:09 PM

gmodena claimed this task.Sep 14 2023, 7:01 PM

gmodena added a project: Event-Platform.

Restricted Application added a project: Data-Engineering. · View Herald TranscriptSep 14 2023, 7:02 PM

gmodena removed gmodena as the assignee of this task.Sep 27 2023, 3:03 PM

lbowmaker moved this task from Sprint 2 to Sprint 3 on the Data Engineering and Event Platform Team board.Sep 29 2023, 1:28 PM

lbowmaker edited projects, added Data Engineering and Event Platform Team (Sprint 3); removed Data Engineering and Event Platform Team (Sprint 2).

Would it be worth partitioning mediawiki.page_change.v1 too? Just so we can run multiple consumers to process for backfills if/when we need (cough cough T347676).

gmodena claimed this task.Oct 18 2023, 3:30 PM

gmodena moved this task from Next Up to In progress on the Data Engineering and Event Platform Team (Sprint 3) board.

In T345806#9215806, @Ottomata wrote:

Would it be worth partitioning mediawiki.page_change.v1 too? Just so we can run multiple consumers to process for backfills if/when we need (cough cough T347676).

Probably? Needs SRE coordination because admin requires cli access to kafka brokers. That stream is easier to manager because messages are already produced with a partition key.

Picking this up. I'll do T338231 first, since is a requriement. Moving T338231 into this sprint.

Ahoelzl renamed this task from mediawiki.page_content_change.v1 topic should be partitioned. to [Event Platform] mediawiki.page_content_change.v1 topic should be partitioned..Oct 20 2023, 4:47 PM

Ahoelzl edited projects, added Data Engineering and Event Platform Team (Sprint 4); removed Data Engineering and Event Platform Team (Sprint 3).Oct 23 2023, 6:53 PM

Ahoelzl moved this task from Next Up to In progress on the Data Engineering and Event Platform Team (Sprint 4) board.

gmodena moved this task from In progress to Blocked/Paused on the Data Engineering and Event Platform Team (Sprint 4) board.Oct 30 2023, 1:14 PM

lbowmaker moved this task from Incoming (new tickets) to Event Platform Maintenance (current quarter) on the Data-Engineering board.Nov 10 2023, 1:44 PM

lbowmaker edited projects, added Data-Engineering (Sprint 5); removed Data Engineering and Event Platform Team (Sprint 4), Data-Engineering.Nov 10 2023, 2:10 PM

lbowmaker moved this task from Next Up to Blocked/Paused on the Data-Engineering (Sprint 5) board.

@gmodena let's sync when you when to increase partition count, and I'll happily oblige!

brouberol updated Other Assignee, added: brouberol.Nov 22 2023, 3:27 PM

lbowmaker edited projects, added Data-Engineering (Sprint 6); removed Data-Engineering (Sprint 5).Dec 1 2023, 1:47 PM

lbowmaker moved this task from Next Up to Blocked/Paused on the Data-Engineering (Sprint 6) board.

lbowmaker moved this task from Blocked/Paused to Next Up on the Data-Engineering (Sprint 6) board.Dec 4 2023, 3:45 PM

Pick this up now that mediawiki-event-utilities 1.3.3 has been released.

I'll start by version bumping deps in the python wrapper and downstream enrichment app docker image.

gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/84

make: version bump mediawiki-event-utilities.

gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/77

Draft: requirements: version bump eventutilities.

gmodena moved this task from In progress to In Review on the Data-Engineering (Sprint 6) board.Dec 5 2023, 10:09 AM

Change 980359 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/deployment-charts@master] mw-page-content-enrich: version bump.

https://gerrit.wikimedia.org/r/980359

gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/84

make: version bump mediawiki-event-utilities.

gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/77

requirements: version bump eventutilities.

Ahoelzl moved this task from In Review to Ready to Deploy on the Data-Engineering (Sprint 6) board.Dec 18 2023, 3:10 PM

Change 980359 merged by jenkins-bot:

[operations/deployment-charts@master] mw-page-content-enrich: version bump.

https://gerrit.wikimedia.org/r/980359

Ottomata added a subtask: T338231: [Event Platform] mw-page-content-change-enrich should (re)produce kafka keys.Dec 18 2023, 8:11 PM

Maintenance_bot removed a project: Patch-For-Review.Dec 18 2023, 8:30 PM

To apply these changes:

Suspended flink app: helmfile -e <env> apply --set app.job.state='suspended'
Alter kafka topic partitioning: kafka topics --alter --topic <topic> --partitions 3
Restart flink app: helmfile -e <env> apply

staging & kafka test eqiad. topic
eqiad & kafka jumbo eqiad. topic
codfw + kafka jumbo codfw. topic

Done!

Ottomata mentioned this in T338231: [Event Platform] mw-page-content-change-enrich should (re)produce kafka keys.Dec 18 2023, 9:10 PM

gmodena moved this task from Ready to Deploy to Done on the Data-Engineering (Sprint 6) board.Dec 19 2023, 9:32 AM

lbowmaker closed this task as Resolved.Jan 8 2024, 7:50 PM

lbowmaker closed subtask T338231: [Event Platform] mw-page-content-change-enrich should (re)produce kafka keys as Resolved.

[Event Platform] mediawiki.page_content_change.v1 topic should be partitioned.
Closed, ResolvedPublic
Actions

Description

Success criteria

Dependencies

Details

Related Objects
Search...

Event Timeline

[Event Platform] mediawiki.page_content_change.v1 topic should be partitioned.Closed, ResolvedPublicActions