Page MenuHomePhabricator
Create Task
Maniphest T325303

Deploy mediawiki-page-content-change-enrichment to wikikube k8s
Closed, ResolvedPublic
Actions

Description

User Story
As a platform engineer I need to deploy the PyFlink stream enrichment service to wikikube k8s
Why?

So that it can be deployed to production to help us understand how well things scale

Dependencies:
  • Finalizing page content change schema
Expected Sub Tasks:

Details

SubjectRepoBranchLines +/-
Remove dse mediawiki-page-content-change-enrichment and stream-enrichment-poc nsoperations/deployment-chartsmaster+0 -153
mw-page-content-change-enrich - use kafka at least once delivery guaranteeoperations/deployment-chartsmaster+2 -1
mw-page-content-change-enrich - enable upgradeMode: savepoint, and take periodic savepoints.operations/deployment-chartsmaster+11 -3
mw-page-content-change-enrich - bump image to 1.20.0operations/deployment-chartsmaster+1 -1
eventutillities-flink - kafkaSourceBuilder now allows for specific topicswikimedia-event-utilitiesmaster+76 -13
New wikikube service: mediawiki-page-content-change-enrichment - stagingoperations/deployment-chartsmaster+185 -59
Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
flink - kafka_sink delivery_guarantee optionrepos/data-engineering/eventutilities-python!63ottokafka_sink_delivery_guaranteemain
Fix bug where kafka topics could not be set on sourcerepos/data-engineering/eventutilities-python!59ottoflink_kafka_topicsmain
kafka_source - Allow specific topics to be setrepos/data-engineering/eventutilities-python!54ottokafka_source_topicsmain
Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedgmodenaT307959 [Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content
 ResolvedOttomataT325303 Deploy mediawiki-page-content-change-enrichment to wikikube k8s
 ResolvedgmodenaT325304 Deploy to YARN
 ResolvedOttomataT325305 Deploy mediawiki-event-enrichment flink app to DSE k8s
 ResolvedOttomataT328925 Flink App Deployment monitoring
 ResolvedOttomataT337496 Flink metrics should use a consistent label for each job
 DuplicateNoneT325307 Deploy to production k8s
 ResolvedgmodenaT326731 Deployment pipeline docker image of flink mediawiki stream enrichment pyhon
 ResolvedOttomataT330507 New Service Request mediawiki-page-content-change-enrichment
 ResolvedOttomataT333464 New Service Request: flink-kubernetes-operator
 ResolvedEevansT330693 Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing
 ResolvedgmodenaT336656 mediawiki-page-content-change-enrichment checkpoints should be stored in Swift
 ResolvedOttomataT331282 Upgrade flink-kubernetes-operator to 1.4.0
 OpenNoneT331283 [Event Platform] [NEEDS GROOMING] Store Flink HA metadata in Zookeeper
 DeclinedNoneT341137 Test version compatibility between production Kafka and newer ZooKeeper
 ResolvedbkingT341792 Provision Zookeeper Cluster for storing Flink HA data
 ResolvedbkingT341705 eqiad: 3 VMs requested for Zookeeper
 ResolvedNoneT343715 codfw: 3 VMs requested for Zookeeper
 DeclinedNoneT331417 mediawiki-page-content-change-enrichment should configurably sample events
 ResolvedgmodenaT331526 eventutilities-python should support using Kafka TLS ports
 ResolvedgmodenaT333833 Define Service Level Objective (SLO) for mediawiki-page-content-change-enrichment
 ResolvedgmodenaT336817 Release mediawiki.page_change.v1 stream

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a project: Data-Engineering. · View Herald TranscriptDec 15 2022, 3:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gmodena added a comment.Dec 22 2022, 9:07 AM

I did some initial/prep to add support for EventRowTypeInfo to eventutilities-python. This allows us to reuse all Flink Row` SerDe, Sink and validation from Java enventutils. This is also true for the Source
side of the pipeline.

Right now there's a pyflink version of the mediawiki stream enrichment pipeline deployed on YARN that produces page_content_change into kafka, effectively superseding the Scala one. Deployment is documented
at https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream/Pyflink_Enrichment_Service_Deployment.

• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Jan 6 2023, 11:00 AM
• EChetty moved this task from Backlog to Event Platform on the Data-Engineering-Planning board.Jan 6 2023, 11:23 AM
gmodena mentioned this in T325304: Deploy to YARN.Jan 30 2023, 2:24 PM
lbowmaker closed subtask T325304: Deploy to YARN as Resolved.Jan 30 2023, 3:25 PM
JArguello-WMF closed subtask T326731: Deployment pipeline docker image of flink mediawiki stream enrichment pyhon as Resolved.Feb 7 2023, 1:44 PM
Ottomata updated the task description. (Show Details)Feb 7 2023, 1:59 PM
lbowmaker assigned this task to Ottomata.Feb 8 2023, 3:57 PM
lbowmaker updated the task description. (Show Details)
lbowmaker updated the task description. (Show Details)
Ottomata renamed this task from Productionize PyFlink Enrichment Service to Deploy mediawiki-page-content-change-enrichment to wikikube k8s.Feb 8 2023, 3:59 PM
Ottomata merged a task: T325307: Deploy to production k8s.
lbowmaker closed subtask T325305: Deploy mediawiki-event-enrichment flink app to DSE k8s as Resolved.Feb 17 2023, 2:48 PM
Ottomata added subscribers: JMeybohm, akosiaris.Feb 23 2023, 3:58 PM

@JMeybohm @akosiaris, we plan to deploy to wikikube by the end of this quarter (end of March). Are there any blockers in wikikube to creating new namespaces and deploying the flink-kubernetes operator there?

akosiaris added a comment.Feb 24 2023, 10:38 AM
In T325303#8641318, @Ottomata wrote:

@JMeybohm @akosiaris, we plan to deploy to wikikube by the end of this quarter (end of March). Are there any blockers in wikikube to creating new namespaces and deploying the flink-kubernetes operator there?

Make sure to file the paperwork via https://phabricator.wikimedia.org/project/profile/1305/ but no, we don't anticipate any blockers end of March. An estimation of resources required would be needed.
Mid march (no definite date yet) we will probably be re-initializing the wikikube eqiad cluster though. Keep that in mind.

Ottomata added a comment.Feb 24 2023, 2:58 PM

Done: T330507: New Service Request mediawiki-page-content-change-enrichment

Ottomata updated the task description. (Show Details)Feb 27 2023, 2:20 PM
Ottomata added a subtask: T330507: New Service Request mediawiki-page-content-change-enrichment.Feb 28 2023, 1:12 PM
JArguello-WMF added a project: Data Pipelines (Sprint 11).Mar 2 2023, 2:15 PM
JArguello-WMF edited projects, added Event-Platform (Sprint 09); removed Data Pipelines (Sprint 11), Event-Platform.
JArguello-WMF moved this task from Next Up to In Progress on the Event-Platform (Sprint 09) board.
JArguello-WMF removed a project: Epic.Mar 2 2023, 2:17 PM
JArguello-WMF added a project: Epic.
JArguello-WMF edited projects, added Event-Platform; removed Event-Platform (Sprint 09).Mar 2 2023, 2:26 PM
JArguello-WMF moved this task from Backlog to Parent Tasks/Epics on the Event-Platform board.Mar 2 2023, 2:29 PM
Ottomata updated the task description. (Show Details)Mar 6 2023, 11:48 AM
akosiaris added a comment.Mar 7 2023, 7:49 AM

By the way, which eventgate (and thus which kafka cluster) will this produce to?

Ottomata added a comment.Mar 7 2023, 10:06 AM

This doesn't produce to EventGate, it produces to Kafka directly. It will produce to Kafka main clusters.

akosiaris added a comment.Mar 7 2023, 11:13 AM
In T325303#8671772, @Ottomata wrote:

This doesn't produce to EventGate, it produces to Kafka directly. It will produce to Kafka main clusters.

Thanks. Are the kafka-main clusters ready to receive this ? serviceops owns them but I don't think we can answer that question easily right now.

Ottomata added a comment.Mar 7 2023, 11:23 AM

Good question. I think so, but we should consult.

Perhaps, we should just continue producing to Kafka jumbo-eqiad from wikikube until we feel ready for kafka main. However, this means that the app in wikikube codfw would produce cross DC. I think this will be fine as we are still in 'release candidate' mode...and MirrorMaker already replicates all data in codfw -> eqiad cross DC anyway.

Ottomata added a subscriber: Milimetric.EditedMar 7 2023, 12:38 PM

We did some research on this stream's volume as part of T307944: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP.

I'll start with @Milimetric's handy numbers from a comment below. For 2021, the average revision size was 20623 bytes. (The average is quite inflated by big outliers). Let's just round this up and say 30000 bytes. In 2021 there were 533764132 revisions, so 533764132*30000/365/24/60/60 == 507767 bytes / second. Compared with what kafka main is already doing (<5 MB / second) adding a revision text event stream wouldn't be a significant addition of traffic. I suppose if some bot edits e.g. 2MB pages 10 or 100 times a second this could matter though.

Assuming growth, more use cases, [...], let's just round this up to 10MB per second.

Um, not sure why I wrote 10MB / second in that task. I should have written "round this up to 1MB / second", but more realistically it will be around 0.5MB / second.

Messages throughput will be the same as mediawiki.page_change now, ~30 messages/second

There main worry is outliers of large messages. Here's some context for new search pipeline design.
There is a list of max and average revision sizes per year here.

Kafka main is configured to reject messages > 4MB. We will be using (snappy) compression. There may still be cases where we try to produces messages with content larger than 4MB. We can discuss later if we should increase this limit in Kafka, but for now, we should keep it. We may choose to deal with these large messages differently (not producing them, but instead producing with a URI pointer to content).

Ottomata closed subtask T331417: mediawiki-page-content-change-enrichment should configurably sample events as Declined.Mar 7 2023, 2:54 PM
gerritbot added a comment.Mar 7 2023, 3:03 PM

Change 895241 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] New wikikube service: mediawiki-page-content-change-enrichment - staging

https://gerrit.wikimedia.org/r/895241

gerritbot added a project: Patch-For-Review.Mar 7 2023, 3:03 PM
JArguello-WMF closed subtask T331282: Upgrade flink-kubernetes-operator to 1.4.0 as Resolved.Mar 13 2023, 2:12 PM
JArguello-WMF triaged this task as High priority.May 15 2023, 3:20 PM
JArguello-WMF edited projects, added Event-Platform (Sprint 14 A); removed Event-Platform.
Ottomata moved this task from Next Up to In progress on the Event-Platform (Sprint 14 A) board.May 15 2023, 3:27 PM
Ottomata mentioned this in T336755: Allow gmodena and tchin to merge changes to operation/deployment-charts repo.May 16 2023, 12:02 PM
gerritbot added a comment.May 16 2023, 3:12 PM

Change 895241 merged by jenkins-bot:

[operations/deployment-charts@master] New wikikube service: mediawiki-page-content-change-enrichment - staging

https://gerrit.wikimedia.org/r/895241

Maintenance_bot removed a project: Patch-For-Review.May 16 2023, 3:30 PM
gerritbot added a comment.May 16 2023, 9:15 PM

Change 920377 had a related patch set uploaded (by Ottomata; author: Ottomata):

[mediawiki/extensions/EventBus@master] Change default page_change stream name to use major versioning

https://gerrit.wikimedia.org/r/920377

gerritbot added a project: Patch-For-Review.May 16 2023, 9:15 PM
Ottomata updated the task description. (Show Details)May 16 2023, 9:33 PM
Ottomata added a parent task: T307959: [Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content.May 16 2023, 9:38 PM
Ottomata updated the task description. (Show Details)May 18 2023, 1:03 PM
Ottomata updated the task description. (Show Details)
CodeReviewBot added a comment.May 22 2023, 3:27 PM

otto opened https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/54

kafka_source - Allow specific topics to be set

CodeReviewBot added a comment.May 22 2023, 6:34 PM

otto merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/54

kafka_source - Allow specific topics to be set

gerritbot added a comment.May 31 2023, 2:15 PM

Change 924956 had a related patch set uploaded (by Ottomata; author: Ottomata):

[wikimedia-event-utilities@master] eventutillities-flink - kafkaSourceBuilder now allows for specific topics

https://gerrit.wikimedia.org/r/924956

CodeReviewBot added a comment.May 31 2023, 2:53 PM

otto opened https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/59

Fix bug where kafka topics could not be set on source

gerritbot added a comment.May 31 2023, 2:56 PM

Change 924956 merged by Ottomata:

[wikimedia-event-utilities@master] eventutillities-flink - kafkaSourceBuilder now allows for specific topics

https://gerrit.wikimedia.org/r/924956

CodeReviewBot added a comment.May 31 2023, 3:06 PM

otto merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/59

Fix bug where kafka topics could not be set on source

gerritbot added a comment.May 31 2023, 3:44 PM

Change 924979 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] mw-page-content-change-enrich - bump image to 1.12.0

https://gerrit.wikimedia.org/r/924979

gerritbot added a comment.May 31 2023, 4:03 PM

Change 924979 merged by Ottomata:

[operations/deployment-charts@master] mw-page-content-change-enrich - bump image to 1.20.0

https://gerrit.wikimedia.org/r/924979

Ottomata updated the task description. (Show Details)May 31 2023, 4:14 PM
Ottomata updated the task description. (Show Details)
gerritbot added a comment.Jun 2 2023, 6:50 PM

Change 926601 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] mw-page-content-change-enrich - enable upgradeMode: savepoint, and take periodic savepoints.

https://gerrit.wikimedia.org/r/926601

CodeReviewBot added a comment.Jun 2 2023, 7:26 PM

otto opened https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/63

flink - kafka_sink delivery_guarantee option

JArguello-WMF closed subtask T333833: Define Service Level Objective (SLO) for mediawiki-page-content-change-enrichment as Resolved.Jun 5 2023, 11:31 AM
JArguello-WMF closed subtask T333464: New Service Request: flink-kubernetes-operator as Resolved.
JArguello-WMF closed subtask T330507: New Service Request mediawiki-page-content-change-enrichment as Resolved.
JArguello-WMF closed subtask T331526: eventutilities-python should support using Kafka TLS ports as Resolved.
JArguello-WMF edited projects, added Event-Platform (Sprint 14 B); removed Event-Platform (Sprint 14 A).Jun 5 2023, 11:43 AM
JArguello-WMF moved this task from Next Up to In progress on the Event-Platform (Sprint 14 B) board.
gerritbot added a comment.Jun 5 2023, 1:23 PM

Change 926601 merged by jenkins-bot:

[operations/deployment-charts@master] mw-page-content-change-enrich - enable upgradeMode: savepoint, and take periodic savepoints.

https://gerrit.wikimedia.org/r/926601

CodeReviewBot added a comment.Jun 5 2023, 1:53 PM

otto merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/63

flink - kafka_sink delivery_guarantee option

gerritbot added a comment.Jun 5 2023, 2:41 PM

Change 927219 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] mw-page-content-change-enrich - use kafka at least once delivery guarantee

https://gerrit.wikimedia.org/r/927219

gerritbot added a comment.Jun 5 2023, 2:44 PM

Change 927224 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] Remove dse mediawiki-page-content-change-enrichment and stream-enrichment-poc ns

https://gerrit.wikimedia.org/r/927224

gerritbot added a comment.Jun 5 2023, 2:49 PM

Change 927219 merged by Ottomata:

[operations/deployment-charts@master] mw-page-content-change-enrich - use kafka at least once delivery guarantee

https://gerrit.wikimedia.org/r/927219

JArguello-WMF moved this task from In progress to Done on the Event-Platform (Sprint 14 B) board.Jun 5 2023, 3:10 PM
JArguello-WMF moved this task from Done to In Review on the Event-Platform (Sprint 14 B) board.
Ottomata updated the task description. (Show Details)Jun 6 2023, 4:01 PM
gerritbot added a comment.Jun 20 2023, 5:40 PM

Change 927224 merged by jenkins-bot:

[operations/deployment-charts@master] Remove dse mediawiki-page-content-change-enrichment and stream-enrichment-poc ns

https://gerrit.wikimedia.org/r/927224

Stashbot added a comment.Jun 20 2023, 5:44 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-20T17:44:41Z] <ottomata> remove stream-enrichment-poc namespace and related resources from dse-k8s-eqiad - T325303

Ottomata updated the task description. (Show Details)Jun 20 2023, 7:20 PM
Ottomata moved this task from In Review to Done on the Event-Platform (Sprint 14 B) board.
Ottomata moved this task from Done to In Review on the Event-Platform (Sprint 14 B) board.Jun 21 2023, 1:01 PM
Ottomata moved this task from In Review to Done on the Event-Platform (Sprint 14 B) board.
JArguello-WMF closed this task as Resolved.Jun 26 2023, 12:59 PM
lbowmaker closed subtask T336817: Release mediawiki.page_change.v1 stream as Resolved.Sep 8 2023, 1:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL