Page MenuHomePhabricator
Create Task
Maniphest T307959

[Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content
Closed, ResolvedPublic
Actions

Description

User Story
As a platform engineer, I need to design, implement and deploy a streaming job that produces event streams of mediawiki page changes with raw content.
The service willl:
  • Call MW API to get the wikitext for the article
  • Format the input stream data and wikitext into the new topic format
  • Output the formatted data to a new Kafka topic
Expected Spikes:
  • Data modeling exercise for new consolidated stream - T308017
Why are we doing this?
  • Simplify event stream consumption. Consumers can listen to a single stream that represent the state of a page rather than a page action (current design)
  • Adding content to streams to make them usable by consumers without having to enrich themselves
What is needed for GA internal release
Follow up work that needs to be done

Details

SubjectRepoBranchLines +/-
Increase the kafka-jumbo maximum message size to 10 MBoperations/puppetproduction+2 -0
Increase the max kafka message size for gobblinanalytics/refinerymaster+4 -0
Remove rc1.mediawiki.page_content_change streamoperations/mediawiki-configmaster+0 -17
data-engineering: flink: alert when TM is missing for 5m.operations/alertsmaster+1 -0
mw-page-content-change-enrich: stream version bumpoperations/deployment-chartsmaster+1 -1
Declare v1 of the page_content_change stream.operations/mediawiki-configmaster+17 -0
Set eventgate service for rc0.mediawiki.page_content_change streamoperations/mediawiki-configmaster+4 -0
Declare rc0.mediawiki.page_content_change streamoperations/mediawiki-configmaster+11 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedgmodenaT307959 [Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content
 ResolvedOttomataT308017 Design Schema for page state and page state with content (enriched) streams
 ResolvedOttomataT310082 [Shared Event Platform] - Research Flink Changelog semantics to inform POC MW schema design
DeclinedNoneT336506 mediawiki/page/change event schema - Use single array field for user attributes instead of boolean fields
 ResolvedOttomataT308356 [Shared Event Platform] Ability to use Event Platform streams in Flink without boilerplate
 ResolvedOttomataT310302 Add better support for using Event Platform streams with the Flink DataStream API
 ResolvedOttomataT310495 Add Event Platform timestamp JSONSchema -> Flink type support
 ResolveddcausseT310218 Flink output support for Event Platform events
 ResolvedgmodenaT309699 [Event Platform] Understand, document, and implement error handling and retry logic when fetching data from the MW api
 ResolvedgmodenaT310626 [Shared Event Platform] Rename scala package and setup a multi module maven project.
 ResolvedgmodenaT311084 [Shared Event Platform] Mediawiki Stream Enrichment should consume the consolidated page-change stream.
 ResolvedOttomataT311129 [Shared Event Platform] Produce new mediawiki.page-change stream from MediaWiki EventBus
 OpenNoneT321411 Add $comment and $performer to ArticleRevisionVisibilitySet params
 ResolvedOwenRBT321412 Create PageUndeleteComplete hook, analogous to PageDeleteComplete
 ResolvedOwenRBT328308 Use new PageUndeleteComplete hook to emit mediawiki.page_change undelete event
 ResolvedOttomataT325303 Deploy mediawiki-page-content-change-enrichment to wikikube k8s
 ResolvedgmodenaT325304 Deploy to YARN
 ResolvedOttomataT325305 Deploy mediawiki-event-enrichment flink app to DSE k8s
 ResolvedOttomataT328925 Flink App Deployment monitoring
 ResolvedOttomataT337496 Flink metrics should use a consistent label for each job
 DuplicateNoneT325307 Deploy to production k8s
 ResolvedgmodenaT326731 Deployment pipeline docker image of flink mediawiki stream enrichment pyhon
 ResolvedOttomataT330507 New Service Request mediawiki-page-content-change-enrichment
 ResolvedOttomataT333464 New Service Request: flink-kubernetes-operator
 ResolvedEevansT330693 Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing
 ResolvedgmodenaT336656 mediawiki-page-content-change-enrichment checkpoints should be stored in Swift
 ResolvedOttomataT331282 Upgrade flink-kubernetes-operator to 1.4.0
 OpenNoneT331283 [Event Platform] [NEEDS GROOMING] Store Flink HA metadata in Zookeeper
 DeclinedNoneT341137 Test version compatibility between production Kafka and newer ZooKeeper
 ResolvedbkingT341792 Provision Zookeeper Cluster for storing Flink HA data
 ResolvedbkingT341705 eqiad: 3 VMs requested for Zookeeper
 ResolvedNoneT343715 codfw: 3 VMs requested for Zookeeper
 DeclinedNoneT331417 mediawiki-page-content-change-enrichment should configurably sample events
 ResolvedgmodenaT331526 eventutilities-python should support using Kafka TLS ports
 ResolvedgmodenaT333833 Define Service Level Objective (SLO) for mediawiki-page-content-change-enrichment
 ResolvedgmodenaT336817 Release mediawiki.page_change.v1 stream
 ResolvedgmodenaT338169 mw-page-content-change-enrich should partition by and process by wiki_id,page_id
 ResolvedgmodenaT338231 [Event Platform] mw-page-content-change-enrich should (re)produce kafka keys
 ResolvedgmodenaT338233 mw-page-content-change-enrich should enable HA with k8s ConfigMaps
 ResolvedgmodenaT340666 mediawiki-event-enrichment jobs alerting
 ResolvedBTullisT344688 Increase Max Message Size in Kafka Jumbo

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Nov 3 2022, 3:08 PM

Change 852909 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] Set eventgate service for rc0.mediawiki.page_content_change stream

https://gerrit.wikimedia.org/r/852909

gerritbot added a project: Patch-For-Review.Nov 3 2022, 3:08 PM
gerritbot added a comment.Nov 3 2022, 3:11 PM

Change 852909 merged by jenkins-bot:

[operations/mediawiki-config@master] Set eventgate service for rc0.mediawiki.page_content_change stream

https://gerrit.wikimedia.org/r/852909

Maintenance_bot removed a project: Patch-For-Review.Nov 3 2022, 3:31 PM
Ottomata reopened subtask T308017: Design Schema for page state and page state with content (enriched) streams as Open.Nov 9 2022, 9:32 PM
JArguello-WMF closed subtask T311084: [Shared Event Platform] Mediawiki Stream Enrichment should consume the consolidated page-change stream. as Resolved.Nov 28 2022, 1:39 PM
Ottomata renamed this task from [Shared Event Platform] Design and Implement POC Flink Service to Combine Existing Streams, Enrich and Output to New Topic to [Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content.Jan 23 2023, 6:18 PM
Ottomata updated the task description. (Show Details)
gmodena added a comment.Feb 7 2023, 9:18 AM

I archived the mediawiki-stream-enrichment repo associated with this task. That contained a Scala PoC that help us inform the current (WIP) Python implementation deployed on DSE.

lbowmaker closed subtask T308017: Design Schema for page state and page state with content (enriched) streams as Resolved.Feb 17 2023, 2:50 PM
Ottomata added a subtask: T325303: Deploy mediawiki-page-content-change-enrichment to wikikube k8s.May 16 2023, 9:38 PM
Ottomata updated the task description. (Show Details)Jun 5 2023, 4:49 PM
Ottomata updated the task description. (Show Details)Jun 6 2023, 1:54 PM
Ottomata moved this task from Backlog to Parent Tasks/Epics on the Event-Platform board.
Ottomata updated the task description. (Show Details)Jun 6 2023, 4:03 PM
Ottomata updated the task description. (Show Details)Jun 6 2023, 4:06 PM
Ottomata updated the task description. (Show Details)
xcollazo subscribed.Jun 7 2023, 2:09 PM
JArguello-WMF closed subtask T325303: Deploy mediawiki-page-content-change-enrichment to wikikube k8s as Resolved.Jun 26 2023, 12:59 PM
Ottomata updated the task description. (Show Details)Jun 26 2023, 8:42 PM
Ottomata updated the task description. (Show Details)Jun 28 2023, 3:08 PM
Ottomata mentioned this in T340666: mediawiki-event-enrichment jobs alerting.Jun 28 2023, 3:33 PM
Ottomata updated the task description. (Show Details)
JArguello-WMF closed subtask T338233: mw-page-content-change-enrich should enable HA with k8s ConfigMaps as Resolved.Jun 29 2023, 1:56 PM
JArguello-WMF closed subtask T309699: [Event Platform] Understand, document, and implement error handling and retry logic when fetching data from the MW api as Resolved.
JArguello-WMF moved this task from Epics to Event Platform on the Data-Engineering-Planning board.Jun 29 2023, 9:43 PM
JArguello-WMF removed a project: Data-Engineering-Planning.Jun 29 2023, 9:49 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptJun 29 2023, 9:49 PM
JArguello-WMF moved this task from Incoming (new tickets) to Event Platform Backlog on the Data-Engineering board.Jun 29 2023, 10:30 PM
gmodena updated the task description. (Show Details)Jun 30 2023, 10:39 AM
gmodena updated the task description. (Show Details)Jun 30 2023, 11:35 AM
JArguello-WMF edited projects, added Data Engineering and Event Platform Team; removed Data-Engineering.Jun 30 2023, 4:15 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptJun 30 2023, 4:15 PM
JArguello-WMF moved this task from Data Eng Backlog to Parent Tasks/Epics on the Data Engineering and Event Platform Team board.Jun 30 2023, 4:17 PM
JArguello-WMF removed a project: Data-Engineering.Jun 30 2023, 4:19 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptJun 30 2023, 4:19 PM
Ottomata updated the task description. (Show Details)Jun 30 2023, 4:52 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)Jun 30 2023, 4:56 PM
gmodena updated the task description. (Show Details)Jul 5 2023, 10:35 AM
Ottomata updated the task description. (Show Details)Jul 7 2023, 2:35 PM
gmodena updated the task description. (Show Details)Jul 13 2023, 11:18 AM
gmodena updated the task description. (Show Details)Jul 13 2023, 11:21 AM
gmodena updated the task description. (Show Details)
gmodena updated the task description. (Show Details)Jul 20 2023, 1:47 PM
lbowmaker closed subtask T340666: mediawiki-event-enrichment jobs alerting as Resolved.Jul 25 2023, 12:21 PM
lbowmaker closed subtask T338169: mw-page-content-change-enrich should partition by and process by wiki_id,page_id as Resolved.
lbowmaker edited projects, added Data Engineering and Event Platform Team (Sprint 1); removed Data Engineering and Event Platform Team.Aug 18 2023, 11:57 AM
lbowmaker moved this task from Next Up to In progress on the Data Engineering and Event Platform Team (Sprint 1) board.
gerritbot added a comment.Aug 22 2023, 10:47 AM

Change 951444 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/mediawiki-config@master] Declare v1 of the page_content_change stream.

https://gerrit.wikimedia.org/r/951444

gerritbot added a project: Patch-For-Review.Aug 22 2023, 10:47 AM
gerritbot added a comment.Aug 22 2023, 11:00 AM

Change 951446 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/deployment-charts@master] mw-page-content-change-enrich: stream version bump

https://gerrit.wikimedia.org/r/951446

gerritbot added a comment.Aug 22 2023, 8:12 PM

Change 951444 merged by jenkins-bot:

[operations/mediawiki-config@master] Declare v1 of the page_content_change stream.

https://gerrit.wikimedia.org/r/951444

Stashbot added a comment.Aug 22 2023, 8:16 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T20:16:03Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:951444|Declare v1 of the page_content_change stream. (T307959)]]

Stashbot added a comment.Aug 22 2023, 8:17 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T20:17:34Z] <urbanecm@deploy1002> urbanecm and gmodena: Backport for [[gerrit:951444|Declare v1 of the page_content_change stream. (T307959)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Stashbot added a comment.Aug 22 2023, 8:27 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T20:27:22Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:951444|Declare v1 of the page_content_change stream. (T307959)]] (duration: 11m 19s)

gerritbot added a comment.Aug 23 2023, 12:04 PM

Change 951446 merged by jenkins-bot:

[operations/deployment-charts@master] mw-page-content-change-enrich: stream version bump

https://gerrit.wikimedia.org/r/951446

Maintenance_bot removed a project: Patch-For-Review.Aug 23 2023, 12:10 PM
gmodena updated the task description. (Show Details)Aug 23 2023, 1:46 PM
gerritbot added a comment.Aug 23 2023, 1:48 PM

Change 951929 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/mediawiki-config@master] Remove rc1.mediawiki.page_content_change stream

https://gerrit.wikimedia.org/r/951929

gerritbot added a project: Patch-For-Review.Aug 23 2023, 1:48 PM
gerritbot added a comment.Aug 23 2023, 3:30 PM

Change 951959 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/alerts@master] data-engineering: flink: alert when TM is missing for 5m.

https://gerrit.wikimedia.org/r/951959

gmodena moved this task from In progress to In Review on the Data Engineering and Event Platform Team (Sprint 1) board.Aug 23 2023, 3:36 PM
gerritbot added a comment.Aug 24 2023, 10:23 AM

Change 952160 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Increase the kafka-jumbo maximum message size to 10 MB

https://gerrit.wikimedia.org/r/952160

gerritbot added a comment.Aug 24 2023, 3:51 PM

Change 951959 merged by jenkins-bot:

[operations/alerts@master] data-engineering: flink: alert when TM is missing for 5m.

https://gerrit.wikimedia.org/r/951959

gerritbot added a comment.Aug 31 2023, 1:13 PM

Change 951929 merged by jenkins-bot:

[operations/mediawiki-config@master] Remove rc1.mediawiki.page_content_change stream

https://gerrit.wikimedia.org/r/951929

Stashbot added a comment.Aug 31 2023, 1:14 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-31T13:14:28Z] <sgimeno@deploy1002> Started scap: Backport for [[gerrit:951929|Remove rc1.mediawiki.page_content_change stream (T307959)]]

Stashbot added a comment.Aug 31 2023, 1:16 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-31T13:16:03Z] <sgimeno@deploy1002> gmodena and sgimeno: Backport for [[gerrit:951929|Remove rc1.mediawiki.page_content_change stream (T307959)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Stashbot added a comment.Aug 31 2023, 1:25 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-31T13:25:01Z] <sgimeno@deploy1002> Finished scap: Backport for [[gerrit:951929|Remove rc1.mediawiki.page_content_change stream (T307959)]] (duration: 10m 33s)

gerritbot added a comment.Sep 5 2023, 4:38 PM

Change 954968 had a related patch set uploaded (by Btullis; author: Btullis):

[analytics/refinery@master] Increase the max kafka message size for gobblin

https://gerrit.wikimedia.org/r/954968

gerritbot added a comment.Sep 8 2023, 8:32 AM

Change 954968 abandoned by Joal:

[analytics/refinery@master] Increase the max kafka message size for gobblin

Reason:

Change not actually needed.

https://gerrit.wikimedia.org/r/954968

gmodena updated the task description. (Show Details)Sep 8 2023, 10:39 AM
lbowmaker moved this task from Sprint 1 to Sprint 2 on the Data Engineering and Event Platform Team board.Sep 8 2023, 1:29 PM
lbowmaker edited projects, added Data Engineering and Event Platform Team (Sprint 2); removed Data Engineering and Event Platform Team (Sprint 1).
lbowmaker moved this task from Next Up to In Review on the Data Engineering and Event Platform Team (Sprint 2) board.
lbowmaker moved this task from Sprint 2 to Event Platform Backlog on the Data Engineering and Event Platform Team board.Sep 11 2023, 3:32 PM
lbowmaker edited projects, added Data Engineering and Event Platform Team; removed Data Engineering and Event Platform Team (Sprint 2).
gerritbot added a comment.Sep 19 2023, 9:28 AM

Change 952160 merged by Btullis:

[operations/puppet@production] Increase the kafka-jumbo maximum message size to 10 MB

https://gerrit.wikimedia.org/r/952160

Maintenance_bot removed a project: Patch-For-Review.Sep 19 2023, 9:31 AM
BTullis closed subtask T344688: Increase Max Message Size in Kafka Jumbo as Resolved.Oct 10 2023, 2:07 PM
lbowmaker moved this task to Event Platform Maintenance (current quarter) on the Data Engineering and Event Platform Team board.Oct 20 2023, 2:43 PM
Ottomata added a comment.Oct 23 2023, 3:04 PM

Can we resolve this task?

lbowmaker removed a project: Data Engineering and Event Platform Team.Nov 10 2023, 1:40 PM
Ottomata updated the task description. (Show Details)Nov 21 2023, 7:21 PM
Ottomata added a comment.Feb 19 2024, 3:39 PM

@lbowmaker @gmodena Should we resolve and close this?

lbowmaker closed this task as Resolved.Feb 19 2024, 3:54 PM

Quoted Text

In T307959#9555401, @Ottomata wrote:

@lbowmaker @gmodena Should we resolve and close this?

Yeah!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL