[SPIKE] Prototype of incremental updates for mediawiki history for simplewiki , including reverts using apache hudi
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Nuria
	Jul 21 2020, 8:35 PM

Description

Taking very TDD approach here see if with hudi we can bridge two mediawiki snapshots for simplewiki when it comes to page reverst for example, this means we are going to use one of the event feeds and try to rebuild reverts with incoming data.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	[WIP] Spike so far	analytics/refinery/source	master	+112 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T204950 Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users
Open	None	T258511 Data Lake incremental Data Updates
Resolved	None	T231938 Get "edits hourly" on a daily basis
Resolved	Milimetric	T258532 [SPIKE] Prototype of incremental updates for mediawiki history for simplewiki , including reverts using apache hudi
Declined	None	T262205 Need for new event-type - `user_create` and `user_rename`
Resolved	JAllemandou	T262256 Test hudi and Iceberg as an incremental update system using 2 mediawiki-history snapshots
Declined	None	T262260 Make hudi work with Hive
Resolved	JAllemandou	T262261 Check whether mediawiki production event data is equivalent to mediawiki-history data over a month
Resolved	Milimetric	T215001 Revisions missing from mediawiki_revision_create
Resolved	None	T280538 Capture rev_is_revert event data in a stream different than mediawiki.revision-create
Declined	Milimetric	T263055 Add log entry details to page and user events in EventBus

Event Timeline

• Nuria created this task.Jul 21 2020, 8:35 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2020, 8:35 PM

• Nuria mentioned this in T240387: MW REST API Historical Data Endpoint Needs.Jul 21 2020, 8:40 PM

Milimetric triaged this task as High priority.Jul 31 2020, 3:32 PM

Milimetric moved this task from Incoming to Smart Tools for Better Data on the Analytics board.

Milimetric added a project: Analytics-Kanban.

Milimetric moved this task from Next Up to In Progress on the Analytics-Kanban board.Aug 3 2020, 4:14 PM

Change 618874 had a related patch set uploaded (by Milimetric; owner: Milimetric):
[analytics/refinery/source@master] [WIP] Spike so far

https://gerrit.wikimedia.org/r/618874

gerritbot added a project: Patch-For-Review.Aug 7 2020, 2:03 AM

JAllemandou mentioned this in T262203: Duplicated revision_create events.Sep 7 2020, 12:34 PM

JAllemandou added a subtask: T262205: Need for new event-type - `user_create` and `user_rename`.Sep 7 2020, 12:56 PM

Milimetric moved this task from In Progress to Done on the Analytics-Kanban board.Oct 21 2020, 3:00 PM

• fdans closed this task as Resolved.Oct 29 2020, 9:22 PM

• fdans closed subtask T262261: Check whether mediawiki production event data is equivalent to mediawiki-history data over a month as Resolved.Nov 16 2020, 3:51 PM

• fdans closed subtask T262256: Test hudi and Iceberg as an incremental update system using 2 mediawiki-history snapshots as Resolved.Mar 18 2021, 4:02 PM

Milimetric closed subtask T262260: Make hudi work with Hive as Declined.Oct 28 2021, 9:07 PM

Milimetric closed subtask T263055: Add log entry details to page and user events in EventBus as Declined.Oct 28 2021, 9:10 PM

Milimetric closed subtask T262205: Need for new event-type - `user_create` and `user_rename` as Declined.Sep 20 2022, 9:19 PM

[SPIKE] Prototype of incremental updates for mediawiki history for simplewiki , including reverts using apache hudi Closed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

[SPIKE] Prototype of incremental updates for mediawiki history for simplewiki , including reverts using apache hudi
Closed, ResolvedPublic
Actions

Related Objects
Search...