[Spike] Spark job for digests-only mediawiki-history-reduced
Closed, ResolvedPublic
Actions

Description

This task is about moving the mediawiki_history_reduced dataset from using raw-events and digests-events to digests-events only.
Optimizations are needed for the dataset not to grow exponentially with dimension-denormalization, and for the computation not to generate the overly-big intermediate dataset.

Details

	Project	Branch	Lines +/-	Subject
	analytics/refinery/source	master	+686 -0	[SPIKE] Add mediawiki-reduced spark job

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T120037 Vital Signs: Please provide an "all languages" de-duplicated stream for the Community/Content groups of metrics
Resolved	None	T120036 Vital Signs: Please make the data for enwiki and other big wikis less sad, and not just be missing for most days
Open	None	T130256 Wikistats 2.0.
Resolved	mforns	T186120 Wikistats Beta
Open	None	T186121 Wikistats 2.0 Remaining reports.
Resolved	None	T188550 Wikistats 2.0: allow to view stats for all language versions (a.k.a. Project families)
Open	None	T188265 Active Editors metric per project family
Open	None	T249751 Define reduce calculations needed to compute active editors per project family
Resolved	fdans	T189882 Create reports in wikistats UI for "most prolific editors" (a.k.a "top contributors")
Resolved	JAllemandou	T201617 Add AQS endpoint providing top editors ("most prolific contributors") and top pages (by number of edits, by net-bytes-diff and abs-bytes diff)
Open	None	T204965 Create report for "articles with most contributors" in Wikistats2
Resolved	JAllemandou	T212928 [Spike] Spark job for digests-only mediawiki-history-reduced
Resolved	Amitjoki	T186335 Wikistat Beta: expand topic explorer by default
Duplicate	Erik_Zachte	T191555 stats.wikimedia.org home page should link to wikistats 2

Event Timeline

JAllemandou created this task.Jan 4 2019, 9:56 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 4 2019, 9:56 AM

JAllemandou claimed this task.Jan 4 2019, 9:56 AM

JAllemandou added parent tasks: T204965: Create report for "articles with most contributors" in Wikistats2, T188265: Active Editors metric per project family .

JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.

Milimetric triaged this task as High priority.Jan 7 2019, 4:35 PM

Milimetric moved this task from Incoming to Smart Tools for Better Data on the Analytics board.

JAllemandou moved this task from In Progress to Paused on the Analytics-Kanban board.Feb 7 2019, 8:46 PM

mforns removed a project: Analytics-Kanban.Nov 25 2019, 5:29 PM

mforns lowered the priority of this task from High to Medium.Dec 9 2019, 5:23 PM

JAllemandou moved this task from Smart Tools for Better Data to Analytics Query Service on the Analytics board.Jun 2 2020, 9:10 AM

JAllemandou added a parent task: T249751: Define reduce calculations needed to compute active editors per project family.

Change 601662 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] [SPIKE] Add mediawiki-reduced spark job

https://gerrit.wikimedia.org/r/601662

gerritbot added a project: Patch-For-Review.Jun 2 2020, 9:15 AM

JAllemandou added a project: Analytics-Kanban.Jun 2 2020, 9:15 AM

JAllemandou set Final Story Points to 5.

JAllemandou moved this task from Paused to Done on the Analytics-Kanban board.

Nuria moved this task from Done to Paused on the Analytics-Kanban board.Jun 3 2020, 3:56 PM

JAllemandou moved this task from Paused to In Code Review on the Analytics-Kanban board.Jun 8 2020, 7:27 AM

Milimetric moved this task from In Code Review to Done on the Analytics-Kanban board.Jun 8 2020, 4:47 PM

Nuria closed this task as Resolved.Sep 1 2020, 9:38 PM

[Spike] Spark job for digests-only mediawiki-history-reduced Closed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

[Spike] Spark job for digests-only mediawiki-history-reduced
Closed, ResolvedPublic
Actions

Related Objects
Search...