Page MenuHomePhabricator
Create Task
Maniphest T143924

Replacing standard edit metrics in dashiki with data from new edit data depot
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

Dashiki thus far is the best UI venue to show current work on wikistats.
We will calculate all metrics here but the rolling ones:
https://meta.wikimedia.org/wiki/Research:Metrics_standardization

We will evaluate on a later phase whether we want to do rolling or other metrics (like bytes added)

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT120037 Vital Signs: Please provide an "all languages" de-duplicated stream for the Community/Content groups of metrics
 ResolvedNoneT120036 Vital Signs: Please make the data for enwiki and other big wikis less sad, and not just be missing for most days
 ResolvedodimitrijevicT130256 Wikistats 2.0.
 ResolvedNoneT143924 Replacing standard edit metrics in dashiki with data from new edit data depot
 ResolvedNoneT146489 Calculate edit metrics from history reconmstruction in hadoop intermediate table
ResolvedmfornsT150023 Run Standard metrics on denormalized history and compare with wikistats
 ResolvedmfornsT150024 (subtask) Marcel's standard metrics
 ResolvedMilimetricT150025 (subtask) Dan's standard metrics
 ResolvedMilimetricT152034 Create 1-off tsv files that dashiki would source with standard metrics from datalake
ResolvedJAllemandouT152035 Productionize Edit History Reconstruction and Extraction
 ResolvedMilimetricT141476 Productionize edit history extraction for all wikis using Sqoop
 ResolvedJAllemandouT141473 Productionize loading of edit data into Druid (contingent on success of research spike)
 DeclinedNoneT153923 vet edit data on the data lake
ResolvedJAllemandouT178591 Feedback on hive table mediawiki_history by Erik Z
 ResolvedJAllemandouT219177 Add user_is_bot_by to MediaWiki history
 ResolvedJAllemandouT144717 Extract edit history denormalized data from intermediate data
 ResolvedJAllemandouT160074 Create oozie job for mediawiki edit history job
 ResolvedOttomataT160083 Create cron job in puppet sqooping prod and labs DBs
 ResolvedJAllemandouT160151 Productionise standard metrics from mediawiki denormalized history
 ResolvedJAllemandouT160152 Update sqoop job to add infra and version partition
 ResolvedJAllemandouT160153 Provide 2 static files to differenciate prod and labs projects to sqoop in
ResolvedJAllemandouT160154 Synchronise changes for productionisation of mediawiki history jobs
 ResolvedJAllemandouT160155 Create hive tables and queries for metrics computation out of mediawiki denormalized history

Event Timeline

• Nuria created this task.Aug 25 2016, 6:28 PM
Milimetric moved this task from Incoming to Backlog (Later) on the Analytics board.Aug 29 2016, 3:40 PM
• Nuria renamed this task from Replacing vital signs metrics in dashiki with data from new edit data depot to Replacing standard edit metrics in dashiki with data from new edit data depot.Sep 22 2016, 4:16 PM
• Nuria updated the task description. (Show Details)Sep 22 2016, 4:18 PM
• Nuria created subtask T146489: Calculate edit metrics from history reconmstruction in hadoop intermediate table .Sep 23 2016, 6:21 PM
• Nuria created subtask T146490: Load storage (druid? clickhouse?) with calculated edit metrics data and serve under an external endpoint.Sep 23 2016, 6:24 PM
• Nuria updated the task description. (Show Details)Nov 3 2016, 4:05 PM
• Nuria updated the task description. (Show Details)Nov 3 2016, 4:07 PM
• Nuria updated the task description. (Show Details)Nov 3 2016, 4:11 PM
• Nuria added a comment.Nov 3 2016, 4:22 PM

We will compute standard metrics with canned data (scoop updates are not happening recurrently)

Process will be:

  • run metrics
  • vet data
  • CR

Rinse and repeat

• Nuria edited projects, added Analytics-Kanban; removed Analytics.Nov 3 2016, 4:22 PM
• Nuria moved this task from Next Up to Parent Tasks on the Analytics-Kanban board.Nov 3 2016, 4:29 PM
Milimetric added a subtask: T150023: Run Standard metrics on denormalized history and compare with wikistats.Nov 4 2016, 3:31 PM
• Nuria closed subtask T146489: Calculate edit metrics from history reconmstruction in hadoop intermediate table as Resolved.Nov 30 2016, 7:58 PM
• Nuria created subtask T152034: Create 1-off tsv files that dashiki would source with standard metrics from datalake .
• Nuria closed subtask T152034: Create 1-off tsv files that dashiki would source with standard metrics from datalake as Resolved.Dec 21 2016, 11:49 PM
• Nuria created subtask T153923: vet edit data on the data lake .Dec 21 2016, 11:52 PM
• Nuria removed a subtask: T153923: vet edit data on the data lake .
• Nuria added a subtask: T152035: Productionize Edit History Reconstruction and Extraction.Dec 21 2016, 11:57 PM
• Nuria removed a subtask: T146490: Load storage (druid? clickhouse?) with calculated edit metrics data and serve under an external endpoint.
• Nuria closed subtask T150023: Run Standard metrics on denormalized history and compare with wikistats as Resolved.Jan 24 2017, 9:58 PM
• Nuria closed subtask T152035: Productionize Edit History Reconstruction and Extraction as Resolved.May 4 2017, 5:08 PM
• Nuria added a comment.Jan 4 2018, 6:00 PM

Please see http://stats.wikimedia.org/v2 for edit metrics

• Nuria closed this task as Resolved.Jan 4 2018, 6:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits