Page MenuHomePhabricator
Create Task
Maniphest T156388

Populate aqs with legacy page-counts
Closed, ResolvedPublic13 Estimated Story Points
Actions

Description

To do:

  • Move data on files to hadoop (make sure we have a backup easy to access) /wmf/data/legacy/project-counts
  • Agreggate/transform format
  • Create workflow (no corrdinator needed) to load into aqs so we have a repeteable process
    • Load aqs from hadoop data:
  • Evangelize this is available

Path:
/metrics/legacy/pagecounts

Let's make sure to include mobile pagecounts too

Details

SubjectRepoBranchLines +/-
Fix domain_abbrev_map job to disambiguate wikimedia projectsanalytics/refinerymaster+5 -1
Add oozie workflow to load projectcounts to AQSanalytics/refinerymaster+591 -0
Lowercase domain abbreviations in projectcounts aggregationanalytics/refinery/sourcemaster+2 -1
Add spark job to aggregate historical projectviewsanalytics/refinery/sourcemaster+214 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Nuria created this task.Jan 26 2017, 5:00 PM
• Nuria updated the task description. (Show Details)
MusikAnimal subscribed.Jan 26 2017, 5:01 PM
• Nuria updated the task description. (Show Details)Jan 26 2017, 5:02 PM
• Nuria updated the task description. (Show Details)EditedJan 26 2017, 5:04 PM

Likely to involve some hive to transform data format into something that can be loaded on aqs

Create an AQS loading job, should be productionized so we can rerun it easily.

• Nuria updated the task description. (Show Details)Jan 26 2017, 5:07 PM
• Nuria updated the task description. (Show Details)
• Nuria renamed this task from Populate aqs with legacy pageviews on new endpoint to Populate aqs with legacy pageviews.Jan 26 2017, 5:12 PM
• Nuria created subtask T156391: Create AQS endpoint to serve legacy pageviews .
• Nuria updated the task description. (Show Details)Jan 26 2017, 5:15 PM
• Nuria renamed this task from Populate aqs with legacy pageviews to Populate reportcard with legacy pageviews.Jan 26 2017, 5:20 PM
• Nuria renamed this task from Populate reportcard with legacy pageviews to Populate aqs with legacy pageviews.
• Nuria updated the task description. (Show Details)
• Nuria updated the task description. (Show Details)Jan 26 2017, 5:24 PM
• Nuria set the point value for this task to 8.
• Nuria changed the point value for this task from 8 to 13.Jan 26 2017, 5:26 PM
• Nuria updated the task description. (Show Details)
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.Jan 30 2017, 3:17 PM
Milimetric mentioned this in T146308: Kill limn1 .Feb 13 2017, 7:35 PM
mforns moved this task from In Progress to In Code Review on the Analytics-Kanban board.Feb 14 2017, 3:03 PM
gerritbot subscribed.Feb 14 2017, 3:03 PM

Change 337593 had a related patch set uploaded (by Mforns):
Add spark job to aggregate historical projectviews

https://gerrit.wikimedia.org/r/337593

gerritbot added a project: Patch-For-Review.Feb 14 2017, 3:03 PM
gerritbot added a comment.Feb 23 2017, 3:15 PM

Change 339421 had a related patch set uploaded (by Mforns):
Add oozie workflow to load projectcounts to AQS

https://gerrit.wikimedia.org/r/339421

gerritbot added a comment.Mar 15 2017, 3:43 PM

Change 337593 merged by jenkins-bot:
[analytics/refinery/source] Add spark job to aggregate historical projectviews

https://gerrit.wikimedia.org/r/337593

• Nuria renamed this task from Populate aqs with legacy pageviews to Populate aqs with legacy page-counts.Mar 22 2017, 8:16 PM
• Nuria updated the task description. (Show Details)Mar 24 2017, 5:17 PM
gerritbot added a comment.Mar 24 2017, 5:30 PM

Change 344665 had a related patch set uploaded (by Mforns):
[analytics/refinery/source@master] Lowercase domain abbreviations in projectcounts aggregation

https://gerrit.wikimedia.org/r/344665

gerritbot added a comment.Mar 24 2017, 6:12 PM

Change 344665 merged by jenkins-bot:
[analytics/refinery/source@master] Lowercase domain abbreviations in projectcounts aggregation

https://gerrit.wikimedia.org/r/344665

gerritbot added a comment.Mar 27 2017, 7:48 AM

Change 344914 had a related patch set uploaded (by Mforns):
[analytics/refinery@master] Fix domain_abbrev_map job to disambiguate wikimedia projects

https://gerrit.wikimedia.org/r/344914

• Nuria moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Mar 30 2017, 4:04 PM
gerritbot added a comment.Mar 31 2017, 12:24 PM

Change 339421 merged by Mforns:
[analytics/refinery@master] Add oozie workflow to load projectcounts to AQS

https://gerrit.wikimedia.org/r/339421

gerritbot added a comment.Mar 31 2017, 1:35 PM

Change 344914 merged by Mforns:
[analytics/refinery@master] Fix domain_abbrev_map job to disambiguate wikimedia projects

https://gerrit.wikimedia.org/r/344914

JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Apr 3 2017, 11:41 AM
• Nuria closed subtask T156391: Create AQS endpoint to serve legacy pageviews as Resolved.Apr 3 2017, 4:37 PM
• Nuria added a subtask: T162157: Pagecounts all sites data issues .Apr 7 2017, 6:59 PM
• Nuria closed this task as Resolved.Apr 11 2017, 11:48 PM
• Nuria closed subtask T162157: Pagecounts all sites data issues as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL