Page MenuHomePhabricator
Create Task
Maniphest T192481

Add Mediawiki-History data-quality check stage in oozie using statistics
Closed, ResolvedPublic13 Estimated Story Points
Actions

Description

This oozie step should happen after the spark denormalization job, and before repairing the history hive tables.
This check will use he statistics generated by Accumulator (T155507).

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Add validation step in mediawiki-history jobsJoalanalytics/refinerymaster+748 -174
Update MediawikiHistoryChecker adding reducedJoalanalytics/refinery/sourcemaster+332 -15
Add MediawikiHistoryChecker spark jobJoalanalytics/refinery/sourcemaster+830 -0
Update mediawiki-history statsJoalanalytics/refinery/sourcemaster+204 -166
Customize query in gerrit

Related Objects
Search...

Event Timeline

JAllemandou created this task.Apr 18 2018, 6:45 PM
JAllemandou mentioned this in T177965: Wikistats 2 Backend: Resiliency, Rollback and Deployment of Data.Apr 18 2018, 6:51 PM
Milimetric triaged this task as High priority.May 7 2018, 8:09 PM
Milimetric added a project: Analytics.
Milimetric moved this task from Incoming to Wikistats Beta on the Analytics board.May 10 2018, 8:59 PM
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.May 22 2018, 12:04 PM
gerritbot subscribed.May 24 2018, 6:52 PM

Change 434987 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Update mediawiki-history stats to not compressed

https://gerrit.wikimedia.org/r/434987

gerritbot added a project: Patch-For-Review.May 24 2018, 6:52 PM
gerritbot added a comment.Jun 1 2018, 7:46 AM

Change 434987 merged by Joal:
[analytics/refinery/source@master] Update mediawiki-history stats

https://gerrit.wikimedia.org/r/434987

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Jun 1 2018, 4:11 PM
• Nuria moved this task from Done to Ready to Deploy on the Analytics-Kanban board.Jun 5 2018, 3:00 PM
JAllemandou moved this task from Ready to Deploy to In Progress on the Analytics-Kanban board.Jun 5 2018, 3:01 PM
gerritbot added a comment.Jun 12 2018, 10:13 AM

Change 439869 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Add MediawikiHistoryChecker spark job

https://gerrit.wikimedia.org/r/439869

JAllemandou moved this task from In Progress to In Code Review on the Analytics-Kanban board.Jun 12 2018, 2:58 PM
gerritbot added a comment.Jun 12 2018, 7:28 PM

Change 440005 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Add check step in mediawiki-history jobs

https://gerrit.wikimedia.org/r/440005

JAllemandou moved this task from In Code Review to In Progress on the Analytics-Kanban board.Jun 28 2018, 7:15 AM
gerritbot added a comment.Jul 9 2018, 9:01 PM

Change 441378 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Update MediawikiHistoryChecker adding reduced

https://gerrit.wikimedia.org/r/441378

JAllemandou moved this task from In Progress to In Code Review on the Analytics-Kanban board.Jul 10 2018, 7:59 AM
JAllemandou set the point value for this task to 13.Aug 9 2018, 3:03 PM
JAllemandou removed a project: Analytics.Aug 9 2018, 3:10 PM
JAllemandou moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Aug 21 2018, 6:58 PM
gerritbot added a comment.Aug 27 2018, 2:43 PM

Change 439869 merged by jenkins-bot:
[analytics/refinery/source@master] Add MediawikiHistoryChecker spark job

https://gerrit.wikimedia.org/r/439869

gerritbot added a comment.Aug 28 2018, 8:33 AM

Change 441378 merged by Joal:
[analytics/refinery/source@master] Update MediawikiHistoryChecker adding reduced

https://gerrit.wikimedia.org/r/441378

gerritbot added a comment.Aug 28 2018, 10:23 AM

Change 440005 merged by Joal:
[analytics/refinery@master] Add validation step in mediawiki-history jobs

https://gerrit.wikimedia.org/r/440005

JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Aug 29 2018, 3:03 PM
• Nuria closed this task as Resolved.Sep 25 2018, 11:04 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits