Page MenuHomePhabricator
Create Task
Maniphest T234333

Import siteinfo dumps onto HDFS
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

siteinfo dumps contain, for each wiki, magic-words aliases, meaning multi-language keywords of wikitext (for instance REDIRECTION in French for REDIRECT). Importing these files and possibly transforming them onto an easier-to-query structure (parquet based) is a necessary step toward productionizing historical redirects extraction.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update oozie datasets to match dumps import changeanalytics/refinerymaster+21 -5
Refactor profile::analytics::refinery::job::import_mediawiki_dumpsoperations/puppetproduction+164 -50
Add site-info dump type to importeranalytics/refinerymaster+8 -4
Customize query in gerrit

Related Objects

Event Timeline

JAllemandou created this task.Oct 1 2019, 2:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 1 2019, 2:29 PM
gerritbot added a comment.Oct 1 2019, 2:29 PM

Change 540124 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Add site-info dump type to importer

https://gerrit.wikimedia.org/r/540124

gerritbot added a project: Patch-For-Review.Oct 1 2019, 2:29 PM
gerritbot added a comment.Oct 3 2019, 6:50 PM

Change 540124 merged by Nuria:
[analytics/refinery@master] Add site-info dump type to importer

https://gerrit.wikimedia.org/r/540124

Maintenance_bot removed a project: Patch-For-Review.Oct 3 2019, 7:10 PM
Milimetric assigned this task to JAllemandou.Oct 7 2019, 3:58 PM
Milimetric triaged this task as High priority.
Milimetric moved this task from Incoming to Data Quality on the Analytics board.
Milimetric added a project: Analytics-Kanban.
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.Oct 17 2019, 8:13 AM
gerritbot added a comment.Oct 29 2019, 3:29 PM

Change 546966 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] [WIP] Refactor profile::analytics::refinery::job::import_mediawiki_dumps

https://gerrit.wikimedia.org/r/546966

gerritbot added a project: Patch-For-Review.Oct 29 2019, 3:29 PM
JAllemandou moved this task from In Progress to In Code Review on the Analytics-Kanban board.Oct 29 2019, 7:41 PM
gerritbot added a comment.Oct 30 2019, 11:21 AM

Change 547169 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Update oozie datasets to match dumps import change

https://gerrit.wikimedia.org/r/547169

JAllemandou moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Oct 31 2019, 5:15 PM
gerritbot added a comment.Oct 31 2019, 5:39 PM

Change 546966 merged by Elukey:
[operations/puppet@production] Refactor profile::analytics::refinery::job::import_mediawiki_dumps

https://gerrit.wikimedia.org/r/546966

gerritbot added a comment.Oct 31 2019, 5:48 PM

Change 547169 merged by Joal:
[analytics/refinery@master] Update oozie datasets to match dumps import change

https://gerrit.wikimedia.org/r/547169

Maintenance_bot removed a project: Patch-For-Review.Oct 31 2019, 6:11 PM
JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Nov 1 2019, 9:32 AM

Checked logs from this morning, they look good (nothing to import yet, but no error)

• Nuria subscribed.Nov 7 2019, 11:15 PM

Did we documented this data is available?

• Nuria added a comment.Nov 14 2019, 7:54 PM

ping @JAllemandou to see if any docs need to be corrected/added

JAllemandou added a comment.Nov 22 2019, 9:49 AM

New task: T238858

• Nuria set the point value for this task to 5.Nov 22 2019, 4:22 PM
• Nuria closed this task as Resolved.Mar 5 2020, 8:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits