Page MenuHomePhabricator
Create Task
Maniphest T209655

Copy Wikidata dumps to HDFS + parquet
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

Now that T202489: Copy monthly XML files from public-dumps to HDFS is done, we'd love to see Wikidata dumps in HDFS. We already have one-off dumps in /user/joal/wmf/data/wmf/mediawiki/wikidata_parquet/20180108. This task is about creating dumps periodically in an automated way.

One immediate use case is generating recommendations for article creation. We already have recommendations that are based on the above indicated dumps. But before going to production, we'd like to generate a new set of recommendations.

Details

SubjectRepoBranchLines +/-
Add oozie job converting wikidata dumps to parquetanalytics/refinerymaster+571 -0
Add spark code for wikidata json dumps parsinganalytics/refinery/sourcemaster+1 K -1
profile::analytics::refinery::job::import_wikidata_entities_dumps: cleanupoperations/puppetproduction+1 -13
profile::analytics::refinery::job::import_wikidata_entities_dumps: fix typooperations/puppetproduction+13 -1
Correct profile::analytics::refinery::job::import_wikidata_entities_dumpsoperations/puppetproduction+2 -2
Add profile::analytics::refinery::job::import_wikidata_entites_dumpsoperations/puppetproduction+120 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• bmansurov created this task.Nov 15 2018, 10:22 PM
JAllemandou subscribed.Nov 16 2018, 8:17 AM

Not sure when we'll be able to do that. However there is more recent dump available: /user/joal/wmf/data/wmf/mediawiki/wikidata_parquet/20181001 :)

Addshore added a project: Wikidata.Nov 16 2018, 8:24 AM
Addshore subscribed.
GoranSMilovanovic subscribed.Nov 16 2018, 8:25 AM
Addshore moved this task from incoming to monitoring on the Wikidata board.Nov 16 2018, 8:30 AM
• bmansurov added a comment.Nov 16 2018, 2:00 PM

Thanks, @JAllemandou! The more recent dumps are very useful.

• fdans triaged this task as Medium priority.Nov 19 2018, 5:18 PM
• fdans lowered the priority of this task from Medium to Low.
• fdans moved this task from Incoming to Smart Tools for Better Data on the Analytics board.
• bmansurov mentioned this in T210844: Generate article recommendations in Hadoop for use in production.Dec 4 2018, 8:24 PM
Ottomata added subscribers: • Nuria, Ottomata.Dec 6 2018, 6:24 PM

@Nuria we should fit this in somewhere! Maybe a Q3 goal? :D

• bmansurov added a parent task: T203041: Output 2.1: An improved task recommendation API.Dec 6 2018, 6:25 PM
• Nuria added a comment.Dec 6 2018, 6:28 PM

Having missed most of goals this quarter due to our mw woes i think this might need to be moved to next quarter (q4?)

• bmansurov edited parent tasks, added: T148129: Productization of Recommendation API; removed: T203041: Output 2.1: An improved task recommendation API.Jan 9 2019, 9:46 PM
• bmansurov edited parent tasks, added: T214074: Productionize article recommender systems; removed: T148129: Productization of Recommendation API.Jan 17 2019, 7:11 PM
leila subscribed.Mar 26 2019, 6:51 PM
In T209655#4804046, @Nuria wrote:

Having missed most of goals this quarter due to our mw woes i think this might need to be moved to next quarter (q4?)

@Nuria can your team help us with this task during Q4? Content Translation (currently the biggest user of the translation recommendation API) is aiming to go to production (in at least one language) in Q4: T102107 . We want to make sure the service is productionized by the time they move to production.

JAllemandou added a comment.Mar 26 2019, 7:41 PM

Most of the complicated things already exist for this to work (equicalent of rsync for HDFS, spark job converting wikidata json dumps to parquet).
I wanted for T216160 to be settled before moving into productionization (having the same date for the various dumps we handle simplifies quite a bit), and it takes time.

GoranSMilovanovic mentioned this in T204440: analyze and visualize the identifier landscape of Wikidata.Mar 29 2019, 11:18 AM
• bmansurov added a parent task: T210844: Generate article recommendations in Hadoop for use in production.Apr 8 2019, 2:40 PM
abian subscribed.Apr 24 2019, 10:07 PM

@JAllemandou, do you think this is now unblocked?

• Nuria added a comment.Apr 24 2019, 10:33 PM

@abian : this is still not happening on a recurrent schedule yet.

GoranSMilovanovic added a comment.Jun 7 2019, 11:02 PM

@JAllemandou Thanks for the recent 20190603 dump copy in HDFS.

JAllemandou added a comment.Jun 8 2019, 7:21 AM

@GoranSMilovanovic : You're welcome :) At some point I'll manage to have that productionize ;)

GoranSMilovanovic mentioned this in T217994: WDCM Dashboards Maintenance.Jun 17 2019, 9:04 AM
leila edited projects, added Research-Freezer; removed Research.Jul 11 2019, 4:08 PM
GoranSMilovanovic mentioned this in T227701: Quantify additional information available via external identifiers.Jul 15 2019, 9:52 AM
WMDE-leszek subscribed.Sep 23 2019, 2:41 PM
GoranSMilovanovic added a comment.Sep 30 2019, 7:17 PM

@JAllemandou Would it be possible to have another update (beyond the most recent 20190603) of the dump in hdfs?
I would like to present some of the analytical systems based on this in the WikidataCon 2019, and would be very, very grateful if a new copy in hdfs would appear until... say... October 15?
Please let me know. Many thanks!

JAllemandou added a comment.Oct 3 2019, 9:57 AM

this is done @GoranSMilovanovic.
Raw data is here /user/joal/wmf/data/raw/mediawiki/wikidata/all_jsondumps/20190902 and parquet data is here /user/joal/wmf/data/wmf/mediawiki/wikidata_parquet/20190902

GoranSMilovanovic added a comment.Oct 3 2019, 12:46 PM

@JAllemandou Awesome, thank you!

Ottomata renamed this task from Copy Wikidata dumps to HDFs to Copy Wikidata dumps to HDFS.Nov 7 2019, 2:37 PM
MGerlach subscribed.Nov 19 2019, 7:55 PM
leila moved this task from Backlog to Radar on the Research-Freezer board.Nov 20 2019, 12:30 AM
GoranSMilovanovic added a comment.Dec 4 2019, 2:26 AM

@JAllemandou Do you think it would be possible to produce a new version of this data set?
The latest update seems to be: 2019-10-03 09:29 /user/joal/wmf/data/wmf/mediawiki/wikidata_parquet/20190902 - which you have pointed me at in T209655#5543575.
I would need to update the Wikidata Quality Report soon (Dec 15, say), and the code relies on Spark to process the dump. Thanks.

JAllemandou added a subscriber: Groceryheist.Dec 4 2019, 7:13 PM

New dataset available @GoranSMilovanovic. Pinging @Groceryheist as I also generated the items per page.

hdfs dfs -ls /user/joal/wmf/data/wmf/mediawiki/wikidata_parquet | tail -1
drwxr-xr-x   - analytics joal          0 2019-12-04 18:31 /user/joal/wmf/data/wmf/mediawiki/wikidata_parquet/20191202

hdfs dfs -ls /user/joal/wmf/data/wmf/wikidata/item_page_link/ | tail -1
drwxr-xr-x   - joal joal          0 2019-12-04 18:50 /user/joal/wmf/data/wmf/wikidata/item_page_link/20191202
GoranSMilovanovic added a comment.Dec 5 2019, 10:42 AM

@JAllemandou Thank you - as ever!

GoranSMilovanovic mentioned this in T237013: Refactor the Wikidata Data Quality Report analytics procedures.Dec 5 2019, 10:49 AM
Isaac subscribed.Jan 14 2020, 6:46 PM

@JAllemandou Thank you - as ever!

+1: these wikidata parquet (specifically item_page_link) dumps are super useful for us!

gerritbot added a comment.Jan 28 2020, 2:39 PM

Change 567954 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Add profile::analytics::refinery::job::import_wikidata_entites_dumps

https://gerrit.wikimedia.org/r/567954

gerritbot added a project: Patch-For-Review.Jan 28 2020, 2:39 PM
JAllemandou claimed this task.Jan 28 2020, 2:41 PM
JAllemandou added a project: Analytics-Kanban.
JAllemandou set the point value for this task to 5.
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.
JAllemandou added a subtask: T243832: Fix hdfs-rsync`prune-empty-dirs` feature.
gerritbot added a comment.Feb 4 2020, 9:04 AM

Change 569836 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Add oozie job converting wikidata dumps to parquet

https://gerrit.wikimedia.org/r/569836

gerritbot added a comment.Feb 4 2020, 11:03 AM

Change 346726 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Add spark code for wikidata json dumps parsing

https://gerrit.wikimedia.org/r/346726

JAllemandou moved this task from In Progress to In Code Review on the Analytics-Kanban board.Feb 4 2020, 11:03 AM
gerritbot added a comment.Feb 8 2020, 9:27 AM

Change 567954 merged by Elukey:
[operations/puppet@production] Add profile::analytics::refinery::job::import_wikidata_entites_dumps

https://gerrit.wikimedia.org/r/567954

JAllemandou moved this task from In Code Review to Done on the Analytics-Kanban board.Feb 10 2020, 8:10 AM
JAllemandou moved this task from Done to In Code Review on the Analytics-Kanban board.
JAllemandou moved this task from In Code Review to In Progress on the Analytics-Kanban board.
gerritbot added a comment.Feb 10 2020, 8:29 AM

Change 571237 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Correct profile::analytics::refinery::job::import_wikidata_entities_dumps

https://gerrit.wikimedia.org/r/571237

gerritbot added a comment.Feb 10 2020, 8:39 AM

Change 571238 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Correct profile::analytics::refinery::job::import_wikidata_entities_dumps

https://gerrit.wikimedia.org/r/571238

gerritbot added a comment.Feb 10 2020, 8:40 AM

Change 571237 abandoned by Joal:
Correct profile::analytics::refinery::job::import_wikidata_entities_dumps

Reason:
New patch at https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /571238/

https://gerrit.wikimedia.org/r/571237

gerritbot added a comment.Feb 10 2020, 8:51 AM

Change 571238 merged by Elukey:
[operations/puppet@production] profile::analytics::refinery::job::import_wikidata_entities_dumps: fix typo

https://gerrit.wikimedia.org/r/571238

gerritbot added a comment.Feb 10 2020, 9:35 AM

Change 571253 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] profile::analytics::refinery::job::import_wikidata_entities_dumps: cleanup

https://gerrit.wikimedia.org/r/571253

gerritbot added a comment.Feb 10 2020, 9:45 AM

Change 571253 merged by Elukey:
[operations/puppet@production] profile::analytics::refinery::job::import_wikidata_entities_dumps: cleanup

https://gerrit.wikimedia.org/r/571253

JAllemandou moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.Feb 10 2020, 10:14 AM
gerritbot added a comment.Feb 17 2020, 9:52 PM

Change 346726 merged by jenkins-bot:
[analytics/refinery/source@master] Add spark code for wikidata json dumps parsing

https://gerrit.wikimedia.org/r/346726

JAllemandou renamed this task from Copy Wikidata dumps to HDFS to Copy Wikidata dumps to HDFS + parquet.Feb 18 2020, 11:33 AM
gerritbot added a comment.Feb 20 2020, 4:32 PM

Change 569836 merged by Fdans:
[analytics/refinery@master] Add oozie job converting wikidata dumps to parquet

https://gerrit.wikimedia.org/r/569836

JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Feb 25 2020, 10:15 AM
• Nuria closed subtask T243832: Fix hdfs-rsync`prune-empty-dirs` feature as Resolved.Feb 27 2020, 7:26 PM
• Nuria closed this task as Resolved.Feb 27 2020, 7:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits