Page MenuHomePhabricator
Create Task
Maniphest T232843

Can we add ORES data so it can be easily retrieved per revision present on mediawiki history?
Open, Needs TriagePublic
Actions

Description

From the survey on the format of the mediawiki history dumps:

"Going on a bit of a tangent, it would be helpful to have the Parsoid and ORES data (for all models available at the time of the extraction, optimized sufficiently for querying even if just via JSON key-value mapping convention only) available in the pertinent files. I find myself often wanting to have this data readily available instead of having to make API calls."

Parsoid data is available via dumps but ORES data is at this time not available anywhere. It seems that it would not be hard to make a dump of ores scores per revision id per wiki and those could be released in a daly schedule such api calls are not needed.
cc @Halfak in case his team is interested in grabbing this item

Related Objects

Event Timeline

mforns created this task.Sep 13 2019, 3:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 13 2019, 3:16 PM
mforns mentioned this in T224459: Recommend the best format to release public data lake as a dump.Sep 13 2019, 3:20 PM
• Nuria subscribed.Sep 13 2019, 3:22 PM

I do not think it will be a wise decision (even if we could do it technically) to have mediawiki history be the monolith of all the things, with revision ids you should be able to retrieve easily ores scores and content.

JAllemandou subscribed.EditedSep 13 2019, 4:00 PM

I mildly disaggre with @Nuria for ORES scores - I think it could be very cool to have them (some models only, one model version only). Maybe in a separate table, or in different dumps. As for parsoid, I don't really understand what that is.
Edited for typos.

• Nuria added a comment.Sep 13 2019, 5:02 PM

I do not disagree that ores score would be useful in a table accessible by revision, +1 to that . I just do not think that the process that retrieves them and maintains them should be related to the mediawiki history one. I think the use case here is to be able to retrieve those in an easier format than it is possible now, makes sense.

JAllemandou added a comment.Sep 13 2019, 8:02 PM

+1 to that :)

• fdans moved this task from Incoming to Machine Learning Platform on the Analytics board.Sep 16 2019, 3:40 PM
• Nuria renamed this task from Can we add Parsoid and ORES data to mediawiki history? to Can we add ORES data so it can be easily retrieved per revision present on mediawiki history?.Sep 16 2019, 5:45 PM
• Nuria updated the task description. (Show Details)Sep 16 2019, 5:47 PM
• Nuria moved this task from Machine Learning Platform to Radar on the Analytics board.
• Nuria added a subscriber: Halfak.
Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:33 AM
Aklapper added a project: Data-Engineering-Icebox.Feb 10 2023, 5:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL