Maniphest T209739

Produce dump files for ORES scores
Open, LowestPublic
Actions

Assigned To

None

Authored By

	awight
	Nov 16 2018, 11:57 PM

Tags

Referenced Files

None

Subscribers

Description

Ideas for what to dump:

enwiki-20181107-ores-scores.jsonlines.bz2 – Full dump of all ORES scores available for the wiki.
enwiki-20181107-ores-scores-editquality.jsonlines.bz2 - Dump of all scores for a specific model.
wikidata-20181107-ores-scores-itemquality-history-monthly.jsonlines.bz2 - Dump of all historical scores for a content quality model, but sampled down to one revision per page per month.
wikidata-20181107-ores-scores-itemquality-current.jsonlines.bz2 - Much smaller dump that only has scores for the latest revision of each page.

These will require some processing to create, a MapReduce query should be fine.

The last time we made a dump like this, it was created manually. About 300 people have downloaded and it's cited by 2 papers, so it seems to be useful.
https://figshare.com/articles/Monthly_Wikipedia_article_quality_predictions/3859800

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T209611 [Epic] Make ORES scores for wikidata available as a dump
		Open		None	T209739 Produce dump files for ORES scores
		Declined		None	T213496 Investigate formal test framework for Oozie jobs

Event Timeline

Pointers from @Nuria:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Oozie
https://github.com/wikimedia/analytics-refinery/tree/master/oozie/mediacounts

awight updated the task description. (Show Details)Nov 16 2018, 11:58 PM

awight updated the task description. (Show Details)

• fdans moved this task from Incoming to Radar on the Analytics board.Nov 19 2018, 5:16 PM

Ladsgroup triaged this task as High priority.Nov 28 2018, 6:35 AM

Ladsgroup raised the priority of this task from High to Needs Triage.

Ladsgroup moved this task from Unsorted to New development on the Machine-Learning-Team board.

What sort of help do you need from my end, or is it too early?

awight unsubscribed.Mar 21 2019, 4:04 PM

ArielGlenn moved this task from Backlog to Other teams on the Dumps-Generation board.Mar 22 2019, 1:59 PM

Harej triaged this task as Lowest priority.Mar 26 2019, 9:19 PM

Halfak added a subtask: T213496: Investigate formal test framework for Oozie jobs.Nov 6 2019, 9:53 PM

Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:33 AM

calbon closed subtask T213496: Investigate formal test framework for Oozie jobs as Declined.Sep 23 2020, 4:37 PM

• ACraze moved this task from New development to Backlog/ORES on the Machine-Learning-Team board.Jan 20 2021, 12:52 AM

Isaac mentioned this in T312518: Migrate ORES clients to LiftWing.Sep 15 2022, 1:54 PM

Aklapper added a project: Data-Engineering-Icebox.Feb 10 2023, 5:44 PM