Page MenuHomePhabricator

Allow custom dump types to be defined per-wiki
Closed, ResolvedPublic

Description

Currently, dumps.wikimedia.org has a set of standard dumps for each wiki, e.g. https://dumps.wikimedia.org/wikidatawiki/20150307/

However, some wikis have a need for custom dump types, like JSON dumps for Wikidata. Currently, there is no way to get them created along with the standard dumps, or to make them, show up in the standard dump directory.

Providing such a way would make such "extra" dumps (which, for wikidata, are actually the most important and useful type of dumps) more discoverable, and their generation more reliable and in-line with the rest of the system, as opposed to an extra cron job that writes a file somewhere.

Feature wish list: I wrote a little wish list at https://www.mediawiki.org/wiki/Wikimedia_MediaWiki_Core_Team/Backlog/Improve_dumps#Configuring_Additional_Dumps

Event Timeline

daniel created this task.Mar 17 2015, 3:03 PM
daniel raised the priority of this task from to Needs Triage.
daniel updated the task description. (Show Details)
daniel added a subscriber: daniel.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 17 2015, 3:03 PM
Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.Mar 17 2015, 3:51 PM

Does this block T27602? If not, I'd set "Low" priority, as it would be only a nice to have which doesn't impact availability of content.

daniel added a comment.Apr 9 2015, 9:38 AM

It doesn't block T27602 as far as I can tell.

For wikidata, this is more than just nice to have - we need this to generate our primary kind of dumps, to enable our primary kind of re-use.

daniel updated the task description. (Show Details)Jun 30 2015, 10:12 PM
daniel set Security to None.

Dumps like these get generated as "other" datasets, typically via weekly cron jobs, as has been done for wikidata. Can we consider that sufficient? They can't really run with the standard dumps, and nor should they, as they are generated by entirely unrelated code.

Lydia_Pintscher closed this task as Resolved.Jan 24 2019, 2:36 PM
Lydia_Pintscher claimed this task.
Lydia_Pintscher added a subscriber: Lydia_Pintscher.

Let's call it sufficient.