Page MenuHomePhabricator

download_dump.py: Support for "date specified" dumps
Open, NormalPublic

Description

Currently the download_dump.py script just can download the file from latest revision (f.ex https://dumps.wikimedia.org/idwiki/latest/ )[1]. We can add support to this script to download from "date specified" revision f.ex https://dumps.wikimedia.org/idwiki/20171001/.

For the implementation, we can perhaps add a new parameter -revision to the script, and then if the user don't specify the -revision we will assume that the revision is latest.

^1. https://github.com/wikimedia/pywikibot/blob/ca7c0ce89f2b2e96ebc5bb7b5b8aef2ccd04c2c3/scripts/maintenance/download_dump.py#L65

Event Timeline

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptDec 29 2017, 9:59 PM

I think if the user don't specify the -revision we will assume that the revision is latest and find out the latest date from the site. See T183667#3864150 for details.

Change 401091 had a related patch set uploaded (by Rafidaslam; owner: rafid):
[pywikibot/core@master] download_dump: Add -revision parameter

https://gerrit.wikimedia.org/r/401091

Xqt closed this task as Resolved.Dec 31 2017, 10:26 AM

Change 401091 merged by jenkins-bot:
[pywikibot/core@master] download_dump: Add -revision parameter

https://gerrit.wikimedia.org/r/401091

zhuyifei1999 reopened this task as Open.Jan 1 2018, 5:10 AM
zhuyifei1999 added a subscriber: zhuyifei1999.

I just realized, does this not work in the case of toolforge?

Hmm, I also a bit doubt of that, since I can't test it in toolforge. But I'll improve the script again based on my comment at https://phabricator.wikimedia.org/T183667#3865397 , I think that will work well on the toolforge

Change 401377 had a related patch set uploaded (by Rafidaslam; owner: rafid):
[pywikibot/core@master] download_dump: Resolve latest revision pointer to a date revision

https://gerrit.wikimedia.org/r/401377

jayvdb added a subscriber: jayvdb.Jan 1 2018, 1:09 PM

Hmm, I also a bit doubt of that, since I can't test it in toolforge.

Why cant you test it in toolforce?
If you dont have an account, you should create one.

Why cant you test it in toolforce?
If you dont have an account, you should create one.

Oh, just know that I can create a toolforge account. I thought it was an invite only, I'll submit a membership request then.

D3r1ck01 moved this task from Backlog to Needs Review on the Pywikibot board.Nov 5 2018, 11:27 AM