Page MenuHomePhabricator

Dumps (or dump progress page) stuck since 28 Jan
Closed, ResolvedPublic

Description

Dumps (or dump progress page) are stuck since 28 Jan.

The latest entry at http://dumps.wikimedia.org/backup-index.html is fiwiki from 2015-01-28 01:46:09. http://dumps.wikimedia.org/fiwiki/20150127/ seems to confirm that the dump is stuck. So are other dumps.

Event Timeline

matmarex raised the priority of this task from to Needs Triage.
matmarex updated the task description. (Show Details)
matmarex added a subscriber: matmarex.
Aklapper set Security to None.
Eloquence added subscribers: Tnegrin, ArielGlenn, mark.
Eloquence added a subscriber: Eloquence.

@mark @ArielGlenn @Tnegrin Just adding you all to this since it seems to be the master ticket for the issue.

Joe added a project: acl*sre-team.

The dumps stopped after we rebooted the servers for protecting against the GHOST vulnerability. I'm investigating this.

Dumps for bigwikis (ie all but enwiki and possibly dewiki) finally started and seem to have resumed operations. I won't guarantee they will work completely, but I'll keep an eye on those

Monitor has restarted correctly as well, updated info should show up as soon as dumps start flowing again.

Dumps for smallwikis have restarted as well. Right now the only process that is still blocked is for enwiki AFAICT.

By the way, how do I file such reports in the future so that they get noticed even when they are not blocking WMF activities? I assume that pinging people directly is not the right solution.

By the way, how do I file such reports in the future so that they get noticed even when they are not blocking WMF activities? I assume that pinging people directly is not the right solution.

Here in Phab, tag with SRE.

I don't think T85970: Some Wikidata XML dumps are failing is directly related to this task, but while people are poking at dumps, I'll just mention it.

After some sweat and cursing (and horrible local hacks, and invocations of the Old Ones) it seems dumps have restarted on enwiki as well. I'm monitoring it and may declare victory shortly

Document, puppetize etc pretty pretty please :-)

Please see https://phabricator.wikimedia.org/T85970 for stats on wikidata dumps.

Nutshell: run time almost doubled in 6 months
June 2014: 195 hrs
November 2014: 367 hrs

Is wikidatawiki on same queue as small wikis?
For many of those dumps data back until mid December.

Reopen or create new bug?