Page MenuHomePhabricator

handle dump run start properly across multiple hosts
Closed, ResolvedPublic

Description

Because more than one hosts does the dumps of the 'regular' (not en wikipedia) wikis, if one host is still running a previous dump run but the other has completed it, we can have problems. Specifically, one host will prepare all unlocked wikis for the new run, but it won't be able to prepare the wikis with dumps in progress. The second host, when it finally completes its jobs from the previous dump run, will run the cron job and try to prepare the wikis again with a different start date.

If the pervious jobs have all completed before the next run starts, this is of course not a problem.

Event Timeline

ArielGlenn claimed this task.
ArielGlenn raised the priority of this task from to Medium.
ArielGlenn updated the task description. (Show Details)
ArielGlenn added a project: Dumps-Generation.
ArielGlenn added subscribers: ArielGlenn, Aklapper.

The proper way to fix this would be to permit jobs from different dates to run at the same time (locks per date), so that the createdir job can happen for all wikis, and to use the date instead of 'last' in the scripts invoked by the cron job, so that previous runs finish running against the date intended.

Change 274997 had a related patch set uploaded (by ArielGlenn):
lock wikis for dump runs by date, permitting runs across multiple dates

https://gerrit.wikimedia.org/r/274997

Change 299448 had a related patch set uploaded (by ArielGlenn):
lock wikis for dump runs by date, permitting runs across multiple dates

https://gerrit.wikimedia.org/r/299448

Change 299448 merged by ArielGlenn:
lock wikis for dump runs by date, permitting runs across multiple dates

https://gerrit.wikimedia.org/r/299448

Leaving this open til we see a successful production run.

Change 274997 abandoned by ArielGlenn:
lock wikis for dump runs by date, permitting runs across multiple dates

Reason:
another copy already merged

https://gerrit.wikimedia.org/r/274997

New run jsut about complete (last wiki doing last jobs), closing this.