skip existing page content dump output files in bz2 and 7z generation jobs for each batch of commands
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ArielGlenn
	Apr 15 2020, 10:17 AM

Description

This is prep work for being able to split page content jobs, either bz2 or 7z, across multiple hosts. We need this for the wikidata run asap.

Details

	Subject	Repo	Branch	Lines +/-
	check bz2 page content files for existence before running command batch	operations/dumps	master	+162 -7
	for 7z production in batches, skip files that exist at beginning of each batch	operations/dumps	master	+210 -25

Customize query in gerrit

Event Timeline

ArielGlenn triaged this task as Medium priority.Apr 15 2020, 10:17 AM

ArielGlenn created this task.

Change 565301 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] for 7z production in batches, skip files that exist at beginning of each batch

https://gerrit.wikimedia.org/r/565301

gerritbot added a project: Patch-For-Review.Apr 15 2020, 12:55 PM

Change 589032 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] check bz2 page content files for existence before running command batch

https://gerrit.wikimedia.org/r/589032

ArielGlenn moved this task from Backlog to Active on the Dumps-Generation board.Apr 16 2020, 10:22 AM

Change 565301 merged by ArielGlenn:
[operations/dumps@master] for 7z production in batches, skip files that exist at beginning of each batch

https://gerrit.wikimedia.org/r/565301

Change 589032 merged by ArielGlenn:
[operations/dumps@master] check bz2 page content files for existence before running command batch

https://gerrit.wikimedia.org/r/589032

Maintenance_bot removed a project: Patch-For-Review.Apr 19 2020, 8:10 PM

Keeping this open until the code path gets used, probably during the May run.

ArielGlenn moved this task from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.Apr 21 2020, 8:26 AM

Well. This got kept open rather longer than needed. This code has been running in production for months, closing!

ArielGlenn moved this task from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.Dec 21 2020, 5:08 PM

skip existing page content dump output files in bz2 and 7z generation jobs for each batch of commandsClosed, ResolvedPublicActions

Description

Details

Event Timeline

skip existing page content dump output files in bz2 and 7z generation jobs for each batch of commands
Closed, ResolvedPublic
Actions