Change 436511 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps/mwbzutils@master] allow writeuptopageid to write multiple output files

https://gerrit.wikimedia.org/r/436511

Change 436956 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] generate temp stubs for page ranges serially from same input stub file

https://gerrit.wikimedia.org/r/436956

Here's where we are on this;

I have done a variety of timing tests to be sure that the changes to writeuptopageid do not slow down the job; with/without zcat from stdin, with/without gzip to stdout, writing only to /dev/null, on local disks, on an nf filesystem. All of these tests look good.

Summary: because we can now pass all args to witeuptopageid directly instead of using a pipe for stdin/stdout, cpu time on an unloaded system has dropped. However real time has gone up a small amount (1.5 minutes for a 3GB compressed file of stubs for the input). That is to be expected, since other cores will not now be used for the other parts of the pipeline. This is fine for us, as we manage parallelism ourselves in the scripts.

I checked to see if there is much difference in performance between -O2 and -O3 in the Makefile; the difference is negligible. This also is to be expected, since the libraries that do most of the work (e.g. lz) remain untouched.

I may convert other of the C utils to be able to read/write files passed as command-line args as well.

The bz2 additions to the code are not needed for our production dump scripts, but folks processing dump xml files may find it useful, particularly the ability to write multiple output files on one run.

Change 436511 merged by ArielGlenn:
[operations/dumps/mwbzutils@master] allow writeuptopageid to write multiple output files

https://gerrit.wikimedia.org/r/436511

Package for writeuptopageid (mwbzutils) is ready and the files are sitting in home dir on install1002 awaiting deployment. The binary extracted from the package has been tested with old and new argument style.

Change 442828 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] generate multiple temp stub files at once for larger wikis

https://gerrit.wikimedia.org/r/442828

Change 436956 merged by ArielGlenn:
[operations/dumps@master] generate temp stubs for page ranges serially from same input stub file

https://gerrit.wikimedia.org/r/436956

Change 442828 merged by ArielGlenn:
[operations/dumps@master] generate multiple temp stub files at once for larger wikis

https://gerrit.wikimedia.org/r/442828

Mentioned in SAL (#wikimedia-operations) [2018-06-30T21:32:44Z] <ariel@deploy1001> Started deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063

Mentioned in SAL (#wikimedia-operations) [2018-06-30T21:35:11Z] <ariel@deploy1001> Finished deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063 (duration: 02m 27s)

While I was in here I updated the scripts to generate temp stubs in parallel for larger wikis. That's deployed too and running.