@Melderick Restarts are done automatically by the script running the specific dump, up to a certain number of times. That is independent of the starting date. If we have one wrapper script run them one after another, the same will still hold.

I've sent mail to wikitech-l, xmldatadumps-l and research-internal (not sure if that last arrived, I may not be subscribed and tbh I am full up on subscriptions). See https://lists.wikimedia.org/pipermail/xmldatadumps-l/2019-February/001455.html

The last iteration of this utility is both fast and seemingly accurate at generating the file crc from the block crcs. The code is still a bit rough but it does the trick. Next up: a utility that takes a sequence of filespecs plus block ranges to extract and merge into a new file; this will be a general bz2 file rewriter with no decompression.

Merged and deployed, with a commit message that I forgot to update, though I double-checked everything else (of course). No longer a work in progress and has been tested with large files. We'll see how everything goes on the run of the 20th.

The last fixup jobs are complete, and all updated files should be available for download along with corrected checksums later today, if they are not already.

Wikidatawiki's run is complete. I am now running the no-op job for that; the enwiki noop job is still running but should complete in some hours.

The directories and any links or status files would be as they are now, but the date could I presume be passed into the script. We need again to see what existing users need and expect though.

What I'd prefer to do, if we are changing things around, is to do one right after another, so: all, truthy, lexemes back to back, and decide on a starting date for the first one. This ensures we aren't running more than one type at once (control over resource use on the server) and that there's no dead time between runs (so server maintenance etc can be scheduled easier).

Status and sha1/md5sum files are being regenerated for enwiki now and should be available for download later in the day. Wikidata's dump run should complete later today and then these files can be rebuilt for it as well.

Adding @hoo and @Smalyshev because they may have an idea of the needs of current users of these entity dumps.

New files have been generated for all of the above wikis; new sha1/md5 sums and status files are being generated for all but enwiki and wikidatawiki right now, and should be available for download sometime tomorrow.

Not I. Anyone else?

https://gist.github.com/apergos/49c2812292c0ec26ccedc8bc9d5d69e5 is a standalone script which does what the gerrit patch does but regenerates just the combined index file, to an arbitrary location as opposed to the live dump directory.

Correct, it needed python3, the directory exists , with python3 the error is different and it's likely the error jcrespo mentioned earlier.

When I go to the directory and do the import manually I get:

Only the so-called 'big wikis' are subjec to this bug, since only for them do we generate smaller index files and recombine them. Of those, the only ones big enough to have the issue are:
en, wikidata, commons, de, fr, es, it, ja, ru wikis. Once there is a verified standalone script to just fix up the index file, I'll run it followed by removal of the sha1 and md5sums for the bad index files, and then a no-op on all of these wikis so that status files are regenerated. Wikidata will need to wait until it's completed the 7z step that it's currently in the middle of, since we can't regenerate status files for a wiki currently in the middle of a dump run.

@Marostegui /srv/sqldata has 39G on it on db04, presumably that's pretty close to the amount of data on the master.

This turns out to be an annoying limitation of (m)awk, that numbers are actually doubles. I'll have to write a tiny python script to do the index file merge.

Hmm, that will be an artifact of the new recombine code. I'll have a look.

This run is complete. Typically when you see that a run has been aborted, you simply need to wait a couple of days for the job to have been restarted and some output and status files to be produced and copied out to the public web servers. I have the advantage here because I was able to check the internal files at the time and see that everything was ok. Thanks for the report.

Do we know which of those spaces in particular was the culprit, or if all of them were? Asking just out of curiosity.

Ah note also that if you click through on the (empty) map in the wp article, you do see a properly rendered full screen map. Huh!

After removal of that one trailing comma, I have done the sandbox trick and stuffed the wikidata id entry in where it should be (Q867944 in place of (if(?id = wd:blah...) and I get a nice little rendered map with two stroke colors and the whole thing. This was taking a copy of the OSM template and plugging some values directly into it (lat/long, frame width/height, zoom, plain=yes). So uh? I must be overlooking something obvious.

Poking at this carefully: afaict the wikidata query that gets run is https://query.wikidata.org/#SELECT%20%3Fid%20%3Flength%0A%20%20%28if%28%3Fid%20%3D%20wd%3AQ867944%2C%20%27%23C12838%27%2C%20%27%2307c63e%27%29%20as%20%3Fstroke%29%0A%20%20%28concat%28%27Line%20length%3A%20%27%2C%20str%28%3Flength%29%2C%20%27%20km%27%29%20as%20%3Fdescription%29%0A%20%20%28if%28BOUND%28%3Flink%29%2C%0A%20%20%20%20%20%20concat%28%27%5B%5B%27%2C%20substr%28str%28%3Flink%29%2C31%2C500%29%2C%20%27%7C%27%2C%20%3FidLabel%2C%20%27%5D%5D%27%29%2C%0A%20%20%20%20%20%20%3FidLabel%29%0A%20%20%20as%20%3Ftitle%29%0AWHERE%20%7B%20%20%0A%20%20%7B%3Fid%20wdt%3AP16%20wd%3AQ5228578.%7D%0A%20%20OPTIONAL%20%7B%20%3Fid%20wdt%3AP2043%20%3Flength.%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%0A%20%20%20%20bd%3AserviceParam%20wikibase%3Alanguage%20%27en%27%20.%0A%20%20%20%20%3Fid%20rdfs%3Alabel%20%3FidLabel%20.%0A%20%20%7D%0A%20%20OPTIONAL%20%7B%3Flink%20schema%3Aabout%20%3Fid.%0A%20%20%3Flink%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fen.wikipedia.org%2F%3E.%7D%0A%7D%20GROUP%20BY%20%3Fid%20%3Flink%20%3FidLabel%20%3Flength%0A
I have gotten this by plugging in Q5228578 for the whole if clause with highway_system_qid and {{wikidata|property|raw|P16}}, Q867944 for {{wikidata|label|raw}}, (grabbed this value by evaluating the templatthe pipe on a preview copy of the Dōtō Expressway article), and by replacing {{!}} with the pipe symbol. But the result gives a list of all the expressways in Japan, not just the one we want. Did I get the query right? Surely this is not what's desired.

Everything sure looks good. Closing.

Spot checked the content of combined files, also checked for links on the web page, everything looks good. Closing.

These files look good; I spot-checked a few for content. Closing this task.

Wikidata's web page is now updated. This ticket will remain open until we see multistream files generated and recombined *and published properly* for the next run.

The wikidata fix is now running and should complete sometime later today.

commonswiki and enwiki fixupsare running now; only wikidata left, which will be taken care of at the end of its current run.

The manual reruns are complete except for commons and wikidata.

The fix has been deployed and I am manually running updates for all big wikis now except for commons and wikidata; these will need to wait until their runs complete.

The html links for the recombined multistream files for the big wikis are wrong. Woops! I'll fix these manually once the runs for these are complete, and get a fix to the code in before the next run.

I should point out that in the original files enwiki-20190101-pages-meta-history10.xml-p2534537p2554779.bz2 and wikidatawiki-20190101-pages-meta-history27.xml-p56428595p56649675.bz2, the revison counts were comparable. 1500850 vs 1484162, but the line counts of the uncompressed content were not: 693941886 vs 28694664 lines. That's due to the very particular structure of a wikidata entry as stored, and it seems that structure is one of the worst cases for the standard bzip2 implementation. Anyways, more testing soon!

It does indeed look like the specific compression implementation.

I don't see a big difference in prefetch percentages for the two wikis. It's worth checking if the compression itself is slower for wikidata revisions due to their content structure.

This has been deployed and is in effect for the xml/sql dumps run that started yesterday. Will wait for the end of the run and check results before closing the ticket.

Watching these til the end of the current run. If all looks good, the ticket can be closed then.

Watching these til the end of the current run. If all looks good, the ticket can be closed then.

Installed updated mwbzutils and deployed all code. This will be in effect for the next xml/sql dumps run, which starts tomorrow.

Installed updated mwbzutils and deployed all code. This will be in effect for the next xml/sql dumps run, which starts tomorrow.

This is looking good; it is ready to go once the current dump run ends. Wikidata is finishing up the last pages-meta-history file; then there will be the 7z files and it will be complete.

At a blocksize of 256k we get 19 minutes for a dd of a 51GB file over NFS, as opposed to the 3 hours it takes to gunzip/gzip files for the stubs recombine step. That's good enough to move forward. Larger blocksizes show no appreciable gain.

These are full html of the pages or 'just' of the parsed/rendered wikitext, or...? And, is there a notion of what the code looks like or what components are involved, so we can know if it can be folded into our infrastructure? Note there is draft code already for parsed/rendered wikitext, as pulled from Restbase.

Doing dd over nfs timing tests now to make sure we gain a reasonable amount of time back.

Run is completed. I am still trying to create smaller input files that reproduce the problem, no luck yet. In the meantime, the revision that breaks everything is this one: https://zh.wikipedia.org/w/index.php?title=Template:X2&oldid=35165097 (WARNING, this is a 12 megabyte text revision so it may break your browser!)

Ne bz2 file has been copied into place and current zhwiki processes shot; the scheduler should pick it up later today or tomorrow and complete the run.

Uncompressed stubs and output file don't help; uncompressed prefetch files seem to make a difference. The output file is now about halfway complete and much farther on than the two attempts that hung. I'll go ahead and let this complete, compress the file and move it into place so that the regular run can continue on, while I continue looking into the cause of the problem.

Right now I'm checking to see if compression/decompression of any of the files plays a role; I've also noted which revision of which page is the last to be written. The following revision, at least in the stub file, sure looks harmless enough.

Results of timing tests:

$ time (bzcat /mnt/dumpsdata/xmldatadumps/public/enwiki/20181201/enwiki-20181201-pages-articles.xml.bz2 | ./recompressxml_prod --pagesperstream 100 --buildindex prod_index.bz2 > prod_pages.bz2)

https://github.com/apergos/misc-wmf-crap/commit/b973e9a1b6d8d73d63e933ffb362c02b25b9db2c Here's a link to that test code; the real thing will of course be added to the recombinejobs class and live in gerrit with the rest of the dumps scripts.

The right way to do this seems to be:

• for each numbered stub output file for big wikis, gzip the xml header and write to output file, gzip the body as it gets appended to the output file, and gzip the footer and cat that on the end.
• for the combined file, look at each numbered stub output file in turn, looking for gzip file headers; note the body and footer offsets for each; then with that info cat the heder from first file, body from all files, footer from last file.

Timing tests are still ongoing. If those pan out, packages are ready to go on install1002 in my home directory. We need to wait until the current run is complete, merge and deploy everything, push out the package to the repo, and then update the package on all snapshots.

While these changes have all been thoroughly tested to make sure they work as advertised, I need to do timing tests yet, to make sure they don't slow down the appropriate dump steps appreciably.

Run back to normal output levels, closing.

Tests looked good but I'll leave it open for a day so we can doublecheck that the cronspam only contains errors and that the run otherwise looks good.