No-op is done, and the files are on the web server, the cloud nfs server, and the dumpsdata fallback server. Closing this task.

The hashes are done. I have edited dumpruninfo.txt for the run and set the status for the meta history jobs to 'done', adding appropriate time stamps according to when the last bz2 and 7z files were completed.

It finished up and the md5 and sha1 hashes are being generated now.

The 7z file is still being written. If all goes well it should finish up today.

The above patch has been deployed everywhere and manually copied into the directory for the run already underway. It won't save us tons of time on the run but still, every savings is useful.

The missing bz2 file is now available, although not linked for download yet. The last md5 and sha1 sums are running now, and because a couple of the files are quite big (193GB, 65GB), it's going to take a bit for those to finish up before I can run a no-op and have them show up with links.

Today's report:

<13>Nov 18 16:27:19 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201116/frwiki-20201116-cirrussearch-content.json.gz

In T267037#6631825, @Ladsgroup wrote:

If it's any help, Traffic workboard has a column called "Bad Herald", you can do a similar thing.

Note that I ran a little dumps test on a non-latin1 wiki in deployment-prep (ruwiki to be precise) and the results look just fine. But I didn't do any sort of comprehensive testing like MW integration tests or whatever they do over there.

The varnishes in beta are broken right now, see T267561

I was able to get further along by doing things manually as root on one of the instances, deployment-cache-text06.

I guess that T267439 might be related.

From looking at the code, it seems like the user list ought to have three fields, the first one being the name of the wiki. That appears to be missing. Someone can correct me on that later if they know better.

Update in deployment-prep is now complete, assuming I did not miss any hosts.

Puppet sync back to working. Back on track to continue with the update in deployment-prep.

This is apparently from T267439 After some discussion with jbond and dancy in irc, I am going to revert that and hope I'm not making the varnish situation worse. Also adding @thcipriani so he is informed.

Can't proceed at the moment, puppet sync to deployment-prep has been broken since Nov 6. Log excerpts from the earliest error:

2020-11-06T20:00:06Z INFO     git.cmd: git rev-parse --abbrev-ref HEAD -> 0; stdout: 'master'
2020-11-06T20:00:06Z INFO     git.cmd: git rev-parse master -> 0; stdout: 'c52d47210553bde2e89735a73637b918911d0226'
2020-11-06T20:00:06Z INFO     git.cmd: git merge-base 63bd567cfb4a5dfbeeb28e1908fb3f8b976d6a4e HEAD -> 0; stdout: '63bd567cfb4a5dfbeeb28e1908fb3f8b976d6a4e'
2020-11-06T20:00:06Z INFO     sync-upstream: Up-to-date: /var/lib/git/labs/private
2020-11-06T20:10:01Z INFO     git.cmd: git diff --abbrev=40 --full-index -M --raw --no-color
2020-11-06T20:10:01Z ERROR    sync-upstream: Local diffs detected.  Commit your changes!
2020-11-06T20:20:01Z INFO     git.cmd: git diff --abbrev=40 --full-index -M --raw --no-color
2020-11-06T20:20:01Z ERROR    sync-upstream: Local diffs detected.  Commit your changes!

git info:

root@deployment-puppetmaster04:/var/lib/git/operations/puppet/modules/profile/manifests/mediawiki# git status
HEAD detached from 0bd812c4d7
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

Note that I'm running

cumin  'O{project:deployment-prep  name:^deployment-mediawiki-[0-9]+$ } or O{project:deployment-prep  name:^deployment-parsoid[0-9]+$ } or O{project:deployment-prep  name:^deployment-snapshot[0-9]+$ } or O{project:deployment-prep  name:^deployment-jobrunner[0-9]+$ } or O{project:deployment-prep  name:^deployment-mwmaint[0-9]+$ }'

which targets the following instances:

deployment-jobrunner03.deployment-prep.eqiad1.wikimedia.cloud,deployment-mediawiki-[07,09].deployment-prep.eqiad1.wikimedia.cloud,
deployment-mwmaint01.deployment-prep.eqiad1.wikimedia.cloud,deployment-parsoid11.deployment-prep.eqiad1.wikimedia.cloud,
deployment-snapshot02.deployment-prep.eqiad1.wikimedia.cloud

Upgrade plan on deployment-prep:

• add profile::mediawiki::php::icu63: true to hiera for deployment-prep project prefix; this will only have impact on hosts including the appropriate mediawiki manifest
• apt-get update on deploy*, mediawiki*, jobrunner*, parsoid*, snapshot*
• export DEBIAN_FRONTEND=noninteractive; apt-get -y libicu63 libxml2 on all of the above
• export DEBIAN_FRONTEND=noninteractive; apt-get install php7.2-bcmath php7.2-bz2 php7.2-cli php7.2-common php7.2-curl php7.2-dba php7.2-fpm php7.2-gd php7.2-gmp php7.2-intl php7.2-json php7.2-mbstring php7.2-mysql php7.2-opcache php7.2-readline php7.2-xml php-apcu php-cli php-common php-excimer php-geoip php-igbinary php-luasandbox php-memcached php-mongodb php-msgpack php-redis php-tideways-xhprof php-wmerrors -y on all of the above

running pages-meta-history for wikidata parts 18 - 23 in screen on snapshot1009, and 23-27 on snapshot1010, one last time, hoping to have te above deployed by Dec 1.

I will updating to icu63 in deployment-prep, with Moritz looking on. This will likely happen later today, and I'll post updates about the progress.

Welp. In the year 2020 it turns out I still need a long int in order to store byte counts that could get up to 3.5GB. So revsperpage gave values that were much too small for cumulative rev lengths and so on, and the code to split up page content jobs into smallish page rangs, mostly didn't.

Today's report:

<13>Nov  3 17:45:52 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201102/commonswiki-20201102-cirrussearch-file.json.gz

Aaaaand I'm wrong. There was a reference to it in the defaults, I swear I checked that but oh well. So the revinfo files are all created as they should be. I'll know if they have the desired effect when I see how the wikidata pagecontent meta-history jobs start off, probably a few days yet.

Dang it, I was so excited to check if the revinfo stuff was working today but I forgot to add an entry to the path for it in the production configs, after all my careful local testing >_< I'll check in on the jobs again tomorrow.

This looks a lot better, still a spike but half the size. There/s probably more that could be done yet.

In T267077#6598886, @nnikkhoui wrote:

Thanks @ArielGlenn! I put a really simple/generic blurb in the attached patchset, please feel free to comment/amend however you think would be best.

I can say something about the "dump" group if someone points me at a location and tells me an appropriate format.

Yeah fine, we are talking about the rule right now. Please give us a few minutes though :-P

In fact I had to deploy a couple quick fixes to manage directory creation at the start of a dump run, see https://gerrit.wikimedia.org/r/c/operations/dumps/+/637855 and https://gerrit.wikimedia.org/r/c/operations/dumps/+/637856

All of those tables are there: see https://gerrit.wikimedia.org/r/c/operations/puppet/+/527505 and current https://github.com/wikimedia/puppet/blob/production/modules/snapshot/files/dumps/table_jobs.yaml#L142

I believe all of the functionality including error handling is there. I need a few more integration-ish tests yet before this can go live, and then there will be a puppet patch needed to add a stage to the regular snapshots (not running wikidata or enwiki) to be secondary batch workers on wikidata for the full run.

In T264298#6511634, @Lucas_Werkmeister_WMDE wrote:

We also realized that the tablejobs.yaml file didn’t mention the new tables (the replacement for wb_terms: wbt_{item,property}_terms, wbt_{term,text}_in_lang, wbt_text, wbt_type). If wb_terms was worth dumping, then presumably the new tables should be dumped too. Is it enough to add them to the YAML file or do you need some extra setup for new tables?

Last week's report that i added in a comment and forgot to save:

<13>Oct 14 01:47:12 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/enwiki-20201012-cirrussearch-general.json.gz
<13>Oct 15 00:41:35 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/srwiki-20201012-cirrussearch-content.json.gz
<13>Oct 15 00:49:59 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/srwikisource-20201012-cirrussearch-content.json.gz
<13>Oct 15 00:55:24 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/svwiki-20201012-cirrussearch-content.json.gz

Deployed and live everywhere, I'll leave this task open for the first few days of the next run to make sure that tables jobs run ok and that we have a smaller spike in etcd calls.

Everything done except the puppet run. This will be live everywhere in ~ 30 minutes, well in time for the new run on the 1st of the month. Leaving this task open until that run completes for some large wikis.

In T199890#6586848, @Dzahn wrote:

While this ticket has high prio I'm not that confident it will be resolved based on the history and timeframe so far. Should we keep it open @ArielGlenn ?

It might be nice to prune some of the existing bot entry rows too. There's precedent for this sort of action, see T184485

See T265989 where I have collected a bunch of bad revisions with timestamps across all the wikis. This may let us make some headway.

OIC, this is not using rev_page_id as an index when it really ought to be.

The above patch has been tested locally, in deployment-prep, and has "unit" (not really unit) tests which pass. It can be deployed once the current dump run finishes, probably Monday or Tuesday.

To be done when the current run completes, probably on Monday:

I looked at the config settings for maxrevbytes and revsPerJob and they look ok to keep everywhere for a first run and we'll see what happens.

More can be done, see T266333 which should reduce the numbers even further.

Deployment plan for the above patch, all to be done when the current run is complete, perhaps the 27th or 28th:

• test in deployment-prep:
  • make a copy of the conf file on snapshot02 in deployment-prep
  • add or double-check settings for maxrevbytes, revsPerJob, revinfostash
  • do some test runs to be sure everything still looks good
• merge the mwbzutils patch
• merge the deb packaging patch for mwbzutils
• push the new package up to our repo
• manually install the new package on all snapshot hosts
• merge the python patch, deploy dumps via scap
• add patchset to puppet with new or double-checked settings for bigwikis, en, wd:
  • • maxrevbytes
    • revsPerJob
    • revinfostash
  • add the tested changes to conf file for WMCS as well
  • merge and deploy puppet patches, run puppet everywhere

A note about the range of affected revisions on the wikis that have a large number of them:

• dewiki, from 2002-09-13 to 2005-05-14 and then also 2006-04-09 and 2009-03-09
• enwiki, from 2001-10-01 to 2005-08-25 and then also 2006-04-09 and 2009-03-09
• eswiki, from 2004-02-19 through 2004-11-25 and then 2009-03-09
• nlwiktionary: 2003-12-23, and then 2004-04-03 to 2004-07-25

The following wikis have revisions with an empty sha1:
anwiki azwiki cawiki commonswiki dewiki dewikiversity dewiktionary elwiki enwiki eswiki eswiktionary etwiki frwiki hrwiki nlwiki nlwiktionary nowiki ocwiki plwiki ptwiki slwiki viwiki zhwiki

In T263587#6563095, @jcrespo wrote:

<snip>

I'm running a crap bash script on the fallback dumps nfs server (dumpsdata1003), crunching metadata xml files to see the pattern of empty sha1s for revisions across all the wikis. I'll drop a report in here when it's complete. Running in screen session from ariel, as dumpsgen user.

Note that the initial report via IRC of over a thousand errors seems to be only these sorts of errors:

dumpsgen@snapshot1005:/mnt/dumpsdata/xmldatadumps/public/nlwiktionary/20201020$ zcat nlwiktionary-20201020-stub-meta-history.xml.gz  | grep '<sha1 />' | wc -l
1817

Note that these errors can be regenerated at any time by going to a relatively idle snapshot host and, as the dumpsgen user, running

php /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=nlwiktionary --full --stub --report=1000 --output=file:/mnt/dumpsdata/temp/dumpsgen/nlwiktionary-20201020-stub-meta-history.xml --start=37917 --end 37918

Might be a case for findBadBlobs.php and I really think we'll see this on a bunch of wikis.

So sometime in 2004-06 things were broken.

Page id: 37917 on nlwiktionary

Today's report:

<13>Oct 14 01:47:12 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/enwiki-20201012-cirrussearch-general.json.gz
<13>Oct 15 00:41:35 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/srwiki-20201012-cirrussearch-content.json.gz
<13>Oct 15 00:49:59 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/srwikisource-20201012-cirrussearch-content.json.gz
<13>Oct 15 00:55:24 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/svwiki-20201012-cirrussearch-content.json.gz

No, not ready. The problem remains that these scripts are in production puppet, and cloning all of puppet info mw-vagrant seems bad, but so does copying the scripts wholesale and committing the copies to the mw-vagrant repo.

I'd like eventually to run it across all wikis and get a sense of the bad timeframes. Just my 2 cents.

In case we wanted to cannibalise some servers from the restbase cluster as we move their content to parsercache backends, assuming such a thing were feasible on the software side:

So... there's the new approach, which once cleaned up, takes about 1 hour and 20 minutes to generate all wikidata page ranges, with 0 db calls. That's the way we like it. The actual patchset has not been tested whatsoever, but rather the revsperpage util and a standalone script making use of it.

They cannot be fixed in the dump; they are truncated on the wiki itself. That's what the sql query shows. Someone will have to go onto rowiki and find out what is going on with the entry of those sortkeys and why they are bad.

You have a couple of options. You can replace/ skip / ignore the bad charcters, see https://docs.python.org/3/howto/unicode.html and look for the paragraph starting "The errors argument specifies the response when the input string can’t be converted according to the encoding’s rules." Alternatively you can split the line into entries yourself and just pull out the ones you want. Whichever is easiest for you!

It looks to me like there are truncated values for the cl_sortkey for rowiki, which prevent the utf8 conversion on line 55 of the pastebin from working. This leads to your lines all remaining essentially byte-encoded with the results you see when displaying the content. I would look into the sortkeys as stored on rowiki and see what's going on. By contrast when I look at e.g. elwiki's category links, there is plenty of non-ascii text there but no bad entries in the table.

I see some stuff on stat1007 in /srv/geoip/archive/ by date; are those what you need?

mwbzutils is packaged and a commit is ready for that: https://gerrit.wikimedia.org/r/c/operations/debs/mwbzutils/+/633167

This has been done, following the procedure described here: https://wikitech.wikimedia.org/wiki/SRE_Clinic_Duty#Mail_aliases Changes are now live.