That time would be ok for me (my evening but it's not too late). @hoo?

OK, it looks like everyone's weighed in. so I'll suggest:

<revision>
  <id>308722154</id>
  ...
  <format>text/x-wiki</format>
  <text location="tt:305112983" sha1="xxxx" bytes="143" />   <-- contains sha1 of content in main slot
  <sha1>a9kdtqq3buy5tribez2u0ad4b6fdxq2</sha1>               <-- revision sha1

Let's shoot for Oct 31 then. @ayounsi What time would the window be? @hoo would you prefer stopping and restarting scripts or just skipping the run for the week?

In T206535#4675178, @hoo wrote:

...

Have you tested importing that via php (and/or anything else that uses the libzip2 compat stuff)?

In T206743#4675755, @Banyek wrote:

on db1124 with instance s8 we have a repliation error as

Last_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log db1087-bin.003073, end_log_pos 582738698

Adding @hoo to see if we can work out timing; we could work out sometime on Oct 30th or 31st but the wikidata weeklies would be interrupted and need restarting manually.

I've enabled the use of lbzip2 for the xml/sql dumps starting with the Oct 20th run; we could consider using this for the wikidata weeklies recompression into bz2 files, at, say, four threads (half the number of shards). As far as I can tell it puts out binary-format compatible dumps to those produced by bzip2, though not byte-identical output. What do folks think?

These changes are now live on the snapshot hosts and will be in effect for the Oct 20th run. The hosts should be monitored more closely than usual to check load and disk i/o (including especially the nfs servers).

Whoever has clinic duty should probably take this and hand it to the right person. I think @Dzahn and/or @MoritzMuehlenhoff may oversee the ubuntu mirror boxes (if not, please excuse the ping).

@lmarlier, can you sign off please?

This is now deployed on snapshot1008 (where cron jobs run). We'll know next Monday if this took care of the problem; let's leave the task open til then.

The above has been tested and is ready for merge and deployment as soon as the current dump run completes, probably in 2-3 days. I"ll prep a changeset for the config setting update in puppet too.

I've done some followup testing with lbzip2, using multiple processors with read from stdin via pipe, examined the low level format of the output files as compared to bzip2 output, and checked memory use. A sample memory use comparison:

The run is working fine. Now we still don't backfill missing runs; I'm going to say that this should be a manual process, which can be done by one command from a screen session should we have such a problem again. That way it can be run on an idle host without doubling the length of the regular daily runs, which already take 15 hours to complete.

In T199121#4643642, @daniel wrote:

Re the ìd`attribute being optional or not: turns out, it's optional already: The "use" attribute of the <attribute> element in an xml schema is indeed optional, and its default value is "optional" :) The fact that this is declared for all other attributes but omitted for id is confusing, though, and should be fixed.

Going to wait for today's run to complete to make sure everything's ok before closing the ticket.

Today's test run looks good. Merging and deploying.

Tests look good so far; I want to run one more tomorrow which will duplicate the circumstances of the duplicate revisions of the Oct 3rd run; if that pans out then this can be deployed.

Adding @hoo and @Smalyshev in hopes that they will have some good ideas.

In T199121#4643036, @daniel wrote:

...

Anything using the existing revision level sha1 for revert detection will miss-detect a revert (or a null-edit) for *all* revisions that did not affect the main slot. While analysis on the slot level may be useful, existing analysis is on the revision level (by definition - slots are new). So it seems reasonable to keep revision-level semantics intact.

Whatever we do, we should definitely include both hashes (main slot and revision), to make the distinction obvious, and the path forward clear, and give consumers the option to change their code to consistently do the thing they want. Only if both hashes are available can we support both kinds of consumers - those that focus on the revision as a whole, and those that focus on individual slots.

There are a few things I've found on looking at the code.

• It doesn't backfill automatically; we'd be able to run the job for those dates and it should 'just work'. This needs testing; potentially the main index.html file for the adds changes dumps might be rewritten with info from these older runs.
• Logging at verbose level is broken (patch coming).
• It correctly finds the max rev id for the previous day's run from the db, even if the previous day's run did not complete. But it writes this information in the current run's max rev id file, which means that the next day's run will have duplicate revisions in it. (Patch coming)

(Sorry for the near-stream-of-consciousness updates here, just trying to Get **It Done.) To get the sha1 discussion started:

https://www.mediawiki.org/wiki/Requests_for_comment/Schema_update_for_multiple_content_objects_per_revision_(MCR)_in_XML_dumps#Schema This has now been updated.

In T174031#4641372, @daniel wrote:

...

Do we currently have Flow content in dumps at all? I mean, the actual content, not the references?

Clarifying question: what slot is Flow content considered to be in, a 'main', or a 'secondary'? All that content has such a different schema in the db tables.

I'll make the changes agreed upon in last night's meeting to the RFC a bit later today and will note here when they are done.

Following up on the deduplication issue raised above:

That fixed the problem, thanks!

https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/459885/ seems to have broken Flow dumps, which use streams:

2018-10-03 11:23:53 failed history content of flow pages in xml format
[48f79972c1cea21e104aa6a5] [no req] Error from line 82 of /srv/mediawiki/php-1.32.0-wmf.24/extensions/Flow/maintenance/dumpBackup.php: Undefined class constant 'STREAM' 
Backtrace: 
#0 /srv/mediawiki/php-1.32.0-wmf.24/extensions/Flow/maintenance/dumpBackup.php(62): FlowDumpBackup->dump(integer) 
#1 /srv/mediawiki/php-1.32.0-wmf.24/maintenance/doMaintenance.php(94): FlowDumpBackup->execute() 
#2 /srv/mediawiki/php-1.32.0-wmf.24/extensions/Flow/maintenance/dumpBackup.php(144): require_once(string) 
#3 /srv/mediawiki/multiversion/MWScript.php(100): require_once(string) 
#4 {main}

In T203075#4637351, @Krinkle wrote:

...

@ArielGlenn Do you know if there are any side-effect or impact from this error that we need to be aware of? Or does the snapshot script account for this issue and recovers without issue?

While not strictly a blocker, some related work is at T205825.

This will be discussed at the TechCom meeting Wednesday, October 3rd at 2pm PST(21:00 UTC, 23:00 CET). The announcement was sent to Wikitech-l: https://lists.wikimedia.org/pipermail/wikitech-l/2018-September/090881.html

Two of the symlink fixes have been deployed, one to go.

In T199121#4619450, @daniel wrote:

...

Sounds mostly good, except that we have to bump the XML schema when we want to do T183490: MCR schema migration stage 4: Migrate External Store URLs (wmf production). If we made id optional & deprecated right away, we wouldn't have to do that. Perhaps this is one of the questions to answer during the RFC discussion.

Yes, sorry I wasn't clear. I meant to include some base data and a script, rather than force an import. since some users may not want that data. Info on how to do the import would go in the README. As a side note.importing can be done as a one-time thing in puppet; the dumps role does this already.

I wasn't happy with the different formats for the same attribute name either, using a different name for the content text id is grand! Using anyURI makes for a simpler schema too.

Soooo... @Legoktm were you able to give it a try, and if so, what happened?

I wish I'd had a pointer to the changes in Storage/Blobstore.php and SqlBlobStore.php earlier, didn't realize this new form of text addressing was baked in already. Anyways, I've revised the draft accordingly, sorry for the long delay. Main slot still only gets an id number for text id (this is required for backwards compat); text id in content element gets schema:location format. Is that pattern regex in the schema ok?

I've moved the dumps output to /var/www/dumps and the webroot to /var/www/dumps/public; what do folks think about that? (Note I have not tested the module with these new changes.)

@Krinkle Of course. But let's not reinstall svn everywhere just to get diffs working on one box, that's all I'm saying.

See T116948: this extension is long gone. It had a wonderful life, and now it rests in peace; let's not devote resources to putting it back. There was a proposal to get an HTML dump of the contents and put them somewhere (T116948). @Legoktm ?

The above change is now live on snapshot1008 (where this job runs) and will take effect for the next run on Monday morning.

@awight What was your thinking behind the choice of /vagrant/srv/dumps/output as the location for dumps output files, as opposed to someplace not on the nfs mount?

What do folks think about putting all the 'misc dump scripts' (puppet/modules/snapshot/files/cron) in their own repo, rather than having them in puppet? They'd have to be deployed by scap I suppose, but they could then be easily cloned into any testing platform instead of each tester having to copy in the scripts to e.g. vagrant or wherever else.

https://apergos.wordpress.com/2018/09/19/xml-sql-dumps-and-mediawiki-vagrant-two-great-tastes-that-taste-great-together/ This is much longer than anyone here cares about, just skip to the last few sections and especially 'next steps'.

Hey @Aklapper, the bot is on toolforge though not currently running. Is that what you're asking?

Sorry, I've been trying to get a couple other things off my plate. Will be coming back to this shortly.

The latest version of the patchset should allow one to apply the wikidata role, then apply the dumps role, then dump any of the wikis configured, including wikidata, and (once the wikidata json, rdf, and dumps_functions and dcat scripts/config are copied in) dump those too.

Well, the mystery of 'why eqiad' is solved; the choice of whether to parse db-eqiad.php or db-codfw.php is determined by the global $wmfDatacenter. This is set in multiversion/MWRealm.php from the value of $wmfCluster, which is taken from the contents of /etc/wikimedia-cluster on the server itself. Since the dumps are run from servers in eqiad, there we have it.

Adding @hoo to see what insights he may have.

Well those were some fun rabbit holes. I'll document what I did, but the long and short of it is, after sorting out github issues and then composer issues, it turns out that there are some annoyances with CommonSettings for the wikidata role. I need to think about the minimal workaround needed for the wikidata role and the dumps role to play well together, as well as the dumps role all by itself.

The wikidata weeklies are running now; the 'regular' (xml/sql) dumps completed on Sept 15th for the first run of the month.

There is not a project for it (yet). There could be. @rosalieper what do you think?

well in applying the wikidata role I got:

==> default: Notice: /Stage[main]/Role::Wikidata/Mediawiki::Extension[Wikibase]/Git::Clone[mediawiki/extensions/Wikibase]/Exec[git_clone_mediawiki/extensions/Wikibase]/returns: executed successfully
==> default: Error: Command exceeded timeout
==> default: Error: /Stage[main]/Role::Wikidata/Mediawiki::Extension[Wikibase]/Php::Composer::Install[/vagrant/mediawiki/extensions/Wikibase]/Exec[composer-install--vagrant-mediawiki-extensions-Wikibase]/returns: change from notrun to 0 failed: Command exceeded timeout
==> default: Notice: /Stage[main]/Role::Wikidata/Mediawiki::Extension[Wikibase]/Mediawiki::Settings[Wikibase]/File[/vagrant/settings.d/puppet-managed/10-Wikibase.php]: Dependency Exec[composer-install--vagrant-mediawiki-extensions-Wikibase] has failures: true
==> default: Warning: /Stage[main]/Role::Wikidata/Mediawiki::Extension[Wikibase]/Mediawiki::Settings[Wikibase]/File[/vagrant/settings.d/puppet-managed/10-Wikibase.php]: Skipping because of failed dependencies

so that's a bit of an issue. Any ideas, @Smalyshev ? This is after applying the dumps role but I can't imagine that makes a difference.

It looks like I might be able to get the job done with importDump.php and a small selection of content from an existing wd dump. I'll experiment some.

In T201478#4572715, @Smalyshev wrote:

Check out also https://www.mediawiki.org/wiki/User:Smalyshev_(WMF)/Dump_Test which describes how I tested dumps and which tweaks may be useful.

Finally. The current patchset works with xml/sql dumps, and I've tested it for category rdf dumps by copying in dump_functions.sh, dumpcategoriesrdf-shared.sh, and dumpcategoriesrdf.sh to /usr/local/etc and/or /usr/local/bin, and also making the log dir /var/log/categoriesrdf owned by vagrant. The log dirs for the various misc scripts can be added in a later patchset; this is just verification that the current code works.

Huh, it needed some changes for that. One more round of testing needed for the latest patch.

The above patch is almost ready for review. I have run the regular xml/sql dumps with it applied, and they run clean. I'd like to try one of the 'misc' dumps that uses the misc crons functions and make sure it runs correctly.

I thought we were capped at 2048k for revision text, so how would we exceed that 32M packet size? But you're right, it would be nice to log the query somehow.

Can we get manager sign-off on this please? Thanks!

We should support testing the 'misc' cron job dumps (from the command line); @Smalyshev did a little work on this for wikibase dumps, see https://gerrit.wikimedia.org/r/#/c/mediawiki/vagrant/+/456673/ and I think there might be more patchsets coming (hopefully :-) )

Pinging @gabriel-wmde, this is just waiting on your input.

It's a week later and the longest runs take about half an hour. I think we can safely close this; I have pending some work to restructure the rsyncs so that we can go at full speed to the labstore hosts, thus reducing issues if we should have a long downtime for those servers for some reason, which means we should never fall too far behind unless one or the other of the dumpsdata hosts dies. And in that case having some delay in recovery is expected.

Frwiki is the only one left, and that is only because it's still running other steps. Closing this task.

Welp, I'm closing it, since we have a ticket for enhancements now!