@elukey On our previous server we let people pull from us and it was very difficult to manage upgrades or any sort of maintenance. Somewhere there's a ticket with the awfulness.

https://github.com/apergos/misc-wmf-crap/tree/master/glyph-image-generator Starting to get clever about this: ability to generate 50k small images with metadata that can be extracted for using in depicts and/or caption statements.

I'm looking at deployment-db05 now, and there are 63332 rows in the revision table, with 53250 rows in the content table. I guess we need to double the number of revisions and then add the structured data for those entries. we can probably be clever about this via a script.

@Smalyshev Do you know how many entries have structured data on deployment-prep? Is that a useful testing ground right now or should we be populating the data over there first?

This appears to be working as it should. Closing.

I'm not thinking about the amount of time it takes, but rather the load on the database servers. Reasonable sized batched queries will be better, as I've seen already with stub dumps and slot retrieval.

I think T222497 should be resolved before this goes live. I can test it in deployment-prep before then, but I don't want to do production tests until there is some sort of batching.

Thanks a lot! I've updated the patch above to remove those entries. Now just waiting on the wb_terms migration to get further along.

The entry in the text row points to a non-existent blob in a cluster.

wikiadmin@10.64.48.34(zhwiki)> select * from text where old_id = 9375723;
+---------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
| old_id  | old_namespace | old_title | old_text         | old_comment | old_user | old_user_text | old_timestamp | old_minor_edit | old_flags           | inverse_timestamp |
+---------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
| 9375723 |             0 |           | DB://cluster20/0 |             |        0 |               |               |              0 | utf-8,gzip,external |                   |
+---------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
1 row in set (0.00 sec)

The id of 0 after the cluster20 address is the issue, just like other entries on this ticket.

The python scripts at the dump end are (mostly) protected against exceptions from MediaWiki generally and from this failure case in particular. Since we have problematic data in production I've re-opened the ticket so that the WikiBase issue can somehow be resolved.

Most of these were handled in https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/524666/ but not quite all.

I should clarify; I think the only thing that is needed is to set the content model column for those rows in the content table to 1 (or whichever model is 'wikitext'). The dewikiversity tickets are similar.

I expected that; this requires direct intervention at the db level. I was sort of hoping you were volunteering to do it :-D

@MarcoAurelio Wonderful! It's not the pages though; in the page entry for each of the bad revisions, the content model is listed there as wikitext. It's only in the content table where the wrong content model (4) is shown.
The entries in the content table are listed in T207627#5105046 (double-checked just now to be sure the list is still the same). The slot, revision and pages infocorresponding to each of those is listed in the same comment, double-checked just now to be sure none of that changed either.

@aaron This revision rEFLR848ef073fa89036c40c440016a8092690ddcf56b for FlaggedRevs seems to indicate that flaggedrevs_stats and flaggedrevs_stats2 are no longer used. Do you know or can you point me to someone who could verify that this is the case? If they aren't used, I will add them to my 'don't ever dump these' list. Thanks!

I'm going to decline this because we would have to walk through and decide which entries can be published and which ones not .
This should be done by letting MediaWiki do the work, rather than re-implementing the logic in the python scripts and needing to keep it in sync.
The pages-logging xml dumps already do that for us.

I sincerely apologize: this weekend the heat baked my brain and I did nothing related to computers at all. And Friday evening I was out. I'll set a notification to remind me this coming Friday earlier in the day, so that this gets done.

The category links are the fallback that was designed, so this is a net positive. Going to go update the CR on the patch.

Ran the following with old and new code for abstracts:

/usr/bin/php7.2 /srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=anwiki /srv/v/mediawiki/php-1.34.0-wmf.15/extensions/ActiveAbstract/includes/AbstractFilter.php  --full --report=1 --output=file:/mnt/dumpsdata/temp/dumpsgen/abstracts-anwiki-cr-testing.txt.test  --filter=noredirect --filter=abstract --skip-header --start=36142 --skip-footer --end 36150

is this still an issue? It wasn't even on my radar but I saw it now by a change search in phab.

The wikis ran to completion, but I forgot to close this. Doing so now!

The refactor patchset now checks out with all the wikidata dumps including json. I'd like to deploy it this weekend, giving plenty of time to make sure it's ok, test the structured data patchset, and then be able to deploy that separately.

If this gets done, stub dumps will go back to working for urwikibooks and dewikiversity. The other option at the moment is to do run the jobs manually with a hacked copy of XmlDumpWriter, which is not ideal. I'd like to not be doing that for more than a few more runs (say, a month).

It looks like there is already such a ticket: T220608 which if I read it correctly would return the raw tex as it appears in the database, for purposes of jobs like the dumps. Should I move this to the Inbox column then?

All but two wikis (commons, wikidata) have completed flow dumps successfully, which is good enough for me. Closing.

The fix looks good but I'll keep this open until one of the big wikis like frwiki completes this step.

@Daimona Thank you!

I have tested the above on snapshot1008 for kabwiki and the job completes with normal output.

Adding @Daimona as the person who did the phan patch, and can correct the fix if something else is more appropriate.

These dumps need to complete by the end of the month, so we should get the fix in and deployed by the 30th.

Broken in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Flow/+/495966/ at line 25 (patched) https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Flow/+/495966/20/includes/Search/Iterators/TopicIterator.php, I guess this needs to be "public $orderByUUID = false;"

I have run dewikiveristy and urwikibooks stubs manually with the patched XmlDumpWriter to unblock the current dump run.

In T207626#5365568, @Jdforrester-WMF wrote:

List of sysops on dewikiversity: https://de.wikiversity.org/wiki/Spezial:Benutzer?username=&group=sysop&wpsubmit=&wpFormIdentifier=mw-listusers-form&limit=50
Link they need to use: https://de.wikiversity.org/wiki/Spezial:ChangeContentModel/Thema:Was_wir_h%C3%B6ren_und_sehen

The revision, the only one for the page, was created in 2011 with the unfortunate prefix 'Thema:' while Flow was enabled in 2015 on the wiki.

In T207626#4866435, @Nemo_bis wrote:

In T207626#4865526, @Zoranzoki21 wrote:

Should https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Flow/+/master/maintenance/convertToText.php resolve this job?

No. The page already has a wikitext content model. It only needs to be returned to namespace 0 by running the usual namespaceDupes.php.

Copying here the comments from the patchset since it's getting a bit long.

Daniel Kinzler
12:02 AM

Stubs for frwiki finished a little bit ago. Closing!

Note that the actual content of these revisions is in fact wikitext; when Flow was disabled the content model was not altered properly. See T220594#5101026 for extracted text for one of them.

Previous stubs had this entry for the example revision in dewikiversity:

<page>
  <title>Was wir hören und sehen</title>
  <ns>2600</ns>
  <id>47279</id>
  <revision>
    <id>274772</id>
    <timestamp>2011-08-07T10:46:42Z</timestamp>
    <contributor>
      <username>MartinKurz</username>
      <id>11256</id>
    </contributor>
    <comment>Automatische Zusammenfassung: Die Seite wurde neu angelegt.</comment>
    <model>wikitext</model>
    <format>text/x-wiki</format>
    <text id="269201" bytes="352" />
    <sha1>o9mhxk86c2bxcvymcwsiirc2vyczgnp</sha1>
  </revision>
</page>

Woops, I codged the above together before seeing your comment, @daniel Feel free to reject, replace, whatever is appropriate.

Looking at where this occurs, I'd really prefer to have the Flow revisions set to the right content model for these two wikis, if it's something that can be done in the next 2-3 days. See T220594#5100772 for one of these revisions where the model name is wrong.

One of the wikis named on the ticket, frwiki, is still running stubs; I'll close this when it completes.

Yes for enwiki and dewiki, but now there is T228921 just to keep the fun coming.

It's still a side effect of https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/464768/ We could add one more exception to invokeLenient() in XmlDumpWriter.php I guess, so that the stubs for these last wikis can run (and the rest of the dump steps for those wikis too). But the underlying issue should be fixed as well. This Flow issue was reported earlier at T220793

Same request id also has

PHP Notice: Trying to get property 'gb_anon_only' of non-object

at

#0 /srv/mediawiki/php-1.34.0-wmf.15/extensions/GlobalBlocking/includes/api/ApiGlobalBlock.php(46): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.34.0-wmf.15/includes/api/ApiMain.php(1583): ApiGlobalBlock->execute()
#2 /srv/mediawiki/php-1.34.0-wmf.15/includes/api/ApiMain.php(531): ApiMain->executeAction()
#3 /srv/mediawiki/php-1.34.0-wmf.15/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#4 /srv/mediawiki/php-1.34.0-wmf.15/api.php(86): ApiMain->execute()
#5 /srv/mediawiki/w/api.php(3): require(string)
#6 {main}

Note that this will currently fail if used with revisions with garbage in them, because getSha1() and getSize() can both force a content reload.

The above has been tested with the problematic frwiki and slwiki pages, and processes the bad revisions appropriately.

So very close. There are some instances of InvalidArgumentException thrown when going down the getSha1() -> getContent() rabbithole, for text addresses like DB://cluster16/54423 where cluster16 leads nowhere. Seen today for testwiki, slwiki, and frwiki. Sample stack trace:

InvalidArgumentException from line 226 of /srv/mediawiki/php-1.34.0-wmf.14/includes/libs/rdbms/lbfactory/LBFactoryMulti.php: Wikimedia\Rdbms\LBFactoryMulti::newExternalLB: Unk\
nown cluster "cluster16"
#0 /srv/mediawiki/php-1.34.0-wmf.14/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(246): Wikimedia\Rdbms\LBFactoryMulti->newExternalLB('cluster16')
#1 /srv/mediawiki/php-1.34.0-wmf.14/includes/externalstore/ExternalStoreDB.php(146): Wikimedia\Rdbms\LBFactoryMulti->getExternalLB('cluster16')
#2 /srv/mediawiki/php-1.34.0-wmf.14/includes/externalstore/ExternalStoreDB.php(156): ExternalStoreDB->getLoadBalancer('cluster16')
#3 /srv/mediawiki/php-1.34.0-wmf.14/includes/externalstore/ExternalStoreDB.php(259): ExternalStoreDB->getSlave('cluster16')
#4 /srv/mediawiki/php-1.34.0-wmf.14/includes/externalstore/ExternalStoreDB.php(65): ExternalStoreDB->fetchBlob('cluster16', '54423', false)
#5 /srv/mediawiki/php-1.34.0-wmf.14/includes/externalstore/ExternalStoreAccess.php(52): ExternalStoreDB->fetchFromURL('DB://cluster16/...')
#6 /srv/mediawiki/php-1.34.0-wmf.14/includes/Storage/SqlBlobStore.php(427): ExternalStoreAccess->fetchFromURL('DB://cluster16/...', Array)
#7 /srv/mediawiki/php-1.34.0-wmf.14/includes/libs/objectcache/WANObjectCache.php(1412): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(false, 604800, Array, NULL)
#8 /srv/mediawiki/php-1.34.0-wmf.14/includes/libs/objectcache/WANObjectCache.php(1258): WANObjectCache->fetchOrRegenerate('global:BlobStor...', 604800, Object(Closure), Array)
#9 /srv/mediawiki/php-1.34.0-wmf.14/includes/Storage/SqlBlobStore.php(431): WANObjectCache->getWithSetCallback('global:BlobStor...', 604800, Object(Closure), Array)
#10 /srv/mediawiki/php-1.34.0-wmf.14/includes/Storage/SqlBlobStore.php(358): MediaWiki\Storage\SqlBlobStore->expandBlob('DB://cluster16/...', Array, 'tt:1236282')
#11 /srv/mediawiki/php-1.34.0-wmf.14/includes/Storage/SqlBlobStore.php(286): MediaWiki\Storage\SqlBlobStore->fetchBlob('tt:1236282', 0)
#12 /srv/mediawiki/php-1.34.0-wmf.14/includes/libs/objectcache/WANObjectCache.php(1412): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(false, 604800, Array, NULL\
)
#13 /srv/mediawiki/php-1.34.0-wmf.14/includes/libs/objectcache/WANObjectCache.php(1258): WANObjectCache->fetchOrRegenerate('global:BlobStor...', 604800, Object(Closure), Array\
)
#14 /srv/mediawiki/php-1.34.0-wmf.14/includes/Storage/SqlBlobStore.php(288): WANObjectCache->getWithSetCallback('global:BlobStor...', 604800, Object(Closure), Array)
#15 /srv/mediawiki/php-1.34.0-wmf.14/includes/Revision/RevisionStore.php(1464): MediaWiki\Storage\SqlBlobStore->getBlob('tt:1236282', 0)
#16 /srv/mediawiki/php-1.34.0-wmf.14/includes/Revision/RevisionStore.php(1673): MediaWiki\Revision\RevisionStore->loadSlotContent(Object(MediaWiki\Revision\SlotRecord), NULL, \
NULL, NULL, 0)
#17 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Object(MediaWiki\Revision\SlotRecord))
#18 /srv/mediawiki/php-1.34.0-wmf.14/includes/Revision/SlotRecord.php(307): call_user_func(Object(Closure), Object(MediaWiki\Revision\SlotRecord))
#19 /srv/mediawiki/php-1.34.0-wmf.14/includes/Revision/SlotRecord.php(551): MediaWiki\Revision\SlotRecord->getContent()
#20 /srv/mediawiki/php-1.34.0-wmf.14/includes/Revision/RevisionSlots.php(200): MediaWiki\Revision\SlotRecord->getSha1()
#21 [internal function]: MediaWiki\Revision\RevisionSlots->MediaWiki\Revision\{closure}(NULL, Object(MediaWiki\Revision\SlotRecord))
#22 /srv/mediawiki/php-1.34.0-wmf.14/includes/Revision/RevisionSlots.php(202): array_reduce(Array, Object(Closure), NULL)
#23 /srv/mediawiki/php-1.34.0-wmf.14/includes/Revision/RevisionStoreRecord.php(174): MediaWiki\Revision\RevisionSlots->computeSha1()
#24 /srv/mediawiki/php-1.34.0-wmf.14/includes/export/XmlDumpWriter.php(309): MediaWiki\Revision\RevisionStoreRecord->getSha1()
#25 /srv/mediawiki/php-1.34.0-wmf.14/includes/export/XmlDumpWriter.php(389): XmlDumpWriter->invokeLenient(Object(MediaWiki\Revision\RevisionStoreRecord), 'getSha1', Array, 'fa\
iled to deter...')
#26 /srv/mediawiki/php-1.34.0-wmf.14/includes/export/WikiExporter.php(530): XmlDumpWriter->writeRevision(Object(stdClass), Array)
#27 /srv/mediawiki/php-1.34.0-wmf.14/includes/export/WikiExporter.php(473): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\ResultWrapper), Object(stdClass))
#28 /srv/mediawiki/php-1.34.0-wmf.14/includes/export/WikiExporter.php(287): WikiExporter->dumpPages('page_id >= 1200...', false)
#29 /srv/mediawiki/php-1.34.0-wmf.14/includes/export/WikiExporter.php(172): WikiExporter->dumpFrom('page_id >= 1200...', false)
#30 /srv/mediawiki/php-1.34.0-wmf.14/maintenance/includes/BackupDumper.php(289): WikiExporter->pagesByRange(12002, 12003, false)
#31 /srv/mediawiki/php-1.34.0-wmf.14/maintenance/dumpBackup.php(82): BackupDumper->dump(1, 1)
#32 /srv/mediawiki/php-1.34.0-wmf.14/maintenance/doMaintenance.php(99): DumpBackup->execute()
#33 /srv/mediawiki/php-1.34.0-wmf.14/maintenance/dumpBackup.php(144): require_once('/srv/mediawiki/...')
#34 /srv/mediawiki/multiversion/MWScript.php(101): require_once('/srv/mediawiki/...')
#35 {main}

I had to log in again a few days ago but the week before that was fine. Maybe my login just expired, as they do.

With the above patch, the problematic command in the task description ran to completion and output was (almost) identical to an earlier run without the bug. Besides the issue raised in T228763 which is completely separate, the empty sha1 tag is written ',sha1 />' with a space before the closing slash, as opposed to the old code, which we can ignore.

We might want both of the above. Seen from the dumps lens:

In T228720#5356422, @daniel wrote:

...

What we could also do is tell the SlotRecord that it shouldn't try to auto-calculate, at the time it is being constructed. Calculating the sha1 on the fly is only needed when reading from an old pre-MCR database, and when constructing a revision programmatically. When reading from the MCR schema, this really doesn't make much sense.

In T228720#5356451, @daniel wrote:

I updated my post while you responded. Sorry about that!

In T228720#5356430, @ArielGlenn wrote:

When the revision content is nonexistent,/unreachable what will the sha1 be? We have plenty of these in the db already from old bugs. Reading revision metadata should not load the content if we don't want it to. Note that if we caught all exceptions in XMLDumpWriter we would have missed yesterday's issue.

Only if we catch and ignore. We should catch and report.

In an ideal world we would watch logstash output closely and see things like this right away. Unfortunately, we're living in this crappy one...

When the revision content is nonexistent,/unreachable what will the sha1 be? We have plenty of these in the db already from old bugs. Reading revision metadata should not load the content if we don't want it to. Note that if we caught all exceptions in XMLDumpWriter we would have missed yesterday's issue. I agree that we should catch them but I want to make sure we're not loading content where we aren't asking for it, first.

Done and deployed.

@daniel Relying on https://meta.wikimedia.org/wiki/Cunningham%27s_Law here is a patch. It fiddles with getSize() as well because that can also load content when it should not. I think getSize() and getSha1() are the only such methods but if you know differently, please add them to the mix.

This looks like it's caused by https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/464768/ at line 355 of the new XmlDumpWriter.php, which calls $rev->getSha1(). This method will try to load content of the revision and compute the sha1 directly if the sha1 is NULL. We need a way to override that behavior, probably within the method itself. Perhaps an argument load_content normally set to True. All those class attributes are private so we can't just grab the field value directly like the XMLDumpWriter code used to do.

@Krinkle I'm suspicious that this can fail on different days with different versions of MediaWiki and on different hosts but in the exact same spot in the code. I would not expect opcache issues to lead to this sort of predictable breakage in the general case. Can we get the values of what MediaWiki thinks are in there for $batchKey, $this->titleInfo and $pageNames at the point where that exception is thrown? Would it be awful to put a try catch around that little stanza, log an entry with those values and then raise?

From IRC conversation, Krinle says he ran

php7adm /opcache-free

and the problem immediately went away in the specific instance.

Thanks to Daniel and James for review and merge. This has been deployed. I verified that the new code runs to completion on the page with the problematic revision, and that Special:Export for pages with revision history produces what we expect. I won't close the ticket just yet, I'd like to watch the run through tomorrow and make sure there are no unexpected consequences.

This should have been fixed in https://gerrit.wikimedia.org/r/#/c/mediawiki/tools/mwdumper/+/191555/ if someone can verify.