Seeing this message climb in error logs at roughly the same rate as T281784: Flow\Exception\InvalidDataException: Revisions for w3m0hl06e77t05d6 could not be found, considering they're both coming from Flow there may be a root common cause, see: https://www.mediawiki.org/wiki/MediaWiki_1.43/wmf.4/Changelog#Flow

It looks like all Flow pages are inaccessible, or at least I can't find any that works. I think this warrants a train rollback.

I can't reproduce locally or on https://patchdemo.wmflabs.org/.

Some of the pages no longer show errors after I refreshed them a couple of times. For example, https://www.mediawiki.org/wiki/User_talk:Trizek_(WMF)/Flow loads for me now (it didn't work previously per one of the dupes), and https://en.wikipedia.beta.wmflabs.org/wiki/Talk:Flow_QA also loads (t didn't work for me before). However, https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Talk:Flow_QA&action=history still shows an error. I don't know if this is some stupid cache fixing itself, or some stupid cache hiding the problem.

I reproduced locally after running update.php, which purged caches.

I bisected, it's caused by rEFLW5f16f4fd308d: Migrate to IReadableDatabase::newSelectQueryBuilder.

Change #1029560 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/Flow@master] Revert "Migrate to IReadableDatabase::newSelectQueryBuilder"

https://gerrit.wikimedia.org/r/1029560

I haven't found yet which part of that large commit caused the problem, but it reverts cleanly, and the revert fixes the problem for me locally.

Also, the Flow\Exception\WikitextException is just one of many exceptions – this seems to be coming from some code that tries (and fails) to rescue the situation where it can't find some posts. Locally I'm also seeing errors like "RuntimeException: Did not load root post u21d6c1jv3hpselj", which seems closer to the real problem.

Change #1029562 had a related patch set uploaded (by Zabe; author: Bartosz Dziewoński):

[mediawiki/extensions/Flow@wmf/1.43.0-wmf.4] Revert "Migrate to IReadableDatabase::newSelectQueryBuilder"

https://gerrit.wikimedia.org/r/1029562

In T364499#9784376, @matmarex wrote:

Also, the Flow\Exception\WikitextException is just one of many exceptions – this seems to be coming from some code that tries (and fails) to rescue the situation where it can't find some posts. Locally I'm also seeing errors like "RuntimeException: Did not load root post u21d6c1jv3hpselj", which seems closer to the real problem.

In logstash there are also errors like the following, which are propably the more interresting ones.

• service.version: 1.43.0-wmf.4
• trace.id: ed835db3-5f64-4358-8774-2d1e64a85f81
• Find trace.id in Logstash

[{reqId}] {exception_url}   Flow\Exception\InvalidDataException: Revisions for w7mw4zwj9kdzaede could not be found

from /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/Collection/AbstractCollection.php(148)
#0 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/Collection/LocalCacheAbstractCollection.php(32): Flow\Collection\AbstractCollection->getAllRevisions()
#1 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/Collection/LocalCacheAbstractCollection.php(123): Flow\Collection\LocalCacheAbstractCollection->getAllRevisions()
#2 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/Formatter/AbstractQuery.php(340): Flow\Collection\LocalCacheAbstractCollection->getLastRevision()
#3 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/Formatter/AbstractQuery.php(220): Flow\Formatter\AbstractQuery->getCurrentRevision(Flow\Model\PostRevision)
#4 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/Formatter/BoardHistoryQuery.php(46): Flow\Formatter\AbstractQuery->buildResult(Flow\Model\PostRevision, string)
#5 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/Data/Pager/HistoryPager.php(47): Flow\Formatter\BoardHistoryQuery->getResults(Flow\Model\UUID, integer, NULL, string)
#6 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/Block/BoardHistoryBlock.php(68): Flow\Data\Pager\HistoryPager->doQuery()
#7 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/View.php(233): Flow\Block\BoardHistoryBlock->renderApi(array)
#8 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/View.php(72): Flow\View->buildApiResponse(Flow\WorkflowLoader, array, string, array)
#9 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/Actions/FlowAction.php(103): Flow\View->show(Flow\WorkflowLoader, string)
#10 /srv/mediawiki/php-1.43.0-wmf.4/extensions/Flow/includes/Actions/FlowAction.php(50): Flow\Actions\FlowAction->showForAction(string)
#11 /srv/mediawiki/php-1.43.0-wmf.4/includes/actions/ActionEntryPoint.php(731): Flow\Actions\FlowAction->show()
#12 /srv/mediawiki/php-1.43.0-wmf.4/includes/actions/ActionEntryPoint.php(508): MediaWiki\Actions\ActionEntryPoint->performAction(Article, MediaWiki\Title\Title)
#13 /srv/mediawiki/php-1.43.0-wmf.4/includes/actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest()
#14 /srv/mediawiki/php-1.43.0-wmf.4/includes/MediaWikiEntryPoint.php(199): MediaWiki\Actions\ActionEntryPoint->execute()
#15 /srv/mediawiki/php-1.43.0-wmf.4/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#16 /srv/mediawiki/w/index.php(3): require(string)
#17 {main}

Change #1029562 merged by jenkins-bot:

[mediawiki/extensions/Flow@wmf/1.43.0-wmf.4] Revert "Migrate to IReadableDatabase::newSelectQueryBuilder"

https://gerrit.wikimedia.org/r/1029562

Found the bug: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/1026106/comment/e2e9c566_9bbceb5f/

This would be obvious in testing, if it weren't for an annoying caching later covering up the mistake.

Let's go with the revert now, but we can try to re-apply the patch with the fix next week.

Change #1029560 merged by jenkins-bot:

[mediawiki/extensions/Flow@master] Revert "Migrate to IReadableDatabase::newSelectQueryBuilder"

https://gerrit.wikimedia.org/r/1029560

Mentioned in SAL (#wikimedia-operations) [2024-05-09T19:42:03Z] <jhuneidi@deploy1002> Started scap: Backport for [[gerrit:1029562|Revert "Migrate to IReadableDatabase::newSelectQueryBuilder" (T312418 T364499)]]

Mentioned in SAL (#wikimedia-operations) [2024-05-09T19:44:43Z] <jhuneidi@deploy1002> jhuneidi and zabe: Backport for [[gerrit:1029562|Revert "Migrate to IReadableDatabase::newSelectQueryBuilder" (T312418 T364499)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-05-09T19:59:41Z] <jhuneidi@deploy1002> Finished scap: Backport for [[gerrit:1029562|Revert "Migrate to IReadableDatabase::newSelectQueryBuilder" (T312418 T364499)]] (duration: 17m 37s)

As a follow-up, I also wrote a patch to remove the useless system that made it harder to find the cause of the problem: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/1029708