Will come back to this on Friday or next week. Once 6394b76d67d57b9708ae7ede1d388152f67c284b is in production, we'll have better insight into the exception that's thrown as well as some more information in the logs. We can also see if rolling back the DBs then works as intended.

Also, here's a debug script to print out the errors (we need to fill in the user ID for users we want to look at): P7615. But let's wait until after the train is in production so that exceptions will be caught properly.

Change 464871 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/Flow@master] Log block errors in OptIn/OptOut process

https://gerrit.wikimedia.org/r/464871

Change 463791 abandoned by Kosta Harlan:
WIP: Add job for fixing boards when InvalidDataException is thrown

Reason:
We can handle these errors on a case by case basis.

https://gerrit.wikimedia.org/r/463791

We have some additional logging in place now, and are properly catching exceptions in the OptInController, so we will come back to this if/when fresh instances of the problem are reported.

Change 464871 merged by jenkins-bot:
[mediawiki/extensions/Flow@master] Log block errors in OptIn/OptOut process

https://gerrit.wikimedia.org/r/464871

Change 455897 had a related patch set uploaded (by Krinkle; owner: Kosta Harlan):
[mediawiki/extensions/Flow@master] WIP: Check permissions before attempting to move talk page

https://gerrit.wikimedia.org/r/455897

Still seen in recent logs. The latest repro case is https://www.wikidata.org/w/index.php?title=User_talk:Seojeongah80&action=history.

[W@JlZApAICwAAFTvFcQAAAAX] 2018-11-07 04:09:08: Fatal exception of type "Flow\Exception\InvalidDataException"

@Krinkle I triggered that one while looking at T201654. I could be wrong but I think that this user's talk page was broken before the various fixes we implemented around this bug were deployed. The user is blocked, so they probably tried to convert their talk page to Flow, an exception was thrown and not caught (since HHVM didn't recognize the catch Error $error syntax.

I can run the script to manually fix this user's page but I don't think it's something that will come up in the future when enabling/disabling flow.

I am leaning towards abandoning https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/455897, 1) rollback on exception is working properly now and 2) in the patch, there's no UI to show to the user that we didn't actually enable Flow when the permissions check failed.

Fixed the user's board just now:

INCONSISTENT: Core title for 'uadxde2amz2ypajk' is 'User talk:Seojeongah80', but Flow title is 'User talk:Seojeongah80/Structured Discussions Archive 1'
FIXED: Updated 'uadxde2amz2ypajk' to match core title, 'User talk:Seojeongah80'

Checked a total of 1200 Flow boards.  Of those, 1 boards had an inconsistent title; 1 were fixed.

Change 455897 abandoned by Kosta Harlan:
WIP: Check permissions before attempting to move talk page

https://gerrit.wikimedia.org/r/455897

Recent cases:

• 2019-01-13 ERROR [XFMPKQpAMFQAAEFAjZQAAACB] mw1249
• 2019-02-01 ERROR [XFSPxwpAAD8AAFTaB90AAACE] mw1268

From https://www.mediawiki.org/wiki/Topic:050e2fb772abdae40bd7842b2b77d26b, which still reproduces the error when viewed, at time of writing.

From https://www.mediawiki.org/wiki/Topic:050e2fb772abdae40bd7842b2b77d26b, which still reproduces the error when viewed, at time of writing.

I looked at this and other examples, it looks like it's all older content. I ran mwscript extensions/Flow/maintenance/FlowFixInconsistentBoards.php --wiki=mediawikiwiki --dry-run but it didn't pick up anything to be fixed.

I think the options here are either 1) a maintenance script to clean up this older, inconsistent content or 2) an alternative to throwing InvalidDataException in WorkflowLoaderFactory.php#loadWorkflowById, perhaps redirecting the user somewhere.

A more recent one T224098

In T224098#5339034, @kostajh wrote:

This part is fine, the application is recognising a certain state it does not support, and reports it properly to the user. It appears up to this point, the response could be HTTP 200 or 40x.

So, perhaps in includes/Actions/Action.php in showForAction, we should do something like

try {
	/** @var WorkflowLoaderFactory $factory */
	$factory = $container['factory.loader.workflow'];
	$loader = $factory->createWorkflowLoader( $title );

	if ( $title->getNamespace() === NS_TOPIC && $loader->getWorkflow()->getType() !== 'topic' ) {
		// @todo better error handling
		throw new FlowException( 'Invalid title: uuid is not a topic' );
	}

	$view->show( $loader, $action );
} catch ( FlowException $e ) {
	$e->setOutput( $output );
	if ( $e instanceof InvalidDataException ) {
		// XXX: output the error message, then return
		return;
	}
	throw $e;
}

There are only six instances in the last 30 days so tentatively setting this for Q2. Contributions to fix this are welcome.

https://www.wikidata.org/wiki/User_talk:PokestarFan

2020-03-21 17:01:00 [XnZITApAADgAABkcr98AAAEX] mw1261 wikidatawiki 1.35.0-wmf.24 exception ERROR: [XnZITApAADgAABkcr98AAAEX] /wiki/User_talk:PokestarFan   Flow\Exception\InvalidDataException from line 130 of /srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/WorkflowLoaderFactory.php: Flow workflow is for different page {"exception_id":"XnZITApAADgAABkcr98AAAEX","exception_url":"/wiki/User_talk:PokestarFan","caught_by":"mwe_handler"} 
[Exception Flow\Exception\InvalidDataException] (/srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/WorkflowLoaderFactory.php:130) Flow workflow is for different page
  #0 /srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/WorkflowLoaderFactory.php(103): Flow\WorkflowLoaderFactory->loadWorkflowById(Title, Flow\Model\UUID)
  #1 /srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/Actions/Action.php(105): Flow\WorkflowLoaderFactory->createWorkflowLoader(Title)
  #2 /srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/Actions/ViewAction.php(20): Flow\Actions\FlowAction->showForAction(string, OutputPage)
  #3 /srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/Actions/Action.php(50): Flow\Actions\ViewAction->showForAction(string)
  #4 /srv/mediawiki/php-1.35.0-wmf.24/includes/MediaWiki.php(519): Flow\Actions\FlowAction->show()
  #5 /srv/mediawiki/php-1.35.0-wmf.24/includes/MediaWiki.php(305): MediaWiki->performAction(Article, Title)
  #6 /srv/mediawiki/php-1.35.0-wmf.24/includes/MediaWiki.php(973): MediaWiki->performRequest()
  #7 /srv/mediawiki/php-1.35.0-wmf.24/includes/MediaWiki.php(535): MediaWiki->main()
  #8 /srv/mediawiki/php-1.35.0-wmf.24/index.php(47): MediaWiki->run()
  #9 /srv/mediawiki/w/index.php(3): require(string)
  #10 {main}

Can someone fix https://www.wikidata.org/wiki/User_talk:PokestarFan / https://www.wikidata.org/w/index.php?title=User_talk:PokestarFan/Structured_Discussions_Archive_1&action=edit&redlink=1 ? Thanks.

In T70526#5989308, @Reedy wrote:

https://www.wikidata.org/wiki/User_talk:PokestarFan

2020-03-21 17:01:00 [XnZITApAADgAABkcr98AAAEX] mw1261 wikidatawiki 1.35.0-wmf.24 exception ERROR: [XnZITApAADgAABkcr98AAAEX] /wiki/User_talk:PokestarFan   Flow\Exception\InvalidDataException from line 130 of /srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/WorkflowLoaderFactory.php: Flow workflow is for different page {"exception_id":"XnZITApAADgAABkcr98AAAEX","exception_url":"/wiki/User_talk:PokestarFan","caught_by":"mwe_handler"} 
[Exception Flow\Exception\InvalidDataException] (/srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/WorkflowLoaderFactory.php:130) Flow workflow is for different page
  #0 /srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/WorkflowLoaderFactory.php(103): Flow\WorkflowLoaderFactory->loadWorkflowById(Title, Flow\Model\UUID)
  #1 /srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/Actions/Action.php(105): Flow\WorkflowLoaderFactory->createWorkflowLoader(Title)
  #2 /srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/Actions/ViewAction.php(20): Flow\Actions\FlowAction->showForAction(string, OutputPage)
  #3 /srv/mediawiki/php-1.35.0-wmf.24/extensions/Flow/includes/Actions/Action.php(50): Flow\Actions\ViewAction->showForAction(string)
  #4 /srv/mediawiki/php-1.35.0-wmf.24/includes/MediaWiki.php(519): Flow\Actions\FlowAction->show()
  #5 /srv/mediawiki/php-1.35.0-wmf.24/includes/MediaWiki.php(305): MediaWiki->performAction(Article, Title)
  #6 /srv/mediawiki/php-1.35.0-wmf.24/includes/MediaWiki.php(973): MediaWiki->performRequest()
  #7 /srv/mediawiki/php-1.35.0-wmf.24/includes/MediaWiki.php(535): MediaWiki->main()
  #8 /srv/mediawiki/php-1.35.0-wmf.24/index.php(47): MediaWiki->run()
  #9 /srv/mediawiki/w/index.php(3): require(string)
  #10 {main}

^ This one popped up in logs again today.

I ran the maintenance script to fix user talk Flow boards on wikidata.

kharlan@mwmaint1002:~$ mwscript extensions/Flow/maintenance/FlowFixInconsistentBoards.php --wiki=wikidatawiki --namespaceName='USER_TALK'

Checked a total of 300 Flow boards.  Of those, 0 boards had an inconsistent title; 0 were fixed.
INCONSISTENT: Core title for 't7r3cj6wabdh0zwd' is 'User talk:Slowking4', but Flow title is 'User talk:Slowking4/Structured Discussions Archive 1'
FIXED: Updated 't7r3cj6wabdh0zwd' to match core title, 'User talk:Slowking4'

Checked a total of 600 Flow boards.  Of those, 1 boards had an inconsistent title; 1 were fixed.
INCONSISTENT: Core title for 'tlwrkelibfuejpsm' is 'User talk:PokestarFan', but Flow title is 'User talk:PokestarFan/Structured Discussions Archive 1'
FIXED: Updated 'tlwrkelibfuejpsm' to match core title, 'User talk:PokestarFan'

Checked a total of 900 Flow boards.  Of those, 2 boards had an inconsistent title; 2 were fixed.
INCONSISTENT: Core title for 'ufkqmeeo0aqdjzac' is 'User talk:Broery2000', but Flow title is 'User talk:Broery2000/Structured Discussions Archive 1'
FIXED: Updated 'ufkqmeeo0aqdjzac' to match core title, 'User talk:Broery2000'

Checked a total of 1200 Flow boards.  Of those, 3 boards had an inconsistent title; 3 were fixed.
INCONSISTENT: Core title for 'v73q4742vi3wb49x' is 'User talk:Slowking4/Structured Discussions Archive 1', but Flow title is 'User talk:Slowking4'
ERROR: 'User talk:Slowking4/Structured Discussions Archive 1' has page ID '66908402', but no workflow is linked to this page ID

Checked a total of 1500 Flow boards.  Of those, 4 boards had an inconsistent title; 3 were fixed.

Checked a total of 1663 Flow boards.  Of those, 4 boards had an inconsistent title; 3 were fixed.

Another report today (6/17/2020, wmf.37) for zhwiki - Phatality ID: ceb774c61be79b30fd3680f027cce1474941960b85057ee11d587b4379ac1e9b

#0 /srv/mediawiki/php-1.35.0-wmf.37/extensions/Flow/includes/WorkflowLoaderFactory.php(103): Flow\WorkflowLoaderFactory->loadWorkflowById(Title, Flow\Model\UUID)
#1 /srv/mediawiki/php-1.35.0-wmf.37/extensions/Flow/includes/Actions/FlowAction.php(99): Flow\WorkflowLoaderFactory->createWorkflowLoader(Title)
#2 /srv/mediawiki/php-1.35.0-wmf.37/extensions/Flow/includes/Actions/ViewAction.php(26): Flow\Actions\FlowAction->showForAction(string, OutputPage)
#3 /srv/mediawiki/php-1.35.0-wmf.37/extensions/Flow/includes/Actions/FlowAction.php(49): Flow\Actions\ViewAction->showForAction(string)
#4 /srv/mediawiki/php-1.35.0-wmf.37/includes/MediaWiki.php(527): Flow\Actions\FlowAction->show()
#5 /srv/mediawiki/php-1.35.0-wmf.37/includes/MediaWiki.php(313): MediaWiki->performAction(Article, Title)
#6 /srv/mediawiki/php-1.35.0-wmf.37/includes/MediaWiki.php(986): MediaWiki->performRequest()
#7 /srv/mediawiki/php-1.35.0-wmf.37/includes/MediaWiki.php(543): MediaWiki->main()
#8 /srv/mediawiki/php-1.35.0-wmf.37/index.php(47): MediaWiki->run()
#9 /srv/mediawiki/w/index.php(3): require(string)
#10 {main}

Per chat with @Krinkle, a band-aid solution would be to catch this error so that 5xx level errors aren't logged, SRE isn't pinged, etc. We would still show a generic error message to the user.

@MMiller_WMF @Etonkovidova moving this into our needs discussion column, perhaps we could find some time to implement the quick bandaid solution (show a generic error message to the user and suppressing the underlying error) so this doesn't show up in the production error logs.

Saw a few of these error messages today for https://zh.wikipedia.org/wiki/User_talk:Franklin_Yu

MW: 1.36.0-wmf.9

Impact:
Error message shows up in that page and in production logs.

Change 631267 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/Flow@master] Catch InvalidDataException errors to avoid log spam

https://gerrit.wikimedia.org/r/631267

Change 631267 merged by jenkins-bot:
[mediawiki/extensions/Flow@master] Catch InvalidDataException errors to avoid log spam

https://gerrit.wikimedia.org/r/631267

Bandaid patch was merged, so next is QA to see if that works well in muting those errors.

Checked today (10/14/2020) - wmf.11 is on zhwiki

• the count for the error "Flow workflow is for different page" is 467

@Etonkovidova -- since you've checked this already, should it be moved to a different column? Which one?

In T70526#6558838, @MMiller_WMF wrote:

@Etonkovidova -- since you've checked this already, should it be moved to a different column? Which one?

The patch was in wmf.12, so it needs to be rechecked now that wmf.13 was deployed. And I don't see the error in Logstash anymore since wmf.13 was deployed.

Change 634947 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/Flow@master] Switch permissions errors logging to debug channel

https://gerrit.wikimedia.org/r/634947

Change 634947 merged by jenkins-bot:
[mediawiki/extensions/Flow@master] Switch permissions errors logging to debug channel

https://gerrit.wikimedia.org/r/634947