It is in an inconsistent state, apparently due to being deleted (2015-02-15T09:36:43Z) then immediately automatically recreated on the next visit (2015-02-15T09:36:56Z), but using the same workflow.

The workflow shown in the 'revisions' request (rlam6pl51ksr5uuw) translates to '050bad020d1d3cafb69678':

mattflaschen@tin:~$ mwscript eval.php --wiki=testwiki
> $workflowId = Flow\Model\UUID::create( 'rlam6pl51ksr5uuw' );

> var_export( $workflowId->getHex() );
'050bad020d1d3cafb69678'

The new page ID is 79845:

> var_export( Title::newFromText( 'Talk:Sandbox' )->getArticleId() );
79845

This workflow (see above) is still associated with the old (wrong) page ID (see first API request, logevents), 69155.

mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(workflow_id)), workflow_wiki, workflow_namespace, workflow_page_id, workflow_title_text, workflow_type FROM flow_workflow WHERE LOWER(HEX(workflow_id)) = '050bad020d1d3cafb69678';
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
| LOWER(HEX(workflow_id)) | workflow_wiki | workflow_namespace | workflow_page_id | workflow_title_text | workflow_type |
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
| 050bad020d1d3cafb69678  | testwiki      |                  1 |            69155 | Sandbox             | discussion    |
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
1 row in set (0.06 sec)

This is the only workflow associated with the page:

mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(workflow_id)), workflow_wiki, workflow_namespace, workflow_page_id, workflow_title_text, workflow_type FROM flow_workflow WHERE workflow_wiki = 'testwiki' AND workflow_namespace = 1 AND workflow_title_text = 'Sandbox' AND workflow_type = 'discussion';
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
| LOWER(HEX(workflow_id)) | workflow_wiki | workflow_namespace | workflow_page_id | workflow_title_text | workflow_type |
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
| 050bad020d1d3cafb69678  | testwiki      |                  1 |            69155 | Sandbox             | discussion    |
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
1 row in set (0.01 sec)

The fix is the same as for T97664: A18. Moving Talk:Search to Talk:Search/Old gives a fatal exception, re-running the new version of T96888: A14. Run maintenance/FlowUpdateWorkflowPageId.php in production. I wanted to make sure that would fix the problem, not confuse matters further. I'm now confident it will. It will cause old topics from before the deletion/re-creation to be re-associated with the board; however, they can still be deleted individually, and this is a test page anyway.

There are old topics that are associated with the

None of this should happen again with Matthias's recent fixes, such as d56426484232753d709a2ddc4f7b5170a8d3b250 and d2f37d2cfeca75fc25ac9687d5b3e93bc8ecf300 .

I ran the script. It's associated with the right page ID, now, but it didn't fix the problem.

I've been debugging through (thanks to @EBernhardson for helping me get that set up). It is a problem with one topic, https://test.wikipedia.org/wiki/Topic:S9rxfaw5bcigtq25:

It looks like this topic was once deleted:

mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(rev_id)), LOWER(HEX(rev_type_id)) as root_post_id, rev_type, rev_change_type, LOWER(HEX(rev_parent_id)) FROM flow_revision WHERE LOWER( HEX( rev_type_id ) ) = '052b82b9c46b055935412d';
+------------------------+------------------------+----------+-----------------+---------------------------+
| LOWER(HEX(rev_id))     | root_post_id           | rev_type | rev_change_type | LOWER(HEX(rev_parent_id)) |
+------------------------+------------------------+----------+-----------------+---------------------------+
| 052b82b9c46f055935412d | 052b82b9c46b055935412d | post     | new-post        | NULL                      |
| 052cdc8c2ad7055935412d | 052b82b9c46b055935412d | post     | delete-topic    | 052b82b9c46f055935412d    |
+------------------------+------------------------+----------+-----------------+---------------------------+
2 rows in set (0.16 sec)

Okay, through more debugging in production I determined that there is a bogus External Store URL.

The max blob_id for testwiki in External Store cluster24 is 66849. However, testwiki has pointers to 73361200.

This may be related to the recovery from T95580: Flow data missing on Wikimedia production wikis.

mysql:wikiadmin@10.64.16.154 [testwiki]> select blob_id from blobs_cluster24 ORDER by blob_id DESC LIMIT 1;
+---------+
| blob_id |
+---------+
|   66849 |
+---------+
1 row in set (0.00 sec)

mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(rev_id)), LOWER(HEX(rev_type_id)), rev_user_wiki, rev_content, rev_flags, rev_type, rev_change_type, LOWER(HEX(rev_parent_id)) FROM flow_revision WHERE LOWER( HEX( rev_type_id ) ) = '052b82b9c46b055935412d' ORDER BY rev_id DESC;
+------------------------+-------------------------+---------------+-------------------------+-----------+----------+-----------------+---------------------------+
| LOWER(HEX(rev_id))     | LOWER(HEX(rev_type_id)) | rev_user_wiki | rev_content             | rev_flags | rev_type | rev_change_type | LOWER(HEX(rev_parent_id)) |
+------------------------+-------------------------+---------------+-------------------------+-----------+----------+-----------------+---------------------------+
| 052cdc8c2ad7055935412d | 052b82b9c46b055935412d  | testwiki      | DB://cluster24/73361200 | external  | post     | delete-topic    | 052b82b9c46f055935412d    |
| 052b82b9c46f055935412d | 052b82b9c46b055935412d  | testwiki      | DB://cluster24/73361200 | external  | post     | new-post        | NULL                      |
+------------------------+-------------------------+---------------+-------------------------+-----------+----------+-----------------+---------------------------+
2 rows in set (0.29 sec)

There are actually 84 references to that row:

73361200 does exist on enwiki, and that's one of the broken rows that caused the original issue.

mysql:wikiadmin@10.64.16.154 [(none)]> SELECT * FROM enwiki.blobs_cluster24 WHERE blob_id = 73361200;
+----------+-----------+
| blob_id  | blob_text |
+----------+-----------+
| 73361200 | NULL      |
+----------+-----------+
1 row in set (0.00 sec)

I still don't know why testwiki would be pointing to that, though. (wikidata, another Flow wiki, also has a 73361200, but it's not NULL and is probably just a coincidence)

In summary, to deal with this there are at least two possibilities:

1. Write a maintenance script to put a content marker here (and anywhere else there is missing content despite recovery) so it's no longer missing. E.g. "Missing data due to https://phabricator.wikimedia.org/T95580" as the content". @EBernhardson suggested this as a possibility.
2. Change the validation somehow so it doesn't fail here.

See also T98935: D12. Run recovery process on test2wiki.

Gory details:

This happens because validation fails, so the topic title is effectively missing when loaded e.g. at https://test.wikipedia.org/wiki/Topic:S9rxfaw5bcigtq25 .

I think we didn't see this before with the original problem, because it was still in cache.

In T98547#1274135, @Mattflaschen wrote:

It looks like this topic was once deleted:

mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(rev_id)), LOWER(HEX(rev_type_id)) as root_post_id, rev_type, rev_change_type, LOWER(HEX(rev_parent_id)) FROM flow_revision WHERE LOWER( HEX( rev_type_id ) ) = '052b82b9c46b055935412d';
+------------------------+------------------------+----------+-----------------+---------------------------+
| LOWER(HEX(rev_id))     | root_post_id           | rev_type | rev_change_type | LOWER(HEX(rev_parent_id)) |
+------------------------+------------------------+----------+-----------------+---------------------------+
| 052b82b9c46f055935412d | 052b82b9c46b055935412d | post     | new-post        | NULL                      |
| 052cdc8c2ad7055935412d | 052b82b9c46b055935412d | post     | delete-topic    | 052b82b9c46f055935412d    |
+------------------------+------------------------+----------+-----------------+---------------------------+
2 rows in set (0.16 sec)

In alpha-decimal form, those two revision IDs are:

> echo Flow\Model\UUID::create( '052b82b9c46f055935412d' )->getAlphadecimal();
s9rxfaw6vgy45wrx
> echo Flow\Model\UUID::create( '052cdc8c2ad7055935412d' )->getAlphadecimal();
satbq50jha0ngqi5

@EBernhardson has been helping me work on this. At his suggestion, I re-ran repair_missing_revision_content.php (note that one just creates a CSV, which was empty in this case), P643. It was unable to recover anything (it was apparently run on testwiki back then).

Note satbq50jha0ngqi5 inherits the parent revision, and s9rxfaw6vgy45wrx is a multiple match (from P643), all of which are tests.

********************

Processing revision s9rxfaw6vgy45wrx
Considering core revisions from 20150112230116 to 20150113090107
Expected length: 7
Checking 13 es urls
Checking 2 es urls
Checking 1 es urls
Checking 4 es urls
Checking 88 es urls
Checking 2 es urls
Checking 12 es urls
Checking 7 es urls
Checking 76 es urls
Checking 13 es urls
Checking 1 es urls
Checking 1 es urls
MULTIPLE POTENTIAL MATCHES:
 DB://cluster24/60698 : Topic 1
 DB://cluster24/60701 : topic 1
 DB://cluster24/60707 : topic 3

I prefer #1 (put a content marker).

@Mattflaschen: Any progress / news here?

(Asking as this has "Unbreak now" priority and no updates for more than two weeks.)

There has been some progress on T98935: D12. Run recovery process on test2wiki, which is related (it's a pre-requisite for the solutions we're discussing here).

Change 217505 had a related patch set uploaded (by Matthias Mullie):
Show stub content where we're missing content instead of failing

https://gerrit.wikimedia.org/r/217505

Change 217505 merged by jenkins-bot:
Show stub content where we're missing content instead of failing

https://gerrit.wikimedia.org/r/217505

• https://gerrit.wikimedia.org/r/217505 (branch master): WMF-deploy-2015-06-16_(1.26wmf10)

Change 218587 had a related patch set uploaded (by Mattflaschen):
Allow invalid content in ContributionsQuery

https://gerrit.wikimedia.org/r/218587

I think the same needs to be done for ContributionsQuery. Done as follow-up.

Change 218587 merged by jenkins-bot:
Allow invalid content in ContributionsQuery

https://gerrit.wikimedia.org/r/218587

It's now failing in getContent. We'll have to test locally with a full External Store setup, and a bogus External Store URL (that's the original issue).

@Mattflaschen: how and where exactly does it fail?

It's intended to fail in AbstractRevision::getContent: it should throw an exception if the content could not be loaded.
Templating::getContent however, should catch that exception (only for format=fixed-html, for rendering) & return the flow-stub-post-content i18n msg instead.

I assume it's failing in some other place that calls AbstractRevision::getContent directly? Where did you see the failure?

https://test.wikipedia.org/wiki/Talk:Sandbox ; see the task description . The issue is probably that plaintext format is used for rendering topic titles.

Also, if we only fix this at fetch-time for rendering, what about API users?

Change 219476 had a related patch set uploaded (by Matthias Mullie):
Output error message instead of failing for more than just html

https://gerrit.wikimedia.org/r/219476

API consumers will also get RevisionFormatter::formatApi output, which should come from Templating::getContent, so they should also get the "couldn't fetch content because of technical issues" error message.

Change 219476 merged by jenkins-bot:
Output error message instead of failing for more than just html

https://gerrit.wikimedia.org/r/219476

In T98547#1384107, @gerritbot wrote:

Change 219476 merged by jenkins-bot:
Output error message instead of failing for more than just html
https://gerrit.wikimedia.org/r/219476

Does this fix the bug?

It should, but this time I want to fix T95870: E10. Have Flow use ExternalStore on MediaWiki-Vagrant and test it locally.

In T98547#1384171, @Catrope wrote:

In T98547#1384107, @gerritbot wrote:

Change 219476 merged by jenkins-bot:
Output error message instead of failing for more than just html
https://gerrit.wikimedia.org/r/219476

Does this fix the bug?

Yes