Page MenuHomePhabricator

D11. Put marker in place of any content lost in data loss incident that could not be recovered
Closed, ResolvedPublic

Description

Visiting https://test.wikipedia.org/wiki/Talk:Sandbox results in:

1An error has occurred.
2
3Return to Main Page.
4
5[304f7bf3] /wiki/Talk:Sandbox Flow\Exception\InvalidDataException from line 349 of /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/Model/AbstractRevision.php: Failed to load the content
6
7Backtrace:
8
9#0 /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/Templating.php(156): Flow\Model\AbstractRevision->getContent(string)
10#1 /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/Formatter/RevisionFormatter.php(241): Flow\Templating->getContent(Flow\Model\PostRevision, string)
11#2 /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/Formatter/TopicListFormatter.php(69): Flow\Formatter\RevisionFormatter->formatApi(Flow\Formatter\TopicRow, Flow\View)
12#3 /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/Formatter/TopicListFormatter.php(41): Flow\Formatter\TopicListFormatter->buildResult(Flow\Model\Workflow, array, array, Flow\View)
13#4 /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/Block/TopicList.php(247): Flow\Formatter\TopicListFormatter->formatApi(Flow\Model\Workflow, array, array, Flow\Data\Pager\PagerPage, Flow\View)
14#5 /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/View.php(180): Flow\Block\TopicListBlock->renderApi(array)
15#6 /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/View.php(71): Flow\View->buildApiResponse(Flow\WorkflowLoader, array, string, array)
16#7 /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/Actions/Action.php(101): Flow\View->show(Flow\WorkflowLoader, string)
17#8 /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/Actions/ViewAction.php(15): Flow\Actions\FlowAction->showForAction(string, OutputPage)
18#9 /srv/mediawiki/php-1.26wmf10/extensions/Flow/includes/Actions/Action.php(46): Flow\Actions\ViewAction->showForAction(string)
19#10 /srv/mediawiki/php-1.26wmf10/includes/MediaWiki.php(413): Flow\Actions\FlowAction->show()
20#11 /srv/mediawiki/php-1.26wmf10/includes/MediaWiki.php(291): MediaWiki->performAction(Article, Title)
21#12 /srv/mediawiki/php-1.26wmf10/includes/MediaWiki.php(634): MediaWiki->performRequest()
22#13 /srv/mediawiki/php-1.26wmf10/includes/MediaWiki.php(431): MediaWiki->main()
23#14 /srv/mediawiki/php-1.26wmf10/index.php(41): MediaWiki->run()
24#15 /srv/mediawiki/w/index.php(3): include(string)
25#16 {main}

Old symptoms:


Visiting https://test.wikipedia.org/wiki/Talk:Sandbox results in:

1An error has occurred.
2
3Return to Main Page.
4
5[5348f422] /wiki/Talk:Sandbox Flow\Exception\InvalidDataException from line 118 of /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Collection/AbstractCollection.php: Revisions for rugqcrsrc16jd7al could not be found
6
7Backtrace:
8
9#0 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Collection/LocalCacheAbstractCollection.php(32): Flow\Collection\AbstractCollection->getAllRevisions()
10#1 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Collection/LocalCacheAbstractCollection.php(120): Flow\Collection\LocalCacheAbstractCollection->getAllRevisions()
11#2 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Collection/CollectionCache.php(44): Flow\Collection\LocalCacheAbstractCollection->getLastRevision()
12#3 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Templating.php(169): Flow\Collection\CollectionCache->getLastRevisionFor(Flow\Model\PostRevision)
13#4 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Formatter/RevisionFormatter.php(180): Flow\Templating->getModeratedRevision(Flow\Model\PostRevision)
14#5 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Formatter/TopicListFormatter.php(69): Flow\Formatter\RevisionFormatter->formatApi(Flow\Formatter\TopicRow, Flow\View)
15#6 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Formatter/TopicListFormatter.php(41): Flow\Formatter\TopicListFormatter->buildResult(Flow\Model\Workflow, array, array, Flow\View)
16#7 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Block/TopicList.php(234): Flow\Formatter\TopicListFormatter->formatApi(Flow\Model\Workflow, array, array, Flow\Data\Pager\PagerPage, Flow\View)
17#8 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/View.php(170): Flow\Block\TopicListBlock->renderApi(array)
18#9 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/View.php(70): Flow\View->buildApiResponse(Flow\WorkflowLoader, array, string, array)
19#10 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Actions/Action.php(101): Flow\View->show(Flow\WorkflowLoader, string)
20#11 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Actions/ViewAction.php(15): Flow\Actions\FlowAction->showForAction(string, OutputPage)
21#12 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Actions/Action.php(46): Flow\Actions\ViewAction->showForAction(string)
22#13 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(413): Flow\Actions\FlowAction->show()
23#14 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(291): MediaWiki->performAction(Article, Title)
24#15 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(583): MediaWiki->performRequest()
25#16 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(432): MediaWiki->main()
26#17 /srv/mediawiki/php-1.26wmf5/index.php(46): MediaWiki->run()
27#18 /srv/mediawiki/w/index.php(3): include(string)
28#19 {main}

Attempting to purge (https://test.wikipedia.org/wiki/Talk:Sandbox?action=purge) results in:

1[e1ec9e2d] /wiki/Talk:Sandbox?action=purge MWException from line 328 of /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Formatter/AbstractQuery.php: Did not load root post s9rxfaw5bcigtq25
2
3Backtrace:
4
5#0 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Formatter/AbstractQuery.php(224): Flow\Formatter\AbstractQuery->getRootPost(Flow\Model\PostRevision)
6#1 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Formatter/TopicListQuery.php(67): Flow\Formatter\AbstractQuery->buildResult(Flow\Model\PostRevision, NULL, Flow\Formatter\TopicRow)
7#2 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Actions/PurgeAction.php(145): Flow\Formatter\TopicListQuery->getResults(array)
8#3 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Actions/PurgeAction.php(124): Flow\Actions\PurgeAction->fetchTopics(array)
9#4 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Actions/PurgeAction.php(63): Flow\Actions\PurgeAction->fetchDiscussion(Flow\Model\Workflow)
10#5 /srv/mediawiki/php-1.26wmf5/includes/actions/PurgeAction.php(74): Flow\Actions\PurgeAction->onSubmit(array)
11#6 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(413): PurgeAction->show()
12#7 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(291): MediaWiki->performAction(Article, Title)
13#8 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(583): MediaWiki->performRequest()
14#9 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(432): MediaWiki->main()
15#10 /srv/mediawiki/php-1.26wmf5/index.php(46): MediaWiki->run()
16#11 /srv/mediawiki/w/index.php(3): include(string)
17#12 {main}

Related Objects

Event Timeline

Mattflaschen-WMF updated the task description. (Show Details)
Mattflaschen-WMF raised the priority of this task from to Unbreak Now!.
Restricted Application added a project: Collaboration-Team-Triage. · View Herald TranscriptMay 7 2015, 11:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Mattflaschen-WMF set Security to None.

It is in an inconsistent state, apparently due to being deleted (2015-02-15T09:36:43Z) then immediately automatically recreated on the next visit (2015-02-15T09:36:56Z), but using the same workflow.

The workflow shown in the 'revisions' request (rlam6pl51ksr5uuw) translates to '050bad020d1d3cafb69678':

mattflaschen@tin:~$ mwscript eval.php --wiki=testwiki
> $workflowId = Flow\Model\UUID::create( 'rlam6pl51ksr5uuw' );

> var_export( $workflowId->getHex() );
'050bad020d1d3cafb69678'

The new page ID is 79845:

> var_export( Title::newFromText( 'Talk:Sandbox' )->getArticleId() );
79845

This workflow (see above) is still associated with the old (wrong) page ID (see first API request, logevents), 69155.

mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(workflow_id)), workflow_wiki, workflow_namespace, workflow_page_id, workflow_title_text, workflow_type FROM flow_workflow WHERE LOWER(HEX(workflow_id)) = '050bad020d1d3cafb69678';
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
| LOWER(HEX(workflow_id)) | workflow_wiki | workflow_namespace | workflow_page_id | workflow_title_text | workflow_type |
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
| 050bad020d1d3cafb69678  | testwiki      |                  1 |            69155 | Sandbox             | discussion    |
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
1 row in set (0.06 sec)

This is the only workflow associated with the page:

mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(workflow_id)), workflow_wiki, workflow_namespace, workflow_page_id, workflow_title_text, workflow_type FROM flow_workflow WHERE workflow_wiki = 'testwiki' AND workflow_namespace = 1 AND workflow_title_text = 'Sandbox' AND workflow_type = 'discussion';
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
| LOWER(HEX(workflow_id)) | workflow_wiki | workflow_namespace | workflow_page_id | workflow_title_text | workflow_type |
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
| 050bad020d1d3cafb69678  | testwiki      |                  1 |            69155 | Sandbox             | discussion    |
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
1 row in set (0.01 sec)

The fix is the same as for T97664: A18. Moving Talk:Search to Talk:Search/Old gives a fatal exception, re-running the new version of T96888: A14. Run maintenance/FlowUpdateWorkflowPageId.php in production. I wanted to make sure that would fix the problem, not confuse matters further. I'm now confident it will. It will cause old topics from before the deletion/re-creation to be re-associated with the board; however, they can still be deleted individually, and this is a test page anyway.

There are old topics that are associated with the

None of this should happen again with Matthias's recent fixes, such as d56426484232753d709a2ddc4f7b5170a8d3b250 and d2f37d2cfeca75fc25ac9687d5b3e93bc8ecf300 .

I ran the script. It's associated with the right page ID, now, but it didn't fix the problem.

I've been debugging through (thanks to @EBernhardson for helping me get that set up). It is a problem with one topic, https://test.wikipedia.org/wiki/Topic:S9rxfaw5bcigtq25:

1Could not find the topic title for current workflow.
2
3Return to Main Page.
4
5[10e8ee2a] /wiki/Topic:S9rxfaw5bcigtq25 Flow\Exception\InvalidDataException from line 786 of /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Block/Topic.php: Every workflow must have an associated topic title
6
7Backtrace:
8
9#0 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Block/Topic.php(471): Flow\Block\TopicBlock->loadTopicTitle()
10#1 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/View.php(170): Flow\Block\TopicBlock->renderApi(array)
11#2 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/View.php(70): Flow\View->buildApiResponse(Flow\WorkflowLoader, array, string, array)
12#3 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Actions/Action.php(101): Flow\View->show(Flow\WorkflowLoader, string)
13#4 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Actions/ViewAction.php(15): Flow\Actions\FlowAction->showForAction(string, OutputPage)
14#5 /srv/mediawiki/php-1.26wmf5/extensions/Flow/includes/Actions/Action.php(46): Flow\Actions\ViewAction->showForAction(string)
15#6 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(413): Flow\Actions\FlowAction->show()
16#7 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(291): MediaWiki->performAction(Article, Title)
17#8 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(583): MediaWiki->performRequest()
18#9 /srv/mediawiki/php-1.26wmf5/includes/MediaWiki.php(432): MediaWiki->main()
19#10 /srv/mediawiki/php-1.26wmf5/index.php(46): MediaWiki->run()
20#11 /srv/mediawiki/w/index.php(3): include(string)
21#12 {main}

It looks like this topic was once deleted:

mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(rev_id)), LOWER(HEX(rev_type_id)) as root_post_id, rev_type, rev_change_type, LOWER(HEX(rev_parent_id)) FROM flow_revision WHERE LOWER( HEX( rev_type_id ) ) = '052b82b9c46b055935412d';
+------------------------+------------------------+----------+-----------------+---------------------------+
| LOWER(HEX(rev_id))     | root_post_id           | rev_type | rev_change_type | LOWER(HEX(rev_parent_id)) |
+------------------------+------------------------+----------+-----------------+---------------------------+
| 052b82b9c46f055935412d | 052b82b9c46b055935412d | post     | new-post        | NULL                      |
| 052cdc8c2ad7055935412d | 052b82b9c46b055935412d | post     | delete-topic    | 052b82b9c46f055935412d    |
+------------------------+------------------------+----------+-----------------+---------------------------+
2 rows in set (0.16 sec)

Okay, through more debugging in production I determined that there is a bogus External Store URL.

The max blob_id for testwiki in External Store cluster24 is 66849. However, testwiki has pointers to 73361200.

This may be related to the recovery from T95580: Flow data missing on Wikimedia production wikis.

mysql:wikiadmin@10.64.16.154 [testwiki]> select blob_id from blobs_cluster24 ORDER by blob_id DESC LIMIT 1;
+---------+
| blob_id |
+---------+
|   66849 |
+---------+
1 row in set (0.00 sec)
mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(rev_id)), LOWER(HEX(rev_type_id)), rev_user_wiki, rev_content, rev_flags, rev_type, rev_change_type, LOWER(HEX(rev_parent_id)) FROM flow_revision WHERE LOWER( HEX( rev_type_id ) ) = '052b82b9c46b055935412d' ORDER BY rev_id DESC;
+------------------------+-------------------------+---------------+-------------------------+-----------+----------+-----------------+---------------------------+
| LOWER(HEX(rev_id))     | LOWER(HEX(rev_type_id)) | rev_user_wiki | rev_content             | rev_flags | rev_type | rev_change_type | LOWER(HEX(rev_parent_id)) |
+------------------------+-------------------------+---------------+-------------------------+-----------+----------+-----------------+---------------------------+
| 052cdc8c2ad7055935412d | 052b82b9c46b055935412d  | testwiki      | DB://cluster24/73361200 | external  | post     | delete-topic    | 052b82b9c46f055935412d    |
| 052b82b9c46f055935412d | 052b82b9c46b055935412d  | testwiki      | DB://cluster24/73361200 | external  | post     | new-post        | NULL                      |
+------------------------+-------------------------+---------------+-------------------------+-----------+----------+-----------------+---------------------------+
2 rows in set (0.29 sec)

There are actually 84 references to that row:

1mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(rev_id)), LOWER(HEX(rev_type_id)), rev_user_wiki, rev_content, rev_flags, rev_type, rev_change_type, LOWER(HEX(rev_parent_id)) FROM flow_revision WHERE rev_user_wiki = 'testwiki' AND rev_content LIKE 'DB://cluster24/______%' LIMIT 100;
2+------------------------+-------------------------+---------------+-------------------------+-----------+----------+-----------------+---------------------------+
3| LOWER(HEX(rev_id)) | LOWER(HEX(rev_type_id)) | rev_user_wiki | rev_content | rev_flags | rev_type | rev_change_type | LOWER(HEX(rev_parent_id)) |
4+------------------------+-------------------------+---------------+-------------------------+-----------+----------+-----------------+---------------------------+
5| 050beaf9a1ad3cafb69678 | 050beaf87d453cafb69678 | testwiki | DB://cluster24/73361200 | external | post | flow-edit-post | 050beaf87d453cafb69678 |
6| 050c20d745f53cafb69678 | 050b9d862c3d3cafb69678 | testwiki | DB://cluster24/73361200 | external | post | restore-topic | 050c20d42af53cafb69678 |
7| 050c5ed2ab713cafb69678 | 050c5ed2ab693cafb69678 | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
8| 050c5ed2ab753cafb69678 | 050c5ed2ab753cafb69678 | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
9| 050c5edc23393cafb69678 | 050c5edc23393cafb69678 | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
10| 050c5ee898a53cafb69678 | 050c5ee898a53cafb69678 | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
11| 051799ac2cdd665935412d | 051799ac2cdd665935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
12| 051bd232cfcd665935412d | 051bd232cfc9665935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
13| 051bec868e75665935412d | 051bec868e71665935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
14| 051bec868e79665935412d | 051bec868e79665935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
15| 05218f099a93675935412d | 05218f0999f7675935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
16| 05218f116cab675935412d | 05218f099ae3675935412d | testwiki | DB://cluster24/73361200 | external | post | restore-post | 05218f0ff553675935412d |
17| 05297e66bc3b055935412d | 05297e66bc37055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
18| 05297e66bc3b065935412d | 05297e66bc3b065935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
19| 05297e68423b055935412d | 05297e684237055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
20| 05297e68423f055935412d | 05297e68423f055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
21| 05297e7d7d2b055935412d | 05297e7d7d23055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
22| 05297e7d7d2b065935412d | 05297e7d7d2b065935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
23| 05297f54f6a3055935412d | 05297f54f6a3055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
24| 052b828fc3ab055935412d | 052b828fc2e7055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
25| 052b828fc41b055935412d | 052b828fc41b055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
26| 052b828fe077055935412d | 052b828fe073055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
27| 052b828fe07b055935412d | 052b828fe07b055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
28| 052b828feacf055935412d | 052b828feacb055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
29| 052b828fead3055935412d | 052b828fead3055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
30| 052b828ff77f055935412d | 052b828ff77b055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
31| 052b828ff783055935412d | 052b828ff783055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
32| 052b829002c3055935412d | 052b829002bf055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
33| 052b829002c7055935412d | 052b829002c7055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
34| 052b82900def055935412d | 052b82900deb055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
35| 052b82900df3055935412d | 052b82900df3055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
36| 052b82979037055935412d | 052b82979033055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
37| 052b82979037065935412d | 052b82979037065935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
38| 052b82979da3055935412d | 052b82979d9f055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
39| 052b82979da7055935412d | 052b82979da7055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
40| 052b8297a78f055935412d | 052b8297a78b055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
41| 052b8297a793055935412d | 052b8297a793055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
42| 052b8297b147055935412d | 052b8297b143055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
43| 052b8297b14b055935412d | 052b8297b14b055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
44| 052b8297bd7f055935412d | 052b8297bd7b055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
45| 052b8297bd83055935412d | 052b8297bd83055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
46| 052b8297ce5f055935412d | 052b8297ce5b055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
47| 052b8297ce5f065935412d | 052b8297ce5f065935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
48| 052b82b9c46f055935412d | 052b82b9c46b055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
49| 052b82bb35c7055935412d | 052b82bb35c3055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
50| 052b82bb4d37055935412d | 052b82bb4d33055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
51| 052b82bb57b7055935412d | 052b82bb57b3055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
52| 052b82bb628b055935412d | 052b82bb6287055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
53| 052b82bb7393055935412d | 052b82bb738f055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
54| 052b82bb7c1f055935412d | 052b82bb7c17055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
55| 052cb602993f055935412d | 052cb602993b055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
56| 052cb602993f065935412d | 052cb602993f065935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
57| 052cb6a3e37b055935412d | 052cb6a3e377055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
58| 052cb6a3e37f055935412d | 052cb6a3e37f055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
59| 052cca73a3f3055935412d | 052cca73a3ef055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
60| 052cca73a3f7055935412d | 052cca73a3f7055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
61| 052cca73bc2b055935412d | 052cca73bc27055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
62| 052cca73bc2b065935412d | 052cca73bc2b065935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
63| 052cca73cb37055935412d | 052cca73cb33055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
64| 052cca73cb3b055935412d | 052cca73cb3b055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
65| 052cca85f7a7055935412d | 052cca85f7a3055935412d | testwiki | DB://cluster24/73361200 | external | post | new-post | NULL |
66| 052cca85f7ab055935412d | 052cca85f7ab055935412d | testwiki | DB://cluster24/73361200 | external | post | reply | NULL |
67| 052cdc87bf37055935412d | 052b828fc2e7055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b828fc3ab055935412d |
68| 052cdc8849a7055935412d | 052b828fe073055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b828fe077055935412d |
69| 052cdc88cb7b055935412d | 052b828feacb055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b828feacf055935412d |
70| 052cdc892c37055935412d | 052b828ff77b055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b828ff77f055935412d |
71| 052cdc8989b7055935412d | 052b829002bf055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b829002c3055935412d |
72| 052cdc89d997055935412d | 052b82900deb055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b82900def055935412d |
73| 052cdc8a2d0f055935412d | 052b82979033055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b82979037055935412d |
74| 052cdc8a775f055935412d | 052b82979d9f055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b82979da3055935412d |
75| 052cdc8ae0cf055935412d | 052b8297a78b055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b8297a78f055935412d |
76| 052cdc8b2183055935412d | 052b8297b143055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b8297b147055935412d |
77| 052cdc8b8da7055935412d | 052b8297bd7b055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b8297bd7f055935412d |
78| 052cdc8bdacb055935412d | 052b8297ce5b055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b8297ce5f055935412d |
79| 052cdc8c2ad7055935412d | 052b82b9c46b055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b82b9c46f055935412d |
80| 052cdc8c8c97055935412d | 052b82bb35c3055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b82bb35c7055935412d |
81| 052cdc8cf33b055935412d | 052b82bb4d33055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b82bb4d37055935412d |
82| 052cdc8d357b055935412d | 052b82bb57b3055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b82bb57b7055935412d |
83| 052cdc8d8c07055935412d | 052b82bb6287055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b82bb628b055935412d |
84| 052cdc8dcafb055935412d | 052b82bb738f055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b82bb7393055935412d |
85| 052cdc8e212b055935412d | 052b82bb7c17055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052b82bb7c1f055935412d |
86| 052cdcb5edef055935412d | 052cca73cb33055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-topic | 052cca73cb37055935412d |
87| 052cdcb70913055935412d | 052cca73cb3b055935412d | testwiki | DB://cluster24/73361200 | external | post | delete-post | 052cca73cb3b055935412d |
88| 052e326a4bcf055935412d | 050bad020d1d3cafb69678 | testwiki | DB://cluster24/73361200 | external | header | edit-header | 050ec6af462d3cafb69678 |
89+------------------------+-------------------------+---------------+-------------------------+-----------+----------+-----------------+---------------------------+
9084 rows in set (0.22 sec)

73361200 does exist on enwiki, and that's one of the broken rows that caused the original issue.

mysql:wikiadmin@10.64.16.154 [(none)]> SELECT * FROM enwiki.blobs_cluster24 WHERE blob_id = 73361200;
+----------+-----------+
| blob_id  | blob_text |
+----------+-----------+
| 73361200 | NULL      |
+----------+-----------+
1 row in set (0.00 sec)

I still don't know why testwiki would be pointing to that, though. (wikidata, another Flow wiki, also has a 73361200, but it's not NULL and is probably just a coincidence)

In summary, to deal with this there are at least two possibilities:

  1. Write a maintenance script to put a content marker here (and anywhere else there is missing content despite recovery) so it's no longer missing. E.g. "Missing data due to https://phabricator.wikimedia.org/T95580" as the content". @EBernhardson suggested this as a possibility.
  2. Change the validation somehow so it doesn't fail here.

See also T98935: D12. Run recovery process on test2wiki.


Gory details:

This happens because validation fails, so the topic title is effectively missing when loaded e.g. at https://test.wikipedia.org/wiki/Topic:S9rxfaw5bcigtq25 .

I think we didn't see this before with the original problem, because it was still in cache.

In T98547#1274135, @Mattflaschen wrote:

It looks like this topic was once deleted:

mysql:research@x1-analytics-slave [flowdb]> SELECT LOWER(HEX(rev_id)), LOWER(HEX(rev_type_id)) as root_post_id, rev_type, rev_change_type, LOWER(HEX(rev_parent_id)) FROM flow_revision WHERE LOWER( HEX( rev_type_id ) ) = '052b82b9c46b055935412d';
+------------------------+------------------------+----------+-----------------+---------------------------+
| LOWER(HEX(rev_id))     | root_post_id           | rev_type | rev_change_type | LOWER(HEX(rev_parent_id)) |
+------------------------+------------------------+----------+-----------------+---------------------------+
| 052b82b9c46f055935412d | 052b82b9c46b055935412d | post     | new-post        | NULL                      |
| 052cdc8c2ad7055935412d | 052b82b9c46b055935412d | post     | delete-topic    | 052b82b9c46f055935412d    |
+------------------------+------------------------+----------+-----------------+---------------------------+
2 rows in set (0.16 sec)

In alpha-decimal form, those two revision IDs are:

> echo Flow\Model\UUID::create( '052b82b9c46f055935412d' )->getAlphadecimal();
s9rxfaw6vgy45wrx
> echo Flow\Model\UUID::create( '052cdc8c2ad7055935412d' )->getAlphadecimal();
satbq50jha0ngqi5

@EBernhardson has been helping me work on this. At his suggestion, I re-ran repair_missing_revision_content.php (note that one just creates a CSV, which was empty in this case), P643. It was unable to recover anything (it was apparently run on testwiki back then).

Note satbq50jha0ngqi5 inherits the parent revision, and s9rxfaw6vgy45wrx is a multiple match (from P643), all of which are tests.

********************

Processing revision s9rxfaw6vgy45wrx
Considering core revisions from 20150112230116 to 20150113090107
Expected length: 7
Checking 13 es urls
Checking 2 es urls
Checking 1 es urls
Checking 4 es urls
Checking 88 es urls
Checking 2 es urls
Checking 12 es urls
Checking 7 es urls
Checking 76 es urls
Checking 13 es urls
Checking 1 es urls
Checking 1 es urls
MULTIPLE POTENTIAL MATCHES:
 DB://cluster24/60698 : Topic 1
 DB://cluster24/60701 : topic 1
 DB://cluster24/60707 : topic 3

I prefer #1 (put a content marker).

@Mattflaschen: Any progress / news here?

(Asking as this has "Unbreak now" priority and no updates for more than two weeks.)

Mattflaschen-WMF lowered the priority of this task from Unbreak Now! to High.Jun 3 2015, 3:18 AM

Perhaps High is more accurate, since it only affects a test wiki.

There has been some progress on T98935: D12. Run recovery process on test2wiki, which is related (it's a pre-requisite for the solutions we're discussing here).

Mattflaschen-WMF renamed this task from Data problem on https://test.wikipedia.org/wiki/Talk:Sandbox to Put marker in place of any content lost in data loss incident that could not be recovered.Jun 4 2015, 6:17 PM
Mattflaschen-WMF removed Mattflaschen-WMF as the assignee of this task.

Change 217505 had a related patch set uploaded (by Matthias Mullie):
Show stub content where we're missing content instead of failing

https://gerrit.wikimedia.org/r/217505

Change 217505 merged by jenkins-bot:
Show stub content where we're missing content instead of failing

https://gerrit.wikimedia.org/r/217505

Change 218587 had a related patch set uploaded (by Mattflaschen):
Allow invalid content in ContributionsQuery

https://gerrit.wikimedia.org/r/218587

I think the same needs to be done for ContributionsQuery. Done as follow-up.

Change 218587 merged by jenkins-bot:
Allow invalid content in ContributionsQuery

https://gerrit.wikimedia.org/r/218587

DannyH renamed this task from Put marker in place of any content lost in data loss incident that could not be recovered to D11. Put marker in place of any content lost in data loss incident that could not be recovered.Jun 16 2015, 7:25 PM

It's now failing in getContent. We'll have to test locally with a full External Store setup, and a bogus External Store URL (that's the original issue).

@Mattflaschen: how and where exactly does it fail?

It's intended to fail in AbstractRevision::getContent: it should throw an exception if the content could not be loaded.
Templating::getContent however, should catch that exception (only for format=fixed-html, for rendering) & return the flow-stub-post-content i18n msg instead.

I assume it's failing in some other place that calls AbstractRevision::getContent directly? Where did you see the failure?

https://test.wikipedia.org/wiki/Talk:Sandbox ; see the task description . The issue is probably that plaintext format is used for rendering topic titles.

Also, if we only fix this at fetch-time for rendering, what about API users?

Change 219476 had a related patch set uploaded (by Matthias Mullie):
Output error message instead of failing for more than just html

https://gerrit.wikimedia.org/r/219476

API consumers will also get RevisionFormatter::formatApi output, which should come from Templating::getContent, so they should also get the "couldn't fetch content because of technical issues" error message.

Change 219476 merged by jenkins-bot:
Output error message instead of failing for more than just html

https://gerrit.wikimedia.org/r/219476

Change 219476 merged by jenkins-bot:
Output error message instead of failing for more than just html

https://gerrit.wikimedia.org/r/219476

Does this fix the bug?

Change 219476 merged by jenkins-bot:
Output error message instead of failing for more than just html

https://gerrit.wikimedia.org/r/219476

Does this fix the bug?

Yes

Catrope closed this task as Resolved.Jun 25 2015, 5:36 PM