mysql:research@s3-analytics-slave [flowdb]> SELECT LOWER(HEX(workflow_id)), workflow_wiki, workflow_namespace, workflow_page_id, workflow_title_text, workflow_type FROM flow_workflow WHERE workflow_id = UNHEX('050b9046b262b234bf3684');
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
| LOWER(HEX(workflow_id)) | workflow_wiki | workflow_namespace | workflow_page_id | workflow_title_text | workflow_type |
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
| 050b9046b262b234bf3684  | mediawikiwiki |                  1 |           438062 | Sandbox             | discussion    |
+-------------------------+---------------+--------------------+------------------+---------------------+---------------+
1 row in set (0.02 sec)

mysql:research@s3-analytics-slave [mediawikiwiki]> SELECT ar_id, ar_timestamp ar_namespace, ar_title, ar_rev_id, ar_page_id, ar_content_model, ar_content_format FROM archive WHERE ar_namespace = 1 AND ar_title = 'Sandbox' ORDER by ar_id DESC LIMIT 10;
+--------+----------------+----------+-----------+------------+------------------+-------------------+
| ar_id  | ar_namespace   | ar_title | ar_rev_id | ar_page_id | ar_content_model | ar_content_format |
+--------+----------------+----------+-----------+------------+------------------+-------------------+
| 222048 | 20150424135738 | Sandbox  |   1623104 |     581614 | NULL             | NULL              |
| 179353 | 20140911184629 | Sandbox  |   1149598 |     261786 | NULL             | NULL              |
| 179352 | 20140911184625 | Sandbox  |   1149597 |     261786 | NULL             | NULL              |
| 179351 | 20140911184210 | Sandbox  |   1149591 |     261786 | NULL             | NULL              |
| 179350 | 20140911184208 | Sandbox  |   1149590 |     261786 | NULL             | NULL              |
| 179349 | 20140826153557 | Sandbox  |   1121681 |     261786 | NULL             | NULL              |
| 150117 | 20140826153415 | Sandbox  |   1121679 |     261784 | NULL             | NULL              |
| 150112 | 20140708194825 | Sandbox  |   1061647 |     186417 | NULL             | NULL              |
| 150111 | 20140708180045 | Sandbox  |   1061600 |     186417 | NULL             | NULL              |
| 150110 | 20140703191752 | Sandbox  |   1057636 |     186417 | NULL             | NULL              |
+--------+----------------+----------+-----------+------------+------------------+-------------------+
10 rows in set (0.00 sec)

workflow_page_id (or maybe ar_page_id) still isn't always being updated properly.

Change 289221 had a related patch set uploaded (by Mattflaschen):
Title->getContentModel: Get new content model with GAID_FOR_UPDATE

https://gerrit.wikimedia.org/r/289221

Change 289239 had a related patch set uploaded (by Mattflaschen):
WIP: Use ArticleRevisionUndeleted instead of ArticleUndelete

https://gerrit.wikimedia.org/r/289239

Ran:

UPDATE flow_workflow SET workflow_page_id = 581614 WHERE workflow_wiki = 'mediawikiwiki' AND workflow_page_id=438062

to workaround in production. Hopefully this patch will fix this going forward.

Mentioned in SAL [2016-05-17T18:37:05Z] <matt_flaschen> Ran manual SQL write in production to work around T122262; see task for query.

Change 289221 merged by jenkins-bot:
Title->getContentModel: Get new content model with GAID_FOR_UPDATE

https://gerrit.wikimedia.org/r/289221

This is ready for review now. I tested the test plans listed at T122261, plus some other stuff I thought of along the way.

The only weirdness I found was that the core history page is sometimes cached somehow, so it doesn't reflect your latest undeletion. Hard refresh works around it. I don't think this is Flow-specific, but I don't know for sure.

As part of this, I also found that MySQL auto_increment primary keys are not as forever-unique as a thought. Apparently, it is possible to end up with primary key duplicates (not in the table at the same time, but giving out an auto_increment ID that was previously given out), but it is apparently only with scenarios that are unlikely in production, like restarting the database server right after deleting the topmost pages. While running, InnoDB keeps track of the latest auto_increment primary key *ever* given out by the table, but it doesn't keep this on disk. When you start it up, it just checks the *current* highest page Id.

AFAICT, the MySQL thing led to some slight weirdness (e.g. RevisionDelete once offering to change visibility on two archived revisions with the same ar_rev_id), but as stated, I think it is very unlikely to occur in production.

In T122262#2310857, @Mattflaschen-WMF wrote:

The only weirdness I found was that the core history page is sometimes cached somehow, so it doesn't reflect your latest undeletion. Hard refresh works around it. I don't think this is Flow-specific, but I don't know for sure.

That suggests that there might be some Last-Modified / 304 stuff going on. If core is in fact computing the Last-Modified value correctly (I would be impressed if that was the case), it might not be hookable or we might not be hooking into its computation.

As part of this, I also found that MySQL auto_increment primary keys are not as forever-unique as a thought. Apparently, it is possible to end up with primary key duplicates (not in the table at the same time, but giving out an auto_increment ID that was previously given out), but it is apparently only with scenarios that are unlikely in production, like restarting the database server right after deleting the topmost pages. While running, InnoDB keeps track of the latest auto_increment primary key *ever* given out by the table, but it doesn't keep this on disk. When you start it up, it just checks the *current* highest page Id.

AFAICT, the MySQL thing led to some slight weirdness (e.g. RevisionDelete once offering to change visibility on two archived revisions with the same ar_rev_id), but as stated, I think it is very unlikely to occur in production.

Still a bit worrying though. But you say that this is a bug in core too? What would happen if you tried to undelete both of those revisions?

Change 289239 merged by jenkins-bot:
Use ArticleRevisionUndeleted instead of ArticleUndelete

https://gerrit.wikimedia.org/r/289239

In T122262#2311390, @Catrope wrote:

Still a bit worrying though. But you say that this is a bug in core too?

Yes. It's not specific to Flow. Filed as T135851: Preserve InnoDB table auto_increment on restart

What would happen if you tried to undelete both of those revisions?

Good question. I first just saw this on RevisionDelete, but I just tested (with two separate regular pages, like T135851), and it crashes hard.

It shows only "Error undeleting page" (undelete-error) (nothing else, not an error message or an exception code), and then it's gone from archive *and* not in revision!

Filed as T135852: Failed restore and data loss on ar_rev_id collisions, will flesh out later.

I reviewed test scenarios in T122261: Come up with scenarios for deletion/undeletion of Flow boards and figure out how to handle them and previously fixed T104591: Deleted wikitext Talk page cannot be restored after EnableFlow creates the same page. It seems that we need clearer specs on which specific use cases should be addressed.

Two scenarios were tested:
First scenario: Flow board is deleted --> wikitext with the same title was created --> restore deleted Flow board
Second scenario: wikitext page is deleted --> Flow board with the same name is created --> restore the deleted wikitext page

Attempting to restore the deleted Flow board in betalabs - the familiar error is displayed.
An error has occurred.
Return to Main Page
[V1XBXApEEaoAAGt9rt4AAAAJ] /w/index.php?title=Special:Undelete&action=submit Flow\Exception\FlowException from line 85 of /srv/mediawiki/php-master/extensions/Flow/includes/BoardMover.php: Could not locate workflow for 142438
Backtrace:
#0 /srv/mediawiki/php-master/extensions/Flow/Hooks.php(1617): Flow\BoardMover->move(integer, Title)
#1 /srv/mediawiki/php-master/includes/Hooks.php(195): FlowHooks::onArticleRevisionUndeleted(Title, Revision, string)
#2 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(621): Hooks::run(string, array)
#3 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(391): PageArchive->undeleteRevisions(array, boolean, string)
#4 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(1716): PageArchive->undelete(array, string, array, boolean, User)
#5 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(845): SpecialUndelete->undelete()
#6 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(479): SpecialUndelete->execute(NULL)
#7 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(591): SpecialPage->run(NULL)
#8 /srv/mediawiki/php-master/includes/MediaWiki.php(282): SpecialPageFactory::executePath(Title, RequestContext)
#9 /srv/mediawiki/php-master/includes/MediaWiki.php(748): MediaWiki->performRequest()
#10 /srv/mediawiki/php-master/includes/MediaWiki.php(520): MediaWiki->main()
#11 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#12 /srv/mediawiki/w/index.php(3): include(string)
#13 {main}

2. It's the same scenario as in T104591: Deleted wikitext Talk page cannot be restored after EnableFlow creates the same page : wikitext page is deleted --> create a Flow board with the same title --> restore the deleted wikitext page. The result: no errors, but the deleted wikitext page is not restored - the Flow board with the same title is displayed.

To summarize my previous comment - the scenarios I tested are the following from T122261: Come up with scenarios for deletion/undeletion of Flow boards and figure out how to handle them

Fail to restore wikitext page where Flow page is now

• only Flow page is displayed not the restored wikitext page.

Fail to restore Flow page where Flow page is now

• the error displayed

In T122262#2358599, @Etonkovidova wrote:

I reviewed test scenarios in T122261: Come up with scenarios for deletion/undeletion of Flow boards and figure out how to handle them and previously fixed T104591: Deleted wikitext Talk page cannot be restored after EnableFlow creates the same page. It seems that we need clearer specs on which specific use cases should be addressed.

All of the ones I listed there, at least. Thanks for catching this exception. We will have to send this back to dev. Maybe a regression from T135883: Better transactionality for Flow undeletion.

2. It's the same scenario as in T104591: Deleted wikitext Talk page cannot be restored after EnableFlow creates the same page : wikitext page is deleted --> create a Flow board with the same title --> restore the deleted wikitext page. The result: no errors, but the deleted wikitext page is not restored - the Flow board with the same title is displayed. It's the same as:

I would have to check http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&prop=revisions&titles=TITLEHERE&rvlimit=10 , but I think this is by design. The wikitext revision is older, so the Flow one is supposed to show.

For non-Flow:

1. Create a page with 'abc'.
2. Delete it.
3. Recreate the page with 'def'.
4. Undelete the original page. 'def' still shows, but the old revision is in the history.

Same for Flow, except that for Flow, the core-level history is not visible except through http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&prop=revisions&titles=TITLEHERE&rvlimit=10 .

In T122262#2358599, @Etonkovidova wrote:

Two scenarios were tested: 1) Flow board is deleted --> wikitext with the same title was created --> restore deleted Flow board

Can not reproduce. I did:

1. EnableFlow
2. Delete Flow
3. Create wikitext
4. Restore deleted Flow

Please give exact steps.

In T122262#2358819, @Etonkovidova wrote:

Fail to restore Flow page where Flow page is now

• the error displayed

This is different from what you said above. I tried:

1. EnableFlow with text 'Flow 1'
2. Delete Flow
3. EnableFlow on same page with text 'Flow 2'
4. Restore previous Flow

I also don't get an error.

You definitely found something, but I need the steps to reproduce. Maybe it depends on topics, or something else.

Yes, it turned out trickier than I thought - it seems that the # of revision matters.
Does not happen on empty Flow board
Does not happen when a Flow page was created and a topic(s) were just added.

You may try the last steps below with my example with Talk:ET10

Step-by-step description:

1. Go to an existing Flow board - should not be extensive in length; I used Talk:ET10 - there are 5 revisions.
2. Delete the Flow page
3. Create a wikitext with the same title
4. Go to Special->View deleted pages (under Page tools)
5. Special:Undelete will be displayed - perform search for the deleted Flow page. Following my example with Talk:ET10, you'll see

The following page has been deleted but is still in the archive and can be restored. The archive may be periodically cleaned out.

    Talk:ET10 (5 revisions deleted)

6. Clicking on the Talk:ET10, you'll be redirected to Special:Undelete&target=Talk%3AET10 page.
7. Under 'Restore revision' section, click on Restore. Do not select any specific revisions to restore in Page history section.

The error will be displayed:
[V1hBswpEEaoAAE7RX-UAAAAD] /w/index.php?title=Special:Undelete&action=submit Flow\Exception\FlowException from line 85 of /srv/mediawiki/php-master/extensions/Flow/includes/BoardMover.php: Could not locate workflow for 131861

Backtrace:

#0 /srv/mediawiki/php-master/extensions/Flow/Hooks.php(1617): Flow\BoardMover->move(integer, Title)
#1 /srv/mediawiki/php-master/includes/Hooks.php(195): FlowHooks::onArticleRevisionUndeleted(Title, Revision, string)
#2 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(621): Hooks::run(string, array)
#3 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(391): PageArchive->undeleteRevisions(array, boolean, string)
#4 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(1716): PageArchive->undelete(array, string, array, boolean, User)
#5 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(845): SpecialUndelete->undelete()
#6 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(479): SpecialUndelete->execute(NULL)
#7 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(591): SpecialPage->run(NULL)
#8 /srv/mediawiki/php-master/includes/MediaWiki.php(282): SpecialPageFactory::executePath(Title, RequestContext)
#9 /srv/mediawiki/php-master/includes/MediaWiki.php(748): MediaWiki->performRequest()
#10 /srv/mediawiki/php-master/includes/MediaWiki.php(520): MediaWiki->main()
#11 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#12 /srv/mediawiki/w/index.php(3): include(string)
#13 {main}

Also got the following error when attempting to view diff for deleted Mavetuna5813.

[V1g86wpEEaoAAElIcioAAAAJ] /w/index.php?title=Special:Undelete&target=Mavetuna5813&diff=prev&timestamp=20160606183845 MWException from line 811 of /srv/mediawiki/php-master/includes/diff/DifferenceEngine.php: Diff not implemented for Flow\Content\BoardContent; override generateContentDiffBody to fix this.

Backtrace:

#0 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(1135): DifferenceEngine->generateContentDiffBody(WikitextContent, Flow\Content\BoardContent)
#1 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(1001): SpecialUndelete->showDiff(Revision, Revision)
#2 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(826): SpecialUndelete->showRevision(string)
#3 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(479): SpecialUndelete->execute(NULL)
#4 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(591): SpecialPage->run(NULL)
#5 /srv/mediawiki/php-master/includes/MediaWiki.php(282): SpecialPageFactory::executePath(Title, RequestContext)
#6 /srv/mediawiki/php-master/includes/MediaWiki.php(748): MediaWiki->performRequest()
#7 /srv/mediawiki/php-master/includes/MediaWiki.php(520): MediaWiki->main()
#8 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#9 /srv/mediawiki/w/index.php(3): include(string)
#10 {main}

In T122262#2364847, @Etonkovidova wrote:

1. Go to an existing Flow board - should not be extensive in length; I used Talk:ET10 - there are 5 revisions.

This was originally created back in July 2015 (see bottom of http://en.wikipedia.beta.wmflabs.org/wiki/Special:Undelete/Talk:ET10). Also, it was deleted and restored in September 2015. 2 Flow revisions were restored (both the 2015 revisions are Flow), and back then we only updated the latest revision.

So it's already in an inconsistent state.

I guess the question is whether we want to try to solve these old inconsistent cases programmatically.

Based on my testing, I don't think such a case can be created on a fresh board created now, but if I'm wrong, we should definitely reopen this.

CC @Catrope @matthiasmullie .

The diff one is T110679: MediaWiki error for deleted Flow board, when clicking 'Show changes'.

In T122262#2365382, @Mattflaschen-WMF wrote:

I guess the question is whether we want to try to solve these old inconsistent cases programmatically.

That depends: is it hard to do that? Do we think many of these inconsistent cases exist in the wild?

In T122262#2367305, @Catrope wrote:

That depends: is it hard to do that?

The first thing I thought of is checking the workflow ID of all Flow revision (meaning the core revision table) or archive (also core) rows (which I think BoardContent knows from the JSON), and setting the workflow_page_id to rev_page or ar_page_id if it's wrong.

That would be slow, but maybe still runnable (it can be batched).

I started thinking of alternatives to that, but need to continue.

Do we think many of these inconsistent cases exist in the wild?

Don't know. Probably "not that many" real ones. It wouldn't even happen for every deletion, or every undeletion. It depends exactly the history of the page, and what happened when.

In T122262#2373268, @Mattflaschen-WMF wrote:

In T122262#2367305, @Catrope wrote:

That depends: is it hard to do that?

The first thing I thought of is checking the workflow ID of all Flow revision (meaning the core revision table) or archive (also core) rows (which I think BoardContent knows from the JSON), and setting the workflow_page_id to rev_page or ar_page_id if it's wrong.

That would be slow, but maybe still runnable (it can be batched).

OK, so that would be a maintenance script then, not an on-the-fly correction.

Do we think many of these inconsistent cases exist in the wild?

Don't know. Probably "not that many" real ones. It wouldn't even happen for every deletion, or every undeletion. It depends exactly the history of the page, and what happened when.

In that case, maybe it's not worth fixing for now, unless we have evidence that it's a significant problem?

Per my discussion with @Catrope today, moving this into QA. If we decide to later, we can write the maintenance script as a separate task.

We should reopen now if it can be triggered on a new board (created in June 2016 or later).

Re-checked straightforward scenarios in betalabs
(1) Delete an empty Flow board - Restore it
(2) Delete a new non-empty Flow board with one revision - Restore it
(3) Delete Flow board with some substantial content - Restore it
(4) Delete wikitext page - Restore it

All seem to be correctly restored.

The cases where Flow board was deleted and, then a wikitext page with the same title was created, and then the Flow board was attempted to be restored (and other similar cases) can be addressed in further tasks.

In T122262#2538474, @Etonkovidova wrote:

The cases where Flow board was deleted and, then a wikitext page with the same title was created, and then the Flow board was attempted to be restored (and other similar cases) can be addressed in further tasks.

That still shouldn't cause an exception, if everything is being done now. IIRC, the Flow board will get restored behind the wikitext page, though.

In T122262#2538488, @Mattflaschen-WMF wrote:

That still shouldn't cause an exception, if everything is being done now. IIRC, the Flow board will get restored behind the wikitext page, though.

I tested this. It works as I recalled it would, without an exception.

Re-checked the scenario @Mattflaschen-WMF described:

1. Search db for deleted Flow boards - a simplified query since I knew what I was searching.

mysql> select ar_namespace, ar_user, ar_page_id, ar_rev_id, ar_title from 
archive where ar_title like 'ET14%';
+--------------+---------+------------+-----------+----------+
| ar_namespace | ar_user | ar_page_id | ar_rev_id | ar_title |
+--------------+---------+------------+-----------+----------+
|            1 |    4841 |     131866 |    253692 | ET14     |
|            1 |    4841 |     131866 |    253693 | ET14     |
+--------------+---------+------------+-----------+----------+
2 rows in set (0.01 sec)

2. Check if the page is in Special:Undelete - it is there.
   
3. Create a wikitext page with the title Talk:ET14 and check what page table:

mysql> select page_id,page_namespace,page_content_model,  page_title, 
page_touched from page where page_title like 'ET14%';
+---------+----------------+--------------------+---------------------+----------------+
| page_id | page_namespace | page_content_model | page_title          | page_touched   |
+---------+----------------+--------------------+---------------------+----------------+
|  129599 |              3 | flow-board         | ET14/Flow_Archive_1 | 20160220221857 |
|  133558 |              2 | wikitext           | ET14                | 20160804210833 |
|  165637 |              3 | wikitext           | ET14                | 20160220222607 |
|  182131 |              1 | wikitext           | ET14                | 20160809223604 |
+---------+----------------+--------------------+---------------------+----------------+
4 rows in set (0.09 sec)

The last row indicates that the page Talk:ET14 was just created.

4. Go back to Special:Undelete for Talk:ET14 page (https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Special:Undelete&target=Talk%3AET14) ad under 'Restore revisions', click on 'Restore'

Error
An error has occurred.

Return to Main Page

[V6pbGQpEEaoAAA28hC4AAAAR] /w/index.php?title=Special:Undelete&action=submit Flow\Exception\FlowException from line 81 of /srv/mediawiki/php-master/extensions/Flow/includes/BoardMover.php: Could not locate workflow for 131866

Backtrace:

#0 /srv/mediawiki/php-master/extensions/Flow/Hooks.php(1606): Flow\BoardMover->move(integer, Title)
#1 /srv/mediawiki/php-master/includes/Hooks.php(195): FlowHooks::onArticleRevisionUndeleted(Title, Revision, string)
#2 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(621): Hooks::run(string, array)
#3 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(391): PageArchive->undeleteRevisions(array, boolean, string)
#4 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(1716): PageArchive->undelete(array, string, array, boolean, User)
#5 /srv/mediawiki/php-master/includes/specials/SpecialUndelete.php(845): SpecialUndelete->undelete()
#6 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(505): SpecialUndelete->execute(NULL)
#7 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(598): SpecialPage->run(NULL)
#8 /srv/mediawiki/php-master/includes/MediaWiki.php(283): SpecialPageFactory::executePath(Title, RequestContext)
#9 /srv/mediawiki/php-master/includes/MediaWiki.php(749): MediaWiki->performRequest()
#10 /srv/mediawiki/php-master/includes/MediaWiki.php(521): MediaWiki->main()
#11 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#12 /srv/mediawiki/w/index.php(3): include(string)
#13 {main}

In T122262#2538645, @Etonkovidova wrote:

Re-checked the scenario @Mattflaschen-WMF described:

Just for clarity, this is also an old board (2015-08-10). If you start the scenario from scratch now (nothing involving existing or earlier-deleted boards or wikitext pages), it shouldn't happen:

mysql> SELECT ar_id, ar_user, ar_page_id, ar_rev_id, ar_namespace, ar_title, ar_timestamp, ar_content_model FROM archive WHERE ar_namespace = 1 AND ar_title = 'ET14';
+-------+---------+------------+-----------+--------------+----------+----------------+------------------+
| ar_id | ar_user | ar_page_id | ar_rev_id | ar_namespace | ar_title | ar_timestamp   | ar_content_model |
+-------+---------+------------+-----------+--------------+----------+----------------+------------------+
| 30545 |    4841 |     131866 |    253692 |            1 | ET14     | 20150810191759 | flow-board       |
| 30546 |    4841 |     131866 |    253693 |            1 | ET14     | 20150810191759 | flow-board       |
+-------+---------+------------+-----------+--------------+----------+----------------+------------------+
2 rows in set (0.00 sec)

In T122262#2538083, @Mattflaschen-WMF wrote:

Per my discussion with @Catrope today, moving this into QA. If we decide to later, we can write the maintenance script as a separate task.

Looks like this is also needed/wanted for T148057: Fix user talk pages already in inconsistent state due to to T138310.