Page MenuHomePhabricator

Fatal IncompleteRevisionException when restoring an article (undelete page)
Closed, ResolvedPublicPRODUCTION ERROR

Description

We get the following internal error when trying to restore a deleted article:

[XKJHAgpAIDAAAEflR2MAAABC] 2019-04-01 17:14:42: Fataler Ausnahmefehler des Typs „MediaWiki\Revision\IncompleteRevisionException“

We tried to restore this article: Böotien

2019-04-01 17:14:42 [XKJHAgpAIDAAAEflR2MAAABC] mw1327 dewikivoyage 1.33.0-wmf.23 exception ERROR: [XKJHAgpAIDAAAEflR2MAAABC] /w/index.php?title=Spezial:Wiederherstellen&action=submit   MediaWiki\Revision\IncompleteRevisionException from line 427 of /srv/mediawiki/php-1.33.0-wmf.23/includes/Revision/RevisionStore.php: sha1 field must not be ''! {"exception_id":"XKJHAgpAIDAAAEflR2MAAABC","exception_url":"/w/index.php?title=Spezial:Wiederherstellen&action=submit","caught_by":"mwe_handler"} 
[Exception MediaWiki\Revision\IncompleteRevisionException] (/srv/mediawiki/php-1.33.0-wmf.23/includes/Revision/RevisionStore.php:427) sha1 field must not be ''!
  #0 /srv/mediawiki/php-1.33.0-wmf.23/includes/Revision/RevisionStore.php(478): MediaWiki\Revision\RevisionStore->failOnEmpty(string, string)
  #1 /srv/mediawiki/php-1.33.0-wmf.23/includes/page/PageArchive.php(776): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\RevisionArchiveRecord, Wikimedia\Rdbms\DatabaseMysqli)
  #2 /srv/mediawiki/php-1.33.0-wmf.23/includes/page/PageArchive.php(506): PageArchive->undeleteRevisions(array, boolean, string)
  #3 /srv/mediawiki/php-1.33.0-wmf.23/includes/specials/SpecialUndelete.php(1175): PageArchive->undelete(array, string, array, boolean, User)
  #4 /srv/mediawiki/php-1.33.0-wmf.23/includes/specials/SpecialUndelete.php(204): SpecialUndelete->undelete()
  #5 /srv/mediawiki/php-1.33.0-wmf.23/includes/specialpage/SpecialPage.php(569): SpecialUndelete->execute(NULL)
  #6 /srv/mediawiki/php-1.33.0-wmf.23/includes/specialpage/SpecialPageFactory.php(558): SpecialPage->run(NULL)
  #7 /srv/mediawiki/php-1.33.0-wmf.23/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
  #8 /srv/mediawiki/php-1.33.0-wmf.23/includes/MediaWiki.php(867): MediaWiki->performRequest()
  #9 /srv/mediawiki/php-1.33.0-wmf.23/includes/MediaWiki.php(517): MediaWiki->main()
  #10 /srv/mediawiki/php-1.33.0-wmf.23/index.php(42): MediaWiki->run()
  #11 /srv/mediawiki/w/index.php(3): include(string)
  #12 {main}

Event Timeline

Krinkle renamed this task from Error message when restoring an article to Fatal IncompleteRevisionException when restoring an article (undelete page).Apr 5 2019, 11:08 PM
Krinkle added a subscriber: Krinkle.

(Tagging steward per mw:Maintainers for initial triage.)

Some of the deleted revisions for this page have ar_sha1 set to an empty string, which is what causes this error. For this page, this only affects the revisions dated November 2012 and earlier. The ones after November 2012 do have ar_sha1 set, and I was able to undelete those successfully.

archive table rows with missing ar_sha1 fields are not common on most wikis, but for some reason dewikivoyage (39k out of 79k) and itwikivoyage (46k out of 77k) have huge numbers of them. There are 92k rows with a blank ar_sha1 across 47 wikis, but these two wikis account for 85k of them (92%). What this means in practice is that more than half of the deleted revisions on these wikis (50% on dewikivoyage, 59% on itwikivoyage) are not undeletable right now.

1arcwiki: 1
2arwiki: 1
3azwiki: 3
4bgwiki: 5
5cawiki: 1
6commonswiki: 282
7cswiki: 1
8dawiki: 2
9dewiki: 2718
10dewikiversity: 1
11dewikivoyage: 39105
12enwiki: 1684
13enwikibooks: 2
14enwikiquote: 2
15eswiki: 276
16eswiktionary: 64
17etwiki: 1
18fawiki: 1
19fiwiki: 1
20frwiki: 488
21frwikisource: 2
22hrwiki: 1
23huwiki: 14
24idwiki: 3
25itwiki: 262
26itwikivoyage: 45940
27jawiki: 51
28kowiki: 3
29kvwiki: 1
30metawiki: 3
31nlwiki: 8
32nlwiktionary: 605
33nowiki: 6
34oswiki: 1
35plwiki: 192
36ptwiki: 4
37ruwiki: 23
38ruwiktionary: 1
39slwiki: 4
40sourceswiki: 1
41srwiki: 4
42svwiki: 168
43tawiki: 1
44testwiki: 11
45trwiki: 1
46ukwiki: 1
47zhwiki: 18

The populateRevisionSha1.php maintenance script includes code that finds rows with a blank ar_sha1 and populates them, but there's currently no way to run only that step without also recomputing the SHA1 for every single revision and archive row in the entire database. I'll write a patch so that we can do this next week (this week is a no-deploys week).

Change 516576 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/core@master] populateRevisionSha1.php: Add --missing-archive-only flag

https://gerrit.wikimedia.org/r/516576

Change 516576 abandoned by Catrope:
populateRevisionSha1.php: Add --missing-archive-only flag

Reason:
Looks like I was completely off base here and we don't need this change at all, sorry.

https://gerrit.wikimedia.org/r/516576

The populateRevisionSha1.php maintenance script includes code that finds rows with a blank ar_sha1 and populates them, but there's currently no way to run only that step without also recomputing the SHA1 for every single revision and archive row in the entire database. I'll write a patch so that we can do this next week (this week is a no-deploys week).

It looks like I was wrong and the script will only repopulate rows with no SHA-1. Going to try this on testwiki first, and if that works well, I'll run it on the other wikis.

Mentioned in SAL (#wikimedia-operations) [2019-06-18T00:15:38Z] <RoanKattouw> Running populateRevisionSha1.php on testwiki for T219816

Mentioned in SAL (#wikimedia-operations) [2019-06-18T00:22:52Z] <RoanKattouw> Running populateRevisionSha1.php on dewikivoyage for T219816

The testwiki run failed to remedy any of the 11 missing ar_sha1 entries because those archive rows point to broken text rows, with the external store URL DB://cluster20/0. I suspect this is the story behind most of the wikis that only have a handful of missing SHA1s. It's now running for dewikivoyage and that does seem to be doing something.

The script finished on dewiktionary, and fixed all the rows with missing ar_sha1 values. I've undeleted the remaining revision on Böotien, and this bug should no longer affect undeletions on dewiktionary. I'll run the script for the other wikis tomorrow.

Doing the others now:

  • arcwiki
  • arwiki
  • azwiki
  • bgwiki
  • cawiki
  • commonswiki
  • cswiki
  • dawiki
  • dewiki
  • dewikiversity
  • dewikivoyage
  • enwiki
  • enwikibooks
  • enwikiquote
  • eswiki
  • eswiktionary
  • etwiki
  • fawiki
  • fiwiki
  • frwiki
  • frwikisource
  • hrwiki
  • huwiki
  • idwiki
  • itwiki
  • itwikivoyage
  • jawiki
  • kowiki
  • kvwiki
  • metawiki
  • nlwiki
  • nlwiktionary (has lots of revision and archive rows with missing content, yikes)
  • nowiki
  • oswiki
  • plwiki
  • ptwiki
  • ruwiki
  • ruwiktionary
  • slwiki
  • sourceswiki
  • srwiki
  • svwiki
  • tawiki
  • testwiki
  • trwiki
  • ukwiki
  • zhwiki
daniel added a subscriber: daniel.

tagging cpt for review/investigation

Catrope claimed this task.

Change 520707 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] SlotRecord:compute sha1 if empty.

https://gerrit.wikimedia.org/r/520707

Change 520707 merged by jenkins-bot:
[mediawiki/core@master] SlotRecord:compute sha1 if empty.

https://gerrit.wikimedia.org/r/520707

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM