Page MenuHomePhabricator
Create Task
Maniphest T219816

Fatal IncompleteRevisionException when restoring an article (undelete page)
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

We get the following internal error when trying to restore a deleted article:

[XKJHAgpAIDAAAEflR2MAAABC] 2019-04-01 17:14:42: Fataler Ausnahmefehler des Typs „MediaWiki\Revision\IncompleteRevisionException“

We tried to restore this article: Böotien

2019-04-01 17:14:42 [XKJHAgpAIDAAAEflR2MAAABC] mw1327 dewikivoyage 1.33.0-wmf.23 exception ERROR: [XKJHAgpAIDAAAEflR2MAAABC] /w/index.php?title=Spezial:Wiederherstellen&action=submit   MediaWiki\Revision\IncompleteRevisionException from line 427 of /srv/mediawiki/php-1.33.0-wmf.23/includes/Revision/RevisionStore.php: sha1 field must not be ''! {"exception_id":"XKJHAgpAIDAAAEflR2MAAABC","exception_url":"/w/index.php?title=Spezial:Wiederherstellen&action=submit","caught_by":"mwe_handler"} 
[Exception MediaWiki\Revision\IncompleteRevisionException] (/srv/mediawiki/php-1.33.0-wmf.23/includes/Revision/RevisionStore.php:427) sha1 field must not be ''!
  #0 /srv/mediawiki/php-1.33.0-wmf.23/includes/Revision/RevisionStore.php(478): MediaWiki\Revision\RevisionStore->failOnEmpty(string, string)
  #1 /srv/mediawiki/php-1.33.0-wmf.23/includes/page/PageArchive.php(776): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\RevisionArchiveRecord, Wikimedia\Rdbms\DatabaseMysqli)
  #2 /srv/mediawiki/php-1.33.0-wmf.23/includes/page/PageArchive.php(506): PageArchive->undeleteRevisions(array, boolean, string)
  #3 /srv/mediawiki/php-1.33.0-wmf.23/includes/specials/SpecialUndelete.php(1175): PageArchive->undelete(array, string, array, boolean, User)
  #4 /srv/mediawiki/php-1.33.0-wmf.23/includes/specials/SpecialUndelete.php(204): SpecialUndelete->undelete()
  #5 /srv/mediawiki/php-1.33.0-wmf.23/includes/specialpage/SpecialPage.php(569): SpecialUndelete->execute(NULL)
  #6 /srv/mediawiki/php-1.33.0-wmf.23/includes/specialpage/SpecialPageFactory.php(558): SpecialPage->run(NULL)
  #7 /srv/mediawiki/php-1.33.0-wmf.23/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
  #8 /srv/mediawiki/php-1.33.0-wmf.23/includes/MediaWiki.php(867): MediaWiki->performRequest()
  #9 /srv/mediawiki/php-1.33.0-wmf.23/includes/MediaWiki.php(517): MediaWiki->main()
  #10 /srv/mediawiki/php-1.33.0-wmf.23/index.php(42): MediaWiki->run()
  #11 /srv/mediawiki/w/index.php(3): include(string)
  #12 {main}

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
SlotRecord:compute sha1 if empty.mediawiki/coremaster+15 -2
populateRevisionSha1.php: Add --missing-archive-only flagmediawiki/coremaster+14 -5
Customize query in gerrit

Related Objects

Event Timeline

DerFussi created this task.Apr 1 2019, 5:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 1 2019, 5:19 PM
Reedy added projects: MediaWiki-Page-deletion, Wikimedia-production-error.Apr 1 2019, 5:21 PM
Reedy updated the task description. (Show Details)
Krinkle renamed this task from Error message when restoring an article to Fatal IncompleteRevisionException when restoring an article (undelete page).Apr 5 2019, 11:08 PM
Krinkle moved this task from Untriaged to Mar 2019 / 1.33wmf20-23 on the Wikimedia-production-error board.
Krinkle added a project: Growth-Team.Apr 19 2019, 7:40 PM
Krinkle subscribed.

(Tagging steward per mw:Maintainers for initial triage.)

Krinkle mentioned this in Blog Post: Production Excellence #9: March 2019.Apr 21 2019, 6:51 PM
kostajh moved this task from Inbox to Needs Discussion on the Growth-Team board.Apr 22 2019, 2:45 PM
JTannerWMF added subscribers: Catrope, JTannerWMF.Apr 30 2019, 6:25 PM

@Catrope will investigate

Krinkle mentioned this in Blog Post: Production Excellence #10: April 2019.May 31 2019, 7:21 PM
Catrope added a comment.Jun 11 2019, 11:32 PM

Some of the deleted revisions for this page have ar_sha1 set to an empty string, which is what causes this error. For this page, this only affects the revisions dated November 2012 and earlier. The ones after November 2012 do have ar_sha1 set, and I was able to undelete those successfully.

archive table rows with missing ar_sha1 fields are not common on most wikis, but for some reason dewikivoyage (39k out of 79k) and itwikivoyage (46k out of 77k) have huge numbers of them. There are 92k rows with a blank ar_sha1 across 47 wikis, but these two wikis account for 85k of them (92%). What this means in practice is that more than half of the deleted revisions on these wikis (50% on dewikivoyage, 59% on itwikivoyage) are not undeletable right now.

P8607 Number of archive rows with blank ar_sha1 on each wiki
1arcwiki: 1
2arwiki: 1
3azwiki: 3
4bgwiki: 5
5cawiki: 1
6commonswiki: 282
7cswiki: 1
8dawiki: 2
9dewiki: 2718
10dewikiversity: 1
11dewikivoyage: 39105
12enwiki: 1684
13enwikibooks: 2
14enwikiquote: 2
15eswiki: 276
16eswiktionary: 64
17etwiki: 1
18fawiki: 1
19fiwiki: 1
20frwiki: 488
21frwikisource: 2
22hrwiki: 1
23huwiki: 14
24idwiki: 3
25itwiki: 262
26itwikivoyage: 45940
27jawiki: 51
28kowiki: 3
29kvwiki: 1
30metawiki: 3
31nlwiki: 8
32nlwiktionary: 605
33nowiki: 6
34oswiki: 1
35plwiki: 192
36ptwiki: 4
37ruwiki: 23
38ruwiktionary: 1
39slwiki: 4
40sourceswiki: 1
41srwiki: 4
42svwiki: 168
43tawiki: 1
44testwiki: 11
45trwiki: 1
46ukwiki: 1
47zhwiki: 18

The populateRevisionSha1.php maintenance script includes code that finds rows with a blank ar_sha1 and populates them, but there's currently no way to run only that step without also recomputing the SHA1 for every single revision and archive row in the entire database. I'll write a patch so that we can do this next week (this week is a no-deploys week).

gerritbot added a comment.Jun 12 2019, 12:25 AM

Change 516576 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/core@master] populateRevisionSha1.php: Add --missing-archive-only flag

https://gerrit.wikimedia.org/r/516576

gerritbot added a project: Patch-For-Review.Jun 12 2019, 12:25 AM
gerritbot added a comment.Jun 17 2019, 6:55 PM

Change 516576 abandoned by Catrope:
populateRevisionSha1.php: Add --missing-archive-only flag

Reason:
Looks like I was completely off base here and we don't need this change at all, sorry.

https://gerrit.wikimedia.org/r/516576

Catrope added a comment.Jun 17 2019, 11:21 PM
In T219816#5251887, @Catrope wrote:

The populateRevisionSha1.php maintenance script includes code that finds rows with a blank ar_sha1 and populates them, but there's currently no way to run only that step without also recomputing the SHA1 for every single revision and archive row in the entire database. I'll write a patch so that we can do this next week (this week is a no-deploys week).

It looks like I was wrong and the script will only repopulate rows with no SHA-1. Going to try this on testwiki first, and if that works well, I'll run it on the other wikis.

Stashbot added a comment.Jun 18 2019, 12:15 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-18T00:15:38Z] <RoanKattouw> Running populateRevisionSha1.php on testwiki for T219816

Stashbot added a comment.Jun 18 2019, 12:22 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-18T00:22:52Z] <RoanKattouw> Running populateRevisionSha1.php on dewikivoyage for T219816

Catrope added a comment.Jun 18 2019, 12:24 AM

The testwiki run failed to remedy any of the 11 missing ar_sha1 entries because those archive rows point to broken text rows, with the external store URL DB://cluster20/0. I suspect this is the story behind most of the wikis that only have a handful of missing SHA1s. It's now running for dewikivoyage and that does seem to be doing something.

Catrope added a comment.Jun 18 2019, 2:49 AM

The script finished on dewiktionary, and fixed all the rows with missing ar_sha1 values. I've undeleted the remaining revision on Böotien, and this bug should no longer affect undeletions on dewiktionary. I'll run the script for the other wikis tomorrow.

Catrope added a comment.EditedJun 18 2019, 10:11 PM

Doing the others now:

  • arcwiki
  • arwiki
  • azwiki
  • bgwiki
  • cawiki
  • commonswiki
  • cswiki
  • dawiki
  • dewiki
  • dewikiversity
  • dewikivoyage
  • enwiki
  • enwikibooks
  • enwikiquote
  • eswiki
  • eswiktionary
  • etwiki
  • fawiki
  • fiwiki
  • frwiki
  • frwikisource
  • hrwiki
  • huwiki
  • idwiki
  • itwiki
  • itwikivoyage
  • jawiki
  • kowiki
  • kvwiki
  • metawiki
  • nlwiki
  • nlwiktionary (has lots of revision and archive rows with missing content, yikes)
  • nowiki
  • oswiki
  • plwiki
  • ptwiki
  • ruwiki
  • ruwiktionary
  • slwiki
  • sourceswiki
  • srwiki
  • svwiki
  • tawiki
  • testwiki
  • trwiki
  • ukwiki
  • zhwiki
Shizhao added a project: Chinese-Sites.Jun 19 2019, 3:35 AM
Krinkle removed a project: Patch-For-Review.Jun 24 2019, 11:20 PM
daniel added a project: Platform Engineering.Jun 26 2019, 8:23 PM
daniel subscribed.

tagging cpt for review/investigation

Catrope mentioned this in T217829: Update populateRevisionSha1.php to run over the content table.Jun 26 2019, 8:27 PM
Catrope closed this task as Resolved.Jun 28 2019, 8:59 PM
Catrope claimed this task.
Shizhao moved this task from Backlog to Closed on the Chinese-Sites board.Jul 1 2019, 3:51 AM
gerritbot added a comment.Jul 4 2019, 8:56 AM

Change 520707 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] SlotRecord:compute sha1 if empty.

https://gerrit.wikimedia.org/r/520707

gerritbot added a project: Patch-For-Review.Jul 4 2019, 8:56 AM
gerritbot added a comment.Jul 5 2019, 6:56 AM

Change 520707 merged by jenkins-bot:
[mediawiki/core@master] SlotRecord:compute sha1 if empty.

https://gerrit.wikimedia.org/r/520707

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.13; 2019-07-09).Jul 5 2019, 7:00 AM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM
tstarling mentioned this in T281325: Text rows containing DB://cluster20/0, causing RevisionAccessException when the affected pages are viewed.Apr 28 2021, 12:20 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits