Page MenuHomePhabricator

Unable to view some pages due to fatal RevisionAccessException: "Failed to load data blob from tt"
Open, NormalPublic

Description

Examples at which this is consistently reproducible:

  • https://os.wikipedia.org/wiki/1888
  • https://de.wikipedia.org/w/index.php?oldid=5780804

Error

Request ID: W7KoHArAIEAAALCQVJQAAABU

message
Failed to load data blob from tt:106822: Failed to load blob from address tt:106822
exception
MediaWiki\Storage\RevisionAccessException:
 Failed to load data blob from tt:106822

#0 /srv/mediawiki/php-1.32.0-wmf.23/includes/Storage/RevisionStore.php(1350): MediaWiki\Storage\RevisionStore->loadSlotContent(MediaWiki\Storage\SlotRecord, NULL, NULL, NULL, integer)
#1 [internal function]: Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2(MediaWiki\Storage\SlotRecord)
#2 /srv/mediawiki/php-1.32.0-wmf.23/includes/Storage/SlotRecord.php(306): call_user_func(Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2;519, MediaWiki\Storage\SlotRecord)
#3 /srv/mediawiki/php-1.32.0-wmf.23/includes/Storage/RevisionRecord.php(174): MediaWiki\Storage\SlotRecord->getContent()
#4 /srv/mediawiki/php-1.32.0-wmf.23/includes/Revision/RenderedRevision.php(205): MediaWiki\Storage\RevisionRecord->getContent(string, integer, NULL)
#5 /srv/mediawiki/php-1.32.0-wmf.23/includes/Revision/RevisionRenderer.php(169): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#6 /srv/mediawiki/php-1.32.0-wmf.23/includes/Revision/RevisionRenderer.php(122): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#7 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2(MediaWiki\Revision\RenderedRevision, array)
#8 /srv/mediawiki/php-1.32.0-wmf.23/includes/Revision/RenderedRevision.php(176): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2;723, MediaWiki\Revision\RenderedRevision, array)
#9 /srv/mediawiki/php-1.32.0-wmf.23/includes/poolcounter/PoolWorkArticleView.php(188): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#10 /srv/mediawiki/php-1.32.0-wmf.23/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#11 /srv/mediawiki/php-1.32.0-wmf.23/includes/page/Article.php(771): PoolCounterWork->execute()
#12 /srv/mediawiki/php-1.32.0-wmf.23/includes/actions/ViewAction.php(68): Article->view()
#13 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(501): ViewAction->show()
#14 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
#15 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(868): MediaWiki->performRequest()
#16 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(525): MediaWiki->main()
#17 /srv/mediawiki/php-1.32.0-wmf.23/index.php(42): MediaWiki->run()
previous.exception
MediaWiki\Storage\BlobAccessException: 
 Failed to load blob from address tt:106822

#0 /srv/mediawiki/php-1.32.0-wmf.23/includes/Storage/RevisionStore.php(1433): MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#1 /srv/mediawiki/php-1.32.0-wmf.23/includes/Storage/RevisionStore.php(1350): MediaWiki\Storage\RevisionStore->loadSlotContent(MediaWiki\Storage\SlotRecord, NULL, NULL, NULL, integer)
#2 [internal function]: Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2(MediaWiki\Storage\SlotRecord)
#3 /srv/mediawiki/php-1.32.0-wmf.23/includes/Storage/SlotRecord.php(306): call_user_func(Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2;519, MediaWiki\Storage\SlotRecord)
#4 /srv/mediawiki/php-1.32.0-wmf.23/includes/Storage/RevisionRecord.php(174): MediaWiki\Storage\SlotRecord->getContent()
#5 /srv/mediawiki/php-1.32.0-wmf.23/includes/Revision/RenderedRevision.php(205): MediaWiki\Storage\RevisionRecord->getContent(string, integer, NULL)
#6 /srv/mediawiki/php-1.32.0-wmf.23/includes/Revision/RevisionRenderer.php(169): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#7 /srv/mediawiki/php-1.32.0-wmf.23/includes/Revision/RevisionRenderer.php(122): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#8 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2(MediaWiki\Revision\RenderedRevision, array)
#9 /srv/mediawiki/php-1.32.0-wmf.23/includes/Revision/RenderedRevision.php(176): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2;723, MediaWiki\Revision\RenderedRevision, array)
#10 /srv/mediawiki/php-1.32.0-wmf.23/includes/poolcounter/PoolWorkArticleView.php(188): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#11 /srv/mediawiki/php-1.32.0-wmf.23/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#12 /srv/mediawiki/php-1.32.0-wmf.23/includes/page/Article.php(771): PoolCounterWork->execute()
#13 /srv/mediawiki/php-1.32.0-wmf.23/includes/actions/ViewAction.php(68): Article->view()
#14 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(501): ViewAction->show()
#15 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
#16 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(868): MediaWiki->performRequest()
#17 /srv/mediawiki/php-1.32.0-wmf.23/includes/MediaWiki.php(525): MediaWiki->main()
#18 /srv/mediawiki/php-1.32.0-wmf.23/index.php(42): MediaWiki->run()

Notes

The fatal RevisionAccessException exception happens during the building of a page view in MediaWiki for both unregistered users and logged-in users. Triaging as at least High priority given it exposes a fatal error on a public GET url, which is a risk for false alarms and increased load.

The error seems to be nested behind a related problem that is handled, but then re-throws. The above previous.exception happens first, and then re-throws as RevisionAccessException, which is is currently unhandled by any UI layer and results in the top-level error handling reporting is as an application problem.

Seems to be a regression that started with 1.32.0-wmf.19. (I didn't find matches among messages from previous versions). Although it is much more frequent on wmf.22 than on wmf.20 and wmf.19.

Just speculating as to why it is growing, but:

  • It might mean that the same pages are receiving more traffic than before, or;
  • It could be that the same problem is slowly affecting a growing number of pages, or;
  • It could be that another bug was introduced last week that made it affect more pages.

Consistently reproducible at https://os.wikipedia.org/wiki/1888.

See also:

Event Timeline

Krinkle triaged this task as High priority.Oct 1 2018, 10:44 PM
Krinkle created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 1 2018, 10:44 PM
Krinkle updated the task description. (Show Details)Oct 1 2018, 11:06 PM
Anomie added a subscriber: Anomie.Oct 4 2018, 4:26 PM

Regarding the specific example,

wikiadmin@10.64.16.191(oswiki)> select * from text where old_id=106822;
+--------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
| old_id | old_namespace | old_title | old_text         | old_comment | old_user | old_user_text | old_timestamp | old_minor_edit | old_flags           | inverse_timestamp |
+--------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
| 106822 |               |           | DB://cluster20/0 |             |          |               |               |                | utf-8,gzip,external |                   |
+--------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+

That "DB://cluster20/0" is definitely not right, the number after the slash is supposed to be a blob_id in the ExternalStore blobs table and should always be positive.

wikiadmin@10.64.32.185(oswiki)> select * from blobs_cluster20 where blob_id=0;
Empty set (0.00 sec)

I spot checked several other instances of the error, on multiple wikis, and interestingly they all have the same old_text and all are on revisions dated 2009-03-09, around 21:00-23:00. https://wikitech.wikimedia.org/wiki/Server_admin_log/Archive_13#March%209 mentions some issues with ExternalStore at about that time (and in particular cluster20 is mentioned on March 10 at 00:48).

I suspect it's generally another issue where MCR code throws/logs exceptions on database inconsistencies where pre-MCR code just displayed an error page to the user without logging anything.

CCicalese_WMF lowered the priority of this task from High to Normal.Nov 26 2018, 4:21 PM
CCicalese_WMF added a subscriber: CCicalese_WMF.

Will need to add better user level error handling. The database corruption should also be addressed.

Krinkle updated the task description. (Show Details)

https://de.wikipedia.org/w/index.php?oldid=5780804 reported last year:

Interner Fehler [W9n5NApAMEoAAJqHzzYAAABM] 2018-10-31 18:49:24: Fataler Ausnahmefehler des Typs „MediaWiki\Revision\RevisionAccessException“

Caught on BETA, but might help here.

  • I understood that External Storage is used to store text blobs of productive WMF wikis.
  • Perhaps some entries in mapping tables are corrupt.
  • The following error report would result from a problem in the other direction.
[XTHCuKwQBHcAADFkv-IAAAAK] /w/index.php?title=Wikipedia:Spielwiese&action=submit MediaWiki\Storage\BlobAccessException from line 261 of /srv/mediawiki/php-master/includes/Storage/SqlBlobStore.php: Unable to store text to external storage

Backtrace:

  • 0 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(950): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
  • 1 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(686): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
  • 2 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(621): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
  • 3 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(519): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
  • 4 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(3952): Closure$MediaWiki\Revision\RevisionStore::insertRevisionOn(Wikimedia\Rdbms\DatabaseMysqli, string)
  • 5 /srv/mediawiki/php-master/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure$MediaWiki\Revision\RevisionStore::insertRevisionOn;3293)
  • 6 /srv/mediawiki/php-master/includes/libs/rdbms/database/DBConnRef.php(645): Wikimedia\Rdbms\DBConnRef->__call(string, array)
  • 7 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(521): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure$MediaWiki\Revision\RevisionStore::insertRevisionOn;3293)
  • 8 /srv/mediawiki/php-master/includes/Storage/PageUpdater.php(985): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
  • 9 /srv/mediawiki/php-master/includes/Storage/PageUpdater.php(756): MediaWiki\Storage\PageUpdater->doModify(CommentStoreComment, User, integer)
  • 10 /srv/mediawiki/php-master/includes/page/WikiPage.php(1921): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
  • 11 /srv/mediawiki/php-master/includes/EditPage.php(2316): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
  • 12 /srv/mediawiki/php-master/includes/EditPage.php(1596): EditPage->internalAttemptSave(array, boolean)
  • 13 /srv/mediawiki/php-master/includes/EditPage.php(681): EditPage->attemptSave(array)
  • 14 /srv/mediawiki/php-master/includes/actions/EditAction.php(55): EditPage->edit()
  • 15 /srv/mediawiki/php-master/includes/actions/SubmitAction.php(38): EditAction->show()
  • 16 /srv/mediawiki/php-master/includes/MediaWiki.php(499): SubmitAction->show()
  • 17 /srv/mediawiki/php-master/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
  • 18 /srv/mediawiki/php-master/includes/MediaWiki.php(884): MediaWiki->performRequest()
  • 19 /srv/mediawiki/php-master/includes/MediaWiki.php(515): MediaWiki->main()
  • 20 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
  • 21 /srv/mediawiki/w/index.php(3): include(string)
  • 22 {main}
WDoranWMF moved this task from MCR to mop on the Core Platform Team board.Jul 26 2019, 6:39 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Wed, Aug 28, 11:08 PM